On the Convergence of Stochastic Low-Rank Adaptation

Low-rank adaptation (LoRA) optimizes J(B,A)=\mathcal L(W_base+sBA) over two adapters B \in \mathbb{R}^{m \times r} and A \in \mathbb{R}^{r \times n} that form a low-rank update to a frozen pretrained weight matrix W_base \in \mathbb{R}^{m \times n}. The prior analysis shows LoRA-GD takes \exp{O(ε^{-2})} oracle calls to find an ε-stationary point such that |\nabla J(B,A)|\leq ε in the deterministic setting. We sharpen the analysis and show that O(ε^{-4}) full-gradient evaluations suffice for the same first-order criterion. We further study stochastic LoRA under unbiased gradient estimates and finite variance. We propose LoRA-NSGDM, which finds an ε-stationary point with O(ε^{-8}) stochastic oracle complexity. Under the additional mean-square smoothness condition, we use variance reduction strategy and propose LoRA-STORM, which improves the stochastic oracle complexity to O(ε^{-6}).