A Unified Analysis of Importance-Corrected Momentum Actor-Critic over a Markovian Trajectory

We present a generic actor-critic method that combines a ridge-regularized pathwise least-squares temporal-difference (LSTD) critic with importance-corrected momentum (ICM) actors. The critic solves a ridge-regularized system per frame by Newton-Schulz inversion and needs no stepsize, no knowledge of the mixing time, and no rank condition. Averaging stochastic policy gradients over frames of T consecutive transitions, the ICM actors incur a frame-dependent gradient bias of order \frac{1}{T}. To our knowledge, we establish the first finite-time bounds for ICM actor-critic methods with normalized and unnormalized steps along a single Markovian trajectory. Up to a stationary-to-discounted distribution mismatch, the averaged gradient norm over K updates is bounded by {\cal O}(K^{-1/3}+\frac{1}{T}) for normalized steps and by {\cal O}(K^{-1/4}+\frac{1}{T}) for unnormalized steps. We specialize these bounds to Muon, normalized stochastic gradient descent (SGD) with momentum, SGD with momentum, and AdaNorm, and compare them in the nuclear norm. With a dimension-free stepsize, Muon confines the factor \min{m,n}^{1/2} associated with an m\times n actor parameter matrix to the noise and correction terms. The comparison also characterizes when orthogonalization guarantees a larger noiseless ascent than normalization: exactly when the gradient's nuclear rank exceeds the spectral-to-Frobenius curvature ratio.