Minimax Optimal Estimator and Improved Error Rate for the MLE in Logistic Regression with Gaussian Design

We study finite-sample parameter estimation in logistic regression with Gaussian design, where the goal is to estimate \mathbfθ^\in \mathbb{R}^d with R=|\mathbfθ^|2\ge 1 from i.i.d. samples {(x_i,y_i)}{i=1}^n, x_i \sim N(0,I_d), y_i\mid x_i \sim Bernoulli((1+\exp(-x_i^\top \mathbfθ^))^{-1}). In this paper, we provide the first minimax optimal estimator, and improve on the best known finite-sample error rate for the maximum likelihood estimator (MLE). These two accomplishments are due to a minimax optimal estimator for the parameter norm R. First, we establish the minimax lower bound Ω(\sqrt{R^3/n}) for norm estimation. We then improve the best known norm estimation error rate of the MLE, i.e., O(\sqrt{R^3d/n}) from Chardon, Lerasle and Mourtada (2024), to \tilde{O}(\sqrt{R^3/n}+R^2d/n). The additional term, R^2d/n, appears to be the intrinsic bias of the MLE, as evidenced by the high-dimensional asymptotic theory of Zhao, Sur and Candes (2022) and numerical examples. We show that, however, this additional term is not information-theoretically necessary. To this end, we construct an efficient debiased norm estimator that achieves the error rate O(\sqrt{R^3/n}) and is therefore minimax optimal. Combining this with the optimal direction estimator given by the MLE, we establish the minimax optimal rate Θ(\sqrt{Rd/n}+\sqrt{R^3/n}) for estimating \mathbfθ^, as well as the improved finite-sample error rate \tilde{O}(\sqrt{Rd/n}+\sqrt{R^3/n}+R^2d/n) for the MLE. Numerical experiments demonstrate that the proposed minimax optimal estimators outperform the MLE.