Symmetric Behavior Regularized Policy Optimization

Behavior Regularized Policy Optimization (BRPO) leverages asymmetric divergence regularization to mitigate distribution shift in offline reinforcement learning. This paper is the first to study the open question of symmetric BRPO. Using didactic examples, we show that symmetric regularization can outperform asymmetric regularization in addressing one-sided bias, near-boundary policy updates, and projection geometry consistency. However, symmetric divergences do not fit BRPO naturally: they do not permit a closed-form solution when used as regularizers, and can lead to numerical instability when used as optimization objectives. We first introduce a universal BRPO framework using an infinite series of Pearson-Vajda divergences to represent any f-divergence, which includes both symmetric and asymmetric divergences. We use a finite-series approximation to obtain the following results for symmetric BRPO: (1) a closed-form optimal policy expression; (2) a numerically stable optimization surrogate; and (3) a tight upper bound on the approximation quality. On the D4RL benchmark and in didactic examples, we show that the proposed method achieves consistently strong results and is robust to the number of terms in the approximation.