强化学习后训练中的覆盖坍缩与参考重置
"A Theory of Policy Optimization in Post-Training: Coverage Collapse and Reference Resets"
摘要
论文提出“覆盖坍缩”现象：策略优化会逐步降低模型生成最优回答的概率，即使平均奖励仍在上升，最优回答也越来越难被重新发现。作者提出近似策略镜像下降（APMD），反复将 KL 参考模型重置为最新模型而非固定原始基座模型，从而在受控改进的同时持续逼近更优解。参考重置也为简化版 PPO、GRPO 和 Kimi 风格算法提供了更强的收敛保证。
摘要由模型根据原文撰写，以原文为准。
正文 · 原文
"A Theory of Policy Optimization in Post-Training: Coverage Collapse and Reference Resets"
RL can actually make an LLM forget how to generate its best answers, even while its average reward continues improving.
This paper identifies a phenomenon called coverage collapse, where policy optimization slowly reduces the probability of generating optimal responses, making them increasingly difficult to rediscover during training.
They then proposed Approximate Policy Mirror Descent (APMD), which repeatedly resets the KL reference to the latest model instead of always using the original base model. This lets RL to make controlled improvements while continuing to move toward better solutions.
Reference resets provide stronger convergence guarantees for simplified PPO, GRPO, and Kimi-style algorithms too.
来源：alphaXiv · x.com
引用原文
askalphaxiv. "A Theory of Policy Optimization in Post-Training: Coverage Collapse and Reference Resets"[EB/OL]. (2026-11-13)[访问日期]. https://x.com/askalphaxiv/status/2121087406627508531.