论文提出 Unified Training：用单一算法统一预训练、监督微调与强化学习

"Training Foundation Models with a Single Algorithm"

LLMs are typically trained in separate stages, from pretraining and supervised fine-tuning to reinforcement learning. But this paper shows that these stages can be unified under a single training algorithm.

This paper proposes Unified Training, which treats RL as learning from reward-weighted model generations. This allows the model to continuously learn from a changing mixture of pretraining data, demonstrations, and successful RL solutions using the same likelihood-based objective.

In experiments with 124M-parameter models, unified training outperforms conventional staged training on reasoning accuracy and generalization to harder unseen problems in its best configurations, while better preserving previously learned capabilities.

https://alphaxiv.org/abs/2610.40564