引用Dr. Cheems Wang 🏡@AlbertW24045555
(1/5) New preprint: TreeOPSD, critic-free RL for long-horizon LLM agents 🌳
Long-horizon agents receive a single outcome reward after many turns, and each rollout batch feeds several sequential mini-batch updates while the policy drifts from the one that sampled it.
TreeOPSD backs terminal rewards up a rollout tree and splits each update into two steps: construct a target for every turn, then match the current policy to it with forward KL.
This leads to:
• closed-form, bounded turn targets that commute with tree backup, so shared prefixes are trained once with no loss of credit
• the exact tree policy gradient on the first mini-batch, and updates that stop at the target and reverse after overshoot
• consistent gains over GRPO, GSPO and a matched tree baseline, largest on the longest-horizon tasks
On Deep Research, TreeOPSD lifts Qwen3-30B-A3B-Thinking-2507 from 16.23 (GRPO) to 21.20 average accuracy.
Title: Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning
Paper: https://arxiv.org/pdf/2610.23181
Together with @TencentHunyuan
#LLM #RLVR #AIAgents