"Multi-Token Prediction Done Right"
摘要
论文发现多Token预测（MTP）约80%的训练梯度来自干扰学习的噪声成分，提出Soft-MTP将其去除，改为让MTP头匹配模型自身的下一Token预测，仅需一行代码、无额外参数或可测训练开销。在167M-1.2B模型上，Soft-MTP提升了困惑度、下游准确率和投机解码速度；1.2B模型HumanEval Pass@64从40.85%升至43.90%，投机解码比标准MTP快约1.5%。
摘要由模型根据原文撰写，以原文为准。
正文 · 原文
"Multi-Token Prediction Done Right"
Multi-token prediction (MTP) helps LLMs predict a few future tokens, but this paper finds that around 80% of its training gradient comes from a noisy component that interferes with learning.
They introduce Soft-MTP, which removes this noisy gradient and instead trains MTP heads to match the model's own next-token predictions. This change requires just one line of code, with no additional parameters or measurable training overhead.
Across 167M-1.2B models, Soft-MTP improves perplexity, downstream accuracy, and speculative decoding speed.
At 1.2B, HumanEval Pass@64 improves from 40.85% to 43.90%, while speculative decoding becomes roughly 1.5% faster than standard MTP
来源：alphaXiv · x.com
引用原文
askalphaxiv. "Multi-Token Prediction Done Right"[EB/OL]. (2026-11-03)[访问日期]. https://x.com/askalphaxiv/status/2117522225691873623.