Soft-MTP：一行代码修正多Token预测

摘要

论文发现多Token预测（MTP）约80%的训练梯度来自干扰学习的噪声成分，提出Soft-MTP将其去除，改为让MTP头匹配模型自身的下一Token预测，仅需一行代码、无额外参数或可测训练开销。在167M-1.2B模型上，Soft-MTP提升了困惑度、下游准确率和投机解码速度；1.2B模型HumanEval Pass@64从40.85%升至43.90%，投机解码比标准MTP快约1.5%。

摘要由模型根据原文撰写，以原文为准。

"Multi-Token Prediction Done Right"

Multi-token prediction (MTP) helps LLMs predict a few future tokens, but this paper finds that around 80% of its training gradient comes from a noisy component that interferes with learning.

They introduce Soft-MTP, which removes this noisy gradient and instead trains MTP heads to match the model's own next-token predictions. This change requires just one line of code, with no additional parameters or measurable training overhead.

Across 167M-1.2B models, Soft-MTP improves perplexity, downstream accuracy, and speculative decoding speed.

At 1.2B, HumanEval Pass@64 improves from 40.85% to 43.90%, while speculative decoding becomes roughly 1.5% faster than standard MTP

https://alphaxiv.org/abs/2610.38331

Soft-MTP：一行代码修正多Token预测

摘要

正文 · 原文

引用原文