Lafa 将预训练 LLM 转为并行生成流模型

摘要

Lafa 把预训练 LLM 转成连续流语言模型，通过逐步去噪嵌入并行生成多个 token，并用同一模型原有的下一 token 预测能力校验这些并行草稿，而非完全取代自回归解码。研究者用不到 30B 训练 token 将 Lafa 扩展到 8B；7B 规模下生成速度最高达同类自回归模型的 2.54 倍，准确率相近。

摘要由模型根据原文撰写，以原文为准。

"Large Flow Language Models"

What if we could turn existing LLMs into diffusionLMs that generate multiple tokens in parallel without losing accuracy?

So this paper Lafa converts pretrained LLMs into continuous flow language models that generate multiple tokens together by progressively denoising their embeddings. Instead of replacing autoregressive decoding entirely, it uses these parallel predictions as drafts, then verifies them using the same model's original next-token prediction capability.

The researchers scaled Lafa to 8B using fewer than 30B training tokens. At 7B, it achieves up to 2.54x faster generation than a comparable autoregressive model, while maintaining similar accuracy.

https://www.alphaxiv.org/abs/2610.28559

Lafa 将预训练 LLM 转为并行生成流模型

摘要

正文 · 原文

引用原文