混合架构中循环层抢先学会检索，BLD 冻结训练法改善注意力层
"Competing Learning in Hybrid Architectures"
摘要
论文发现混合架构中循环层学习检索更快，抑制了注意力层掌握同一能力，为此提出 Better Layer Division（BLD）：训练早期暂时冻结循环层，迫使注意力层先学会检索，再让两部分共同训练。在 180M-1B 参数模型上 BLD 一致提升检索表现，750M 模型 RULER 准确率从 26.53% 升至 32.82%，长上下文困惑度从 12.68 降至 8.08。
摘要由模型根据原文撰写，以原文为准。
正文 · 原文
"Competing Learning in Hybrid Architectures"
Hybrid models combine recurrent layers with full attention, but this paper finds that recurrent layers learn retrieval faster, preventing attention layers from learning the same capability effectively.
So they proposed Better Layer Division (BLD) which temporarily freezes recurrent layers early in training, forcing attention layers to learn retrieval first before allowing both components to train together.
Across 180M-1B models, BLD consistently improves retrieval performance. At 750M, RULER accuracy increases from 26.53% to 32.82%, while long-context perplexity drops from 12.68 to 8.08.
来源：alphaXiv · x.com
引用原文
askalphaxiv. "Competing Learning in Hybrid Architectures"[EB/OL]. (2026-10-31)[访问日期]. https://x.com/askalphaxiv/status/2116469477852508204.