删除不到0.05%权重即可让LLM学会新能力

摘要

新论文提出 Deletion Tuning：学习一个二值掩码，决定从前馈层删除哪些权重，而不学习新权重值。在 RL、SFT 和蒸馏中，该方法在最大测试规模上追平全量微调；30B 参数模型上达 58.54%，全量微调为 58.43%，适配器约小 4,000 倍。它还更好地保留原有能力，而常规微调最多损失 5.4% 的先前任务准确率。

摘要由模型根据原文撰写，以原文为准。

"The Surprising Effectiveness of Deleting Weights in LLM Post-Training"

LLMs can learn entirely new capabilities by simply deleting less than 0.05% of their pretrained weights, without modifying any of the remaining parameters...?

This new paper proposes Deletion Tuning, which learns a binary mask that determines which weights to remove from the model's feedforward layers. Unlike conventional fine-tuning or LoRA, it doesn't learn new weight values, but instead learns which existing connections to disable.

Across RL, SFT, and distillation, Deletion Tuning matches full fine-tuning at the largest scales tested.

On a 30B-parameter model, it achieves 58.54% vs 58.43% for full fine-tuning, with an adapter roughly 4,000x smaller.

It also preserves previous capabilities better, while conventional fine-tuning methods lose up to 5.4% of prior-task accuracy.

The reason this might work is because full fine-tuning tends to preserve the model's main weight structure while making changes outside its dominant components. So selectively deleting a tiny fraction of weights can produce a similar pattern of changes without disturbing the rest of the model.

https://www.alphaxiv.org/abs/2611.10866

删除不到0.05%权重即可让LLM学会新能力

摘要

正文 · 原文

引用原文