用逐层随机特征模型模拟深度神经网络训练损失尖峰
"Simulating Training Spikes of Deep Neural Networks by Layer-wise Random Feature Models"
摘要
该论文用简单二次模型说明重尾数据与噪声梯度即可触发 LLM 训练中的损失尖峰，无需依赖复杂神经网络行为。在一个 1.3B Transformer 上，其模拟可在 CPU 上每层仅用 5 秒识别易出现尖峰的训练配置。
摘要由模型根据原文撰写，以原文为准。
正文 · 原文
"Simulating Training Spikes of Deep Neural Networks by Layer-wise Random Feature Models"
The sudden loss spikes during LLM training might be more predictable than we thought.
This paper uses simple quadratic models to show how heavy-tailed data and noisy gradients can trigger these spikes, without requiring complex neural network behavior.
On a 1.3B Transformer, it identifies spike-prone training configurations using simulations that take just 5 seconds on a CPU per layer.
来源：alphaXiv · x.com
引用原文
askalphaxiv. "Simulating Training Spikes of Deep Neural Networks by Layer-wise Random Feature Models"[EB/OL]. (2026-10-24)[访问日期]. https://x.com/askalphaxiv/status/2113885250929807487.