Latent Feedback 可省 48% 训练算力

"Latent Feedback Is a Compute and Data Amplifier"

Repeatedly training LLMs on the same data gives diminishing returns, but latent feedback provides a more efficient way to use that compute.

This paper revisits Latent Feedback Transformers, which feed each token's final hidden state into the next token's input, letting the model to reuse information across consecutive positions.

Across models from 60M to 778M params, latent feedback saves up to 48% of training compute or 41% of unique training data in measured comparisons at matched validation loss. A new scaling law also predicts performance on a held-out 778M model within 0.3%.

https://www.alphaxiv.org/abs/2611.latent-feedback-transformers-scaling-law