"Unbundling Multimodal Vision Encoding"
摘要
论文显示，多模态大模型不必单独预训练视觉编码器：从零训练视觉编码器与单独预训练效果相当，把视觉专用层移入语言模型前几层也能达到独立编码器水平。在0.6B至6.4B参数模型上，无编码器架构需约3倍训练算力才能匹配视觉理解性能，而所提in-fusion方法在报告的0.4B视觉容量设置下精度相当，首token时间最多降低13%。
摘要由模型根据原文撰写，以原文为准。
正文 · 原文
"Unbundling Multimodal Vision Encoding"
Multimodal LLMs don't need a separately pretrained vision encoder to understand images, but they still benefit from having dedicated layers for processing visual information.
This paper shows that training vision encoders from scratch works just as well as separate pretraining, while moving vision-specific layers into the language model's early layers can match a standalone encoder.
Across models from 0.6B to 6.4B parameters, encoder-free architectures require around 3x more training compute to match visual understanding performance, while the proposed in-fusion approach achieves comparable accuracy with up to 13% lower time to first token in the reported 0.4B-vision-capacity setup.
来源：alphaXiv · x.com
引用原文
askalphaxiv. "Unbundling Multimodal Vision Encoding"[EB/OL]. (2026-11-15)[访问日期]. https://x.com/askalphaxiv/status/2121884408450077153.