Rethinking Projector Training in Multimodal LLMs

The paper investigates the projector training stage in multimodal large language models, showing that only a small fraction of data is needed and that projectors from different datasets can be interpolated with little loss. It proposes PORTAL, a training‑free initialization that uses optimal transport between Gaussian approximations of vision and language features, eliminating the need for paired image‑caption data. PORTAL matches or surpasses the standard pipeline across multiple backbones, encoders, and benchmarks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• multimodal
• benchmarks
• safety