GAVAGAI: Referential Alignment for Infant-Scale Vision-Language Models via Balanced Optimal Transport

The paper introduces GAVAGAI, a method that adds a referential alignment variable to vision‑language models by solving a small assignment problem via entropic optimal transport. By allowing words to be assigned to a special NOTHING column and enforcing a capacity constraint on regions, the approach improves word‑to‑picture accuracy and localization on the SAYCam infant video corpus. Experiments show that the alignment loss boosts performance over a standard next‑token captioning baseline, especially under high non‑reference rates.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

• llms
• multimodal
• benchmarks
• safety