arXiv AI
VIF-Bench: Evaluating Visual Instruction Following in Multi-Reference Image Generation
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computer Vision
Sep 1
Instruction Distillation: Text Instructions as Visual Examples
arXiv:2608.28696v1 Announce Type: new Abstract: Visual in-context learning (ICL) with multimodal large language models (MLLMs) is effective for fine-grained visual classification, but each retrieved...
arXiv AI
Sep 7
Cross-Task Generalization Between Understanding and Generation in Unified Vision-Language Models: A Controlled Study
The study investigates how unified vision‑language models (VLMs) can simultaneously support visual understanding and generation. Using controlled benchmarks (SmartWatch and modified CelebA) that pair VQA, captioning, and text‑to‑image tasks, the authors evaluate several LLM‑based architectures built on SigLIP and VQ‑VAE visual spaces. Results show that mixed training can improve both understanding and generation, but the gains depend on how well the visual input and output spaces are aligned; misaligned or distorted visual spaces can weaken or reverse these benefits. The paper also demonstrates that balancing data across tasks and controlling attribute frequencies can help recover underrepresented visual concepts, and that the transfer is driven more by the base language model’s learned relationships than by visual adapters.
arXiv Machine Learning
Jun 3
Visual Instruction Tuning Aligns Modalities through Abstraction
arXiv:2606. 03871v1 Announce Type: cross Abstract: Visual instruction tuning effectively adapts a pre-trained Large Language Model (LLM) to process image information alongside text.
arXiv AI
Jul 9
T2T-VICL: Cross-Task Visual In-Context Learning via Implicit Text-Driven VLMs
arXiv:2511. 16107v3 Announce Type: replace-cross Abstract: Visual in-context learning (VICL) solves visual tasks by conditioning on a few input-output demonstrations without any model training.
arXiv Computer Vision
Aug 28
Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Information
The paper introduces Visual Retrieval Heads (VRHs), a small fraction of attention heads in vision‑language models that are causally responsible for grounding text descriptions to image regions. By recasting head‑scoring methods and evaluating across eleven VLMs and five benchmarks, the authors show that masking the top 20 VRHs can drop grounding accuracy by up to 80 percentage points, while random masking has little effect. VRHs generalize across various visual reference tasks, preserve output format while corrupting localization, and transfer causally across models sharing an LLM backbone.
arXiv Computer Vision
Aug 25
When Does Visual Generation Help Visual Understanding in Unified Multimodal Models?
arXiv:2608.22174v1 Announce Type: new Abstract: Unified multimodal models (UMMs) can perform both understanding and generation, raising a central question: can visual generation improve understanding...