← Back to all news
arXiv Computation and Language By Chanseul Cho, Noori Bae, Dongheon Lee, Youngmin Ro

GRIT: Gradient Masking for Visually Misgrounded Tokens in Text-Rich Reasoning

Read the original on arXiv Computation and Language →

The paper introduces GRIT, a training framework that adds visual grounding to reinforcement learning with verifiable rewards (RLVR) for vision-language models, addressing the issue of Correct-but-Visually-Misgrounded Reasoning (CVMR). CVMR occurs when models produce correct final answers while misperceiving visual information during intermediate reasoning steps, a problem that standard RLVR rewards cannot detect. By analyzing token-level attention patterns, the authors show that CVMR tokens differ from well-grounded tokens and use this insight to guide GRIT, which improves both reasoning accuracy and grounding fidelity. Additionally, the authors present GRIT-Bench, a new benchmark for evaluating text-rich reasoning across varying text lengths and reasoning depths.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Computer Vision
Oct 16

Perception or Reasoning? Diagnosing and Tackling Reinforcement Learning Bottlenecks in VLMs

The paper introduces Perception and Reasoning Attribution (PARA), a method that quantifies how much of a vision‑language model’s errors can be fixed by better perception versus stronger reasoning. PARA reveals that reinforcement learning (RL) benefits more from improving reasoning than perception, and motivates a new group‑relative attention distillation objective called GRAD. GRAD directly guides weaker rollouts toward the visual attention patterns of the best rollouts, consistently boosting performance across multiple model scales and benchmarks.

By Zeyuan Yang, Zhuohao Chen, Jiaben Chen, Yao Zhang, Maohao Shen, Chuang Gan
llmsreinforcement-learningefficiencymultimodalbenchmarks
More like this →
arXiv AI
Sep 17

Anchoring What Matters: A Dual-Level Learning Framework for Visually-Grounded Multimodal Reasoning

The paper introduces PIVOT, a dual-level learning framework designed to improve visually-grounded multimodal reasoning in large vision-language models. PIVOT employs a self‑calibrated experience replay mechanism to selectively reuse valuable visual reasoning trajectories, and a vision‑guided advantage allocation scheme that assigns extra rewards to tokens with strong visual support. Experiments on multiple benchmarks show that PIVOT enhances the multimodal reasoning performance of these models.

By Xinxin Song, Siyuan Li, Tingxiong Xiao, Jinli Suo
llmsreinforcement-learningmultimodalbenchmarks
More like this →
arXiv Computer Vision
Oct 23

RECAP: Looking Once Is Not Enough for Vision-Language Reasoning

The paper introduces RECAP, a training method that enhances vision‑language models by incorporating temporally localized reactivation of image evidence—called visual revisit—into the reinforcement learning process. Visual revisit peaks are shown to predict correctness and causally improve performance; masking them hurts accuracy more than masking non‑peaks, while adding revisit peaks to failed reasoning partially restores correct answers. RECAP applies revisit‑conditioned advantage estimation, dynamic discounting, and gated value prediction to propagate final rewards through visually important reasoning steps, achieving significant gains on HallusionBench and other benchmarks without extra rewards or inference‑time changes.

By Zhaolu Kang, Tailong Luo, Chenxin Li, Yingjie He, Zhenyu Yu, Fengyu Zhou, Jiachen Qian, Lei Wei, Shuang Chen, Jiachen Li, Eric Hanchen Jiang, Rongchao Zhang, Zhengtao Yao, Hoi Leong Lee, Guansu Wang, Kaiyue Zhou
llmsreinforcement-learningfine-tuningmultimodalbenchmarks
More like this →