arXiv Computer Vision
Aug 31

Spectral Query-Key Product Weight Steering for Training-Free VLM Hallucination Mitigation

The paper introduces QK Product Steering, a data‑free, training‑free method that edits the query‑key product in vision‑language models to reduce object hallucination. By suppressing a few dominant singular modes in selected middle layers and mapping the edited product back to query weights, the approach lowers hallucination rates without affecting inference cost. Experiments on three GQA‑based VLMs show a 4.0% average reduction in CHAIR$_s$, with the effect localized to symmetric mutual‑attention channels.

By Karn Tiwari, Varnith Chordia, Prathosh A P