Uni-LaDiR (Unified Latent Diffusion Reasoner) is a new framework that unifies multimodal reasoning by mapping teacher reasoning steps from different modalities into a shared latent space of thought tokens. It employs a diffusion model to predict the next block of thought tokens, jointly training the encoder and reasoner with shared weights to ensure tokens are both useful and predictable. The approach achieves relative gains of 7.3% on visual reasoning benchmarks and 6.1% on robot manipulation tasks compared to the strongest baselines.
By Haoqiang Kang, Yizhe Zhang, Nikki Lijing Kuang, Yian Ma, Lianhui Qin
DocAttriBench (DAB) is a large-scale benchmark that provides fine-grained, element-level source attribution for Document Visual Question Answering (VQA). It introduces MAPPET, a Mask-based Perplexity-Derived Attribution method that uses document layout and language modeling to identify the most informative layout element for each answer. The benchmark contains 237k documents and 296k question-answer pairs with grounding annotations, and it evaluates multimodal LLMs on answer accuracy, attribution accuracy, and overall answer quality.
AgriScope is a unified pixel‑grounded multimodal framework designed for agricultural image understanding, supporting image‑level, region‑level, and pixel‑level tasks such as grounded caption generation, referring expression segmentation, and multi‑turn multimodal interaction. It incorporates biologically specialized semantic representations with dense spatial grounding through biological‑semantic encoding, dense spatial representations, and pixel decoding. The authors also introduce AgriGround, a large‑scale dataset of over 500K images and 11M instruction‑following samples, created via an automatic annotation pipeline that combines caption generation, phrase‑level grounding, segmentation mask generation, and task‑oriented instruction synthesis to provide densely grounded supervision for agricultural vision‑language learning.
The paper introduces FCA‑Guided Counterfactual (FCA‑CF) explanations for multi‑modal breast cancer diagnosis, leveraging a Formal Concept Analysis lattice as a hard structural constraint to generate counterfactuals. On the TCGA‑BRCA dataset, FCA‑CF achieves perfect validity (100% prediction flips), the lowest average feature changes (2.37), and competitive proximity (0.900), outperforming four established counterfactual methods. Ablation studies show the lattice constraint and a greedy refinement phase are key to its sparsity and validity.
VākQA is a newly introduced benchmark for Telugu spoken factoid question answering, comprising 2,001 question‑answer pairs across six domains, 2.53 hours of speech audio, bilingual transcriptions, and human‑verified reference answers. The study validates automatic evaluation methods against human judgments, finding that Gemini‑as‑a‑judge best approximates human ratings but is inconsistently strict, while open‑weight judges tend to penalize correct Telugu answers that differ in surface form. Using this validated setup, the authors benchmark proprietary and open‑weight models, highlighting challenges such as cultural specificity loss in translation, phonetic confusions from speech input, and compounded errors from cascaded ASR‑MT pipelines.
SnapPhysics is a training‑free framework that reconstructs 3D objects and estimates their physical properties—such as mass, friction, and center of gravity—from a single image. It combines instance‑level 3D reconstruction with a physics‑aware scene graph that encodes inter‑object relationships, providing structured context for vision‑language model reasoning. Experiments on 3D‑FRONT and real captured scenes show significant improvements over prior methods, reducing errors in mass estimation and enhancing scene‑level F‑Score.
The paper introduces a context‑aware neural grapheme‑to‑phoneme (G2P) system for unsegmented languages like Japanese, combining a discriminative conditional random field over a dictionary‑based word lattice with large language model‑generated training data. By generating over two million synthetic sentences, the method addresses data scarcity and achieves superior performance compared to traditional morphological analyzers and neural sequence models. On the Joyo‑Kanji‑Yomi benchmark, it attains 99.62% target word reading accuracy, 0.32% target word phoneme error rate, and 0.14% sentence phoneme error rate.
DeliveryGym is a 3D reinforcement learning environment that simulates continuous courier shifts, integrating multimodal tool interaction and persistent world dynamics to compute trajectory rewards based on simulator events. It provides feedback on resource consumption—time, energy, and money—across entire delivery trajectories, enabling agents to learn planning that balances immediate task success with long‑term constraints. The environment also adapts future training shifts to a policy’s weaknesses while keeping evaluation fixed, demonstrating that both learning from complete shifts and targeted training improve agent performance on complex delivery tasks.
Autoregressive image generation has emerged as a paradigm for multimodal AI systems due to its compatibility with transformer-based LLM serving infrastructures. However, generating thousands of visual...
MiX: Micro-Inverted-Scaling for End-to-End Low-Bit Vision-Language Model Acceleration proposes a new quantization format that inverts the traditional microscaling approach by assigning private exponents to each element and a shared mantissa. The adaptive dual-format MiX-MX inference framework maps this format to a custom accelerator, replacing multipliers with shifters. Evaluations show that 4.5-bit MiX matches or surpasses NVFP4 accuracy on multimodal benchmarks while improving area efficiency by 25% and delivering 2.3–4.5× speedup with 1.4–2.9× energy reduction compared to the Focus accelerator.
The paper investigates how adversarial patches affect Vision‑Language‑Action (VLA) policies, revealing that such patches can cause immediate action corruption and persistent state effects that linger after the patch is removed. A state‑restoration protocol is introduced to isolate these effects by removing the patch at action‑chunk boundaries and measuring recoverability within the remaining step budget. Experiments on OpenVLA-OFT with EDPA attacks show that only 36.2% of episodes recover after five chunks, whereas controls recover at 89.9% and 87.0%. A recovery adapter trained on attack‑induced states improves recovery from 7.7% to 47.4% at one‑chunk latency, but its effectiveness drops sharply with delayed intervention, underscoring the importance of timely recovery.
The paper introduces ${M}^2$Tok, a Multi-head Multi-codebook Action Tokenizer that reduces reconstruction loss in discrete action tokenization for Vision‑Language‑Action models. By decomposing latent action features into multiple heads and assigning independent codebooks to each, the tokenizer expands representational expressivity and improves policy performance. Experiments on RoboTwin, Simpler‑Env, and zero‑shot real‑world tasks show superior reconstruction fidelity and higher success rates compared to prior methods.
By Chunpu Xu, Zhixuan Liang, Yuhao Zhang, Chi-Min Chan, Jessie Wang, Yang Xiao, Mengkang Hu, Xiaokang Yang, Yao Mu
Mem2Ego introduces a vision‑language model for embodied navigation that combines global memory with egocentric visual inputs. By adaptively retrieving task‑relevant cues from a global memory module and aligning them with local perception, the framework improves spatial reasoning and decision‑making over long horizons. The method outperforms prior state‑of‑the‑art approaches on the HSSD and HM3D benchmarks and shows strong performance on a real robot.
By Lingfeng Zhang, Yuecheng Liu, Zhanguang Zhang, Matin Aghaei, Yixin Xiao, Yaochen Hu, Mohammad Ali Alomrani, David Gamaliel Arcos Bravo, Hongjian Gu, Zhiyuan Li, Yangzheng Wu, Zhanpeng Zhang, Raika Karimi, Atia Hamidizadeh, Guowei Huang, Haoping Xu, Tongtong Cao, Weichao Qiu, Xingyue Quan, Jianye Hao, Yuzheng Zhuang, Yingxue Zhang
The paper audits vision‑language models (CLIP) for gender bias using 1,500 artworks from the Metropolitan Museum of Art, focusing on zero‑shot logit differences for prompts like "masterpiece," "quality," and "influence." Unadjusted results show no significant gender effect and statistical equivalence across models, but high residual variance suggests that global zero‑shot metrics are largely noise‑dominated and may miss fine‑grained biases. The study underscores the need for multivariate confound control, equivalence testing, and provenance auditing when evaluating AI fairness in cultural heritage data.
By Manpreet Singh, Rhythm Bhatia, Rahul Joshi
The paper introduces the Mult2EMo dataset, which gathers annotations from both authors and readers on multimodal social media posts and the real‑world events that triggered them. It investigates how well readers can reconstruct the authors’ emotional experience from the post content, emphasizing the importance of both text and image modalities. The study finds that accurate emotion reconstruction is possible but remains challenging, especially when images dominate the expression and when understanding the triggering event is essential.
By Christopher Bagdon, Carina Silberer, Roman Klinger
MiRAGE is a new evaluation framework for retrieval‑augmented generation (RAG) that handles multimodal sources such as audiovisual media. It uses a claim‑centric approach with two metrics: InfoF1, which measures factuality and information coverage, and CiteF1, which measures citation support and completeness. Human evaluation shows MiRAGE aligns well with extrinsic quality judgments, and an automatic implementation outperforms three text‑centric RAG metrics (ALCE, ARGUE, RAGAS) on text while uniquely generalizing to multimodal inputs.
By Alexander Martin, William Walden, Reno Kriz, Dengjia Zhang, Kate Sanders, Eugene Yang, Chihsheng Jin, Benjamin Van Durme
The study evaluates whether vision‑language models (VLMs) can reliably assess sidewalk accessibility attributes—effective width, longitudinal slope, cross slope, and pavement condition—from pedestrian‑level images. Using sampling‑based conformal prediction on 514 images from Seoul, the authors find that calibrated models achieve nominal 90% coverage, but only effective width yields informative estimates; other attributes remain too uncertain for compliance assessment. The work also demonstrates that raw sampling dispersion is not a trustworthy uncertainty measure without calibration and releases annotated images with ground‑truth measurements.
By Seung Jae Lieu, Diego Morra, Chiara Cadoni, Wonseop Song, Martina Mazzarello, Carlo Ratti
ActiveScale is a framework that enhances active perception for robots by integrating model, data, and hardware innovations. It augments vision‑language‑action models with historical video observations and explicit camera‑pose supervision, and introduces a scalable human‑robot mid‑training recipe using 1000 hours of egocentric and robotic data. The Active‑perception Mobile‑manipulation Platform (AMP) enables single‑operator teleoperation for scalable demonstration collection, leading to improved success rates on active‑perception tasks.
By Shuai Zhou, Kaisheng Pang, Wenxuan Song, Wenjie Zhang, Xinhu Zheng, Haoang Li
arXiv:2609.18680v1 Announce Type: new
Abstract: Contextual ASR can benefit from semantic cues or from target words explicitly provided in the context. We introduce HearInContext, a Mandarin--English...
By Yifan Gao, Yao Tian, Hongbin Suo
The paper introduces a framework for collaborative memory in multi‑agent vision‑language model (VLM) systems, addressing how agents share and update visual context across distributed perception and reasoning tasks. It outlines a memory hierarchy, cross‑agent sharing protocols, and consistency mechanisms to reconcile differing interpretations and recover missing visual information. The design emphasizes preserving not only raw images or textual summaries but also the dependencies among observations, interpretations, and subsequent reasoning, thereby shaping information flow across agents.
By Huixin Zhang, Shao-Jun Xia, Di Wang, Liangxi Liu, Hainan Xiong, Zihao Wang