DeicticVLA unifies three instruction modes—Language Instruction, Vision‑Language Instruction, and Visual Instruction—into a single text prompt and deictic mask framework, allowing a single pretrained Vision‑Language‑Action model to handle all modes. The approach uses text‑prompt completion and deictic gesture grounding, and evaluates various visual prompting methods and training strategies in simulation and real‑world tasks. Results show that two‑stage training improves deictic mask usage, and that Vision‑Language and Visual Instruction outperform Language Instruction on unseen expressions, appearance changes, and novel objects, achieving 100% success on unseen categories.
By Kango Yanagida, Tatsuya Aoki, Yuichiro Yoshikawa, Takato Horii
The paper introduces Meta-Adaptive Multimodal Jailbreaking (MAMJ), a method that jointly optimizes an attack strategy prompt and attacker weights to generate more effective jailbreaks against vision‑language models. Using an LLM‑based critique to refine the strategy and group‑level success‑rate rewards to update the weights, MAMJ achieves high attack success rates on MM‑SafetyBench, outperforming existing baselines by up to 24.1 percentage points. The learned attacker also transfers to unseen models and remains robust against typical defenses, highlighting a systemic vulnerability in current VLMs.
By Benlei Cui, Shen Pang, Yuke Wang, Xuemei Dong, Yuwen Zhai, Jingqun Tang, Haiyang Yu, Hui Xue, Longtao Huang, Haiwen Hong
Iron is a new framework for training generalist virtual agents that aligns low‑level actions with high‑level intents using a stepwise cycle‑consistent reward. It also repurposes failed trajectories through a hindsight reproduction mechanism to improve learning efficiency and task diversity. Experiments show Iron‑trained agents outperform those trained with three times more data, achieving a 25.06% relative improvement on unseen web tasks and better performance on complex tasks.
By Jiahe Ying, Wendong Bu, Kaihang Pan, Bingchen Miao, Siyu Chen, Wen Wang, Xueming Jiang, Juncheng Li, Siliang Tang
The paper introduces a Mixture of Multicenter Experts (MoME) framework that leverages diverse clinical strategies to reduce bias in medical AI without sharing data across institutions. MoME integrates specialized expertise from multiple centers, improving generalizability and adaptability of a multimodal target volume delineation model for prostate cancer radiotherapy. The model, trained with few-shot imaging and clinical notes, outperformed baselines, especially in high inter‑center variability or limited data scenarios, and allows local customization without cross‑institutional data exchange.
By Yujin Oh, Sangjoon Park, Xiang Li, Pengfei Jin, Yi Wang, Jonathan Paly, Jason Efstathiou, Annie Chan, Jun Won Kim, Hwa Kyung Byun, Ik Jae Lee, Jaeho Cho, Chan Woo Wee, Peng Shu, Peilong Wang, Caiwen Jiang, Nathan Yu, Jason Holmes, Jong Chul Ye, Quanzheng Li, Wei Liu, Woong Sub Koom, Jin Sung Kim, Kyungsang Kim
SDGBiasBench is a large-scale benchmark suite designed to evaluate and mitigate biases in vision–language models (VLMs) when reasoning about Sustainable Development Goals (SDGs). It contains 500k expert‑involved multiple‑choice questions and 50k regression tasks, allowing assessment of both decision‑level and estimation‑level bias. Experiments show that current VLMs exhibit intrinsic SDG bias, often relying on priors rather than multimodal evidence, and the proposed CADE method significantly reduces this bias, improving accuracy and reducing mean absolute error.
By Zihang Lin, Huaiyuan Qin, Muli Yang, Hongyuan Zhu
SemEnrich introduces a self‑supervised method to enrich radiology reports by clustering sentences semantically and adding positive or neutral observations from different clusters. The enriched data consistently improves supervised fine‑tuning across multiple vision‑language metrics, with gains ranging from 3% to 7.5% on COMET, Bert, Sentence Bleu, CheXbert‑F1, and RadGraph‑F1. The authors also demonstrate that incorporating cluster information into the reward design for GRPO training yields additional performance boosts.
By Halil Ibrahim Gulluk, Olivier Gevaert
arXiv:2608.27885v1 Announce Type: new
Abstract: Multimodality translation (e.g., text-to-image) is a core generative AI task. However, existing approaches (1) follow generative paths that do not dire...
By Gabe Guo, Elon Litman, Thanawat Sornwanee, Jose Blanchet, Stefano Ermon
arXiv:2608.27578v1 Announce Type: cross
Abstract: Training effective foundation models requires massive and organized datasets, yet scientific domains such as nuclear fusion present unique challenges...
By Nathaniel Chen, Kouroche Bouchiat, Peter Steiner, Azarakhsh Jalalvand, SangKyeun Kim, Egemen Kolemen
arXiv:2608.27505v1 Announce Type: new
Abstract: Reinforcement learning from human feedback (RLHF) has become the dominant paradigm for aligning large language models (LLMs) with human preferences. Ho...
By Zifei Shan, Fangning Shao
arXiv:2608.27881v1 Announce Type: new
Abstract: Recently, many streaming video understanding methods have been proposed by constructing an external memory to store historical data for computational r...
By Yuxin Liu, Peiqin Zhuang, Yali Wang
arXiv:2608. 28389v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) augments LLMs with external documents, but public or user-editable sources expose RAG systems to data poisoning: attackers can inject malicious documents to steer outputs toward targeted answers.
By Jaewon Jung, Haizhong Zheng, Hongsun Jang, Jaeyong Song, Beidi Chen, Jinho Lee
The paper introduces a data‑free learning objective called relative trajectory balance for training diffusion models to sample from a posterior defined by a diffusion prior and an arbitrary black‑box constraint or likelihood. It proves asymptotic correctness of this objective and demonstrates its use across vision, language, and multimodal tasks, including classifier guidance, language infilling, and text‑to‑image generation. Additionally, the method is applied to continuous control with a score‑based behavior prior, achieving state‑of‑the‑art results in offline reinforcement learning.
By Siddarth Venkatraman, Moksh Jain, Luca Scimeca, Minsu Kim, Marcin Sendera, Mohsin Hasan, Luke Rowe, Sarthak Mittal, Pablo Lemos, Emmanuel Bengio, Alexandre Adam, Jarrid Rector-Brooks, Yoshua Bengio, Glen Berseth, Esmeralda S. Whitammer
OmniFusion is an end‑to‑end multilingual multimodal translation system that fuses a pretrained multimodal foundation model (Omni 2.5‑7B) with a translation large language model (SeedX PPO‑7B). By connecting hidden states from multiple layers of the multimodal model to the translation LLM, OmniFusion can translate speech, speech‑and‑image, and text‑and‑image inputs while reducing simultaneous speech‑translation latency by about one second compared to cascaded pipelines. The approach improves overall translation quality by leveraging both audio and visual context.
By Sai Koneru, Matthias Huck, Jan Niehues
MemoryCard is a video-memory-based augmentation framework designed to improve long-video question answering for Vision‑Language Models. It segments lengthy videos into semantically coherent units—each representing a distinct topic or event—by performing a self‑reading process over the video and aligned utterances. For each unit, the framework generates an event‑level video gist and selects representative visual moments, which are compiled into unified Memory Cards that are used for retrieval and answering questions, yielding up to a 21.8% relative accuracy improvement under comparable visual‑token budgets.
By Qing Yang, Pengcheng Huang, Xinze Li, Zhenghao Liu, Yukun Yan, Yu Gu, Ge Yu, Gang Li, Maosong Sun
arXiv:2510.17198v2 Announce Type: replace
Abstract: Riverbank erosion is a serious environmental problem in Bangladesh, causing land loss, damage to infrastructure, and displacement of local communit...
By M. Saifuzzaman Rafat, Akif Islam, Mohd Ruhul Ameen, Momen Khandoker Ope, Abu Saleh Musa Miah, Jungpil Shin
arXiv:2608.28216v1 Announce Type: new
Abstract: Locating a specific object instance in a cluttered scene using a single reference image and a short description, and reporting when that instance is ab...
By Kishor Datta Gupta, Ahmed Rafi Hasan, Md. Mahfuzur Rahman, Md. Sadman Haque, Mohd Ariful Haque
arXiv:2608.27871v1 Announce Type: new
Abstract: Long-video understanding remains challenging for Multimodal Large Language Models (MLLMs) due to limited context length. Uniform sampling may miss cruc...
By Ziling Huang, Shin'ichi Satoh
GraSP‑VL demonstrates that the length of frozen vision‑language embeddings can serve as a controllable semantic interface. By learning a shared near‑orthogonal prefix transform, the method creates a Semantic Matryoshka where short prefixes encode coarse semantics and longer prefixes reveal finer language‑grounded distinctions, all while preserving the original embedding geometry. Experiments on COCO/Flickr30K and SugarCrepe‑clean show strong performance with negligible drift in the full embedding space.
By Zesheng Li, Chengchang Pan, Honggang Qi
The paper examines OCR adaptation for low‑resource languages, noting that fine‑tuning often hits a performance ceiling in data‑scarce settings. It identifies that lower layers of language‑specific models learn redundant features while higher layers capture script nuances, leading to a structural inefficiency. To address this, the authors propose PSMC, a framework that pre‑trains a base model, specializes it per language, merges the experts via task arithmetic, and co‑trains a unified multilingual backbone, achieving about a 2% improvement in Word Recognition Rate across 10 Indian scripts without adding parameters.
By Achyuth P, Kahaan Shah, Chetan Arora
The paper introduces Short‑Films 20K (SF20K), a large publicly available movie dataset comprising 20,143 amateur films totaling 3,582 hours, with an average length of 12 minutes per film. Accompanying the dataset is SF20K‑Test, a manual open‑ended question‑answering benchmark featuring 95 movies and 979 question‑answer pairs. Analysis of the benchmark shows limited data leakage, highlights the necessity of long‑term reasoning, and demonstrates that instruction tuning on the large‑scale dataset significantly boosts vision‑language model performance.
By Ridouane Ghermi, Xi Wang, Vicky Kalogeiton, Ivan Laptev