The paper investigates how to evaluate audio‑language models by separating the use of acoustic evidence from the need to invoke a generative audio model. Using a controlled call‑decision framework, the authors compare policies that rely on transcript labels, encoder outputs from CLAP, AST, or WavLM, and optional calls to generative models such as Qwen2‑Audio, Qwen2.5‑Omni, or MOSS‑Audio. Results on the VocalSound dataset show that while transcript‑only accuracy is low (0.296), encoder‑only controls achieve high accuracy (≈0.85) without any generative calls, and adding generative calls yields only a marginal improvement (0.925 vs. 0.921).
By Mengzhe Geng
The paper introduces STITCH, a training‑free method that partitions videos into semantically meaningful temporal chunks using a frozen video‑text backbone. By detecting changes in the embedding sequence of short video windows, STITCH produces reusable temporal abstractions that can be applied to multiple tasks such as event boundary detection, language‑based moment retrieval, and frame selection for vision‑language models. Experiments show that STITCH performs competitively with specialized methods while requiring no task‑specific training, especially when processing is limited to a few frames or tokens.
By Etienne Casanova, Sevan Brodjian, Pietro Perona
Iron is a new framework for training generalist virtual agents that aligns low‑level actions with high‑level intents using a stepwise cycle‑consistent reward. It also repurposes failed trajectories through a hindsight reproduction mechanism to improve learning efficiency and task diversity. Experiments show Iron‑trained agents outperform those trained with three times more data, achieving a 25.06% relative improvement on unseen web tasks and better performance on complex tasks.
By Jiahe Ying, Wendong Bu, Kaihang Pan, Bingchen Miao, Siyu Chen, Wen Wang, Xueming Jiang, Juncheng Li, Siliang Tang
PACE is an agentic framework that learns publisher‑specific extraction configurations from representative web pages and user requirements. During training it employs LLMs to analyze page structure and gather reusable extraction patterns, then creates a deterministic extractor template for inference that eliminates the need for further LLM calls. Experiments on article bodies, metadata, images, and tables show that PACE surpasses scalable non‑manual baselines and approaches the quality of manually engineered publisher‑specific parsers.
By Zhanlin Liu, Munirathnam Srikanth
arXiv:2608. 28389v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) augments LLMs with external documents, but public or user-editable sources expose RAG systems to data poisoning: attackers can inject malicious documents to steer outputs toward targeted answers.
By Jaewon Jung, Haizhong Zheng, Hongsun Jang, Jaeyong Song, Beidi Chen, Jinho Lee
OceanGym is the first comprehensive benchmark for underwater embodied agents, featuring eight realistic task domains and a unified agent framework powered by Multi‑modal Large Language Models (MLLMs). It challenges agents to process optical and sonar data, navigate complex environments, and achieve long‑horizon goals amid low visibility and dynamic currents. Experiments show significant performance gaps between current MLLM agents and human experts, underscoring the difficulty of perception, planning, and adaptability in ocean settings.
By Yida Xue, Mingjun Mao, Xiangyuan Ru, Yuqi Zhu, Baochang Ren, Shuofei Qiao, Mengru Wang, Shumin Deng, Xinyu An, Ningyu Zhang, Ying Chen, Huajun Chen
arXiv:2608.27885v1 Announce Type: new
Abstract: Multimodality translation (e.g., text-to-image) is a core generative AI task. However, existing approaches (1) follow generative paths that do not dire...
By Gabe Guo, Elon Litman, Thanawat Sornwanee, Jose Blanchet, Stefano Ermon
arXiv:2608.27881v1 Announce Type: new
Abstract: Recently, many streaming video understanding methods have been proposed by constructing an external memory to store historical data for computational r...
By Yuxin Liu, Peiqin Zhuang, Yali Wang
DeicticVLA unifies three instruction modes—Language Instruction, Vision‑Language Instruction, and Visual Instruction—into a single text prompt and deictic mask framework, allowing a single pretrained Vision‑Language‑Action model to handle all modes. The approach uses text‑prompt completion and deictic gesture grounding, and evaluates various visual prompting methods and training strategies in simulation and real‑world tasks. Results show that two‑stage training improves deictic mask usage, and that Vision‑Language and Visual Instruction outperform Language Instruction on unseen expressions, appearance changes, and novel objects, achieving 100% success on unseen categories.
By Kango Yanagida, Tatsuya Aoki, Yuichiro Yoshikawa, Takato Horii
OmniFusion is an end‑to‑end multilingual multimodal translation system that fuses a pretrained multimodal foundation model (Omni 2.5‑7B) with a translation large language model (SeedX PPO‑7B). By connecting hidden states from multiple layers of the multimodal model to the translation LLM, OmniFusion can translate speech, speech‑and‑image, and text‑and‑image inputs while reducing simultaneous speech‑translation latency by about one second compared to cascaded pipelines. The approach improves overall translation quality by leveraging both audio and visual context.
By Sai Koneru, Matthias Huck, Jan Niehues
arXiv:2608.28216v1 Announce Type: new
Abstract: Locating a specific object instance in a cluttered scene using a single reference image and a short description, and reporting when that instance is ab...
By Kishor Datta Gupta, Ahmed Rafi Hasan, Md. Mahfuzur Rahman, Md. Sadman Haque, Mohd Ariful Haque
arXiv:2604.06079v2 Announce Type: replace
Abstract: Graphics Program Synthesis is pivotal for interpreting and editing visual data, effectively facilitating the reverse-engineering of static visuals...
By Juekai Lin, Yun Zhu, Honglin Lin, Sijing Li, Tianwei Lin, Zheng Liu, Xiaoyang Wang, Wenqiao Zhang, Lijun Wu
arXiv:2608.28455v1 Announce Type: new
Abstract: Contrastive vision-language learning uses paired chest CT volumes and radiology reports to learn abnormality classifiers without manually annotated lab...
By Huseyin Umut Isik, Mehmet Alp Ozaydin, Sila Kurugol, \c{S}eyda Ertekin
The paper introduces Meta-Adaptive Multimodal Jailbreaking (MAMJ), a method that jointly optimizes an attack strategy prompt and attacker weights to generate more effective jailbreaks against vision‑language models. Using an LLM‑based critique to refine the strategy and group‑level success‑rate rewards to update the weights, MAMJ achieves high attack success rates on MM‑SafetyBench, outperforming existing baselines by up to 24.1 percentage points. The learned attacker also transfers to unseen models and remains robust against typical defenses, highlighting a systemic vulnerability in current VLMs.
By Benlei Cui, Shen Pang, Yuke Wang, Xuemei Dong, Yuwen Zhai, Jingqun Tang, Haiyang Yu, Hui Xue, Longtao Huang, Haiwen Hong
arXiv:2608.28517v1 Announce Type: new
Abstract: Cross-modal image translation in remote sensing must preserve source-observed content while matching the target-domain distribution. Existing methods j...
By Keyan Hu, Mingtao Wang, Ziyu Zhou, Tiandong Shi, Haifeng Li, Ji Qi, Chao Tao
arXiv:2509.24192v2 Announce Type: replace
Abstract: Vision-language models (VLMs) have advanced multimodal perception, demonstrated by open-vocabulary object detection with simple language queries. S...
By Sojung An, Kwanyong Park, Yong Jae Lee, Donghyun Kim
The paper presents an automated pipeline that generates high‑quality Quranic datasets, including 848 hours of audio and 286,000 annotated utterances, by collecting recitations, segmenting at pause points with a fine‑tuned wav2vec2‑BERT model, transcribing segments, and verifying transcripts using a novel Tasmeea algorithm. It introduces qdat_bench, a benchmark covering phonemes, diacritization, and Tajweed rules, and a custom Quran Phonetic Script (QPS) for encoding Tajweed. A multi‑level CTC model trained on this data achieves a 0.21% phoneme error rate on the test set and 1.94% on qdat_bench, with a 75.8% Tajweed F1 score.
By Abdullah Abdelfattah, Mahmoud I. Khalil, Hazem Abbas
arXiv:2608.27871v1 Announce Type: new
Abstract: Long-video understanding remains challenging for Multimodal Large Language Models (MLLMs) due to limited context length. Uniform sampling may miss cruc...
By Ziling Huang, Shin'ichi Satoh
arXiv:2608.27505v1 Announce Type: new
Abstract: Reinforcement learning from human feedback (RLHF) has become the dominant paradigm for aligning large language models (LLMs) with human preferences. Ho...
By Zifei Shan, Fangning Shao
The paper introduces the Semantic Drift Protocol (SDP), a Telephone Game-inspired method to evaluate how well unified models preserve meaning when alternating between image-to-text (I2T) and text-to-image (T2I) generation over multiple generations. It defines Mean Cumulative Drift (MCD) and Multi-Generation GenEval (MGG) as metrics for semantic retention, and presents a new benchmark of 400 image‑text pairs from NoCaps and DOCCI to stress-test models beyond COCO. Applying SDP to seven models shows that models with strong single‑pass scores can still suffer severe semantic drift, revealing catastrophic failure modes that isolated benchmarks miss.
By Sabbir Mollah, Rohit Gupta, Sirnam Swetha, Qingyang Liu, Ahnaf Munir, Mubarak Shah