Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.
arXiv:2609.21663v1 Announce Type: new Abstract: Word Error Rate (WER), the most commonly used metric for Automatic Speech Recognition (ASR), treats every lexical deviation from the reference as equal...
arXiv:2609.21844v1 Announce Type: new Abstract: Long transcripts are costly inputs for downstream NLP systems and often contain irrelevant context. We study query-conditioned topic localization: pred...
arXiv:2609.20904v1 Announce Type: cross Abstract: Hybrid motor-imagery brain-computer interfaces (MI-BCIs) combining EEG and fNIRS can outperform EEG-only systems by exploiting complementary electrop...
arXiv:2609.21522v1 Announce Type: new Abstract: Recent pre-trained foundation models provide rich multi-modal priors for downstream 3D vision tasks. However, the effectiveness of these representation...
We present M3GA-Wild, the first benchmark for multi-modal, multi-session ground-to-aerial place recognition in forests. M3GA-Wild unifies and extends existing forest localisation datasets, providing a...
LaSR (Latent Speech Reasoning) is a new training paradigm for context‑aware speech recognition that uses a latent reasoning trajectory instead of explicit intermediate tokens. It aligns chain‑of‑thought supervision around the acoustic region of target words and introduces latent reasoning periods for grounding context and guiding transcription transitions. Experiments on Fun‑Audio‑Chat show that LaSR improves terminology recognition without added latency and outperforms standard fine‑tuning baselines, demonstrating the promise of latent reasoning for efficient, context‑aware speech assistants.
The paper introduces IMFD, an end‑to‑end multi‑face forgery detector that uses instruction‑based Large Vision‑Language Models (LVLMs). IMFD jointly localizes faces and predicts forgery labels in a single stage, explicitly incorporating predicted face bounding boxes into the textual instruction to improve grounding and detection. Experiments on converted multi‑face forgery datasets show that IMFD outperforms several state‑of‑the‑art methods.
BinoGen is an automated framework that generates large-scale, embodiment-aware egocentric binocular visual experiences in indoor environments. It models environmental and observer variation through generative scene synthesis, probabilistic object instantiation, appearance randomization, stochastic trajectory generation, and configurable binocular camera setups, producing synchronized videos with dense multimodal supervision such as depth maps, optical flow, surface normals, semantic maps, object coordinates, and camera poses. Using BinoGen, the authors created a dataset of over 20 million annotated images, demonstrating that incorporating this data improves real-world visual perception tasks like depth estimation, object detection, and video object tracking, and that embodiment-specific adaptation enhances performance while joint training enables a single model to perform competitively across different observer embodiments.
The paper introduces a context‑aware neural grapheme‑to‑phoneme (G2P) system for unsegmented languages like Japanese, using a discriminative conditional random field over a word lattice built from dictionaries. It addresses data scarcity by generating over two million sentences with large language models. Experiments show the method surpasses traditional morphological analyzers and neural sequence models, achieving 99.62% target word reading accuracy and very low phoneme error rates on the Joyo‑Kanji‑Yomi benchmark.
The Public Discourse Corpus (PDC) is the first dataset of public‑figure interview speech annotated for affective valence and epistemic modality. It contains 998 videos from 100 speakers across seven professional domains, yielding 186,642 sentences (3.1 million words). A key methodological contribution is Target Speaker Participation (TSP), a five‑category annotation taxonomy with documented inter‑annotator reliability (κ = 0.616), and an audio‑first diarization pipeline that separates target‑speaker turns from interviewer and third‑party speech. The corpus, annotation tools, validation sample, and processing pipeline are released as open source.
The paper introduces FCA‑Guided Counterfactual (FCA‑CF) explanations for multi‑modal breast cancer diagnosis, leveraging a Formal Concept Analysis lattice as a hard structural constraint to search for counterfactuals. On the TCGA‑BRCA dataset, FCA‑CF achieves perfect validity (100% prediction flips), the lowest average feature changes (2.37), and competitive proximity (0.900) compared to four other methods. Ablation studies show the lattice constraint drives sparsity, while a greedy refinement phase further improves results.
The paper investigates whether open‑source Vision‑Language Models (VLMs) can perform zero‑shot action quality assessment (AQA) on Olympic diving videos. Using the AQA‑7 benchmark, the authors propose a regression framework that combines VLM‑generated semantic reasoning, phase‑level sub‑scores, TF‑IDF vectorization, dimensionality reduction, and ensemble learning to predict final competition scores. While individual VLMs achieve moderate Spearman correlations (<0.32), the ensemble approach boosts performance to 0.67, demonstrating that VLM‑derived textual reasoning features are more informative than raw numerical sub‑scores for AQA. whyItMatters":"The study shows that VLMs can serve as explainable, semi‑automated tools for evaluating sports performance, potentially aiding expert judging in complex, subjective Olympic events."
The survey "From Models to Systems: A Comprehensive Survey of Efficient Multimodal Learning" reviews over 300 works on efficient multimodal learning (EML), proposing a structured taxonomy that spans model, algorithm, and system layers. It synthesizes how cross‑layer co‑design addresses the Efficiency‑Utility‑Privacy trade‑off and illustrates this through a case study of multimodal large language models. The paper also offers optimization blueprints for various domains, discusses a shift toward self‑regulating intelligence, and outlines open challenges for future EML research.
DeliveryGym is a 3D reinforcement learning environment that simulates continuous courier shifts, integrating multimodal tool interaction, persistent world dynamics, and trajectory‑based rewards derived from simulator events. It allows agents to learn how their decisions affect time, energy, and money across an entire shift, and it adapts future training shifts to the policy’s weaknesses while keeping evaluation fixed. Experiments on six models and 13 city maps show a significant gap between task execution and optimal sequencing, with RL improving Qwen3‑VL‑4B’s net income by 54.3% and adaptive training boosting test income by 16.5% over uniform sampling.
AVTrace is a diagnostic suite designed to evaluate audio‑visual temporal reasoning in omni models. It covers tasks such as onset and span grounding, synchronization, next‑step prediction, cross‑modal localization, chain parsing, and event‑conditioned comprehension, providing 34,114 training examples and balanced development and test splits. Five open omni models were tested, all scoring below the majority‑label baseline on synchronization verification and showing low performance on chain parsing and event‑conditioned tasks, while parameter‑efficient temporal post‑training improved some metrics.
The paper introduces DB‑SUiT, a surface‑based diffusion bridge that translates cortical MRI to PET images directly on the cortical manifold. It employs a conditional spherical U‑shaped vision Transformer to capture multi‑scale surface features and long‑range dependencies while incorporating demographic and subcortical information. Evaluations on two dementia datasets show that the synthesized PET surfaces outperform MRI and PET volumes in automated classification and achieve high diagnostic accuracy in a blinded reader study.
The paper introduces VLN on the Fly, an onboard vision‑language navigation stack for aerial robots that separates grounding, planning, and control into inspectable stages. A quantized vision‑language model grounds instructions to a coarse image cell, depth estimation lifts this to a 3D goal, a fast B‑spline planner generates a feasible trajectory, and a pretrained reinforcement learning policy translates the trajectory into motor commands. In controlled indoor flights, the stack achieved the target in 13 of 15 trials with a mean goal error of 5.72 cm and 39.3% GPU utilization, and successfully tracked collision‑free trajectories in cluttered environments.
The paper introduces a modular, model‑agnostic pipeline to enhance automatic speech recognition for FarmerChat, an AI agricultural advisory assistant used by smallholder farmers in their native languages. The pipeline integrates gated audio enhancement, speaker diarization with target‑speaker selection, domain‑aware lexicon correction, and a quality gate, requiring fine‑tuning only at the diarization stage. Evaluations on Hindi, Telugu, and Odia recordings show significant reductions in word error rate—up to 42% on multi‑speaker cloud ASR models—demonstrating that targeted preprocessing and domain‑specific post‑processing can markedly improve transcription quality without altering the core ASR model.
HIL-UMI is a policy-guided Universal Manipulation Interface that enables robot‑free, human‑in‑the‑loop post‑training of vision‑language‑action models. By querying the current policy during handheld demonstrations and using an Energy Score to detect out‑of‑distribution states, it selectively collects new data and refines a progress‑based advantage estimator. The updated estimator then drives advantage‑conditioned behavioral cloning, improving performance on long‑horizon and precise manipulation tasks while reducing per‑frame collection time compared to HG‑DAgger.
GeoAAC introduces a geometry-based adaptive action chunking technique for Vision‑Language‑Action policies, dynamically adjusting the action horizon based on the reliability of current action predictions. By leveraging the geometric variation in Flow Matching denoising trajectories, GeoAAC constructs a horizon‑wise geometric profile that determines the action horizon during a single generation without extra training. Experiments on LIBERO, LIBERO‑Pro, RoboCasa365, and real‑world manipulation tasks demonstrate consistent gains over fixed‑horizon baselines, achieving up to 8.7 percentage points improvement in simulation and raising real‑world success rates from 53.3% to 74.4%.