Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.
arXiv:2601.09050v2 Announce Type: replace Abstract: Tonal low-resource languages are widely spoken but remain underserved by modern speech technologies. A central challenge is learning speech represe...
The paper investigates intersectional biases in multimodal clinical predictions using Electronic Healthcare Records (EHR). It introduces datasets MIMIC-Eye1 and MIMIC-IV ED, applies unified text representations from pre‑trained clinical language models, and benchmarks bias mitigation at the intersectional subgroup level. Results show that subgroup‑specific mitigation is robust across datasets, subgroups, and embeddings, effectively addressing intersectional biases in multimodal settings.
The paper reports a submission to the SHROOM-Visions shared task, aiming to detect and classify hallucinated character spans in vision‑language model outputs across four languages. The authors use multiple fine‑tuned vision‑language models as independent annotators, combine their predictions via character‑level majority voting, and also investigate activation probes. Their method achieved first place in three of the four languages and consistently ranked on the podium for all languages and metrics, with analysis showing that model disagreement mirrors human annotator disagreement.
arXiv:2609.16841v1 Announce Type: cross Abstract: Increasing image resolution produces ever-longer visual-token sequences in vision-language models (VLMs), substantially raising their inference cost....
MUUNRiver-Bench is a diagnostic benchmark for music retrieval that uses natural‑language instructions to define relevance for reference‑audio queries. It contains 3,440 tracks across 13 genres and 116 sub‑genres and covers seven tasks such as similar‑music, style‑preserving lyric‑rewriting, cover, and segment retrieval. Experiments with six models in eight configurations show that acoustic encoders favor local identity while text‑aligned encoders favor semantic relations, and that instruction‑aware and audio‑text fusion systems do not consistently outperform their backbones.
The paper introduces PosterVisor, a framework that replaces transient prompts with persistent control for scientific poster generation. It uses an Orchestrator to compile rubrics into Semantic‑Geometric Contracts that bind claims, sources, visuals, budgets, and spatial commitments, and employs Recursive Contract Enforcement to dynamically trigger checks and prevent silent regressions. Implementations in HTML/CSS and PPTX show improved QA accuracy and higher human preference on benchmark datasets.
OmniHarness is a framework that enables generalizable visual generation by learning symbolic policies from verified executions. It abstracts shared procedures and applicability conditions, allowing these policies to be instantiated, adapted, and composed for new tasks while keeping model parameters fixed. The system uses intermediate verification for refinement, self-directed inquiry to generate practice tasks, and continuous feedback to expand capabilities, achieving strong results on multiple benchmarks and outperforming baselines on Creative tasks.
The paper presents a multimodal machine learning framework that classifies Emirati residential architectural styles by combining visual features from images and textual descriptions using OpenAI's CLIP model. The unified 512‑dimensional embeddings are reduced with UMAP, clustered with K‑Means, and then used to train an SVM classifier, achieving a 98% accuracy across eight style clusters. This approach outperforms previous studies and demonstrates the value of integrating visual and textual data for cultural heritage analysis.
SyncVoice is a new automatic video dubbing framework that adds a lightweight Text‑Visual Fusion Module to a pretrained TTS system, aligning visual features with linguistic representations to produce temporally synchronized speech. The approach avoids complex architectural changes and achieves state‑of‑the‑art performance on the LRS3 dataset in zero‑shot dubbing. When further trained on a large bilingual audio‑visual corpus, SyncVoice improves vocal fidelity while maintaining synchronization, enabling a single model to dub both Chinese and English videos.
DGSG-Mind introduces a hybrid instance-aware 3D Gaussian dynamic scene graph system that integrates open‑vocabulary semantic information into dynamic 3D scene representations. By coupling a probabilistic voxel grid with explicit 3D Gaussians, it achieves robust cross‑modal instance fusion, incremental semantic mapping, and dynamic change handling through Gaussian‑based relocalization and masked refinement. The system builds a hierarchical scene graph and a 3D Gaussian Mind for multimodal reasoning, achieving state‑of‑the‑art zero‑shot 3D visual grounding and strong performance in open‑vocabulary semantic segmentation and scene reconstruction, and is demonstrated on real‑world robots.
arXiv:2609.16722v1 Announce Type: new Abstract: Scaling Multimodal Large Language Models (MLLMs) to long-form video understanding is bottlenecked by the explosion of visual tokens, which saturates co...
The survey "World-Action Models for Robot Learning and Control" reviews recent advances in coupling future world prediction with executable action generation for robots in open environments. It clarifies the scope of World-Action Models (WAMs) relative to conventional world models, model-based RL, and Vision‑Language‑Action policies, and organizes existing methods through a unified taxonomy covering representations, transition modeling, action interfaces, architectures, training pipelines, data modalities, and scaling strategies. The paper also surveys applications in manipulation, navigation, and autonomous driving, summarizes datasets, benchmarks, and metrics, and discusses key challenges such as action alignment, spatial consistency, long‑horizon memory, and efficient inference.
EviDep is a multimodal evidential regression framework for estimating depression severity from audio–visual recordings, incorporating multi‑scale temporal modeling and shared–private representation learning. It uses frequency‑aware feature extraction to decompose behavioral sequences into multiple frequency bands, refined by scale‑specific experts, and applies disentangled evidential learning to separate cross‑modal shared and modality‑specific information. The model outputs Normal‑Inverse‑Gamma distributions via multi‑branch evidential regression, enabling estimation of depression severity along with aleatoric and epistemic uncertainty, and demonstrates competitive accuracy on several benchmark datasets.
The paper introduces Relation‑Orbit, a symmetry‑aware method for aggregating likelihoods from frozen vision‑language models to verify fine‑grained left‑right claims. It uses a closed‑form contrast that assigns eight normalized likelihoods based on reflection, inverse relation, and entity exchange, and asserts a claim only when the signed contrast exceeds a threshold chosen via Clopper‑Pearson bounds. Experiments on VSR, GQA, and LLaVA‑1.5/COCO show that Relation‑Orbit achieves higher mean test coverage at a 10% selective‑risk calibration target compared to an all‑eight Orbit‑Max baseline across multiple dataset‑backbone settings.
The paper introduces Affect-Prototype Guided Fusion (APCF), a framework for open‑vocabulary multimodal emotion recognition that handles incomplete and unsynchronized modal data. APCF builds an affect‑prototype library to model how different emotions contribute across modalities, enabling dynamic fusion of available features. The fused representations are then decoded by an LLM to generate open‑vocabulary emotion labels, achieving superior performance on OV‑MERD+ and MER‑FG datasets compared to existing methods.
arXiv:2609.16795v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) can answer knowledge-intensive visual questions by combining visual evidence from images with facts retrieved...
The paper introduces PolyLatentFlow, a continuous‑time flow‑matching framework for polymer generation, and LlamaUni, a multimodal representation that fuses polymer sequences with 3D structural data. In unconditional generation, the combination yields the highest number of valid, novel candidates while preserving diversity, and in conditional settings it systematically shifts property distributions across a 200 °C target range. Across multi‑property tasks, the representation choice affects validity, training‑set replay, and structural proximity, with PolyLatentFlow + LlamaUni achieving the best balance of high validity, low replay, and high target hit yield.
The paper introduces ViD, a vision‑dominant gender bias mitigation framework for large vision‑language models. ViD uses causal analysis of attention patterns and dual mechanisms—backdoor adjustment and refined token selection—to suppress bias while preserving reasoning and generation quality. Experiments show a 14.7% reduction in gender bias on FACET and significant improvements on MS COCO image captioning, all without extra training overhead.
arXiv:2602.06652v2 Announce Type: replace Abstract: The robustness of Vision Language Models (VLMs) is commonly assessed through output-level invariance, implicitly assuming that stable predictions r...
SPEAR NeXT is a compact, pixel‑wise multimodal spectral‑temporal foundation model that learns temporal self‑supervision by predicting future latent Earth states from past observations. It encodes instantaneous states from optical, radar, and environmental data into 32‑dimensional embeddings, then models their evolution with a causally masked transformer that forecasts multiple future horizons. The model uses Rotary Position Embeddings to capture relative temporal order and month/year embeddings to encode seasonal and interannual context.