Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

8,364 stories · RSS feed

arXiv Computation and Language
Sep 11

VectraYX-Vision-1B: A Sub-2B Spanish/LATAM Cybersecurity Vision-Language Model with Structured Visual Reasoning and Native Tool Use

The paper announces VectraYX‑Vision‑1B, a Spanish/LATAM cybersecurity vision‑language model that uses a natively‑trained visual tower (Qwen2‑VL) and a frozen decoder. The authors report that transplanting the visual tower improves the failing 8‑nibble address field from 0.00 to 0.81 exact, using a coarser token budget than 2×2 tiling, and that a second pre‑registered field shows 63% contamination and is demoted. The model’s B6/B7 tool‑id remains at floor, and code/checkpoints are available on Hugging Face.

By Juan S. Santillana
arXiv Machine Learning
Sep 11

Robust Multimodal Sentiment Analysis with Incomplete Modalities via Semantic-aware Completeness based Reconstruction

The paper presents a method for robust multimodal sentiment analysis that handles incomplete or noisy modalities. It introduces a completeness estimation technique to measure how much sentiment-relevant information remains in partial data, guiding the reconstruction of missing semantics. A joint training strategy stabilizes multi-task learning for sentiment prediction and completeness estimation, and experiments on three benchmark datasets show improved semantic reconstruction and sentiment accuracy.

By Han-Jun Choi, Byunggill Joe, Saim Shin, Jin Yea Jang
arXiv Computation and Language
Sep 11

Learning to Think Like a Cartoon Captionist: Incongruity-Resolution Supervision for Multimodal Humor Understanding

The paper introduces IRS (Incongruity-Resolution Supervision), a framework that breaks humor understanding into three parts: identifying mismatches in a visual scene, creating coherent reinterpretations of those mismatches, and aligning these interpretations with human preferences. IRS uses structured reasoning traces to guide models from visual perception to humorous interpretation, and it is evaluated on the New Yorker Cartoon Caption Contest. Experiments on 7B, 32B, and 72B models show that IRS improves caption matching and ranking, with the 72B model achieving 76.10% ranking accuracy—outperforming non-expert humans and all other multimodal baselines—and demonstrates transferable reasoning patterns in zero‑shot settings.

By Hatice Merve Vural, Doga Kukul, Ege Erdem Ozlu, Demir Ekin Arikan, Bob Mankoff, Erkut Erdem, Aykut Erdem
arXiv Computer Vision
Sep 11

Recursive Code World Models: Building Complex Worlds through Recursive Scene Programs

Recursive Code World Models (RCWM) present a framework that reconstructs complex 3D worlds from a single reference image by combining a Recursive Scene Program (RSP) representation with a construction solver that recursively calls itself. The RSP encodes the world as compositional scene code, while each solver invocation follows a global‑local‑global recursion: it first establishes the whole scene, then recursively reconstructs unresolved parts, and finally revisits the whole to refine composition. This approach preserves scene‑wide geometry and relationships, propagates a shared camera projection across levels, and uses a vision‑language coding agent to compare reference images with scene renders, enabling fine‑scale structure refinement and improved reconstruction over prior methods.

By Zhiqi Li, Yuxuan Liao, Bo Zhu
arXiv Machine Learning
Sep 11

HuRo: Robotizing Human Videos for Scalable VLA Pretraining

The paper introduces HuRo, a dataset of 630K robotized episodes derived from diverse human videos, created via a pipeline that aligns observations and actions for robotic use. Experiments on four real‑world manipulation tasks show that scaling robotized pretraining boosts task completion from 51.5% to 80.3% and improves out‑of‑distribution performance under spatial and visual shifts. Ablation studies reveal that visual robotization enhances robustness and that end‑to‑end pretraining with retargeted actions outperforms visual‑only transfer.

By Jinho Jeong, Se June Joo, Jaehyun Kang, Dongyun Kim, Yena Kim, Hanjung Kim, Seon Joo Kim
Hugging Face Trending Papers
Sep 10

The Eloquence submission for Task 2 of the Interspeech 2026 MLC-SLM challenge

The Eloquence team presents three strategies for the Interspeech 2026 MLC‑SLM Task 2, a multilingual MCQA challenge across 21 languages. They fine‑tune Voxtral‑Mini‑3B with LoRA and cross‑lingual augmentations, achieving 0.72 macro‑accuracy; they use multimodal in‑context learning on the frozen Voxtral‑24B to correct label bias, reaching 0.81; and they deploy a training‑free retrieval system with a voice‑anchored memory, scoring 0.68. All approaches surpass the official baseline.

Hugging Face Trending Papers
Sep 10

Combining Synthetic and Real Data for Low-Resource Historical OCR: A Manchu Case Study

The paper investigates how to best combine synthetic and real historical Manchu word images for low‑resource OCR. Using 60,000 synthetic and 20,306 real images, the authors compare three pretrained vision‑language models and a compact CRNN across synthetic‑only, real‑only, joint, and sequential training regimes. Adding real data boosts word accuracy to 95–96%, and ensembling the best recognizers raises it to 98.27% without extra training.

Hugging Face Trending Papers
Sep 10

DINO-Med: A Unified Patch-Based Adaptation Framework for Multi-Modal Medical Image Analysis Applied to Liver Fibrosis Staging

The paper introduces DINO-Med, a patch‑based framework that adapts natural‑image foundation models to multi‑modal medical imaging, specifically for liver fibrosis staging. It processes raw multimodal scans through training‑free registration, automated localization, and mask‑filtered patch extraction, then aggregates patch‑level insights into subject‑level diagnostics. Using the CARE 2025 Liver Track 4 cohort, the DINOv3‑based approach achieved the highest classification accuracy (78.4% for mild fibrosis and 75.8% for cirrhosis) compared to other feature representations.

Hugging Face Trending Papers
Sep 10

2AM: Grounding Agent-Side Memory as Guidance for Steerable Action Models in Long-Horizon Manipulation

The paper introduces 2AM, a system that keeps task memory solely within a multimodal Agent while using a single RGB‑based, stateless Action Model to execute motions. By compiling interaction history into subtask language and optional 2D grasp/place/move hints, the Agent steers the Action Model, which is trained to tolerate imperfect guidance through dropout, noise, and jitter. On the LIBERO‑Mem benchmark, 2AM achieves 76.3% average completion without depth, geometry, or planners, vastly outperforming the best baseline.

arXiv AI
Sep 10

NOAH: Learning the Full Patient Journey. A Longitudinal Multimodal Time-Aware Model for Representation and Forecasting

NOAH is a generative transformer that learns the full multimodal patient journey by integrating bidirectional time and a variational latent space. Trained on over 559 million clinical events from 431,000 hospital visits, it processes medical images, time‑series, numeric signals, categorical events, and both structured and unstructured records. The model supports autoregressive forecasting, zero‑shot classification, and counterfactual intervention simulation, yielding strong performance on clinical outcomes, ICD chapters, comorbidities, and time‑to‑event prediction.

By Tobias Susetzky, Raphael Rehms, Dmitrii Seletkov, \"Ozg\"un Turgut, Michelle Espranita Liman, Lisa Steinhelfer, Rickmer Braren, Daniel Rueckert
arXiv AI
Sep 10

S$^3$F-Net: A Multi-Modal Approach to Medical Image Classification via Spatial-Spectral Summarizer Fusion Network

The paper introduces S$^3$F-Net, a dual‑branch network that fuses spatial and spectral representations for medical image classification. It combines a deep spatial CNN with a shallow spectral encoder, SpectraNet, which uses a learnable SpectralFilter layer to process the full Fourier spectrum efficiently. Evaluated on four medical imaging datasets, S$^3$F-Net consistently outperforms spatial‑only baselines, achieving state‑of‑the‑art accuracy on BRISC2025 and surpassing deeper models on the Chest X‑Ray Pneumonia dataset.

By Md. Saiful Bari Siddiqui, Mohammed Imamul Hassan Bhuiyan
arXiv Machine Learning
Sep 10

CoRL: Co-Evolutionary Reinforcement Learning for Adaptive Indirect Prompt-Injection Attacks and Defenses

The paper introduces CoRL, a co-evolutionary reinforcement learning framework designed to defend against adaptive indirect prompt-injection attacks on tool-augmented language agents. CoRL operates in three stages—attacker fine‑tuning, bilateral Co‑PPO training, and defender fine‑tuning—using verifier‑grounded repairs to adapt to changing attack strategies. Experiments on 1,514 executions show that CoRL reduces attack success rates to 0% while improving task utility, demonstrating its effectiveness against adaptive adversaries.

By Boyang Zhang, Qingxin Xiao, Lingwei Dang, Qingyao Wu
arXiv Computation and Language
Sep 10

From Symbolic Perception to Logical Deduction: A Framework for Guiding Language Models in Geometric Reasoning

The paper introduces a framework that combines a Geometric Vision Parser and a Symbolic Solver to enable a Large Language Model to solve complex plane geometry problems. By translating diagrams into symbolic representations and performing formal deductions, the approach reduces hallucinations and produces interpretable, human-like solutions. Experiments on a new benchmark from 2025 Chinese Zhongkao exams show performance comparable to Gemini 2.5 Pro.

By Weichen Dai, Rafael Medeiros Cabral, Ziyi Shou, Yan Cao, Xin Shen, Dongcai Lu, Yi Zhou
arXiv Computer Vision
Sep 10

Freezing of Gait Prediction Under Spatial Occlusion: An IMU-Supervised Cross-Modal Distillation Approach

The paper introduces a cross‑modal distillation framework that combines the accuracy of inertial measurement unit (IMU) data with the practicality of video‑based gait analysis to detect freezing of gait (FOG) in Parkinson’s patients. By extracting invariant latent topologies from a pre‑trained kinematic oracle, the method supervises a visual architecture and fuses skeletal graph nodes with continuous spatial pixels to handle severe spatial occlusion during continuous 360° turns. Experiments on a public multimodal dataset show that this approach reduces tracking entropy and achieves precise FOG predictions without requiring wearable sensors.

By Chandan Biswas, Aryan Singh, Anabik Pal
arXiv Computer Vision
Sep 10

VideoTIR: Accurate Understanding for Long Videos with Efficient Tool-Integrated Reasoning

VideoTIR introduces a reinforcement‑learning approach to improve long‑video understanding by encouraging multimodal large language models to use comprehensive multi‑level toolkits efficiently. It combines Zero‑RL and SFT cold‑starting strategies to help models retrieve and focus on meaningful video segments, images, and regions, thereby reducing hallucinations. The method includes Toolkit Action Grouped Policy Optimization (TAGPO) to streamline tool‑calling and a sandbox‑based trajectory synthesis framework for high‑quality data, achieving strong results on three long‑video QA benchmarks.

By Zhe Gao, Shiyu Shen, Taifeng Chai, Weinong Wang, Haotian Xu, Xing Wu, Wenbin Li, Qi Fan, Yang Gao, Dacheng Tao
arXiv AI
Sep 10

LoGAN: Multilingual Font Localization with Generative Agents

LoGAN is a VLM-based agentic framework designed for few-shot multilingual font localization. It takes a handful of glyphs or logo letters and generates complete character sets across many languages, including CJK, by combining a glyph-level diffusion model, style finetuning, spacing/kerning transfer, and texture expansion. The method outperforms specialized font generators and state‑of‑the‑art image editors in glyph fidelity, style, texture, and kerning consistency on datasets covering more than 27 languages.

By Zhuoning Yuan, Ta-Ying Cheng, Benjamin Klein
arXiv AI
Sep 10

3DWay: Generalizing Robot Manipulation via 3D Consistent Waypoints

The paper introduces 3DWay, a method that predicts 3D-consistent waypoints for robot manipulation by first generating multi‑view consistent 2D waypoints and then triangulating them. This approach addresses the 3D ambiguity inherent in 2D trajectory predictions and leverages pretrained vision‑language models to provide explicit 3D motion specifications. Experiments demonstrate that 3DWay improves 3D spatial grounding and vision‑language reasoning, enhancing generalization for robot manipulation tasks.

By Ziqin Huang, Yingyue Li, Chenyangguang Zhang, Ruida Zhang, Yuxin Chen, Gu Wang, Xingyu Liu, Masayoshi Tomizuka, Xiangyang Ji
arXiv AI
Sep 10

Segment Any Motion with Radar: Robust Multimodal Moving-Object Segmentation and Tracking

The paper introduces RGBTR‑Motion, a new benchmark that synchronizes RGB, thermal, and radar data with dense moving‑instance masks and consistent identities for surveillance scenes. It also presents SAM‑Radar, a segmentation and tracking framework that fuses calibrated RGBT features with radar returns, using radar‑aware detection and motion supervision to reject clutter and maintain identity continuity during low visibility or occlusion. SAM‑Radar achieves state‑of‑the‑art performance, improving IoU, F1‑50, MOTA, HOTA, and IDF1 metrics over existing methods.

By Jue Wang, Xuan Wang, Hao Zhou, Ruixiang Zhou, Yixuan Zhou, Tianshuo Yuan, Jieming Ma, Jie Zhang, Fei Luo
arXiv AI
Sep 10

PopResume: Causal Fairness Evaluation of LLM/VLM Resume Screeners with Population-Representative Dataset

PopResume is a population‑representative resume dataset designed for causal fairness auditing of large language model (LLM) and vision‑language model (VLM) resume screeners. It grounds fairness evaluation in real population statistics and preserves natural attribute relationships, enabling path‑specific effect (PSE) analysis that separates business‑necessity from redlining pathways. Using PopResume, the authors evaluated eight models on 60.8K resumes across five occupations and uncovered five discrimination patterns that aggregate metrics missed, demonstrating the value of causally‑grounded auditing.

By Sumin Yu, Juhyeon Park, Taesup Moon