Multimodal models

Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.

7,981 stories · RSS feed

arXiv Machine Learning
Sep 17

Beyond Random Couplings: Contrastive Noise Alignment in Generative Flows

The paper introduces Contrastive Noise Alignment (CNA), a training-time method for generative flow models that dynamically aligns Gaussian noise with data samples using a cross-modal InfoNCE objective. By modeling noise as an interacting particle system and regularizing with angular entropy and radial norm penalties, CNA reduces arbitrary data-noise couplings and flow curvature. Empirical results show that CNA improves generation quality, lowering FID by over 50% for few-step pixel-space generation compared to standard rectified flow and outperforming optimal transport baselines by at least 24%.

By Lennart Wittke, Vinicius Azevedo
arXiv Machine Learning
Sep 17

Mixed-Integer Nonlinear Differentiable Predictive Control for Underground Pumped Hydro Energy Storage Systems

The paper extends Mixed-Integer Differentiable Predictive Control (MI‑DPC) to handle multi‑modal discrete decisions and nonconvex polynomial dynamics in Underground Pumped Hydro Energy Storage Systems (UPHES). It introduces a neural policy that maps problem parameters to continuous setpoints and integer mode selections using a Gumbel‑Softmax layer, trained via self‑supervised differentiation of the finite‑horizon control objective. Key methodological advances include a parallel differentiable simulator, a Transformer encoder for long‑range temporal dependencies, and a Gumbel‑Softmax temperature annealing schedule, enabling the framework to achieve only 1.6% suboptimality compared to a mixed‑integer quadratic programming baseline while providing a five‑order‑of‑magnitude speedup in online scheduling.

By Honghui Zheng, J\'an Boldock\'y, Yury Dvorkin, J\'an Drgo\v{n}a
arXiv Machine Learning
Sep 17

TabPFN-3.5: Technical Report

The report introduces TabPFN‑3.5, a new flagship tabular foundation model that outperforms its predecessor TabPFN‑3 and all existing baselines on a wide range of tabular tasks. It achieves state‑of‑the‑art performance on standard tabular prediction in TabArena and extends to practical scenarios such as non‑i.i.d. data, temporal or grouped splits, tables containing strings, text, images, high‑cardinality categorical features, and wide tables. Variants like TabPFN‑3.5‑Fast, TabPFN‑3.5‑Plus, and TabPFN‑3.5‑Thinking offer faster inference, expanded multimodal capabilities, and further speed improvements up to 12× faster than the previous Thinking mode.

By Benjamin J\"ager, Nick Erickson, L\'eo Grinsztajn, Felix Birkel, Klemens Fl\"oge, Oscar Key, K\"ur\c{s}at Kaya, Jonas K\"ubler, Ad\`ele Frankel, Tobias Schr\"oder, Anurag Garg, Jan Hendrik Metzen, David Salinas, Simon Bing, Kristina Collins, Tuana \c{C}elik, Vahid Balazadeh, Lydia Sidhoum, Tom\'as Pereda, Brendan Roof, Andrej Tschalzev, Siyuan Guo, Philipp Singer, Lennart Purucker, Jake Robertson, Marie Salmon, Philipp Jund, Jerry Chen, Diana Kriuchkova, Arthur Cahu, Eliott Kalfon, Adrian Hayler, Georg Grab, Vitor Monteiro, Lilly Wehrhahn, Dominik Safaric, Clara Cornu, Alan Arazi, Rylee Grace, Simone Alessi, Mihir Manium, Bernhard Sch\"olkopf, Yann LeCun, Madelon Hulsebos, Sauraj Gambhir, Noah Hollmann, Frank Hutter
arXiv Machine Learning
Sep 17

MCLC-NET: Multimodal Continual Learning for Leaf Counting

The paper introduces MCLC‑NET, a multimodal continual learning framework for leaf counting that sequentially learns tasks using a memory buffer to retain key samples. It also presents MMLC, a new real‑world dataset containing RGB, depth, and thermal images across different crops and environmental conditions, organized in crop‑wise, time‑wise, and mixed orderings. Experiments show that MCLC‑NET outperforms existing methods on all three task orderings, achieving the lowest average mean squared errors.

By Ruchi Bhatt, Pratibha Kumari, Shreya Bansal, Vedant Agnihotri, Dwarikanath Mahapatra, Mukesh Saini
arXiv Computation and Language
Sep 17

TeleAntiFraud 2.0: A Refreshable, Profile-Grounded, and Audio-Based Benchmark for Telecom Fraud Detection

TeleAntiFraud 2.0 is a monthly‑frozen, audio‑based benchmark for telecom fraud detection that incorporates newly observed scam patterns while preserving earlier test sets. It uses a Mixed‑Tree Anti‑Fraud Generation Pipeline to create profile‑grounded scenarios, expands them into mixed‑tree dialogues, and renders validated speech for 900 Chinese calls (600 fraud, 300 near‑domain non‑fraud) each month. Experiments show that classifiers perform well against unrelated negatives but drop significantly against near‑domain negatives, highlighting the need for near‑domain construction and collapse‑aware reporting in realistic evaluation settings.

By Huiyuan Liu, Zhiming Ma, Yanxing Liu, Shun Zhang, Qifan Wang, Di Liu, Yifan Wang, Yuyang Deng, Haoyang Meng, Yijin Zhou, Yuxi Zhao, Chengxian Hu, Peidong Wang, Peng Chen
arXiv AI
Sep 17

Decodable but Misrouted: Sparse Features Uncover a Readout Gap in Vision-Language Models for Harmful Meme Detection

The paper investigates why large vision‑language models sometimes misclassify harmful memes, attributing failures to either missing internal evidence or poor routing of evidence to the output. Using sparse autoencoders, role‑conditioned probes, and causal interventions on Gemma‑3 and Qwen3.5, the authors show that sparse readouts consistently outperform native predictions across six harmful content benchmarks, revealing a readout gap that is largely due to routing rather than representation. The study also demonstrates that calibration‑only routing recovers most of the performance gap and that the issue persists across languages and is not solely driven by OCR signals.

By Girish A. Koushik, Diptesh Kanojia, Helen Treharne
arXiv AI
Sep 17

Label-free steering: Compressing test-time reinforcement learning into bias-only subspaces

The paper introduces label‑free bias‑only test‑time reinforcement learning (TTRL), which uses majority‑vote pseudo‑labels as rewards and optimizes only about 100 K bias parameters while keeping the pretrained backbone frozen. On the MATH‑500 benchmark it achieves 76.67 % accuracy, slightly better than a labeled bias‑steering baseline, and improves performance on several vision‑language and audio reasoning tasks. The authors also show that the learned steering vectors transfer to 4,500 held‑out MATH problems and analyze why such a highly restricted adaptation works, linking majority‑vote reliability to rollout consensus and gradient energy in bias subspaces.

By Naveen Vakada, Mingyuan Li, Shaoxiong Ji
arXiv AI
Sep 17

rMuscle: Robotic Muscle Memory for Efficient Vision-Language-Action Model Inference

The paper introduces rMuscle, a real‑time Vision‑Language‑Action inference framework that mimics human muscle memory to accelerate robotic decision making. By exploiting repeated task similarity, rMuscle uses a dual‑phase cache: a Context Cache reuses visual‑token outputs and an Action Cache reuses neuron activation patterns, reducing computation and weight accesses. Experiments on RTX 4090 and Jetson Thor show 1.29–1.42× speedups on LIBERO, RoboTwin, and physical manipulation tasks while preserving success rates on real robots.

By Kaijun Zhou, Zhiyang Li, Le Chen, Jinyu Gu