Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.
arXiv:2608.00239v2 Announce Type: replace Abstract: Learning transferable representations from CT-report pairs requires combining whole-volume context with anatomy-specific evidence. Existing methods...
arXiv:2608.00231v2 Announce Type: replace Abstract: Volumetric CT vision-language pretraining learns 3D representations from scan-report pairs, but global and anatomy-aware objectives supervise only...
arXiv:2603.16130v5 Announce Type: replace Abstract: Overexposure caused by strong daylight and oncoming headlights frequently overwhelms visible sensors, resulting in critical information loss in vis...
ShotFinder introduces a new benchmark for open‑domain video shot retrieval, formalizing editing requirements as keyframe‑oriented shot descriptions and adding five controllable constraints—temporal order, color, visual style, audio, and resolution. The benchmark comprises 1,210 high‑quality YouTube samples across 20 themes, generated with large models and verified by humans. A three‑stage retrieval pipeline—query expansion via video imagination, candidate video retrieval, and description‑guided shot localization—shows a notable performance gap to humans, especially for color and visual style constraints.
The paper introduces the Wide-area Spatio-temporal Scene Understanding (WSTU) problem, which demands simultaneous wide-area coverage, per-target resolution, and temporal continuity—capabilities lacking in existing datasets. To address this, the authors present HARD, an ultra‑high‑resolution (12768×9564) UAV dataset annotated for object detection, multi‑object tracking, and scene‑level visual question answering. They also propose a latency‑aware metric, streaming‑HOTA (s‑HOTA), and show through baseline experiments that high resolution and processing latency significantly impact detection, tracking, and VQA performance, revealing gaps in current methods for WSTU.
ActionPiece rethinks how actions are tokenized for autoregressive vision‑language‑action models by introducing physical rank consistency (PRC) to evaluate relational fidelity of reconstructed actions. The method jointly supervises representation learning and quantization to preserve local physical distance rankings, improving both PRC and policy success. Experiments on LIBERO, LIBERO‑Plus, SimplerEnv, and VLA‑Arena show significant gains over baseline tokenizers.
The paper demonstrates that a single-pass multimodal model struggles to produce faithful, comprehensive reviews of long recordings or documents, often omitting a third of the content and embellishing the rest. By splitting the task into two passes—first transcribing the source and then reviewing the transcript—the authors show improved faithfulness and coverage across a diverse set of 21 sources. The benefit is most pronounced for longer or weaker baseline cases, while the approach introduces new failure modes such as space constraints and memory confabulation.
MIRAGE is a controlled study that examines how multimodal personal agents use historical evidence when conversation state changes. The study keeps evidence, questions, and scoring constant while varying only the conversation state, then checks if agents can determine answerability, recover the correct source, and answer from it. Results across seven multimodal backbones show distinct failure regimes before and after compaction, heavy reliance on context continuity by open-weight models, and mixed effects of retrieval pressure on source attribution.
The paper introduces LM Fight Arena, a new benchmark that pits large multimodal models against each other in the fighting game Mortal Kombat II to evaluate real‑time visual understanding and sequential decision‑making. Six leading open‑ and closed‑source models were tested in a controlled tournament where each controlled the same character, ensuring a fair comparison. The framework offers a fully automated, reproducible, and objective assessment of an LMM’s strategic reasoning in a dynamic setting.
The paper presents a generative framework that estimates category-level 6D pose and 3D size of objects from a single RGB image, using score-based diffusion models to produce a multi-hypothesis pose distribution. It replaces costly likelihood pruning with a Mean Shift approach to isolate the mode as the final pose estimate, achieving state-of-the-art results on the REAL275 benchmark. The method also decouples detection from pose estimation, enabling robust zero-shot generalisation on the Wild6D dataset and extending naturally to video sequences by propagating the pose distribution over time.
NeMo Data Designer (NDD) is an open‑source framework for generating multimodal synthetic data. It uses a declarative configuration format that lets users define dataset columns—text, code, structured outputs, images, embeddings, and statistical samplers—to steer diversity. The system supports a preview‑and‑revision workflow, dependency resolution, and retry logic, and can be extended via plugins. Case studies demonstrate its use for structured, agentic, multimodal, and domain‑specialized tasks, including datasets for Nemotron model development and enterprise deployments.
WetRobo is a reproducible robot kit designed to enable wet‑lab researchers to delegate tasks to coding agents without teleoperation or neural‑network training. The kit includes a robot arm, essential lab equipment, pre‑recorded teleoperation demos, and a skill file, allowing a coding agent to observe the lab, write, and execute programs using external tools. Experiments with OpenAI Codex on tasks such as lifting a Petri dish lid, removing a bottle cap, and opening an incubator door demonstrated successful performance in two different laboratories, outperforming a fine‑tuned vision‑language‑action policy that failed to transfer.
The paper introduces "cunning questions"—non‑safety prompts that contain misleading premises or subtle inconsistencies—to train large language models (LLMs) to scrutinize underlying intent and assumptions. Experiments show that incorporating these questions improves robustness against out‑of‑distribution jailbreak attacks and enhances subsequent safety fine‑tuning, achieving a new state‑of‑the‑art reduction in mean ASR from 17.40% to 15.05% across nine backbone–benchmark combinations. The authors argue that this training fosters vigilance, enabling models to prioritize safety judgments before engaging in harmful planning.
The paper introduces AWED, a word‑level emission‑delay metric, and demonstrates that post‑training a delayed‑streaming model with a joint reward (GRPO) improves both accuracy and latency. Using a single operating point (τ=6 frames), the method outperforms both its supervised baseline and the Voxtral Realtime backbone across all lookahead budgets, reducing WER by up to 30.8% at 80 ms delay and lowering median AWED from 1.17 s to 1.04 s.
The paper introduces Mask 2D-3D, an Adaptive Dual-Masked Autoencoder Network designed for image-to-point cloud registration. It proposes an Intermodal Dual-MAE Framework (ID-MAE) with a Similarity-based RL Masking Strategy (SRLM) that adaptively masks informative positions using cross-modal similarity and reinforcement learning. Experiments on RGB-D Scenes v2 and 7-Scenes benchmarks demonstrate state-of-the-art performance in this registration task.
The paper surveys Generative Physical Artificial Intelligence (GPAI), a field where large foundation models are integrated with physical robots. It introduces a taxonomy of five approaches—Robot Foundation Models, Vision‑Language Action models, Large Behavior Models, Diffusion Policy Models, and World Foundation Models—and discusses how they complement each other across domains such as autonomous vehicles, industrial automation, healthcare robotics, and humanoid systems. The review highlights performance gains, data‑efficient learning, sim‑to‑real transfer, edge‑compatible architectures, and safety frameworks as key research directions.
The paper introduces Contrastive Noise Alignment (CNA), a training-time method for generative flow models that dynamically aligns Gaussian noise with data samples using a cross-modal InfoNCE objective. By modeling noise as an interacting particle system and regularizing with angular entropy and radial norm penalties, CNA reduces arbitrary data-noise couplings and flow curvature. Empirical results show that CNA improves generation quality, lowering FID by over 50% for few-step pixel-space generation compared to standard rectified flow and outperforming optimal transport baselines by at least 24%.
The paper extends Mixed-Integer Differentiable Predictive Control (MI‑DPC) to handle multi‑modal discrete decisions and nonconvex polynomial dynamics in Underground Pumped Hydro Energy Storage Systems (UPHES). It introduces a neural policy that maps problem parameters to continuous setpoints and integer mode selections using a Gumbel‑Softmax layer, trained via self‑supervised differentiation of the finite‑horizon control objective. Key methodological advances include a parallel differentiable simulator, a Transformer encoder for long‑range temporal dependencies, and a Gumbel‑Softmax temperature annealing schedule, enabling the framework to achieve only 1.6% suboptimality compared to a mixed‑integer quadratic programming baseline while providing a five‑order‑of‑magnitude speedup in online scheduling.
The report introduces TabPFN‑3.5, a new flagship tabular foundation model that outperforms its predecessor TabPFN‑3 and all existing baselines on a wide range of tabular tasks. It achieves state‑of‑the‑art performance on standard tabular prediction in TabArena and extends to practical scenarios such as non‑i.i.d. data, temporal or grouped splits, tables containing strings, text, images, high‑cardinality categorical features, and wide tables. Variants like TabPFN‑3.5‑Fast, TabPFN‑3.5‑Plus, and TabPFN‑3.5‑Thinking offer faster inference, expanded multimodal capabilities, and further speed improvements up to 12× faster than the previous Thinking mode.
The paper introduces MCLC‑NET, a multimodal continual learning framework for leaf counting that sequentially learns tasks using a memory buffer to retain key samples. It also presents MMLC, a new real‑world dataset containing RGB, depth, and thermal images across different crops and environmental conditions, organized in crop‑wise, time‑wise, and mixed orderings. Experiments show that MCLC‑NET outperforms existing methods on all three task orderings, achieving the lowest average mean squared errors.