Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.
Data augmentation is fundamental to training modern deep vision and multimodal models. While individual methods, such as RandAug, CutMix, Mixup, RandErase, and DropPath, offer strong regularization ef...
Robust speech-to-text translation systems should perform reliably across diverse acoustic conditions, yet practical pipelines lack controllable tools for systematic environment exploration. Large spee...
Modern TTS systems approach human quality for high-resource languages but degrade when clean speech data is scarce. Modern Greek exemplifies this, lacking the curated corpora behind state-of-the-art s...
Malocclusion skeletal grading is a fundamental task in orthodontics, critical for diagnosis and treatment planning. Traditionally, cone-beam computed tomography (CBCT) is used for visual measurement,...
Large Multimodal Models (LMMs) large-scale deployment in industrial warehouse settings specifically necessitates that models exhibit human-expert-level hazard-oriented perception, understanding, and r...
Text-aligned speech tokenization methods have emerged to better align speech tokens with LLM token spaces, enabling more effective utilization of pretrained LLMs. However, they rely on offline automat...
The rapid advancement of audio and multimodal large language models has unlocked transformative speech understanding capabilities, yet evaluation frameworks remain predominantly English-centric, leavi...
Sixth-generation (6G) wireless networks are expected to provide a key infrastructure for large-scale embodied intelligence, where heterogeneous robots collaborate through low-latency connectivity, edg...
We introduce BuzzASR, a collection of language-specialized fine-tuned Whisper models adapted for automatic speech recognition (ASR) in 102 languages. Large end-to-end Transformer-based ASR models such...
The paper investigates image tokenizers as the visual language of unified multimodal models by creating a controlled autoregressive testbed that tracks task‑specific validation losses during multimodal continual pretraining across text, image, text‑to‑image, and image‑to‑text predictions. It shows that losses must be analyzed by task, that the loss–performance relationship varies with the token space, and that better reconstruction does not always lead to stronger downstream performance. The study also demonstrates how tokenizer design choices—such as discriminator use, semantic supervision, and vocabulary size—affect joint modeling and downstream results.
NOAH is a generative transformer that models the entire multimodal patient journey by integrating bidirectional time and a variational latent space to capture continuous, stochastic clinical trajectories. Trained on over 559 million events from 431,000 hospital visits, it processes medical images, time‑series, numeric signals, categorical events, and both structured and unstructured records. The model supports autoregressive forecasting, zero‑shot classification, and counterfactual simulations, yielding strong predictive performance across 15 ICD chapters, 29 comorbidities, and time‑to‑event outcomes.
The paper investigates how vision encoders and Vision‑Language Models (VLMs) encode conceptual information by using canonical color as a test case. By creating a dataset of objects with canonical colors and probing encoders with both color and grayscale images, the authors show that canonical color can still be decoded from grayscale inputs and is linked to predicted object identity. They further demonstrate that post‑training of VLMs can significantly influence color decodability within the vision encoder, suggesting that canonical color is a useful tool for tracing conceptual semantics in these models.
DeCAL is a vision‑language‑action model designed for dexterous manipulation that incorporates tactile sensing through adaptive visuo‑tactile fusion and latent co‑imagination. It uses a Mixture‑of‑Transformers architecture with specialized experts for understanding, imagination, and action, enabling efficient information flow and dynamic regulation of tactile inputs. Experiments show DeCAL achieves state‑of‑the‑art performance, with a 71% average success rate and 83.4% progress success rate, and generalizes well to unseen scenarios.
DXPR is a depth‑based cross‑modal place recognition framework that matches monocular camera queries to a LiDAR map using a single vision foundation model backbone. By converting both modalities into a unified depth image representation, DXPR learns modality‑invariant global descriptors without modality‑specific encoders. A geometry‑aware overlap miner refines pairwise metric learning by computing pixel‑level overlap scores, and extensive tests on KITTI and Boreas show strong performance across seasons, weather, and day/night conditions, outperforming prior CMPR baselines.
TontaubeV1 is a streaming text‑to‑speech model that maintains natural prosody while running on a single consumer GPU. It encodes speech with a hierarchical DualCodec representation at 12.5 Hz, separating a semantic stream from successive acoustic refinements, and uses Qwen3‑derived transformers to predict the semantic stream and add refinements. The model supports up to one minute of reference audio for voice conditioning, streams with a 200 ms latency to first audio, and achieves real‑time factors of 0.08 (single input) and 0.02 (eight concurrent inputs).
Embodied planning increasingly relies on vision-language models (VLMs) to translate instructions and visual observations into executable action sequences. However, fluent plans are not always executab...
The paper investigates zero‑shot out‑of‑distribution (OOD) detection in medical imaging using vision‑language models (VLMs). It shows that intermediate layers, rather than only the final layer, provide valuable OOD signals and that the best layer depends on the imaging modality. To overcome instability in entropy‑based layer selection, the authors introduce a multi‑resolution entropy estimation that aggregates histogram statistics across scales, achieving consistent improvements over state‑of‑the‑art methods on the MIDOG and OASIS benchmarks.
The paper introduces 3DWay, a method that predicts 3D consistent waypoints for robot manipulation using multi‑view images. By first generating 2D waypoints that are consistent across views and then triangulating them, the approach provides explicit 3D motion specifications while leveraging pretrained vision‑language models. Experiments demonstrate that 3DWay improves 3D spatial grounding and vision‑language reasoning, enhancing the generalization of robot manipulation policies.
The paper introduces AeroBelief, a dual‑layer semantic‑spatial belief mapping framework for aerial object goal navigation. It separates broad contextual plausibility (intuition layer) from target‑specific evidence (evidence layer) and fuses them into persistent spatial belief hotspots, while also employing object‑conditioned visual reasoning and temporally stable regional guidance. Experiments on the UAV‑ON benchmark show AeroBelief outperforms prior methods in success rate, object success rate, and SPL.
The article outlines the emerging field of Magnetoencephalography (MEG) foundation models, explaining how these reusable, pretrained models can surpass traditional task‑specific decoding pipelines. It reviews current design choices—such as tokenization, sensor versus source representations, and self‑supervised objectives—and notes the limited number of existing MEG‑specific models and datasets. The authors propose a roadmap that includes native MEG pretraining, adaptation of EEG models, transfer from generic time‑series models, and multimodal integration with other neuroimaging and behavioral data, while emphasizing the need for coordinated infrastructure, rigorous evaluation, and responsible data‑sharing practices.