Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.
arXiv:2608.23921v1 Announce Type: new Abstract: Recent Vision-Language Models encode high-resolution images into long visual token sequences, incurring prohibitive prefill costs. To compress them, ex...
The paper introduces the Boot-and-Feedback (BooF) framework, which facilitates collaboration between multimodal large language models (MLLMs) and expert vision models for breast ultrasound (BUS) diagnosis. In the Boot Stage, the MLLM is guided by the BI-RADS lexicon and preliminary expert predictions to generate reliable textual descriptions, while the Feedback Stage fuses these descriptions with visual features using an Attention-Gated Cross-Modality Fusion Module, allowing the expert model to incorporate textual insights and filter out noise. Experiments on multiple BUS datasets show that BooF improves both diagnostic accuracy and interpretability compared to existing methods.
arXiv:2608.23723v1 Announce Type: new Abstract: Few-shot anomaly detection (FSAD) has recently benefited from vision-language models such as CLIP, which enable anomaly de?tection by aligning visual f...
arXiv:2608.23869v1 Announce Type: new Abstract: While state-of-the-art generative models produce high-fidelity 3D meshes, these outputs lack the physical properties required for interactive simulatio...
The paper introduces a new evaluation protocol for deep multivariate imputation models on wearable device data, addressing the issue of structured missingness where sensor features drop out together. Using a Garmin smartwatch dataset from an epilepsy patient, the authors generate realistic block-missing patterns from training data and show that matching the training protocol to this distribution reduces BRITS’ mean absolute error by 43%. They also extend BRITS with time‑of‑day encoding and compare it to linear interpolation and SAITS, finding that no single model dominates and that model rankings vary with evaluation design.
The paper evaluates the tablet‑2 long‑term memory engine on multilingual text benchmarks and cross‑lingual photo retrieval without lexical matching. Tablet‑2 achieves high accuracy on LongMemEval‑S (95.7%) and moderate accuracy on BEAM‑1M (67.5%), with minimal variance across runs. In multimodal tests, it outperforms BM25 on image‑cell recall and shows significant language‑dependent performance gaps, especially for low‑resource languages.
The paper surveys the evolution of smart glasses from simple capture devices to first‑person intelligence platforms that integrate human perception, context, and action. It introduces a unified framework that formalizes data flow, hardware capabilities, and seven foundational capabilities, and presents an L0‑L5 hierarchy for capture to embodied action. The study also maps nine application scenes, proposes a nine‑dimensional deployment framework, and outlines an evidence ladder for evaluation and trustworthiness.
The paper introduces DoublesEval, a diagnostic framework that uses professional doubles badminton to test visual‑language models’ ability to reason about dynamic multi‑agent interactions. It decomposes rallies into key moments and evaluates models across four dimensions—atomic recognition, intra‑segment composite understanding, cross‑segment causal reasoning, and high‑level tactical abstraction—highlighting specific reasoning failures. The authors also propose TacticCheck, a lightweight consistency checker that improves performance without retraining the models, yet significant gaps remain in tactical reasoning.
arXiv:2608.24763v1 Announce Type: cross Abstract: Procedural video-language models must solve heterogeneous tasks from the same visual evidence, including action recognition, forecasting, and procedu...
RT-NeuS is a neuro‑symbolic framework for long‑form video question answering that retains the accuracy and formal guarantees of temporal‑logic‑guided methods while dramatically reducing inference latency. It achieves this by using coarse‑to‑fine adaptive sampling to focus on query‑relevant frames and batched proposition detection with KV‑cache reuse, enabling all propositions to be evaluated in a single forward pass. Experiments on LongVideoBench, Video‑MME, and MLVU show up to a 13× speed‑up on an NVIDIA H200 GPU while matching or surpassing prior neuro‑symbolic accuracy.
The paper presents a systematic study of scientific image synthesis, comparing pixel‑based generation and programmatic approaches. It introduces ImgCoder, a logic‑driven framework that follows an "understand‑plan‑code" workflow to enhance structural precision, and SciGenBench, a benchmark that evaluates images for information utility and logical validity. The authors find that pixel‑based models exhibit systematic failure modes and that fine‑tuning large multimodal models on rigorously verified synthetic images consistently improves downstream reasoning performance.
MolEmb is a lightweight framework that adapts multimodal large language models (MLLMs) to serve as general molecular embedding models. By aligning molecular profiles with textual descriptions in a shared embedding space using a bidirectional contrastive objective, MolEmb produces embeddings conditioned on both a molecular profile and a natural‑language semantic context. The model performs competitively on molecular property prediction and enables cross‑modal molecule‑text retrieval, while the newly introduced MolCAR benchmark demonstrates that context‑aware molecular embedding is largely a data property of the supervision.
Prompt learning modifies vision‑language models by optimizing continuous prompt vectors, yet the resulting prompts are hard to interpret in natural language. PromptSpLiCE is a post‑hoc method that rewrites each class‑conditioned text embedding as a sparse mix of concepts from a fixed dictionary, enabling a direct comparison of concept profiles before and after prompt learning. Across 11 image‑classification datasets, the method shows that only about 1.6 of the initial top‑10 concepts remain after learning, and that larger profile changes correlate with higher accuracy gains, while a derived gradient expression offers geometric insight into loss sensitivity.
arXiv:2608.24795v1 Announce Type: new Abstract: Recently, the rapid advancement of multimodal domains has driven a data-centric paradigm shift in graph ML, transitioning from text-attributed to multi...
Luce is a 3D representation that unifies geometry and physically based rendering (PBR) materials within a voxelized multimodal Gaussian cloud, using dedicated Gaussian primitives for each modality. A variational autoencoder compresses this representation into a unified material‑aware latent space, which a rectified‑flow transformer generates from a single image conditioned on multi‑layer features from a pretrained image encoder. The latent decodes into relightable PBR Gaussians and an optional textured mesh with a tangent‑space normal map, achieving state‑of‑the‑art single‑image‑to‑3D generation on Toys4K and improving CLIP image‑alignment scores on a benchmark of AI‑generated images.
The paper tackles the challenge of predicting student engagement from online tutoring videos, noting that engagement is a complex, multidimensional construct influenced by behavioral, emotional, and cognitive states. By analyzing the CASED dataset, the authors highlight the difficulty posed by high inter‑person variability and subjective annotations. They propose a multimodal framework that fuses implicit spatiotemporal features from pretrained video, audio, and image encoders with structured behavioral cues such as head pose, gaze, facial action units, emotion, and wavelet‑based audio features, integrating them via a Perceiver IO bottleneck and modeling participant personalities with variational posteriors. The system employs evidential regression and spectral‑normalized Gaussian process classification heads to provide uncertainty‑aware predictions, achieving competitive performance on the CASED challenge test set while offering well‑calibrated uncertainty metrics.
LUCAID is an agentic multimodal AI system designed for precision lung cancer pathology, integrating nine modules that cover the entire routine workflow—from quality control and tumor detection to histological subtyping, microenvironment profiling, cellularity quantification, and biomarker scoring (PD‑L1, MET, TROP‑2). The system generates automated structured reports and allows interactive querying of module outputs. In prospective clinical validation, LUCAID achieved 93.0% concordance with an expert‑panel reference standard for clinically actionable decisions, outperforming five experienced thoracic pathologists who ranged from 68.3% to 81.1% concordance.
arXiv:2406.05375v4 Announce Type: replace Abstract: Root cause analysis (RCA) is crucial for enhancing the reliability and performance of complex systems. However, progress in this field has been hin...
The paper investigates whether joint language‑audio embedding models encode human perceptual timbre semantics. It evaluates several state‑of‑the‑art models, finding that LAION‑CLAP aligns best with human‑perceived timbre across instrumental sounds and descriptor‑conditioned audio effects, yet the overall alignment remains limited. The study also notes that reverb‑induced timbre semantics are more consistently captured than equalization‑induced ones.
arXiv:2608.24138v1 Announce Type: new Abstract: Large vision-language models have shown strong progress in UI-to-code generation, yet their test-time self-evolution remains unstable. We first identif...