arXiv AI
Jul 28
Like a Baby: Visually Situated Neural Language Acquisition
arXiv:1805. 11546v3 Announce Type: replace-cross Abstract: We examine the benefits of visual context in training neural language models to perform next-word prediction.
Multimodal models
Vision-language models, speech and cross-modal systems that read, look and listen in the same forward pass.
arXiv AI
Jul 28
MarineEVT: Advancing Event-Centric Marine Video Understanding via Visual Tool Reasoning
arXiv:2607. 24064v1 Announce Type: cross Abstract: Recent Vision-Language Models (VLMs) have achieved remarkable success in visual understanding, driven by the growing availability of high-quality image-text pairs.
arXiv AI
Jul 28
WCM: World-Cognition Model for Generalizable Human-Robot Interaction
arXiv:2607. 22999v1 Announce Type: cross Abstract: Language agents can now interact fluently with users in software, but robots still struggle to bring comparable interaction to physical tasks.
arXiv AI
Jul 28
The Visual Bottleneck: Sparse-Frame Adaptation of MLLMs for Joint Spatial-Temporal Video Grounding
arXiv:2607. 24570v1 Announce Type: cross Abstract: Large-scale video platforms process millions of uploads hourly, requiring moderation systems that can localize when and where policy violations occur within each video.
Hugging Face Trending Papers
Jul 27
ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding
Multimodal large language models (MLLMs) hold immense potential to revolutionize clinical practice, yet deploying them in the medical domain is fundamentally a vision-centric challenge: models must absorb knowledge from heterogeneous 2D and 3D medical images, and evaluation protocols must align with radiologists' clinical practice and provide an accurate, fine-grained and factualness-driven assessment. In this paper, we introduce ClinFusion, a vision-centric MLLM designed for holistic medical understanding that systematically addresses these limitations.
Hugging Face Trending Papers
Jul 27
DSCH-Loss: A Dynamic Semantic Channel Objective for Deep Semantic Hashing
Semantic hashing methods for generating short binary hash codes that allow efficient approximate nearest neighbor search in high-dimensional data spaces have gained extensive consideration in recent years. Deep learning-based methods offer better semantic capturing capabilities than traditional approaches relying on manual feature engineering.
Hugging Face Trending Papers
Jul 27
EgoPlay: Event-Triggered Video Editing for Egocentric Streams
We introduce EgoPlay, an event-triggered video-to-video editor for egocentric streams, obtained by fine-tuning a pretrained V2V diffusion transformer on event-conditioned data built primarily from Ego4D. Given a monocular video and an event-triggered prompt of the form "when X happens, do Y," EgoPlay infers whether and when event X occurs, preserves pre-event frames, and applies edit Y only to the post-event continuation.
Hugging Face Trending Papers
Jul 27
MAViE: A Multi-scale Adaptive Vision Encoder for Fine-grained Visual Perception and Efficient Multimodal Reasoning
Vision-language models commonly project all tokens produced by a pretrained vision encoder into a large language model. However, final-layer features can discard text, local attributes, and spatial relationships, while high-resolution inputs substantially increase context length and inference latency.
Hugging Face Trending Papers
Jul 27
FilmBench: A Film-Grade Benchmark for Cinematic Video Generation
Progress in video generation keeps narrowing the visual gap between AI-generated and professionally produced footage, yet most benchmarks still draw prompts from web sources or LLM templates and score them with untrained, generic multimodal models. More fundamentally, their evaluation taxonomies remain rudimentary (overall visual quality, coarse text alignment and temporal smoothness) rather than the professional Cinematic Language criteria by which films are actually made and judged, so they assess basic video plausibility rather than film-grade craft.
Hugging Face Trending Papers
Jul 27
MATS: A novel multi-modality multi-task learning framework for 3D perception in autonomous driving
Multi-modality data from different sensors provides rich complementary information for 3D perception, becoming an essential component in reliable autonomous driving systems. Current research typically designs intricate and complex fusion strategies to integrate information from multimodal data on a unified bird's-eye-view (BEV) feature map for the joint learning of multiple perception tasks.
Hugging Face Trending Papers
Jul 27
DeVA: Decoupled Video-Action Model with physical guidance for robot policy learning
Generalizable robot manipulation requires policies that can anticipate how visual scenes evolve while executing language instructions. While recent Vision-Language-Action models benefit from large-scale pretraining, their predominantly static pretraining objectives provide limited supervision for physical dynamics and temporal causality, leaving control-relevant knowledge to be learned from downstream robot demonstrations.
Hugging Face Trending Papers
Jul 27
UniGen-AR: Unifying Visual Generation with Auto-Regressive Modeling
Modern computer vision pipelines remain fragmented, with tasks such as text-to-image generation, editing, restoration, and classical perception handled by separate models. We study Unified Visual Generation (UVG), where a single model produces diverse image-valued outputs through a unified multimodal interface.
Hugging Face Trending Papers
Jul 27
A Motion-Aware Vector Quantization Framework with Centroid Reuse for Efficient VLA Inference
Vision-Language-Action (VLA) models have demonstrated strong potential for embodied AI, yet their high inference latency on GPUs limits real-time deployment. Existing accelerators, such as Dadu-Corki, improve efficiency but treat VLA models as full-precision workloads, leaving substantial redundancy in both memory and computation underexploited.
Hugging Face Trending Papers
Jul 27
Scaling GUI Agents with Visual State Transitions
We introduce State Transition Pretraining (STP) as a new scaling axis for GUI agents. During the STP stage, we continually pretrain a unified multimodal model on visual state transitions by jointly optimizing inverse dynamics (predicting actions from state changes) and forward dynamics (predicting next states from current states and actions).
Hugging Face Trending Papers
Jul 27
BeyondFusion: Self-Aligned Latent Diffusion for Calibration-Free Infrared Super-Resolution and Infrared-Visible Fusion
Mobile infrared-visible imaging typically pairs a compact infrared sensor with a high-resolution visible camera for complementary perception. While cross-sensor misalignment caused by different optics, viewpoints, fields of view, and exposure timings hinders practical deployment.
Hugging Face Trending Papers
Jul 27
Towards High-Level Semantic Intelligence
Recent advances in AI have substantially expanded its cognitive and reasoning capabilities. From the perspective of semantic complexity, the development of AI reveals a clear trajectory from simple to complex semantic processing.
Hugging Face Trending Papers
Jul 27
FinAbstain: Uncertainty-Calibrated Multimodal RAG for Selective Financial Forecasting
Large language models (LLMs) can synthesize financial narratives but may express high confidence when evidence is sparse, stale, or contradictory. This failure is especially consequential in forecasting, where filings, news, prices, volume, and technical signals can disagree.
Hugging Face Trending Papers
Jul 27
Multimodal Semantic-Probabilistic Objectness for Open World Object Detection
Open-world object detection (OWOD) requires a detector to recognize known categories, discover unnamed objects from unseen categories, and incrementally learn newly annotated classes. PROB improves unknown discovery by modeling class-agnostic probabilistic objectness in the decoder-query space.
arXiv Machine Learning
Jul 27
Autoregressive EHR Foundation Models with Multimodal Inputs
arXiv:2607. 22264v1 Announce Type: new Abstract: Autoregressive foundation models trained on tokenized electronic health records (EHRs) can support zero-shot clinical prediction, yet most operate on structured event codes alone, and do not incorporate multiple modalities in a principled way.
arXiv Machine Learning
Jul 27
FBLayout: Optimizing Memory Layout for Efficient LLM Finetuning on Mobile GPUs
arXiv:2607. 21624v1 Announce Type: cross Abstract: Transformer-based models have enabled unprecedented capabilities across language, vision, and multimodal tasks.