Large language models

Model releases, architecture work and prompting research on large language models — from frontier-lab announcements to the arXiv papers behind them.

31,091 stories · RSS feed

arXiv Computer Vision
Sep 23

The Temporal Moderation Gap: Text-to-Video Safety Filters Are Blind to Harm in Motion

The paper identifies a flaw in current text-to-video safety filters that rely on keyword prompts and per-frame checks, showing that these systems miss harms that arise solely from the ordering of frames. Experiments reveal that a significant portion of generated videos contain harmful content that is undetected because each individual frame appears safe, and prompt engineering does not mitigate this issue. The authors propose an order-aware detector that can distinguish harmful videos from benign shuffles, achieving an AUC of 0.74 compared to chance for per-frame checking.

By Yuxin Cao, Fusen Guo, Yuezhong Wu, Huadong Mo, Wei Song
arXiv Computer Vision
Sep 23

From Token Importance to Conditional Removability: Rethinking Visual Token Pruning in Multimodal Large Language Models

The paper argues that token importance alone is insufficient to determine safe removal of visual tokens in multimodal large language models, because removability depends on representation depth and the surrounding deletion set. Through controlled experiments, the authors show that the same tokens can have different effects when removed at different depths or contexts. They introduce CoRePrune, a training‑free two‑stage pruning framework that refreshes deletion effects as visual representations evolve and refines candidate tokens based on the current deletion set, achieving high performance retention across multiple backbones and reducing prefill time significantly.

By Shengli He, Yongchao Liang, Roumeng He, Junjie Zeng, Jiyuan He, Xin Fang, Can Wu, Li Zheng
arXiv Computer Vision
Sep 23

Semantically-Guided Domain Randomization for Industrial Object Detection in Low-Image-Budget Regimes

Semantically-Guided Domain Randomization (S‑GDR) is an annotation‑free pipeline that uses vision‑language model captioning of a small real reference set, diffusion‑based background synthesis, and mask‑based object composition to generate synthetic training data. In a high‑mix, low‑volume automotive detection benchmark, S‑GDR achieves a mAP50‑95 of 0.739 with only 200 synthetic images, outperforming a domain‑randomized render baseline and several other synthetic data methods under the same budget. These results suggest S‑GDR is a viable alternative for training visual perception systems when annotation, energy, and time resources are severely limited.

By Jose Moises Araya-Martinez, Gautham Mohan, Jens Lambrecht
arXiv Computer Vision
Sep 23

Virtual Encoders in Multimodal Transformers

The paper investigates how multimodal language models can generate perceptual representations without dedicated encoders. It shows that the shared transformer can internally create these representations in its early-to-middle layers, a structure termed a Virtual Encoder. Experiments with linear probing, similarity metrics, and causal analysis reveal that this encoder-like computation emerges even when models receive only perceptual tokens, indicating that perception and language processing can be decoupled within a single architecture.

By Katsuya Ogata, Yuta Nakashima
arXiv Computer Vision
Sep 23

HARMONY: Hierarchical Agentic Reasoning for MONocular Image-to-Scene Synthesis

HARMONY is a hierarchical chain-of-thought framework that reconstructs complete 3D indoor scenes from a single monocular image. It combines agentic reasoning with visual geometry foundation models, starting with camera calibration and semantic layout recovery, then placing objects hierarchically while refining geometry with point cloud estimations. The method achieves semantically consistent scenes that align perceptually with the input image, outperforming existing baselines on synthetic and real-world data.

By Shufan Sun, Chen Wang, Enxin Song, Jiatao Gu, Lingjie Liu
arXiv Computer Vision
Sep 23

NPLSD: Accelerating Line-Segment Detection on NPU Microcontrollers

The paper introduces NPLSD, a pair of line‑segment detectors optimized for the Neural‑ART NPU on STM32N6 microcontrollers. NPLSD‑H retains a convolutional backbone and replaces transformer components with a fully‑convolutional head, while NPLSD‑M adapts a lightweight trunk to the NPU’s supported operators. Trained on ImageNet and ShanghaiTech Wireframe, the models achieve competitive accuracy with 2.63 M and 0.62 M parameters, respectively, and ablation studies show initialization contributes significantly to performance.

By Parsa Hassani Shariat Panahi, Amir Hossein Jalilvand, M. Hassan Najafi
arXiv Computer Vision
Sep 23

TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Action

arXiv:2505.01583v2 Announce Type: replace Abstract: Understanding causal event relationships and achieving fine-grained temporal grounding in videos remain challenging for vision-language models (VLM...

By Jen-Hao Cheng, Yi-Hao Peng, Huapeng Zhou, Vivian Wang, Huayu Wang, Hsiang-Wei Huang, Wenhao Chai, Hou-I Liu, Kuang-Ming Chen, Cheng-Yen Yang, Yi-Ling Chen, Vibhav Vineet, Qin Cai, Jenq-Neng Hwang
arXiv Computation and Language
Sep 23

MultiViewDx: Evidence-Linked Multi-View Clinical Diagnosis

MultiViewDx is a physician‑validated multimodal instruction dataset that links medical imaging studies with patient context and normalizes heterogeneous reports into an evidence‑linked workflow (evidence → findings → differential discussion → diagnosis). The dataset covers a wide range of imaging modalities and uses a unified image‑text retriever to ensure that instruction synthesis is grounded in source‑supported evidence. Fine‑tuned models on MultiViewDx achieve the highest average accuracy on four MedVQA benchmarks and receive the strongest overall rating on JAMA Clinical Challenge cases, with ablations confirming the importance of case‑level multi‑view organization and evidence‑linked reasoning.

By Junda Wang, Zonghai Yao, Yujan Ting, Eric Z. Chen, Hieu Tran, Hong Yu, Weijing Huang, Terrence Chen