Large language models

Model releases, architecture work and prompting research on large language models — from frontier-lab announcements to the arXiv papers behind them.

31,046 stories · RSS feed

arXiv AI
Sep 28

Coding Agents Aren't Enough! Evaluating an Enterprise Security Brain for Agentic Cloud Investigations

The article evaluates the Sola Security Brain, a purpose-built security intelligence layer, against a general-purpose coding agent (Claude Code) on 28 cloud‑security investigation tasks. The Sola Security Brain achieved 0.693 coverage versus 0.387 for the coding agent, a 79.2% relative gain, and outperformed the agent on 25 of 28 tasks while incurring far lower reasoning and cost per unit of coverage. The study also identifies a ‘sample‑and‑generalise’ pattern where the live agent reports universal negatives based on limited sampling, illustrating a potential efficiency trade‑off in cloud investigations.

By Leon Goldberg, Gal Engelberg, Eden Yavin, Elad Elouz, Ariel Zadok, Konstantin Koutsyi
arXiv AI
Sep 28

Strategic Self-Consistency

The paper "Strategic Self-Consistency" investigates how large language model providers might exploit the self‑consistency technique—generating multiple reasoning paths and selecting the majority answer—to overcharge users. The authors present a simple, efficient algorithm that strategically generates and reorders extra reasoning paths so that each appears necessary for the majority vote, thereby evading detection by auditors. Experiments on Llama, Qwen, and DeepSeek-R1 models across math, science, and QA benchmarks show that the added paths follow a heavy‑tailed distribution and that significant overcharging can persist even under stringent audits with a false‑positive rate below 0.1.

By Tori Qiu, Ander Artola Velasco, Manuel Gomez-Rodriguez
arXiv AI
Sep 28

Cost-Aware Best-LLM Identification using Dueling Feedback

The paper introduces a new variant of the multi‑armed bandit problem that incorporates dueling feedback—pairwise comparisons of model responses—and heterogeneous sampling costs to identify the best large language model (LLM) from a set with varying query costs. Assuming a Condorcet winner, the authors propose a Track‑and‑Stop style algorithm that guarantees asymptotically optimal cost as the error probability approaches zero. Extensive experiments on synthetic and real‑world data show that this cost‑aware approach consistently outperforms both classical cost‑unaware algorithms and other cost‑aware extensions.

By Sarvesh Gharat, Nikhil Karamchandani, Jayakrishnan Nair
arXiv AI
Sep 28

Actively Resolving Contextual Uncertainty for Underspecified Tasks in Natural Language

The paper introduces CLUE, a framework that lets robots actively resolve contextual uncertainty for underspecified natural language tasks. CLUE employs an LLM-derived policy to generate task-relevant hypotheses and plans, then uses an online language-embedded map to ground these into actions, refining its plan through closed-loop interaction. Experiments on a Boston Dynamics Spot across diverse indoor and outdoor settings show CLUE achieving near-oracle performance and outperforming LLM planners without closed-loop feedback by a significant margin.

By Zachary Ravichandran, Jonathan Diller, Fernando Cladera, Varun Murali, George J. Pappas, Vijay Kumar
arXiv AI
Sep 28

A Benchmarking Framework for Context-aware XR Interfaces

The paper introduces ContextXR, a benchmarking framework designed to evaluate context‑aware interfaces in Extended Reality (XR) systems. It models an XR application as a graph of functional facets—semantically coherent groups of capabilities that support a shared user intent—and uses this representation to build MineXR++, a dataset enriched with facet‑level annotations. The framework defines three core tasks—context factor analysis, initial facet suggestion, and next facet suggestion—and evaluates suggestion methods via a simulated interaction metric that measures navigation and search cost to reach desired functionality.

By Hyunsung Cho, Sarah Yewon Yun, Nancy Ruonan Sun, Ben Lafreniere, Mark Parent, Kashyap Todi, Tanya R. Jonker, Hrvoje Benko, Sherry Tongshuang Wu, David Lindlbauer
arXiv AI
Sep 28

Breaking Homogeneity: Diversifying Persona Sets for Creative LLM Outputs

The paper introduces persona diversification as a set‑level conditioning strategy to reduce homogeneity in large language model outputs. It explores two design axes—selecting versus generating personas and space‑filling versus frontier‑seeking diversity—implementing four methods that span coverage and dispersion subset selections, uniform‑coverage sampling, and evolutionary persona generation. Experiments on tasks such as the Alternative Uses Task, Infinity‑Chat, and Divergent Association Task demonstrate significant gains in response diversity, originality, flexibility, and overall creativity, while maintaining high validity.

By Sang Bin Moon, Nicole Cho, Daniel Borrajo, Sumitra Ganesh, Abolfazl Hashemi
arXiv AI
Sep 28

PolicyAttention: Softmax Attention Implements Policy Mirror Descent for Closed-Loop Control

The paper investigates whether causal softmax attention can realize policy mirror descent (PMD) as a repeated controller rather than a one‑step algebraic identity. It constructs a fixed causal‑softmax actor–environment–one‑step‑critic protocol, detailing actor, routing, sampling, and normalization residuals, and shows that a frozen one‑step audit model closely approximates PMD. Empirical results demonstrate that the learned actor with an exact one‑step critic achieves median policy loss only about 5% higher than the exact PMD oracle across multiple control settings.

By Yuhe Sui, Yingzhi Tang, Shufang Chen
arXiv AI
Sep 28

The Hard Part Comes After Search: Benchmarking Web Agents on Synthesizing, Organizing, and Displaying Knowledge

The paper introduces KNOWS, a benchmark for evaluating web agents that act as assistants by retrieving, synthesizing, and presenting information across complex, multi-step browser tasks. It outlines a task design rubric, evaluation protocol combining deterministic checks with LLM judgments, and reports that current agents achieve only modest success, with the best performing agent succeeding on less than 3% of tasks. The study highlights significant gaps in agents’ tool use, visual understanding, and long‑horizon reasoning.

By Alexander Gill, Md Farhan Ishmam, Xuyen Nguyen, Neha Bhat, Parker Henry DeYoung, Fateme Hashemi Chaleshtori, Nathan Stringham, Kenneth Marino, Ana Marasovi\'c
arXiv AI
Sep 28

MedTokenBudget: Lesion-Preserving Token Routing for Dermoscopic Image Classification

MedTokenBudget introduces a supervised token routing framework for Vision Transformers applied to dermoscopic image classification. Its Lesion-Aware Token Scoring (LATS) module combines attention entropy, feature norm, and local feature contrast to select the top‑K patches under a target budget, trained with curriculum learning, diversity regularization, attention distillation, and lesion‑mask supervision. On the ISIC 2019 dataset, mask‑supervised LATS outperforms Random and ToMe at headline budgets while retaining more lesion patches.

By Zhexiang Li
arXiv AI
Sep 28

A Framework for Identifying, Categorizing, and Explaining Bias in AI-Generated Code

The paper presents a taxonomy-driven framework for identifying, categorizing, and explaining bias in AI-generated Python code. By extending an existing dataset and manually annotating bias categories and justifications, the authors evaluate both proprietary and open-source large language models (LLMs) for automated bias detection and explanation. Results show that models such as Gemini and Qwen3-coder achieve high classification accuracy and produce justification and code identification similarities that closely match human-authored reasoning.

By Manaal Basha, Aimee M. Ribeiro, Gema Rodriguez-Perez
arXiv AI
Sep 28

VLALight: Lightweight Vision-Language-Action Models for Emergency-Aware Traffic Signal Control

VLALight is a lightweight end‑to‑end vision‑language‑action framework designed for traffic signal control. It fuses multiple camera views and textual instructions to directly predict signal actions, avoiding intermediate image‑to‑text conversions. The model, with only 0.5 B parameters, achieves superior emergency vehicle service, cutting pooled waiting time by 21.1% compared to cascaded methods while running in real time on local hardware.

By Kemou Jiang, Maonan Wang, Xingchen Zou, Jiayue Zhu, Yuhang Fu, Sicheng Wang, Xi Chen, Yirong Chen, Zhiyong Cui
arXiv AI
Sep 28

Words Speak Louder Than Order: A Behavioral Evaluation of Gemma 4

The study investigates how Google’s Gemma 4‑e4b language model resolves conflicts between two documents. Using a counterbalanced design, researchers found that the semantic framing of a source (e.g., labeling it as an official guideline) dominates over the order in which documents appear. While the model shows a primacy bias toward the first document, this bias varies widely with wording and is amplified only when the documents are structurally identical.

By Amanda Fitch
arXiv AI
Sep 28

Beyond the Last Truffula Tree: SustainAI - A Water-Aware, Closed-Loop Framework for Environmentally Accountable AI

SustainAI is a water‑aware, closed‑loop framework that embeds environmental accountability into AI deployment. It combines real‑time water metering, a hallucination‑aware penalty model, and a water‑aware routing algorithm that considers regional water stress. In tests with small language models, water footprints varied 11‑fold across data centers, and 1,335 inference runs consumed about 399 mL of water but yielded only 240 correct outputs, highlighting the resource cost of inaccurate responses.

By Farnaz Farid, Tashfia Towkee, Sania Nasreen, Sami bin Azad
arXiv AI
Sep 28

Skip the Talk, Re-Focus on Vision: Latent Reasoning for Reasoning Segmentation in Multimodal Large Language Models

The paper introduces LIRSeg, a method that replaces explicit Chain-of-Thought reasoning in multimodal large language models with a compact set of learnable latent tokens for reasoning segmentation. LIRSeg is trained in two stages—spatial alignment and GRPO—while employing extreme-advantage sampling, decoupled exploration-stability updates, and latent diversity amplification to enhance token informativeness. Experiments show that LIRSeg improves segmentation accuracy and reasoning efficiency, achieving significant gIoU gains over the VisionReasoner baseline and reducing reasoning tokens by about 16×.

By Tianhang Guo, Yulin He, Wei Chen, Wenjuan Zhou, Yuhang Li, Xinbiao Gan
arXiv AI
Sep 28

Warned alike, AI agents avoid the less-crowded road while people take it

AI agents built on shared models tend to follow a common forecast, causing them to cluster on one road while avoiding a less crowded alternative. In a two-road congestion game, adding a warning that others might follow a routing tip led 50 GPT agents to crowd one road, raising average travel time from 64 to 95 minutes. Human participants largely maintained balance, but in mixed groups the imbalance grew as more agents were involved, with agents incurring higher travel times than humans.

By Takahiro Ezaki, Naoto Imura, Katsuhiro Nishinari
arXiv AI
Sep 28

UltraG-Bench: A Multi-task Benchmark for assessing Large Vision-Language Models on Pixel-level Evidence Grounding in Ultrasound

UltraG-Bench is a large‑scale, multi‑task benchmark designed to evaluate pixel‑level evidence grounding in ultrasound images. It comprises 40 public segmentation datasets covering 13 anatomical categories and includes three progressive tasks—instruction‑guided segmentation, evidence‑grounded VQA, and evidence‑grounded report generation—with a total of 736,726 annotations. Evaluation of 14 state‑of‑the‑art models shows a significant gap between semantic understanding and fine‑grained pixel‑level localization, and the authors propose UltraG‑Agent, which combines a vision‑language model with the ultrasound‑specific segmentation model UltraSAM to improve both semantic prediction and visual grounding.

By Quanhao Zhu, Bo Xu, Rui Lin, Chenyuan Wang, Yu Shao, Boling Zhu, Jiuyan Sun, Liang Zhao, Hongfei Lin, Feng Xia
arXiv AI
Sep 28

Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Models

The paper introduces EoupCT, a framework that estimates and orthogonalizes unknown pre‑training gradients to mitigate catastrophic forgetting during continual fine‑tuning of large language models. It generates pseudo data most susceptible to forgetting using a learnable soft prompt with Gumbel‑Softmax, then jointly optimizes model parameters and the prompt via a first‑order Pareto optimizer to enforce orthogonality between new task updates and the estimated gradients. Experiments on multiple LLMs show that EoupCT preserves both task‑specific performance and the models’ inherent general‑purpose knowledge.

By Bing Wang, Changchun Li, Xin-Qiang Cai, Lin Yuanbo Wu, Ximing Li, Gang Niu, Masashi Sugiyama
arXiv AI
Sep 28

MVVBench: Benchmarking 4D Reasoning in Vision-Language Models

MVVBench is a new benchmark for multi‑view video reasoning that tests vision‑language models on tasks requiring integration of spatial and temporal evidence across multiple, often non‑overlapping camera streams. The benchmark contains questions that cannot be answered from any single view or single moment, forcing models to jointly reason across views and time. It evaluates six capabilities—including attribute identification, relative distance, camera pose, and compositional counting—and provides human‑authored QA, rigorous verification, and detailed error analysis. "whyItMatters":"The benchmark offers a rigorous evaluation of 4D multi‑view reasoning and a foundation for future progress toward reliable embodied perception."

By Hyungjin Chung, Byeongjun Park, Joonseok Lee, Hojun Kim, Jaeho Choi, Byung-Hoon Kim
arXiv AI
Sep 28

The Linear Representation Hypothesis for Vision-Language-Action Models

The paper proposes a theoretical framework called the Linear Representation Hypothesis (LRH) for vision‑language‑action (VLA) models, extending the concept from large language models to systems where physical quantities of interest (QoI) evolve with the dynamics. It introduces a signature‑based formulation that unifies representations and policies, proving that future QoI evolution can be linearly probed from representations and that a generalized linear model for stochastic action chunks allows monotonic steering of QoI. The authors validate their theory with an explicit oracle representation in a planar control‑affine navigation experiment, demonstrating the predicted linear probing and steering mechanisms.

By Minseok Jeong, Hyewon Choi, Hiroyasu Tsukamoto, SooJean Han
arXiv AI
Sep 28

G$^2$PTQ: Improving LLM Post-Training Quantization with Generalized Gradient Compensation

G$^2$PTQ is a post‑training quantization framework that improves large language models by combining first‑ and second‑order information in a globally supervised, block‑wise optimization. It refreshes gradient and Hessian estimates before each Transformer block and uses a trust‑region scaling mechanism to stabilize gradient steps, preventing exploding weight updates. The method achieves better alignment with full‑precision models and outperforms state‑of‑the‑art baselines across various model families and bit‑widths.

By Ruikang Liu, Haoli Bai, Yuxuan Sun, Qian Zhang, Wenzheng Cai, Yanqi Hao, Feiyu Wang, Weidong Zhong, Zhuang Wang, Tong Yang, Xiangsheng Zhou