Large language models

Model releases, architecture work and prompting research on large language models — from frontier-lab announcements to the arXiv papers behind them.

31,384 stories · RSS feed

arXiv AI
Sep 18

From Rollout to Reset: A Graph-Based Harness for Autonomous Long-Horizon Manipulation Evaluation

The paper introduces HALTER, a graph-based system that automates the reset and evaluation of long-horizon robot manipulation tasks. HALTER constructs a spatial scene graph from point clouds and vision models, uses an LLM to score rollouts, plan resets, and verify success, all without labeled success images. In experiments on a Franka arm, HALTER restores scenes in 76% of episodes, improves skill completion estimation, and reduces operator time by 72% compared to manual reset.

By Jing Jiang, Yue Yang, Xinkai Jiang, Gedas Bertasius, Daniel J. Szafir, Rudolf Lioutikov
arXiv AI
Sep 18

From Models to Systems: A Comprehensive Survey of Efficient Multimodal Learning

The survey "From Models to Systems: A Comprehensive Survey of Efficient Multimodal Learning" reviews over 300 works on efficient multimodal learning (EML), proposing a structured taxonomy that spans model, algorithm, and system layers. It synthesizes how cross‑layer co‑design addresses the Efficiency‑Utility‑Privacy trade‑off and illustrates this through a case study of multimodal large language models. The paper also offers optimization blueprints for various domains, discusses a shift toward self‑regulating intelligence, and outlines open challenges for future EML research.

By Pan Wang, Siwei Song, Hui Ji, Siqi Cao, Heng Yu, Zhijian Liu, Huanrui Yang, Yingyan Celine Lin, Beidi Chen, Mohit Bansal, Xiaoming Liu, Pengfei Zhou, Ming-Hsuan Yang, Tianlong Chen, Jingtong Hu
arXiv AI
Sep 18

Efficiently Linking Unstructured Data for Multi-step Reasoning

The paper introduces DASE, a query engine designed to efficiently link unstructured data for multi-step reasoning tasks. DASE combines a multi-step reasoning model, a sparse materialized embedding-similarity join index (SemJI), and a co‑designed execution layer to perform multi‑attribute filtering, multi‑vector search, exact relational joins, and thresholded embedding‑similarity joins. In scientific discovery workloads, DASE outperforms traditional RDBMS, rerank, and vector‑database baselines by 6x to 46x in retrieval speed while maintaining comparable recall, and it serves as a high‑recall prefilter that reduces downstream LLM evaluation cost and improves accuracy on benchmarks such as SemBench E‑Commerce.

By Jiaming Liang, Haydn Jones, Jacob R. Gardner, Mark Yatskar, Zachary Ives
arXiv AI
Sep 18

For Your Eyes Only: Evaluating Coordination Between Isolated Language Model Instances

For Your Eyes Only: Evaluating Coordination Between Isolated Language Model Instances explores whether a language model can embed a signal in natural language that another independent instance can detect without shared memory or coordination training. The study introduces a cooperative signalling game where a Sender describes two words, one hidden, and a Receiver must identify the target. Seven contemporary models from four architectural families were tested on 300 word pairs, revealing that most struggle to coordinate when signals must be undetectable, though one frontier model performs near-perfectly even after filtering, and that models can also use this capability for deliberate misdirection.

By Alexander Shirnin, Aleksey Kudelya
arXiv AI
Sep 18

AURORA: A Natural Language-Driven Agentic Framework for Understanding, Reasoning, and Orchestrating Reliable Air-Ground Co-Simulation

AURORA is a natural‑language‑driven framework that treats air‑ground scenario generation as a compilation process with verification. It introduces the Air‑Ground Scenario Graph (AGSG), a typed intermediate representation linking agents, missions, events, communication, and success conditions, enabling joint grounding, temporal planning, pre‑execution checks, runtime verification, failure localization, and bounded repair. The authors also present AURORA‑Bench to evaluate not only execution but faithful realization of requested interactions, showing that structured execution and runtime verification improve reliability and that explicit intermediate representations facilitate verifiable and repairable co‑simulation.

By Keshu Wu, Hao Zhang, Rui Gan, Xiangbo Gao, Xiaopeng Li, Zhengzhong Tu, Yang Zhou
arXiv AI
Sep 18

DeltaSelect: Affordable A/B Testing for Coding Agents

DeltaSelect is an open‑source method that selects a small, cost‑effective set of tasks for A/B testing coding agents, using Pearson correlation to identify tasks whose single‑run results reliably reflect full‑benchmark performance. It maps fractional verifier outputs to a common score via linear regression and operates within a specified dollar budget, making it suitable for repeated baseline‑versus‑candidate comparisons during development rather than for model ranking. In a case study with gpt‑5.6‑luna, DeltaSelect reduced costs by 58.1% and improved calibrated scores across 13 evaluations.

By Nicholas J. Conn
arXiv AI
Sep 18

SoK: Trading Agents or Market Crashers? Dissecting Robustness and Security Failures in Academic Financial LLM Trading Schemes

The paper introduces FARSIGHT, a framework for evaluating the robustness and security of financial trading agents powered by large language models. It assesses agents on their resilience to market turbulence, such as flash crashes, and their vulnerability to three types of attacks: on information sources, on the agents themselves, and on agents acting as attackers. Applying FARSIGHT to 15 academic trading schemes reveals that 80% fail at least one robustness test and all exhibit security weaknesses, highlighting the risk of market-wide crashes from both accidental misjudgments and deliberate attacks.

By Mengxiao Wang, Nitesh Saxena
arXiv AI
Sep 18

Semantic Layer Induction from Raw Telemetry via Hierarchical LLM and RAG Abstraction

The paper introduces an end‑to‑end framework that automatically builds a business semantic layer from raw application logs. It uses a two‑stage abstraction: first, large language models identify high‑level business features with industry knowledge, and second, a structured pipeline refines data, retrieves relevant information, filters, clusters semantically, and assigns canonical names. Evaluation on production‑scale telemetry shows significant gains in semantic quality, noise reduction, and maintenance effort, with a 0.87 Cohen’s kappa in an LLM‑as‑Judge assessment.

By Yuanzhe Jia, Ali Anaissi
arXiv AI
Sep 18

DataCanvas-EDU: An Agentic Framework for Instructor-Guided Synthetic Data Generation in Business Analytics Education

DataCanvas-EDU is an agentic framework that lets instructors guide the creation of synthetic datasets for business analytics courses. Instructors set teaching goals and desired patterns via conversation, and an AI agent writes generation code, verifies the data, and produces assignments, reference solutions, and rubrics. The process is organized into four phases—Plan, Create, Verify/Test Analysis, and Evaluate—to streamline case preparation and enable students to explore new patterns with AI.

By Bang An, Maria Hamdani, Joseph Fox
arXiv AI
Sep 18

Evolution or Illusion? Rethinking Evaluation in LLM Evolutionary Search

The paper critiques the common practice of evaluating large‑language‑model (LLM) evolutionary search methods using a single seed and fixed iteration budget, arguing that this approach is insufficient. By testing three search strategies across five optimization tasks and varying both the number of seeds (width) and iterations (depth), the authors find that optimal budget allocation depends on the strategy, task, and total budget, and that strategy rankings shift with different budgets. They propose a measurement protocol that maps the seeds‑by‑iterations frontier and offers practical guidance for researchers.

By Tal Oved, Roi Pony, Oshri Naparstek, Udi Barzelay
arXiv AI
Sep 18

Zarya: A Hybrid Autoregressive--Masked Diffusion Language Model with Flexible Training and Dual-Mode Inference

Zarya is a hybrid language model that jointly trains an autoregressive objective and a masked-diffusion objective within a single architecture. It structures training data into variable-size slots and uses a curriculum that gradually increases slot granularity, allowing a smooth transition from fine-grained AR learning to coarse-grained diffusion learning. At inference, Zarya offers two decoding modes—MDM sampling with first-hitting denoising and slotted speculative decoding that interleaves diffusion-based selection with autoregressive infilling—while fully decoupling training and inference regimes and supporting extensive configurability.

By Leonid Sinev, Ilya Koziev, Vladislav Leshchuk
arXiv Machine Learning
Sep 18

DeliveryGym: An RL Environment for Long-Horizon Embodied Agent Planning with Adaptive Curriculum

DeliveryGym is a 3D reinforcement learning environment that simulates continuous courier shifts, integrating multimodal tool interaction, persistent world dynamics, and trajectory‑based rewards derived from simulator events. It allows agents to learn how their decisions affect time, energy, and money across an entire shift, and it adapts future training shifts to the policy’s weaknesses while keeping evaluation fixed. Experiments on six models and 13 city maps show a significant gap between task execution and optimal sequencing, with RL improving Qwen3‑VL‑4B’s net income by 54.3% and adaptive training boosting test income by 16.5% over uniform sampling.

By Haoqiang Kang, Yiming Zhang, Yiyang Guo, Chuying Li, Jianzhi Shen, Tianruo Rose Xu, Xiaokang Ye, Lianhui Qin
arXiv AI
Sep 18

MATCH: Model-Aware Tool Learning with Curriculum Scheduling and Hierarchically Gated Rewards

MATCH is a closed‑loop framework for model‑aware tool learning that combines curriculum scheduling with hierarchically gated rewards. It introduces Model‑Aware Curriculum Learning (MACL), which dynamically adjusts sample difficulty based on reward signals, and Hierarchical Tool‑call Gated Reward (HTGR), which allocates credit at the tool name, argument key, and argument value levels only when prerequisites are met. Experiments on API‑Bank and BFCL V3 show MATCH achieving 72.19% and 62.87% overall accuracy, outperforming both supervised and RL‑based baselines across multiple backbone models.

By Shihao Liu, Hao Yin, Lijun Liu, Zhengzong Chen, Yuanyuan Zhao, Fei Huang
arXiv AI
Sep 18

AdaRepair-Mem: Adaptive Experience Orchestration for Repository-Level Program Repair

AdaRepair-Mem introduces an adaptive experience retrieval framework for repository-level program repair, addressing three key limitations in existing memory-augmented methods: imbalanced episodic memory, non-monotonic success with increased memory, and phase misalignment of memory types. The framework employs coverage-aware retrieval, quality-aware selection, and stage-aware routing to better match repair contexts with relevant memories. Evaluations on SWE-Bench-Lite and SWE-Bench-Verified show improved performance on under-covered repositories, reduced noisy retrieval, and enhanced patch refinement support.

By Z. C. Luo, J. C. Guo, W. J. He, S. Y. Wang, J. C. Yu, F. M. Zhao, Y. Chen, T. Cao, L. Q. Liu, N. Zheng, W. Xu, J. Jiang, Z. M. Zhao
arXiv AI
Sep 18

Designing Against Deskilling: Metacognitive Feedback Reduces Cognitive Offloading to LLM Assistants

The study investigates how to prevent skill loss from relying on AI assistants by testing two interventions. In a preregistered online experiment with 704 participants, metacognitive feedback—explicitly highlighting the consequences of offloading—significantly lowered the likelihood of users offloading answers to an LLM and improved their performance on a subsequent unaided test. No effect was found for an effort-based reward incentive. The findings suggest that metacognitive feedback is an effective design strategy to reduce cognitive offloading to AI.

By Sebastian Maier, Kai Schwabe, Manuel Schneider, Stefan Feuerriegel
arXiv AI
Sep 18

Bridging Modalities on the Cortex: Surface-based MRI to PET Translation with a Diffusion Bridge

The paper introduces DB‑SUiT, a surface‑based diffusion bridge that translates cortical MRI to PET images directly on the cortical manifold. It employs a conditional spherical U‑shaped vision Transformer to capture multi‑scale surface features and long‑range dependencies while incorporating demographic and subcortical information. Evaluations on two dementia datasets show that the synthesized PET surfaces outperform MRI and PET volumes in automated classification and achieve high diagnostic accuracy in a blinded reader study.

By Yitong Li, Alexandra Samoylova, Fabian Bongratz, Timo Grimmer, Dennis M. Hedderich, Igor Yakushev, Christian Wachinger
arXiv AI
Sep 18

Music Hallucination in Audio-Language Models: A Hierarchical Formulation and Empirical Study

The paper presents the first music‑specific, layer‑wise empirical study of hallucination in audio‑language models, framing it as a hierarchical perceptual grounding failure across five layers: sound events, temporal properties, tonal attributes, style, and emotion. It introduces MuseDiag, a diagnostic framework that evaluates nine models and finds universal vocal misperception, significant tonal perception differences, and identifies Audio‑Flamingo‑3 as the most stable model. The study also proposes two training‑free mitigation methods, ADD‑M and TPA, which reduce hallucination in probing but show variable effectiveness in free‑form generation, highlighting the need for multi‑paradigm evaluation.

By Yu Liu, Jiahui Liu, Zhilin Liu, Cong Cao, Fangfang Yuan, Yuling Yang, Pin Xu, Yanbing Liu
arXiv AI
Sep 18

Is It Still Worth Training a Classical Model in the Era of LLMs? A Crossover Benchmark on Tabular Data

The paper investigates whether training classical machine learning models remains worthwhile when large language models (LLMs) can label tabular data without training. By defining a labeled‑data crossover point (N*) where a trained classical model surpasses a frozen LLM’s flat error, the authors analyze 126 student evaluations of GPT models across 18 datasets and compare them to power‑law learning curves of six classical model families. Results show that in 86% of cases a classical model outperforms the LLM with no more labeled data than already available, and the crossover occurs at a median of about 6% of the training set, suggesting that collecting a few hundred labels and training a gradient‑boosted model is typically advantageous.

By Kaihua Ding
arXiv AI
Sep 18

LLM-Guided Transformation of Non-Critical Driving Scenes into Safety-Critical Scenarios Using Augmented Reality

The paper introduces an automated pipeline that converts non‑critical driving scenes into safety‑critical scenarios by integrating computer vision, Large Language Models (LLMs), and Augmented Reality (AR). It detects and tracks road users, extracts safety features such as distance, velocity, motion direction, and Time‑to‑Collision (TTC), and evaluates scene criticality. Safe scenes are then modified by an LLM, which generates realistic collision‑inducing objects and behaviors that are overlaid onto the original scene using AR, achieving 97.52% safety classification accuracy on the nuScenes dataset and producing realistic scenarios like pedestrian crossings, rear overtaking vehicles, and sudden‑stop events.

By Noura Fady, Farah Khaled, Catherine M. Elias
arXiv AI
Sep 18

STR-Agent: An LLM-Driven Agent for QoS-Aware Routing in LEO Satellite Networks

STR-Agent is an LLM-driven framework designed for QoS-aware routing in Low Earth Orbit satellite networks. It integrates intent perception, tool-based execution, experience accumulation, and reflection-based policy adaptation to translate natural-language service requests into adaptive routing decisions. In simulations on a Walker-Delta constellation, STR-Agent reduces end-to-end delay by up to 60% compared with DQ-Dijkstra and improves intent-understanding accuracy from 45.4% to 92.45% after fine-tuning, with the Reflection Module providing additional delay reductions.

By Bowen Lu, Mugen Peng, Yaohua Sun, Hongyu Wang, Kerui Guo, Wenjia Xu