Large language models

Model releases, architecture work and prompting research on large language models — from frontier-lab announcements to the arXiv papers behind them.

26,035 stories · RSS feed

arXiv AI

Jun 30

Faults in Our Formal Benchmarking: Dataset Defects and Evaluation Failures in Lean Theorem Proving

arXiv:2606. 29493v1 Announce Type: new Abstract: Benchmarks for LLM-assisted theorem proving in Lean are often treated as intrinsically reliable because every solved instance comes with a machine-checked proof.

By Pawan Sasanka Ammanamanchi, Siddharth Bhat, Stella Biderman

llms benchmarks safety

arXiv AI

Jun 30

It Lied to a Doctor to Buy Poison Ingredients: Quantifying Real-World Misuse of Phone-use Agents

arXiv:2606. 27944v1 Announce Type: cross Abstract: Phone-use Agents can execute complex tasks end to end across real mobile applications.

By Yiming Sun, Chen Chen, Zifan Zhou, Mi Zhang

llms agents robotics safety

arXiv AI

Jun 30

Open Problems in Constitutional Preference Reconstruction

arXiv:2606. 30116v1 Announce Type: new Abstract: Pairwise preference data is widely used for training and evaluating language models (e.

By Eleanor Clifford, Michael Amir, Arduin Findeis, Aaron Zhao, Robert Mullins

llms reinforcement-learning safety

arXiv AI

Jun 30

Geo-Expert: Towards Expert-Level Geological Reasoning via Parameter-Efficient Fine-Tuning

arXiv:2605. 24844v2 Announce Type: replace Abstract: While general-purpose Large Language Models (LLMs) applied to Geology often hallucinate when reasoning about subsurface structures and deep-time evolution, current AI in Earth sciences predominantly targets surface remote sensing and GIS.

By Chenyou Guo, Zongqi Liu, Yizhou Zhang, Zhaorui Jiang, Ze Liu

llms fine-tuning benchmarks

arXiv AI

Jun 30

Carolina Guide: A Multi-Agent RAG System with Institutional Guardrails for Academic Policy Assistance

arXiv:2606. 28360v1 Announce Type: cross Abstract: University students often struggle to navigate complex academic policies, leading to advising bottlenecks and delayed access to critical information.

By Ben Torsion, Jun Zhou

llms rag agents benchmarks safety

arXiv AI

Jun 30

Conversational Query Engine for Mixed-Modality Heterogeneous Enterprise Data Sources

arXiv:2606. 28370v1 Announce Type: cross Abstract: Enterprise business intelligence queries span structured warehouses and unstructured document repositories -- modalities with fundamentally different access methods, cost profiles, and correctness semantics.

By Darshita Rathore, Vineet Kumar, Vaibhav Singal, Ankur Vivek Singh, Anindya Moitra

llms rag agents fine-tuning benchmarks

arXiv AI

Jun 30

When Does Overlap Help? OSU-Mem and a Cell-Conditional Analysis of Trajectory Memory for LLM Agents

arXiv:2606. 28376v1 Announce Type: cross Abstract: Long-horizon large language model (LLM) agents accumulate interaction trajectories that quickly exceed any practical prompt budget, and existing memory methods either truncate aggressively and lose non-local evidence or retain boilerplate that degrades decision quality.

By Mellow Baixuan Chen, Xiangguo Sun

llms agents benchmarks

arXiv AI

Jun 30

How Do LLMs Cite? A Mechanistic Interpretation of Attribution in Retrieval-Augmented Generation

arXiv:2606. 28358v1 Announce Type: cross Abstract: Retrieval-Augmented Generation (RAG) aims to enhance the trustworthiness of Large Language Models (LLMs) by grounding their outputs in external documents, often using inline citations for verifiability.

By Ian van Dort (University of Amsterdam), Maria Heuss (University of Amsterdam)

llms rag benchmarks

arXiv AI

Jun 30

SearchSkill: Teaching LLMs to Use Search Tools with Evolving Skill Banks

arXiv:2605. 09038v3 Announce Type: replace Abstract: Teaching language models to use search tools is not only a question of whether they search, but also of whether they issue good queries.

By Jinchao Hu, Meizhi Zhong, Kehai Chen, Min Zhang

llms nlp fine-tuning benchmarks

arXiv Machine Learning

Jun 30

RA-QA: A Benchmarking System for Respiratory Audio Question Answering Under Real-World Heterogeneity

arXiv:2602. 18452v3 Announce Type: replace-cross Abstract: As conversational multimodal AI tools are increasingly adopted to process patient data for health assessment, robust benchmarks are needed to measure progress and expose failure modes under realistic conditions.

By Gaia A. Bertolino, Yuwei Zhang, Tong Xia, Domenico Talia, Cecilia Mascolo

llms nlp multimodal benchmarks

arXiv AI

Jun 30

SA-VLA: State-aware tokenizer for improving Vision-Language-Action Models' performance

arXiv:2606. 30113v1 Announce Type: cross Abstract: Discrete action tokenization provides a compact interface for autoregressive VLA policies, but accurately recovering continuous robot actions from discrete codes remains challenging.

By Tengyue Jiang, Chunpu Xu, Jiayue Kang, Yao Mu

llms nlp robotics multimodal

arXiv AI

Jun 30

Multi-Agent Routing as Set-Valued Prediction: A WildChat Benchmark and Cost-Aware Evaluation

arXiv:2606. 28925v1 Announce Type: cross Abstract: Tool and agent routing from natural-language prompts is naturally a set-valued prediction problem: a single query may require multiple agents, while over-selection increases execution cost.

By Ananto Nayan Bala, Faisal Muhammad Shah

llms agents fine-tuning benchmarks

arXiv AI

Jun 30

AB-RAG: Adaptive Budgeted Retrieval-Augmented Generation for Reliable Question Answering

arXiv:2606. 29090v1 Announce Type: cross Abstract: Retrieval-Augmented Generation (RAG) has become the standard way to ground large language models in external knowledge, yet most systems retrieve a fixed number of passages for every question regardless of its difficulty.

By Ansh Kamthan

llms rag nlp

arXiv AI

Jun 30

MedDiffuseMix: Preserving Diagnostic Evidence with Saliency-Aware Diffusion Medical Image Data Augmentatio

arXiv:2606. 28419v1 Announce Type: cross Abstract: Limited data availability, class imbalance, and domain variability remain major barriers to reliable medical image classification.

By Teerath Kumar, Raja Vavekanand, Muhammad Turab

llms diffusion computer-vision benchmarks

arXiv AI

Jun 30

Evidence-Driven LLM Agent for C-to-Synthesizable-C Conversion and Verification

arXiv:2606. 28409v1 Announce Type: cross Abstract: Software-compilable C programs routinely fail to complete the four-stage pipeline of a high-level synthesis (HLS) toolchain -- compilation, C simulation (CSim), synthesis, and C/RTL co-simulation (CoSim) -- because HLS accepts only a synthesizable subset of C (HLS-C).

By Zhe Zhao, Hongbing Lang, Zhihan Xiao, Luke Ztz Hu, John Imoleayo Adebisi, Songping Mai

llms rag agents benchmarks

arXiv Machine Learning

Jun 30

An Integrated Two-Stage Deep-Learning Tool for Rapid Post-Hurricane Damage Identification and Repair Scheduling

arXiv:2606. 29117v1 Announce Type: cross Abstract: Post-hurricane damage assessment and repair scheduling can require computationally intensive simulation and optimization.

By Hooman Torkaman, Ellis Oti Boateng, Jignesh Solanki, Anurag Srivastava

llms benchmarks

arXiv AI

Jun 30

The Verbose Context Problem in Medical Records

arXiv:2606. 29503v1 Announce Type: cross Abstract: The verbose context problem occurs when structured concepts have token-inefficient textual representations.

By Shiva Kaul, Min-Gyu Kim, Anjum Khurshid, Sriram Vishwanath

llms agents benchmarks

arXiv AI

Jun 30

Towards Generalizable and Evidential Nuclear Magnetic Resonance-Based Molecular Structure Elucidation via Large Language Model Agent

arXiv:2606. 29776v1 Announce Type: cross Abstract: Nuclear Magnetic Resonance (NMR) spectroscopy is the gold standard for molecular structure elucidation, yet interpreting complex spectra for unknown molecules remains a bottleneck reliant on human expertise.

By Zheng Fang, Chen Yang, Yusen Tan, Yunpeng Zhao, Fanjie Xu, Hongxin Xiang, Hanyu Sun, Hanyu Gao, Xiaojian Wang, Wenjie Du, Yuqiang Li, Jun Xia

llms agents benchmarks safety

arXiv AI

Jun 30

Neural Procedural Memory: Empowering LLM Agents with Implicit Activation Steering

arXiv:2606. 29824v1 Announce Type: cross Abstract: While Large Language Models (LLMs) excel as static solvers, transforming them into autonomous agents remains challenging.

By Chengfeng Zhao, Yuqiao Tan, Shizhu He, Yequan Wang, Jun Zhao, Kang Liu

llms rag agents benchmarks

arXiv AI

Jun 30

OptiMUS-0.3: Using Large Language Models to Model and Solve Optimization Problems at Scale

arXiv:2407. 19633v4 Announce Type: replace Abstract: Optimization problems are pervasive in sectors from manufacturing and distribution to healthcare.

By Ali AhmadiTeshnizi, Wenzhi Gao, Herman Brunborg, Shayan Talaei, Connor Lawless, Madeleine Udell

llms fine-tuning benchmarks

Get the daily briefing