Large language models

Model releases, architecture work and prompting research on large language models — from frontier-lab announcements to the arXiv papers behind them.

32,584 stories · RSS feed

arXiv AI

Sep 15

K-Bench: A Benchmark for LLM Unlearning in Agentic Deployments

K-Bench is a new benchmark designed to evaluate large language model (LLM) unlearning when the models are deployed as agents. Unlike previous benchmarks that only inspect the final answer, K-Bench examines all six channels of a ReAct agent—including chain-of-thought, tool calls, tool observations, and elicited summaries—to determine if a secret is leaked. The benchmark measures leakage for secrets placed in the model weights, prompt, or retrieval store, and finds that many existing unlearning methods fail to prevent leaks in deployed agents, especially when secrets reside in the prompt or retrieval store.

By Guangsheng Yu, Yanna Jiang, Qin Wang, Baihe Ma, Xu Wang

llms agents benchmarks

arXiv AI

Sep 15

ANASSA: An Agentic AI Orchestration Framework for Spatial Intelligence

The paper introduces ANASSA, an agentic AI orchestration framework designed for spatial intelligence in geographic information systems. It addresses gaps in current systems by integrating structured spatial reasoning, multi‑agent workflow orchestration, execution feedback, authoritative validation, provenance, uncertainty handling, and human decision authority. The architecture is detailed with eleven components across four layers, a six‑step Geospatial AI Cognitive Loop, cross‑component contracts, and governance mechanisms to ensure traceability, reproducibility, and accountability.

By Constantinos Papantoniou, Brian Hilton

llms agents multimodal benchmarks

arXiv AI

Sep 15

AlgoEvo: Self-Evolving Agentic Search for Automated Algorithm Discovery

AlgoEvo introduces a unified agentic framework for automated algorithm discovery that replaces rigid search pipelines with an interactive, knowledge‑accumulating process. An autonomous agent inspects, diagnoses, and edits code using runtime feedback, while a design skill hub decouples paradigm‑specific knowledge from the core engine, enabling a single workflow to handle single‑objective, multi‑objective, and multi‑component design tasks. The hierarchical experience mechanism organizes search trajectories into a task‑level tree, guiding exploration and consolidating cross‑task patterns into reusable skills, resulting in performance that matches or surpasses specialized methods with fewer evaluations and reduced token consumption.

By Junhao Qiu, Qinglong Hu, Xialiang Tong, Mingxuan Yuan, Liyong Lin, Qingfu Zhang

llms agents benchmarks

arXiv AI

Sep 15

Identity Is More Than Recall: A Benchmark for Persistent Identity in Deployed AI Agents

The paper introduces PAI‑Bench, a benchmark designed to evaluate persistent AI agents on how faithfully they adhere to a versioned identity contract. It separates several dimensions—recall, composition, behavioral enactment, resistance, persistence, lineage, and role‑conditioned updates—while keeping scoring oracles independent of the target process. Experiments on synthetic profiles show that explicit cues can significantly alter the presence of identity identifiers, revealing prompt‑dependent component selection and sensitivity to startup cues.

By Zhenyu Zhao, Roy Zhao

llms agents benchmarks

arXiv AI

Sep 15

ClinAgent: A ReAct-Based Agent for Conversational Access to Clinical Trial Information

ClinAgent is a conversational system that uses a ReAct-based LLM agent to retrieve and synthesize clinical trial information from multiple sources such as ClinicalTrials.gov, PubMed, and a local dataset. The agent iteratively reasons over user queries, selects appropriate tools, and refines its actions to provide grounded, up-to-date responses in natural language across multi-turn interactions. Evaluation across three phases shows that DeepSeek (thinking mode) excels in planning quality while Gemini 3.0 Flash delivers the highest overall performance and expert ratings, demonstrating the promise of agentic AI for improving clinical trial data access.

By Antonino Vaccarella, Riccardo Cantini, Domenico Talia, Paolo Trunfio, Marianna Talia, Rosamaria Lappano, Marcello Maggiolini

llms rag agents

arXiv AI

Sep 15

Forty Shades of Blue: Quality-Diversity Alignment via Mode-Conditioned Reinforcement Learning

arXiv:2609.14896v1 Announce Type: cross Abstract: A notable byproduct of LLM alignment training is mode collapse: the progressive loss of output diversity that narrows a model's expressivity at infer...

By Jiayi Yuan, Hangoo Kang, James Jihao Liu, Yejin Choi, Vikram Iyer, Liwei Jiang, Natasha Jaques

llms agents reinforcement-learning benchmarks safety

arXiv AI

Sep 15

Beyond Numerical Time Series: A Unified Benchmark for Multimodal Forecasting with Heterogeneous Context

arXiv:2609.15087v1 Announce Type: cross Abstract: Most time series forecasting benchmarks remain numerical-centric and provide limited support for evaluating contextual information that shapes real-w...

By Peng Chen, Zhihao Zhuang, Hongzhou Chen, Junhao Huang, Aiping Yang, Mengsen Wu, Yiding Liu, Xilin Dai, Zewei Dong

llms multimodal benchmarks

arXiv AI

Sep 15

Predictive Likelihood Ratios for Language Model Watermark Detection

arXiv:2609.15657v1 Announce Type: cross Abstract: Keyed watermark detection tests dependence between observed tokens and pseudorandom variables reconstructed from a secret key. Building on the pivota...

By Li Ma

llms

arXiv AI

Sep 15

When the World Lies: Backdoor Attacks on Latent World Models for Downstream Control

arXiv:2609.15781v1 Announce Type: cross Abstract: Pretrained world models, learned simulators that encode an observation into a latent state and predict how it evolves under actions, are beginning to...

By Roberto Ria\~no, Gorka Abad, Stjepan Picek, Aitor Urbieta

llms fine-tuning

arXiv AI

Sep 15

Per-Matrix Optimality Is Not Enough: Three-Level Optimization for Low-Rank LLM Compression

arXiv:2609.15838v1 Announce Type: cross Abstract: Per-matrix singular value decomposition (SVD) truncation is Eckart-Young optimal in the whitened Frobenius norm, but errors from independently compre...

By Huicheng Zhang, Xiyao Feng, Ze-Tong Li, Chengkai Zhu, Xiao Shi, Xiwei Pan, Jinguo Liu, Ge Bai, Xin Wang

llms benchmarks

arXiv AI

Sep 15

Testing the Limits of Truth Directions in LLMs

arXiv:2604.03754v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have been shown to encode truth of statements in their activation space along a linear truth direction. Previous...

By Angelos Poulis, Mark Crovella, Evimaria Terzi

llms

arXiv Machine Learning

Sep 15

WaVeFuse: Regime-Adaptive Equity Index Forecasting via Channel-Wise Wavelet Denoising and Vertical Attention Fusion

arXiv:2609.14733v1 Announce Type: new Abstract: Hybrid Deep Learning for equity index forecasting is limited by three problems: propagation of OHLCV noise into derived technical indicators (TIs), cha...

By Aashish Bohra, Vivek Vijay

llms benchmarks

arXiv Machine Learning

Sep 15

GroundBench: A Factorized, Counterfactual Benchmark for Locating VLM Affordance Failures

arXiv:2609.13308v1 Announce Type: cross Abstract: A companion evaluation found that naming the target part in a manipulation prompt increased action accuracy by 0.32-0.63 across eight vision-language...

By Sarthak Sattigeri

llms robotics multimodal benchmarks

arXiv Computation and Language

Sep 15

Toward Complete Hospital Discharge Summarization with Abstract Meaning Representation

arXiv:2609.13581v1 Announce Type: new Abstract: Discharge summaries are lengthy medical documents that summarize a hospital in-patient visit. Automatically generating them can reduce documentation bu...

By Paul Landes, Sitara Rao, Aaron Jeremy Chaise, Barbara Di Eugenio

llms nlp safety

arXiv Computation and Language

Sep 15

SHIFT-M3: Pre-fusion Alignment-based Consistency Screening for Multimodal ECG Record Integrity

arXiv:2609.13874v1 Announce Type: new Abstract: Multimodal clinical AI typically assumes that the waveform, report, metadata, and downstream predictions attached to a record belong to the same patien...

By Md Ashik Khan, Md Nahid Siddique

llms multimodal benchmarks safety

arXiv Computation and Language

Sep 15

Reducing the Output-Mode Gap in Speech Language Models via Joint-Output On-Policy Distillation

arXiv:2609.15313v1 Announce Type: cross Abstract: Autoregressive generation of interleaved text and acoustic tokens is a common approach to spoken-response generation in speech large language models....

By Daxin Tan, Dehua Tao, Chengxi Deng, Hanlin Zhang, Xiao Chen

llms fine-tuning efficiency multimodal

arXiv Computer Vision

Sep 15

RxScribe Bench: A Multi-Axis Benchmark for Evaluating Vision-Language Models on Indian Outpatient Prescriptions

arXiv:2609.13280v1 Announce Type: new Abstract: Prescription transcription errors are not interchangeable. A model that fabricates a drug and a model that misreads a legible dose pose very different...

By Somil, Vijay Saini, Vidit Verma, Riya, Aastha Batta, Vibhuti Malhotra, Chayan Khetan, Piyush Mittal, Puneet Poonia

llms multimodal benchmarks safety

arXiv Computer Vision

Sep 15

Selective Tool Use for Agentic Change Visual Question Answering in Remote Sensing

arXiv:2609.14523v1 Announce Type: new Abstract: Change visual question answering (Change VQA) requires understanding semantic changes across bi-temporal remote sensing images. Although vision languag...

By Yakoub Bazi, Mohamad M. Al Rahhal, Mohamed A. Mekhtiche, Mansour Zuair

llms agents nlp fine-tuning multimodal

arXiv Computer Vision

Sep 15

Perceive, Refine, Reason: A Calibrated Pipeline for Measuring Indicators in Strategic Visual Communication on Social Media

arXiv:2609.14699v1 Announce Type: new Abstract: Visual content shapes audience perception and opinion on social media, and computational social science increasingly relies on automated tools to analy...

By Weihong Qi, Chen Ling

llms multimodal

arXiv AI

Sep 15

ZEBRAARENA: A Diagnostic Simulation Environment for Studying Reasoning-Action Coupling in Tool-Augmented LLMs

arXiv:2603.18614v2 Announce Type: replace Abstract: Tool-augmented large language models (LLMs) must tightly couple multi-step reasoning with external actions, yet existing benchmarks often confound...

By Wanjia Zhao, Ludwig Schmidt, Yejin Choi, James Zou, Vidhisha Balachandran, Lingjiao Chen

llms agents benchmarks

Get the daily briefing