Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv AI

Sep 15

K-Bench: A Benchmark for LLM Unlearning in Agentic Deployments

K-Bench is a new benchmark designed to evaluate large language model (LLM) unlearning when the models are deployed as agents. Unlike previous benchmarks that only inspect the final answer, K-Bench examines all six channels of a ReAct agent—including chain-of-thought, tool calls, tool observations, and elicited summaries—to determine if a secret is leaked. The benchmark measures leakage for secrets placed in the model weights, prompt, or retrieval store, and finds that many existing unlearning methods fail to prevent leaks in deployed agents, especially when secrets reside in the prompt or retrieval store.

By Guangsheng Yu, Yanna Jiang, Qin Wang, Baihe Ma, Xu Wang

llms agents benchmarks

arXiv AI

Sep 15

ANASSA: An Agentic AI Orchestration Framework for Spatial Intelligence

The paper introduces ANASSA, an agentic AI orchestration framework designed for spatial intelligence in geographic information systems. It addresses gaps in current systems by integrating structured spatial reasoning, multi‑agent workflow orchestration, execution feedback, authoritative validation, provenance, uncertainty handling, and human decision authority. The architecture is detailed with eleven components across four layers, a six‑step Geospatial AI Cognitive Loop, cross‑component contracts, and governance mechanisms to ensure traceability, reproducibility, and accountability.

By Constantinos Papantoniou, Brian Hilton

llms agents multimodal benchmarks

arXiv AI

Sep 15

AlgoEvo: Self-Evolving Agentic Search for Automated Algorithm Discovery

AlgoEvo introduces a unified agentic framework for automated algorithm discovery that replaces rigid search pipelines with an interactive, knowledge‑accumulating process. An autonomous agent inspects, diagnoses, and edits code using runtime feedback, while a design skill hub decouples paradigm‑specific knowledge from the core engine, enabling a single workflow to handle single‑objective, multi‑objective, and multi‑component design tasks. The hierarchical experience mechanism organizes search trajectories into a task‑level tree, guiding exploration and consolidating cross‑task patterns into reusable skills, resulting in performance that matches or surpasses specialized methods with fewer evaluations and reduced token consumption.

By Junhao Qiu, Qinglong Hu, Xialiang Tong, Mingxuan Yuan, Liyong Lin, Qingfu Zhang

llms agents benchmarks

arXiv AI

Sep 15

Token Efficient Task Execution via Application Behavior Modeling for Web Agents

The paper introduces OdoBot, a web‑agent architecture that reduces token usage by modeling application behavior from successful task demonstrations. Compared to two state‑of‑the‑art agents, OdoBot uses 44% and 80% fewer tokens on 45 tasks in the Canvas Learning Management System, and it also achieves a higher task success rate than WebVoyager.

By Alexandru Ianta, Eleni Stroulia

agents benchmarks

arXiv AI

Sep 15

Identity Is More Than Recall: A Benchmark for Persistent Identity in Deployed AI Agents

The paper introduces PAI‑Bench, a benchmark designed to evaluate persistent AI agents on how faithfully they adhere to a versioned identity contract. It separates several dimensions—recall, composition, behavioral enactment, resistance, persistence, lineage, and role‑conditioned updates—while keeping scoring oracles independent of the target process. Experiments on synthetic profiles show that explicit cues can significantly alter the presence of identity identifiers, revealing prompt‑dependent component selection and sensitivity to startup cues.

By Zhenyu Zhao, Roy Zhao

llms agents benchmarks

arXiv AI

Sep 15

Forty Shades of Blue: Quality-Diversity Alignment via Mode-Conditioned Reinforcement Learning

arXiv:2609.14896v1 Announce Type: cross Abstract: A notable byproduct of LLM alignment training is mode collapse: the progressive loss of output diversity that narrows a model's expressivity at infer...

By Jiayi Yuan, Hangoo Kang, James Jihao Liu, Yejin Choi, Vikram Iyer, Liwei Jiang, Natasha Jaques

llms agents reinforcement-learning benchmarks safety

arXiv AI

Sep 15

Beyond Numerical Time Series: A Unified Benchmark for Multimodal Forecasting with Heterogeneous Context

arXiv:2609.15087v1 Announce Type: cross Abstract: Most time series forecasting benchmarks remain numerical-centric and provide limited support for evaluating contextual information that shapes real-w...

By Peng Chen, Zhihao Zhuang, Hongzhou Chen, Junhao Huang, Aiping Yang, Mengsen Wu, Yiding Liu, Xilin Dai, Zewei Dong

llms multimodal benchmarks

arXiv AI

Sep 15

Per-Matrix Optimality Is Not Enough: Three-Level Optimization for Low-Rank LLM Compression

arXiv:2609.15838v1 Announce Type: cross Abstract: Per-matrix singular value decomposition (SVD) truncation is Eckart-Young optimal in the whitened Frobenius norm, but errors from independently compre...

By Huicheng Zhang, Xiyao Feng, Ze-Tong Li, Chengkai Zhu, Xiao Shi, Xiwei Pan, Jinguo Liu, Ge Bai, Xin Wang

llms benchmarks

arXiv Machine Learning

Sep 15

WaVeFuse: Regime-Adaptive Equity Index Forecasting via Channel-Wise Wavelet Denoising and Vertical Attention Fusion

arXiv:2609.14733v1 Announce Type: new Abstract: Hybrid Deep Learning for equity index forecasting is limited by three problems: propagation of OHLCV noise into derived technical indicators (TIs), cha...

By Aashish Bohra, Vivek Vijay

llms benchmarks

arXiv Machine Learning

Sep 15

Admissable: Training Reinforcement Learning Agents against Adversarial Missingness

arXiv:2609.15297v1 Announce Type: new Abstract: In order to make Reinforcement Learning algorithms applicable in real world scenarios, safety must be ensured even under adverse operating conditions....

By Paul Stahlhofen, Luca Hermes, Tim Kochs, Markus Vieth, Barbara Hammer

agents reinforcement-learning benchmarks safety

arXiv Machine Learning

Sep 15

PRISM-UDE: Physics-Regularized Iterative Symbolic Modeling of 3nm FinFETs via Universal Differential Equation

arXiv:2609.13200v1 Announce Type: cross Abstract: Compact transistor models are the mathematical backbone of circuit simulation. However, at advanced nodes such as 3nm, transport physics becomes too...

By Pranavanath Balamurali, Prathamesh Dinesh Joshi, Raj Abhijit Dandekar, Rajat Dandekar, Sreedath Panat

benchmarks safety

arXiv Machine Learning

Sep 15

GroundBench: A Factorized, Counterfactual Benchmark for Locating VLM Affordance Failures

arXiv:2609.13308v1 Announce Type: cross Abstract: A companion evaluation found that naming the target part in a manipulation prompt increased action accuracy by 0.32-0.63 across eight vision-language...

By Sarthak Sattigeri

llms robotics multimodal benchmarks

arXiv Machine Learning

Sep 15

From matrix inversion to constraints: provably tighter confidence regions for importance weights in label shift

arXiv:2609.14802v1 Announce Type: cross Abstract: Importance weights are essential in domain adaptation under label shift, yet their utility is often undermined by the finite sample uncertainty assoc...

By Mushan Li, Kihyun Han, Yanyuan Ma

agents robotics fine-tuning multimodal benchmarks

arXiv Computation and Language

Sep 15

SHIFT-M3: Pre-fusion Alignment-based Consistency Screening for Multimodal ECG Record Integrity

arXiv:2609.13874v1 Announce Type: new Abstract: Multimodal clinical AI typically assumes that the waveform, report, metadata, and downstream predictions attached to a record belong to the same patien...

By Md Ashik Khan, Md Nahid Siddique

llms multimodal benchmarks safety

arXiv Computation and Language

Sep 15

Parameter-Efficient Quantum NLP for Paraphrase Detection: Performance, Robustness, and Entanglement

arXiv:2609.14529v1 Announce Type: cross Abstract: Rigorous empirical validation of quantum machine learning on natural language tasks remains scarce. We evaluate a 10-qubit hybrid quantum-classical v...

By Farha Nausheen, Khandakar Ahmed, Farina Riaz

nlp benchmarks safety

arXiv Computer Vision

Sep 15

EMCStereo: Attention-Enhanced Stereo Matching for Thin-Structure Depth Estimation with a Synthetic Tree-Branch Benchmark

arXiv:2609.13233v1 Announce Type: new Abstract: Thin structures such as tree branches are among the hardest cases for stereo matching: a branch is only a few pixels wide, the background is cluttered,...

By Yida Lin, Bing Xue, Mengjie Zhang, Sam Schofield, Richard Green

computer-vision benchmarks

arXiv Computer Vision

Sep 15

RxScribe Bench: A Multi-Axis Benchmark for Evaluating Vision-Language Models on Indian Outpatient Prescriptions

arXiv:2609.13280v1 Announce Type: new Abstract: Prescription transcription errors are not interchangeable. A model that fabricates a drug and a model that misreads a legible dose pose very different...

By Somil, Vijay Saini, Vidit Verma, Riya, Aastha Batta, Vibhuti Malhotra, Chayan Khetan, Piyush Mittal, Puneet Poonia

llms multimodal benchmarks safety

arXiv Computer Vision

Sep 15

FFVO: A Feedforward Pose Decoder for Long-Horizon Visual Odometry

arXiv:2609.13733v1 Announce Type: new Abstract: Stable and reliable 4D spatial understanding is fundamental for autonomous driving systems. While feedforward reconstruction networks can estimate came...

By Meng-Li Shih, Shih-Yang Su, Yuliang Zou, Hao Xiang, Haidong Zhu, Vincent Casser, Brian Curless, Dmitry Kalenichenko, Mingxing Tan, Dragomir Anguelov

agents computer-vision robotics benchmarks

arXiv Computer Vision

Sep 15

DynEoMT: Learning Object Dynamicity from Online Segmentation Queries

arXiv:2609.14466v1 Announce Type: new Abstract: Video segmentation models recognize and track objects over time, but they do not indicate whether each segmented region moves independently of the obse...

By Calvin Galagain, Martyna Poreba, Fran\c{c}ois Goulette

computer-vision benchmarks

arXiv Computer Vision

Sep 15

Reconstructing Is Not Acting: Action-Centric Latent Dynamics Modeling

arXiv:2609.15189v1 Announce Type: new Abstract: Latent action models (LAMs) learn action representations from unlabeled videos by inferring latent actions from visual transitions and reconstructing f...

By Dingjie Fu, Dianxing Shi, Yangyang Xu, Jun Yu

robotics benchmarks

Get the daily briefing