Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

14,014 stories · RSS feed

arXiv AI

Aug 11

Not All Turns Are Equally Hard: Adaptive Thinking Budgets For Efficient Multi-Turn Reasoning in Agents

arXiv:2604. 05164v3 Announce Type: replace-cross Abstract: As LLM reasoning performance plateaus, improving inference-time compute efficiency is crucial to mitigate overthinking and long thinking traces even for simple queries.

By Neharika Jali, Anupam Nayak, Gauri Joshi

llms agents reinforcement-learning benchmarks

arXiv AI

Aug 11

Continual Visual Anomaly Detection on the Edge: Benchmark and Efficient Solutions

arXiv:2604. 06435v2 Announce Type: replace-cross Abstract: Visual Anomaly Detection (VAD) is a critical task for many applications including industrial inspection and healthcare.

By Manuel Barusco, Francesco Borsatti, David Petrovic, Davide Dalle Pezze, Gian Antonio Susto

efficiency benchmarks

arXiv AI

Aug 11

TraceSafe: A Systematic Assessment of LLM Guardrails on Multi-Step Tool-Calling Trajectories

arXiv:2604. 07223v2 Announce Type: replace-cross Abstract: As large language models (LLMs) evolve from static chatbots into autonomous agents, the primary vulnerability surface shifts from final outputs to intermediate execution traces.

By Yen-Shan Chen, Sian-Yao Huang, Cheng-Lin Yang, Yun-Nung Chen

llms agents benchmarks safety

arXiv AI

Aug 11

Evaluating Jailbreaking Vulnerabilities in LLMs Deployed as Assistants for Smart Grid Operations: A Benchmark Against NERC Standards

arXiv:2604. 23341v3 Announce Type: replace-cross Abstract: The deployment of Large Language Models (LLMs) as assistants in electric grid operations promises to streamline compliance and decision-making but exposes new vulnerabilities to prompt-based adversarial attacks.

By Taha Hammadia, Lucas Rea, Ahmad Mohammad Saber, Amr Youssef, Deepa Kundur

llms multimodal benchmarks safety

arXiv AI

Aug 11

Path-Lock Expert: Separating Reasoning Mode in Hybrid Thinking via Architecture-Level Separation

arXiv:2604. 27201v3 Announce Type: replace-cross Abstract: Hybrid-thinking language models expose explicit /think and /no_think modes, but current designs do not separate them cleanly.

By Shouren Wang, Wang Yang, Chuang Ma, Debargha Ganguly, Vikash Singh, Chaoda Song, Xinpeng Li, Xianxuan Long, Vipin Chaudhary, Xiaotian Han

llms rag fine-tuning benchmarks

arXiv AI

Aug 11

DeltaPrompts: Escaping the Zero-Delta Trap in Multimodal Distillation

arXiv:2605. 15532v3 Announce Type: replace-cross Abstract: Distillation enables compact Vision-Language Models (VLMs) to obtain strong reasoning capabilities, yet the prompts driving this process are typically chosen via simple heuristics or aggregated from off-the-shelf datasets.

By Jaehun Jung, Hyunwoo Kim, Brandon Cui, Ximing Lu, David Acuna, Prithviraj Ammanabrolu, Yejin Choi

llms fine-tuning efficiency multimodal benchmarks

arXiv AI

Aug 11

Beyond Questions: Evaluating LLM's Knowledge Expression

arXiv:2605. 26937v2 Announce Type: replace-cross Abstract: Parametric knowledge in large language models (LLMs) is a cornerstone of their success, yet remains poorly understood.

By Luca Giordano, Simon Razniewski

llms benchmarks safety

arXiv AI

Aug 11

ATMA: Long-Context Language Modeling via Polar Attention and Gated-Delta Compression Memory

arXiv:2606. 25156v3 Announce Type: replace-cross Abstract: Native length extrapolation remain a weakly solvable problem in language modeling due to trade-off balancing between exact retrieval fidelity, long-document likelihood, and inference efficiency.

By Habibullah Akbar

benchmarks

arXiv AI

Aug 11

Symbolic Attack Chain Generation from Atomic Red Team Techniques: An Empirical Study of Predicate Representation Granularity

arXiv:2608. 00143v2 Announce Type: replace-cross Abstract: Automated attack chain generation is critical for modern cybersecurity, yet manual construction fails to scale as adversary behaviors expand.

By Ramya Varunsegar

llms benchmarks safety

arXiv Machine Learning

Aug 11

From Benchmark Performance to Tool Deployment: Human-in-the-Loop Anomaly Detection

arXiv:2608. 07770v1 Announce Type: new Abstract: Automated anomaly detection methods often report strong performance on curated academic benchmarks, but their behavior under real-world industrial conditions is less clear.

By Mike Szklarzewski, CJ George, Gavin Smithson, Christopher Stokes, Dakota Fulp, William M. Jones, Benjamin Wynn, Alexander Ur, Agit Yesiloz, Clint Kallenbach, Mark Swartz, Nathan DeBardeleben, Sharmistha Chakrabarti

benchmarks data-engineering

arXiv Machine Learning

Aug 11

Adaptive KappaSharp: Condition-Number Shaping for Preferential Bayesian Optimization

arXiv:2608. 07859v1 Announce Type: new Abstract: Preferential Bayesian optimization (PBO) optimizes objectives accessible only through pairwise user comparisons.

By Ketong Shao, Jialu Wang, Xuekai Pei, Ali Mesbah

benchmarks

arXiv Machine Learning

Aug 11

LLM-Based Embeddings for Program Analysis and Optimization

arXiv:2608. 07894v1 Announce Type: new Abstract: Recent advances have highlighted the potential of machine learning, particularly Large Language Models (LLMs), for analyzing and optimizing programs.

By Calvin Higgins, Marco Alvarez

llms rag benchmarks

arXiv Machine Learning

Aug 11

SPECTRA: Pushing the KV Cache Beyond the 2-Bit Cliff via Spectral Transform Coding

arXiv:2608. 07915v1 Announce Type: new Abstract: Large language models (LLMs) increasingly read long inputs in the agentic era, from whole documents and codebases to conversations across many turns.

By Jiamu Zhang, Liang Wu, Kelly Wan, Hanjie Chen, Liangjie Hong

llms agents efficiency benchmarks

arXiv Machine Learning

Aug 11

Adaptive Supervised Anchoring for On-Policy Self-Distillation

arXiv:2608. 07935v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) adapts a language model by distilling guidance from a frozen teacher on trajectories sampled from the student.

By Meilin Yang (Renmin University of China, Beijing, China), Zixuan Ding (Renmin University of China, Beijing, China), Jianhao Nie (Renmin University of China, Beijing, China), Weite Zhang (Renmin University of China, Beijing, China), Yuxin Zhang (Renmin University of China, Beijing, China), Zhiming Shao (Renmin University of China, Beijing, China), Li Yu (Renmin University of China, Beijing, China), Zhe Fu (Renmin University of China, Beijing, China)

llms efficiency benchmarks safety

arXiv Machine Learning

Aug 11

Control-Diverse Reinforcement Fine-Tuning: Decoupling the Shared Control Bottleneck of RL Post-Training

arXiv:2608. 08224v1 Announce Type: new Abstract: Reinforcement learning post-training unlocks complex reasoning in LLMs.

By Binwen Tan, Jingchao Wang, Dengzhe Hou, Lingyu Jiang, Zeyuan Wu, Yunhan Shen, Fangzhou Lin, Kazunori Yamada, Atsushi Koike

llms reinforcement-learning fine-tuning efficiency benchmarks safety

arXiv Machine Learning

Aug 11

The Replay Gap: Static Evaluation of Model Switching in LLM Agents Scores the Wrong World

arXiv:2608. 08239v1 Announce Type: new Abstract: LLM routers promise efficiency by matching each request to the cheapest adequate model, and are increasingly applied per step inside multi-step agents.

By Ashritha Gonuguntla

llms agents benchmarks

arXiv Machine Learning

Aug 11

Stateful CARS: Exact Cross-History Reuse for Policy-Constrained LLM Agents

arXiv:2608. 08282v1 Announce Type: new Abstract: Tool-using language-model agents face constraints whose meaning changes with observations and prior actions.

By Ibne Farabi Shihab, Md Najmus Swaqeeb, Abu Sa-Adat Mohamed Moon-Im Al Ahsan

llms agents benchmarks

arXiv Machine Learning

Aug 11

Task-to-Model Optimization for Enterprise LLM Coding Assistants: A Data-Driven Framework for Cost-Optimal Routing

arXiv:2608. 08528v1 Announce Type: new Abstract: Enterprise AI coding assistants incur substantial inference spend, and naive token-cost minimization often fails to reduce end-to-end cost once retries, escalations, and developer wait time are included.

By Srinivasan Manoharan, Junhua Zhao, Fangbo Tu, Haifeng Wu, Jian Wan, Maliah Rajan M, Ashwin Hegde, Mithun Sasidharan, Kalyan Chakravarthi Podamekala

llms benchmarks

arXiv Machine Learning

Aug 11

Can Graph Learning Learn Circuits?

arXiv:2608. 08536v1 Announce Type: new Abstract: Circuit localization is a mechanistic interpretability task whose goal is to identify a sparse subgraph of a transformer's computation graph sufficient to reproduce a particular behavior.

By Chester Tan, Moritz Lampert, Courtney Maynard, Ankit Ramakrishnan, Tina Eliassi-Rad, Ingo Scholtes

llms benchmarks safety

arXiv Machine Learning

Aug 11

When Skills Meet Safety: Benchmarking and Characterizing the Adaptive Jailbreak Robustness of Skill-Merged LLMs

arXiv:2608. 08542v1 Announce Type: new Abstract: Model merging has become the default way to give an aligned language model new skills without retraining: a practitioner folds task vectors from math, code, or domain specialists into a safety-aligned base using task arithmetic, TIES, or DARE.

By Yu Ma, Hongli Shi, Jing Li, Xinran Xu, Weiwei Hou

llms benchmarks safety

Get the daily briefing