Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

14,220 stories · RSS feed

arXiv AI

Aug 5

SphUnc: Hyperspherical Uncertainty Decomposition and Causal Identification via Information Geometry

arXiv:2603. 01168v3 Announce Type: replace-cross Abstract: Reliable decision-making in complex multi-agent systems requires calibrated predictions and interpretable uncertainty.

By Rong Fu, Chunlei Meng, Jinshuo Liu, Dianyu Zhao, Yongtai Liu, Yibo Meng, Xiaowen Ma, Wangyu Wu, Yangchen Zeng, Shuaishuai Cao, Simon Fong

agents benchmarks

arXiv AI

Aug 5

Injection-Execution Dissociation: A Mechanistic Evaluation of Persistent Memory Attacks and Defenses in Stateful LLM Agents

arXiv:2605. 08442v5 Announce Type: replace-cross Abstract: We discover that prompt-injection success and tool-execution success are separable safety properties: defenses that block injection do not necessarily block execution, and vice versa.

By Jun Wen Leong

llms rag agents benchmarks safety

arXiv AI

Aug 5

Lean Refactor: Multi-Objective Controllable Proof Optimization via Agentic Strategy Search

arXiv:2605. 20244v2 Announce Type: replace-cross Abstract: We present Lean Refactor, a plug-and-play retrieval-augmented agentic framework for multi-objective, controllable, and version-robust refactoring of Lean proofs.

By Jialin Lu, Soonho Kong, Rodrigo Stehling, Kaiyu Yang, Zhangyang Wang, Weiran Sun, Wuyang Chen

llms rag agents fine-tuning benchmarks

arXiv AI

Aug 5

Ranking Image Fusion the Way Humans Do: A Learned Pairwise Preference Metric for Infrared-Visible Fusion Assessment

arXiv:2608. 01301v2 Announce Type: replace-cross Abstract: Infrared-visible image fusion (IVIF) has no ideal fused reference, so fusion algorithms are routinely ranked by scalar objective metrics that formalize different proxies for information transfer, structure, or source similarity.

By Haoran Liu, Mingzhe Liu, Peng Li, Guibin Zan

benchmarks

arXiv Machine Learning

Aug 5

Evaluation Blindness: How Silent Measurement Failures Corrupt AI Systems from Training to Deployment

arXiv:2608. 02786v1 Announce Type: new Abstract: AI systems can fail silently.

By Priyanka Bajaj (Independent Researcher)

llms fine-tuning benchmarks

arXiv Machine Learning

Aug 5

NOMADD: Numerical Optimization of Models Adapting to Data Drift

arXiv:2608. 02845v1 Announce Type: new Abstract: Tabular model performance degrades when feature distributions change over time or the relationship between features and outcome variables change over time, known as data drift and concept drift, respectively.

By Swapn Shah, Keith Burghardt

benchmarks

arXiv Machine Learning

Aug 5

GoT-CD: Graph-of-Thoughts Causal Discovery and the Fragility of Post-hoc Path-Specific Fairness Audits

arXiv:2608. 02877v1 Announce Type: new Abstract: Causal discovery recovers directed structure from observational data and is increasingly used in clinical settings to support mechanism reasoning and fairness audits of predictive models.

By Nitish Nagesh, Elahe Khatibi, Thomas Dean Hughes, Mahdi Bagheri, Pratik Gajane, Amir M. Rahmani

llms benchmarks safety

arXiv Machine Learning

Aug 5

ATFlash: Per-RoPE-Wavelength Attention Windows for Compute/Memory-Efficient LLM Inference

arXiv:2608. 02947v1 Announce Type: new Abstract: The attention score with rotary position embeddings (RoPE) decomposes exactly into a sum over its 2D-rotation frequency pairs, and each pair's wavelength limits how far it can discriminate position.

By Shun-ichiro Hayashi, Daichi Mukunoki, Tetsuya Hoshino, Takahiro Katagiri

llms rag efficiency benchmarks

arXiv Machine Learning

Aug 5

FedCritic-MIMO: Communication-Efficient Serverless Federated Critic Learning for Massive-MIMO Resource Control in Open and Disaggregated 6G RANs

arXiv:2608. 03852v1 Announce Type: new Abstract: This paper proposes FedCritic-MIMO, a communication-efficient serverless federated multi-agent reinforcement learning framework for AI-native resource control across independently deployable cell-level controllers in open and disaggregated 6G RANs.

By Amin Farajzadeh, Melike Erol-Kantarci

agents reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Aug 5

Automatic Patient-Specific Microwave Ablation Planning Accelerated by a Physics-Guided Deep Learning Model

arXiv:2608. 03086v1 Announce Type: cross Abstract: Microwave ablation (MWA) is a promising minimally invasive treatment for liver tumors, but its therapeutic outcome strongly depends on patient-specific planning of antenna insertion trajectory, power, and treatment duration.

By Seonaeng Cho, Minjee Seo, Minju Seol, Juil Park, Joon Ho Kwon, Kyungho Yoon

benchmarks

arXiv Machine Learning

Aug 5

VIVID: A Culturally Grounded Benchmark Exposing the Figurative Language Gap in Vietnamese NLP

arXiv:2608. 03095v1 Announce Type: cross Abstract: We present VIVID (Vietnamese Idioms for Validation and Interpretation Depth), the first systematic benchmark for evaluating culturally grounded figurative language understanding in Vietnamese.

By Tu Tran Do, Nhat Ngoc Nguyen, Khanh-Tung Tran, Hoang D. Nguyen, Tu Minh Phuong, Long Hoang Dang

llms nlp benchmarks

arXiv Machine Learning

Aug 5

Don't Walk the Line: Boundary Guidance for Filtered Generation

arXiv:2510. 11834v3 Announce Type: replace Abstract: Generative models are increasingly paired with safety classifiers that filter harmful or undesirable outputs.

By Sarah Ball, Andreas Haupt

llms reinforcement-learning fine-tuning benchmarks safety

arXiv Machine Learning

Aug 5

When Classes Evolve: A Benchmark and Framework for Stage-Aware Class-Incremental Learning

arXiv:2602. 00573v2 Announce Type: replace Abstract: Class-Incremental Learning (CIL) aims to sequentially learn new classes while mitigating catastrophic forgetting of previously learned knowledge.

By Zheng Zhang, Tao Hu, Xueheng Li, Yang Wang, Rui Li, Jie Zhang, Chengjun Xie

benchmarks

arXiv AI

Aug 5

On the missing data layer and a potential solution

arXiv:2608. 02949v1 Announce Type: new Abstract: Latin America is missing two foundational layers of AI infrastructure: the dataset layer and the benchmark layer.

By Francis F Daniel, Mauro Iba\~nez, Francis Perelman, Marian Basti

benchmarks

arXiv AI

Aug 5

Beyond Average Performance: Dynamic Instance Clustering and Specialized Algorithm Design in LLM-Assisted Evolutionary Search

arXiv:2608. 03129v1 Announce Type: new Abstract: Large Language Model-assisted Evolutionary Search (LES) has emerged as a powerful paradigm for automated algorithm design.

By Qinglong Hu, Qingfu Zhang, Fei Liu, Xialiang Tong, Kun Mao, Mingxuan Yuan

llms benchmarks

arXiv AI

Aug 5

Reachability Is Not Realization: Tracing the Sources of LLM Benchmark Gains

arXiv:2608. 03219v1 Announce Type: new Abstract: Benchmark gains are often treated as evidence of greater LLM capability.

By Yanchao Li, Wanhao Liu, Jiaqing Xie, Ben Gao, Yanbo Wang, Tianfan Fu, Yuqiang Li

llms benchmarks

arXiv AI

Aug 5

Intertemporal Preference Steering in Qwen3 via Contrastive Activation Addition

arXiv:2608. 03892v1 Announce Type: new Abstract: We study linear representations of temporal horizon in the large language model Qwen3-32B and use them to change the model's time-related preferences, recommendations, and capabilities.

By Michal Mr\'az, Justin Shenk

llms benchmarks safety

arXiv AI

Aug 5

Instruction Stacking Collapse: A Benchmark and the Capability-Dependent Value of Prompt Compilation

arXiv:2608. 02639v1 Announce Type: cross Abstract: Production prompts rarely carry a single instruction.

By Atul Anand, Sourav Chattaraj

llms benchmarks

arXiv AI

Aug 5

Training Documents Reranker with Search Rubrics for Deep Research Agent

arXiv:2608. 03527v1 Announce Type: cross Abstract: Retrieval systems help deep research agents generate high-quality answers by providing relevant documents.

By Wenhan Liu, Yu Lu, Qiaolin Xia, Hui Xu, Tong Zhao, Jian Xi, Yutao Zhu, Haijin Liang, Haibo Shi, Hao Wang, Zhicheng Dou

llms rag agents reinforcement-learning fine-tuning benchmarks

arXiv AI

Aug 5

On the Limits of Layer Pruning for Generative Reasoning in Large Language Models

arXiv:2602. 01997v3 Announce Type: replace-cross Abstract: Recent work has shown that layer pruning can effectively compress large language models (LLMs) while retaining strong performance on classification benchmarks, often with little or no finetuning.

By Safal Shrestha, Anubhav Shrestha, Minwu Kim, Aadim Nepal, Keith Ross

llms fine-tuning efficiency benchmarks

Get the daily briefing