Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

14,048 stories · RSS feed

arXiv Machine Learning

Aug 5

When Classes Evolve: A Benchmark and Framework for Stage-Aware Class-Incremental Learning

arXiv:2602. 00573v2 Announce Type: replace Abstract: Class-Incremental Learning (CIL) aims to sequentially learn new classes while mitigating catastrophic forgetting of previously learned knowledge.

By Zheng Zhang, Tao Hu, Xueheng Li, Yang Wang, Rui Li, Jie Zhang, Chengjun Xie

benchmarks

arXiv AI

Aug 5

On the missing data layer and a potential solution

arXiv:2608. 02949v1 Announce Type: new Abstract: Latin America is missing two foundational layers of AI infrastructure: the dataset layer and the benchmark layer.

By Francis F Daniel, Mauro Iba\~nez, Francis Perelman, Marian Basti

benchmarks

arXiv AI

Aug 5

Beyond Average Performance: Dynamic Instance Clustering and Specialized Algorithm Design in LLM-Assisted Evolutionary Search

arXiv:2608. 03129v1 Announce Type: new Abstract: Large Language Model-assisted Evolutionary Search (LES) has emerged as a powerful paradigm for automated algorithm design.

By Qinglong Hu, Qingfu Zhang, Fei Liu, Xialiang Tong, Kun Mao, Mingxuan Yuan

llms benchmarks

arXiv AI

Aug 5

Reachability Is Not Realization: Tracing the Sources of LLM Benchmark Gains

arXiv:2608. 03219v1 Announce Type: new Abstract: Benchmark gains are often treated as evidence of greater LLM capability.

By Yanchao Li, Wanhao Liu, Jiaqing Xie, Ben Gao, Yanbo Wang, Tianfan Fu, Yuqiang Li

llms benchmarks

arXiv AI

Aug 5

Intertemporal Preference Steering in Qwen3 via Contrastive Activation Addition

arXiv:2608. 03892v1 Announce Type: new Abstract: We study linear representations of temporal horizon in the large language model Qwen3-32B and use them to change the model's time-related preferences, recommendations, and capabilities.

By Michal Mr\'az, Justin Shenk

llms benchmarks safety

arXiv AI

Aug 5

Instruction Stacking Collapse: A Benchmark and the Capability-Dependent Value of Prompt Compilation

arXiv:2608. 02639v1 Announce Type: cross Abstract: Production prompts rarely carry a single instruction.

By Atul Anand, Sourav Chattaraj

llms benchmarks

arXiv AI

Aug 5

Training Documents Reranker with Search Rubrics for Deep Research Agent

arXiv:2608. 03527v1 Announce Type: cross Abstract: Retrieval systems help deep research agents generate high-quality answers by providing relevant documents.

By Wenhan Liu, Yu Lu, Qiaolin Xia, Hui Xu, Tong Zhao, Jian Xi, Yutao Zhu, Haijin Liang, Haibo Shi, Hao Wang, Zhicheng Dou

llms rag agents reinforcement-learning fine-tuning benchmarks

arXiv AI

Aug 5

On the Limits of Layer Pruning for Generative Reasoning in Large Language Models

arXiv:2602. 01997v3 Announce Type: replace-cross Abstract: Recent work has shown that layer pruning can effectively compress large language models (LLMs) while retaining strong performance on classification benchmarks, often with little or no finetuning.

By Safal Shrestha, Anubhav Shrestha, Minwu Kim, Aadim Nepal, Keith Ross

llms fine-tuning efficiency benchmarks

arXiv AI

Aug 5

Cross-Layer Interaction under Weight-Space Ablation: A Closed-Form Attention Jacobian Bound and a Test on a Real Pretrained Model

arXiv:2608. 03629v1 Announce Type: new Abstract: A companion paper studies when activation patching and weight-space ablation agree, inside an idealized model where a conditional computation is carried additively through a residual stream.

By Abdallah Khemais

llms benchmarks

arXiv AI

Aug 5

Beyond Representational Similarity: Source-Conditioned Description-Length Gain for Generative Plagiarism Detection and Candidate Source Reranking

arXiv:2608. 03859v1 Announce Type: cross Abstract: Large language models (LLMs) pose challenges to academic integrity and peer review.

By Peijia Guo, Wenxuan Xie, ZiGuang Li, Ming Li

llms rag benchmarks

arXiv AI

Aug 5

Externally Validated Breast Ultrasound Segmentation via Multi-task Learning with BI-RADS-Consistent Morphological Priors

arXiv:2511. 15968v2 Announce Type: replace-cross Abstract: External validation of breast ultrasound segmentation models remains limited because internal train--test splits do not capture domain shifts across imaging systems, acquisition protocols, and patient populations.

By Jingru Zhang, Saed Moradi, Ashirbani Saha

computer-vision benchmarks

arXiv AI

Aug 5

Output-Aware Rotation for INT2 KV-Cache Quantization

arXiv:2608. 02691v1 Announce Type: cross Abstract: The key-value (KV) cache has become a major memory and bandwidth bottleneck in long-context large language model inference, making ultra-low-bit quantization increasingly important.

By Vincent-Daniel Yun, Woosang Lim, Minsoo Cheong, Sunwoo Lee, Murali Annavaram, Sai Praneeth Karimireddy, Sungjoo Yoo

llms efficiency benchmarks

arXiv Machine Learning

Aug 5

Operationally Feasible Synthetic Power-Grid Scenarios via Learning the AC-Operable Joint Distribution

arXiv:2608. 03878v1 Announce Type: new Abstract: Synthetic power-grid scenarios are essential for planning, resilience assessment, contingency analysis, and data-driven power-system applications.

By Chenhan Xiao, Xinyu He, Haoran Li, Hanghang Tong, Yang Weng

diffusion benchmarks

arXiv AI

Aug 5

Can LLMs Test Terminal User Interfaces?

arXiv:2608. 03743v1 Announce Type: cross Abstract: Terminal User Interfaces (TUIs) combine the stateful, screen-oriented behaviour of GUIs with terminal deployment and are now common in developer tools.

By Chao Peng, Ruida Hu, Ajitha Rajan, Tegawend\'e F Bissyand\'e, Jacques Klein, Cuiyun Gao

llms efficiency benchmarks

arXiv AI

Aug 5

CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement

arXiv:2608. 03890v1 Announce Type: cross Abstract: A clinically useful chest X-ray system must go beyond fluent report generation: it should classify findings with tunable decision thresholds, localize them spatially, and derive the anatomical measurements upon which many diagnoses depend.

By Mercy Prasanna Ranjit, Anirban Porya, Sathvik Joel, Niharika Vadlamudi, Nikhilesh Chowdary Eathamukkala, Prasanth V V, Abhyuday Kumara Swamy, Pranay Narhari Umredkar, Pradeep Narayan, Vivek Rajagopal, Tanuja Ganu

llms nlp multimodal benchmarks

arXiv Machine Learning

Aug 5

Crayotter: Learning Long-Horizon Video Editing Agents via Group-Relative Preference Backpropagation

arXiv:2608. 02694v1 Announce Type: cross Abstract: Long-horizon video editing agents receive final-product feedback only after many interdependent decisions.

By Lecheng Yan, Jianze Lin, Yichong Zhang, Ben Pan, Wenxi Li, Chenyang Lyu, Liting Zhou, Cathal Gurrin

agents benchmarks

arXiv Machine Learning

Aug 5

Joint Affine Spectral Shaping: Coupling Weight and Bias Updates Beyond Weight-Only Muon

arXiv:2608. 02991v1 Announce Type: new Abstract: Matrix spectral optimizers reshape weight-update spectra but usually delegate vector-valued biases to a separate optimizer.

By Gongyue Zhang, Honghai Liu

benchmarks safety

arXiv Machine Learning

Aug 5

SAKI: Score-Aware Low-Rank Key Indexing for Long-Context KV Retrieval

arXiv:2608. 03228v1 Announce Type: new Abstract: Existing low rank KV cache methods preserve either model weights or key variance, neither of which directly reflects the attention scores used during inference.

By Lin Zhang

llms efficiency benchmarks

arXiv Machine Learning

Aug 5

AgentStream: How Well Do Self-Evolving LLM Agents Perform Under Streaming Tasks?

arXiv:2608. 00155v1 Announce Type: cross Abstract: Large language model (LLM) agents can self-evolve by continually improving from their own accumulated experience.

By Dong Yan, Jian Liang, Dapeng Hu, Ran He, Nicholas Jing Yuan, Qi Zhang, Tieniu Tan

llms agents benchmarks

arXiv AI

Aug 5

TumorBoard: Evidence-Grounded Multi-Agent Decision Support for Longitudinal Neuro-Oncology

arXiv:2608. 03190v1 Announce Type: new Abstract: Neuro-oncology decisions require coordinated interpretation of serial MRI, pathology, molecular markers, treatment history, performance status, and evolving guidelines.

By Yantong Liu, Zheyu Zhang, Runpeng Liu, Mu Xitang, Seong-Yoon Shin, Hyun-Ae Lee

agents benchmarks safety

Get the daily briefing