arXiv AI August 17, 2026 By Yanshi Li, Xueru Bai, Shuman Liu, Long Zhang

RepBench: Compiling Benchmarks into Capability Representations for Large Language Models

arXiv:2607. 28008v2 Announce Type: replace-cross Abstract: Representation engineering reads and steers capability directions in large language models, yet methods are typically evaluated on paper-specific synthetic data.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv AI.

arXiv AI

Jun 2

Capturing LLM Capabilities via Evidence-Calibrated Query Clustering

arXiv:2605. 17110v2 Announce Type: replace Abstract: Query clustering organizes queries into groups that reflect shared latent capability demands, enabling capability-aware LLM evaluation.

By Fangzhou Wu, Sandeep Silwal, Qiuyi Zhang

llms rag

arXiv AI

Jun 26

The Capability Frontier: Benchmarks Miss 82% of Model Performance

arXiv:2606. 26836v1 Announce Type: new Abstract: Existing benchmarks typically report accuracy for a single model on a single run.

By Bradley Fowler, Ryan Smith, Daniel Thi Graviet, William Myers, Joshua Greaves, Narmeen Fatimah Oozeer, Ant\'ia Garc\'ia, Philip Quirke, Amirali Abdullah, Fazl Barez, Shriyash Kaustubh Upadhyay

llms agents benchmarks

arXiv AI

2d ago

Don't Claim Benchmark-Oriented Optimization Improves General Coding Capability -- Diverse Evaluation Is Required

arXiv:2608. 13566v1 Announce Type: cross Abstract: Post-training papers, model cards, and blog posts often treat scores on a small set of coding benchmarks (e.

By Egor Shibaev, Vera Kudrevskaia, Timur Galimzyanov, Mikhail Evtikhiev, Ana Terna, Rastislav Rabatin, Timur Kudashev, Timofey Bryksin, Arina Puchkova, Patrik Bartak, Egor Bogomolov, Sergey Titov

llms agents fine-tuning benchmarks

arXiv AI

Jul 17

OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios

arXiv:2607. 14989v1 Announce Type: cross Abstract: Large language models are increasingly evolving from text generators into general agents capable of understanding user requests, invoking external tools, and completing complex tasks through interaction.

By Chengyu Shen, Yujie Fu, Gangtao Xin, Yanheng Hou, Wenlong Fei, Guojie Zhu, Jiawei Li, Hongcheng Gao, Runming He, Zhen Hao Wong, Meiyi Qiang, Hao Liang, Zhao Cao, Hao Jiang, Chong Chen, Wentao Zhang

llms agents benchmarks

arXiv AI

Jun 2

Uncovering Competency Gaps in Large Language Models and Their Benchmarks

arXiv:2512. 20638v2 Announce Type: replace-cross Abstract: The evaluation of large language models relies heavily on standardized benchmarks.

By Maty Bohacek, Nino Scherrer, Nicholas Dufour, Thomas Leung, Christoph Bregler, Stephanie C. Y. Chan

llms benchmarks

arXiv AI

Jul 20

CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Data

arXiv:2607. 16122v1 Announce Type: new Abstract: Evaluations should do more than measure a models current performance.

By Vipul Gupta, Zihao Wang, Razvan-Gabriel Dumitru, MohammadHossein Rezaei, Aakash Sabharwal, Yunzhong He

llms fine-tuning benchmarks

Related stories

Capturing LLM Capabilities via Evidence-Calibrated Query Clustering

The Capability Frontier: Benchmarks Miss 82% of Model Performance

Don't Claim Benchmark-Oriented Optimization Improves General Coding Capability -- Diverse Evaluation Is Required

OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios

Uncovering Competency Gaps in Large Language Models and Their Benchmarks

CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Data