arXiv AI August 13, 2026 By Rodrigo Guedes de Souza, Alison R. Panisson

Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation

arXiv:2608. 12150v1 Announce Type: new Abstract: Standard evaluation of large language models assumes stable model rankings across inference conditions.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv AI.

arXiv AI

Jun 26

The Capability Frontier: Benchmarks Miss 82% of Model Performance

arXiv:2606. 26836v1 Announce Type: new Abstract: Existing benchmarks typically report accuracy for a single model on a single run.

By Bradley Fowler, Ryan Smith, Daniel Thi Graviet, William Myers, Joshua Greaves, Narmeen Fatimah Oozeer, Ant\'ia Garc\'ia, Philip Quirke, Amirali Abdullah, Fazl Barez, Shriyash Kaustubh Upadhyay

llms agents benchmarks

arXiv AI

Jun 24

CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression

arXiv:2606. 24083v1 Announce Type: cross Abstract: "Talk short.

By Morayo Danielle Adeyemi, Ryan A. Rossi, Franck Dernoncourt

llms benchmarks

arXiv Machine Learning

Jul 10

Resample or Reroute? Budget-Aware Test-Time Model Selection for Large Language Models

arXiv:2607. 08665v1 Announce Type: new Abstract: Routing among large language models (LLMs) trades response quality against serving cost, motivated by the reported gap between deployed routers and a per-instance oracle.

By Teng-Ruei Chen

llms benchmarks

arXiv AI

2d ago

Not All Tokens Are Equal: Inflation-Aware Routing for Agentic LLM Systems

arXiv:2608. 13571v1 Announce Type: cross Abstract: When a language model fails to answer a query on the first attempt, an agentic system retries, consuming additional tokens each time.

By Heming Fu, Shan Lin, Qianqian Xie, Guojun Xiong

llms agents nlp

arXiv AI

Jun 9

SLMJury: Can Small Language Models Judge as Well as Large Ones?

arXiv:2606. 07810v1 Announce Type: cross Abstract: Large language models (LLMs) are widely used as judges for evaluating model outputs, but their high cost, latency, and opacity limit scalability.

By Anish Laddha, Nitesh Pradhan, Gaurav Srivastava

llms agents nlp benchmarks safety

arXiv Machine Learning

Aug 5

M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models

arXiv:2608. 03803v1 Announce Type: cross Abstract: Multilingual language models are deployed across a hundred or more languages, yet most benchmarks test whether a model can perform a task _in_ a language rather than whether it commands the language itself, conflating fluency with proficiency.

By Tom\'a\v{s} Burkert, Angelika Peljak-{\L}api\'nska, David Zelen\'y

llms nlp benchmarks safety

Related stories

The Capability Frontier: Benchmarks Miss 82% of Model Performance

CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression

Resample or Reroute? Budget-Aware Test-Time Model Selection for Large Language Models

Not All Tokens Are Equal: Inflation-Aware Routing for Agentic LLM Systems

SLMJury: Can Small Language Models Judge as Well as Large Ones?

M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models