arXiv AI
Jun 26
The Capability Frontier: Benchmarks Miss 82% of Model Performance
arXiv:2606. 26836v1 Announce Type: new Abstract: Existing benchmarks typically report accuracy for a single model on a single run.
Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation
arXiv:2608. 12150v1 Announce Type: new Abstract: Standard evaluation of large language models assumes stable model rankings across inference conditions.
Related stories
arXiv AI
Jun 24
CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression
arXiv:2606. 24083v1 Announce Type: cross Abstract: "Talk short.
arXiv Machine Learning
Jul 10
Resample or Reroute? Budget-Aware Test-Time Model Selection for Large Language Models
arXiv:2607. 08665v1 Announce Type: new Abstract: Routing among large language models (LLMs) trades response quality against serving cost, motivated by the reported gap between deployed routers and a per-instance oracle.
arXiv AI
2d ago
Not All Tokens Are Equal: Inflation-Aware Routing for Agentic LLM Systems
arXiv:2608. 13571v1 Announce Type: cross Abstract: When a language model fails to answer a query on the first attempt, an agentic system retries, consuming additional tokens each time.
arXiv AI
Jun 9
SLMJury: Can Small Language Models Judge as Well as Large Ones?
arXiv:2606. 07810v1 Announce Type: cross Abstract: Large language models (LLMs) are widely used as judges for evaluating model outputs, but their high cost, latency, and opacity limit scalability.
arXiv Machine Learning
Aug 5
M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models
arXiv:2608. 03803v1 Announce Type: cross Abstract: Multilingual language models are deployed across a hundred or more languages, yet most benchmarks test whether a model can perform a task _in_ a language rather than whether it commands the language itself, conflating fluency with proficiency.
arXiv AI
Jul 28
Understanding Tone-Dependent Inference Cost in Large Language Models
arXiv:2607. 23915v1 Announce Type: cross Abstract: We examine how prompt tone affects both accuracy of the LLM answers and inference cost as reflected in output-token consumption.
arXiv AI
Aug 10
FutureBridge: Token Selection Beyond Local Preference in Collaborative Decoding
arXiv:2608. 06819v1 Announce Type: cross Abstract: Token-level collaboration allows a large language model (LLM) to assist a small language model (SLM) when their predictions diverge.
arXiv Machine Learning
Jul 20
VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs
arXiv:2607. 15498v1 Announce Type: cross Abstract: The key-value (KV) cache is the main memory bottleneck in long-context large language model (LLM) inference.
arXiv Machine Learning
Jun 25
Brevity is the Soul of Inference Efficiency: Inducing Concision in VLMs via Data Curation
arXiv:2606. 25432v1 Announce Type: new Abstract: Inference efficiency is typically pursued by shrinking the model: distillation, pruning, quantization, and sparse routing each lower per-token cost while treating token count as fixed.
arXiv AI
Jul 14
Filtered Reasoning Score: Evaluating Reasoning Quality on a Model's Most-Confident Traces
arXiv:2604. 11996v2 Announce Type: replace-cross Abstract: Should we trust Large Language Models (LLMs) with high accuracy?
arXiv AI
Jul 21
Retain or Consolidate? Budget-Dependent Operator Selection for Language Agent Memory
arXiv:2607. 17545v1 Announce Type: new Abstract: Language agents depend on memory across interactions.