arXiv Machine Learning
One Capability or Many? Testing the Economic Validity of Frontier AI Evaluation
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Sep 29
One Capability or Many? Structural and Predictive Tests of Benchmark Validity Disagree About Economic Benchmarks for Frontier AI
The paper examines whether economic benchmarks used in frontier AI leaderboards measure a distinct capability or merely reflect general test-taking ability. Using a structural factor analysis and a predictive leave-one-benchmark-out test on a snapshot of 421 model configurations, the authors find that economic benchmarks do not form a separate factor but are better predicted by a multi‑factor representation than by a single general index, especially for linear learners. They argue that construct validity should be evaluated with both structural and predictive tests and provide a two‑test protocol along with data and code.
arXiv Machine Learning
Sep 24
Evaluation Choices Decide the Forecasting Leaderboard: Evidence from a Production Marketplace Panel
The paper demonstrates that the outcome of a forecasting leaderboard is largely determined by the evaluator’s design choices rather than the models themselves. By fixing the data, horizon, and period, the authors varied three key evaluation decisions—unit of analysis, error pooling, and scoring metric—and showed that each can reverse or eliminate the apparent superiority of any forecasting method. The study also evaluates the practical impact of these choices on a deployed system, revealing that the selection rule captures a significant portion of the potential performance gain, and confirms the findings on an external public dataset.
arXiv AI
Aug 18
Frontier AI Forecasting Has a Measurement Problem: An Audit of Progress Evidence
arXiv:2608. 14903v1 Announce Type: new Abstract: Quantitative forecasts of frontier artificial intelligence often connect dated targets to trends in benchmark scores, training compute, release time, or expert belief.
arXiv AI
Aug 13
Deployment Decision Reliability: A Generalizability-Theory Framework for Sizing Long-Horizon Agent Evaluations
arXiv:2608. 11323v1 Announce Type: new Abstract: Enterprise practitioners read agent leaderboards as if they ranked agent capability.
arXiv AI
Jul 3
Meta-Benchmarks for Financial-Services LLM Evaluation
arXiv:2607. 01740v1 Announce Type: new Abstract: Public LLM leaderboards optimise for global average performance and do not capture the specific cognitive demands of financial-services work: a model that leads on MMLU-Pro may underperform on document-grounded compliance reasoning, and a coding leader may handle multi-turn customer interactions poorly.
arXiv AI
Oct 5
Knowledge or Calculator? Decomposing the Skill Premium in Verifiable Financial Agent Workflows
The paper introduces FinSkillBench, a benchmark of 2,603 financial AI agent episodes across portfolio construction, risk management, and fundamental analysis, evaluated with deterministic verifiers. Experiments on 9 models and 3 resource settings show that curated skill packages improve performance by +16.2 points, while in‑episode skill generation adds only +0.5 points. Decomposing the premium reveals that executable tools contribute +19.5 points, documentation +5.6 points, and their combination is subadditive, with the effect varying by workflow type.