Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

23,305 stories · RSS feed

arXiv AI
Jun 3

ASAP: Exploiting the Satisficing Generalization Edge in Neural Combinatorial Optimization

arXiv:2501. 17377v4 Announce Type: replace-cross Abstract: Deep Reinforcement Learning (DRL) has emerged as a promising approach for solving Combinatorial Optimization (CO) problems, such as the 3D Bin Packing Problem (3D-BPP), Traveling Salesman Problem (TSP), or Vehicle Routing Problem (VRP), but these neural solvers often exhibit brittleness when facing distribution shifts.

By Han Fang, Paul Weng, Yutong Ban
arXiv AI
Jun 3

WebRISE: Requirement-Induced State Evaluation for MLLM-Generated Web Artifacts

arXiv:2606. 03220v1 Announce Type: cross Abstract: Existing benchmarks for MLLM-generated web artifacts assess interaction through local evidence and miss the requirement-induced states and transitions that determine whether a page works.

By Yuxin Meng, Yuhan Suo, Junjie Wang, Yuhan Sun, Yiyao Yu, Ruixu Zhang, Ruining Hu, Yubin Wang, Shouwei Ruan, Bin Wang, Yuxiang Zhang, Yujiu Yang
arXiv AI
Jun 3

BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents

arXiv:2606. 03829v1 Announce Type: new Abstract: Financial-research answers are decision-relevant only when another analyst can audit how they were produced: which source was chosen, which period and accounting definition were used, which assumptions were made, and how the calculation was performed.

By Alex Wang, Georg Meinhardt, Jacob Katz, Joseph H. Kim, Pratyush K. Chaudhary, Chase Blagden, Eric Xu
arXiv Machine Learning
Jun 3

Testing Most Influential Sets

arXiv:2510. 20372v4 Announce Type: replace-cross Abstract: Small influential data subsets can dramatically impact model conclusions, with a few data points overturning key findings.

By Lucas D. Konrad, Nikolas Kuschnig