Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv Computation and Language
Oct 1

SpanUQ: Span-Level Uncertainty Quantification for Large Language Model Generation

arXiv:2607.05721v2 Announce Type: replace Abstract: Uncertainty estimation is essential not only for the trustworthy deployment of large language models (LLMs) but also as a foundation for self-refin...

By Yimeng Zhang, Yingying Zhuang, Ziyi Wang, Yuxuan Lu, Pei Chen, Aman Gupta, Zhe Su, Ming Tan, Zhilin Zhang, Qun Liu, Manikandarajan Ramanathan, Rajashekar Maragoud, Edward Vul, Jing Huang, Dakuo Wang
arXiv AI
Oct 1

MILO: Automated Harness Discovery via Orchestrated Multi-Agent Evolution

arXiv:2609.38349v1 Announce Type: cross Abstract: Modern agentic systems combine an AI model with a harness that controls execution and environmental interactions. Harness design strongly affects lon...

By Prithwish Jana, Mononito Goswami, Hao Liu, Xinyu Li, Langlin Huang, Zhehui Huang, Zhishen Huang, Patrick Bl\"obaum, Anoop Deoras, Purak Jain, Nikos Kanakaris, Sahika Genc
arXiv Computer Vision
Oct 1

TrafficSignBench: Rule-Centric Closed-Loop Evaluation of Traffic-Sign Compliance in Autonomous Driving

arXiv:2609.38463v1 Announce Type: cross Abstract: Autonomous driving planners are typically evaluated using aggregate metrics such as driving score, destination rate, and collision rate, which do not...

By Victoria Smirnova, Viktoriia Zinkovich, Gregorii Bukhtuev, Artem Belyaev, Andrey Kuznetsov, Denis Shepelev, Vlad Shakhuro