Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

13,227 stories · RSS feed

arXiv AI
Aug 14

DiG-bench: Discovery in Games

arXiv:2608. 12593v1 Announce Type: new Abstract: Discovery---formulating novel generalizations---is a central part of the scientific process.

By Ruairidh M. Battleday, Kai Sandbrink, Jimi Cullen-Drohan, Zihan Yan, Timothy Muller, Clare Maguire, Ales Kubicek, Fraser Greenlee-Scott, Sukrit Sumant, Tri Dao, J\"urgen Schmidhuber, Michal Valko, Joshua Tenenbaum, Thomas L. Griffiths, Zeb Kurth-Nelson, James C. R. Whittington
arXiv AI
Aug 14

Robust Dempster-Shafer Evidence Fusion with Chaos-Conflict Measurement and Historical-Experience Weighting

arXiv:2608. 13108v1 Announce Type: new Abstract: Multi-source evidence fusion under Dempster-Shafer theory faces two persistent challenges: existing conflict measures assess inter-evidence inconsistency and intra-evidence uncertainty independently, yielding incomplete evaluations, and current fusion methods evaluate evidence sources exclusively through instantaneous comparisns without exploiting their long-term reliability across diverse decision contexts.

By Huiyu Li, Weibo Liu, Xinru Xu, Dongchen Gao, Meng Zhang, Junhua Hu
arXiv AI
Aug 14

SSPO: Structure-Aware Similarity-Weighted Preference Optimization for Neural Combinatorial Optimization

arXiv:2608. 12443v1 Announce Type: cross Abstract: Neural combinatorial optimization (NCO) relies on parallel solution sampling for training, yet existing methods fail to fully exploit the rich information latent in a co-sampled solution group.

By Yuanyu Li, Jintao Xu, Zijiang Liu, Yongzhi Qi, Ningxuan Kang, Jianshen Zhang, Wei Qi, Chen Xie, Zuo-Jun Max Shen
arXiv AI
Aug 14

Error-Aware Reverse Auction Mechanism for Large Language Model Routing

arXiv:2608. 12719v1 Announce Type: cross Abstract: Routing each query to a cost-effective large language model (LLM) is critical for balancing quality and cost, yet most routers rely on a centralized task center to predict model performance, creating an information-risk mismatch and a scalability bottleneck as the model pool grows.

By Haolong Chen, Zhengyuan Xin, Liang Zhang, Lei Xue, Guangxu Zhu
arXiv Machine Learning
Aug 14

FlowLOB: Efficient and Controllable Limit Order Book Generation with Flow Matching

arXiv:2608. 13096v1 Announce Type: new Abstract: Limit order book (LOB) simulators are most useful to practitioners when they combine realistic market dynamics, computationally efficient sampling, controllable scenario generation, and the ability to generalize beyond the instruments seen during training---properties that existing agent-based and deep generative simulators provide only partially.

By Zhuohan Wang, Andreea Bacalum, Ollie Olby, Carmine Ventre, Namid Stillman