Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv Machine Learning
Oct 15

Beyond Equation Search: Conjecture-Guided Discovery with SCOPE and ClosureBench

The paper introduces SCOPE, a framework that enables multiple scientist agents to pursue conjecture-driven research trajectories by modeling equations, formulating questions, and conducting active experiments, then sharing evidence to refine their conjectures. It also presents ClosureBench, an interactive benchmark of 60 closure-discovery tasks across five scientific domains, designed to evaluate equation discovery beyond accuracy by measuring evidence faithfulness, mechanism insight, and experimental effectiveness. Experiments show that SCOPE outperforms diverse baselines on both ClosureBench and a real-world scientific discovery case, yielding more accurate, reliable, and effective scientific discoveries.

By Yuxiao Hu, Qian Li, Ding Wang, Siheng Chen, Linfeng Zhang, Jinyue Yan, Yuntian Chen
arXiv AI
Oct 15

Is Complex Query Answering Really Complex?

arXiv:2410.12537v4 Announce Type: replace-cross Abstract: Complex query answering (CQA) on knowledge graphs (KGs) is gaining momentum as a challenging reasoning task. In this paper, we show that the...

By Cosimo Gregucci, Bo Xiong, Daniel Hernandez, Lorenzo Loconte, Pasquale Minervini, Steffen Staab, Antonio Vergari
arXiv Machine Learning
Oct 15

Are We Overthinking One-Step Generation?

The paper introduces Priming Models, a two‑stage framework for training one‑step generative models. First, the model is trained on a reconstruction objective such as pixel‑space denoising at a fixed noise level, then it undergoes feature‑space distributional post‑training. This simple reconstruction stage alone is shown to eliminate the difficulty of distributional training, leading to significant improvements in training compute versus generation quality across tasks like class‑conditional image generation, text‑to‑image generation, and world modeling, with a state‑of‑the‑art FID of 1.19 on ImageNet 256×256 using about one‑tenth the compute of previous methods.

By Woobin Park, Jaeyeon Kim, Yilun Du, Sitan Chen, Bohyung Han
arXiv AI
Oct 15

FDE-Bench: Evaluating End-to-End Delivery from Underspecified Real-World Business Requests

arXiv:2610.14539v1 Announce Type: new Abstract: Real-world business requests often leave key requirements unstated because frontline practitioners struggle to articulate rules they know from experien...

By Huaiming Li, Can Huang, Chufan Wu, Danyu Liu, Xiaomin Yuan, Qinze Li, Yui Lo, Wenhui Bai, Qianlong Wang, Zengle Ge, Xiyu Yan, Yuheng Cheng, Jinhui Ren, Guokai Chen, Jiaqun Liu, Haoran Li, Qianhui Liu, Annan Li, Jianmin Wu, Dawei Yin, Dou Shen
arXiv AI
Oct 15

SkillHorizon: Constructing Long-Horizon Agent Tasks through Skill Composition

arXiv:2610.14966v1 Announce Type: new Abstract: Long-horizon agents must complete workflows in which later activities depend on earlier outputs and changes to shared state. A key challenge for evalua...

By Yuxuan Liu, Zhaochen Su, Hangyu Guo, Shijue Huang, Zhixiong Zhang, Kawai Chung, Zhongwei Xie, Tianshi Zheng, Wei Fan, Qing Zong, Haochen Shi, Baixuan Xu, Haoran Li, Yangqiu Song