Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

23,288 stories · RSS feed

arXiv AI

Jun 4

Learning to Evaluate: Cost-Effective Model Evaluation on Unlabeled Data with Meta-Learning

arXiv:2605. 23595v2 Announce Type: replace-cross Abstract: The rapid advancement of machine learning has led to an unprecedented expansion of model ecosystems, making it increasingly difficult to assess the reliability of newly released models on unseen and unlabeled data.

By Trinh Pham, Viet Huynh, Hongzhi Yin, Quoc Viet Hung Nguyen, Thanh Tam Nguyen

fine-tuning benchmarks

arXiv AI

Jun 4

Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning

arXiv:2505. 17315v2 Announce Type: replace Abstract: Recent language models exhibit strong reasoning capabilities, yet the influence of long-context capacity on reasoning remains underexplored.

By Wang Yang, Zirui Liu, Hongye Jin, Qingyu Yin, Vipin Chaudhary, Xiaotian Han

llms fine-tuning benchmarks

arXiv AI

Jun 4

Temporal Order Matters for Agentic Memory: Segment Trees for Long-Horizon Agents

arXiv:2606. 04555v1 Announce Type: cross Abstract: Long-horizon conversational agents need to interact with users through evolving events, tasks, and goals.

By Yifan Simon Liu, Liam Gallagher, Faeze Moradi Kalarde, Jiazhou Liang, Armin Toroghi, Scott Sanner

llms agents benchmarks

arXiv Machine Learning

Jun 4

SpliceBind: Isoform-Aware Prediction of Binding Pocket Druggability

arXiv:2606. 04020v1 Announce Type: cross Abstract: Splice-mediated drug resistance occurs in up to 40% of patients on targeted kinase inhibitors, yet state-of-the-art druggability tools operate on single structures and cannot compare across isoforms.

By Bryan Cheng, Austin Jin, Joshua Chang

rag benchmarks

arXiv AI

Jun 4

Bilevel Autoresearch: Meta-Autoresearching Itself

arXiv:2603. 23420v2 Announce Type: replace Abstract: If autoresearch is itself a form of research, then autoresearch can be applied to research itself.

By Yaonan Qu, Meng Lu

llms agents benchmarks

arXiv AI

Jun 4

HighTide: An Agent-Curated Open-Source VLSI Benchmark Suite

arXiv:2606. 04126v1 Announce Type: cross Abstract: We introduce HighTide, an evolving AI-assisted benchmark suite.

By Benjamin Goldblatt, Paolo Pedroso, Farhad Modaresi, Ethan Sifferman, Matthew R. Guthaus

agents benchmarks

arXiv AI

Jun 4

The Variance Brain Foundation Models Forgot: Third-Order Statistics Predict Cognition Where Billion-Parameter Models Fail

arXiv:2606. 04010v1 Announce Type: cross Abstract: Brain foundation models (BFMs) are self-supervised Transformers pretrained on fMRI data.

By Giovanni Marraffini, Gabriel Mahuas, Trinidad Borrell, Victoria Shevchenko, Demian Wassermann

llms fine-tuning benchmarks

arXiv AI

Jun 4

Parthenon Law: A Self-Evolving Legal-Agent Framework

arXiv:2606. 04602v1 Announce Type: new Abstract: As agents grow more capable, legal-domain LLM agents promise to turn document-heavy matters into reviewable work products -- yet reliable deployment faces three obstacles: no large-scale evidence on how today's strongest model-and-harness combinations behave on end-to-end legal matters; no agent architecture adapted to the legal vertical, only general-purpose harnesses; and, in a setting that keeps shifting with new facts, authorities, and deadlines, no mechanism for systems to learn from their own outcomes.

By Hejia Geng, Leo Liu

llms agents benchmarks

arXiv AI

Jun 4

LoopFM: Learning frOm HistOrical RePresentations of Foundation Model for Recommendation

arXiv:2605. 29280v2 Announce Type: replace-cross Abstract: Knowledge distillation (KD) transfers a single scalar prediction from a large foundation model (FM) to compact vertical models (VMs), suffering from diminishing transfer ratio -- the fraction of FM improvement captured by the VM -- as a single scalar cannot convey the rich intermediate knowledge that larger FMs learn.

By Shali Jiang, Hua Zheng, Boyang Liu, Laming Chen, Kenny Lov, Chuanqi Xu, Lisang Ding, Qinghai Zhou, Can Cui, Xiaolong Liu, Xiaoyi Liu, Yasmine Badr, Xin Xu, Jiyan Yang, Ellie Dingqiao Wen, Gerard Jonathan Mugisha Akkerhuis, Chenxiao Guan, Rong Jin, Ruichao Qiu, Xian Chen, Shifu Xu, Zhehui Zhou, Ping Chen, Rui Yang, Haicheng Chen, Xiangge Meng, Song Zhou, Dharak Kharod, Shuyu Xu, Qiang Jin, Qiao Yang, Wankun Zhu, Qin Huang, Yuzhen Huang, Darren Liu, Parish Aggarwal, Hui Zhou, Erzhuo Wang, Shuo Chang, Xiaorui Gan, Wenlin Chen, Santanu Kolay, Huayu Li

rag efficiency benchmarks

arXiv AI

Jun 4

Scaling Novel Graph Generation via Lightweight Structure-Guided Autoregressive Models

arXiv:2606. 04287v1 Announce Type: cross Abstract: Generating realistic and diverse graphs is a key problem in machine learning, with applications in molecular discovery, circuit design, cybersecurity, and beyond.

By Alessio Barboni, Massimiliano Lupo Pasini, Bishal Lakha, Edoardo Serra

diffusion benchmarks

arXiv AI

Jun 4

Luminol-AIDetect: Fast Zero-shot Machine-Generated Text Detection based on Perplexity under Text Shuffling

arXiv:2604. 25860v2 Announce Type: replace-cross Abstract: Machine-generated text (MGT) detection requires identifying structurally invariant signals across generation models, rather than relying on model-specific fingerprints.

By Lucio La Cava, Andrea Tagarelli

llms benchmarks safety

arXiv Machine Learning

Jun 4

RIDE: An Open Dataset and Benchmark for Train Delay Prediction

arXiv:2606. 05070v1 Announce Type: new Abstract: Train delay prediction is an important problem for both passengers and railway operators, yet progress in the field remains difficult to assess due to the lack of standardized datasets, prediction targets, and evaluation protocols.

By Cl\'ement Elliker, Mathis Le Bail, Cl\'ement Mantoux, Jesse Read, Sonia Vanier

benchmarks data-engineering

arXiv AI

Jun 4

dMX: Differentiable Mixed-Precision Assignment for Low-Precision Floating-Point Formats

arXiv:2606. 04115v1 Announce Type: cross Abstract: Quantizing large language models (LLMs) to low-precision floating-point representations is central to efficient deployment, yet applying a single bit-width uniformly across all layers is sub-optimal in terms of both performance and accuracy.

By Giuseppe Franco, Ian Colbert, Pablo Monteagudo-Lago, Felix Marty, Nicholas Fraser

llms efficiency benchmarks

arXiv AI

Jun 4

Can Generalist Agents Automate Data Curation?

arXiv:2606. 04261v1 Announce Type: new Abstract: Curating training data is among the most consequential yet labor-intensive parts of modern AI development: practitioners iteratively propose, implement, evaluate, and revise data policies against noisy benchmark feedback.

By Feiyang Kang, Hanze Li, Adam Nguyen, Mahavir Dabas, Jiaqi W. Ma, Frederic Sala, Dawn Song, Ruoxi Jia

llms agents multimodal benchmarks

arXiv Machine Learning

Jun 4

Provably Reduced Sample Cost in Prior-Guided Hyperparameter Optimization

arXiv:2606. 04866v1 Announce Type: new Abstract: Large-scale hyperparameter optimization (HPO) in automated machine learning (AutoML) consumes substantial computational resources, raising growing concerns about scalability and energy efficiency.

By Leona Hennig, Jasmin Brandt, Lukas Fehring, Barbara Hammer, Marius Lindauer, Marcel Wever

benchmarks

arXiv Machine Learning

Jun 4

It's TIME: Towards the Next Generation of Time Series Forecasting Benchmarks

arXiv:2602. 12147v4 Announce Type: replace Abstract: Time series foundation models (TSFMs) are revolutionizing the forecasting landscape from specific dataset modeling to generalizable task evaluation.

By Zhongzheng Qiao, Sheng Pan, Anni Wang, Viktoriya Zhukova, Yong Liu, Xudong Jiang, Qingsong Wen, Mingsheng Long, Ming Jin, Chenghao Liu

llms benchmarks

arXiv AI

Jun 4

AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?

arXiv:2606. 05080v1 Announce Type: new Abstract: Scientific and engineering progress is fundamentally a long-horizon iterative process: proposing changes, running experiments, measuring outcomes, and continuously refining artifacts.

By Zhangchen Xu, Junda Chen, Yue Huang, Dongfu Jiang, Jiefeng Chen, Hang Hua, Zijian Wu, Zheyuan Liu, Zexue He, Lichi Li, Shizhe Diao, Jiaxin Pei, Jinsung Yoon, Hao Zhang, Mengdi Wang, Radha Poovendran, Misha Sra, Alex Pentland, Zichen Chen

llms agents benchmarks

arXiv AI

Jun 4

UniCAD: A Unified Benchmark and Universal Model for Multi-Modal Multi-Task CAD

arXiv:2606. 05058v1 Announce Type: cross Abstract: Computer-Aided Design (CAD) underpins modern engineering and manufacturing by enabling the creation of precise, editable 3D models.

By Jingyuan Chen, Sheng Jin, Haopeng Sun, Wentao Liu, Chen Qian

llms nlp multimodal benchmarks

arXiv AI

Jun 4

LiftQuant: Continuous Bit-Width LLM via Dimensional Lifting and Projection

arXiv:2606. 04050v1 Announce Type: cross Abstract: Existing quantization methods are fundamentally limited by rigid, integer-based bit-widths (e.

By Liulu He, XuanAng Liu, Juntao Liu, Taolue Feng, Ting Lu, Chunsheng Gan, Zhiyv Peng, Yuan Du, Huanrui Yang, Yijiang Liu, Li Du

llms efficiency benchmarks

arXiv Machine Learning

Jun 4

Towards A Generative Protein Evolution Machine with DPLM-Evo

arXiv:2605. 00182v3 Announce Type: replace Abstract: Proteins are shaped by gradual evolution under biophysical and functional constraints.

By Xinyou Wang, Liang Hong, Jiasheng Ye, Zaixiang Zheng, Yu Li, Shujian Huang, Quanquan Gu

llms diffusion benchmarks safety

Get the daily briefing