Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,071 stories · RSS feed

arXiv AI

Jun 19

Hard or Just Unreached? Diagnosing the Sampling Blind Spot in Math-Reasoning Difficulty Estimation

arXiv:2606. 19636v1 Announce Type: cross Abstract: Math and science reasoning benchmarks rely on pass@k, the fraction of sampled chains that reach gold, as the canonical per-example difficulty signal.

By Luca Zhou, Sajel Shah, Emanuele Rodol\`a, Roberto Dess\`i

reinforcement-learning benchmarks

arXiv AI

Jun 19

Process-Verified Reinforcement Learning for Theorem Proving via Lean

arXiv:2606. 20068v1 Announce Type: new Abstract: While reinforcement learning from verifiable rewards (RLVR) typically has relied on a single binary verification signal, symbolic proof assistants in formal reasoning offer rich, fine-grained structured feedback.

By Minsu Kim, Se-Young Yun

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jun 19

A Model-Driven Approach for Developing Families of Reinforcement Learning Environments

arXiv:2606. 20324v1 Announce Type: cross Abstract: Virtual training environments are software-intensive systems in which reinforcement learning (RL) agents learn, adapt, and demonstrate meaningful behavior.

By Xiaoran Liu, Istvan David

agents reinforcement-learning benchmarks

arXiv Machine Learning

Jun 19

Quantifying Aleatoric Uncertainty of In-Context Learning for Robust Measure of LLM Prediction Confidence

arXiv:2606. 19353v1 Announce Type: cross Abstract: In-Context Learning (ICL) allows LLMs to adapt to new tasks from a few demonstrations, but its reliability remains a concern: predictions are highly sensitive to both prompt design and the model's ability to understand the context, obscuring whether failures arise from data properties or model limitations.

By Jinseok Chung, Minkyoung Song, Hyunji Jung, Namhoon Lee

llms benchmarks safety

arXiv AI

Jun 19

Sign-Language Datasets at Scale: A Comprehensive Survey on Resources, Benchmarks, and Annotation Standards

arXiv:2606. 19352v1 Announce Type: cross Abstract: Sign languages are expressive visual languages used by Deaf and Hard-of-Hearing (DHH) communities.

By Yiming Ni, Zhi-Qi Cheng, Jiayu Li, Wei Cheng

benchmarks safety

arXiv AI

Jun 19

PCBSchemaGen: Reward-Guided LLM Code Synthesis for Printed Circuit Boards (PCB) Schematic Design with Structured Verification

arXiv:2602. 00510v2 Announce Type: replace Abstract: Most LLM code-synthesis benchmarks rely on unit tests as the reward oracle, but PCB schematic design has none: correctness is defined by structured physical constraints over real IC packages and pin-level assignments, per-task golden references are unavailable, and SPICE simulation does not validate schematic-level correctness.

By Huanghaohe Zou, Peng Han, Emad Nazerian, Mafu Zhang, Zhicheng Guo, Alex Q. Huang

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jun 19

FreshRetailNet-50K: A Stockout-Annotated Censored Demand Dataset for Latent Demand Recovery and Forecasting in Fresh Retail

arXiv:2505. 16319v5 Announce Type: replace Abstract: Accurate demand estimation is critical for the retail business in guiding the inventory and pricing policies of perishable products.

By Yangyang Wang, Jiawei Gu, Li Long, Xin Li, Li Shen, Zhouyu Fu, Xiangjun Zhou, Xu Jiang

benchmarks safety

arXiv AI

Jun 19

Cross-Dataset, Age, and Gender Generalization: A Comprehensive Analysis of Fine-Tuning Strategies for Low-Resource Children's ASR

arXiv:2606. 19791v1 Announce Type: cross Abstract: The challenge associated with recognizing dysarthric speech primarily arises from pronounced acoustic variability attributed to impaired articulatory precision.

By Paban Sapkota, Hemant Kumar Kathania, Mikko Kurimo, Sudarsana Reddy Kadiri, Shrikanth Narayanan

fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 19

CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training

arXiv:2510. 18784v3 Announce Type: replace Abstract: Despite significant work on low-bit quantization-aware training (QAT), there is still an accuracy gap between such techniques and native training.

By Soroush Tabesh, Mher Safaryan, Andrei Panferov, Alexandra Volkova, Dan Alistarh

llms fine-tuning efficiency benchmarks

arXiv AI

Jun 19

LLM agent safety, multi-turn red-teaming, jailbreak benchmarks, adversarial robustness, safety-critical systems

arXiv:2606. 20408v1 Announce Type: cross Abstract: Large language model (LLM) agents are increasingly proposed as supervisory components for safety-critical systems, yet their robustness under sustained, adaptive adversarial pressure remains poorly characterized.

By Hanwool Lee, Dasol Choi, Bokyeong Kim, Seung Geun Kim, Haon Park

llms agents benchmarks safety

arXiv Machine Learning

Jun 19

Spectral DPPs via NEPv: A Scalable Continuous Relaxation of Determinantal MAP for Diversity-Aware Data Selection

arXiv:2606. 19411v1 Announce Type: new Abstract: Selecting a small, diverse, high-quality subset from a massive pool of candidates is a recurring primitive in modern machine learning -- data curation and coreset selection for training and fine-tuning large models, active-learning batch acquisition, prompt and exemplar selection for in-context learning, retrieval diversification, and experimental design.

By Richard Yi Da Xu

llms rag fine-tuning benchmarks

arXiv Machine Learning

Jun 19

Predicting gestational age at birth in the context of preterm birth from multi-modal fetal MRI

arXiv:2606. 20172v1 Announce Type: new Abstract: Preterm birth is associated with significant mortality and a risk for lifelong morbidity.

By Diego Fajardo-Rojas, Megan Hall, Daniel Cromb, Mary A. Rutherford, Lisa Story, Emma C. Robinson, Jana Hutter

multimodal benchmarks

arXiv AI

Jun 19

NRITYAM: Language Models Meet Art and Heritage of Dance

arXiv:2606. 19727v1 Announce Type: cross Abstract: Language models have become essential tools in shaping modern workflows.

By Punit Kumar Singh, Niladri Ghosh, Advait Joshi{\i}nst, Shailee Choudhary, Michael F\"arber, Haiqin Yang

llms multimodal benchmarks

arXiv AI

Jun 19

Learning to Prompt: Improving Student Engagement with Adaptive LLM-based High-School Tutoring

arXiv:2606. 20138v1 Announce Type: new Abstract: LLMs can personalize education, although current static-prompt tutoring systems struggle to adapt to diverse academic disciplines.

By Po-Chin Chang, Nicholas Hogan, Aske Plaat, Michiel T. van der Meer

llms robotics benchmarks

arXiv AI

Jun 19

How LLMs Fail and Generalize in RTL Coding for Hardware Design?

arXiv:2606. 19347v1 Announce Type: cross Abstract: Translating sequential programming priors into the parallel temporal logic of hardware design remains a crucial bottleneck for large language models(LLM).

By Guan-Ting Liu, Chao-Han Huck Yang, Chenhui Deng, Zhongzhi Yu, Brucek Khailany, Yu-Chiang Frank Wang

llms benchmarks safety

arXiv Machine Learning

Jun 19

PaAno+: Multiscale Encoding and Cross-Variable Attention for Time Series Anomaly Detection

arXiv:2606. 20055v1 Announce Type: new Abstract: Time-series anomaly detection has significant practical value for industrial and medical monitoring, as well as other critical domains.

By Youji Zhu, Hongbing Wang, Wenchao Liu, Xiaodong Liu, Xiangguang Xiong

llms rag benchmarks

arXiv AI

Jun 19

MEAL: A Benchmark for Continual Multi-Agent Reinforcement Learning

arXiv:2506. 14990v3 Announce Type: replace Abstract: Benchmarks play a central role in reinforcement learning (RL) research, yet their computational constraints often shape what is studied.

By Tristan Tomilin, Luka van den Boogaard, Samuel Garcin, Constantin Ruhdorfer, Bram Grooten, Fabrice Kusters, Yali Du, Andreas Bulling, Mykola Pechenizkiy, Meng Fang

agents reinforcement-learning benchmarks

arXiv AI

Jun 19

UniMM: A Unified Mixture Model Framework for Multi-Agent Simulation

arXiv:2501. 17015v2 Announce Type: replace Abstract: Simulation plays a crucial role in assessing autonomous driving systems, where the generation of realistic multi-agent behaviors is a key aspect.

By Longzhong Lin, Xuewu Lin, Kechun Xu, Haojian Lu, Lichao Huang, Rong Xiong, Yue Wang

agents robotics multimodal benchmarks safety

arXiv AI

Jun 19

Which Pairs to Compare for LLM Post-Training?

arXiv:2606. 19607v1 Announce Type: new Abstract: Preference-based post-training has become a central paradigm for aligning language models.

By Jiangze Han, Vineet Goyal, Will Ma

llms reinforcement-learning benchmarks

arXiv AI

Jun 19

The MAMA-MIA Challenge: Advancing Generalizability and Fairness in Breast MRI Tumor Segmentation and Treatment Response Prediction

arXiv:2603. 01250v2 Announce Type: replace-cross Abstract: Breast cancer is the most frequently diagnosed malignancy among women worldwide and a leading cause of cancer-related mortality.

By Lidia Garrucho, Smriti Joshi, Kaisar Kushibar, Richard Osuala, Maciej Bobowicz, Xavier Bargall\'o, Paulius Jaru\v{s}evi\v{c}ius, Kai Geissler, Raphael Sch\"afer, Muhammad Alberb, Tony Xu, Anne Martel, Daniel Sleiman, Navchetan Awasthi, Hadeel Awwad, Joan C. Vilanova, Robert Mart\'i, Daan Schouten, Jeong Hoon Lee, Mirabela Rusu, Eleonora Poeta, Luisa Vargas, Eliana Pastor, Maria A. Zuluaga, Jessica K\"achele, Dimitrios Bounias, Alexandra Ertl, Katarzyna Gwo\'zdziewicz, Maria-Laura Cosaka, Pasant M. Abo-Elhoda, Sara W. Tantawy, Shorouq S. Sakrana, Norhan O. Shawky-Abdelfatah, Amr Muhammad Abdo-Salem, Androniki Kozana, Eugen Divjak, Gordana Ivanac, Katerina Nikiforaki, Michail E. Klontzas, Rosa Garc\'ia-Dosd\'a, Meltem Gulsun-Akpinar, O\u{g}uz Lafc{\i}, Carlos Mart\'in-Isla, Oliver D\'iaz, Laura Igual, Karim Lekadir

computer-vision benchmarks safety

Get the daily briefing