Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,090 stories · RSS feed

arXiv Machine Learning

Jun 5

SHALA-LLM: Smartly Handling Ambiguous Labels in Aligning LLMs

arXiv:2606. 05376v1 Announce Type: new Abstract: Many human-centered tasks, including natural language inference (NLI) and emotion recognition (ER), have multiple plausible interpretations, leading to label ambiguity and challenging disagreements across human annotators.

By Jingyao Wu, Ashley Wang, Keane Ong, Paul Pu Liang, Rosalind Picard

llms reinforcement-learning benchmarks safety

arXiv Machine Learning

Jun 5

ERRORQUAKE: Heavy-Tailed Error Severity Distributions in Open-Weight Large Language Models

arXiv:2606. 05170v1 Announce Type: new Abstract: At matched accuracy, open-weight LLMs differ substantially in the shape of their error severity distribution -- a difference invisible to the scalar error rate.

By Jason Z Wang

llms benchmarks safety

arXiv Machine Learning

Jun 5

Scalable Reinforcement Learning via Adaptive Batch Scaling

arXiv:2605. 21557v2 Announce Type: replace-cross Abstract: Conventional wisdom holds that large-batch training is fundamentally incompatible with Reinforcement Learning (RL) - beyond a modest threshold, increasing batch sizes typically yields diminishing returns or performance degradation due to the inherent non-stationarity of the data distribution.

By Jongchan Park

reinforcement-learning benchmarks

arXiv Machine Learning

Jun 5

HEIST: A Graph Foundation Model for Spatial Transcriptomics and Proteomics Data

arXiv:2506. 11152v4 Announce Type: replace-cross Abstract: Single-cell transcriptomics and proteomics have become a great source for data-driven insights into biology, enabling the use of advanced deep learning methods to understand cellular heterogeneity and gene expression at the single-cell level.

By Hiren Madhu, Jo\~ao Felipe Rocha, Tinglin Huang, Siddharth Viswanath, Smita Krishnaswamy, Rex Ying

llms rag benchmarks

arXiv Machine Learning

Jun 5

CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction

arXiv:2606. 05799v1 Announce Type: new Abstract: Existing calibration methods for Large Language Models (LLMs) often overlook a critical dimension of trustworthiness: a model's {\em behavioral robustness} to irrelevant or misleading information.

By Mohammad Anas Jawad, Cornelia Caragea

llms benchmarks

arXiv Machine Learning

Jun 5

A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth

arXiv:2601. 21817v2 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on open-ended tasks without ground-truth labels is increasingly done via the LLM-as-a-judge paradigm.

By Mingyuan Xu, Xinzi Tan, Jiawei Wu, Doudou Zhou

llms benchmarks

arXiv Machine Learning

Jun 5

AlloGen: Conformation-Selective Binder Generation with Differential State Scoring

arXiv:2606. 05474v1 Announce Type: cross Abstract: Protein binder design has largely optimized for affinity alone, leaving conformational selectivity unaddressed: for allosteric targets such as kinases, nuclear receptors, and GPCRs, a binder that engages both active and inactive states provides no functional specificity regardless of how tightly it binds.

By Hanqun Cao, Zachary Quinn, Aastha Pal, Sumi Kimura, Jingjie Zhang, Pheng Ann Heng, Pranam Chatterjee

llms benchmarks

arXiv Machine Learning

Jun 5

LLMs for Secure Hardware Design and Related Problems: Opportunities and Challenges

arXiv:2605. 10807v4 Announce Type: replace-cross Abstract: The integration of Large Language Models (LLMs) into Electronic Design Automation (EDA) and hardware security is rapidly reshaping the semiconductor industry.

By Johann Knechtel, Ozgur Sinanoglu, Ramesh Karri

llms agents benchmarks safety

arXiv Machine Learning

Jun 5

DiffSlack: Learning under Nonlinear Inequality Constraints via Learnable Slack Variables

arXiv:2606. 05247v1 Announce Type: new Abstract: Enforcing nonlinear inequality constraints in neural networks remains challenging, especially when the output is subject to many coupled constraints.

By Ziqian Wang, Chenxi Fang, Zhen Zhang

rag benchmarks

arXiv Machine Learning

Jun 5

TS-ICL: A Flexible Time-Indexed Foundation Model for Time Series via In-Context Learning

arXiv:2606. 05878v1 Announce Type: new Abstract: Foundation models mark a profound paradigm shift in time series modeling, with task-specific models being superseded by general-purpose zero-shot models.

By Etienne Le Naour, Tahar Nabil, Adrien Petralia

llms benchmarks

arXiv Machine Learning

Jun 5

CuTeGen: An LLM-Based Agentic Framework for Generation and Optimization of High-Performance GPU Kernels using CuTe

arXiv:2604. 01489v2 Announce Type: replace Abstract: High-performance GPU kernels are critical to modern machine learning systems, yet developing them remains a manual, expert-driven process.

By Tara Saba, Zhiyang Chen, Jikai Jason Li, Anne Ouyang, Xujie Si, Fan Long

llms agents benchmarks

arXiv Machine Learning

Jun 5

Multi-Fidelity Learning with Shallow Recurrent Decoders for Reactor Physics

arXiv:2606. 05202v1 Announce Type: cross Abstract: In reactor physics, neutronics can be treated with different fidelity levels, according to the needs of the user.

By Stefano Riva, Carolina Introini, J. Nathan Kutz, Antonio Cammi

diffusion benchmarks

arXiv Machine Learning

Jun 5

From Prediction to Self: Developmental Conditions for Agency in Minimal Neural Systems

arXiv:2606. 05605v1 Announce Type: new Abstract: How does a system that merely predicts the world come to distinguish its own causal influence from everything else?

By Evan Ye

benchmarks

arXiv Machine Learning

Jun 5

Statistically Reliable LLM-Based Ranking Evaluation via Prediction-Powered Inference

arXiv:2606. 05308v1 Announce Type: new Abstract: With PRECISE, we extended Prediction-Powered Inference to produce bias-corrected estimates of ranking evaluation metrics by combining a small human-labeled set with a large LLM-judged set.

By Abhishek Divekar

llms benchmarks safety

arXiv Machine Learning

Jun 5

On the training of physics-informed neural operators for solving parametric partial differential equations

arXiv:2606. 06164v1 Announce Type: new Abstract: Physics-informed neural operators (PINOs) aim to learn solution operators for partial differential equations by using the governing physics as supervision, rather than relying solely on paired input-output simulation data.

By Nanxi Chen, Chuanjie Cui, Airong Chen, Sifan Wang, Rujin Ma

llms computer-vision benchmarks

arXiv Machine Learning

Jun 5

Cluster-Aware Causal Mixer for Online Anomaly Detection in Multivariate Time Series

arXiv:2506. 00188v2 Announce Type: replace Abstract: Early and accurate detection of anomalies in time-series data is critical due to the substantial risks associated with false or missed detections.

By Md Mahmuddun Nabi Murad, Yasin Yilmaz

rag benchmarks

arXiv Machine Learning

Jun 5

Compress-Distill: Reasoning Trace Compression for Efficient Knowledge Distillation

arXiv:2606. 05988v1 Announce Type: new Abstract: Reasoning models produce long chain-of-thought traces that are costly to distill and encourage verbose student outputs.

By Maxime Griot, Paul Steven Scotti, Tanishq Mathew Abraham

llms fine-tuning efficiency benchmarks

arXiv Machine Learning

Jun 5

Q-GNN: Query-Conditioned Graph Neural Networks with Type Awareness for Knowledge Graph Completion

arXiv:2606. 05639v1 Announce Type: new Abstract: Knowledge Graph Completion (KGC) aims at predicting missing triplets from incomplete knowledge graphs, which is crucial for downstream applications.

By Dongxiao He, Ruqiong Zhang, Zhizhi Yu, Ling Ding, Di Jin, Guangquan Xu, Zhiyong Feng

llms benchmarks

arXiv Machine Learning

Jun 5

GIPO: Gaussian Importance Sampling Policy Optimization

arXiv:2603. 03955v2 Announce Type: replace Abstract: Post-training with reinforcement learning (RL) has recently shown strong promise for advancing multimodal agents beyond supervised imitation.

By Chengxuan Lu, Zhenquan Zhang, Shukuan Wang, Qunzhi Lin, Yanjie Li, Baigui Sun, Yang Liu

agents reinforcement-learning multimodal benchmarks safety

arXiv Machine Learning

Jun 5

TLA-Prover: Verifiable TLA+ Specification Synthesis via Preference-Optimized Low-Rank Adaptation

arXiv:2606. 06133v1 Announce Type: cross Abstract: TLA+ is a formal specification language for verifying distributed systems and safety-critical protocols.

By Eric Spencer, Arslan Bisharat, Brian Ortiz, Khushboo Bhadauria, TaiNing Wang, George K. Thiruvathukal, Konstantin Laufer, Mohammed Abuhamad

llms reinforcement-learning fine-tuning benchmarks safety

Get the daily briefing