Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

23,305 stories · RSS feed

arXiv AI

Jun 2

When Softmax Fails at the Top: Extreme Value Corrections for InfoNCE

arXiv:2606. 00262v1 Announce Type: cross Abstract: InfoNCE is the standard contrastive learning objective, but its softmax form is not only a computational convenience: it also encodes a statistical assumption about how the top-scoring example is selected.

By Melihcan Erol, Suat Evren, Oktay Ozel, Alexander Morgan, Jongha Jon Ryu, Lizhong Zheng

rag benchmarks

arXiv AI

Jun 2

Experiments in Agentic AI for Science

arXiv:2605. 26305v2 Announce Type: replace Abstract: This paper details two novel frameworks for developing autonomous, agentic AI in scientific workflows.

By Judy Fox, Geoffrey Fox

llms rag agents benchmarks

arXiv AI

Jun 2

Consistent and Distinctive: LLM Benchmark Efficiency via Maximum Independent Set Prompt Selection on Similarity Graphs

arXiv:2606. 01400v1 Announce Type: cross Abstract: Evaluating large language models (LLMs) across comprehensive benchmarks is expensive and time-consuming.

By Denica Kjorvezir, Marko Djukanovi\'c, Ana Gjorgjevikj, Gjorgjina Cenikj, Tome Eftimov

llms rag benchmarks

arXiv AI

Jun 2

LLM-Evolved Pattern Generators for Optimal Classical Planning

arXiv:2606. 02438v1 Announce Type: new Abstract: Learned heuristics have recently become a competitive alternative to traditional domain-independent heuristics for satisficing planning.

By Windy Phung, Dominik Drexler, Arnaud Lequen, Jendrik Seipp

llms benchmarks

arXiv AI

Jun 2

Detection vs. Execution: Single-Bucket Probes Miss Half the Mamba-2 State Sink

arXiv:2606. 00930v1 Announce Type: cross Abstract: Mechanistic interpretability often assumes that probes identifying a representational signature also identify the circuit executing the corresponding computation.

By Yuhang Jiang

benchmarks safety

arXiv AI

Jun 2

Asking Is Not Enough: Protocol Sensitivity in LLM Confidence Calibration

arXiv:2605. 27752v2 Announce Type: replace Abstract: LLM confidence calibration is often evaluated by comparing two signals: token-probability scores and verbalized confidence.

By Hankyeol Kim, Pilsung Kang

llms benchmarks

arXiv AI

Jun 2

LLM-Guided Communication for Cooperative Multi-Agent Reinforcement Learning

arXiv:2605. 18077v2 Announce Type: replace Abstract: Communication is a key component in multi-agent reinforcement learning (MARL) for mitigating partial observability, yet prior approaches often rely on inefficient information exchange or fail to transmit sufficient state information.

By Sangjun Bae, Yisak Park, Sanghyeon Lee, Seungyul Han

llms agents reinforcement-learning benchmarks

arXiv AI

Jun 2

Cooperation of Experts: Fusing Heterogeneous Information with Large Margin

arXiv:2505. 20853v3 Announce Type: replace-cross Abstract: Fusing heterogeneous information remains a persistent challenge in modern data analysis.

By Shuo Wang, Shunyang Huang, Jinghui Yuan, Zhixiang Shen, Zhao Kang

benchmarks

arXiv AI

Jun 2

Prototype Transformer: Towards Language Model Architectures Interpretable by Design

arXiv:2602. 11852v2 Announce Type: replace Abstract: While state-of-the-art language models (LMs) surpass most humans in certain domains, their reasoning remains largely opaque, reducing trust and increasing the risk of deception and hallucination.

By Yordan Yordanov, Matteo Forasassi, Bayar Menzat, Ruizhi Wang, Chang Qi, Markus Kaltenberger, Amine M'Charrak, Tommaso Salvatori, Thomas Lukasiewicz

llms benchmarks safety

arXiv AI

Jun 2

From Graph Retrieval to Schema Realization: Counterfactual Validation for Text-to-SPARQL over Heterogeneous Knowledge Graphs

arXiv:2508. 01815v2 Announce Type: replace-cross Abstract: Text-to-SPARQL maps natural-language questions to executable SPARQL queries over RDF knowledge graphs.

By Yang Zhao, Chengxiao Dai, Yue Xiu, Dusit Niyato

agents nlp benchmarks safety

arXiv Machine Learning

Jun 2

d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation

arXiv:2509. 21474v4 Announce Type: replace Abstract: While diffusion language models (DLMs) have achieved competitive performance in text generation, improving their reasoning ability with reinforcement learning remains an active research area.

By Guanghan Wang, Gilad Turok, Yair Schiff, Marianne Arriola, Volodymyr Kuleshov

llms diffusion reinforcement-learning benchmarks

arXiv AI

Jun 2

FreqLite: A Lightweight Frequency-Decomposed Linear Model with Adaptive Reversible Normalization for Robust Long-Term Time-Series Forecasting

arXiv:2606. 01339v1 Announce Type: cross Abstract: Long-term time-series forecasting needs models that are accurate yet efficient enough for commodity hardware.

By Mirza Samad Ahmed Baiga, Syeda Anshrah Gillani

llms benchmarks

arXiv Machine Learning

Jun 2

Beyond Additive Decompositions: Interpretability Through Separability

arXiv:2605. 31200v2 Announce Type: replace Abstract: Interpretable machine learning requires models that are accurate and structurally faithful to the data.

By Jinyang Liu, Munir Eberhardt Hiabu

benchmarks safety

arXiv Machine Learning

Jun 2

CAREF: Calibration-Aware Regularization for Explanation Faithfulness Without Rationale Supervision

arXiv:2605. 27835v2 Announce Type: replace Abstract: We introduce CAREF, a parameter-efficient fine-tuning framework that jointly optimizes predictive accuracy and explanation faithfulness via calibration-aware regularization.

By Naphat Nithisopa, Teerapong Panboonyuen

llms fine-tuning efficiency benchmarks safety

arXiv AI

Jun 2

ChronosAD: Leveraging Time Series Foundation Models for Accurate Anomaly Detection

arXiv:2606. 01300v1 Announce Type: cross Abstract: Time series anomaly detection is a crucial task in various domains, including finance, healthcare, and industry.

By Uzair Khan, Luigi Capogrosso, Francesco Biondani, Michele Magno, Franco Fummi, Francesco Setti, Marco Cristani

rag benchmarks

arXiv Machine Learning

Jun 2

GRPO-TTA: Test-Time Visual Tuning for Vision-Language Models via GRPO-Driven Reinforcement Learning

arXiv:2605. 03403v2 Announce Type: replace-cross Abstract: Group Relative Policy Optimization (GRPO) has recently shown strong performance in post-training large language models and vision-language models.

By Yujun Li, Hongyuan Zhang, Yuan Yuan

llms reinforcement-learning multimodal benchmarks safety

arXiv AI

Jun 2

Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying

arXiv:2606. 00151v1 Announce Type: cross Abstract: In reinforcement learning (RL), agents benefit from exploration only because they repeatedly encounter similar states: trying different actions can improve performance or reduce uncertainty; without such retries, a greedy policy is optimal.

By Soichiro Nishimori, Paavo Parmas, Sotetsu Koyamada, Tadashi Kozuno, Toshinori Kitamura, Shin Ishii, Yutaka Matsuo

agents reinforcement-learning benchmarks

arXiv AI

Jun 2

AI-PROPELLER: Warehouse-Scale Interprocedural Code Layout Optimization with AlphaEvolve

arXiv:2606. 00131v1 Announce Type: cross Abstract: Post-link optimizers (PLOs) such as Propeller and BOLT have demonstrated that precise, profile-guided code layout can extract significant performance gains from heavily optimized binaries.

By Chaitanya Mamatha Ananda, Rajiv Gupta, Mircea Trofin, Aiden Grossman, Sriraman Tallam, Xinliang David Li, Amir Yazdanbakhsh

agents benchmarks

arXiv AI

Jun 2

Zero-Shot Off-Policy Learning

arXiv:2602. 01962v2 Announce Type: replace-cross Abstract: Off-policy learning methods seek to derive an optimal policy directly from a fixed dataset of prior interactions.

By Arip Asadulaev, Maksim Bobrin, Salem Lahlou, Dmitry Dylov, Fakhri Karray, Martin Takac

agents reinforcement-learning benchmarks safety

arXiv AI

Jun 2

Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performance

arXiv:2304. 11127v5 Announce Type: replace-cross Abstract: Recent scientific advances require complex experiment design, necessitating the meticulous tuning of many experiment parameters.

By Shuhei Watanabe

benchmarks

Get the daily briefing