Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

22,912 stories · RSS feed

arXiv AI

Jun 9

Where Instruction Hierarchy Breaks: Diagnosing and Repairing Failures in Reasoning Language Models

arXiv:2606. 07808v1 Announce Type: new Abstract: Reasoning language models deployed in agentic workflows must follow an instruction hierarchy: when instructions from different sources conflict, the model should obey the highest-privilege applicable instruction.

By Sanjay Kariyappa, G. Edward Suh

llms agents benchmarks

arXiv Machine Learning

Jun 9

Querying Counterfactuals on Tissue Graphs with Supervised Disentanglement

arXiv:2606. 08493v1 Announce Type: cross Abstract: \textit{Tissue graph counterfactuals} ask how a cell's expression would change under altered spatial neighbor contexts.

By Abdul Moeed, Stefan Schrod, Martin Rohbeck, Marc Jan Bonder, Pavlo Lutsik, Oliver Stegle, Daniel Dimitrov

benchmarks

arXiv AI

Jun 9

When Does Delegation Beat Majority? A Delegation-Based Aggregator for Multi-Sample LLM Inference

arXiv:2606. 08098v1 Announce Type: new Abstract: Majority voting over sampled answers is the dominant unsupervised aggregator for multi-sample LLM inference.

By Yasushi Sakai, Allen Song, Kent Larson

llms rag benchmarks

arXiv Machine Learning

Jun 9

Beyond Convolution: Advancing Hypergraph Neural Networks with Hypergraph U-Nets

arXiv:2606. 09051v1 Announce Type: new Abstract: Convolutions have successfully transitioned from image processing to the complex realm of non-Euclidean higher-order domains, particularly in hypergraphs.

By Fuli Wang, Wei Qian, Daniel L. Lau, Gonzalo R. Arce

benchmarks

arXiv Machine Learning

Jun 9

Operator learning for the 2D incompressible Navier-Stokes equations: a conformal prediction approach in the data-scarce regime

arXiv:2606. 08654v1 Announce Type: new Abstract: In this paper, we propose a perturbation-based conformal prediction framework for uncertainty quantification in operator learning, with a focus on the 2D Navier--Stokes equations.

By Weinan Wang, Bowen Gang, Hao Deng

benchmarks

arXiv AI

Jun 9

Shared Latent Structures Enable Unified Backdoor Detection and Mitigation in LLMs

arXiv:2606. 07963v1 Announce Type: new Abstract: Backdoor attacks in large language models (LLMs) are often treated as isolated trigger-response failures, motivating defenses tailored to specific triggers or behaviors.

By Omar Mahmoud, Aly M. Kassem, Thommen George Karimpanal, Buddhika Laknath Semage, Negar Rostamzadeh, Golnoosh Farnadi, Santu Rana

llms robotics fine-tuning benchmarks safety

arXiv Machine Learning

Jun 9

Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families

arXiv:2606. 09456v1 Announce Type: new Abstract: On-Policy Distillation (OPD) has become a core technique in the post-training of Large Language Models (LLMs) for transferring knowledge from domain experts to student models.

By Yifan Niu, Han Xiao, Dongyi Liu, Zelong Wang, Dihong Gong, Yasheng Wang, Jia Li

llms nlp fine-tuning efficiency benchmarks

arXiv Machine Learning

Jun 9

Causal Semantic Alignment for LLM-based Time Series Forecasting

arXiv:2606. 08262v1 Announce Type: new Abstract: Recent advances in Large Language Models (LLMs) have opened new possibilities for time series forecasting by enabling alignment between temporal patterns and pretrained word embeddings.

By Kexuan Zhang, Xiaobei Zou, Cesare Alippi, Gary G. Yen, Yang Tang

llms rag benchmarks safety

arXiv Machine Learning

Jun 9

Model-Based Learning of Whittle indices

arXiv:2511. 20397v2 Announce Type: replace Abstract: We present BLINQ, a new model-based algorithm that learns the Whittle indices of an indexable, communicating and unichain Markov Decision Process (MDP).

By Jo\"el Charles-Rebuff\'e, Nicolas Gast, Bruno Gaujal

reinforcement-learning benchmarks

arXiv Machine Learning

Jun 9

MOLOT System Card: Malicious Operational Logic Observation Transformer

arXiv:2606. 07792v1 Announce Type: cross Abstract: MOLOT (Malicious Operational Logic Observation Transformer) is a static malicious-code detection system designed for SAST setup where package metadata, maintainer history, and dynamic execution traces may be unavailable or unreliable.

By Daniil Lopatkin, Maksim Mitrofanov, Stanislav Rakovsky, Aleksandr Khalikov

llms benchmarks

arXiv AI

Jun 9

Discovering heuristics in a complex SAT solver with large language models

arXiv:2507. 22876v2 Announce Type: replace Abstract: The Satisfiability problem (SAT) is fundamental in computational complexity theory and has a wide range of industrial applications.

By Yiwen Sun, Furong Ye, Zhihan Chen, Ke Wei, Shaowei Cai

llms benchmarks

arXiv AI

Jun 9

FuseFSS: Efficient Secure LLM Inference with Function Secret Sharing

arXiv:2606. 09551v1 Announce Type: cross Abstract: Two-server secure inference allows a client to query a hosted large language model (LLM) without revealing prompts or embeddings.

By Yuhan Ma, Yong Li, Stefan Schmid

llms rag benchmarks

arXiv AI

Jun 9

A Dataset for Dynamic Human Preferences for Vision Language Models

arXiv:2606. 07653v1 Announce Type: cross Abstract: Given the increased adoption of Vision Language Models (VLMs) in human-interactive settings, it is important that we evaluate how well these models can adapt to real-time preferences for different users.

By Hannah Gao (Massachusetts Institute of Technology), Dylan Hadfield-Menell (Massachusetts Institute of Technology), Rachel Ma (Massachusetts Institute of Technology)

llms multimodal benchmarks

arXiv Machine Learning

Jun 9

SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Sampling

arXiv:2606. 09304v1 Announce Type: cross Abstract: On-policy distillation (OPD) trains a student on its own trajectories with dense per-token supervision from a stronger teacher, and often outperforms off-policy distillation and standard reinforcement learning.

By Haoran Xu, Hongyu Wang, Yifei Gao, Jiaze Li, Xiaofeng Zhang, Xiaosong Yuan

reinforcement-learning efficiency benchmarks safety

arXiv Machine Learning

Jun 9

Deep reinforcement learning for process design: Review and perspective

arXiv:2308. 07822v2 Announce Type: replace Abstract: The transformation towards renewable energy and feedstock supply in the chemical industry requires new conceptual process design approaches.

By Qinghe Gao, Artur M. Schweidtmann

agents reinforcement-learning benchmarks

arXiv AI

Jun 9

Structured Neuron Pruning in Deep Neural Networks Using Multi-Armed Bandits

arXiv:2606. 07615v1 Announce Type: cross Abstract: Deep neural networks often contain redundant hidden units.

By Salem Ameen, Sunil Vadera

reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Jun 9

Improving the sharpness in neural network-based parametric post-processing of ensemble forecasts

arXiv:2606. 08587v1 Announce Type: cross Abstract: Statistical post-processing has proven to be an effective tool in improving ensemble forecast of different weather variables.

By \'Agnes Baran, M\'at\'e Mihalina

benchmarks

arXiv Machine Learning

Jun 9

Differentiable Weightless Controllers: Learning Logic Circuits for Continuous Control

arXiv:2512. 01467v2 Announce Type: replace Abstract: Controlling autonomous systems under real-world conditions often requires policies that can be evaluated with low latency and minimal energy consumption.

By Fabian Kresse, Christoph H. Lampert

agents efficiency benchmarks

arXiv Machine Learning

Jun 9

Bayesian Optimization of a Multi-Product Chemical Reactor Using Composite Models and Partial Physics Knowledge

arXiv:2606. 08611v1 Announce Type: cross Abstract: We study data-driven real-time economic optimization of a multi-product chemical reactor when no reliable first-principles model is available beyond a steady-state energy balance.

By Liqiu Dong, Marta Zag\'orowska, Mehmet Mercang\"oz

benchmarks

arXiv AI

Jun 9

TempoBench: Evaluating Temporal Causal Reasoning in Large Language Models

arXiv:2510. 27544v2 Announce Type: replace Abstract: Temporal reasoning involves understanding how systems evolve over time through input-driven state transitions.

By Nikolaus Holzer, William Fishell, Baishakhi Ray, Mark Santolucito

llms fine-tuning benchmarks

Get the daily briefing