Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

22,245 stories · RSS feed

arXiv AI

Jun 10

Using Probabilistic Programs to Train Inductive Reasoning in Large Language Models

arXiv:2606. 09856v1 Announce Type: cross Abstract: Post-training Large Language Models (LLMs) for reasoning typically focuses on deductive tasks such as mathematics and coding where correctness is verifiable.

By Liyi Zhang, Akshay K. Jagadish, Brenden M. Lake, Thomas L. Griffiths

llms agents fine-tuning benchmarks safety

arXiv AI

Jun 10

LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake

arXiv:2606. 10460v1 Announce Type: cross Abstract: Recent large language models (LLMs) have shown rapid progress in reading-based question answering (QA), where evidence is explicitly provided or can be trivially retrieved.

By Haonan Wang, Jiaxiang Liu, Yurong Liu, Austin Senna Wijaya, Tianle Zhou, Eden Wu, Yijia Chen, Wanting You, Reya Vir, Daniela Pinto, Grace Fan, Yusen Zhang, Juliana Freire, Eugene Wu

llms agents nlp benchmarks

arXiv AI

Jun 10

Mind the Gap: Can Frontier LLMs Pass a Standardized Office Proficiency Exam?

arXiv:2606. 10956v1 Announce Type: new Abstract: The deployment of Large Language Model (LLM) agents for computer automation is accelerating, yet their ability to navigate complex, professional-grade productivity software is largely untested.

By Tengchao Lv, Dongdong Zhang, Jiayu Ding, Yilin Jia, Yuzhong Zhao, Yupan Huang, Wenshan Wu, Xiangyang Zhou, Shaohan Huang, Nan Yang, Li Dong, Lei Cui, Furu Wei

llms agents benchmarks

arXiv AI

Jun 10

Machine Learning Methods for Studying Latent Neural Activity Dynamics

arXiv:2606. 10530v1 Announce Type: cross Abstract: Recent developments in brain recording are driving a demand for machine learning tools capable of decoding the latent structure of large populations of neurons.

By Shufeng Kong, Fumei Deng, Xinyi Dong, Caihua Liu, Weiwei Chen, Yingheng Wang, Daniel Cao, Azahara Oliva, Antonio Fernandez-Ruiz, Carla Gomes

llms diffusion benchmarks

arXiv AI

Jun 10

PhantomBench: Benchmarking the Non-existential Threat of Language Models

arXiv:2606. 11105v1 Announce Type: cross Abstract: Hallucinations, where language models (LMs) generate factually ungrounded responses, pose serious risks, as users tend to blindly rely on them.

By Haeji Jung, Hila Gonen

llms benchmarks safety

arXiv AI

Jun 10

Temporal Sheaf Neural Networks with Dynamic Orthogonal Transport

arXiv:2606. 10071v1 Announce Type: cross Abstract: We introduce Temporal Sheaf Neural Networks (TSNN), a temporal link prediction framework that equips each node with a time-varying orthogonal frame and compares node states only after explicit transport between local coordinate systems.

By Md Sadek Hossain Asif, Tanzila Khan, Md. Mosaddek Khan

rag diffusion benchmarks

arXiv Machine Learning

Jun 10

Dexterous Point Policy: Learning Point-based Dexterous Hand Policies from Human Demonstrations

arXiv:2606. 10614v1 Announce Type: cross Abstract: Robotic foundation models pre-trained on human demonstration videos have shown promise, but a significant embodiment gap remains when the resulting policies are deployed on real robots.

By Beomjun Kim, Seong Hyeon Park, Seunghoon Sim, Seungjun Moon, Sanghyeok Lee, Jinwoo Shin

llms agents robotics fine-tuning benchmarks

arXiv Machine Learning

Jun 10

Multi-task LLMs for Bug Classification: Efficient Inference with Auxiliary Decoding Heads

arXiv:2606. 09956v1 Announce Type: cross Abstract: The rapid adoption of LLM-powered code generation has dramatically accelerated software development, yet effective verification methods remain severely underdeveloped.

By Nikolai Rozanov

llms agents nlp efficiency benchmarks safety

arXiv Machine Learning

Jun 10

What Demonstration Curation Metrics Do to Your Policy

arXiv:2606. 10229v1 Announce Type: cross Abstract: We study whether demonstration-curation metrics that detect defective training episodes also improve the downstream behavior-cloning policy that trains on the curated data.

By Aarav Bedi

benchmarks

arXiv AI

Jun 10

Dep-LLM: Training-Free Depression Diagnosis via Evidence-Guided Structured Multi-factor with Reliable LLM Reasoning

arXiv:2606. 10796v1 Announce Type: cross Abstract: Automatic Depression Detection (ADD) from clinical interviews is a pivotal task in computational mental health, yet it remains challenging due to two critical obstacles: 1) difficulty in modeling complex but sparsely distributed depression clues within lengthy, multi-topic clinical interviews, leading to superficial and unreliable reasoning; 2) scarcity of labeled data due to clinical privacy, together with high cost of training and fine-tuning, limiting the deployment of supervised ADD systems.

By Yiqing Lyu, Xianbing Zhao, Buzhou Tang, Ronghuan Jiang

llms fine-tuning benchmarks safety

arXiv AI

Jun 10

ERAlign: Energy-based Representation Alignment of GNNs and LLMs on Text-attributed Graphs

arXiv:2606. 10461v1 Announce Type: cross Abstract: Text-attributed Graphs (TAGs) incorporate textual node attributes with graph structures to describe rich relational semantics.

By Xianlin Zeng, Fan Xia, Xiangyu Chen

llms rag diffusion benchmarks safety

arXiv AI

Jun 10

Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation

arXiv:2606. 10749v1 Announce Type: cross Abstract: Large language model (LLM) agents are rapidly moving from conversational interfaces to software components that plan, invoke tools, maintain memory, and act on external environments.

By Yuchen Ling, Shengcheng Yu, Zhenyu Chen, Chunrong Fang

llms agents benchmarks

arXiv Machine Learning

Jun 10

Magnetic HIP-NN for spin dynamics in disordered itinerant magnets

arXiv:2606. 10349v1 Announce Type: cross Abstract: We present a magnetic extension of the Hierarchically Interacting Particle Neural Network (HIP-NN) that enables large-scale simulations of electron-mediated spin dynamics in disordered itinerant magnets.

By Supriyo Ghosh, Yunhao Fan, Sheng Zhang, Kipton Barros, Gia-Wei Chern

benchmarks

arXiv AI

Jun 10

What Fits (Into Few Tokens) Doesn't Overfit: Compression and Generalization in ML Research Agents

arXiv:2606. 11045v1 Announce Type: new Abstract: Reusing a held-out benchmark adaptively should, in principle, invite overfitting.

By Martin Andres Bertran, Aaron Roth, Zhiwei Steven Wu

llms agents diffusion reinforcement-learning benchmarks

arXiv AI

Jun 10

NOVA: Symbolic Regression Discovery of Interpretable Car-Following and Lane-Change Models with Driver Heterogeneity

arXiv:2606. 10583v1 Announce Type: cross Abstract: We present NOVA, an autonomous symbolic regression framework that identifies interpretable car-following and lane-change structures from raw trajectory data with minimal behavioral priors.

By Ishak Abassi, Nassim Ali Bouazzouni, Farah Ibelaiden, Nadir Farhi

llms agents benchmarks

arXiv Machine Learning

Jun 10

Efficient AI-Inspired Reduction of Feynman Integrals via Tube Seeding

arXiv:2606. 10698v1 Announce Type: cross Abstract: In this paper, we use machine learning to discover a new seeding strategy for integration-by-parts reduction of Feynman integrals, which is a frequent bottleneck in state-of-the-art calculations in theoretical particle and gravitational-wave physics.

By Justin Berman, Francois Charton, Andres Luna, Matthias Wilhelm, Mao Zeng

efficiency benchmarks

arXiv Machine Learning

Jun 10

Effective Training Principles of Physical Reservoirs

arXiv:2606. 10130v1 Announce Type: cross Abstract: Reservoir computers benefit from the inherent complexity of optical phenomena, which provide rich, often nonlinear dynamics.

By Sobhi Saeed, Mehmet M\"uft\"uoglu, Glitta R. Cheeran, Juliane Heim, Bennet Fischer, Mario Chemnitz

diffusion efficiency benchmarks

arXiv AI

Jun 10

A Constrained Natural-Language Interface for Variational Multi-Physics Finite Element Simulations in FEniCS

arXiv:2606. 10928v1 Announce Type: cross Abstract: Large language models can reduce the manual effort required to set up finite element simulations, but they introduce reliability risks when generated solver code lies on the critical path.

By Nilay Upadhyay, Wesley F. Reinhart

llms agents benchmarks

arXiv Machine Learning

Jun 10

mlr3mbo: Bayesian Optimization in R

arXiv:2603. 29730v2 Announce Type: replace-cross Abstract: We present mlr3mbo, a modular toolbox for Bayesian optimization in R.

By Marc Becker, Lennart Schneider, Martin Binder, Lars Kotthoff, Bernd Bischl

benchmarks

arXiv AI

Jun 10

CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs

arXiv:2606. 11063v1 Announce Type: new Abstract: AI control protocols oversee untrusted models by monitoring their actions and modifying potentially unsafe steps, often using a trusted model.

By Joachim Schaeffer, Thomas Jiralerspong, Alexander Panfilov, Guillaume Lajoie, Jonas Geiping, Yoshua Bengio, Roland S. Zimmermann

llms benchmarks

Get the daily briefing