Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

22,530 stories · RSS feed

arXiv AI

Jun 9

Ablation-Reversible Heads Don't Transfer: A Stress Test for Mechanistic Role Claims in Transformers

arXiv:2606. 08292v1 Announce Type: new Abstract: In mechanistic interpretability, attention heads are commonly elevated to role claims (e.

By Philip Quirke

llms benchmarks safety

arXiv AI

Jun 9

LFNO: Bridging Laplace and Fourier via Transient-Steady Decomposition

arXiv:2606. 07601v1 Announce Type: cross Abstract: We introduce the Laplace-Fourier Neural Operator (LFNO), a unified framework for modeling dynamical systems across transient and steady-state regimes by integrating the spectral advantages of Laplace and Fourier Neural Operators.

By Jeongun Ha, Sanga Yoon, Donghun Lee

diffusion benchmarks safety

arXiv AI

Jun 9

How Small Can You Go? LoRA Fine-Tuning 270M-8B Models for Merchant Information Extraction in Financial Transactions

arXiv:2606. 08051v1 Announce Type: new Abstract: Financial transaction processing requires extracting structured merchant information from noisy, abbreviated bank transaction strings at scale.

By Donghao Huang, Tomas Drietomsky, Benjamin Barrett, Zhaoxia Wang

llms fine-tuning efficiency benchmarks

arXiv Machine Learning

Jun 9

SoK: Reconstruction Attacks on Synthetic Tabular Data (Insights from Winning the NIST CRC)

arXiv:2606. 08372v1 Announce Type: cross Abstract: Synthetic data is increasingly promoted as a privacy-preserving substitute for releasing sensitive tabular records, yet its central adversarial threat ("reconstruction", the recovery of an individual's hidden attribute values from a synthetic release and a handful of known quasi-identifiers) has been studied only in scattered, hard-to-compare settings.

By Steven Golob, Sikha Pentyala, Martine De Cock

benchmarks safety

arXiv AI

Jun 9

ACTIVE-o3: Empowering MLLMs with Active Perception via Pure Reinforcement Learning

arXiv:2505. 21457v2 Announce Type: replace-cross Abstract: Active vision, also known as active perception, refers to actively selecting where and how to look in order to gather task-relevant information.

By Muzhi Zhu, Hao Zhong, Canyu Zhao, Zongze Du, Mingyu Liu, Zheng Huang, Anzhou Li, Hao Chen, Cheng Zou, Jingdong Chen, Ming Yang, Chunhua Shen

llms agents computer-vision reinforcement-learning robotics multimodal benchmarks

arXiv Machine Learning

Jun 9

Rank Intervals for Leaderboards: A Hierarchical Framework for Model Evaluation

arXiv:2606. 08679v1 Announce Type: cross Abstract: Pretrained models are often evaluated on multi-task leaderboards to measure their applicability in diverse contexts.

By Bitya Neuhof, Yuval Benjamini

benchmarks

arXiv Machine Learning

Jun 9

Multi-View Speech Representation Learning for Parkinson's Disease Detection Using Context-guided Cross-modal Attention

arXiv:2606. 09271v1 Announce Type: cross Abstract: Parkinson's disease (PD) is a progressive neurodegenerative disorder that frequently causes speech impairments associated with hypokinetic dysarthria.

By George Theodosiou, Loukas Ilias, Dimitris Askounis

rag multimodal benchmarks

arXiv AI

Jun 9

Voting Protocols as Coordination Mechanisms for Role-Constrained Multi-Agent Tutoring Systems

arXiv:2606. 08030v1 Announce Type: cross Abstract: Agentic tutoring systems introduce a coordination challenge: multiple agents may propose different but reasonable interventions, yet only one response can be delivered to the learner.

By Eric S. Qiu, Joyce Gill

agents benchmarks

arXiv Machine Learning

Jun 9

Model-Based Learning of Whittle indices

arXiv:2511. 20397v2 Announce Type: replace Abstract: We present BLINQ, a new model-based algorithm that learns the Whittle indices of an indexable, communicating and unichain Markov Decision Process (MDP).

By Jo\"el Charles-Rebuff\'e, Nicolas Gast, Bruno Gaujal

reinforcement-learning benchmarks

arXiv AI

Jun 9

BLM-SGAN: Bidirectional Language Modeling for Semantic-Spatial Text-to-Image Generation

arXiv:2606. 08847v1 Announce Type: cross Abstract: Despite the success of image generation from text descriptions, it still faces challenges that are difficult to overcome in domains such as natural language processing (NLP) and computer vision (CV).

By Ahmed Abdelmoneim Mazrou, Haidy Maher El-Amir, Ali Hamdi

diffusion computer-vision nlp benchmarks safety

arXiv AI

Jun 9

Unifying Object-Centric World Models and Diffusion Policy: A Hierarchical Framework for Multi-Stage Robotic Tasks

arXiv:2606. 08775v1 Announce Type: cross Abstract: Visual world models have shown great potential in learning complex system dynamics.

By Raktim Gautam Goswami, Prashanth Krishnamurthy, Yann LeCun, Farshad Khorrami

diffusion robotics benchmarks

arXiv Machine Learning

Jun 9

Nonparametric LLM Evaluation from Preference Data

arXiv:2601. 21816v2 Announce Type: replace Abstract: Evaluating the performance of large language models (LLMs) from human preference data is crucial for obtaining LLM leaderboards.

By Dennis Frauen, Athiya Deviyani, Mihaela van der Schaar, Stefan Feuerriegel

llms benchmarks

arXiv AI

Jun 9

A Dataset for Dynamic Human Preferences for Vision Language Models

arXiv:2606. 07653v1 Announce Type: cross Abstract: Given the increased adoption of Vision Language Models (VLMs) in human-interactive settings, it is important that we evaluate how well these models can adapt to real-time preferences for different users.

By Hannah Gao (Massachusetts Institute of Technology), Dylan Hadfield-Menell (Massachusetts Institute of Technology), Rachel Ma (Massachusetts Institute of Technology)

llms multimodal benchmarks

arXiv Machine Learning

Jun 9

RepoLaunch: Automating Build and Management of Code Repositories across Languages and Platforms

arXiv:2603. 05026v2 Announce Type: replace-cross Abstract: Language model (LM) agents have driven substantial progress in automated software engineering (SWE), yet building and testing software repositories at scale remains a largely manual and labor-intensive bottleneck.

By Kenan Li, Rongzhi Li, Linghao Zhang, Qirui Jin, Liao Zhu, Xiaosong Huang, Geng Zhang, Yikai Zhang, Shilin He, Chengxing Xie, Xin Zhang, Zijian Jin, Bowen Li, Chaoyun Zhang, Yu Kang, Yufan Huang, Elsie Nallipogu, Saravan Rajmohan, Qingwei Lin, Dongmei Zhang

llms agents benchmarks

arXiv AI

Jun 9

Reconstructing and forecasting disease trajectories of patients with Alzheimer's disease using routine data in resource-constrained settings

arXiv:2606. 07798v1 Announce Type: new Abstract: Alzheimer's disease is a progressive neurodegenerative disorder, and its progression varies substantially across patients.

By Ratnadeep Das, Atri Chatterjee, Sitikantha Roy

benchmarks

arXiv AI

Jun 9

XCR-Bench: Benchmarking Cross-Cultural Reasoning in LLMs via Culture-Specific Items and Hall's Triad

arXiv:2601. 14063v2 Announce Type: replace-cross Abstract: Cross-cultural competence in large language models (LLMs) requires understanding and adapting Culture-Specific Items (CSIs) across varying cultural contexts.

By Mohsinul Kabir, Tasnim Ahmed, Md Mezbaur Rahman, Shaoxiong Ji, Hassan Alhuzali, Yuechen Jiang, Jimin Huang, Sophia Ananiadou

llms nlp benchmarks

arXiv Machine Learning

Jun 9

ATLAS: Verifier-Guided Adaptive Latent Activation Steering for Efficient LLM Reasoning

arXiv:2601. 03093v2 Announce Type: replace Abstract: Recent work on activation and latent steering has demonstrated that modifying internal representations can effectively guide large language models (LLMs) toward improved reasoning and efficiency without updating model parameters.

By Tuc Nguyen, Thai Le

llms reinforcement-learning benchmarks

arXiv AI

Jun 9

Diverse Thinking Schemata Elicit Better Reasoning in Large Language Models

arXiv:2606. 08974v1 Announce Type: new Abstract: Large reasoning models (LRMs) have attracted increasing attention for their ability to solve complex mathematical problems by generating extended reasoning chains.

By Xinyue Liang, Yizhe Yang, Yu Bai, Bin Xu, Jiawei Li, Yang Gao

llms reinforcement-learning benchmarks

arXiv AI

Jun 9

The Last Visible Pixel: Probing Fine-Scale Perception in Vision-Language Models

arXiv:2606. 07861v1 Announce Type: cross Abstract: Recent vision-language models (VLMs) excel at multimodal understanding and reasoning, yet their fine-grained visual perception remains underexplored.

By Lujun Li, Lama Sleem, Niccolo Gentile, Yangjie Xu, Yewei Song, Wenbo Wu, Radu State

llms multimodal benchmarks

arXiv Machine Learning

Jun 9

Beyond Fixed Rounds: Data-Free Early Stopping for Practical Federated Learning

arXiv:2601. 22669v3 Announce Type: replace Abstract: Federated Learning (FL) facilitates decentralized collaborative learning without transmitting raw data.

By Youngjoon Lee, Hyukjoon Lee, Seungrok Jung, Andy Luo, Jinu Gong, Yang Cao, Joonhyuk Kang

benchmarks safety

Get the daily briefing