Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,477 stories · RSS feed

arXiv Machine Learning

Jun 30

FlexTab: A Flexible Encoder-Decoder Architecture for In-Context Learning Across Diverse Tabular Tasks

arXiv:2606. 30336v1 Announce Type: new Abstract: We introduce FlexTab, a flexible encoder-decoder architecture for in-context learning on tabular data that pairs a single, task-agnostic encoder with a suite of task-specific decoders.

By Marek Polewczyk, Maximilian Schambach, Marco Spinaci, Sam Thelin, Johannes H\"ohne

llms rag benchmarks

arXiv Machine Learning

Jun 30

DataComp-VLM: Improved Open Datasets for Vision-Language Models

arXiv:2606. 28551v1 Announce Type: cross Abstract: Building performant Vision-Language Models (VLMs) requires carefully curating large-scale training datasets, yet the community lacks systematic benchmarks for evaluating such curation strategies.

By Matteo Farina, Vishaal Udandarao, Thao Nguyen, Selim Kuzucu, Maximilian B\"other, Andreas Hochlehnert, Adhiraj Ghosh, Marianna Nezhurina, Karsten Roth, Joschka Struber, Yuhui Zhang, Sebastian Dziadzio, Elaine Sui, Soumya Jahagirdar, Dhruba Ghosh, Hasan Hammoud, Thomas De Min, Simone Caldarella, Jehanzeb Mirza, Sedrick Keh, Mehdi Cherti, Hilde Kuehne, Bernt Schiele, Serena Yeung-Levy, Muhammad Ferjad Naeem, Federico Tombari, Ana Klimovic, Elisa Ricci, Matthias Bethge, Sewoong Oh, Ameya Prabhu, Alessio Tonioni, Jenia Jitsev, Massimiliano Mancini, Ludwig Schmidt, Nikhil Parthasarathy

llms multimodal benchmarks

arXiv AI

Jun 30

Pondering the Way: Spatial-perceiving World Action Model for Embodied Navigation

arXiv:2606. 29908v1 Announce Type: cross Abstract: Existing world model-based planners for visual navigation typically follow a verification-centric paradigm, decoupling goal intent from trajectory synthesis.

By Hong Chen, Daqi Liu, Zehan Zhang, Haiguang Wang, Tianhao Lu, Longfei Yan, Haiyang Sun, Fangzhen Li, Hongwei Xie, Bing Wang, Guang Chen, Hangjun Ye, Yihua Tan

robotics benchmarks safety

arXiv Machine Learning

Jun 30

Improving Coherence in Hierarchical Time Series Forecasting using Structured Temporal Fusion

arXiv:2606. 28553v1 Announce Type: new Abstract: In many real-world applications, such as retail sales, energy usage, and supply chain planning, forecasting is performed across hierarchical structures.

By Ruchi Pakhle

llms rag benchmarks

arXiv AI

Jun 30

MotionAtlas: Detailed Region Captioning for Motion-Centric Videos

arXiv:2606. 29531v1 Announce Type: cross Abstract: We propose MotionAtlas, a system for detailed captioning of motion-centric videos, comprising (1) a dedicated human-annotated benchmark, (2) a scalable, high-quality pipeline to construct training samples, and (3) a family of powerful Video-MLLMs.

By Weisong Liu, Haochen Wang, Kuan Gao, Yuhao Wang, Yikang Zhou, Zhongwei Ren, Jacky Mai, Anna Wang, Yanwei Li, Jason Li, Zhaoxiang Zhang

benchmarks safety data-engineering

arXiv Machine Learning

Jun 30

Boundary Degree as a Node-level Feature for Epidemic Scenario Identification in Agent-based Cascade Simulations

arXiv:2606. 29596v1 Announce Type: cross Abstract: Characterizing the scenario underlying an epidemic from its disease cascade is an important task in simulation analytics.

By Amro Alabsi Aljundi, Galen Harrison, Jiangzhuo Chen, Abhijin Adiga, Anil Kumar Vullikanti, Madhav V. Marathe

agents benchmarks

arXiv Machine Learning

Jun 30

HyperDFlash: Hyper-Connection-Aligned Block Speculative Decoding with Gated Residual Reduction

arXiv:2606. 26744v2 Announce Type: replace Abstract: We present HyperDFlash, a block-parallel speculative decoding framework tailored to DeepSeek-V4's Hyper-Connections (HC).

By Luxi Lin, Shuang Peng, Rui Ma, Junhao Hua, Shuwei Fan, Zhengda Qin, Qiang Wang, Hongjian Sun, Fangmin Chen, Songwei Liu

efficiency benchmarks safety

arXiv AI

Jun 30

DyGnROLE: Asymmetric Pretraining for Edge Classification on Dynamic Graphs

arXiv:2602. 23135v2 Announce Type: replace-cross Abstract: Edge classification on directed dynamic graphs requires modeling interactions between source and destination nodes exhibiting asymmetrical behavioral patterns and temporal dynamics.

By Tyler Bonnet, Marek Rei

llms rag benchmarks safety

arXiv Machine Learning

Jun 30

Beyond Trajectory Matching: Reflow with Marginal Distribution Alignment

arXiv:2606. 29287v1 Announce Type: new Abstract: Diffusion and continuous-flow generative models achieve high-quality generation, and their deterministic sampling can be formulated as solving learned ODE dynamics.

By Chen Wang, Peiran Yun, Pan Xie, Ke Deng

diffusion efficiency benchmarks safety

arXiv AI

Jun 30

Data and Evaluation Closed-Loop for Model Capability Enhancement

arXiv:2606. 28471v1 Announce Type: new Abstract: Model capability is the central variable in LLM pre-training, yet is never observed directly: data shapes it prospectively, while evaluation reveals it only retrospectively, compressing samples, prompts, decoding, and scoring rules into one noisy score.

By Zhixuan Li, Jiangan Yuan, Han Xu

llms benchmarks

arXiv Machine Learning

Jun 30

Representational Depth of Evaluation Awareness Shifts With Scale in Open-Weight Language Models

arXiv:2606. 29196v1 Announce Type: new Abstract: Do language models know when they are being tested?

By Archit Manek

llms benchmarks safety

arXiv AI

Jun 30

SEVA: Self-Evolving Verification Agent with Process Reward for Fact Attribution

arXiv:2606. 29713v1 Announce Type: cross Abstract: Hallucination is the reliability bottleneck for LLM-based agents, and fact attribution verifiers are the last line of defense -- yet today's verifiers emit only opaque binary labels, leaving agents unable to self-correct and operators unable to audit.

By Aojie Yuan, Yi Nian, Haiyue Zhang, Zijian Su, Yue Zhao

llms agents reinforcement-learning benchmarks safety

arXiv AI

Jun 30

When More Sampling Hurts: The Modal Ceiling and Correlation Ceiling of Test-Time Scaling

arXiv:2606. 28661v1 Announce Type: cross Abstract: People overthink; language models over-sample, and the extra effort can talk both into a worse answer.

By Yong Yi Bay, Kathleen A. Yearick

llms benchmarks

arXiv AI

Jun 30

Recursive Self-Evolving Agents via Held-Out Selection

arXiv:2606. 28374v1 Announce Type: new Abstract: LLM agents are increasingly improved without weight updates by evolving a natural-language artifact, such as reflections, workflows, playbooks, cheatsheets, or optimized prompts, that conditions a frozen policy.

By Michael Nguyen, Quoc Nguyen, Paul Vuong

llms agents benchmarks

arXiv Machine Learning

Jun 30

DiLaServe: High SLO Attainment Serving for Diffusion Language Models

arXiv:2606. 29094v1 Announce Type: new Abstract: Diffusion language models (DLMs) have recently emerged as a promising alternative to conventional autoregressive language models.

By Tzu-Tao Chang, Benjamin Yuanyang Hong, Kiet Pham, Shivaram Venkataraman

llms diffusion efficiency benchmarks

arXiv AI

Jun 30

DuoMem: Towards Capable On-Device Memory Agents via Dual-Space Distillation

arXiv:2606. 29961v1 Announce Type: cross Abstract: Large Language Model (LLM)-based agents can solve complex procedural tasks by interacting with environments over multiple turns, but this ability typically depends on large models, long contexts, and repeated inference calls.

By Peyman Hosseini, Ondrej Bohdal, Ahmed Alajrami, Andrea Maracani, Ignacio Castro, Matthew Purver, Mete Ozay, Savas Ozkan, Taha Ceritli

llms agents robotics fine-tuning efficiency benchmarks

arXiv AI

Jun 30

Diff-Based Code Corruption using LLMs for Large-Scale Bugfix Benchmarking

arXiv:2606. 29088v1 Announce Type: cross Abstract: There are various benchmarks to evaluate bugfixing capabilities of Large Language Models.

By Bal\'azs Szalontai, \'Abel Szauter, Bal\'azs M\'arton, P\'eter Verebics, Bal\'azs Pint\'er, Tibor Gregorics

llms fine-tuning benchmarks

arXiv Machine Learning

Jun 30

Exploring the Effects of Entanglement on Quantum Machine Learning of Pathogen Epitope-Receptor Binding

arXiv:2606. 28655v1 Announce Type: cross Abstract: Parameterized quantum circuits (PQCs) provide a flexible substrate for hybrid quantum machine learning (QML), but their practical value on Noisy Intermediate-Scale Quantum (NISQ) devices remains an empirical question, especially because training depth and scale can introduce optimization challenges such as barren plateaus.

By Aspen Erlandsson Brisebois, Luis Pablo Gonzalez Dominguez, Shivansi Prajapati, Zahed Khatooni, Heather L. Wilson, Connor Burbridge, Brook Byrns, Sureesh Tikoo, Christophe Pere, Steven Rayan, Gordon Broderick

rag benchmarks

arXiv Machine Learning

Jun 30

A Neuroimaging Simulation Framework for Developing and Evaluating Causal AI

arXiv:2606. 28684v1 Announce Type: cross Abstract: Causally linking disease-related factors to image-derived biomarkers provides a powerful pathway to understanding disease mechanisms.

By Eryn Libert-Scott, Emma A. M. Stanley, Vibujithan Vigneshwaran, Matthias Wilms, Erik Y. Ohara, Nils D. Forkert

benchmarks

arXiv AI

Jun 30

How Alignment Routes: Localizing, Scaling, and Controlling Policy Circuits in Language Models

arXiv:2604. 04385v5 Announce Type: replace-cross Abstract: We localize the policy routing mechanism in alignment-trained language models.

By Gregory N. Frank

llms benchmarks safety

Get the daily briefing