Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,494 stories · RSS feed

arXiv AI

Jun 3

FFR: Forward-Forward Learning for Regression

arXiv:2606. 03927v1 Announce Type: cross Abstract: The Forward-Forward (FF) algorithm offers a computationally efficient and biologically plausible alternative to backpropagation (BP) by training neural networks through purely local, layer-wise optimization.

By Xinyang Liu, Xuanyu Liang, Shiqi Ding, Boyang Li, Zhiqiang Que, Jiayang Li, Guosheng Hu

benchmarks

arXiv AI

Jun 3

Spike-Aware C++ INT8 Inference for Sparse Spiking Language Models on Commodity CPUs

arXiv:2606. 03026v1 Announce Type: cross Abstract: Spiking language models expose activation sparsity that dense Transformer runtimes do not directly exploit.

By Ting Liu

llms agents robotics efficiency benchmarks

arXiv AI

Jun 3

Assistax: A Multi-Agent Hardware-Accelerated Reinforcement Learning Benchmark for Assistive Robotics

arXiv:2507. 21638v2 Announce Type: replace Abstract: The development of reinforcement learning (RL) algorithms has been largely driven by ambitious challenge tasks and benchmarks.

By Leonard Hinckeldey, Elliot Fosong, Rimvydas Rubavicius, Elle Miller, Trevor McInroe, Fan Zhang, Patricia Wollstadt, Stefano V. Albrecht, Subramanian Ramamoorthy

agents reinforcement-learning robotics benchmarks

arXiv AI

Jun 3

P\textsuperscript{2}-DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization

arXiv:2606. 03376v1 Announce Type: cross Abstract: Hallucination has recently garnered significant research attention in Large Vision-Language Models (LVLMs).

By Ruipeng Zhang, Zhihao Li, Haozhang Yuan, C. L. Philip Chen, Tong Zhang

llms reinforcement-learning multimodal benchmarks safety

arXiv AI

Jun 3

Towards Compact Autonomous Driving Perception with Balanced Learning and Multi-sensor Fusion

arXiv:2606. 02979v1 Announce Type: cross Abstract: We present a novel compact deep multi-task learning model to handle various autonomous driving perception tasks in one forward pass.

By Oskar Natan, Jun Miura

agents computer-vision robotics benchmarks

arXiv AI

Jun 3

CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks

arXiv:2606. 03650v1 Announce Type: cross Abstract: Choosing or ranking language models for a specific application is hardest when no task-specific labeled data exists, and standard public benchmarks cannot be trusted, their items having likely leaked into pretraining, so scores reflect memorization rather than fitness.

By Alexander Apartsin, Yehudit Aperstein

llms benchmarks safety

arXiv AI

Jun 3

PrimeSVT: An Automated Memory-aware Pruning Framework with Prioritized Compression Policy for Spiking Vision Transformers

arXiv:2606. 03428v1 Announce Type: cross Abstract: The large sizes of Spiking Vision Transformers (SViTs) still hinder their embedded implementation, highlighting the need for model compression.

By Rachmad Vidya Wicaksana Putra, Achyuta Muthuvelan, Alberto Marchisio, Muhammad Shafique

llms fine-tuning efficiency benchmarks

arXiv AI

Jun 3

TurtleAI: Benchmarking Multimodal Models for Visual Programming in Turtle Graphics

arXiv:2606. 03626v1 Announce Type: cross Abstract: Vision-language models (VLMs) have been explored for visual programming, where they generate code to solve visual tasks.

By Chao Wen, Jacqueline Staub, Adish Singla

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jun 3

Efficient ASR Training with Conversations that Never Happened

arXiv:2606. 03957v1 Announce Type: cross Abstract: Conversational ASR for lower-resource languages and niche domains is limited by the scarcity of domain-matched multi-speaker training data.

By M\'at\'e Gedeon, P\'eter Mihajlik

llms multimodal benchmarks

arXiv AI

Jun 3

Constitutional On-Policy Safe Distillation

arXiv:2606. 03089v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD) has emerged as an efficient post-training paradigm by using a teacher conditioned on privileged information to provide dense token-level supervision.

By Ming Wen, Yuxuan Liu, Kun Yang, Yunhao Feng, Zhuoer Xu, Yuhao Sun, Shiwen Cui, Xiang Zheng, Xingjun Ma, Yu-Gang Jiang

fine-tuning efficiency benchmarks safety

arXiv AI

Jun 3

Conditional Latent Diffusion Model with Fourier-based Motion Modelling for Virtual Population Synthesis

arXiv:2606. 03827v1 Announce Type: cross Abstract: In-silico trials of medical devices require the generation of virtual populations of anatomies.

By Shaokun Lan, Haoran Dou, Jinghan Huang, Arezoo Zakeri, Fengming Lin, Zherui Zhou, Jinming Duan, Alejandro F. Frangi

diffusion benchmarks

arXiv AI

Jun 3

When to Re-Plan: Subgoal Persistence in Hierarchical Latent Reasoning

arXiv:2606. 03741v1 Announce Type: new Abstract: Long-horizon reasoning requires a system to commit to medium-horizon intent without becoming rigid: re-plan too often and computation never coheres into multi-step structure; commit too long and the plan goes stale.

By Ayushi Chadha

benchmarks safety

arXiv AI

Jun 3

Agentic Chain-of-Thought Steering for Efficient and Controllable LLM Reasoning

arXiv:2606. 03965v1 Announce Type: cross Abstract: Large language models improve final-answer accuracy through extended chain-of-thought reasoning, but often spend tokens inefficiently and offer little inference-time control.

By Yu Xia, Zhouhang Xie, Xin Xu, Byungkyu Kang, Prarit Lamba, Xiang Gao, Julian McAuley

llms agents reinforcement-learning benchmarks

arXiv AI

Jun 3

A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks

arXiv:2605. 28556v2 Announce Type: replace Abstract: As agent capabilities advance, existing benchmarks, such as $\tau^2$-Bench, are becoming increasingly saturated.

By Tomer Keren, Nitay Calderon, Asaf Yehudai, Yotam Perlitz, Michal Shmueli-Scheuer, Roi Reichart

llms agents benchmarks

arXiv AI

Jun 3

Automatic Layer Selection for Hallucination Detection

arXiv:2605. 26366v3 Announce Type: replace Abstract: Recent studies on hallucination detection have shown that hallucination-related signals are more strongly encoded in intermediate layers than in the final layer of large language models (LLMs).

By Xinpeng Wang, William X. Cao, Andrew Gordon Wilson, Zhe Zeng

llms nlp benchmarks safety

arXiv AI

Jun 3

The Reliability Gap in Benchmark Auditing: Distribution Shift and Scale as Failure Modes of Contamination Detection

arXiv:2606. 03305v1 Announce Type: new Abstract: Benchmark contamination, where evaluation examples appear in a model's training data, threatens the validity of LLM assessment.

By Wojciech Zarzecki, Jan Dubi\'nski, Sebastian Cygert

llms benchmarks

arXiv Machine Learning

Jun 3

WaterSIC: Information-Theoretically (Near) Optimal Linear Layer Quantization

arXiv:2603. 04956v2 Announce Type: replace Abstract: This paper considers the problem of converting a given dense linear layer to low precision.

By Egor Lifar, Semyon Savkin, Or Ordentlich, Yury Polyanskiy

llms efficiency benchmarks

arXiv Machine Learning

Jun 3

A Cartesian-3j Framework for Machine Learning Interatomic Potentials

arXiv:2512. 16882v2 Announce Type: replace-cross Abstract: Machine learning interatomic potentials (MLIPs) have brought substantial gains in the extrapolation capability in computational chemistry.

By Zemin Xu, Chenyu Wu, Wenbo Xie, P. Hu

benchmarks safety

arXiv Machine Learning

Jun 3

Zero-Shot 3D Question Answering via Hierarchical View-to-Token Transportation

arXiv:2606. 03100v1 Announce Type: cross Abstract: Recently, zero-shot 3D scene understanding via 2D Vision-Language Models (VLMs) has gained increasing research interest due to their promising spatial reasoning capabilities.

By Dongsheng Wang, Dawei Su, Hui Huang

llms rag nlp multimodal benchmarks

arXiv AI

Jun 3

Evaluating LLMs' Effectiveness on Real-World Consumer Device Repair Questions

arXiv:2606. 03331v1 Announce Type: cross Abstract: Consumer device repair is an important but underexplored testbed for large language models (LLMs).

By Atm Mizanur Rahman (University of Illinois Urbana-Champaign), Md Arid Hasan (University of Toronto), Syed Ishtiaque Ahmed (University of Toronto), Sharifa Sultana (University of Illinois Urbana-Champaign)

llms benchmarks safety

Get the daily briefing