Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

22,245 stories · RSS feed

arXiv Machine Learning

Jun 10

CodeAlchemy: Synthetic Code Rewriting at Scale

arXiv:2606. 10087v1 Announce Type: cross Abstract: Pre-training on raw code teaches syntax but provides sparse signal for diverse real-world task formats.

By Ankit Gupta, Aditya Prasad, Rameswar Panda

llms benchmarks

arXiv AI

Jun 10

GRID: Scaling Task-Agnostic Inference in Continual Prompt Tuning

arXiv:2507. 14725v4 Announce Type: replace-cross Abstract: Prompt-based continual learning (CL) offers a parameter-efficient way to adapt large language models (LLMs) across task sequences.

By Anushka Tiwari, Sayantan Pal, Rohini K. Srihari, Kaiyi Ji

llms benchmarks

arXiv Machine Learning

Jun 10

Blind denoising diffusion models and the blessings of dimensionality

arXiv:2602. 09639v2 Announce Type: replace Abstract: Denoising diffusion models (DDMs) are state-of-the-art methods for learning densities from data across numerous domains, yet many aspects of the training and sampling pipeline remain poorly understood.

By Zahra Kadkhodaie, Aram-Alexandre Pooladian, Sinho Chewi, Eero Simoncelli

rag diffusion benchmarks

arXiv AI

Jun 10

Non-Parametric Structural Priors for Geometry Theorem Prediction

arXiv:2603. 04852v2 Announce Type: replace Abstract: Multi-step theorem prediction is a central challenge in geometry problem solving.

By Junbo Zhao, Ting Zhang, Can Li, Wei He, Jingdong Wang, Hua Huang

llms rag benchmarks

arXiv Machine Learning

Jun 10

Inverse Probability Weighting and Age-of-Information Aggregation for Decentralized Federated Learning under Partial Reception

arXiv:2606. 10774v1 Announce Type: new Abstract: Decentralized Federated Learning (DFL) over lossy wireless networks faces two key challenges: selection bias, where updates from poor-quality links are systematically underrepresented due to partial model reception, and update staleness, where asynchronous nodes contribute outdated information.

By Chanuka A. S. Hewa Kaluannakkage, Rajkumar Buyya

benchmarks safety

arXiv Machine Learning

Jun 10

It\^o maps for any-step SDEs

arXiv:2606. 11156v1 Announce Type: cross Abstract: Recent one-step generative models accelerate sampling by learning deterministic flow maps of the underlying dynamics.

By Zhengkai Pan, Peter Potaptchik, Wenxi Yao, Michael S. Albergo, Jakiw Pidstrigach

efficiency benchmarks

arXiv Machine Learning

Jun 10

FailureScope: Cross-Regime Behavioral Diagnosis of Language Model Weaknesses

arXiv:2606. 09878v1 Announce Type: new Abstract: Standard benchmarks report aggregate accuracy, but practitioners need to know which specific capabilities a model lacks.

By Nicholas Saban

llms agents multimodal benchmarks safety

arXiv Machine Learning

Jun 10

SPDM: Geometry-Modulated State Space Modeling with Manifold Constraints for Time Series Forecasting

arXiv:2606. 09917v1 Announce Type: new Abstract: Multivariate time series forecasting requires capturing the continuously evolving correlation structure among interacting variables.

By Xingsheng Chen, Siu-Ming Yiu

rag benchmarks

arXiv Machine Learning

Jun 10

Symmetrization of Loss Functions for Robust Training of Neural Networks in the Presence of Noisy Labels

arXiv:2605. 20347v2 Announce Type: replace Abstract: Labeling a training set is often expensive and susceptible to errors, making the design of robust loss functions for label noise an important problem.

By Alexandre Lemire Paquin, Brahim Chaib-Draa, Philippe Gigu\`ere

benchmarks

arXiv AI

Jun 10

Anomaly Detection and Root Cause Analysis for Microservice Systems

arXiv:2606. 09942v1 Announce Type: cross Abstract: Microservice systems are widely used to build cloud applications, yet their complexity makes failures inevitable, degrading user experience and causing economic loss.

By Luan Pham

multimodal benchmarks

arXiv AI

Jun 10

The Interlocutor Effect: Why LLMs Leak More Personal Data to Agents Than Humans

arXiv:2606. 09844v1 Announce Type: cross Abstract: Large Language Models (LLMs) alter their privacy behavior based on the perceived identity of their interlocutor.

By Faouzi El Yagoubi, Godwin Badu-Marfo, Ranwa Al Mallah

llms agents benchmarks safety

arXiv Machine Learning

Jun 10

TRAPS: Therapeutic Response Analysis via Pathway-informed Stratification

arXiv:2606. 09898v1 Announce Type: new Abstract: Cancer treatment planning requires decisions across multiple clinical dimensions at once.

By Sujoy Banik, Sayantan Chakraborty, Boishakhi Das Toma, Zainab Ghafoor, Ushashi Bhattacharjee, Koushik Howlader, Tirtho Roy

benchmarks

arXiv AI

Jun 10

AuRA: Internalizing Audio Understanding into LLMs as LoRA

arXiv:2606. 11033v1 Announce Type: cross Abstract: Recent efforts to extend large language models (LLMs) to speech inputs typically rely on cascaded ASR-LLM pipelines, end-to-end speech-language models, or bridge/distillation-based adaptation.

By Bo Cheng, Lei Shi, Zhanyu Ma, Yuan Wu, Jun Xu, Jiuchong Gao, Jinghua Hao, Renqing He

llms rag fine-tuning efficiency multimodal benchmarks

arXiv Machine Learning

Jun 10

When to Align, When to Predict: A Phase Diagram for Multimodal Learning

arXiv:2606. 11190v1 Announce Type: new Abstract: Cross-modal alignment (CA) and cross-modal prediction (CP) are the dominant paradigms for multimodal representation learning, yet there is no systematic understanding of when each succeeds, when each fails, and when cross-modal training helps at all -- a gap that leaves practitioners, especially in scientific domains like biomedicine or astrophysics, with heterogeneous instruments and multiple levels of organization and measurement, unable to diagnose why standard methods underperform the best single modality.

By Ilay Kamai, Hugues Van Assel, Aviv Regev, Hagai B. Perets, Randall Balestriero

multimodal benchmarks safety

arXiv AI

Jun 10

Benchmarking Knowledge Editing using Logical Rules

arXiv:2606. 10554v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly deployed in real-world applications that require access to up-to-date knowledge.

By Tatiana Moteu Ngoli, NDah Jean Kouagou, Hamada M. Zahera, Axel-Cyrille Ngonga Ngomo

llms benchmarks

arXiv AI

Jun 10

Lost in Serialization: Invariance and Generalization of LLM Graph Reasoners

arXiv:2511. 10234v3 Announce Type: replace-cross Abstract: While promising, graph reasoners based on Large Language Models (LLMs) lack built-in invariance to symmetries in graph representations.

By Daniel Herbst, Lea Karbevska, Divyanshu Kumar, Akanksha Ahuja, Fatemeh Gholamzadeh Nasrabadi, Fabrizio Frasca

llms fine-tuning benchmarks

arXiv AI

Jun 10

Vision-Assisted Foundation Model for Solving Multi-Task Vehicle Routing Problems

arXiv:2606. 10431v1 Announce Type: cross Abstract: Multi-task vehicle routing problems play a critical role in enhancing efficiency across various industries and service sectors.

By Shuangchun Gui, Zhiguang Cao, Wen Song, Yew-Soon Ong

rag benchmarks

arXiv AI

Jun 10

RAT: Reference-Augmented Training for ASV Anti-Spoofing

arXiv:2606. 10908v1 Announce Type: cross Abstract: We introduce a spoofing countermeasure architecture conditioned on speaker-reference recordings, but observe that it converges to a solution that effectively ignores the reference during inference.

By Vojt\v{e}ch Stan\v{e}k, Anton Firc, Jakub Re\v{s}, Kamil Malinka

benchmarks

arXiv Machine Learning

Jun 10

From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents

arXiv:2606. 09863v1 Announce Type: new Abstract: LLM agents can fail silently by asserting task completion when the environment state shows otherwise.

By Laksh Advani

llms agents benchmarks

arXiv AI

Jun 10

CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks

arXiv:2606. 09833v1 Announce Type: cross Abstract: AI agents are reshaping the workspace, leading to drastic change of how humans work.

By Yijia Shao, Zora Zhiruo Wang, Neel Ahuja, Yicheng Wang, Bowen Liu, Diyi Yang

llms agents benchmarks

Get the daily briefing