Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,880 stories · RSS feed

arXiv Machine Learning

Jun 10

Time series forecasting from partial observations via Non-negative Matrix Factorization

arXiv:2102. 05314v2 Announce Type: replace Abstract: In modern time series problems, one aims at forecasting multiple time series with possible missing and noisy values.

By Yohann de Castro (ICJ, PSPM, CERMICS UMR 9032, ECL, IUF), Luca Mencarelli (CERMICS UMR 9032)

llms benchmarks

arXiv Machine Learning

Jun 10

Decision-Making under Combinatorial Risk

arXiv:2606. 10092v1 Announce Type: new Abstract: Decision-making under risk is typically studied through single-shot lottery choices.

By Yifan Hong, Hongmiao Fan, Chen Wang

benchmarks

arXiv AI

Jun 10

SPACE: Source-free Proxy Anchor Concept Erasure for MLLMs

arXiv:2606. 09868v1 Announce Type: cross Abstract: As Multimodal Large Language Models (MLLMs) face growing privacy risks and regulatory constraints, machine unlearning (MU) has emerged as a crucial solution for removing sensitive data while preserving model performance.

By Zhijing Zhang, Jiaqi Ding, Qianshan Wei, Nan Zhou, Jiaqi Li, Yongliang Wu, Tongxin Zhu, Xiaolin Fang

llms multimodal benchmarks safety

arXiv AI

Jun 10

Piper: A Programmable Distributed Training System

arXiv:2606. 11169v1 Announce Type: cross Abstract: Large-scale model training increasingly relies on composing multiple parallelism strategies, such as data, pipeline, and expert parallelism, together with memory-saving optimizations like ZeRO.

By Megan Frisella, Shubham Tiwari, Andy Ruan, Yi Pan, Parker Gustafson, Mat Jacob, Gilbert Bernstein, Stephanie Wang

benchmarks

arXiv AI

Jun 10

GRID: Scaling Task-Agnostic Inference in Continual Prompt Tuning

arXiv:2507. 14725v4 Announce Type: replace-cross Abstract: Prompt-based continual learning (CL) offers a parameter-efficient way to adapt large language models (LLMs) across task sequences.

By Anushka Tiwari, Sayantan Pal, Rohini K. Srihari, Kaiyi Ji

llms benchmarks

arXiv AI

Jun 10

Non-Parametric Structural Priors for Geometry Theorem Prediction

arXiv:2603. 04852v2 Announce Type: replace Abstract: Multi-step theorem prediction is a central challenge in geometry problem solving.

By Junbo Zhao, Ting Zhang, Can Li, Wei He, Jingdong Wang, Hua Huang

llms rag benchmarks

arXiv Machine Learning

Jun 10

It\^o maps for any-step SDEs

arXiv:2606. 11156v1 Announce Type: cross Abstract: Recent one-step generative models accelerate sampling by learning deterministic flow maps of the underlying dynamics.

By Zhengkai Pan, Peter Potaptchik, Wenxi Yao, Michael S. Albergo, Jakiw Pidstrigach

efficiency benchmarks

arXiv Machine Learning

Jun 10

Symmetrization of Loss Functions for Robust Training of Neural Networks in the Presence of Noisy Labels

arXiv:2605. 20347v2 Announce Type: replace Abstract: Labeling a training set is often expensive and susceptible to errors, making the design of robust loss functions for label noise an important problem.

By Alexandre Lemire Paquin, Brahim Chaib-Draa, Philippe Gigu\`ere

benchmarks

arXiv Machine Learning

Jun 10

Validation-Stage Combinatorial Fusion Analysis for Imbalanced Credit-Card Fraud Detection

arXiv:2606. 10393v1 Announce Type: new Abstract: Credit-card fraud detection is difficult because fraudulent transactions are rare, costly, and unevenly distributed.

By Xiao Han, Chenyu Wu

benchmarks

arXiv Machine Learning

Jun 10

Gradient-Guided Reward Optimization for Inference-time Alignment

arXiv:2606. 09635v1 Announce Type: cross Abstract: Ensuring the reliability of Large Language Models (LLMs) under distribution drift requires inference-time adaptation.

By Hankun Lin, Ruqi Zhang

llms reinforcement-learning benchmarks safety

arXiv AI

Jun 10

Anomaly Detection and Root Cause Analysis for Microservice Systems

arXiv:2606. 09942v1 Announce Type: cross Abstract: Microservice systems are widely used to build cloud applications, yet their complexity makes failures inevitable, degrading user experience and causing economic loss.

By Luan Pham

multimodal benchmarks

arXiv AI

Jun 10

Expert-Level Crisis Detection in Mental Health Conversations

arXiv:2606. 10380v1 Announce Type: cross Abstract: Real-world crisis intervention is inherently conversational, yet existing research largely focuses on static texts.

By Grace Byun, Abigail Lott, Rebecca Lipschutz, Sean T. Minton, Elizabeth A. Stinson, Jinho D. Choi

benchmarks

arXiv AI

Jun 10

Attention-Discounted Adaptive Sampler for Masked Diffusion Language Models

arXiv:2606. 10829v1 Announce Type: cross Abstract: Masked diffusion language models can reduce inference steps by revealing multiple tokens per denoising iteration, but this parallelism is fragile: positions that are individually confident may be unsafe to commit together when their predictions are coupled.

By Yusuf Sahin, Ahmed Rockey Saikia, Volkan Cevher, Paolo Favaro

llms rag diffusion benchmarks

arXiv AI

Jun 10

Does Capability Transfer to Subjective Behavior -- and Would Our Instruments Tell Us? A Self-Evolving, Trust-by-Construction Evaluation Paradigm

arXiv:2605. 27914v2 Announce Type: replace-cross Abstract: Benchmarking is mature where answers are verifiable -- math, code, reasoning -- but the fastest-growing uses of LLMs are subjective and human-facing: companionship, emotional support, counseling.

By Yuming (Rapheal), Huang, Yao Liu, Pengjie Ding, Lei Wang, Junchen Wan

llms benchmarks

arXiv AI

Jun 10

Conformal Prediction for Neural Operators: Distribution-Free Uncertainty Quantification in Physics Simulation

arXiv:2606. 09923v1 Announce Type: cross Abstract: Neural operators such as the Fourier Neural Operator (FNO) have emerged as powerful surrogates for solving partial differential equations (PDEs), achieving speedups of several orders of magnitude over traditional numerical solvers.

By Michael Chin

benchmarks safety

arXiv AI

Jun 10

Rotate2Think: Geometric Priming via Orthogonal Rotation to Improve Language Model Reasoning

arXiv:2606. 09873v1 Announce Type: cross Abstract: Reasoning models achieve strong performance on challenging tasks by generating explicit intermediate reasoning traces before producing a final answer.

By Aditya Sharma, Christopher J. Pal, Amal Zouaq

llms rag multimodal benchmarks

arXiv AI

Jun 10

Bypassing Copyright Protection in Diffusion-based Customization via Two-Stage Latent Feature Optimization

arXiv:2606. 09909v1 Announce Type: cross Abstract: With the growing concerns over copyright infringement in diffusion-based customization, adversarial attacks have emerged as a prominent defense strategy to prevent malicious content forgery in personalized image generation.

By Ziang Xu, Wenbo Yu, Hongyao Yu, Hao Fang, Jiawei Kong, Bin Chen, Hao Wu, Shu-Tao Xia, Zhiyong Wu

diffusion benchmarks safety

arXiv Machine Learning

Jun 10

Disjoint or Overlapping? Inference Windowing for Reconstruction-Based Time Series Anomaly Detection

arXiv:2606. 09874v1 Announce Type: new Abstract: Reconstruction-based methods are widely used for time series anomaly detection, where models are trained to reconstruct subsequences, and anomalies are identified through reconstruction errors.

By Guillaume Coulaud (UM, IROKO), Reza Akbarinia (IROKO), Florent Masseglia (IROKO)

llms benchmarks

arXiv AI

Jun 10

A Theory on Flow Matching with Neural Networks

arXiv:2606. 10089v1 Announce Type: cross Abstract: In this work, we develop theoretical foundation for flow matching with neural-network-parameterized conditional velocity fields.

By Yihan He, Qishuo Yin, Yuan Cao, Jianqing Fan, Han Liu

diffusion benchmarks

arXiv AI

Jun 10

Can Image Models Imagine Time? ImageTime: A Novel Benchmark for Probing Visual World Modeling Through Spatiotemporal Consistency

arXiv:2606. 10620v1 Announce Type: cross Abstract: Image generation models now produce high-quality static images, yet their ability to represent how a visual world changes over time remains poorly understood.

By Xinrui Wu, Lichen Huang

llms diffusion multimodal benchmarks safety

Get the daily briefing