Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,859 stories · RSS feed

arXiv Machine Learning

Jun 15

Deep Doubly Debiased Longitudinal Effect Estimation with ICE G-Computation

arXiv:2602. 12379v2 Announce Type: replace Abstract: Estimating longitudinal treatment effects is essential for sequential decision-making but is challenging due to treatment-confounder feedback.

By Wenxin Chen, Weishen Pan, Kyra Gan, Fei Wang

llms benchmarks safety

arXiv Machine Learning

Jun 15

Aligned but Stereotypical? How System Prompts Shape Demographic Bias in LLM-Based Text-to-Image Models

arXiv:2512. 04981v2 Announce Type: replace-cross Abstract: Text-to-image (T2I) systems increasingly rely on Large Language Model (LLM)-based text conditioning to interpret and expand user prompts.

By NaHyeon Park, Na Min An, Kunhee Kim, Soyeon Yoon, Jiahao Huo, Hyunjung Shim

llms rag diffusion benchmarks safety

arXiv AI

Jun 15

Same-Origin Policy for Agentic Browsers

arXiv:2606. 14027v1 Announce Type: cross Abstract: Agentic browsers integrate autonomous AI agents into web browsers, enabling users to accomplish web tasks through natural-language instructions.

By Xilong Wang, Xiaoxing Chen, Patrick Li, Dawn Song, Neil Gong

agents benchmarks

arXiv AI

Jun 15

Poker Arena: Multi-Axis Profiling of Strategic Reasoning and Memory in LLMs

arXiv:2606. 13815v1 Announce Type: new Abstract: Strategic reasoning under uncertainty underpins consequential decisions in negotiation, finance, and policy, but prevailing game-play benchmarks collapse heterogeneous reasoning dimensions into a single scalar, leaving the capability structure of frontier LLMs unexamined.

By Pratham Singla, Shivank Garg, Vihan Singh

llms benchmarks

arXiv AI

Jun 15

AFFORDANCE20Q: Evaluating Affordance Reasoning from Physical Properties

arXiv:2606. 14240v1 Announce Type: new Abstract: Affordance reasoning, the inference of an object's action possibilities from its physical properties (e.

By Yifan Jiang, Meige Yang, Zitong Li, Jay Pujara

llms benchmarks

arXiv AI

Jun 15

FAConformer: Frequency-Aware Convolutional Transformer for Auditory Attention Decoding

arXiv:2606. 14120v1 Announce Type: cross Abstract: Auditory attention decoding (AAD) aims to infer the attended speaker from neural responses in multi-speaker acoustic environments and is a key problem for neuro-steered hearing systems.

By Ziwei Wang, Xingyi He, Tianwang Jia, Hongbin Wang, Dongrui Wu

llms benchmarks safety

arXiv AI

Jun 15

DIFF-ERO: A Conformance-Aware Loss for Deep Learning in Process Mining

arXiv:2606. 14283v1 Announce Type: cross Abstract: Deep learning has driven many recent advances in process analytics, especially for predictive and prescriptive monitoring.

By Johannes De Smedt, Jari Peeperkorn, Artem Polyvyanyy, Jochen De Weerdt

llms benchmarks

arXiv AI

Jun 15

Learning optimal policies from event logs through reinforcement learning: a comparison of deep and MDP-based approaches

arXiv:2303. 09209v2 Announce Type: replace Abstract: Prescriptive Process Monitoring is an emerging area within Process Mining that focuses on recommending actions to optimize business outcomes.

By Stefano Branchi, Andrei Buliga, Chiara Di Francescomarino, Chiara Ghidini, Riccardo Graziosi, Francesca Meneghello, Massimiliano Ronzani

reinforcement-learning benchmarks

arXiv AI

Jun 15

Token-Level LLM Collaboration via FusionRoute

arXiv:2601. 05106v5 Announce Type: replace Abstract: Large language models (LLMs) exhibit strengths across diverse domains.

By Nuoya Xiong, Yuhang Zhou, Hanqing Zeng, Zhaorun Chen, Furong Huang, Shuchao Bi, Lizhu Zhang, Zhuokai Zhao

llms fine-tuning benchmarks

arXiv AI

Jun 15

EmoMind: Decoding Affective Captions from Human Brain fMRI

arXiv:2605. 16739v2 Announce Type: replace-cross Abstract: Decoding visual experience from brain activity has advanced substantially, but current brain-to-text systems largely recover semantic content while discarding affect.

By Bilal A. Mohammed, Lin Gu, Ruogu Fang

llms benchmarks

arXiv AI

Jun 15

SAFformer:Improving Spiking Transformer via Active Predictive Filtering

arXiv:2605. 08270v2 Announce Type: replace-cross Abstract: Spiking Neural Networks (SNNs) offer notable advantages in biological plausibility and energy efficiency, making them promising candidates for building low-power Transformers.

By Zequan Xie, Weiming Zeng, Yunhua Chen, Sichang Ling, Tongyang Chen, Jinsheng Xiao

llms benchmarks

arXiv Machine Learning

Jun 15

Non-Parametric Machine Text Detection via Multi-View Gaussian Processes

arXiv:2606. 14060v1 Announce Type: new Abstract: Adversarial conditions such as paraphrasing and targeted style transfer sharply degrade the accuracy of machine text detectors.

By Aleem Khan, Nicholas Andrews

llms benchmarks safety

arXiv Machine Learning

Jun 15

Learning the Context of Errors: Black-Box Online Adaptation of Time Series Foundation Models

arXiv:2606. 14222v1 Announce Type: new Abstract: The rapid evolution of Time Series Foundation Models (TSFMs) has advanced zero-shot forecasting across diverse domains.

By Xilin Dai, Yiding Liu, Hongjie Xia, Yifan Hu, Zewei Dong, Jiang-Ming Yang, Qiang Xu

llms fine-tuning benchmarks

arXiv Machine Learning

Jun 15

Neural ARFIMA model for forecasting BRIC exchange rates with long memory

arXiv:2509. 06697v3 Announce Type: replace-cross Abstract: Exchange rate forecasting remains a challenging problem, particularly for emerging economies, where the observed time series exhibit pronounced long-memory dependence, nonlinear dynamics, and sensitivity to macro-financial drivers.

By Donia Besher, Madhurima Panja, Shovon Sengupta, Tanujit Chakraborty

benchmarks

arXiv Machine Learning

Jun 15

Efficient Rationale-based Retrieval: On-policy Distillation from Generative Rerankers based on JEPA

arXiv:2604. 23336v3 Announce Type: replace-cross Abstract: Unlike traditional fact-based retrieval, rationale-based retrieval typically necessitates cross-encoding of query-document pairs using large language models, incurring substantial computational costs.

By Teng Chen, Sheng Xu, Feixiang Guo, Xiaoyu Wang, Qingqing Gu, Hongyan Li, Luo Ji

llms rag robotics efficiency benchmarks

arXiv AI

Jun 15

CADET: Physics-Grounded Causal Auditing and Training-Free Deconfounding of End-to-End Driving Planners

arXiv:2606. 14438v1 Announce Type: cross Abstract: End-to-end (E2E) autonomous-driving planners trained by imitation are prone to statistical shortcuts: they associate scene elements that merely co-occur with expert actions (a roadside object, a building facade) with driving decisions, rather than the variables that causally determine them.

By Zikun Guo

agents benchmarks

arXiv AI

Jun 15

Scalable Production Scheduling: Linear Complexity via Unified Homogeneous Graphs

arXiv:2604. 23841v2 Announce Type: replace-cross Abstract: Efficiently solving the Job Shop Scheduling Problem in real-world industrial applications requires policies that are both computationally lean and topologically robust.

By Jonathan Hoss, Moritz Link, Noah Klarmann

agents diffusion reinforcement-learning benchmarks

arXiv Machine Learning

Jun 15

Learning Variable-Length Tokenization for Generative Recommendation

arXiv:2605. 17779v2 Announce Type: replace Abstract: Generative recommendation reformulates recommendation as next-token prediction over discrete semantic identifiers (IDs).

By Minhao Wang, Bowen Wu, Wei Zhang

nlp efficiency benchmarks

arXiv Machine Learning

Jun 15

Dense Supervision, Sparse Updates: On the Sparsity and Geometry of On-Policy Distillation

arXiv:2606. 13657v2 Announce Type: replace Abstract: On-policy distillation (\textsc{OPD}) has recently become a prominent post-training recipe by combining two desirable ingredients: on-policy student trajectories and dense teacher supervision.

By Guo Yu, Wenlin Liu, Yulan Hu, Hao-Xuan Ma, Jun-Peng Jiang, Han-Jia Ye

llms efficiency multimodal benchmarks

arXiv AI

Jun 15

Succeeding at Scale: Enterprise Retrieval Benchmark Construction and Index-Preserving Query Adaptation for Multi-Tenant Search

arXiv:2601. 04646v4 Announce Type: replace-cross Abstract: Large-scale multi-tenant retrieval systems generate extensive query logs but lack curated relevance labels for effective domain adaptation, resulting in substantial underutilized "dark data.

By Prateek Jain, Shabari S Nair, Ritesh Goru, Prakhar Agarwal, Ajay Yadav, Yoga Sri Varshan Varadharajan, Constantine Caramanis

llms fine-tuning benchmarks

Get the daily briefing