Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

23,305 stories · RSS feed

arXiv AI

Jun 2

FreqLite: A Lightweight Frequency-Decomposed Linear Model with Adaptive Reversible Normalization for Robust Long-Term Time-Series Forecasting

arXiv:2606. 01339v1 Announce Type: cross Abstract: Long-term time-series forecasting needs models that are accurate yet efficient enough for commodity hardware.

By Mirza Samad Ahmed Baiga, Syeda Anshrah Gillani

llms benchmarks

arXiv AI

Jun 2

From Graph Retrieval to Schema Realization: Counterfactual Validation for Text-to-SPARQL over Heterogeneous Knowledge Graphs

arXiv:2508. 01815v2 Announce Type: replace-cross Abstract: Text-to-SPARQL maps natural-language questions to executable SPARQL queries over RDF knowledge graphs.

By Yang Zhao, Chengxiao Dai, Yue Xiu, Dusit Niyato

agents nlp benchmarks safety

arXiv AI

Jun 2

Cooperation of Experts: Fusing Heterogeneous Information with Large Margin

arXiv:2505. 20853v3 Announce Type: replace-cross Abstract: Fusing heterogeneous information remains a persistent challenge in modern data analysis.

By Shuo Wang, Shunyang Huang, Jinghui Yuan, Zhixiang Shen, Zhao Kang

benchmarks

arXiv AI

Jun 2

Knowledge-Intensive Video Generation

arXiv:2606. 01285v1 Announce Type: cross Abstract: Text-to-video generation has advanced rapidly in visual quality, but remains under-evaluated for factuality and practical usefulness.

By Chenxu Wang, Mingda Chen

diffusion benchmarks

arXiv Machine Learning

Jun 2

d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation

arXiv:2509. 21474v4 Announce Type: replace Abstract: While diffusion language models (DLMs) have achieved competitive performance in text generation, improving their reasoning ability with reinforcement learning remains an active research area.

By Guanghan Wang, Gilad Turok, Yair Schiff, Marianne Arriola, Volodymyr Kuleshov

llms diffusion reinforcement-learning benchmarks

arXiv AI

Jun 2

Prototype Transformer: Towards Language Model Architectures Interpretable by Design

arXiv:2602. 11852v2 Announce Type: replace Abstract: While state-of-the-art language models (LMs) surpass most humans in certain domains, their reasoning remains largely opaque, reducing trust and increasing the risk of deception and hallucination.

By Yordan Yordanov, Matteo Forasassi, Bayar Menzat, Ruizhi Wang, Chang Qi, Markus Kaltenberger, Amine M'Charrak, Tommaso Salvatori, Thomas Lukasiewicz

llms benchmarks safety

arXiv AI

Jun 2

LLM-Guided Communication for Cooperative Multi-Agent Reinforcement Learning

arXiv:2605. 18077v2 Announce Type: replace Abstract: Communication is a key component in multi-agent reinforcement learning (MARL) for mitigating partial observability, yet prior approaches often rely on inefficient information exchange or fail to transmit sufficient state information.

By Sangjun Bae, Yisak Park, Sanghyeon Lee, Seungyul Han

llms agents reinforcement-learning benchmarks

arXiv AI

Jun 2

Asking Is Not Enough: Protocol Sensitivity in LLM Confidence Calibration

arXiv:2605. 27752v2 Announce Type: replace Abstract: LLM confidence calibration is often evaluated by comparing two signals: token-probability scores and verbalized confidence.

By Hankyeol Kim, Pilsung Kang

llms benchmarks

arXiv AI

Jun 2

Detection vs. Execution: Single-Bucket Probes Miss Half the Mamba-2 State Sink

arXiv:2606. 00930v1 Announce Type: cross Abstract: Mechanistic interpretability often assumes that probes identifying a representational signature also identify the circuit executing the corresponding computation.

By Yuhang Jiang

benchmarks safety

arXiv AI

Jun 2

LLM-Evolved Pattern Generators for Optimal Classical Planning

arXiv:2606. 02438v1 Announce Type: new Abstract: Learned heuristics have recently become a competitive alternative to traditional domain-independent heuristics for satisficing planning.

By Windy Phung, Dominik Drexler, Arnaud Lequen, Jendrik Seipp

llms benchmarks

arXiv AI

Jun 2

Evaluating Interactive Reasoning in Large Language Models: A Hierarchical Benchmark with Executable Games

arXiv:2606. 00103v1 Announce Type: new Abstract: We introduce a multi-turn interactive framework for reasoning evaluation that treats reasoning as active evidence acquisition and belief updating.

By Mingyuan Fan, Weiguang Han, Daixin Wang, Cen Chen, Zhiqiang Zhang, Jun Zhou

llms benchmarks

arXiv AI

Jun 2

Skill-Based Mixture-of-Experts: Adaptive Routing for Heterogeneous Reasoning via Inferred Skills

arXiv:2503. 05641v4 Announce Type: replace-cross Abstract: Combining existing pre-trained LLMs is a promising approach for diverse reasoning tasks.

By Justin Chih-Yao Chen, Sukwon Yun, Elias Stengel-Eskin, Tianlong Chen, Mohit Bansal

llms agents benchmarks

arXiv Machine Learning

Jun 2

Physics-Informed Residuals for Adaptive Mesh Refinement in Finite-Difference PDE Solvers

arXiv:2606. 02475v1 Announce Type: cross Abstract: Classical finite-difference solvers remain reliable tools for partial differential equations, but their efficiency depends on where mesh resolution is placed.

By Henry Kasumba, Ronald Katende

benchmarks

arXiv Machine Learning

Jun 2

Learning Fine-grained Parameter Sharing via Sparse Tensor Decomposition

arXiv:2411. 09816v5 Announce Type: replace Abstract: Large neural networks achieve state-of-the-art performance on many tasks, yet their sheer size hinders deployment on resource-constrained devices.

By Cem \"Uy\"uk, Mike Lasby, Mohamed Yassin, Utku Evci, Yani Ioannou

llms fine-tuning efficiency benchmarks

arXiv AI

Jun 2

DASH: Dual-Branch Score Distillation for Guidance-Calibrated Compact Diffusion Models

arXiv:2606. 00798v1 Announce Type: cross Abstract: Parameter compression of class-conditional diffusion models reveals an underexplored limitation in output-level distillation: the unconditional score branch remains unsupervised, leaving the classifier-free guidance gap underdetermined in the student.

By Abdullah Al Shafi, Kazi Saeed Alam, Sk Imran Hossain, Engelbert Mephu Nguifo

diffusion efficiency benchmarks

arXiv AI

Jun 2

TCAR-Gen: Temporal Graph Retrieval with Evidence Fusion for Knowledge-Grounded Generation

arXiv:2606. 00029v1 Announce Type: cross Abstract: Retrieval-augmented generation systems struggle with temporal reasoning and evidence fusion when answering complex questions over historical criminal case narratives.

By Sidra Nasir, Muhammad Noman Zahid, Rizwan Ahmed Khan

llms rag nlp benchmarks

arXiv AI

Jun 2

Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook

arXiv:2604. 06210v3 Announce Type: replace-cross Abstract: As LLMs are globally deployed, aligning their cultural value orientations is critical for safety and user engagement.

By Jaehyeok Lee, Xiaoyuan Yi, Jing Yao, Hyunjin Hwang, Roy Ka-Wei Lee, Xing Xie, JinYeong Bak

llms benchmarks safety

arXiv AI

Jun 2

STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems

arXiv:2605. 02122v2 Announce Type: replace-cross Abstract: Human evaluation remains the primary standard for assessing modern AI systems, yet annotator disagreement, bias, and variability make system rankings fragile under standard majority vote aggregation.

By Akash Bonagiri, Gerard Janno Anderias, Saee Patil, Angelina Lai, Devang Borkar, Gezheng Kang, Ishant Gandhi, Setareh Rafatirad, Houman Homayoun

agents benchmarks safety

arXiv Machine Learning

Jun 2

CRMA: A Spectrally-Bounded Backbone for Modular Continual Fine-Tuning of LLMs

arXiv:2606. 00382v1 Announce Type: new Abstract: Sequential fine-tuning of large language models forces a choice: let the shared substrate keep learning and accept catastrophic forgetting, or freeze it after task one and foreclose cross-task refinement.

By Kiran Nayudu, Aswini Nutakki, Sai Vinay Naidu, Ashwin Shanmugasundaram

llms fine-tuning efficiency benchmarks

arXiv Machine Learning

Jun 2

PE-means: Improved Differentially Private $k$-means Clustering through Private Evolution

arXiv:2606. 00342v1 Announce Type: new Abstract: We study the problem of differentially private (DP) $k$-means clustering in Euclidean space.

By Thomas Humphries, Zinan Lin, Sergey Yekhanin

benchmarks

Get the daily briefing