Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

22,530 stories · RSS feed

arXiv AI

Jun 10

LC-QAT: Data-Efficient 2-Bit QAT for LLMs via Linear-Constrained Vector Quantization

arXiv:2606. 10531v1 Announce Type: cross Abstract: Quantization-aware training (QAT) is essential for extremely low-bit large language models (LLMs).

By Haoyu Wang, Xingyu Yu, Haiyan Zhao, Fengxiang Wang, Xu Han

llms efficiency benchmarks

arXiv AI

Jun 10

Is Fairness Truly Fair? Towards Reliable Lipschitz Fairness in Multi-Task Learning via Fixed-\texorpdfstring{$\delta$}{delta} Alignment

arXiv:2606. 10632v1 Announce Type: cross Abstract: Lipschitz-style individual fairness formalizes the idea that semantically similar examples should receive similar predictions, but its evaluation in multi-task learning (MTL) can be confounded by method-induced representation scales.

By Junbo Ding, Xin Zang, Chenchen Pan, Donghao Song, Jiaxin Zhu, Danhuai Guo

benchmarks safety

arXiv AI

Jun 10

RoboGPT-R1: Enhancing Robot Task Planning with Reinforcement Learning

arXiv:2510. 14828v3 Announce Type: replace Abstract: Improving the reasoning capabilities of embodied agents is crucial for robots to complete complex human instructions in long-view manipulation tasks successfully.

By Jinrui Liu, Bingyan Nie, Boyu Li, Yaran Chen, Yuze Wang, Shunsen He, Haoran Li

llms agents reinforcement-learning robotics fine-tuning benchmarks

arXiv AI

Jun 10

Do VLMs Reason Like Engineers? A Benchmark and a Stage-wise Evaluation

arXiv:2606. 10833v1 Announce Type: new Abstract: Vision-Language Models (VLMs) demonstrate strong performance on general multimodal reasoning benchmarks, yet their ability to perform engineering reasoning remains largely unexplored.

By Syed Wasiq, Syed Mohamad Tawseeq, Yashwant Pravinrao Bangde, Debaditya Roy

llms nlp multimodal benchmarks

arXiv Machine Learning

Jun 10

Validation-Stage Combinatorial Fusion Analysis for Imbalanced Credit-Card Fraud Detection

arXiv:2606. 10393v1 Announce Type: new Abstract: Credit-card fraud detection is difficult because fraudulent transactions are rare, costly, and unevenly distributed.

By Xiao Han, Chenyu Wu

benchmarks

arXiv Machine Learning

Jun 10

Gradient-Guided Reward Optimization for Inference-time Alignment

arXiv:2606. 09635v1 Announce Type: cross Abstract: Ensuring the reliability of Large Language Models (LLMs) under distribution drift requires inference-time adaptation.

By Hankun Lin, Ruqi Zhang

llms reinforcement-learning benchmarks safety

arXiv AI

Jun 10

Frontier Coding Agents Use Metaprogramming to Adapt to Unfamiliar Programming Languages

arXiv:2606. 10933v1 Announce Type: new Abstract: LLM-based coding agents are usually evaluated in familiar software settings: mainstream languages, common libraries, and public repositories.

By Aman Sharma, Sushrut Thorat, Paras Chopra

llms agents benchmarks

arXiv AI

Jun 10

Evaluating Research-Level Math Proofs via Strict Step-Level Verification

arXiv:2606. 10799v1 Announce Type: new Abstract: Large Language Models (LLMs) struggle to rigorously verify complex mathematical proofs.

By Yifeng Sun

llms agents benchmarks safety

arXiv AI

Jun 10

ComBench: A Benchmark for Rigorous Proof Reasoning and Constructive Realization in Olympiad-Level Combinatorics

arXiv:2606. 10479v1 Announce Type: new Abstract: Combinatorics is central to Olympiad-level mathematical problem solving, requiring deep discrete reasoning, creative constructions, and rigorous structural insight.

By Shunkai Zhang, Haoran Zhang, Yun Luo, Qianjia Cheng, Haodi Lei, Yizhuo Li, Runzhe Zhan, Zhilin Wang, Bangjie Xu, Yucheng Su, Xinmiao Han, Xiaoye Qu, Dongrui Liu, Zhouchen Lin, Yu Qiao, Ning Ding, Yafu Li, Yu Cheng

llms benchmarks

arXiv AI

Jun 10

MMClima: A Framework for Multimodal Climate Science Data and Evaluation

arXiv:2606. 10194v1 Announce Type: cross Abstract: Climate change research increasingly requires AI systems that reason across text, dynamic visual content, and scientific figures, yet existing climate QA benchmarks are small, mostly textual, and cover a narrow range of models.

By Muhammad Umer Sheikh, Hassan Abid, Khawar Shehzad, Ufaq Khan, Muhammad Haris Khan

llms nlp fine-tuning multimodal benchmarks

arXiv AI

Jun 10

ReflectiChain: Epistemic Grounding in LLM-Driven World Models for Supply Chain Resilience

arXiv:2606. 10359v1 Announce Type: new Abstract: AI agents in supply chains face a fundamental epistemic gap: large language models (LLMs) interpret policies but lack physical grounding, while reinforcement learning (RL) optimizes flows but is semantically blind to unstructured constraints.

By Jia Luo

llms agents diffusion reinforcement-learning benchmarks safety

arXiv AI

Jun 10

Bellman-Taylor Score Decoding for Markov Decision Processes with State-Dependent Feasible Action Sets

arXiv:2606. 10979v1 Announce Type: new Abstract: Many Markov decision processes (MDPs) in operations research have feasible actions that are state dependent and defined implicitly by various operational constraints.

By Yi Chen (Lucy), Rushuai Yang (Lucy), Qiang Chen (Lucy), Dongyan (Lucy), Huo

reinforcement-learning benchmarks

arXiv AI

Jun 10

Null-Space Constrained Low-Rank Adaptation for Response-Specified Large Language Model Unlearning

arXiv:2606. 10989v1 Announce Type: new Abstract: Large language model unlearning aims to suppress designated undesirable knowledge while preserving benign capabilities.

By Bocheng Ju, Jianhua Wang, Chengliang Liu, Xiaolin Chang

llms fine-tuning benchmarks safety

arXiv AI

Jun 10

One Token per Multimodal Evidence: Latent Memory for Resource-Constrained QA

arXiv:2606. 10572v1 Announce Type: new Abstract: External memory effectively grounds large language models (LLMs) and vision-language models (VLMs)-based question answering (QA) in relevant multimodal evidence.

By Zhi Zheng, Ziqiao Meng, Hao Luan, Wei Liu, Wee Sun Lee

llms rag nlp efficiency multimodal benchmarks

arXiv Machine Learning

Jun 10

Encoding the Euler Characteristic Transform

arXiv:2606. 10824v1 Announce Type: new Abstract: The Euler Characteristic Curve (ECC) records the Euler characteristic of a linearly embedded cell complex as a function of filtration height in a given direction, and the Euler Characteristic Transform (ECT) is the injective shape descriptor obtained by collecting ECCs over many directions.

By Nello Blaser, Odin Hoff Gardaa, Lars M. Salbu, Elena Xinyi Wang, Bastian Rieck

llms nlp benchmarks safety

arXiv AI

Jun 10

Representational Alignment with Chemical Induced Fit for Molecular Relational Learning

arXiv:2502. 07027v4 Announce Type: replace-cross Abstract: Molecular Relational Learning (MRL) is widely applied in natural sciences to predict relationships between molecular pairs by extracting structural features.

By Peiliang Zhang, Jingling Yuan, Qing Xie, Yongjun Zhu, Chao Che, Lin Li

rag benchmarks safety

arXiv Machine Learning

Jun 10

PADD: Path-Aligned Decompression Distillation for Non-Router Teacher to Guide MoE Student Learning

arXiv:2606. 10369v1 Announce Type: cross Abstract: As large language models (LLMs) continue to scale, it becomes increasingly challenging to grow model capacity under fixed computation budgets.

By Xinyue Peng, Yi Qian, Jiaojiao Lin, Wenjian Shao, Yanming Liu

llms efficiency benchmarks

arXiv AI

Jun 10

ActiveMem: Distributed Active Memory for Long-Horizon LLM Reasoning

arXiv:2606. 10532v1 Announce Type: new Abstract: Memory is essential for enabling large language model (LLM) agents to handle long-horizon reasoning tasks.

By Yunhan Jiang, Wenbin Duan, Shasha Guo, Liang Pang, Xiaoqian Sun, Huawei Shen

llms agents efficiency benchmarks

arXiv AI

Jun 10

HIPIF: Hierarchical Planning and Information Folding for Long-Horizon LLM Agent Learning

arXiv:2606. 10507v1 Announce Type: new Abstract: While Large Language Models (LLMs) have demonstrated strong capabilities as autonomous agents across a wide range of tasks, their performance often degrades in multi-turn long-horizon agentic tasks.

By Juncheng Diao, Zhicong Lu, Peiguang Li, Yongwei Zhou, Changyuan Tian, Qingbin Li, Rongxiang Weng, Jingang Wang, Xunliang Cai

llms agents nlp reinforcement-learning benchmarks

arXiv AI

Jun 10

Less Context, More Accuracy: A Bi-Temporal Memory Engine for LLM Agents Where a Lean Retrieved Context Beats the Full History

arXiv:2606. 09900v1 Announce Type: cross Abstract: Long-term memory is the missing layer for LLM agents: across sessions they forget, and the common workaround -- replaying the whole history into the prompt -- is expensive, slow, and, as distractors accumulate, less accurate.

By Liuyin Wang

llms agents benchmarks

Get the daily briefing