Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv AI

Oct 15

PhysCodeGym: Benchmarking Coding Agents for Physically Verifiable 4D Reconstruction

arXiv:2610.17288v1 Announce Type: new Abstract: Building physical simulations from video requires coding agents to translate observations into executable models and refine them through tool use. We i...

By Shuai Shao, Jinchen He, Cunxin Fan, Zihan Guo, Shijian Wang, Qihan Ren, Kangning Zhang, Zunnan Xu, Haoyuan Li, Hao Wang, Jianghao Lin, Weiwen Liu, Weinan Zhang

agents benchmarks

arXiv AI

Oct 15

AM-Bench: Understanding Agents under Task Underspecification across Models and Harnesses

arXiv:2610.18090v1 Announce Type: new Abstract: Ambiguous or incomplete task specifications complicate both execution and clarification for language-model agents. Yet agents' responses to these infor...

By Kaiyu Huang, Yiting Li, Angyang Gu, Pinjia He, Minchen Yu, Qingjiang Shi

agents benchmarks

arXiv AI

Oct 15

QORH: Learning Query-Specific Ordered Rule Hypergraphs for Inductive Knowledge Graph Link Prediction

arXiv:2610.18198v1 Announce Type: new Abstract: Knowledge graph link prediction involving unseen entities and relations requires learning transferable reasoning patterns from observed graph structure...

By Naixing Xu

benchmarks

arXiv AI

Oct 15

AesCode: Aesthetic Code Generation with Decoupled Cross-Modal Rewards

arXiv:2610.18860v1 Announce Type: new Abstract: Generating information-rich visual artifacts such as slides, posters, and dashboards as code is appealing because the outputs are structured, editable,...

By Xin Gao, Yifan Yang, Xin Li, Pan Wang, Yan Li, Muzhao Tian, Mingxi Cheng, Bei Liu, Kai Qiu, Qi Dai, Dong Chen, Yue Dong, Leandros Tassiulas, Rex Ying, Ji Li, Chong Luo

llms reinforcement-learning multimodal benchmarks

arXiv AI

Oct 15

PreSO: Privileged Representation Isolation for On-Policy Self-Distillation

arXiv:2610.14001v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD) uses a self-teacher conditioned on privileged information to turn the sparse outcome signal of reinforcement learn...

By Yuhang Yang, Mingxuan Xia, Chao Ye, Shuai Zhu, Shouqing Yang, Haobo Wang, Shenzhi Yang, Zhengqing Zang, Chenglong Wang

reinforcement-learning efficiency benchmarks

arXiv AI

Oct 15

UDHRBench: Benchmarking Human Rights in Large Language Models

arXiv:2610.15179v1 Announce Type: cross Abstract: Large language models are increasingly deployed in contexts requiring adherence to ethical and legal norms, yet their alignment with internationally...

By Alak Raad (Hamad Bin Khalifa University, Qatar), Noreldin Morssy (Hamad Bin Khalifa University, Qatar), Zeerak Suhail (Hamad Bin Khalifa University, Qatar), Serin Atiani (Princess Sumaya University for Technology, Jordan), Mustafa Jarrar (Hamad Bin Khalifa University, Qatar)

llms benchmarks safety

arXiv AI

Oct 15

Explicitly Aligned Multi-View Breast Cancer Risk Prediction from Variable-Length Screening Histories

arXiv:2610.16329v1 Announce Type: cross Abstract: Accurate breast cancer risk prediction from screening mammography is essential for improved risk stratification and personalized screening. It can be...

By Solveig Thrun, Zijun Sun, Suaiba A. Salahuddin, Kristoffer Wickstr{\o}m, Elisabeth Wetzer, Stine Hansen, Robert Jenssen, Michael Kampffmeyer

benchmarks safety

arXiv AI

Oct 15

A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge

arXiv:2610.18108v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as judges to predict which of two candidate answers a human evaluator would prefer, providing a sc...

By Cesare Barbera, Luca Stradiotti, Lorenzo Cascioli, Lorenzo Perini, Niek Tax, Daniel Haimovich, Jesse Davis, Andrea Passerini, Andrea Pugnana

llms benchmarks safety

arXiv AI

Oct 15

Lightweight but Linguistically Fair? Quantisation Robustness across Bavarian, German, and English

arXiv:2610.18343v1 Announce Type: cross Abstract: Post-training quantisation reduces the memory cost of large language models, but it remains unclear whether compression preserves language skills of...

By Jophin John, Michael Hoffmann, Piroska Lendvai, Florent Dufour

llms benchmarks

arXiv AI

Oct 15

LIFT: Likelihood-Based Intervention Faithfulness Training

arXiv:2509.13334v2 Announce Type: replace Abstract: Chain-of-thought (CoT) reasoning improves LLM accuracy on complex tasks, but CoTs are often unfaithful; that is, the model does not rely on the sta...

By Anand Swaroop, Maheep Chaudhary, Kevin Zhu, Sunishchal Dev, Vasu Sharma

llms fine-tuning benchmarks

arXiv AI

Oct 15

TADDLE: A Tool-Augmented Agent for Detecting Deficient LLM-Generated Peer Reviews

arXiv:2605.26911v2 Announce Type: replace Abstract: LLM-generated peer reviews are increasingly common at major venues, yet their deficiencies are hard to detect because they are uniformly fluent and...

By Hanqi Duan, Xiang Li

llms agents benchmarks

arXiv AI

Oct 15

SearchMaster: Grounded and Regulated Self-Play for Search Agents

arXiv:2608.01822v2 Announce Type: replace Abstract: Training LLM-based search agents requires high-quality search data: tasks that demand genuine multi-hop retrieval and trajectories that use search...

By Wentao Tan, Qiong Cao, Jiaqi Wang, Nan Duan

llms agents reinforcement-learning benchmarks

arXiv AI

Oct 15

Recursive Improvement of a Differentiable Scientific Software Ecosystem

arXiv:2610.04561v2 Announce Type: replace Abstract: Differentiable programming connects scientific computation with gradient-based inference, learning and design. Extending these capabilities across...

By Pengcheng Hou, Xiaojun Tan, Sihan Hu, Ruisi Wang, Shuo Chen, Lei Wang, Youjin Deng, Kun Chen

agents benchmarks

arXiv AI

Oct 15

A Generalist Hanabi Agent

arXiv:2503.14555v2 Announce Type: replace-cross Abstract: Traditional multi-agent reinforcement learning (MARL) systems can develop cooperative strategies through repeated interactions. However, thes...

By Arjun V Sudhakar, Hadi Nekoei, Mathieu Reymond, Miao Liu, Janarthanan Rajendran, Sarath Chandar

agents reinforcement-learning benchmarks

arXiv AI

Oct 15

JAL-Turn: Joint Acoustic-Linguistic Modeling for Real-Time and Robust Turn-Taking Detection in Full-Duplex Spoken Dialogue Systems

arXiv:2603.26515v2 Announce Type: replace-cross Abstract: Efficient and robust turn-taking detection remains a significant challenge in industrial-grade voice AI agents. Existing systems often rely o...

By Yu Pan, Guangzhao Yang, Muhammad Huzaifah, Ningjie Bai, Shi Qiu

llms agents multimodal benchmarks

arXiv AI

Oct 15

UniT: Toward a Unified Physical Language for Human-to-Humanoid Policy Learning and World Modeling

arXiv:2604.19734v2 Announce Type: replace-cross Abstract: Scaling humanoid foundation models is bottlenecked by the scarcity of robotic data. While massive egocentric human data offers a scalable alt...

By Boyu Chen, Yi Chen, Lu Qiu, Jerry Bai, Yuying Ge, Yixiao Ge

diffusion nlp robotics benchmarks safety

arXiv AI

Oct 15

Hiding in Plain Sight: Detectability-Aware Antidistillation of Reasoning Models

arXiv:2604.23238v3 Announce Type: replace-cross Abstract: Distillation via reasoning traces exposes frontier models to adversarial third parties, who can bypass their guardrails and misappropriate th...

By Max Hartman, Vidhata Jayaraman, Moulik Choraria, Lav R. Varshney

llms efficiency benchmarks safety

arXiv Machine Learning

Oct 15

Do Vision CIL Methods Transfer to Protein? A Diagnostic Benchmark of Frozen Protein Encoders

arXiv:2610.18879v1 Announce Type: new Abstract: Protein function annotation operates over an expanding label space, making class-incremental learning (CIL) a natural framework for incorporating newly...

By Ziyuan Gao

rag benchmarks

arXiv Machine Learning

Oct 15

Divergent Paths: Learning an Outcome-Predictive Representation of LLM Agent Trajectories

arXiv:2610.19614v1 Announce Type: new Abstract: In this work, we reframe agentic trajectories as paths through a shared embedding space of actions and environmental observations. We show that success...

By Alexander Gill, Guram Gogia, Francisco Guzm\'an

llms rag agents benchmarks

arXiv Machine Learning

Oct 15

CTIA-ODE: Continuous-Time Inter-Variable Attention for Long-Term Time Series Forecasting

arXiv:2610.19831v1 Announce Type: new Abstract: Although multivariate time series forecasting has broad application prospects, effectively incorporating exogenous variables remains challenging. Preva...

By Chengyi Pan, Huangqimei Zheng, Hao Xue

efficiency benchmarks safety

Get the daily briefing