Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv AI

Oct 27

RoboReason: Boosting Robotics-Relevant VLM Reasoning Through Scalable Data Generation

arXiv:2610.33778v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have demonstrated impressive performance when integrated into robot manipulation pipelines, but real-world evaluation o...

By Sagnik Anupam, Utkarsh Lal, Aneesh Edara, Botong Zhang, Yao Tang, Jiatao Gu, Dinesh Jayaraman, Osbert Bastani

llms agents robotics fine-tuning multimodal benchmarks

arXiv AI

Oct 27

Recovering Business Rules as Cascade Lookups for Enterprise Field Prediction

arXiv:2610.34258v1 Announce Type: cross Abstract: Enterprise sales-order systems rely on manually maintained configuration rules to determine fields such as payment terms, shipping points, and sales...

By Changxu Duan

llms agents benchmarks data-engineering

arXiv AI

Oct 27

CodeMeepo: A Multi-Agent Workflow for Repository-Level Code Migration and Optimization

arXiv:2610.34633v1 Announce Type: cross Abstract: Repository-level code migration rewrites an entire project in a new programming language while preserving its observable behavior. It is challenging...

By Farzad Habibi, Changrong Wu, Jinzhou Jiang, Guohan Lu

llms agents benchmarks

arXiv AI

Oct 27

Benchmarking Obstacle-Conditioned Safety in Household Robot Navigation

arXiv:2610.34674v1 Announce Type: cross Abstract: Safe household navigation requires robots to regulate clearance and speed according to what an obstacle is and how close it is, without unnecessary c...

By Hyunwoong Bae, Jisu Han, Jiyeon Choi, Dohoon Kim, Taesup Moon

robotics multimodal benchmarks safety

arXiv AI

Oct 27

EEGaze: Multimodal Attention Prediction in Visual Search via EEG-Based Saliency Refinement

arXiv:2610.35757v1 Announce Type: cross Abstract: Visual search is ubiquitous in daily life, from locating keys in a messy apartment to finding a tool in a cluttered workshop or an application icon o...

By Mansi Sharma, Dinar Orazgaliyev, Antonio Kr\"uger, Philipp M\"uller

multimodal benchmarks

arXiv AI

Oct 27

Cross-Domain Sequential Recommendation with Representation-Level User Transfer

arXiv:2610.35550v1 Announce Type: cross Abstract: Cross-domain sequential recommendation aims to improve next-item prediction in a target domain by leveraging user behavior from auxiliary source doma...

By Abdulaziz Samra, Nikita Sukhorukov, Alexey Grishanov, Alexey Vasilev, Evgeny Frolov

rag benchmarks

arXiv AI

Oct 27

Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling

arXiv:2604.28075v3 Announce Type: replace-cross Abstract: Recent research has shown that filtering massive English web corpora into high-quality subsets significantly improves training efficiency. Ho...

By Ansar Aynetdinov, Patrick Haller, Alan Akbik

llms benchmarks

arXiv AI

Oct 27

Web Agents Should Adopt the Compile-and-Execute Paradigm

arXiv:2605.14290v2 Announce Type: replace-cross Abstract: ReAct has become the default architecture across LLM agents, and many existing web agents follow this paradigm. We argue that it is the wrong...

By Julien Piet, Annabella Chow, Yiwei Hou, Muxi Lyu, Sylvie Venuto, Jinhao Zhu, Raluca Ada Popa, David Wagner

llms agents benchmarks

arXiv AI

Oct 27

StableHand: Quality-Aware Flow Matching for World-Space Dual-Hand Motion Estimation from Egocentric Video

arXiv:2605.18553v2 Announce Type: replace-cross Abstract: Recovering world space 4D motion of two interacting hands from egocentric video is a fundamental capability for supervising robot policy lear...

By Huajian Zeng, Chaohua Yao, Yuantai Zhang, Jiaqi Yang, Rolandos Alexandros Potamias, Xingxing Zuo

diffusion robotics benchmarks

arXiv AI

Oct 27

Prompt PerturbAItions: Mechanisms of LLM Response Inconsistencies

arXiv:2605.30913v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly deployed in conversational settings where user phrasing ranges from polite to casual to toxic,...

By Soorya Ram Shimgekar, Agam Goyal, Amruta Parulekar, Joshua Chen, Yian Wang, Navin Kumar, Hari Sundaram, Eshwar Chandrasekharan, Koustuv Saha

llms benchmarks safety

arXiv AI

Oct 27

What the Abstention Option Measures: Refusal Loss, Sampling Churn, and Option Capture in Distilled Small Language Models

arXiv:2607.28639v3 Announce Type: replace-cross Abstract: Small instruction-tuned language models reach phones, cars, and offline assistants through knowledge distillation, and whether a distilled mo...

By Plawan Kumar Rath

llms fine-tuning efficiency benchmarks safety

arXiv AI

Oct 27

Beyond LLM-GA: Secure Fluid Antenna Systems with ReEvo-Designed Memetic Algorithm

arXiv:2610.10235v2 Announce Type: replace-cross Abstract: Fluid antenna systems (FASs) offer significant spatial flexibility, yet securing them against eavesdropping is critical for practical FAS dep...

By Hanyong Xu, Zhaolai Dang, Tong Zhang

llms benchmarks

arXiv Machine Learning

Oct 27

A Tutorial on Physics Constrained Machine Learning for Hybrid Modeling, Optimization, and Control

arXiv:2610.32914v1 Announce Type: new Abstract: Physics-constrained machine learning (PCML) integrates domain knowledge, first-principles, and physical constraints into AI/ML models. In this work, we...

By Rahul Golder, Bimol Nath Roy, MM Faruque Hasan

benchmarks

arXiv Machine Learning

Oct 27

Probing the limits of forward screening and inverse design of crystal materials

arXiv:2610.32678v1 Announce Type: cross Abstract: Crystal materials are designed either by screening known crystals for a target property or by generating new crystals with that property. Both routes...

By Zhilong Song, Zijun Pang, Lixue Cheng

llms agents benchmarks

arXiv Machine Learning

Oct 27

Heterogeneous Agent Collaborative Reinforcement Learning

arXiv:2603.02604v3 Announce Type: replace Abstract: We introduce Heterogeneous Agent Collaborative Reinforcement Learning (HACRL), a new Reinforcement Learning from Verifiable Reward (RLVR) problem t...

By Zhixia Zhang, Zixuan Huang, Gongxun Li, Huaiyang Wang, Chengyi Yuan, Xin Xia, Deqing Wang, Fuzhen Zhuang, Shuai Ma, Ning Ding, Yaodong Yang, Jianxin Li, Yikun Ban

llms agents reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Oct 27

Residual-Based Adaptive Refinement in Hybrid Variational-Collocation PINNs for Singular Elliptic PDEs with Mixed Dirichlet-Neumann Boundary Conditions

arXiv:2610.26458v2 Announce Type: replace-cross Abstract: Solutions of second-order elliptic PDE problems on nonconvex or cracked domains with mixed Dirichlet-Neumann conditions typically lie outside...

By Somnath Giri, Muslim Malik, Rishikesh Yadav

diffusion benchmarks

arXiv Computation and Language

Oct 27

Two Uncertainties Are Not One: Disagreement versus Decision-Risk in Cheap Structured Content Moderation

arXiv:2610.33258v1 Announce Type: new Abstract: Content moderation compresses a graded, contested human judgment into a binary act (leave up / take down) over billions of items, at a scale where runn...

By Karan Yadav

llms benchmarks safety

arXiv Computation and Language

Oct 27

Small Edits, Shifting Decisions: Benchmarking LLM Accuracy and Reliability on French Input Variants

arXiv:2610.33493v1 Announce Type: new Abstract: French users may omit apostrophes and diacritics or make keyboard typos in everyday input. The effects of these common variations on language model per...

By Jiaming Yu, Xiaotian Xie, Jian Ma, Yingjie Li

llms benchmarks

arXiv Computation and Language

Oct 27

Trajectory2Gym: Decision-Grounded Compositional Interaction Synthesis for Tool-Use Agents

arXiv:2610.33568v1 Announce Type: new Abstract: Successful tool-use trajectories are not necessarily useful supervision. Even successful interactions may underrepresent multi-step dependencies or con...

By Yihao Hu, Zhihao Wen, Yitong Lin, Siyuan Lu, Pan Wang, Wei Wu, Jiang-Ming Yang, Tao Lin

agents fine-tuning benchmarks

arXiv Computation and Language

Oct 27

Memory Tax: Dissecting the Net Value of Agent Memory

arXiv:2610.34073v1 Announce Type: new Abstract: Memory is increasingly used in long-horizon agents, yet the net value of building memory over direct retrieval from raw history remains largely unclear...

By Xuan He, Sining Zhoubian, Ruozhen Yang, Yuhao Wen, Zhineng Wang, Yuyang Hu, Ke Yang, Yuyu Zhang, Ping Nie, Jiawei Han, ChengXiang Zhai

rag agents benchmarks

Get the daily briefing