Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv Machine Learning

Oct 15

Diagnosing and Correcting Reward Hacking in Verifiable Instruction Following

arXiv:2610.19525v1 Announce Type: new Abstract: Verifiable instruction-following benchmarks emphasize output-constraint satisfaction but can overlook whether responses meaningfully answer the underly...

By Sangyeon Yoon, Wooseok Seo, Hamish Ivison, Hyesoo Hong, Donghoon Kim, Wonjun Kang, Albert No, Jaehyung Kim

reinforcement-learning benchmarks

arXiv AI

Oct 15

Memory Echo in Single-Agent Long-Term Memory: Lifecycle Evaluation and Provenance-Guided Mitigation

arXiv:2610.17763v1 Announce Type: new Abstract: Long-term memory supports agents across sessions. Repeated retrieval and write-back can let one historical claim occupy multiple retrieval positions an...

By Yufeng Li, Sheng Yao, Peng Zeng, Dezhang Kong, Shi Lin, Changting Lin, Xun Wang, Meng Han

agents nlp benchmarks

arXiv AI

Oct 15

Learning How to Remember: Memory Modality Selection for GUI Agents

arXiv:2610.17928v1 Announce Type: new Abstract: Recent advances in vision-language models (VLMs) have substantially improved the capabilities of GUI agents, while memory augmentation has emerged as a...

By Taeyun Roh, Suhyeong Park, Gijin Kim, Wonjune Jang, Jaewoo Kang

llms agents multimodal benchmarks

arXiv AI

Oct 15

Can Agents Forget When Asked? A Benchmark for Memory Unlearning in LLM Agents

arXiv:2610.18052v1 Announce Type: new Abstract: Long-term memory has become a core component of large language model (LLM) agents. As agents accumulate personal information over long-term use, users...

By Bofan Xie, Chaokun Wang, Guanchen Ge, Songyao Wang, Fang Niu, Hang Zhang, Wei Wang

llms agents benchmarks

arXiv Machine Learning

Oct 15

InstructPro2: Natural Language and Biological Grounding Guided De Novo Protein Design

arXiv:2610.19592v1 Announce Type: new Abstract: The ability to generate proteins with desired functions is a central goal of computational biology. Although natural language can express diverse desig...

By Charles Novak, Zhenqiao Song, Chuan Li, Jianwen Xie, Lei Li

benchmarks

arXiv Machine Learning

Oct 15

Deep Generative Models for Phylogenetic Inference with Complex Evolutionary Processes

arXiv:2610.19053v1 Announce Type: new Abstract: Inference of phylogenetic trees from sequences plays a central role in understanding evolutionary relationships, with applications ranging from trackin...

By Ethan Baron, Alan N. Amin, Andrew Gordon Wilson

diffusion benchmarks

arXiv Machine Learning

Oct 15

When Does Depth Help? Understanding Layerwise Inference in Tabular Foundation Models

arXiv:2610.19814v1 Announce Type: new Abstract: Transformer-based tabular foundation models (TFMs) achieve state-of-the-art performance on small to medium-sized tabular prediction tasks while process...

By Wenwen Gong, Haotian Gu, Zhen Yang, Jiaying Ma, Hanqing Zhang, Zhongshen Wang, Jingjun Wang, Shouyi Xu, Wei Wang

llms benchmarks

arXiv AI

Oct 15

Label-Centric Knowledge Retrieval for Zero-Shot Text Classification

arXiv:2610.16182v1 Announce Type: new Abstract: Flagship large language models (LLMs) offer rich general knowledge and strong reasoning capabilities, but their high inference cost and latency limit t...

By Daoping Wu, Dong Wang, Siyu Liu, Chen Liang, Jian Geng, Ziqi Li, Shuguang Xiong, Ying Zhang

llms rag nlp benchmarks

arXiv AI

Oct 15

Correct Choices, Failed Sequences: Choice2Seq Benchmarks Correctness Retention Across Output Interfaces

arXiv:2610.16191v1 Announce Type: new Abstract: A model may select the correct answer yet fail to construct it through a sequence of local decisions. We introduce Choice2Seq, a diagnostic benchmark t...

By Tianyou Lai, Wentao Yue, Zhiying Deng, Qilei Li

benchmarks

arXiv AI

Oct 15

ABot-ER: Unified Embodied Reasoning through Agentic Data-Model Co-Evolution

arXiv:2610.18355v2 Announce Type: new Abstract: We introduce ABot Embodied Reasoning (ABot-ER), a unified embodied reasoning foundation model that breaks the tension between breadth and depth in embo...

By Qiuyu Ding, Heng-Da Xu, Changda Xia, Jiajun Shen, Zihao Guan, Zhengchen Yuan, Jianfang Zhang, Di Yang, Jiayi Tian, Yuting Xu, Fei Wang, Shiao Liu, Wei Zhang, Xiaolong Wu, Wenbin Tang, Zedong Chu, Mu Xu

agents robotics benchmarks

arXiv AI

Oct 15

Choosing LLMs for Strategic Agent Interactions

arXiv:2610.18354v1 Announce Type: new Abstract: Game-theoretic benchmarks claim that large language model (LLM) strategic decision-making varies across game types, interaction structures, and capabil...

By Arastun Mammadli, Herbert Woisetschl\"ager, Jawad Fayaz, Shiqiang Wang

llms rag agents reinforcement-learning benchmarks safety

arXiv AI

Oct 15

Reading Right, Acting Wrong: Diagnosing Decision Failures of System One Models in Agent Loops

arXiv:2610.18579v1 Announce Type: new Abstract: System One models choose an agent's next action by scoring a supplied set of options in a single pass. This makes them fast and cheap, but when they ch...

By Yubo Li, Ramayya Krishnan, Rema Padman

agents benchmarks

arXiv AI

Oct 15

Compare to Correct: Pairwise-Guided Logit Adjustment for LLM Test-Time Scaling

arXiv:2610.18583v1 Announce Type: new Abstract: Sampling-based test-time scaling enhances LLM reasoning by aggregating multiple reasoning paths for the same query. Existing methods use answer frequen...

By Cong Hua, Qianqian Xu, Zitai Wang, Zhiyong Yang, Shilong Bao, Linchao Zhang, Qingming Huang

llms benchmarks

arXiv AI

Oct 15

PixSearch: Region-Grounded Retrieval for Knowledgeable Large Multimodal Models

arXiv:2601.19060v2 Announce Type: replace-cross Abstract: Multimodal Retrieval-Augmented Generation (MM-RAG) augments visual question answering (VQA) with external knowledge, but many systems rely on...

By Jeonghwan Kim, Renjie Tao, Sanat Sharma, Jiaqi Wang, Kai Sun, Zhaojiang Lin, Seungwhan Moon, Lambert Mathias, Anuj Kumar, Heng Ji, Xin Luna Dong

rag computer-vision nlp fine-tuning efficiency multimodal benchmarks

arXiv Machine Learning

Oct 15

When Non-Adversarial Noise Hurts Learning: Benchmarking Skill Self-Evolution in Agents

arXiv:2610.19910v1 Announce Type: new Abstract: Large language model agents increasingly improve by distilling experience into reusable skills, but errors in that experience can persist in learned ru...

By Tao Yu, Zhongtian Luo, Yuchen Long, Yiheng Lu, Yuxuan Zhou, Yanpei Gong, Siyi Cao, Yuqing Wu, Haopeng Jin, Zhenghao Zhang, Yujia Yang, Jingmin Zhu, Yunyun Hou, Xinming Wang, Hongzhu Yi, Jiachen Liu, Sirui Wang, Nan Duan, Yan Huang, Liang Wang, Tianjin Huang, Yanyan Zou

llms agents benchmarks safety

arXiv Machine Learning

Oct 15

Beyond DFT electronic band structures via dual-space orbital learning

arXiv:2610.19919v1 Announce Type: new Abstract: Accurate electronic band structures are essential for predicting material properties, but high-fidelity calculations are computationally demanding. We...

By Minhyuk Kang, Beom Seok Kang, Nikolai Argatoff, Miguel Nouman, Danish Khan, William A. Goddard III, Anima Anandkumar

benchmarks

arXiv AI

Oct 15

Auditing LLM Program Search for Bermudan Stopping with Primal--Dual Certificates

arXiv:2610.14063v1 Announce Type: cross Abstract: LLM program-search agents are usually judged by the score that drives their search. We study LLM-driven search over Python exercise policies for Berm...

By Yuhua Qian

llms agents benchmarks

arXiv Machine Learning

Oct 15

Empirical Boundaries of Reservoir Computing for Few-Shot Sequence Pattern Tasks

arXiv:2610.19997v1 Announce Type: new Abstract: Efficient sequence modeling remains a fundamental challenge in machine learning. Despite the success of Transformers, recurrent neural networks (RNNs)...

By Mario Mallea, Angela Nebot, Franscisco Mugica

llms benchmarks

arXiv Machine Learning

Oct 15

GBTSVR: Twin Support Vector Regression via Granular Ball Computing

arXiv:2610.20053v1 Announce Type: new Abstract: Regression models used in real-world applications are often challenged by noise, outliers, data heterogeneity, and asymmetric error characteristics, wh...

By A. Quadir, M. Tanveer

benchmarks

arXiv AI

Oct 15

Two-Stage Multi-modal Learning for Dynamic Audio-Visual Navigation: From Physical Alignment to Scene Aggregation

arXiv:2610.14077v1 Announce Type: cross Abstract: Embodied Audio-Visual Navigation (AVN) requires an agent to locate sound targets by jointly leveraging visual and auditory cues. While prior work has...

By Jia Li, Yinfeng Yu

llms agents robotics multimodal benchmarks safety

Get the daily briefing