Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,906 stories · RSS feed

arXiv AI

Jul 14

SCALECUA: Scaling Computer Use Agents with Verifiable Task Synthesis and Efficient Online RL

arXiv:2607. 11185v1 Announce Type: new Abstract: Computer use agents (CUAs) are emerging as a powerful interface for automating complex digital workflows through visual perception and GUI execution.

By Bowen Lv, Xiao Liu, Yanyu Ren, Hanyu Lai, Bohao Jing, Hanchen Zhang, Yanxiao Zhao, Shuntian Yao, Jie Tang, Yuxiao Dong

agents computer-vision reinforcement-learning benchmarks

arXiv AI

Jul 14

UNIT: Unleash Large Language Models Potential for Graph Continual Learning

arXiv:2607. 10159v1 Announce Type: new Abstract: In real-world multimodal web scenarios, graph-structured data often arrives in a streaming manner, making graph continual learning a crucial paradigm for continuously modeling such evolving structures.

By Tairan Huang, Yili Wang, Beibei Hu, Yiting Shi, Qiutong Li, Changlong He, Jianliang Gao

llms fine-tuning multimodal benchmarks

arXiv AI

Jul 14

Looped State-Space Language Models with Adaptive Exit-State Selection

arXiv:2607. 10110v1 Announce Type: new Abstract: Recent work on looped language models suggests that many reasoning problems benefit from greater computational depth rather than from additional independent parameters.

By Zhenxuan Yu, Takeshi Kojima, Yutaka Matsuo, Yusuke Iwasawa

llms robotics benchmarks

arXiv AI

Jul 14

RepTran: Search-Based Repair of Transformer Models

arXiv:2607. 11193v1 Announce Type: cross Abstract: To ensure the overall quality of AI-enabled software, not only traditional software components but also AI components need to be tested and repaired.

By Yuta Ishimoto, Paolo Arcaini, Fuyuki Ishikawa, Masanari Kondo, Naoyasu Ubayashi, Yasutaka Kamei

llms benchmarks

arXiv AI

Jul 14

The evolution of AI from image interpretation toward scientific inference in nanoparticle electron microscopy

arXiv:2607. 10388v1 Announce Type: cross Abstract: Artificial intelligence (AI) is transforming electron microscopy by enabling quantitative analysis of increasingly large and complex datasets for nanoparticle characterization.

By Evropi Toulkeridou, Jiafei Li, Leonardo Lari, Panagiotis Grammatikopoulos

llms agents computer-vision multimodal benchmarks

arXiv AI

Jul 14

AgentAbstain: Do LLM Agents Know When Not to Act?

arXiv:2607. 10059v1 Announce Type: new Abstract: Agent systems based on large language models (LLMs) are increasingly deployed for autonomous tasks, yet existing evaluations mostly focus on task success rather than whether agents know when to abstain.

By Xun Liu, Yi Evie Zhang, Vira Kasprova, Parisa Rabbani, Pardis Sadat Zahraei, Tianyu Zhang, Ali Ebrahimpour-Boroojeny, Varun Chandrasekaran

llms agents benchmarks

arXiv Machine Learning

Jul 14

Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation

arXiv:2607. 10805v1 Announce Type: cross Abstract: On-Policy Self-Distillation (OPSD) has emerged as a crucial paradigm for enhancing and aligning Large Language Models (LLMs).

By Keqin Peng, Chen Li, Yuanxin Ouyang, Yancheng Yuan, Liang Ding

llms efficiency benchmarks

arXiv AI

Jul 14

Lifelong Representations: A Survey on Continual Self-Supervised Learning for Vision Models

arXiv:2607. 09785v1 Announce Type: cross Abstract: Traditionally, continual learning has assumed access to labeled data, yet many real-world applications -- such as lifelong robotics -- require models to adapt continuously from unlabeled streams.

By Sergi Masip, Alicja Dobrzeniecka, Jonathan Swinnen, Joachim Collin, Bart{\l}omiej Twardowski, Szymon {\L}ukasik, Tinne Tuytelaars

robotics efficiency multimodal benchmarks

arXiv AI

Jul 14

What Context Does a Coding Agent Actually Need to Act?

arXiv:2607. 09691v1 Announce Type: cross Abstract: A modern coding agent can hold an entire repository in its context window.

By Brian Sam-Bodden

llms agents benchmarks

arXiv AI

Jul 14

TopoExplore: Topological Discrimination for Archive-Based Exploration

arXiv:2607. 09971v1 Announce Type: new Abstract: Archive-based exploration methods such as Go-Explore select which visited state to return to using visitation rarity, and frontier methods return to the boundary of the unknown; neither asks whether the unexplored region behind a boundary is enterable at all.

By Jason Carlson

benchmarks

arXiv AI

Jul 14

Auditing Belief-Conditioned LLM Agents in Hidden-Information Social Deduction Games

arXiv:2607. 10814v1 Announce Type: cross Abstract: Evaluating LLM agents in hidden-information multi-agent settings is hard: final outcomes are high-variance and rarely reveal why an agent decided as it did.

By Yuan Gao, Jiangyi Yang, Yao Zhao, Yichi Zhang

llms agents benchmarks

arXiv AI

Jul 14

3D-DefectBench: A Controlled Factorial Study of Vision-Language Model Evaluation Pipelines for Fine-Grained 3D Generation Defects

arXiv:2607. 10826v1 Announce Type: cross Abstract: Automated evaluation is essential for scaling generative 3D systems, where exhaustive human review is costly and slow.

By Zhenyu Zhao, Nanshan Jia, Jihyeon Je, Yifu Tang, Alvin Chan, Michael Spedden, Michael V. Palleschi, Sui Huang, Jingshen Wang, Zeyu Zheng

llms multimodal benchmarks

arXiv AI

Jul 14

Measure the Sim-to-Real Gap: Designing an Affordable Real-World Benchmark Platform for Reinforcement Learning in AIoT Systems

arXiv:2607. 10309v1 Announce Type: new Abstract: Reinforcement learning (RL) is commonly employed to enhance the performance of autonomous systems, including the Autonomous Internet of Things (AIoT).

By Rongping Zhou, Omid Tavallaie, Shuaijun Chen, Albert Y. Zomaya

agents reinforcement-learning robotics benchmarks safety

arXiv AI

Jul 14

Valid $\ne$ Necessary: Diagnosing Latent Inefficiency in Chain-of-Thought

arXiv:2607. 11266v1 Announce Type: new Abstract: Chain-of-Thought (CoT) prompting has significantly advanced the reasoning capabilities of Large Language Models (LLMs), yet it often incurs substantial computational costs due to over-reasoning: the generation of redundant, verbose, or irrelevant steps.

By Daeyeop Lee, Hwanjo Yu

llms efficiency benchmarks

arXiv AI

Jul 14

Turbo Connection: Reasoning as Information Flow from Higher to Lower Layers

arXiv:2602. 17993v2 Announce Type: replace-cross Abstract: Complex problems, whether in math, logic, or planning, are solved by humans through a sequence of steps where the result of one step informs the next.

By Mohan Tang, Sidi Lu

llms fine-tuning benchmarks

arXiv AI

Jul 14

STEC: Evidence Compression for Deep Search in Open-domain Multi-Hop QA

arXiv:2607. 10795v1 Announce Type: new Abstract: In open-domain multi-hop question answering (QA), LLM-based search agents offer a promising approach to knowledge-intensive QA by combining retrieval with reasoning.

By Xinkang Li, Rong Jiang, Xin Song, Ye Wang, Yue Han, Changjian Li

llms agents nlp benchmarks

arXiv AI

Jul 14

Debiasing Central Fixation Confounds Reveals a Peripheral "Sweet Spot" for Human-like Scanpaths in Hard-Attention Vision

arXiv:2602. 14834v2 Announce Type: replace-cross Abstract: Human eye movements in visual recognition reflect a balance between foveal sampling and peripheral context.

By Pengcheng Pan, Yonekura Shogo, Yasuo Kuniyosh

benchmarks safety

arXiv AI

Jul 14

Disentangling Intrinsic Importance from Emergent Structure in Multi-Expert Orchestration

arXiv:2602. 04291v3 Announce Type: replace-cross Abstract: Multi-expert systems, where multiple Large Language Models (LLMs) collaborate to solve complex tasks, are increasingly adopted for high-performance reasoning and generation.

By Sudipto Ghosh, Sujoy Nath, Sunny Manchanda, Tanmoy Chakraborty

llms agents benchmarks safety

arXiv AI

Jul 14

Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models

arXiv:2601. 07372v2 Announce Type: replace-cross Abstract: While Mixture-of-Experts (MoE) scales capacity via conditional computation, Transformers lack a native primitive for knowledge lookup, forcing them to inefficiently simulate retrieval through computation.

By Xin Cheng, Rui Tian, Wangding Zeng, Damai Dai, Qinyu Chen, Bingxuan Wang, Zhenda Xie, Kezhao Huang, Xingkai Yu, Chengqi Deng, Shangyan Zhou, Chenggang Zhao, Zhewen Hao, Yukun Li, Han Zhang, Zhengyan Zhang, Yixu Wei, M. Y Xu, Huishuai Zhang, Dongyan Zhao, Wenfeng Liang

llms rag efficiency benchmarks

arXiv AI

Jul 14

Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting

arXiv:2607. 10661v1 Announce Type: cross Abstract: Speculative decoding has significantly accelerated Large Language Model (LLM) inference by alleviating memory-bound bottlenecks.

By Zipeng Gao, Zhi Zheng, Qingrong Xia, Junda Lin, Ziwei Zhao, Tong Xu, Zhefeng Wang, Enhong Chen

llms efficiency benchmarks

Get the daily briefing