Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

17,067 stories · RSS feed

arXiv AI

Jul 8

RoME: Robust Mixture of Low-Rank Experts against Multiple Adversarial Perturbations

arXiv:2607. 06109v1 Announce Type: cross Abstract: Multi-perturbation adversarial training (MAT) aims to achieve robustness against multiple $\ell_p$ perturbations but suffers from robustness trade-offs between different threats.

By Woo Jae Kim, Kyle Min, Suhyeon Ha, Joonsung Jeon, Sung-eui Yoon

benchmarks safety

arXiv AI

Jul 8

Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agents

arXiv:2510. 19771v4 Announce Type: replace Abstract: LLM-based agents are increasingly moving towards proactivity: rather than awaiting instruction, they exercise agency to anticipate user needs and solve them autonomously.

By Gil Pasternak, Dheeraj Rajagopal, Julia White, Dhruv Atreja, Matthew Thomas, George Hurn-Maloney, Ash Lewis

llms agents benchmarks

arXiv Machine Learning

Jul 8

Low-Overhead Error-Corrected QCNNs Using Bivariate Bicycle Codes

arXiv:2607. 05724v1 Announce Type: new Abstract: Quantum convolutional neural networks (QCNNs) combine the power of quantum computing and classical CNN for computational speedup in classification tasks.

By Alejandro Rosales, Animesh Yadav

benchmarks

arXiv Machine Learning

Jul 8

Intuitionistic Fuzzy Graph Embedded Random Vector Functional Link with Multiview Learning

arXiv:2607. 05635v1 Announce Type: new Abstract: Random Vector Functional Link (RVFL) networks are popular due to their fast training and universal approximation capabilities.

By Vrushank Ahire, Yogesh Kumar, M. A. Ganaie

rag benchmarks

arXiv AI

Jul 8

PolyWorkBench: Benchmarking Multilingual Long-Horizon LLM Agents

arXiv:2607. 06008v1 Announce Type: new Abstract: Large language model (LLM) agents have shown strong performance in long-horizon tasks that require planning, tool use, and interaction with external environments.

By Hongliang Li, Yijin Liu, Zhiwei Zhang, Zihe Liu, Xinyue Lou, Jinan Xu, Fandong Meng, Kaiyu Huang

llms agents benchmarks

arXiv AI

Jul 8

Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges

arXiv:2605. 19723v2 Announce Type: replace-cross Abstract: Mathematical reasoning is essential for problem-solving in education, science, and industry, serving as a crucial benchmark for evaluating artificial intelligence systems.

By Husnain Amjad, Raja Khurram Shahzad, Aamir Shahzad, Mehwish Fatima

llms fine-tuning benchmarks

arXiv AI

Jul 8

FootsiesGym: A Fighting Game Benchmark for Two-Player Zero-Sum Imperfect-Information Games

arXiv:2607. 06514v1 Announce Type: new Abstract: We present FootsiesGym, an open-source environment for learning in a non-trivial two-player, zero-sum, imperfect-information game.

By Chase McDonald, Nathan Tsang, Wesley N. Kerr

reinforcement-learning efficiency benchmarks

arXiv AI

Jul 8

FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference

arXiv:2607. 06519v1 Announce Type: new Abstract: Long-context LLM inference is increasingly limited by the memory and bandwidth cost of KV caches, yet aggressive compression can remove the layer-specific evidence needed for retrieval and multi-step reasoning.

By Anna C\'ordoba, Adam Puente Tercero, Nerea Angulo Hijo, Mar Linares Tercero, Julia Barrientos, Ainhoa Miranda, Jes\'us Olivera

llms nlp efficiency benchmarks

arXiv AI

Jul 8

Self-Review Reinforcement Learning (SRRL) with Cross-Episode Memory and Policy Distillation

arXiv:2607. 05541v1 Announce Type: cross Abstract: Reinforcement Learning is commonly used to train large language models using environmental feedback.

By Muhammad Zain Amin, Kibele Sebnem Yildirim

llms reinforcement-learning efficiency benchmarks

arXiv AI

Jul 8

PCBWorld: A Benchmark Environment for Engine-Grounded PCB Design Automation

arXiv:2607. 05915v1 Announce Type: new Abstract: PCB routing is the task of connecting the nets of a board with copper traces under strict design rules, yet learning-based methods still lag behind rule-based routers.

By Hyungseok Song, Junseok Park, Won-Seok Choi, Seohui Bae, Han-Seul Jeong, Youngjoon Park, Soonyoung Lee

llms agents reinforcement-learning benchmarks

arXiv AI

Jul 8

Token-Based Dual-view Fusion and Adaptation of Large Vision Models for Breast Cancer Classification

arXiv:2607. 06309v1 Announce Type: cross Abstract: Accurate breast cancer classification from mammography requires effective integration of complementary information from craniocaudal (CC) and mediolateral oblique (MLO) views, which provide a more complete characterization of breast abnormalities.

By Aysan Ghayouri Pirsoltan, Shima Babakordi, Mohammad Reza Mohammadi

llms computer-vision benchmarks

arXiv AI

Jul 8

CANONIC: Governance Is Compilation

arXiv:2607. 05410v1 Announce Type: cross Abstract: We present CANONIC: governed intelligence that compiles digital artifacts into an evidence ledger at scale.

By Dexter Hadley

llms benchmarks

arXiv AI

Jul 8

IMR: Iterative Mode-World Weighted Regression for Multi-Agent Trajectory Prediction

arXiv:2607. 05705v1 Announce Type: cross Abstract: Multi-agent motion prediction is essential for automated vehicles to understand the intentions of surrounding vehicles.

By Honglin Wang, Shiyao Pan, Yun-Fu Liu

agents benchmarks safety

arXiv AI

Jul 8

Quantifying Frontier LLM Capabilities for Container Sandbox Escape

arXiv:2603. 02277v2 Announce Type: replace-cross Abstract: Large language models (LLMs) increasingly act as autonomous agents, using tools to execute code, read and write files, and access networks, creating novel security risks.

By Rahul Marchand, Art O Cathain, Jerome Wynne, Philippos Maximos Giavridis, Sam Deverett, John Wilkinson, Jason Gwartz, Harry Coppock

llms agents benchmarks safety

arXiv AI

Jul 8

HST-HGN: Heterogeneous Spatial-Temporal Hypergraph Networks with Bidirectional State Space Models for Global Fatigue Assessment

arXiv:2604. 08435v2 Announce Type: replace-cross Abstract: It remains challenging to assess driver fatigue from untrimmed videos under constrained computational budgets, due to the difficulty of modeling long-range temporal dependencies in subtle facial expressions.

By Changdao Chen, Qinqiuhong Ye, Hao Chen, Jinyu Wang

efficiency multimodal benchmarks

arXiv AI

Jul 8

From Passive Observer to Active Critic: Reinforcement Learning Elicits Process Reasoning for Robotic Manipulation

arXiv:2603. 15600v2 Announce Type: replace-cross Abstract: Accurate process supervision remains a critical challenge for long-horizon robotic manipulation.

By Yibin Liu, Yaxing Lyu, Daqi Gao, Zhixuan Liang, Weiliang Tang, Shilong Mu, Xiaokang Yang, Yao Mu

reinforcement-learning robotics fine-tuning benchmarks

arXiv AI

Jul 8

CMDR: Contextual Multimodal Document Retrieval

arXiv:2607. 05927v1 Announce Type: cross Abstract: Multimodal document retrieval aims to retrieve relevant pages while preserving both textual and visual content from the original document.

By Ryota Tanaka, Taku Hasegawa, Kyosuke Nishida

rag multimodal benchmarks

arXiv AI

Jul 8

Uncovering Latent Depression Severity for Binary Depression Detection via Advantage-weighting Ranking

arXiv:2607. 05901v1 Announce Type: new Abstract: Automatic depression detection using audio-visual data faces significant challenges, particularly in disentangling overlapping feature distributions and establishing robust decision boundaries.

By Manning Gao, Tingyi Liu, Leheng Zhang, Haifeng Hu, Yuncheng Jiang, Sijie Mai

llms diffusion multimodal benchmarks

arXiv AI

Jul 8

Multi-Task Instruction Tuning via Data Scheduling for Low-Resource Arabic SpeechLLMs

arXiv:2601. 12494v3 Announce Type: replace-cross Abstract: Audio large language models (LLMs) enable unified speech understanding and generation, but adapting them to linguistically complex and dialect-rich settings such as Arabic-English remains challenging.

By Hunzalah Hassan Bhatti, Firoj Alam, Shammur Absar Chowdhury

llms nlp fine-tuning multimodal benchmarks

arXiv AI

Jul 8

An Experimental Design Approach to Evaluating Agentic AI's Autonomous Model Discovery

arXiv:2607. 06413v1 Announce Type: cross Abstract: Large language model coding agents increasingly perform open-ended data modeling and analysis.

By Hao He, Xueying Liu, Chris J. Kuhlman, Xinwei Deng

llms agents benchmarks

Get the daily briefing