Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,859 stories · RSS feed

arXiv Machine Learning

Jun 16

MIRAGE: Auditing Anti-Muslim Bias in Frontier LLMs Across Reasoning, Agentic, and Time-Coupled Conditions

arXiv:2606. 16562v1 Announce Type: new Abstract: Five years after the discovery of persistent anti-Muslim bias in large language models, most evaluations remain confined to single-turn prompt completion, a setting that no longer reflects how frontier LLMs are deployed.

By Noor Islam S. Mohammad, Tamim Sheikh

llms agents nlp benchmarks safety

arXiv AI

Jun 16

Towards Verifiable Agentic Data Science: Solving Irregular TSQA Via Tool-Grounded Reasoning

arXiv:2606. 15107v1 Announce Type: new Abstract: Time series data in real-world deployments is overwhelmingly irregular.

By Sanhorn Chen, Xiaoyang Chen, Boyu Liu, Roy Zhao

llms agents nlp benchmarks

arXiv AI

Jun 16

Tool-IQA: Augmenting Image Quality Assessment with Simple Tools

arXiv:2606. 16082v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have been increasingly adopted for Image Quality Assessment (IQA).

By Guanyi Qin, Junjie Zhang, Chunming He, Yibing Fu, Jie Liang, Tianhe Wu, Lei Zhang

llms agents multimodal benchmarks

arXiv Machine Learning

Jun 16

Imbalanced Semi-Supervised Learning via Label Refinement and Threshold Adjustment

arXiv:2407. 05370v3 Announce Type: replace Abstract: Semi-supervised learning (SSL) algorithms often struggle to perform well when trained on imbalanced data.

By Zeju Li, Ying-Qiu Zheng, Chen Chen, Saad Jbabdi

benchmarks safety

arXiv Machine Learning

Jun 16

Robust Transformer-Based One-Step Stock Index Forecasting via Shifted Data Augmentation

arXiv:2606. 15701v1 Announce Type: new Abstract: Transformers have shown remarkable success in sequence modeling, yet their direct application to financial time series remains challenging due to noisy signals, short-memory dynamics, and distributional shifts.

By Tien Thanh Thach

llms benchmarks

arXiv Machine Learning

Jun 16

Replay What Matters: Off-Policy Replay for Efficient LLM Reinforcement Unlearning

arXiv:2606. 15333v1 Announce Type: cross Abstract: LLM unlearning has emerged as a cost-effective alternative to full retraining for removing hazardous knowledge from pretrained models while preserving general utility.

By Zirui Pang, Chenlong Zhang, Haosheng Tan, Zhuoran Jin, Jiaheng Wei, Zixin Zhong

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jun 16

EIBench: A Simulator-Based Benchmark and Turn-Credit RL for Emotion Management

arXiv:2606. 15532v1 Announce Type: cross Abstract: Emotional intelligence (EI) in Large Language Models (LLMs) is often evaluated through static understanding tasks or single-response dialogue generation.

By Rongzhi Zhu, Xiang Huang, Yuchuan Wu, Rui Wang, Zequn Sun, Tao Ren, Weiyao Luo, Bingxue Qiu, Jieping Ye, Yongbin Li, Wei Hu

llms reinforcement-learning benchmarks

arXiv AI

Jun 16

MiroBench: Benchmarking Realism in Agentic Simulation of Real-world Discussions

arXiv:2606. 14715v1 Announce Type: cross Abstract: LLM agents are increasingly used to simulate real world interactions, but it remains unclear whether simulated behaviors preserve the content patterns and interaction dynamics of real human behaviors.

By Yaoning Yu, Ye Yu, Haojing Luo, Haohan Wang

llms agents benchmarks safety

arXiv AI

Jun 16

OpenClaw-Skill: Collective Skill Tree Search for Agentic Large Language Models

arXiv:2606. 16774v1 Announce Type: new Abstract: Equipping Large Language Model (LLM) agents with effective skills is crucial for solving complex tasks in real-world systems like OpenClaw.

By Tianyi Lin, Chuanyu Sun, Jingyi Zhang, Changxu Wei, Huanjin Yao, Shunyu Liu, Xikun Zhang, Liu Liu, Jiaxing Huang

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Jun 16

Data-Centric Benchmarking of Exploit Generation in LLMs: Understanding the Impact of Fine-Tuning

arXiv:2606. 15123v1 Announce Type: cross Abstract: We study the task of CVE-conditioned exploit generation, where a model drafts proof-of-concept (PoC) exploits given software vulnerability context.

By Yiwei Chen, Lichi Li, Kai Cheung, Vinny Parla, Ganesh Sundaram

llms fine-tuning benchmarks data-engineering

arXiv AI

Jun 16

Less is More: Improving LLM Reasoning with Minimal Test-Time Intervention

arXiv:2510. 13940v4 Announce Type: replace-cross Abstract: Recent progress in large language models (LLMs) has focused on test-time scaling to improve reasoning via increased inference computation, but often at the cost of efficiency.

By Zhen Yang, Mingyang Zhang, Feng Chen, Ganggui Ding, Liang Hou, Xin Tao, Ying-Cong Chen

llms efficiency benchmarks

arXiv Machine Learning

Jun 16

The Data Manifold under the Microscope

arXiv:2606. 15760v1 Announce Type: new Abstract: A significant gap exists between theory and practice in deep learning.

By Marios Koulakis, Constantin Seibold

benchmarks

arXiv AI

Jun 16

An Attention Mechanism for Robust Multimodal Integration in a Global Workspace Architecture

arXiv:2602. 08597v3 Announce Type: replace Abstract: Robust multimodal systems must remain effective when some modalities are noisy, degraded, or unreliable.

By Roland Bertin-Johannet, Lara Scipio, Leopold Mayti\'e, Rufin VanRullen

multimodal benchmarks

arXiv AI

Jun 16

Landmark-free Assessment of Lower-limb Alignment with Implicit Neural Shape Functions from Knee Radiographs

arXiv:2606. 15250v1 Announce Type: cross Abstract: Radiographic assessment of lower-limb alignment (LLA) is important for predicting joint health and surgical outcomes in total knee arthroplasty.

By Zhisen Hu, Antti Kemppainen, David Johnson, Egor Panfilov, Huy Hoang Nguyen, Timothy Cootes, Claudia Lindner, Aleksei Tiulpin

diffusion benchmarks safety

arXiv Machine Learning

Jun 16

GradPower: Powering Gradients for Faster Language Model Pre-Training

arXiv:2505. 24275v4 Announce Type: replace Abstract: We propose GradPower, a lightweight gradient-transformation technique for accelerating language model pre-training.

By Jinbo Wang, Mingze Wang, Jiaqi Zhang, Wei Wang, Peng Pei, Xunliang Cai, Weinan E, Lei Wu

llms benchmarks

arXiv AI

Jun 16

SkillsVote: Lifecycle Governance of Agent Skills from Collection, Recommendation to Evolution

arXiv:2605. 18401v2 Announce Type: replace-cross Abstract: Long-horizon LLM agents generate traces that could become reusable experience, but raw trajectories are noisy, local, and hard to govern.

By Hongyi Liu, Haoyan Yang, Tao Jiang, Bo Tang, Feiyu Xiong, Yuyu Luo, Zhiyu Li

llms agents benchmarks

arXiv AI

Jun 16

Recurrent Reasoning on Symbolic Puzzles with Sequence Models

arXiv:2606. 15686v1 Announce Type: new Abstract: Large language models often appear strong on symbolic and algorithmic tasks, yet this apparent strength can hide brittle behaviour when problems become longer, harder, or slightly out of distribution.

By Gowrav Mannem, Chowdhury Marzia Mahjabin, Jason Chen, Shivank Garg, Kevin Zhu

llms fine-tuning benchmarks

arXiv AI

Jun 16

Scaling LLM Reasoning from Minimal Labels: A Semi-Supervised Framework with a Lightweight Verifier

arXiv:2606. 16811v1 Announce Type: new Abstract: For the development of Large language models (LLMs), recent approaches to generating pseudo intermediate reasoning have shown remarkable progress.

By Keizo Kato, Chenhui Chu, Yugo Murawaki, Sado Kurohashi

llms agents nlp fine-tuning benchmarks

arXiv AI

Jun 16

A biological vision inspired framework for machine perception of abutting grating illusory contours

arXiv:2508. 17254v2 Announce Type: replace-cross Abstract: Higher levels of machine intelligence demand alignment with human perception and cognition.

By Xiao Zhang, Kai-Fu Yang, Xian-Shi Zhang, Hong-Zhi You, Hong-Mei Yan, Yong-Jie Li

benchmarks safety

arXiv AI

Jun 16

StyleShield: Exposing the Fragility of AIGC Detectors through Continuous Controllable Style Transfer

arXiv:2605. 00924v2 Announce Type: replace-cross Abstract: AI-generated content (AIGC) detectors are increasingly deployed in high-stakes settings such as academic integrity screening, yet their reliability rests on a fundamental paradox: as language models are trained on human-written corpora, the statistical boundary between AI and human writing will inevitably dissolve as models improve.

By Guantian Zheng

llms rag diffusion benchmarks

Get the daily briefing