Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,065 stories · RSS feed

arXiv AI

Jul 24

From Noise to Diversity: Random Embedding Injection in LLM Reasoning

arXiv:2605. 11936v2 Announce Type: replace Abstract: Recent soft prompt research has tried to improve reasoning by inserting trained vectors into LLM inputs, yet whether the gain comes from the learned content or from the act of injection itself has not been carefully separated.

By Heejun Kim, Seungpil Lee, Jewon Yeom, Jaewon Sok, Seonghyeon Park, Jeongjae Park, Taesup Kim, Sundong Kim

llms rag benchmarks

arXiv AI

Jul 24

From Errors to Rules: Iterative Prompt Optimization for Text Classification

arXiv:2607. 20497v1 Announce Type: new Abstract: Prompt optimization for text classification spans diverse approaches, from demonstration selection to exploration-based search to error-driven diagnosis, each with known but incompletely characterized strengths and limitations.

By Yueying Cui, Renhao Xue, Yi Zhang, Mukul Prasad

nlp benchmarks

arXiv AI

Jul 24

RE-AD: Real-Time Requirement Adherence for Data Labeling

arXiv:2607. 20455v1 Announce Type: cross Abstract: Human-annotated data remains fundamental to training frontier Large Language Models (LLMs).

By Siddarth Malreddy, Ishan Nigam, Akshay Arora, Nikhil Mittal, Subrat Sahu

llms benchmarks

arXiv AI

Jul 24

Telco-GAIA: Bilingual Benchmark for Agents in Telecom Domain

arXiv:2607. 20510v1 Announce Type: new Abstract: We introduce Telco-GAIA, a bilingual, multi-modal benchmark for evaluating tool-using agents on the data of a real-world telecommunications operator.

By Dmitrii Khizbullin, Zaid Alyafeai, Abdelrahman Eldesokey, Nourah AlSultan, Raghad Alshalan, David R. Pugh, Bernard Ghanem

llms agents multimodal benchmarks data-engineering

arXiv AI

Jul 24

Deblurring in the Wild: A Real-World Image Deblurring Dataset from Smartphone High-Speed Videos

arXiv:2506. 19445v4 Announce Type: cross Abstract: We introduce the largest real-world image deblurring dataset constructed from smartphone slow-motion videos.

By Syed Mumtahin Mahmud, Mahdi Mohd Hossain Noki, Prothito Shovon Majumder, Abdul Mohaimen Al Radi, Sudipto Das Sukanto, Afia Lubaina, Md. Mosaddek Khan

benchmarks

arXiv AI

Jul 24

OpenForgeRL: Train Harness-native Agents in Any Environment

arXiv:2607. 21557v1 Announce Type: new Abstract: Modern AI agents rely on elaborate inference harnesses such as Claude Code, Codex, and OpenClaw to drive multi-turn reasoning, tool use, and access to external systems.

By Xiao Yu, Baolin Peng, Ruize Xu, Hao Zou, Qianhui Wu, Hao Cheng, Wenlin Yao, Nikhil Singh, Zhou Yu, Jianfeng Gao

llms agents reinforcement-learning fine-tuning multimodal benchmarks

arXiv AI

Jul 24

Agentic Context Management: Solving Agent Memory and Cost by Treating Them as Lifecycle and Architecture Problems

arXiv:2607. 21503v1 Announce Type: new Abstract: Production AI agents' failures are less often due to an inability to reason well and more often because they cannot manage what is in their reasoning context: conversation histories, large prompts, large tool definitions, and ballooning tool outputs.

By Gaurav Dadhich

agents nlp benchmarks

arXiv AI

Jul 24

MIRROR: Learning from the Other View for Multi-Modal Reasoning

arXiv:2607. 21552v1 Announce Type: new Abstract: Unlike large language models (LLMs) that exhibit strong reasoning capabilities, vision-language models (VLMs) struggle with visual reasoning, even on geometry problems that admit equivalent text, diagram, and combined diagram+text views.

By Wen Ye, Yuxiao Qu, Aviral Kumar, Xuezhe Ma

llms reinforcement-learning multimodal benchmarks

arXiv AI

Jul 24

Agentic coding without the cloud: evaluating open-weight large language models on longitudinal data preparation tasks

arXiv:2607. 21482v1 Announce Type: new Abstract: Large language models (LLMs) and agents are now widely used tools in code development, with data typically sent to third-party cloud-based models.

By Mack Nixon, Liam Wright, Yevgeniya Kovalchuk, Alison Fang-Wei Wu, Martin Danka, Andy Boyd, David Bann

llms agents benchmarks

arXiv AI

Jul 24

AREX: Towards a Recursively Self-Improving Agent for Deep Research

arXiv:2607. 21461v1 Announce Type: new Abstract: Deep research requires agents to find answers that jointly satisfy multiple constraints.

By Shuqi Lu, Chaofan Li, Kun Luo, Zhang Zhang, Hui Wang, Hongwang Xiao, Zheng Liu, Lei Xiong, Jiahao Wang, Sen Wang, Xiyan Jiang, Wanli Li, Yuyang Hu, Hongjin Qian, Bingyu Yan, Ziyi Xia, Yingxia Shao, Kang Liu, Zhicheng Dou, Di He, Chaozhuo Li, Qiwei Ye, Zhongyuan Wang, Zheng Liu

agents reinforcement-learning benchmarks

arXiv AI

Jul 24

Multimodal Pretraining for Generalizable EEG Representation Learning

arXiv:2607. 21384v1 Announce Type: new Abstract: Electroencephalography (EEG) models used for epilepsy are often limited to specific datasets and tasks.

By Targol Bakhtiarvand, Jugal Kalita, Adham Atyabi

llms rag computer-vision fine-tuning multimodal benchmarks safety

arXiv AI

Jul 24

ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders

arXiv:2607. 21217v1 Announce Type: new Abstract: The recent emergence of vibe-coding workflows is changing what coding agents are expected to do.

By Zhongyuan Peng, Dan Huang, Chuyu Zhang, Caijun Xu, Changyi Xiao, Shibo Hong, David Lo, Lin Qiu, Xuezhi Cao, Jiyuan He, Yixin Cao

agents benchmarks

arXiv AI

Jul 24

Marking the Wrong Symptoms: Evaluating LLM Watermarks in Medical Texts

arXiv:2607. 20462v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly integrated into clinical workflows, stressing the need for reliable traceability of model-generated output with watermarking.

By Melanie Rieff, Robin Staab, Thibaud Gloaguen, Stefan Hegselmann, Martin Vechev

llms multimodal benchmarks safety

arXiv AI

Jul 24

V-DEAL: Diagnosing Video Safety De-Calibration as an Understanding-Refusal Coupling Failure

arXiv:2607. 21151v1 Announce Type: new Abstract: As Video Large Language Models are increasingly deployed in real-world applications, ensuring their safety alignment has become critical.

By Zhetong Zhang, Honghao Fu, Miao Xu, Yiwei Wang, Yujun Cai

llms fine-tuning benchmarks safety

arXiv AI

Jul 24

When RLVR Shrinks the Reasoning Boundary: Diagnosing Pass@k Inversion

arXiv:2607. 20543v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) can improve one-sample accuracy while making a model worse under repeated sampling.

By Todd Zhou

agents reinforcement-learning multimodal benchmarks

arXiv AI

Jul 24

CAMeR: Keyword-Gated Hybrid Activation for Adaptive Memory Retention in LLM Agents

arXiv:2607. 20458v1 Announce Type: cross Abstract: Large language model (LLM) agents operating over extended dialogues accumulate vast amounts of information, yet existing memory systems either retain everything indiscriminately or apply uniform forgetting heuristics that fail to distinguish relevant from irrelevant knowledge.

By Haowen Lai

llms rag agents benchmarks

arXiv AI

Jul 24

Answer-then-Edit: Reasoning Skeleton Editing for Anti-Distillation with Preserved Utility

arXiv:2607. 20440v1 Announce Type: cross Abstract: Proprietary large language models (LLMs) entail substantial intellectual and financial investment, making them valuable intellectual property (IP).

By Fan Li, Mengting Pan, Sijia Xu, Xiaoyang Wang, Chen Chen, Wenjie Zhang

llms efficiency benchmarks

arXiv AI

Jul 24

From Attention to Frequency: Integration of Vision Transformer and FFT-ReLU for Enhanced Image Deblurring

arXiv:2511. 10806v1 Announce Type: cross Abstract: Image deblurring is vital in computer vision, aiming to recover sharp images from blurry ones caused by motion or camera shake.

By Syed Mumtahin Mahmud, Mahdi Mohd Hossain Noki, Prothito Shovon Majumder, Abdul Mohaimen Al Radi, Md. Haider Ali, Md. Mosaddek Khan

llms computer-vision efficiency benchmarks

arXiv AI

Jul 24

SonicSampler: Unified Tile-Aware Kernels for LLM Sampling and Speculative Verification

arXiv:2607. 20475v1 Announce Type: new Abstract: Sampling in LLM inference comprises a combinatorial set of logit processing, token selection, and verification operations for speculative decoding.

By Pragaash Ponnusamy, Shivam Sahni, Jue Wang, Tri Dao

llms benchmarks safety

arXiv Machine Learning

Jul 24

CLOAK: Contrastive Guidance for Latent Diffusion-Based Data Obfuscation

arXiv:2512. 12086v2 Announce Type: replace Abstract: Data obfuscation is a promising technique for mitigating attribute inference attacks by semi-trusted parties with access to time-series data emitted by sensors.

By Xin Yang, Omid Ardakanian

diffusion benchmarks safety

Get the daily briefing