Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

15,829 stories · RSS feed

arXiv Machine Learning

Jul 24

From Evaluation to Optimisation: Hierarchy-Aware Training Signals for CWE Prediction in Python

arXiv:2607. 21069v1 Announce Type: new Abstract: The original ALPHA benchmark introduced a taxonomy-aware penalty for evaluating CWE-level vulnerability prediction in Python and proposed that the penalty could theoretically also serve as a training signal.

By Muntasir Adnan, Manile Srun, Carlos C. N. Kuhn

reinforcement-learning fine-tuning benchmarks safety

arXiv Machine Learning

Jul 24

A Polynomial Architecture-Attribution Co-Design Framework for Exact Aumann-Shapley Attribution in GNNs

arXiv:2607. 21094v1 Announce Type: new Abstract: We study feature-level and node-level explanations for graph neural networks (GNNs) through the lens of Aumann-Shapley attribution.

By Bizu Feng, Zhimu Yang, Shuming Wang, Shaode Yu, Yuan Cheng, Xiaojun Qian, Zixin Hu

benchmarks

arXiv Machine Learning

Jul 24

Automated Synthesis and Adversarial Validation of Executable Causal Research Pipelines

arXiv:2607. 21173v1 Announce Type: new Abstract: While automated research systems promise to accelerate empirical analysis, they are prone to silent failures: instances in which analysis code executes successfully yet relies on invalid causal assumptions.

By Irena Girshovitz, Dan Zeltzer, Ran Gilad-Bachrach

llms benchmarks safety

arXiv Machine Learning

Jul 24

The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and What Actually Works

arXiv:2607. 21273v1 Announce Type: new Abstract: Dense per-step supervision is an appealing remedy for sparse-reward, long-horizon LLM agents: reward the agent for predicting its next observation, and memory should follow.

By Yu Wang

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Jul 24

Naver-News-KO: A Korean News Summarization Dataset for Open-Source Fine-Tuning of Summarization Models

arXiv:2607. 20442v1 Announce Type: cross Abstract: We release Naver-News-KO, a Korean news summarization dataset of 27,400 (document, summary) pairs collected from Naver News over a ten-day window in July 2022 across two categories (Economy and IT/Science; 77/23 split), with train/validation/test partitions of 22,194 / 2,466 / 2,740 and a mean per-record document-to-summary character-compression ratio of 6.

By Daekeun Kim

nlp fine-tuning benchmarks

arXiv Machine Learning

Jul 24

Domyn-Small: A European 10B Reasoning Language Model

arXiv:2607. 20448v1 Announce Type: cross Abstract: We introduce Domyn-Small, a 10-billion-parameter open-weight reasoning language model released under the MIT license.

By Simone Angarano, Francesco Bertolotti, Federico D'Ambrosio, Michele Resta, Alessandro Rognoni, Nicol\`o Ruggeri, Dario Salvati, Andrea Valenti, Alberto Veneri, Martin Cimmino

llms agents reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jul 24

RadioTrace: Transmitter-Aware Diffusion for Radio Map Estimation without Deployment-Time Fine-Tuning

arXiv:2607. 20909v1 Announce Type: cross Abstract: Radio map (RM) estimation aims to reconstruct the spatial distribution of wireless signal characteristics, such as received signal strength (RSS), from sparse measurements, a task that is critical for spectrum management, interference mitigation, and localization in modern wireless networks.

By Liu Yang, Qiang Li, Zhuo Cao, Weijie Xiong, Guomin Sun, Jingran Lin

diffusion fine-tuning benchmarks

arXiv Machine Learning

Jul 24

Safety-oriented sidewalk and road segmentation for smartphone-based assistive navigation

arXiv:2607. 21137v1 Announce Type: cross Abstract: Independent sidewalk mobility is essential for blind and visually impaired pedestrians (BVIPs), yet smartphone-based assistive navigation requires perception models that distinguish walkable sidewalks from adjacent unsafe regions.

By Hakan Calim, Anamaria Dumitrescu, Adarsh Bhandary Panambur, Huzaifa Asif, Andreas Maier

computer-vision fine-tuning benchmarks safety

arXiv Machine Learning

Jul 24

Tackling Heterogeneity in Federated Learning via Variance-Reduced Boltzmann Sampling within Homogeneous Social Coalitions

arXiv:2506. 02897v3 Announce Type: replace Abstract: Federated Learning (FL) enables privacy-preserving collaborative model training, but its effectiveness is often limited by client data heterogeneity.

By Alessandro Licciardi, Roberta Raineri, Anton Proskurnikov, Lamberto Rondoni, Lorenzo Zino

benchmarks safety

arXiv AI

Jul 24

PATS: Policy-Aware Training Scaffolding for Agentic Reinforcement Learning

arXiv:2607. 21419v1 Announce Type: new Abstract: In long-horizon LLM agent reinforcement learning, weak policies often repeat similar failures, producing uninformative rollout trajectories and limiting effective policy optimization.

By Yipeng Shi, Zhipeng Ma, Yue Wang, Qitai Tan, Yang Li, Peng Chen, Zhengzhou Zhu

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Jul 24

Counterfactual Explainability Framework With CycleGAN And Counterfactual-Classifier Alignnment Score for Retinal Disease Classification

arXiv:2607. 21068v1 Announce Type: new Abstract: Automated detection of vision impairing retina-based ocular conditions from fundus images is important for early screening, timely referral and reducing dependency on specialist-only assessment, for which neural network-based deep learning (DL) models have been widely utilized.

By Kritanu Chattopadhyay, Sayanjit Singha Roy, Soumya Chatterjee

rag benchmarks safety

arXiv Machine Learning

Jul 24

Position Bias is Hidden Behind Ceiling Effects: A Permutation Diagnostic for LLM Benchmarks

arXiv:2607. 20864v1 Announce Type: new Abstract: Position bias in multiple-choice LLM evaluation is widely cited as a confound in capability comparisons, but published measurements rely on single answer-order shuffles whose results confound the bias signal with content-level noise and sampling stochasticity.

By Hiroki Tamba

llms benchmarks safety

arXiv AI

Jul 24

From Errors to Rules: Iterative Prompt Optimization for Text Classification

arXiv:2607. 20497v1 Announce Type: new Abstract: Prompt optimization for text classification spans diverse approaches, from demonstration selection to exploration-based search to error-driven diagnosis, each with known but incompletely characterized strengths and limitations.

By Yueying Cui, Renhao Xue, Yi Zhang, Mukul Prasad

nlp benchmarks

arXiv AI

Jul 24

Multilevel Graph Wavelet Compressed Sensing with Scale-Aware Neural Recovery

arXiv:2607. 20857v1 Announce Type: cross Abstract: Scientific machine learning methods such as neural operators and physics-informed neural networks have advanced engineering applications and inverse problems, but their training typically requires large volumes of simulated data.

By Amirhossein Nouranizadeh, Sarang Rajendra Patil, Alan John Varghese, Varsha Narayanan, Amit Chakraborty, Mengjia Xu

benchmarks

arXiv AI

Jul 24

JAXBench: Benchmarking Autonomous TPU Kernel Optimization

arXiv:2607. 20466v1 Announce Type: new Abstract: Rigorous benchmarks have driven progress in autonomous GPU kernel performance optimization by establishing a shared target to hillclimb on, but no equivalent exists for TPUs.

By Arya Tschand, Charles Hong, Julian Walker, Nina Cai, Shangkun Wang, Suvinay Subramanian, Sundar Dev, Vijay Janapa Reddi, Amir Yazdanbakhsh, Sethu Sankaran

llms agents benchmarks

arXiv AI

Jul 24

Reliability-Aware LLM Alignment from Inconsistent Human Feedback

arXiv:2607. 20515v1 Announce Type: new Abstract: Reinforcement Learning from Human Feedback (RLHF) is critical for aligning Large Language Models (LLMs) with human preferences.

By Jingyi Huang, Ruohan Zong, Yujun Feng, Liran Ma, Lanyu Shang, Yang Zhang

llms reinforcement-learning benchmarks safety

arXiv AI

Jul 24

AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use

arXiv:2607. 20536v1 Announce Type: new Abstract: Tool-use agents that address day-to-day digital tasks such as ordering groceries must not only operate applications, but also interact with the user, e.

By Junzhi Chen, Harsh Trivedi, Jane Pan, Michael JQ Zhang, Tejas Srinivasan, Niranjan Balasubramanian, Ashish Sabharwal

llms agents benchmarks

arXiv AI

Jul 24

From Attention to Frequency: Integration of Vision Transformer and FFT-ReLU for Enhanced Image Deblurring

arXiv:2511. 10806v1 Announce Type: cross Abstract: Image deblurring is vital in computer vision, aiming to recover sharp images from blurry ones caused by motion or camera shake.

By Syed Mumtahin Mahmud, Mahdi Mohd Hossain Noki, Prothito Shovon Majumder, Abdul Mohaimen Al Radi, Md. Haider Ali, Md. Mosaddek Khan

llms computer-vision efficiency benchmarks

arXiv AI

Jul 24

Answer-then-Edit: Reasoning Skeleton Editing for Anti-Distillation with Preserved Utility

arXiv:2607. 20440v1 Announce Type: cross Abstract: Proprietary large language models (LLMs) entail substantial intellectual and financial investment, making them valuable intellectual property (IP).

By Fan Li, Mengting Pan, Sijia Xu, Xiaoyang Wang, Chen Chen, Wenjie Zhang

llms efficiency benchmarks

arXiv AI

Jul 24

CAMeR: Keyword-Gated Hybrid Activation for Adaptive Memory Retention in LLM Agents

arXiv:2607. 20458v1 Announce Type: cross Abstract: Large language model (LLM) agents operating over extended dialogues accumulate vast amounts of information, yet existing memory systems either retain everything indiscriminately or apply uniform forgetting heuristics that fail to distinguish relevant from irrelevant knowledge.

By Haowen Lai

llms rag agents benchmarks

Get the daily briefing