Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,077 stories · RSS feed

arXiv Machine Learning

Jul 21

Backpropagation-Free Trunk Training via the Split Forward Gradients

arXiv:2607. 16612v1 Announce Type: cross Abstract: Backpropagation makes training deep networks memory intensive because it must store intermediate activations.

By Tian Qin, Wei-Min Huang

llms benchmarks

arXiv AI

Jul 21

Fantastic Adaptive Taxonomies and How to Use Them

arXiv:2607. 16387v1 Announce Type: cross Abstract: An agent system's execution traces record how it fails, and procedures that improve such a system without changing model weights (trajectory selection, prompt and workflow optimization, runtime monitoring) read these traces for feedback.

By Mert Cemri, Andrei Cojocaru, Melissa Pan, Shu Liu, Shubham Agarwal, Alexander Krentsel, Jay Tang, Kannan Ramchandran, Joseph E. Gonzalez, Matei Zaharia, Alex Dimakis, Ion Stoica

llms agents benchmarks

arXiv Machine Learning

Jul 21

AGG: Jacobian-Aggregated Group Gradient for Efficient GRPO Training of Diffusion Models

arXiv:2607. 17572v1 Announce Type: new Abstract: Group Relative Policy Optimization (GRPO) is a powerful reinforcement learning algorithm for aligning generative models with human preferences.

By Ruiyi Ding, Jie Li, He Kang, Ziyan Liu, Chengru Song, Yuan chen

llms diffusion reinforcement-learning benchmarks

arXiv AI

Jul 21

A Survey on Knowledge-Oriented Retrieval-Augmented Generation

arXiv:2503. 10677v3 Announce Type: replace-cross Abstract: Retrieval-Augmented Generation (RAG) has gained significant attention in recent years for its potential to enhance natural language understanding and generation by combining large-scale retrieval systems with generative models.

By Mingyue Cheng, Yucong Luo, Jie Ouyang, Qi Liu, Huijie Liu, Li Li, Shuo Yu, Bohou Zhang, Jiawei Cao, Jie Ma, Daoyu Wang, Enhong Chen

rag nlp multimodal benchmarks safety

arXiv AI

Jul 21

Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security

arXiv:2607. 18063v1 Announce Type: cross Abstract: LLM-based agents process external content, exposing them to prompt injection and multi-turn manipulation.

By Devina Jain, David Hartmann, Chuan Li

llms agents robotics multimodal benchmarks safety

arXiv AI

Jul 21

Topological Signatures of Context-Level Reliability in TabPFN

arXiv:2607. 17962v1 Announce Type: cross Abstract: TabPFN is a transformer-based foundation model for tabular prediction that performs inference without task-specific training by conditioning on a support set and query inputs.

By James Hu, Mahdi Ghelichi

llms benchmarks

arXiv AI

Jul 21

The Art of Not Forgetting

arXiv:2607. 17944v1 Announce Type: cross Abstract: We introduce CMP (Cognitive Memory Primitive), an architecture that represents inputs as sparse relational codes, stores them in a two-tier competitive memory, and learns entirely through local, gradient-free updates, with no backpropagation anywhere in the network.

By Ashmith Atmuri, Akshay Kumar, Yashaswini Rao Bhogarajula

llms benchmarks

arXiv AI

Jul 21

Chemical filters for ultra-high-throughput materials screening and generation

arXiv:2607. 17910v1 Announce Type: cross Abstract: Generative artificial intelligence is rapidly transforming materials design by enabling de novo exploration of immense chemical spaces.

By Kinga O. Mastej, Panyalak Detrattanawichai, Hyunsoo Park, Anthony Onwuli, Masahiro Negishi, Aron Walsh

diffusion efficiency benchmarks safety

arXiv AI

Jul 21

ReViV: Reconstructing the Viewer and the View in 4D from Monocular Egocentric Video

arXiv:2607. 17790v1 Announce Type: cross Abstract: Egocentric devices, such as wearable front-facing cameras, provide a unique perspective for capturing the continuous interaction between a human viewer and the surrounding environment.

By Xiaozhong Lyu, Gen Li, Zhiyin Qian, Xucong Zhang, Marc Pollefeys, Siyu Tang

llms computer-vision efficiency multimodal benchmarks

arXiv AI

Jul 21

Environment-free Synthetic Data Generation for API-Calling Agents

arXiv:2607. 16900v1 Announce Type: new Abstract: Training API-calling large language model (LLM) agents demands massive amounts of high-quality trajectories.

By Seanie Lee, Sanjoy Chowdhury, Chao Jiang, Cheng-Yu Hsieh, Ting-Yao Hu, Alexander T Toshev, Oncel Tuzel, Raviteja Vemulapalli

llms agents fine-tuning benchmarks

arXiv AI

Jul 21

"Not in My Backyard": LLMs Uncover Online and Offline Social Biases Against Homelessness

arXiv:2508. 13187v4 Announce Type: replace-cross Abstract: Homelessness is a persistent social challenge, impacting millions worldwide.

By Jonathan A. Karr Jr., Benjamin F. Herbst, Matthew L. Sisk, Xueyun Li, Ting Hua, Matthew Hauenstein, Georgina Curto, Nitesh V. Chawla

llms benchmarks safety

arXiv AI

Jul 21

Retain or Consolidate? Budget-Dependent Operator Selection for Language Agent Memory

arXiv:2607. 17545v1 Announce Type: new Abstract: Language agents depend on memory across interactions.

By Qingcan Kang, Mingyang Liu, Shixiong Kai, Kaichao Liang, Zhentao Tang, Yuqi Cui, Tao Zhong, Mingxuan Yuan

llms agents benchmarks safety

arXiv Machine Learning

Jul 21

Time-Aware Prior Fitted Networks for Zero-Shot Forecasting with Exogenous Variables

arXiv:2603. 15802v2 Announce Type: replace Abstract: In many time series forecasting settings, the target time series is accompanied by exogenous covariates, such as promotions and prices in retail demand; temperature in energy load; calendar and holiday indicators for traffic or sales; and grid load or fuel costs in electricity pricing.

By Andres Potapczynski, Ravi Kiran Selvam, Tatiana Konstantinova, Malcolm Wolff, Kin G. Olivares, Ruijun Ma, Michael W. Mahoney, Andrew Gordon Wilson, Boris N. Oreshkin, Dmitry Efimov

benchmarks

arXiv Machine Learning

Jul 21

SkillRouter: Skill Routing for LLM Agents at Scale

arXiv:2603. 22455v5 Announce Type: replace Abstract: Reusable skills let LLM agents package task-specific procedures, tool affordances, and execution guidance into modular building blocks.

By YanZhao Zheng, ZhenTao Zhang, Chao Ma, YuanQiang Yu, JiHuai Zhu, Yong Wu, Tianze Xu, Baohua Dong, Hangcheng Zhu, Ruohui Huang, Gang Yu

llms rag agents benchmarks

arXiv AI

Jul 21

LookME: Lookup-Based Multimodal Embeddings for Layer Injection in Vision-Language Models

arXiv:2607. 16305v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have achieved strong progress in multimodal understanding.

By Zeyu Xu, Xingzhong Hou, Pengkai Guo, Siling Lin, Xiao Xu, Menghua Zhai, Haoyu Chen, Yunke Zhang, Fei Huang

llms rag multimodal benchmarks

arXiv Machine Learning

Jul 21

SpecXMaster Technical Report

arXiv:2603. 23101v3 Announce Type: replace Abstract: Intelligent spectroscopy serves as a pivotal element in AI-driven closed-loop scientific discovery, functioning as the critical bridge between matter structure and artificial intelligence.

By Yutang Ge, Yaning Cui, Hanzheng Li, Jun-Jie Wang, Fanjie Xu, Jinhan Dong, Yongqi Jin, Dongxu Cui, Peng Jin, Guojiang Zhao, Hengxing Cai, Tianci Yangfeng, Xueqing Chen, Hongshuai Wang, Rong Zhu, Linfeng Zhang, Xiaohong Ji, Zhifeng Gao

agents reinforcement-learning benchmarks safety

arXiv AI

Jul 21

Is Your Model Thinking or Just Stagnating? PUMA: Diagnosing Reasoning Pathology via Phase-Momentum Alignment

arXiv:2607. 17188v1 Announce Type: new Abstract: Test-time scaling empowers Large Reasoning Models (LRMs) to tackle complex tasks via extensive Chain-of-Thought (CoT).

By Cheng Yan, Guangyang Ye, Wuyang Zhang, Fan Xu, Zhijun Fan, Xiang Xia, Yanyong Zhang

benchmarks safety

arXiv AI

Jul 21

A Deep Reinforcement Learning Algorithm for the Vehicle Routing Problem with Stochastic Demands and Outsourcing

arXiv:2607. 16875v1 Announce Type: cross Abstract: We introduce the vehicle routing problem with stochastic demands and outsourcing options (VRP-SDO), in which a logistics service provider partitions customer requests into customers outsourced to a common carrier and customers committed to its fixed fleet.

By Mohsen Dastpak, Fausto Errico, Ola Jabali

reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jul 21

Uncertainty Decomposition for Bayes-Filtered Transformers via Bayesian Predictive Inference

arXiv:2602. 04596v2 Announce Type: replace-cross Abstract: Bayes-filtered transformers are transformers meta-learned on sequences from a prior predictive distribution to approximate the corresponding posterior predictive distribution.

By Sandra Fortini, Kenyon Ng, Sonia Petrone, Judith Rousseau, Susan Wei

llms benchmarks

arXiv Machine Learning

Jul 21

Hippasus: Effective and Efficient Automatic Feature Augmentation for Machine Learning Tasks on Relational Data

arXiv:2602. 02025v2 Announce Type: replace-cross Abstract: ML models critically depend on feature quality, yet in real-world settings, useful features are often distributed across multiple relational tables rather than a single dataset.

By Serafeim Papadias, Kostas Patroumpas, Dimitrios Skoutas

llms benchmarks

Get the daily briefing