Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,906 stories · RSS feed

arXiv AI

Jul 14

FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights

arXiv:2602. 02905v2 Announce Type: replace Abstract: Autonomous agents powered by large language models (LLMs) promise to accelerate scientific discovery end-to-end, but rigorously evaluating their capacity for verifiable discovery remains a central challenge.

By Zhen Wang, Fan Bai, Zhongyan Luo, Jinyan Su, Kaiser Sun, Xinle Yu, Jieyuan Liu, Kun Zhou, Claire Cardie, Mark Dredze, Zhiting Hu, Eric P. Xing

llms agents benchmarks

arXiv AI

Jul 14

Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs

arXiv:2604. 18576v4 Announce Type: replace Abstract: We present the Bayesian Linguistic Forecaster (BLF), an agentic system for binary forecasting that achieves state-of-the-art performance on the ForecastBench benchmark.

By Kevin Murphy

llms agents benchmarks

arXiv AI

Jul 14

A Low-Latency Fraud Detection Layer for Detecting Adversarial Interaction Patterns in LLM-Powered Agents

arXiv:2605. 01143v2 Announce Type: replace Abstract: Large Language Model (LLM)-powered agents demonstrate strong capabilities in autonomous task execution, tool use, and multi-step reasoning.

By Sheldon Yu, Yingcheng Sun, Hanqing Guo, Qianqian Tong

llms agents benchmarks safety

arXiv AI

Jul 14

Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models

arXiv:2601. 07372v2 Announce Type: replace-cross Abstract: While Mixture-of-Experts (MoE) scales capacity via conditional computation, Transformers lack a native primitive for knowledge lookup, forcing them to inefficiently simulate retrieval through computation.

By Xin Cheng, Rui Tian, Wangding Zeng, Damai Dai, Qinyu Chen, Bingxuan Wang, Zhenda Xie, Kezhao Huang, Xingkai Yu, Chengqi Deng, Shangyan Zhou, Chenggang Zhao, Zhewen Hao, Yukun Li, Han Zhang, Zhengyan Zhang, Yixu Wei, M. Y Xu, Huishuai Zhang, Dongyan Zhao, Wenfeng Liang

llms rag efficiency benchmarks

arXiv AI

Jul 14

Disentangling Intrinsic Importance from Emergent Structure in Multi-Expert Orchestration

arXiv:2602. 04291v3 Announce Type: replace-cross Abstract: Multi-expert systems, where multiple Large Language Models (LLMs) collaborate to solve complex tasks, are increasingly adopted for high-performance reasoning and generation.

By Sudipto Ghosh, Sujoy Nath, Sunny Manchanda, Tanmoy Chakraborty

llms agents benchmarks safety

arXiv Machine Learning

Jul 14

Gauge dependence and structured-output corruption in sign-branched repetition penalties: measurements across models, inference stacks, and alternative repetition controls

arXiv:2607. 09791v1 Announce Type: new Abstract: The multiplicative repetition penalty shipped across the LLM inference ecosystem (HuggingFace, vLLM, llama.

By Peter Hollows

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jul 14

Reinforcement Learning for Execution under Dynamic Fees in a Closed-Loop DEX Simulator

arXiv:2607. 10960v1 Announce Type: new Abstract: Trader-facing dynamic fees are increasingly proposed for automated market makers (AMMs), but historical data do not identify how order flow would respond: trader-facing fees do not vary, trader types are latent, and a replayed tape is not a sequential decision environment.

By Wen-Ting Wang

agents reinforcement-learning benchmarks

arXiv Machine Learning

Jul 14

A Multi-Agent Framework for Zero-Dimensional Reduced-Order Model Planning

arXiv:2607. 10994v1 Announce Type: new Abstract: Zero-dimensional reduced-order models (0D ROMs) are central to multi-dimensional design workflows for high-end complex equipment.

By Bingteng Sun, Hao Yin, Yiling Chen, Renjie Xiao, Lei Xie, Shanyou Wang, Ruonan Wang, Shubao Chen, Qingzong Xu, Lin Lu, Qiang Du, Junqiang Zhu

llms rag agents benchmarks

arXiv Machine Learning

Jul 14

ToolAtlas: Learning Once, Reusing Everywhere with Tool-Side Memory

arXiv:2607. 11126v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly rely on external tools served by shared providers and accessed by heterogeneous downstream agents.

By Yue Fang, Zhibang Yang, Fangkai Yang, Xiaoting Qin, Liqun Li, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang

llms agents benchmarks

arXiv Machine Learning

Jul 14

NeuroMem-FHP: A Likelihood-Free Deep Learning Framework for Parameter Estimation of Fractional Hawkes Process

arXiv:2607. 11177v1 Announce Type: new Abstract: In this paper, we propose deep learning based NeuroMem-FHP framework for estimating the parameters of the fractional Hawkes process (FHP), a self-exciting point process that captures long-range dependence through a fractional Mittag-Leffler excitation kernel.

By Neha Gupta, Aditya Maheshwari

llms benchmarks

arXiv Machine Learning

Jul 14

SPARC-Net: A Spectral, Causality-Aware, and Hard-Constrained Physics-Informed Architecture for Stiff and Shock-Dominated Partial Differential Equations

arXiv:2607. 11310v1 Announce Type: new Abstract: Physics-Informed Neural Networks (PINNs) provide a meshless approach for solving partial differential equations (PDEs), but suffer severe degradation in stiff and shock-dominated problems, where small PDE residuals can correspond to globally inaccurate solutions.

By Divyavardhan Singh, Dimple Sonone, Hammad Mohammad, Kishor Upla

benchmarks safety

arXiv Machine Learning

Jul 14

Surprisingly Simple and Effective Multi-Domain Graph Foundation Model through Graph-to-Table Alignment

arXiv:2607. 11374v1 Announce Type: new Abstract: Graph Foundation Models (GFMs) have emerged as a promising paradigm for learning transferable representations across diverse graph domains.

By Chunyu Hu, Tianyin Liao, Ge Lan, Xingxuan Zhang, Jianxin Li, Peng Cui, Ziwei Zhang

llms diffusion benchmarks safety

arXiv Machine Learning

Jul 14

HyperSafe: Inference-Time Safety Recovery for Fine-Tuned Language Models

arXiv:2607. 11475v1 Announce Type: new Abstract: Safety alignment in large language models can be fragile under fine-tuning, as even benign task adaptation may increase harmful compliance.

By Aznaur Aliev, Carlos Hinojosa, Abdelrahman Eldesokey, Bang An, Bernard Ghanem, Yibo Yang

llms fine-tuning benchmarks safety

arXiv Machine Learning

Jul 14

DAG-FM: A Foundation Model for Causal Discovery under Heterogeneous Causal Mechanisms

arXiv:2607. 11510v1 Announce Type: new Abstract: Causal discovery from observational tabular data remains fundamentally challenging, primarily due to the heterogeneity of underlying causal mechanisms and the high-dimensional combinatorial search space of Directed Acyclic Graphs (DAGs).

By Yikang Chen, Zhengkang Guan, Haoyuan Qian, Peng Cui, Yi Yang, Kun Kuang

llms benchmarks

arXiv Machine Learning

Jul 14

Fundamental Limitations of Fixed-Budget Best-Arm Identification

arXiv:2607. 11635v1 Announce Type: new Abstract: In fixed-budget best-arm identification, also known as ranking and selection, an algorithm has a sampling budget to distribute across $K$ arms.

By Motti Goldberger

benchmarks

arXiv Machine Learning

Jul 14

DynaFilter: Cloud-driven Dynamic Filtering for Satellite Edge Intelligence

arXiv:2607. 10098v1 Announce Type: cross Abstract: Modern satellite edge systems, including those performing remote sensing tasks such object detection and tracking, are characterized by severely limited bandwidth and intermittent connections, making continuous data transmission to the cloud impractical.

By Ziyang Zhang, Jie Liu, Luca Mottola

computer-vision efficiency multimodal benchmarks

arXiv Machine Learning

Jul 14

Gradient-Skipping Relevance Propagation for Efficient Explainability of Vision Transformers

arXiv:2607. 10365v1 Announce Type: cross Abstract: Vision Transformers (ViTs) are difficult to interpret because current methods of relevance propagation and attention flow do not fully consider some key architectural features, such as the uneven importance of attention heads and residual connections.

By Christopher Buratti, Michele Marchetti, Federica Parlapiano, Davide Traini, Domenico Ursino, Luca Virgili

llms computer-vision benchmarks safety

arXiv Machine Learning

Jul 14

Training-Free Off-Screen Player Imputation for Broadcast-Based Spatial Football Analytics

arXiv:2607. 11548v1 Announce Type: cross Abstract: Spatial football metrics such as pitch control assume access to the positions of all 22 players, yet the most widely available source of positional data -- the broadcast main camera -- shows only 10-16 of them at any moment.

By Seongjin Choi

benchmarks safety

arXiv Machine Learning

Jul 14

$\mathtt{Q^2SAR}$: overcoming classical bottlenecks in drug discovery via quantum multiple kernel learning

arXiv:2607. 11701v1 Announce Type: cross Abstract: Quantitative Structure-Activity Relationship ($\mathtt{QSAR}$) modeling is a foundational computational methodology in early-stage drug discovery, heavily relied upon for predicting compound toxicity, bioavailability, and therapeutic potential.

By Mariano Caruso, Daniel Ruiz, Alejandro Giraldo, Guido Bellomo

agents benchmarks safety

arXiv AI

Jul 14

Memory-Conditioned Tool Calling for Camera-First Visual Agents

arXiv:2607. 09822v1 Announce Type: cross Abstract: Recognition tells an agent what is in an image; personal memory affects what is worth looking up next.

By Xiaofan Wu (Chance AI), Xi Zeng (Chance AI), Miaoxia Chen (Chance AI), Peishan Chen (Chance AI), Shuyan Li (Chance AI), Jiyun Yao (Chance AI), Hanyong Zhong (Chance AI), Jiahao Zhu (Chance AI)

llms agents benchmarks

Get the daily briefing