Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,077 stories · RSS feed

arXiv Machine Learning

Jul 21

FlashRT: Agent Harness for Guiding Agents to Deploy Real-Time Multimodal Applications

arXiv:2607. 18171v1 Announce Type: new Abstract: Real-time multimodal applications, including voice agents and interactive video generation, compose heterogeneous models into pipelines whose efficient deployment requires application-specific decisions about placement, streaming, and intra-model parallelism.

By Krish Agarwal, Zhuoming Chen, Yanyuan Qin, Zhenyu Gu, Atri Rudra, Beidi Chen

llms agents diffusion efficiency multimodal benchmarks

arXiv AI

Jul 21

Retain or Consolidate? Budget-Dependent Operator Selection for Language Agent Memory

arXiv:2607. 17545v1 Announce Type: new Abstract: Language agents depend on memory across interactions.

By Qingcan Kang, Mingyang Liu, Shixiong Kai, Kaichao Liang, Zhentao Tang, Yuqi Cui, Tao Zhong, Mingxuan Yuan

llms agents benchmarks safety

arXiv Machine Learning

Jul 21

Right Knowledge, Wrong Answer: Characterizing Parametric Temporal Conflict in Open-Weight Language Models

arXiv:2606. 20959v2 Announce Type: replace Abstract: Language models may encode both outdated facts and their newer replacements.

By Elias Hossain, Sourav Saha, Tasfia Nuzhat Ornee, Sanjeda Sara Jennifer, Umesh Chandra Biswas, Shubhashis Roy Dipta, Rajib Rana, Niloofar Yousefi

llms benchmarks

arXiv AI

Jul 21

Natural Language Access to Domain-Specific Metadata: A Reusable Framework for LLM Query Generation

arXiv:2607. 18029v1 Announce Type: cross Abstract: Researchers need to answer ad-hoc questions about the contents of domain-specific archives but often lack the expertise to write structured queries on the metadata.

By Blake G. Fitch, Cato Elia Kurtz

llms agents fine-tuning benchmarks safety data-engineering

arXiv AI

Jul 21

Lookahead Branching for Neural Network Verification

arXiv:2607. 17290v1 Announce Type: cross Abstract: In this work, we investigate the effect of lookahead branching strategies in neural network verification.

By Liam Davis, Duo Zhou, Huan Zhang, Guy Katz, Clark Barrett, Haoze Wu

benchmarks

arXiv AI

Jul 21

When to Use Which? Benchmarking Optimisers for Configurable Systems under Varying Budgets

arXiv:2607. 16476v1 Announce Type: cross Abstract: Software configuration tuning is crucial for optimising system performance, and various optimisers have emerged over the last decade.

By Chao Jiang, Yulong Ye, Tao Chen, Miqing Li

benchmarks

arXiv AI

Jul 21

Geometry-Enhanced Portion Estimation for Multimodal LLMs

arXiv:2607. 16514v1 Announce Type: cross Abstract: Image-based dietary assessment promises to replace costly, bias-prone manual recalls, but portion estimation remains a major blocker.

By Lin Liao, Peng Li

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jul 21

Retrieval is Enough: Training-Free Interpretability with a Tool-Using Agent

arXiv:2607. 16448v1 Announce Type: cross Abstract: Interpretability methods for neural network activations span a wide cost spectrum, from cheap, training-free techniques (such as linear probes, PCA, SVD) to more expensive training-based ones (such as SAEs and activation oracles).

By Sriram Balasubramanian, Soheil Feizi

llms rag agents diffusion benchmarks safety

arXiv AI

Jul 21

AoA: Theorem Proving Agent over Abstract Syntax Tree of Redesigned Language

arXiv:2607. 16372v1 Announce Type: cross Abstract: Interactive theorem proving (ITP) underpins program verification and formalized mathematics, but its manual effort limits scalability.

By Qiyuan Xu, Joshua Ong Jun Leang, Renxi Wang, Wenda Li, Haonan Li, Luke Ong, Conrad Watt

llms agents benchmarks

arXiv AI

Jul 21

Fantastic Adaptive Taxonomies and How to Use Them

arXiv:2607. 16387v1 Announce Type: cross Abstract: An agent system's execution traces record how it fails, and procedures that improve such a system without changing model weights (trajectory selection, prompt and workflow optimization, runtime monitoring) read these traces for feedback.

By Mert Cemri, Andrei Cojocaru, Melissa Pan, Shu Liu, Shubham Agarwal, Alexander Krentsel, Jay Tang, Kannan Ramchandran, Joseph E. Gonzalez, Matei Zaharia, Alex Dimakis, Ion Stoica

llms agents benchmarks

arXiv AI

Jul 21

A Quantum-Classical Hybrid Framework for Multivariate Time-Series Forecasting Complexity-Fidelity Trade-offs and Limitations

arXiv:2607. 16358v1 Announce Type: cross Abstract: This paper presents a unified quantum-classical hybrid framework for multi-horizon time-series forecasting, introducing two model variants Quantum Reservoir Forecaster (QRC-F) and Variational Quantum Forecaster (VQF-F).

By Sanjay Chakraborty, Fredrik Heintz

efficiency benchmarks

arXiv AI

Jul 21

RobustMAD: Evaluating Real-World Robustness of Multimodal Small Language Models for Deployable Anomaly Detection Assistants

arXiv:2607. 16243v1 Announce Type: cross Abstract: Multimodal industrial anomaly inspection assistants are a critical component of next-generation smart factories, enabling interactive vision-language-based querying.

By Anushiya Arunan, Xin Li, Yan Qin, U-Xuan Tan, Nhu Khue Vuong, Xiaoli Li, Chau Yuen

llms multimodal benchmarks safety

arXiv AI

Jul 21

FlowBlock: Wavefront-Parallel Decoding for Self-Correcting Diffusion Language Models

arXiv:2607. 17652v1 Announce Type: new Abstract: Block-wise diffusion large language models (dLLMs) decode sequentially at the block level, enabling effective KV-cache reuse across blocks but making inter-block decoding strictly serial.

By Bing Tian, Haikun Liu, Xiaocheng Zhong, Zhuohui Duan, Zhaokai Luo, Huayi Jin, Zhiyong Wang, Xiaofei Liao

llms diffusion efficiency benchmarks

arXiv AI

Jul 21

ProEvent: An Event-centric Benchmark for Proactive Agents

arXiv:2607. 17701v1 Announce Type: new Abstract: Proactive agents are expected to anticipate user needs and provide autonomous assistance by perceiving environmental context without explicit instructions.

By Guanzhen Li, Liangming Pan, Leye Wang

llms agents benchmarks

arXiv AI

Jul 21

Kernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones

arXiv:2607. 17419v1 Announce Type: cross Abstract: Linear attention promises constant-time recurrent inference but degrades sharply on associative recall.

By Ayoub Ghriss, Sourav Chakraborty

llms rag efficiency benchmarks

arXiv AI

Jul 21

Mechanistic Attention Guidance for Agent Memory Refinement

arXiv:2607. 17621v1 Announce Type: new Abstract: Existing self-evolving memory systems mainly improve agent memory based on textual outputs, such as task trajectories and reflections.

By Yechao Hong, Haiquan Qiu, Yaqing Wang, Quanming Yao

agents benchmarks

arXiv AI

Jul 21

A Dual-Hypothesis Reasoning Framework for LLM Guardrails

arXiv:2607. 17575v1 Announce Type: new Abstract: We propose ARBITER, a novel LLM guardrail framework that introduces two key ideas: (i) dual-hypothesis reasoning, a reasoning method for LLM guardrails that explicitly considers both safe and unsafe interpretations of a prompt before making a safety decision, and (ii) multi-component supervised fine-tuning (MC-SFT), a structured training loss for reasoning-based guardrails that decomposes LLM outputs into logical components and weights them according to their importance.

By Md Asiful Islam, Mihai Surdeanu

llms fine-tuning benchmarks safety

arXiv AI

Jul 21

Chemical filters for ultra-high-throughput materials screening and generation

arXiv:2607. 17910v1 Announce Type: cross Abstract: Generative artificial intelligence is rapidly transforming materials design by enabling de novo exploration of immense chemical spaces.

By Kinga O. Mastej, Panyalak Detrattanawichai, Hyunsoo Park, Anthony Onwuli, Masahiro Negishi, Aron Walsh

diffusion efficiency benchmarks safety

arXiv AI

Jul 21

The Art of Not Forgetting

arXiv:2607. 17944v1 Announce Type: cross Abstract: We introduce CMP (Cognitive Memory Primitive), an architecture that represents inputs as sparse relational codes, stores them in a two-tier competitive memory, and learns entirely through local, gradient-free updates, with no backpropagation anywhere in the network.

By Ashmith Atmuri, Akshay Kumar, Yashaswini Rao Bhogarajula

llms benchmarks

arXiv AI

Jul 21

Shapley Context Pruning: A Cooperative Game Perspective for Context Reranking and Pruning

arXiv:2607. 16209v1 Announce Type: new Abstract: Context reranking and pruning have become essential for improving the efficiency of modern Retrieval-Augmented Generation (RAG) systems, yet an interpretable and unified framework remains underexplored.

By Yanqiao Chen, Dongsheng Hou, Yuhan Rui, Zhen Cao, Yepang Liu

llms rag fine-tuning efficiency benchmarks

Get the daily briefing