Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv AI
Sep 18

AgentPProf: Semantic Profiler for Long Horizon AI Agents

AgentPProf is a new semantic profiler designed for long‑horizon AI agents that aggregates agent trajectories into pprof‑compatible profiles, enabling flame‑graph visualization and hierarchical attribution of tasks and subtasks. It introduces a semantic operation stack model and recursive operation segmentation to replace traditional call‑stack profiling, addressing the challenge of profiling agent intent rather than code paths. In evaluations, AgentPProf achieves high F1 scores against human annotations and significantly improves problem‑localization metrics, demonstrating its effectiveness in attributing resources, locating issues, and optimizing token cost.

By Yusheng Zheng, Chaokun Chang, Yu Mao, Tianyuan Wu, Yuxi Huang, Tao Ma, Wenan Mao, Shuyi Cheng, Andi Quinn, Wei Wang
arXiv AI
Sep 18

PAPC: Platform Mediation for Privacy-Propagation Externalities in AI-Mediated Workflows

PAPC is a platform-mediated mechanism designed to mitigate privacy-propagation externalities in AI‑mediated workflows. It intercepts information‑moving events—such as memory writes, shared‑workspace updates, inter‑agent messages, or tool events—before they update shared state or external channels, using policy, provenance, topology/fanout, privilege, and content signals to decide whether to release a policy‑safe abstraction, quarantine raw content, block a transition, or narrow onward rights. Experiments on retrieval‑memory and multi‑agent workflow benchmarks show that PAPC preserves deterministic task completion while eliminating measured exact raw‑value and external raw‑value exposure, demonstrating that event‑level mediation can serve as a platform‑governance primitive for agent‑mediated online work.

By Tao Huang, Guosen Wu, Chen Hou, Guolong Zheng
arXiv AI
Sep 18

Predict Before You Deploy: Offline Prediction of Quantization-Induced Task Degradation for World Action Models

The paper introduces PreDE, a policy‑calibrated framework that predicts how post‑training quantization will degrade task performance in world action models (WAMs) before deployment. By calibrating two thresholds on a small development set, PreDE can accept, reject, or defer new quantization configurations based on offline action deviations, achieving 75% coverage of decisions that match closed‑loop outcomes. Experiments on five WAMs and real‑robot trials show that PreDE accurately identifies high‑deviation configurations and enables significant speedups and memory reductions without compromising performance.

By Jiuyi Xu, Jinjia Guo, Meida Chen, Jing Du, Yangming Shi
arXiv AI
Sep 18

Detecting Soft Errors in Parallel Software with LLM-tuned Instruction Duplication

The paper introduces PaRID, a software-directed soft error detection framework that uses parallel instruction duplication and LLM-tuned performance modeling. PaRID is designed to work at compile time for multithreaded programs, handling mixed serial and parallel regions while keeping performance overhead low. Experiments on NPB benchmarks show that PaRID cuts protection overhead from 162.79% to 59.84% on average and can achieve up to a 5× speedup while still detecting all errors.

By Yafan Huang, Guanpeng Li
arXiv AI
Sep 18

PetriBench: Benchmarking LLM Reasoning over Dynamic State Spaces

PetriBench is a compact, fully self‑contained, and scalable benchmark that evaluates large language model (LLM) reasoning over dynamic state spaces using Petri nets. It organizes reasoning into four task families with Easy, Medium, and Hard levels, each generated by increasing structural complexity and evaluated against exact ground truth. Experiments across proprietary and open‑weight models show that accuracy consistently drops with difficulty, revealing distinct task‑specific capability profiles, while test‑time compute and procedural generation affect performance differently across tasks.

By Pyrros Koussios, Benjamin J\"ager, John Hua Yao, Ajay Sridhar, Violet Xiang, Chenhao Li
arXiv AI
Sep 18

The Missing Complement: State-Conditioned Minimal Sufficient Evidence for Coding Agents

The paper introduces State‑Conditioned Minimal Sufficient Evidence Recovery (SER), a method that, given a coding agent’s current state, reconstructs a compact set of evidence passages that collectively provide all facts needed for the agent’s next decision. Using the SERBench dataset of 500 held‑out states from 45 repositories, the authors show that their MSS‑Complement approach recovers a complete evidence set for 73.0 % of states with five items and 80.6 % with eight, outperforming baseline ranking methods. The study also demonstrates that this set‑level policy improves downstream performance on AMA‑Bench and highlights the importance of retrieving missing facts rather than merely re‑ranking similar passages.

By Zhexi Feng, Ruiyi Zhang, Yongbo Yang, Pengtao Xie
arXiv AI
Sep 18

When Do Language-Grounded Explanations Help? A Graph-Bottleneck for Farm Monitoring Interpretable Sheep Facial Pain

The study investigates whether language‑grounded explanations improve trust in automated sheep pain recognition from facial expressions. By grounding a model in the Sheep Pain Facial Expression Scale (SPFES) and testing attention‑based explanations, the authors find that such explanations are largely ineffective. They then replace the appearance bypass with a concept bottleneck that reads only SPFES concept scores, which slightly reduces performance but yields demonstrably learned concepts and better recovery of minority pain states.

By Alam Noor, Miguel Guti'errez Gait'an
arXiv AI
Sep 18

Large Language Models as Falsifiers for Cyber-Physical Systems

The paper introduces LLM-Falsifier, a large language model–based method for falsifying cyber‑physical system specifications written in Signal Temporal Logic (STL). By exposing the LLM to semantic cues such as natural‑language names, output trajectories, and critical‑time witnesses, the approach performs smarter, sample‑efficient robustness searches. On ARCH‑COMP benchmarks, LLM‑Falsifier outperforms existing tools across 14 of 21 specifications, requiring fewer simulations to find counterexamples.

By Ali ArjomandBigdeli, Jiawei Zhou, Stanley Bak
arXiv AI
Sep 18

Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generation

The study investigates how document segmentation and chunk representation affect retrieval-augmented generation (RAG) for chemistry texts. Using the ChemQuests corpus, the authors benchmark 41 embedding models and evaluate them across five chunking strategies, seven chunk sizes, and various overlap settings. They find that embedding choice has the largest impact, with models like E5, BGE, and Nomic performing best, and recommend medium-to-large chunks with fixed-token, recursive-token, or hierarchical-section chunking and low overlap for effective chemistry-aware RAG.

By Mahmoud Amiri, Thomas Bocklitz
arXiv AI
Sep 18

SGM: A Statistical Godel Machine for Risk-Controlled Recursive Self-Modification

The paper introduces the Statistical Godel Machine (SGM), a framework that replaces formal proof requirements with statistical confidence tests to safely allow recursive self-modification in machine learning systems. SGM uses e-values and Hoeffding bounds to certify improvements at a chosen confidence level while managing a global error budget, and introduces Confirm-Triggered Harmonic Spending (CTHS) to allocate risk more efficiently. Experiments on supervised learning, reinforcement learning, and black-box optimization demonstrate that SGM can certify genuine gains on CIFAR-100, reject false improvements on ImageNet-100, and maintain robustness across benchmarks.

By Xuening Wu, Shenqin Yin, Yanlan Kang, Xinhang Zhang, Qianya Xu, Zeping Chen, Wenqiang Zhang
arXiv Machine Learning
Sep 18

A Noise Optimum in Rehearsal-Free Continual Learning: Isolation, Mechanism, and Scope

The paper demonstrates that adding stochastic noise to a consolidation rule can improve a neural network’s retention of earlier tasks up to an optimal level, after which performance degrades, forming an inverted‑U relationship. Through simulations on related‑task continual‑learning benchmarks, the authors isolate the conditions that produce this optimum, showing it requires coherent restoration toward consolidated weights and is linked to the noise variance. The study further maps the scope of the effect, noting it depends on shared task structure and diminishes with more tasks, while a single‑seed hardware demonstration is referenced elsewhere.

By Gunner Levi Howe
arXiv Machine Learning
Sep 18

Hypernetwork-Parameterized Spatially Adaptive Neural Operators for PDE Learning

The paper introduces Hypernetwork-Parameterized Spatially Adaptive Neural Operators (SANO) for learning partial differential equations (PDEs) with spatial heterogeneity. SANO replaces spatially shared parameterization with a continuous field of location-dependent operator parameters, generated by a coordinate-conditioned hypernetwork and interpolated via a Hyper-Neural Element mechanism. Experiments on 1‑, 2‑, and 3‑dimensional PDEs and perforated-domain elliptic benchmarks demonstrate that SANO consistently outperforms existing neural‑operator, hypernetwork‑based, and physics‑informed baselines.

By Jiaquan Zhang, Chaoning Zhang, Shuxu Chen, Meng Ye, Xiaofeng Zhang, Qiang He, Weifeng Huang, Guoqing Wang, Yang Yang, Caiyan Qin
arXiv Machine Learning
Sep 18

VisKG-LM: Compiling Knowledge Graphs into Visual Memory for Multiple-Choice Question Answering

VisKG‑LM proposes compiling retrieved knowledge graph subgraphs into static visual memories rather than re‑encoding them during each inference step. The method serializes each subgraph as Relation‑Labeled Paths, renders them as images that preserve the graph’s branching structure, and caches these images for reuse. At inference, a language model processes the question and candidate text first, then consults the cached visual memory only at its final layer, yielding improved performance on CommonsenseQA, OpenBookQA, and MedQA‑USMLE compared to both text‑only baselines and a large vision‑language model.

By Yixin Peng, Er Jin, Shiwei Luo, Diego Collarana, Stefan Decker
arXiv Machine Learning
Sep 18

PrefixBench-H100: Characterizing Prefix Reuse and Time-to-First-Token in H100 LLM Serving

PrefixBench-H100 is a reproducible benchmark that evaluates how reusing prompt prefixes affects LLM serving performance on NVIDIA H100 GPUs. It tests two popular runtimes (vLLM and TensorRT-LLM) across varied workloads, measuring metrics such as time-to-first-token, latency, throughput, cache hits, and GPU memory usage. The study identifies when prefix reuse significantly reduces first‑token latency and when cache pressure diminishes those gains, noting that cache effectiveness is largely unaffected by concurrency or output length, while differences arise mainly in scheduling.

By Omkar Shewale, Deepak Kumar, Divakar Kumar Yadav
arXiv Machine Learning
Sep 18

HyperAMS-Net: Adaptive Multi-Scale Spatial Hypergraph Network for Brain Disorder Classification

HyperAMS‑Net is a deep learning framework that classifies brain disorders from resting‑state fMRI or structural MRI data. It combines adaptive multi‑scale convolution, hypergraph attention, spatial‑channel attention, and adaptive feature fusion to capture complementary patterns across multiple scales and higher‑order dependencies. Evaluated on ABIDE, REST‑meta‑MDD, and ADNI datasets, it achieves state‑of‑the‑art accuracy and AUC, with ablation studies showing hypergraph attention as the most critical component.

By Proloy Kumar Mondal, Md Kamran Hussin Chowdhury, Hoi Leong Lee
arXiv Machine Learning
Sep 18

ResNLS: An Improved Model for Stock Price Forecasting

ResNLS is a hybrid neural model combining ResNet and LSTM to forecast stock prices by emphasizing dependencies between adjacent prices. The model uses the closing prices of the previous five trading days as input, achieving optimal performance and outperforming state‑of‑the‑art baselines by at least 20%. In back‑testing, a trading strategy based on ResNLS‑5 predictions mitigated losses during market declines and generated profits during upturns.

By Yuanzhe Jia, Ali Anaissi, Basem Suleiman
arXiv Machine Learning
Sep 18

Model-Aware Data Cleaning for Tabular Foundation Models

The paper introduces L2C‑TFM, a reinforcement‑learning framework for cleaning tabular data before feeding it to Tabular Foundation Models (TFMs). It proposes a model‑aware reward that regularizes the Wasserstein distance between cleaned and dirty data, aiming to preserve distributional stability. Experiments on ten OpenML datasets show that while some reward designs fail, the model‑aware reward performs comparably to a random‑forest baseline and improves minority‑class macro‑F1 under class imbalance, and a policy trained on one dataset can transfer to others.

By Laure Berti-Equille
arXiv Machine Learning
Sep 18

Interpretable and Calibrated Classification of Clinical Data Using Supervised Feature Binarization

The paper introduces a statistically grounded framework for interpretable, rule-based clinical classification using Bernoulli Naïve Bayes (BNB). It employs supervised chi‑square‑guided binarization to convert continuous medical variables into binary indicators, enabling BNB to handle continuous data while maintaining transparency. On three benchmark datasets—Pima Indians Diabetes, Wisconsin Breast Cancer, and Heart Failure Prediction—the method achieved AUCs of 0.800, 0.984, and 0.919, respectively, and demonstrated reliable probability calibration through cross‑validated analysis and post‑hoc beta calibration.

By Antony Garcia, Adrian Noriega, Gabrielle Britton, Xinming Huang