Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv AI
Sep 24

Learning Local Heterogeneity and Cross-Region Context for Large-Scale Traffic Forecasting

The paper introduces LoReST, a Local-Region Spatial Temporal network designed for large-scale traffic forecasting. LoReST captures local spatial heterogeneity by relation-aware aggregation within node neighborhoods and incorporates cross-region context through mean pooling, inter-region attention, and broadcasting back to nodes. Experiments on the LargeST benchmark demonstrate significant improvements, reducing MAE, RMSE, and MAPE by 4.78%, 3.60%, and 5.75% respectively.

By Qi Feng, Zidong Wang, Bo Li, Xiaoguang Gao, Jiayu Zhang, Chenfeng Wang, Kaifang Wan
arXiv AI
Sep 24

Provably Complete Generalized Planning with LLMs

The paper presents a method for automatically generating generalized plans in Lean, along with formal proofs of their completeness for given domain constraints. It introduces a semantic‑preserving conversion from PDDL to Lean and uses an LLM to produce both the plan and its proof, whose correctness is verified by Lean’s kernel. Evaluated on 13 benchmark domains with GPT‑5.6‑Sol, the approach yields complete plans and valid proofs for 12 of them, marking a significant advance in automated generalized‑plan completeness.

By Katharina Stein, Chaahat Jain, J\"org Hoffmann, Alexander Koller
arXiv AI
Sep 24

XLOG: A CUDA-Native Engine for Neurosymbolic Integration

XLOG is a CUDA‑native logic programming engine that fuses neural perception with deterministic Datalog, probabilistic inference, and epistemic world views. It offers multiple reasoning modes that share device data planes, with host‑orchestrated Datalog and exact inference, and resident recursive and Monte Carlo cores that avoid host‑device transfers. The system supports end‑to‑end gradients through GPU knowledge compilation, achieves significant speedups in MNIST‑addition training and join operations, and demonstrates competitive accuracy on several benchmark tasks.

By Levi Dubrovin, Nikita Pospelov, Kirill Sabitov
arXiv AI
Sep 24

PotARCin: Multi-Dimensional Evaluation of Skill Acquisition in Abstract Reasoning Tasks

PotARCin expands the ARC benchmark by evaluating abstract reasoning across five dimensions—Definition, Classification, Constrained Generation, Editing, and Inversion—using programmatic generation of new task instances. The study shows a 25‑52 percentage‑point performance gap between standard ARC evaluation and PotARCin, and reveals that multi‑dimensional assessment can reorder models that appear equivalent under single‑metric accuracy. Additionally, a new held‑out set, P‑ARC, demonstrates low model accuracy (1‑8%) across all dimensions, highlighting the need for more comprehensive tests of abstract reasoning.

By Claas Beger, Ryan Yi, Melanie Mitchell
arXiv AI
Sep 24

Learn How to Act from Your Own Interactions: On-Policy Self-Distillation for GUI Agents

The paper introduces GUI‑SD‑v2, an on‑policy self‑distillation framework that extends previous methods from GUI grounding to multi‑turn GUI interaction. It employs a two‑stage training process that first improves privilege following by jointly optimizing rollouts with and without privileged guidance, then selectively distills step‑specific reasoning and memory guidance via a privilege‑conditioned self‑teacher. Experiments on AndroidWorld and MobileWorld benchmarks demonstrate that GUI‑SD‑v2 outperforms existing OPSD baselines and state‑of‑the‑art methods in Pass@1 and Pass@3 success rates.

By Yan Zhang, Daiqing Wu, Huawen Shen, Liang Li, Gang Cao, Zhi Gong, Wei Dai, Xiaode Zhang, Can Ma, Yu Zhou
arXiv AI
Sep 24

Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved Mechanisms

The paper introduces VHD-Play, a pipeline that first samples and solves a mathematical model before generating agentic reinforcement learning environments, ensuring that dynamics and evaluation are aligned from the outset. This approach yields 3,300 diverse environments at a low cost and significantly improves the performance of a large language‑model agent (Qwen3.6‑35B‑A3B) across multiple diagnostic families and external benchmarks. The study demonstrates that stateful interaction is a key factor in learning gains and that scaling the training substrate can further enhance performance.

By Xinjie Shen, Wei Fan, Xudong Guo, Jianhong Tu, Yang Su, Chuqiao Kuang, Yinger Zhang, Dayiheng Liu
arXiv AI
Sep 24

Omni-Decision: Evidence-Ledger Planning for Omni-Modal Agents

Omni-Decision is an omni-modal agent that uses evidence-ledger planning to manage noisy multimodal observations. It replaces the growing dialogue history with a compact evidence ledger that tracks missing, confirmed, and conflicting evidence, allowing the planner to operate on a streamlined context. The system achieves state‑of‑the‑art accuracy on OmniGAIA and WorldSense while operating at a fraction of the cost of larger models.

By Ming Ma, Yi Zhu, Yiran Zhong, Feida Zhu, Yuhao Wang, Junhan Shi, Lingrui Mei, Tianming Yang, Steven Hoi
arXiv AI
Sep 24

The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks

The paper introduces the concept of an agent’s "taste"—its ability to make effective long‑horizon decisions—and presents Taste‑Bench, a new benchmark that automatically generates decision‑fork questions from agent trajectories. Taste‑Bench evaluates models on choosing the best path without seeing future outcomes, revealing that top models answer only about 60% of questions correctly and that later‑appearing evidence makes forks harder. The authors also demonstrate that training a student model to mimic a teacher’s judgment improves decision quality and overall success on held‑out software engineering tasks.

By Wenbo Pan, Zhichao Liu, Shujie Liu, Jingying Zeng, Chin-Yew Lin, Xianfeng Tang, Yan Lu, Qi He, Xiaohua Jia
arXiv Machine Learning
Sep 24

When Adaptation Hurts: Split Sensitivity and Person-Level Negative Transfer in Federated Wearable Onboarding

The paper evaluates six onboarding strategies for federated wearable models on five datasets using a leakage‑controlled protocol that fixes source checkpoints and separates calibration from evaluation. Results show that while average accuracy is high, person‑level performance can drop significantly, with some methods causing negative transfer for certain users. The study highlights that mean accuracy alone is insufficient and provides an auditable benchmark and failure map for future development.

By Rahil Aftab, Vineet Kumar Rakesh, Soumya Mazumdar, Tapas Samanta
arXiv Machine Learning
Sep 24

Active Learning for Biodiversity Monitoring: From Label Efficiency to Reliable Ecological Inference

The paper discusses how active learning (AL) can alleviate the expert annotation bottleneck in biodiversity monitoring by selecting the most informative samples under a fixed budget. It highlights that while AL reduces labeling effort, its non-random sample selection complicates model validation, calibration, and ecological inference, issues often overlooked in current studies. The authors review existing AL research across acoustic and image data, identify gaps such as limited species coverage and lack of real-world deployments, and propose a tutorial framework and roadmap for developing AL methods that support efficient training, reliable validation, and trustworthy ecological conclusions.

By Ben McEwen, Shiqi Zhang, Dan Stowell
arXiv Machine Learning
Sep 24

EBRL: Asynchronous Embodied RL by Multi-Grained Resource Management

EBRL is an asynchronous embodied reinforcement learning training system that overlaps rollout and training stages, pipelines simulation and generation across environment groups, and eliminates synchronization stalls. It employs a fine‑grained resource manager that pools CPU cores and GPU streaming multiprocessors, dynamically adjusting resource quotas and batch sizes based on stage profiles and runtime feedback. Experiments on RLinf with four policies and four simulation benchmarks show EBRL improves rollout throughput by 1.30–3.47× and training convergence by 2.5× over state‑of‑the‑art embodied RL systems.

By Liang Mi, Weijun Wang, Bowen Gao, Tianze Yu, Zixu Hao, Han Xiao, Xin Ding, Mingzhe Huang, Xin He, Lu Shi, Hao Wu, Haipeng Dai, Guihai Chen, Yunxin Liu, Ting Cao
arXiv Machine Learning
Sep 24

Limiting-Kernel Q($\lambda$): Bridging Short and Long Horizons

Limiting‑Kernel Q(λ) (LKQL) is an off‑policy value estimator that blends n‑step truncation with a long‑horizon approximation based on the limiting kernel. It maintains the computational efficiency of n‑step methods while improving policy evaluation accuracy, especially for long‑horizon tasks. The authors prove faster convergence of LKQL’s operator under aperiodicity and near‑on‑policy conditions, and demonstrate empirical gains on MuJoCo continuous‑control benchmarks.

By Tolga Ok, Arman Sharifi Kolarijani, Peyman Mohajerin Esfahani, Mohamad Amin Sharifi Kolarijani
arXiv Machine Learning
Sep 24

CAST: Context- and Anomaly Structure-Conditioned Time Series Anomaly Generation

CAST is a framework for generating anomalous time series that addresses the scarcity and heterogeneity of anomaly data. It uses a two‑stage approach: pretraining on abundant normal data to learn system dynamics, then finetuning with anomaly structure representations to capture diverse anomaly morphologies. Experiments on real‑world datasets show that CAST outperforms existing methods in both generation quality and downstream task performance.

By Haochen Zhang, Jie Peng, Songyuan Sui, Yu-Chao Huang, Xiangqi Zhu, Tianlong Chen
arXiv Machine Learning
Sep 24

Evaluation Choices Decide the Forecasting Leaderboard: Evidence from a Production Marketplace Panel

The paper demonstrates that the outcome of a forecasting leaderboard is largely determined by the evaluator’s design choices rather than the models themselves. By fixing the data, horizon, and period, the authors varied three key evaluation decisions—unit of analysis, error pooling, and scoring metric—and showed that each can reverse or eliminate the apparent superiority of any forecasting method. The study also evaluates the practical impact of these choices on a deployed system, revealing that the selection rule captures a significant portion of the potential performance gain, and confirms the findings on an external public dataset.

By Md Rezwanul Islam, Wael Mohammed
arXiv AI
Sep 24

WAInjectBench: Benchmarking Prompt Injection Detections for Web Agents

WAInjectBench introduces the first comprehensive benchmark for detecting prompt injection attacks against web agents, offering a fine‑grained categorization of threats and datasets that include malicious and benign text and image samples. The study systematically evaluates both text‑based and image‑based detection methods across multiple scenarios, revealing that detectors perform well on attacks with explicit instructions or visible perturbations but struggle with subtle or instruction‑free attacks. The authors release the datasets and code to facilitate further research in this area.

By Yinuo Liu, Xilong Wang, Ruohan Xu, Yuqi Jia, Neil Zhenqiang Gong
arXiv Machine Learning
Sep 24

LexLattice: Multilingual Extractive Summarization via Neural Cellular Automata on Document Hierarchies

LexLattice is an extractive summarizer that models a legal act’s hierarchy as a two‑dimensional semantic lattice and consolidates information over it using a masked 2D neural cellular automata before selecting content. The method achieves state‑of‑the‑art ROUGE scores across all 24 languages of EUR‑Lex‑Sum in both multilingual and cross‑lingual settings, outperforming large instruction‑tuned baselines while using only a 1.8 M‑parameter consolidator on a frozen multilingual encoder. A consolidator trained on high‑resource languages transfers almost losslessly to unseen languages, suggesting the model operates on language‑agnostic semantic geometry rather than surface form.

By Sujay Uday Rittikar, Sheela Ramanna
arXiv Machine Learning
Sep 24

Does Graph Structure Earn Its Place in Microservice Root-Cause Analysis? A Controlled Study on RCAEval, and What the Benchmark Was Really Measuring

The paper investigates whether graph structure improves microservice root‑cause analysis by conducting a controlled study on the RCAEval benchmark. Using identical features, optimizers, and evaluation protocols across three model variants, the authors find no consistent advantage for graph‑based models over flat models, with a negligible Avg@5 difference (0.003, p=0.844). They identify two benchmark properties—limited fault injection and a non‑uniform telemetry schema—that bias results, and propose a new model, PSC‑GRCA, which achieves higher Avg@5 mainly through a system prior rather than graph information.

By Imad Bulji\'c
arXiv Machine Learning
Sep 24

PEARL: A Lightweight Prompt-based Feature Interpreter Framework for Real-Time, Anonymous, and Heterogeneous Collaborative Perception

PEARL is a lightweight, prompt‑embedding framework designed for real‑time, anonymous, and heterogeneous collaborative perception. It uses two parallel, low‑rank visual prompt interpreters—sparse‑detection (LWSD) and dense, domain‑invariant (LWDDI)—to align features and select the appropriate interpreter for newly joining agents without needing their configurations. Experiments on simulated and real datasets show that PEARL improves average precision by 8.2% over random selection, runs in 1.67 ms, and reduces communication cost by up to 34.7× while outperforming state‑of‑the‑art offline methods by 5.6% AP.

By Armin Maleki, Hayder Radha
arXiv Machine Learning
Sep 24

Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models

The paper investigates how attention dynamics evolve across recurrent depth in language models, finding that attention support stabilizes early while hidden states and outputs take longer. It proposes WISE, a training‑free method that uses full attention in early steps and then reuses the discovered sparse working set for later steps, preserving performance on multi‑hop QA tasks. Experiments show that WISE maintains quality up to 2K context, offers measurable speedups, and highlights the importance of recurrent discovery of attention support.

By Ke Wan, Chen Chen
arXiv Machine Learning
Sep 24

"What's That Sound?": A Versatile, Robust, and Lightweight Convolutional Transformer for Environment Sound Recognition

The paper introduces RALCT, a lightweight Convolutional Transformer that combines randomized audio augmentations, MFCCs, and log‑mel spectrograms to extract robust features for environmental sound recognition. With only about 310,000 parameters, RALCT achieves state‑of‑the‑art accuracy—over 93% on UrbanSound8K, peaking at 94.56%—making it suitable for deployment on mobile devices. The authors also develop a mobile app that integrates the model to provide real‑time safety alerts for hearing‑impaired users.

By Julia Huang