Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv AI
Oct 7

Topology-Consistent Task Planning over Cellular Workflow Complexes for LLM-based Agents

The paper introduces TopoPlanner, a framework that transforms tool dependency graphs into cellular workflow complexes to enable topology‑consistent task planning for large language model (LLM) agents. It retrieves request‑relevant closed subcomplexes via cosheaf‑consistent cellular retrieval, performs multidimensional structural reasoning on the topology, and feeds the resulting representation to an LLM for generating tool sequences. Experiments on four tool‑planning benchmarks demonstrate consistent improvements over prompt‑based and graph‑enhanced baselines across various LLM backbones, especially for workflows involving loops, merges, and loop‑merge patterns.

By Sen Zhao, Jia Tang, Ruiqi Kong, Zuyu Zhang, Lifeng Shen, Ding Zou, Xinyu He, Xu Zhang, Junwei Han
arXiv AI
Oct 7

Cylindrical Geodesic Flow Matching for Quasiperiodic Physiological Signal Transformation

The paper introduces cylindrical geodesic flow matching, a method for translating quasiperiodic cardiovascular waveforms between different body locations. By replacing the standard affine path in flow matching with a closed‑form geodesic on a phase–amplitude cylinder, the approach preserves amplitude and instantaneous frequency during interpolation. Experiments on photoplethysmography and seismocardiography data show that this geometry‑aware method outperforms interpolation baselines and matches or exceeds supervised models, reducing error metrics by up to ~15%.

By Onur Selim Kilic, Afra Nawar, Cem Okan Yaldiz, Michael J. Cho, Ahmet Rasim Emirdagi, Demet Tangolar, Amirali Aghazadeh, Amit J. Shah, Omer T. Inan
arXiv AI
Oct 7

Learning When to Refine: Long-Horizon Reinforcement Learning for Budgeted Neural-Operator PDE Solvers

The paper introduces a budgeted adaptive neural‑operator framework for solving time‑dependent PDEs, combining a global Fourier neural operator with a local patch‑wise residual corrector and a set‑aware selector that decides where to refine. A macro policy governs the timing and amount of refinement, and the authors propose rollout‑verified policy improvement (RV‑PI) to evaluate feasible refinement counts via actual rollouts, converting long‑horizon advantages into conservative policy targets and accepting updates only when held‑out trajectory error improves. Experiments on shallow‑water and Brusselator benchmarks show that RV‑PI outperforms immediate‑only and random macro policies, demonstrating that the value of a local correction depends on its downstream effect rather than just its immediate error reduction.

By Ange Tong
arXiv AI
Oct 7

Towards a Unified Misuse Monitoring Benchmark

The paper introduces a unified framework for monitoring misuse of large language model agents in multi‑actor settings, focusing on two attack types: decomposition attacks and prompt injection attacks. It proposes tracking the agent’s externalised actions and determining whether the first detection of harm falls within a defined harm window. Using this formalism, the authors build a benchmark of about 6,200 conversation transcripts and evaluate 17 monitor configurations, finding that action‑framed monitors perform well on both threats while content‑framed monitors struggle with injection attacks, and highlighting limitations of traditional position‑blind metrics.

By Aniruddh Pramod, James Oldfield, Adel Bibi
arXiv AI
Oct 7

CausalBind: Causal Modeling and Learning for Protein-Molecule Virtual Screening

CausalBind introduces a causal modeling framework for protein‑molecule virtual screening that focuses on sparse cross‑modality interactions rather than dense holistic alignment. By formalizing a V‑structure causal model with Heckman‑style selection, the authors prove that latent interaction concepts are identifiable only under sparsity constraints and propose three implementation variants. Experiments on DUD‑E and LIT‑PCBA benchmarks demonstrate that all variants outperform strong retrieval baselines, especially in early enrichment and out‑of‑distribution generalization.

By Loka Li, Jin Tian, Kun Zhang
arXiv AI
Oct 7

SkillPoison: Progressive Skill Poisoning via Successful Experiences

SkillPoison demonstrates that self‑improving large language model agents can be progressively poisoned through successful experiences without any single trajectory being malicious. The framework constructs successful experiences that reinforce a target behavior, then removes contextual constraints so the skill extractor generalizes to harmful behavior when misapplied. Experiments on three benchmarks show a 95.71% attack success rate while all injected experiences remain task‑correct and pass verification and lexical inspection.

By Lizhi Zhang, Xin He, Dianxuan Fu, Yuyuan Feng, Jiatong Li, Qi Wang, Xin Wang, Qinggang Zhang
arXiv AI
Oct 7

SMART: Zero-Shot Sim-to-Real Articulated Object Manipulation via Large-Scale Synthetic Pretraining

SMART introduces a scalable system for zero‑shot sim‑to‑real articulated object manipulation by leveraging large‑scale synthetic pretraining. The core component, SMART‑Sim, is an articulation‑aware simulation platform that facilitates efficient task generation and demonstration collection. Using SMART‑Sim, the authors synthesize SMART‑Data, comprising over 1 million demonstrations across 44 task types, 5 robot setups, and 2,507 articulated objects, and demonstrate that a vision‑language‑action model pretrained on this data achieves competitive simulation performance and zero‑shot transfer to real‑world tasks.

By Jicong Ao, Shuhan Jiang, Yuling Zhong, Yanwen Liu, Yuhan Gao, Jiangyuan Zhao, Yang Zhang, Shiqiang Zhu, Chenjia Bai, Xuelong Li
arXiv Machine Learning
Oct 7

QiYao-I: A Manifold Based Foundation Model for Irregular Multivariate Time Series Forecasting

QiYao-I is a manifold‑based foundation model designed for irregular multivariate time series forecasting. It introduces a sampling‑conditioned temporal manifold attention mechanism that maps real timestamps into a learnable manifold feature space, and a dynamic variable interaction mechanism that selectively passes messages across variables while respecting asynchronous observations. Experiments on real‑world benchmarks show that QiYao‑I outperforms existing foundation and end‑to‑end irregular forecasting models, especially in zero‑shot and few‑shot scenarios.

By Linfeng Wang, Ruitong Zhang, Kai Zhao, Yang Shu, Zhongwen Rao, Meng Wang, Yijie Li, Bin Yang, Chenjun Guo
arXiv Machine Learning
Oct 7

Hub for Outliers, Spokes for Inliers: Uniform Latent Space Construction for Dual-Mismatched Semi-Supervised Learning

The paper introduces a hub‑spoke latent space design for dual‑mismatched semi‑supervised learning, where known classes are uniformly arranged around a central hub and each class forms a compact cluster around its prototype. The hub serves as an anchor for a low‑evidence region that captures high‑uncertainty unknown class samples, thereby reducing majority‑class domination and improving feature discriminability. Experiments demonstrate that this approach outperforms state‑of‑the‑art methods, achieving up to a 3.25% improvement across various settings.

By Li Yuan, Yaxin Hou, Jiawei Tang, Yongbiao Gao, Yuheng Jia
arXiv Machine Learning
Oct 7

Complementary Supervised and Self-Supervised Representations for Out-of-Distribution Graph Learning

The paper investigates how self‑supervised graph representations can complement supervised ones for out‑of‑distribution (OOD) node classification. Two backbone‑agnostic frameworks are introduced: Co‑Train, which jointly learns and integrates supervised and SSL representations during training, and Dual‑Space Retrieval, which performs non‑parametric prediction in both representation spaces and fuses the results at inference. Experiments on four graph benchmarks with various GNN backbones and SSL objectives (DGI and GRACE) show that Co‑Train consistently outperforms strong supervised OOD baselines, while Dual‑Space Retrieval offers competitive, flexible performance, demonstrating that SSL provides complementary signals that improve OOD node classification.

By Qingying Hao, Zikang Chen, Chuxuan Hu, Jinyuan Jia, Bo Li, Gang Wang, Carl Gunter
arXiv Machine Learning
Oct 7

Identity-Conditioned Score Fusion for Open-Set Person Re-Identification

The paper proposes identity-conditioned score fusion for open-set person re-identification, a framework that assigns weights tailored to each gallery identity without requiring additional training. By comparing intra-identity consistency with cross-identity impostors, it derives identity-specific profiles that work with query-conditioned adaptation through a parameter-free rule, enhancing the separation between true and false matches while maintaining score calibration. Experiments on three clothes-changing benchmarks show consistent improvements over statistical, rank-based, and learned baselines, achieving up to an 8.8% absolute reduction in the false non-identification rate.

By Manyi Yao, Jurijs Nazarovs, Eunji Chong, Abhishek Sharma, Rohan Sarkar, Yue Guo, Christian R. Shelton, Amit K. Roy-Chowdhury, Debashish Pal
arXiv Machine Learning
Oct 7

Common-Mode Errors Limit Low-Timestep Deep Spiking Q-Networks

The paper identifies that low‑timestep deep spiking Q‑networks (DSQNs) suffer mainly from common‑mode errors that degrade Q‑value estimation. To address this, the authors introduce Common‑Mode Compensation DSQN (CMC‑DSQN), which employs an auxiliary ANN during training to correct these errors while keeping inference energy‑efficient. Experiments on Atari and MiniAtar show that CMC‑DSQN improves performance by nearly 20 % at two timesteps and outperforms both DSQN baselines and the ANN baseline at four timesteps.

By Zijie Xu, Bingrui Guo, Yiding Sun, Yiting Dong, Zhile Yang, Zhaofei Yu
arXiv Machine Learning
Oct 7

Surviving the Router: Optimizing Skill Injections for Retrieval and Execution

The paper introduces CORSA, a router‑aware attack framework that optimizes skill injections for both retrieval and execution in AI agents that use modular third‑party skills. It demonstrates that existing evaluations overestimate attack success because they assume the malicious skill is already selected, whereas realistic multi‑skill environments reduce success rates by 87‑97%. CORSA improves retrieval and end‑to‑end attack success while maintaining user utility and shows transferability across different router architectures and LLM backbones.

By Haneen Najjar, Luca Scionis, Haritz Puerto, Sahar Abdelnabi
arXiv Machine Learning
Oct 7

Beyond Marginal Monitoring: Distributed Joint-Distribution Testing for Data Concept Drift in Large Scale E-Commerce Operations

The paper evaluates five multivariate two-sample drift detectors on large‑scale e‑commerce data, including a 137.5‑million‑row Trendyol feature table. Distributed Maximum Mean Discrepancy with Random Fourier Features on Apache Spark shows strong performance, achieving a Pearson correlation of 0.940 with expected drift magnitude, an 80.4% true positive rate, and a 3.2% false positive rate. In contrast, the per‑dimension Kolmogorov‑Smirnov test fails due to statistic saturation from ID‑like columns, and detectors struggle under weak drift configurations, underscoring limitations in current sampling designs and sensitivity analysis.

By Cagdas Pullu, Mahmut Emir Arslan, Bugra Balkac, Aylin Ondersev Balta, Cihangir Celal Palaci, Fikri Cem Yilmaz, Altan Cakir