Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv AI

Oct 28

Smart Surveillance: Deep Learning and Hybrid Approaches for Dynamic Scene Analysis, Object Detection and Motion Tracking

The paper evaluates event-triggered recording strategies for static surveillance cameras, comparing six motion-based triggers, a CNN detector, strided detection, and hybrid approaches on a benchmark of seven scenes. Detector-based triggers reduce idle frame storage to 19‑28% versus 36‑89% for motion triggers, and remain robust under camera jitter, keeping under 29% idle frames. In addition, the study shows that increasing YOLO11n input resolution improves detection accuracy more than larger models, and that longer track memory boosts tracking IDF1, though detection remains the limiting factor.

By Shahran Rahman Alve

computer-vision efficiency benchmarks

arXiv AI

Oct 28

Counting Clues: A Lightweight Probabilistic Baseline Can Match an LLM

The paper introduces Frequency-Based Probabilistic Ranker (FBPR), a lightweight method that uses smoothed Naive Bayes on concept‑diagnosis co‑occurrence statistics to score multiple‑choice clinical diagnosis questions. When trained on corpora such as Dolma and RedPajama, FBPR matches the performance of large language models (LLMs) on MedQA benchmarks. FBPR and LLMs answer largely different subsets of questions, and a simple hybrid of the two yields a notable performance gain.

By Furong Jia, Yuan Pu, Finn Guo, Monica Agrawal

llms benchmarks

arXiv AI

Oct 28

A swap-adversarial framework for domain generalization in pre- and post-stimulation Parkinsonian state classification

The paper introduces a swap‑adversarial framework designed to address inter‑subject variability and the high‑dimensional low‑sample‑size challenge in electrocorticography (ECoG) data. It combines robust preprocessing, an inter‑subject balanced channel swap (ISBCS) for cross‑subject augmentation, and domain‑adversarial learning (DAL) to suppress subject‑specific bias, enabling robust domain generalization across both ECoG and EEG‑based brain‑computer interface datasets. The authors also present a new reproducible ECoG benchmark derived from long‑term recordings of 6‑hydroxydopamine‑induced rat models, and demonstrate the framework’s superior performance in cross‑subject, cross‑session, and cross‑dataset experiments.

By Seongwon Jin, Hanseul Choi, Sunggu Yang, Sungho Park, Jibum Kim

benchmarks safety

arXiv AI

Oct 28

Efficient Token-Level Policy Optimization for Masked Diffusion Models via Velocity Alignment

The paper introduces Sequence Likelihood-Free Policy Optimization (LFPO), a method that aligns masked diffusion models by optimizing token-level denoising logits through contrastive updates, avoiding the need for sequence-level likelihoods. LFPO leverages discrete velocity alignment inspired by continuous flow matching, providing a stable, low-variance training surrogate. Experiments on reasoning and coding tasks show that LFPO matches or surpasses likelihood-based baselines while significantly speeding up inference.

By Chenxing Wei, Jiazhen Kang, Hong Wang, Jianqing Zhang, Hao Jiang, Xiaolong Xu, Ningyuan Sun, Ying He, F. Richard Yu, Yao Shu, Bo Jiang

llms diffusion reinforcement-learning benchmarks safety

arXiv Statistics ML

Oct 28

Latent Nonlinear Denoising Score Matching for Enhanced Learning of Structured Distributions

The paper introduces Latent Nonlinear Denoising Score Matching (LNDSM), a method that integrates nonlinear, structure-informed latent diffusions with a variational autoencoder framework. LNDSM designs the forward diffusion to admit a chosen reference distribution, such as a Gaussian mixture, and derives a computable approximation to a cross‑entropy upper bound using Euler‑Maruyama discretization. Experiments on benchmark image datasets show that LNDSM improves sample quality, mode balance, and training efficiency compared to conventional latent score‑based generative models.

By Kaichen Shen, Wei Zhu

diffusion benchmarks

arXiv Statistics ML

Oct 28

Temporal difference learning adapts to incomplete coverage: an oracle inequality over subgraphs

The paper demonstrates that ordinary temporal difference (TD) learning in a tabular setting automatically combines the asymptotic efficiency of TD with the visitation requirements of Monte Carlo (MC) methods. By introducing the subgraph Bellman variance—a measure of variance arising from bootstrapping within a subgraph and rollouts outside it—the authors prove a non‑asymptotic oracle inequality: the mean squared error of full TD is bounded by this variance divided by the sample size for every sufficiently covered subgraph containing the target, plus a higher‑order remainder that depends only on that subgraph. Importantly, the algorithm does not need to be supplied with any subgraph, allowing TD to adapt to incomplete coverage without global visitation guarantees.

By Wenlong Mou, Jian Qian

reinforcement-learning benchmarks

arXiv Computer Vision

Oct 28

GradCO: Behavior-Aware Gradient-Conflict Offloading for Text-Based Person Anomaly Search

GradCO is a new method for Text-based Person Anomaly Search (TPAS) that addresses two key challenges: gradient conflict from same-identity, different-behavior hard negatives and the dominance of strong appearance cues over weak behavioral evidence. It introduces a Behavior-Aware Mixture-of-Experts (B-MoE) to separate appearance, normal-behavior, and anomalous-behavior streams, Conflict Offload Learning (COL) to reduce conflicting gradients via pair-conditioned anchors, and Hierarchical Intent Alignment (HIA) to aggregate expert similarities with query-conditioned weights. Experiments on five TPAS/TIPR benchmarks demonstrate that GradCO significantly outperforms state-of-the-art methods, validating its design choices.

By Zhiheng Fu, Zixu Li, Zhiwei Chen, Jiale Huang, Fangxu Liu, Jinhe Bi, Yupeng Hu

benchmarks safety

arXiv AI

Oct 28

Spiking Neural Networks Are Harder to Steal: A Benchmark Study of Model Stealing Attacks

The paper investigates whether spiking neural networks (SNNs) are more resistant to model stealing attacks than conventional artificial neural networks (ANNs). By benchmarking four common attacks on MNIST, Fashion‑MNIST, and CIFAR‑10, the authors find that SNNs consistently yield surrogates with 10%–20% lower agreement and 15%–25% lower adversarial transferability compared to ANNs. The authors attribute this resilience to the spike‑based, time‑dependent dynamics of SNNs, which make their decision boundaries harder for ANN‑based surrogates to approximate.

By Ayana Moshruba, Maryam Parsa

benchmarks safety

arXiv AI

Oct 28

How Stable Are LLM-Generated Proofs Under Program Refactoring? A Stress Test for Verifiable Code Generation

The paper introduces RefactorProof, a benchmark that tests whether proofs generated for code remain valid after program refactoring that preserves behavior. Using 458 Lean-certified variants derived from 185 VERINA tasks, the study finds that structural refactorings expose brittleness in proofs, with task-normalized structural PSR ranging from 4.3% for supplied references to 13.3–26.8% for specialized provers. Despite higher failure rates, specialized prover proofs (DeepSeek-Prover-V2-7B, Goedel-Prover-V2-32B, Goedel-Prover-V2-8B) show significantly higher unchanged-proof survival compared to reference artifacts, and many failures can be repaired with minimal edits.

By Aravind Mandiga, Guoming Li

llms benchmarks

arXiv AI

Oct 28

MAREval: A Multi-Agent Framework for Evaluating Natural Language Recommendation Explanations

MAEval is a multi‑agent framework that evaluates natural language explanations in recommender systems using large language models. It employs a planner agent with a Chain of Debate prompting strategy, a moderator to control prompt drift, and an arbitrator that aggregates multiple evaluation rounds, supplemented by Monte Carlo sampling for robustness. Experiments on public and proprietary datasets show that MAREval outperforms existing baselines, aligns more closely with human judgments, reduces variability, and achieves production‑level quality, leading to measurable gains in user engagement and revenue.

By Reza Yousefi Maragheh, Jayesh Uddhav Kudase, Aysenur Inan, Ramin Giahi, Kai Zhao, Parth Parikh, Jianpeng Xu, Jason Cho, Evren Korpeoglu, Sushant Kumar

llms agents benchmarks safety

arXiv AI

Oct 28

Remembering Is Not Enough: Refining Memories and Learning When to Use Them in Multi-Agent Systems

The paper introduces DiscernMem, a training‑free framework for improving memory quality and usage in LLM‑based multi‑agent systems. It combines Defect‑Aware Memory Refinement (DMR), which diagnoses and repairs flawed memories, with Boundary‑Learning Meta‑Memory (BLM), which learns when to apply or avoid each memory based on context. Experiments with Qwen3‑4B‑Instruct‑2507 on four benchmarks show DiscernMem outperforms existing baselines, achieving higher scores on AutoGen, CAMEL, and MacNet, and improving code generation and symbolic planning performance.

By Yipeng Hao, Yuxuan Cai, Wenkai Ji, Jie Zhou, Liyang Yu, Qin Chen, Zhiguo Yan, Shulei Zheng, Liang He

llms agents benchmarks

arXiv AI

Oct 28

Conservative Policy Iteration For Zero-Sum POSGs

The paper introduces an approximate policy‑iteration algorithm for finite‑horizon zero‑sum partially observable stochastic games (zs‑POSGs). It maintains a separate policy for each player and updates both jointly using conservative policy updates along occupancy trajectories, avoiding the need to approximate a full central value function. The method focuses computation on occupancy regions where the current joint policy can be exploited, and shows competitive performance on small horizons while improving the runtime‑exploitability trade‑off at larger horizons compared to existing solvers such as PBVI, HSVI, and CFR+.

By Erwan C. Escudie, Matthia Sabatelli, Jilles Steeve Dibangoye

benchmarks

arXiv AI

Oct 28

Timestral

Timestral is a family of four time‑series foundation models (5M, 28M, 123M, and 209M parameters) designed for probabilistic forecasting. It achieves state‑of‑the‑art results on GIFT‑Eval, TIME, and fev‑bench, with the 209M and 123M variants topping every metric. The approach relies on Decoupled Horizon Forecasting, a covariate framework for known‑future variables, and a two‑stage training pipeline that first pre‑trains on synthetic data before fine‑tuning on real and synthetic data with extended context.

By Adrien Petralia, Alexandre Gavaudan, Artjom Joosen, Etienne Goffinet, Richard Kurle, Yassine El Ouahidi, Yassir Bendou, Abdelaziz Bounhar, Abhijeet Somani, Aditi Kabra, Adrian Valente, Adrien Sad\'e, Ahmed Djermani, Alan Jeffares, Albert Jiang, Alberto Bietti, Aleksandr Timashov, Alexandre Cahill, Alexandre Laval, Alexandre Sablayrolles, Alexandros Sanchez Vassopoulos, Am\'elie H\'eliou, Amos You, Andr\'e Jonasson, Andreas F\"urst, Andrew Bai, Andrew Ehrenberg, Andrew Zhao, Andy Lo, Angele Lenglemetz, Anmol Agarwal, Antonia Calvi, Arata Suzuki, Arjun Majumdar, Arthur Fournier, Ashwani Kumar, Augustin Lombard, Avinash Sooriyarachchi, Aylin Guliz Akkus, Aysenur Karaduman, Baptiste Bout, Baptiste Rozi\`ere, Barna P\'asztor, Baudouin De Monicault, Benjamin Holzschuh, Benjamin Tibi, Benno Stein, Bernhard Stadlbauer, Bhargavi Bhaskar, Binbing Hou, B{\l}a\.zej Osi\'nski, Boris van Breugel, Camille Le Scao, Chaoran Yu, Charlotte Cronj\"ager, Chen-Yo Sun, Chris Bamford, Christian Wallenwein, Christophe Renaudin, Cl\'emence Lanfranchi, Colin Bonatti, Corentin Barreau, Corentin Sautier, Cristiana-Diana Diaconu, Cyprien Courtot, Daniel Lopes, Daniel Marczak, Darius Dabert, Diego de las Casas, Dominik Nuss, Dylan Rubini, Dzmitry Soupel, Elizaveta Demyanenko, Elliot Chane-Sane, Emilien Fugier, Emmanuel Gottlob, Erik Aas, Etienne Duchesne, Fabian Paischer, Fabian Schlager, Faruk Ahmed, Federico Baldassarre, F\'elix Rosseeuw, Filip Szatkowski, Filippo Vicentini, Florian Wiesner, Gabrielle Berrada, Ga\"etan Ecrepont, Gaspard Blanchet, Gaspard Donada-Vidal, Gaurav Mahajan, Gauthier Guinet, Genevieve Hayes, Georgii Novikov, Georgios Christoglou, Giada Pistilli, Gonzague de Carpentier, Graeme Nail, Guillaume Breton, Guillaume Kunsch, Guillaume Lample, Guillaume Martin, Guillaume Raille, Guillaume Richard, Guillem Simeon, Gunjan Dhanuka, Gunshi Gupta, Hamza Senhaji Rhazi, Han Zhou, H\'edi Hadiji, Himanshu Jaju, Hope McGovern, Hubert de Parseval, Hugo Cisneros, Hugo Germain, Hugo Thimonier, Hugo Venturini, Hyeonju Lee, Hyunwoo Oh, Iliass Lasri, Ilyas Oulkadda, Jacques Sun, Jan Ludziejewski, Jason Rute, Jean de Becdeli\`evre, Jean Quentin, Jean-Malo Delignon, Jessica Chudnovsky, Jianchao Li, Jie Zhang, Jin Bin Ou, Joachim Studnia, Joep Barmentlo, Johannes Brandstetter, John Harvill, Jonas Amar, Jonas Schweizer, Jonathan Roberts, Jos\'ephine Delas, Josselin Somerville, Jules Depersin, Julien Denize, Julien Tauran, Julien Verlaguet, Kartik Khandelwal, Khyathi Raghavi Chandu, Kilian Tep, Konstantin Mishchenko, Kris Szczepaniak, Kush Jain, Larissa Laich, Laura Calem, Laurence Aitchison, Laurent Callot, Laurent Fainsin, L\'eo Cotteleer, L\'eonard Blier, Lo\"ic Chadoutaud, Lo\"ick Chambon, Lorenzo Proietti, Louis Faucon, Louis Martin, Louis Serrano, Lu Li, Luc Boudier, Lucas Fernandes Martins, Lucas Mebille, Lucile Saulnier, Ludovic Ho Fuh, Luis Montero, {\L}ukasz Borchmann, Maarten Buyl, Malik Hassan Raza, Manon Chossegros, Marcin Mo\.zejko, Margaret Jennings, Markus Hennerbichler, Martin Alexandre, Mathieu Poir\'ee, Mathieu Schmitt, Mathilde Guillaumin, Mathis Felardos, Matthieu Andr\'e, Matthieu Dinot, Matthieu Futeral, Maurits Bleeker, Mauro Comi, Max Mynter, Maxim Berman, Maxime Darrin, Maxime Langelier, Maxime Louis, Maximilian Augustin, Maximilian M\"uller, Mehdi Bouaziz, Melina Jingting Laimon, Mert Unsal, Mia Chiquier, Michael Pilcer, Micha{\l} Pietruszka, Micha{\l} Zaj\k{a}c, Micka\"el Seznec, Mikhail Biriuchinskii, Minh-Quang Pham, Minwoo Kang, Mircea Lica, Morgane Rivi\`ere, Namit Katariya, Natasha {\O}rregaard Klingenbrunn, Nathan Bout, Nathan Grinsztajn, Nathan Simpson, Neeraj Aggarwal, Neil Ashton, Neil Mallinar, Nick B\"uhrer, Nicolas Moreau, Nicol\`o Lucchesi, Ola Mysiak, Oliver Leicht, Olivier Bousquet, Olivier Koch, Parag Jain, Patricia Wang, Patrick Blies, Patrick von Platen, Patryk Saffer, Paul Jacob, Paul Teiletche, Paul Wambergue, Paula Kurylowicz, Pavan Kumar Reddy, Pavel Kuksa, Philippe Formont, Philippe Pinel, Philom\`ene Chagniot, Pierre Glandon, Pierre Stock, Pierre-Andr\'e Savalle, Piotr Mi{\l}o\'s, Poojan Shah, Prateek Gupta, Pravesh Agrawal, Pui Ki Chan, Quentin Desreumaux, Quentin Selle, Quentin Torroba, Quentin Wargnier, Quercus Hernandez, Ram Ramrakhya, Randall Isenhour, Ranjit Parva, Rapha\"el Balouka, Rapha\"el Rialland, Raul Perez Pelaez, Reinhard Sonnleitner, Remi Delacourt, Rishi Shah, Rob Romijnders, Rohan Shah, Rohin Arora, Romain Sauvestre, Roman Soletskyi, Ronan Riochet, Rosalie Millner, Rupert Menneer, Sagar Vaze, Samuel Barry, Samuel Belkadi, Samuel Humeau, Samuele Papa, Sanchit Gandhi, Sandeep Cranganore, Sandeep Subramanian, Sarthak Mittal, Saskia Adaime, Scott Matthews, Sean Cha, Sebastian Kaltenbach, Seungjae Jung, Shashwat Verma, Sherif Waly, Shrimai Prabhumoye, Siddhant Waghjale, Siddharth Gandhi, Simon Lepage, Simon Sorg, Soham Ghosh, Sophie Marbach, Srijan Mishra, Stanislas Lange, Steve Hong, Sumukh Aithal, Szymon Antoniak, Tarun Kumar Vangani, Teddy Turquet, Teven Le Scao, Th\'eo Cachet, Thibaut Lavril, Thomas Chabal, Thomas Coste, Thomas Defard, Thomas Foubert, Thomas Loux, Thomas Robert, Thomas Wang, Tianyu Zhang, Tim Lawson, Timoth\'ee Lacroix, Timothy Wallis, Tobias Kronlachner, Tom Bewley, Tom Edwards, Tomas Hodan, Triantafyllos Afouras, Tuhin Das, Tyler Wang, Udit Saxena, Ulrick Bl\'e, Umar Jamil, Umberto Tomasini, Valentin Mac\'e, Van Phung, Venkata Reddy, Vibha Masti, Victor Jouault, Victor Letzelter, Victor Paltz, Victor Poucheret, Vincent Maladi\`ere, Vincent Pfister, Virgile Richard, Vladislav Bataev, Wassim Bouaziz, Wen Ding Li, William Flanagan, William Havard, William Marshall, Witold J\k{e}drzejewski, Xinghui Li, Xingran Guo, Xinyu Yang, Xuanyu Zhang, Yann Dreze, Yann Viegas, Yannic Neuhaus, Yassine Turki, Yessin Moakher, Yi He, Yihan Wang, Yimu Pan, Yuying Zhu, Yves Martin des Taillades, Zaccharie Ramzi, Zhenlin Xu, Zs\'ofia Cs\'ak\'any

benchmarks

arXiv AI

Oct 28

scPyviewer: a Python-native interactive viewer from AnnData single-cell data

scPyviewer is a Python-native interactive web viewer that directly ingests AnnData objects, enabling non-programmers to explore single-cell datasets, embeddings, gene expression, metadata, and marker genes without converting to Seurat. In feature‑parity tests against three R Shiny tools, scPyviewer matches or exceeds baseline capabilities and uniquely supports cross‑dataset comparison over shared genes and matched cell types. Benchmarking on datasets ranging from 22,315 to ~313,000 cells shows scPyviewer renders all core views up to 3.6× faster, uses up to 5.2× less memory on the smallest dataset, and successfully handles 313,000 cells where Seurat fails due to memory limits. The tool also offers a Python API that returns Matplotlib figures and pandas tables for publication‑ready output.

By Hao Xuan, Yu Huang, Jiang Bian, Xiangtao Liu

rag benchmarks data-engineering

arXiv AI

Oct 28

EMMA: Structured Episodic Memory for Multi-Agent Workflow Synthesis

EMMA is a framework that treats the configuration of a multi‑agent system—controller routing, specialist prompts, and procedural skills—as structured episodic memory. It explicitly assigns ownership and limits updates through a reflector and backbone‑independent guards, ensuring edits are scoped, incremental, and bounded. Experiments on KramaBench, AppWorld, and BBEH show EMMA outperforms four baselines, achieving up to 32.5% higher performance and reducing training executions by 17–38%.

By Ruijia Niu, Kaiwen Liu, Rose Yu, Kristina Klinkner

llms agents benchmarks

arXiv AI

Oct 28

DuplexTune: Coupled-State Replay for Memory-Efficient Mixture-of-Experts Adaptation

DuplexTune is a framework that enables memory‑efficient adaptation of mixture‑of‑experts language models by replaying coupled hidden states through pretrained attention and expert modules. It introduces trainable width bridges that connect two residual streams to the backbone, with a bridge warm‑up phase before backbone adaptation while keeping router weights frozen. Experiments on Qwen1.5‑MoE‑A2.7B show that DuplexTune reduces peak memory from 64.8 GB to 38.8 GB and increases throughput from 19.9 to 25.1 samples/s, achieving competitive performance on MMLU, GSM8K, and MT‑Bench.

By Paul Raj, Simon Caldwell, Daniel Whitmore, Mutlu Mete

llms fine-tuning efficiency benchmarks

arXiv AI

Oct 28

LociArt: A Unified Paradigm for Free-Style Local Video Stylization

LociArt introduces a unified framework for free‑style local video stylization, allowing precise specification of where to stylize and which style to apply while preserving non‑target content. The core method, GRNStyle, builds on Generative Refinement Networks and uses a polar style‑aware mask to embed style identities as equal‑norm directions within selected regions. An automated pipeline generates paired supervision, leading to the LociArt‑100K dataset and benchmark, which evaluate regional style quality, non‑target preservation, and temporal consistency, and experiments show GRNStyle outperforms existing methods in these aspects.

By Yuxuan Chen, Feng Xie, Jiagao Hu, Fuhao Li, Zepeng Wang, Fei Wang, Daiguo Zhou

benchmarks

arXiv AI

Oct 28

Dense Representation Learning and Differentiable Shape Modeling for Spine Landmark Detection

The paper introduces a method for detecting spine landmarks in X-ray images by treating landmark detection as a pixel-level contrastive representation learning task. Using a generic U‑Net backbone, the approach generates more discriminative heatmaps than traditional heatmap regression, enabling better differentiation between vertebral corners. The method further incorporates learnable shape priors to resolve location ambiguities, achieving competitive accuracy on the AASCE dataset and two private datasets.

By Augustin Daridon, Sofiane Sifaoui, Ayman Assi, Mohamad Karam, Wafa Skalli, Elsa Angelini, Loic Le Folgoc

rag benchmarks

arXiv AI

Oct 28

Interpretable Deep Learning Models for Breast Cancer Diagnosis Using Medical Imaging

The study evaluates ten ImageNet‑pretrained deep learning models for classifying benign versus malignant breast cancer in the BreakHis histopathology dataset, using a patient‑level split. DenseNet121 achieved the highest performance with 99.4% accuracy and an AUC‑ROC of 0.999, outperforming all other models (p < 0.001). Explainability methods (Grad‑CAM, LIME, Integrated Gradients, LRP) consistently identified nuclei‑rich and glandular regions as key diagnostic features, demonstrating that transfer learning combined with explainable AI can produce accurate and interpretable breast cancer diagnostic tools.

By Sami Alnawali, Muhammad Atif Saeed, Abdul Hadi, Azka Atiq, Akhtar Jamil, Ahmad Din

llms fine-tuning benchmarks safety

arXiv AI

Oct 28

GAS-PFN: Graph-Aware Spatiotemporal Prior-Data Fitted Networks for Zero-Shot Traffic Forecasting

The paper introduces GAS-PFN, a Graph‑Aware Spatiotemporal Prior‑Data Fitted Network that incorporates road topology into synthetic task priors via random‑walk diffusion and uses a two‑mask Transformer architecture to separately model sensor‑specific temporal dynamics and spatiotemporal interactions. This design allows the model to adapt to new traffic networks without parameter updates, achieving a 34.0% average MSE reduction over TimePFN and a 27.7% reduction over recent industry zero‑shot forecasters across six benchmarks at a 90‑minute horizon.

By Joseph Natter, Long Vu, Yifan Zhang, Bobin Deng, Kun Suo, Rui Wu

llms diffusion benchmarks

Get the daily briefing