Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv AI
Sep 24

Escaping Python Dependency Hell: A Hybrid Replay-and-Repair Pipeline for Python Dependency Resolution

The paper introduces PLLM+, a hybrid pipeline for resolving Python dependency conflicts that combines deterministic steps—such as static AST inference, replaying known successful configurations from a solutions database, and live PyPI validation—with an LLM-based repair fallback. Evaluated on the HG2.9K benchmark of 2,891 failing snippets, PLLM+ successfully fixes 1,500 cases, outperforming the baseline PLLM and reducing average runtime from 368.7 to 71.8 seconds per snippet. The majority of fixes (1,495) come from replaying existing configurations, while the LLM fallback contributes only five additional solutions.

By Veronica Poweska, Ariana Oyanguren, Jessica Pourleyli, Sourena Khanzadeh, Manar Alalfi
arXiv AI
Sep 24

CAVEAT: Towards Robust Computer-Use Agents in Incentive-Misaligned Environments

CAVEAT is a new benchmark that tests computer‑use agents (CUAs) in nine online marketplace environments where platform incentives may steer agents away from user goals. The study finds that agents succeed in choosing user‑optimal products only 78.6% of the time in neutral settings, dropping to 17.3% when steering mechanisms are active. By diagnosing three failure points—priority distortion, premature narrowing of options, and early commitment—CAVEAT-Harness interventions raise user‑optimal purchasing success by 55.0%.

By Yuxuan Li, Will Epperson, Wesley Deng, Zezhou Huang
arXiv AI
Sep 24

The Gaussian Is Enough: Flow-Matching Priors Do Not Help When Fine-Tuning Large Behavior Models

The paper investigates whether using non‑Gaussian priors improves fine‑tuning of large behavior models (LBMs) for robot imitation learning. Across more than 100,000 simulation rollouts and 1,250 hardware trials on diverse tasks, the authors find that non‑Gaussian priors do not yield better fine‑tuning performance than standard Gaussian priors, except possibly at very low data fractions. Diagnostic analyses reveal that encoder training dominates fine‑tuning outcomes, while prior choice has minimal impact.

By Chen Xu, Rishi Shah, Hadas Kress-Gazit, Haruki Nishimura, Masha Itkina
arXiv AI
Sep 24

Phonemizing User-Generated Text: A Benchmark, Taxonomy, and Compositional Approach

The paper introduces UGTPhon, a grapheme-to-phoneme benchmark for user‑generated text in English, Vietnamese, and Korean, and presents a taxonomy for diagnosing pronunciation errors. It shows that existing G2P models and large language models struggle with canonical‑to‑non‑canonical text, with errors up to 66.8 PER points. A compositional G2P approach that uses exact‑match lookup and staged decoding reduces these errors and performs competitively with larger few‑shot LLMs.

By MinJu Jeon, Younghan Park, Han Sung Park, Jong-Hwan Kim, Dong-Jin Kim, Hoyeon Lee
arXiv AI
Sep 24

Meet, Compare, or Abstain: LatWeave for Deterministic Multi-Hop Question Answering on Knowledge Lattices

LatWeave is a deterministic multi‑hop question‑answering framework that structures knowledge into a multidimensional lattice and reduces QA to three operators—meet, compare, and abstain—while limiting LLM use to extraction and planning. It achieves near‑lossless performance on complete knowledge benchmarks (e.g., MetaQA) and strong results on templated multi‑hop datasets (e.g., 2WikiMultihopQA), while transparently handling incomplete knowledge through abstention. The approach offers reproducible, auditable answer paths with no performance penalty within its operating envelope.

By Yuze Ren, Shaoheng Fan, Tao Wang, Yabo Yan, Han Han
arXiv AI
Sep 24

Planned Test-Time Scaling with Coordinated Reasoning Paths

The paper introduces Planned Test-Time Scaling (PTTS), a method that replaces independent sampling of reasoning branches with a coordinated joint policy. PTTS uses a planner to generate distinct solution outlines for each branch and an executor to produce full solutions, thereby improving coverage of complementary reasoning modes. Two variants—PTTS‑ZS (zero‑shot) and PTTS‑RL (reinforcement‑learned)—demonstrate significant gains on five mathematical reasoning benchmarks, with PTTS‑RL achieving up to a 13.4‑point improvement in pass@64 over repeated sampling.

By Xueqing Wu, Langxing Bai, Hritik Bansal, Po-Nien Kung, Shuo Li, Hao Liu, Nanyun Peng, Kai-Wei Chang
arXiv AI
Sep 24

Evaluating ADC-only deep learning pipelines for breast cancer detection and segmentation using standalone diffusion-weighted MRI

The paper evaluates deep learning pipelines that use only apparent diffusion coefficient (ADC) maps from diffusion-weighted MRI (DW-MRI) for breast cancer detection and segmentation. It compares various state‑of‑the‑art models and claims to be the first comprehensive assessment of ADC‑only approaches for classification, detection, and segmentation tasks. The study highlights the potential of DW-MRI as a faster, contrast‑free alternative to dynamic contrast‑enhanced MRI for breast cancer imaging.

By Pablo Garc\'ia Marcos, Paula Puerta Gonz\'lez, Guillermo Lorenzo, H\'ector G\'omez, Covadonga del Camino, Angel Rio-Alvarez, V\'ictor M. Gonz\'alez
arXiv AI
Sep 24

TEMPS: Temporal Sentence Embeddings for Temporal Information Retrieval

The paper introduces TEMPS, a modular temporal branch that enhances semantic retrievers by adding a temporal scoring component. TEMPS resolves anchored temporal expressions into intervals, matches them to Gaussian distributions, and trains an anchor-date-conditioned encoder using grounding supervision without hand‑labeled data. On three temporal benchmarks, TEMPS improves MRR across all tested backbones and raises R@1 from 19.92 to 25.39 on the TS‑Retriever, surpassing prior temporal state‑of‑the‑art methods.

By Mourad Hassani, Julien Romero, Amel Bouzeghoub, Christian Jacquelinet
arXiv AI
Sep 24

Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark

The paper introduces a 96‑item paired benchmark for evaluating large language models (LLMs) on backtest auditing, where each flawed backtest is matched with a clean control that keeps strategy, dates, code style, labels, and reporting scaffold constant while altering a single methodological detail. A deterministic scorer distinguishes flaw recall, clean‑control false positives, evidence localization, and fix relevance. Experiments on 1440 cached audits from four text endpoints show that the DeepSeek auditor achieves perfect closed and clean‑aware code recall, yet open prompts over‑flag 93.8% of clean controls, and clean‑aware specificity is 87.5% even when recall saturates. Introducing a clean‑aware warning eliminates 20.8% false positives to 0% without affecting recall, though the budget anchor still flags many clean controls. Reporting clean‑control rates provides a clearer differentiation among models than reporting recall alone.

By Makar Ulesov, Vladislav Smirnov, Omar Ibrahim, Arsenii Bobovnikov
arXiv AI
Sep 24

Controlling Collectives of AI Agents in Reasoning Space with Spatial Transformers

The paper introduces COMPASS, a decentralized architecture that uses spatial transformers to generate local feedback tokens for large collectives of AI agents in robotics. By aggregating multi‑hop messages, COMPASS enables scalable control of up to 1024 robots, achieving cohesive flocking formations and accurate execution of natural language commands. Experiments show that structured diversity in input commands improves performance and that learned feedback tokens outperform hand‑crafted raw state feedback.

By Frederic Vatnsdal, Roshan Gopal, Romina Garcia Camargo, Vijay Kumar, Alejandro Ribeiro
arXiv AI
Sep 24

COMED: The Missing Middle Between Routing and Collaboration in Multi-LLM Inference

COMED (Controlled Model Escalation for Multi-LLM Deliberation) is a post-anchor controller that selectively engages cross-model collaboration in multi-LLM inference. It uses anchor self‑consistency, router margin, and a lightweight peer probe to accept confident answers, verify ambiguous cases, and only escalates when collaboration is likely beneficial. Experiments on medical, scientific, and general reasoning benchmarks show that COMED improves performance across 16 open‑weight settings, achieving up to +10.7 percentage points on MedQA and outperforming dense collaboration while invoking fewer models and decoded tokens.

By Norah Alballa, Wenxuan Zhang, Salma Kharrat, Fares Fourati, Zafar Ayyub Qazi, Mohamed Elhoseiny, Marco Canini
arXiv AI
Sep 24

Hidden not Deleted: How Networks Suppress Entangled Features

The paper investigates how linear concept erasure methods fail when features are densely superposed, showing that such methods destroy both target and non‑target features. It demonstrates that networks trained with gradient descent can avoid this failure but converge to one of two distinct circuit‑level solutions—mirror or shadow—depending on initialization. The authors map this bifurcation as a function of feature entanglement, reveal a stable attractor structure, and use causal interventions to show that both solutions leave a measurable trace of the erased feature that can be recovered with a single scalar patch.

By Akash Samanta, Manish Pratap Singh, Debasis Chaudhuri
arXiv Machine Learning
Sep 24

ChipMEM: Verification-Grounded Memory for EDA Agents

ChipMEM introduces a verification‑grounded memory layer for electronic design automation agents that combines cross‑task procedural memory with within‑trajectory statistical guidance. The procedural component stores a skill only after it passes synthesis, simulation, or formal checks, while a Bayesian component ranks recovery strategies based on tool‑call outcomes. Experiments on RTLRewriter‑Bench and CVDP tasks show that ChipMEM improves equivalence‑passing outputs and area metrics compared to agents without memory.

By Abdulrahman AlRabah, Joshua Mabry, Dilek Hakkani-T\"ur, Abdussalam Alawini, Hamid Shojaei, Kartik Hegde, Sandesh Adhikary
arXiv Machine Learning
Sep 24

Confidence Falls Short: Asymmetric Certainty Gains from Optimization Hinder Multimodal Classification

The paper identifies that in multimodal learning, optimization often produces asymmetric certainty gains, with the stronger modality becoming more confident than the weaker one, which leads to imbalanced contributions and suboptimal performance. The authors attribute this issue to unimodal characteristics and propose a Max Confidence Regularization (MaxCR) method that tracks each modality’s semantic confidence via a nonlinear sparsity measure and applies max suppression and excitation to balance confidence levels. Experiments on standard datasets demonstrate that MaxCR improves overall performance compared to state‑of‑the‑art multimodal baselines.

By Longfei Huang, Xiangyu Wu, Yang Yang
arXiv Machine Learning
Sep 24

NeuroRule: Making Black-Box Neural Networks Explainable through Rule-set Evolution

NeuroRule is a knowledge distillation framework that transforms high‑capacity neural networks into explainable rule‑sets. It adapts the EVOTER rule‑set evolution infrastructure to evolve propositional logic expressions that capture the neural network’s performance. The approach includes a conciseness objective to enhance explainability and demonstrates viability even without access to the original training data.

By Tapaswini Kodavanti, Hormoz Shahrzad, Risto Miikkulainen
arXiv Machine Learning
Sep 24

How Sensitive Are LLM Leaderboard Claims to Hidden Model Selection?

The paper investigates how many hidden model variants can exist while still supporting a published leaderboard margin that shows a provider’s advantage over a fixed comparator. It derives a sensitivity curve for a fixed candidate family under a Gaussian margin model, linking the maximum number of hidden variants to a lower bound on within‑family correlation. Using this framework, an audit of 394 adjacent‑rank claims on the Open LLM Leaderboard found that 391 lack statistical support before any correction, and that certification of the remaining claims depends on assumptions about the hidden family’s correlation.

By Chen Yang, Xianyang Zhang, Jun Chen
arXiv Computer Vision
Sep 24

From ECG Signals to Representative-Morphology Heatmaps for Biometric Recognition

The paper introduces representative‑morphology heatmaps, a deterministic ECG‑to‑image representation that averages the five beats closest to the block mean within each ten‑beat block, producing either a conventional trace or a dense cardiac‑time‑by‑lead heatmap. Experiments on PTB, ECG‑ID, and MIMIC‑IV‑ECG‑DEMO show that heatmaps consistently improve verification and identification performance across 15 compact models, reducing EER by an average of 9.59 percentage points and increasing Rank‑1 by 24.69 points. The study also demonstrates that ImageNet initialization benefits multilead datasets, that performance does not scale monotonically with model size, and that useful channel combinations vary by cohort and biometric task.

By Athanasios Angelakis, Marta Gomez-Barrero
arXiv AI
Sep 24

PhyMo: A Physical-Field Modality for Multimodal AI4Physics

The paper introduces PhyMo, a physics‑grounded multimodal framework that uses a physical‑field modality to represent heterogeneous measurements via PDE‑associated operators. It follows a three‑stage learning process: pretraining a physical‑field encoder with PDE residual supervision, aligning its representations with visual embeddings in a shared latent space, and applying downstream prediction heads to the fused multimodal representations. Experiments on five diverse physical datasets show that PhyMo outperforms the strongest baseline on each dataset, establishing its effectiveness for multimodal representation learning in AI for Physics.

By Henan Sun, Haitao Hu, Jin Liu, Jianfeng Zhang, Lujia Pan, Nuo Chen, Jia Li
arXiv AI
Sep 24

FLEET: From Logits Entropy to Enhanced Trajectories in Text Generation

FLEET is a new method for text generation that adds a memory mechanism to large language models. It represents each generation as a sparse trajectory of high‑entropy states and uses these trajectories to compute per‑token utility scores that adjust the logits. Benchmarks show that FLEET matches the accuracy of repeated sampling while being three times faster and improving accuracy on complex coding tasks, all with minimal changes to existing pipelines.

By Oleksii Streltsov, Oleksandra Vitko
arXiv AI
Sep 24

InGuard: Towards Generalized Inner Guardrail for Safe Text-to-Image Generation

InGuard introduces an inner guardrail for text-to-image generation that operates within the model’s own representations, avoiding external classifiers. It grades prompts using the text encoder’s embeddings, modifies risky embeddings with SAGE to produce safe images, and employs a latent detector to halt generation early. Evaluated on the RevGen Safety Benchmark, InGuard achieves a 97.9–98.8% safety rate across five open-weight models while reducing benign disturbances, model parameters, and denoising steps.

By Zeyu Wang, Xiaodan Li, Zhiwen Li, Yuefeng Chen, Hui Xue