Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv Machine Learning
Oct 12

Deep End-to-End Survival Analysis with Temporal Consistency

The paper introduces a new survival analysis algorithm that leverages reinforcement learning concepts, specifically a delayed target network inspired by Deep Q-Networks, to enforce temporal consistency across consecutive survival predictions. This framework supports complex architectures and mini‑batch, end‑to‑end training, and demonstrates improved performance on long‑sequence datasets compared to benchmarks. Ablation studies reveal that the delay in target updates significantly affects predictive accuracy and training stability.

By Mariana Vargas Vieyra, Pascal Frossard
arXiv Machine Learning
Oct 12

Bayesian Nonparametric Mixed-Effect ODEs with Gaussian Processes

The paper introduces MEGPODE, a Bayesian nonparametric mixed‑effect ODE framework that models each subject’s dynamics as a combination of a shared population component and a subject‑specific deviation, both governed by Gaussian process priors. It avoids repeated ODE solves by representing latent trajectories with state‑space GP priors and penalizing derivative discrepancies on a finite time grid, enabling a single variational objective for joint inference of trajectories and vector fields. Experiments on heterogeneous oscillatory and biomedical ODE benchmarks show that MEGPODE outperforms strong baselines in recovering population‑level fields and predicting subject‑level trajectories.

By Julien Martinelli, Maksim Sinelnikov, Harri L\"ahdesm\"aki, Quentin Clairon, M\'elanie Prague
arXiv Machine Learning
Oct 12

LLM4Trust: Exploring the Capabilities of Large Language Models for Trust Evaluation

The paper introduces LLM4Trust, a benchmark framework that evaluates large language models (LLMs) for trust assessment in cybersecurity. It constructs trust graphs to represent five fundamental trust properties, designs property‑understanding tasks, and tests eight LLMs across nine prompting strategies. The study identifies effective LLM‑prompt combinations, applies them to real datasets, proposes methods to handle context limits, and demonstrates that LLMs can understand trust properties while highlighting vulnerabilities and high inference costs, for which defense mechanisms and batch inference are suggested.

By Jie Wang, Yanbo Sun, Zheng Yan, Jiahe Lan, Elisa Bertino
arXiv Machine Learning
Oct 12

Approximation of differential entropy in Bayesian optimal experimental design

The paper presents a method for approximating differential entropy in Bayesian optimal experimental design, focusing on cases where the likelihood entropy is design‑independent or explicitly computable, such as additive noise models. By reducing the expected information gain to a single integral, the authors propose using Monte Carlo or quasi‑Monte Carlo surrogates for the evidence density while evaluating differential entropy with standard techniques, thereby avoiding extra likelihood evaluations. Theoretical analysis shows convergence rates comparable to or better than existing methods, and numerical experiments support these findings.

By Chuntao Chen, Tapio Helin, Nuutti Hyv\"onen, Yuya Suzuki
arXiv Computation and Language
Oct 12

SWE-Duel: Towards a Self-Scaling Arena for Coding Agents

SWE‑Duel proposes a self‑scaling benchmark for coding agents, where tasks and oracles are generated at evaluation time to prevent data leakage and adapt difficulty to participant performance. In the arena, two agents take turns as Red and Blue: Red proposes a feature, injects a subtle bug, and must repair it blind, while Blue reviews the pull request without knowing the bug and must pass all tests. A tournament of 11 model‑harness participants on 15 repositories across five languages showed GPT‑5.5/​Codex CLI as the top solver and Claude Opus 4.8/​mini‑swe‑agent as the top task author, with newer model versions producing harder tasks for older ones.

By Wei Minn, Zhensu Sun, Lwin Khin Shar, David Lo
arXiv Computation and Language
Oct 12

perturber: A Stateless Service for Semantics-Preserving Text Perturbations

The paper introduces perturber, a stateless service and Python library designed to apply single, semantics‑preserving text perturbations. It offers a catalog of 68 perturbations across 11 families, enabling reproducible, isolated application of each perturbation without coupling to a specific benchmarking stack. The authors evaluate the tool’s correctness, reproducibility, and cost, and provide a live playground and introspectable catalog for benchmark selection.

By Khoi-Nguyen Tran
arXiv Computation and Language
Oct 12

Time Delay System for Sequence Modeling

The paper introduces a new sequence modeling framework based on nonlinear time-delay systems that dynamically estimates effective delay using semantic similarity between current and past inputs. It proposes a differentiable semantic lag estimator for content-aware soft attention, a diagonal state space with O(1) per-step complexity, and an interpretable fuzzy semantic modulation mechanism that balances immediate processing with historical recall. Experiments on text classification and generation show that this approach outperforms state‑of‑the‑art methods.

By Youshan Zhang, Jialu Li
arXiv Computation and Language
Oct 12

Evaluating LLM Creativity through Long-Tail Performance

The paper proposes that creativity in large language models can be measured by performance on the long tail of a reference input-output distribution, using n‑gram absence from pretraining data as a novelty proxy. Across 15 benchmarks and 26 open‑weight models, the authors find that creativity benchmarks occupy further tail regions than standard benchmarks and that higher‑tail subsets better align with human creativity judgments. They introduce a new diagnostic, q@Tail(α), to assess mean task quality on the tail fraction of existing benchmarks, revealing significant potential for improving LLM creativity through stronger tail performance.

By Yichen Wang, Tenghao Huang, Vishakh Padmakumar, Mina Lee
arXiv Computation and Language
Oct 12

KV-Repurpose: A Testbed for Evaluating Non-Prefix KV-Cache Reuse

KV-Repurpose is an open‑source, vLLM‑based system that enables segment‑wise KV‑cache reuse beyond exact prompt prefixes in large language model serving. It integrates cache‑aware segment scheduling, prompt construction, capacity management, workload generation, and runtime monitoring to evaluate non‑prefix KV‑cache reuse. Experiments show that naive segment reuse maintains output quality for single‑document QA and summarization, while significantly improving cache hit rates and reducing time‑to‑first‑token compared to prefix caching, especially under Zipfian RAG‑style workloads.

By Taiga Ikeda, Daisuke Miyashita, Jun Deguchi
arXiv Computation and Language
Oct 12

From Structure to Validity: Mechanism-Guided Generation and Refinement of LLM Rubrics

The paper investigates what makes rubric design effective for evaluating language models, analyzing criterion organization, operationalization, and score aggregation. It finds that a well-structured organization supports comprehensive criteria, semantic specification alone is insufficient for accurate judgments, and complex aggregation yields limited gains. Based on these insights, the authors propose ARTS‑Hybrid, an adaptive tree search with hybrid refinement that improves both evaluation performance and rubric quality, and shows promise for guiding model optimization.

By Feifei Qian, Xuan Chen, Lixin Cui, Chaoyang Mei, Fanlin Meng, Shuyao Xiao, Ming Cui, Junxi Yin, Lu Bai
arXiv Computation and Language
Oct 12

Multi-Stream Attention Residuals

The paper introduces Multi-Stream Attention Residuals (MSAR), an extension of Attention Residuals that applies multi-head depth attention inspired by multi-head sequence attention. MSAR uses multi-scale causal convolutions to provide diverse contextual views for each head, then adaptively mixes the retrieved streams into a single-stream input, avoiding persistent multi-stream storage. Across 12 downstream benchmarks at 18B and 28B MoE scales, MSAR outperforms prior residual variants and achieves a 1.58× compute speedup over the standard residual baseline.

By Xiaohan Qin, Xiangdong Zhang, Tuo Dai, Sunan Zou, Huaijin Wu, Yebin Yang, Zhuo Xia, Lin Yao, Yuliang Liu, Junchi Yan
arXiv Computation and Language
Oct 12

Personalization Without Stereotyping: Do Speech LLMs Know When to Turn a Deaf Ear?

The paper introduces BASIS-Bench, a benchmark that separates demographic robustness, stereotypical bias, and personalization in speech large language models (LLMs). It evaluates eight recent speech LLMs, finding that while many models detect demographic cues, they rarely use them for appropriate personalization, and explicit demographic labels can worsen stereotypical associations. The study underscores the necessity of training speech LLMs to discern when demographic information should influence responses.

By Kshitij Ambilduke, Caroline Brun, Thibaut Thonet, Vassilina Nikoulina, Marcely Zanon Boito
arXiv Computation and Language
Oct 12

REFLECT: Role-Aware Detection of Sentiment, Emotion, Politeness, and Formality

The paper introduces REFLECT, a role-aware transformer that integrates Semantic Role Labeling (SRL) configurations into self‑attention to better detect sentiment, emotion, politeness, and formality (SEPF) in dialogue. By fusing role-aware representations with contextual embeddings through a learned gate, REFLECT achieves significant improvements over BERT baselines on emotion, politeness, and formality tasks, especially for low‑frequency and complex cases, while sentiment gains are minimal. The approach also enhances interpretability and generalizes to real‑world mental‑health data.

By Sangpil Youm, Annabel Stocks Natalias, Hayden Heninger, Leah Jones, Xiaoyang Jin, Xiaoya Zhang, Bonnie Dorr
arXiv Computation and Language
Oct 12

When VLMs Destroy What They Know: Spatial Representations Fall Below Lexical Priors

The paper introduces EICS, a metric for assessing whether Vision‑Language Models (VLMs) consistently encode concepts under controlled one‑concept interventions, and CASP‑VL, a benchmark of causally controlled spatial perturbations derived from Visual Genome scene graphs. Experiments across multiple VLM architectures reveal that spatial relations are encoded far less faithfully than lexical information, even when models perform well on compositional benchmarks. The study shows that relational information is retained at the predicate token level but disappears when sentences are aggregated into a global representation, highlighting a systematic failure invisible to output‑based evaluation.

By Shubh Garg, Chen Shani
arXiv Computation and Language
Oct 12

Les Mis\'erables Dialogue: A film-based network representation of social communication

The paper presents a new dataset called the Les Misérables Dialogue temporal text network, created from the 1998 film adaptation of the novel. This network explicitly records speakers, addressees, other recipients, the content of each utterance, and its timing. Additionally, the authors generate multilayer network representations based on ten social dimensions, and provide a general methodology for constructing similar communication datasets from narrative sources.

By Theodora Moldovan, Georgios Panayiotou, Davide Vega
arXiv Computation and Language
Oct 12

Not All Correct Fixes Run Alike: An Empirical Study of Patch Performance in Repository-Level Issue Resolution

The study examines whether the built‑in FAIL_TO_PASS (F2P) tests in repository‑level issue‑resolution benchmarks can be used to evaluate the runtime performance of patches that pass correctness tests. Using four advanced language models paired with SWE‑agent, the authors measured instruction counts and wall‑clock times for 262 cases and applied Welch ANOVA to detect performance differences. They found that many patches exhibit significant runtime disparities, that performance rankings often contradict issue‑resolution rates, and that some architectural regressions are revealed only through performance analysis.

By Han Peng, Xing Hu, Feifei Niu, Huan Li, Xin Xia
arXiv Computation and Language
Oct 12

OARelatedWork: A Large-Scale Dataset of Related Work Sections with Full-texts from Open Access Sources

OARelatedWork is a large‑scale dataset for generating entire related‑work sections from open‑access papers, comprising 94,450 papers across multiple domains. It includes full texts of cited works and is designed so every source paper is available in full, enabling controlled evaluation of full‑text related‑work generation. The paper benchmarks various models, shows that large language models struggle with massive full‑text contexts, and introduces a statement‑level evaluation framework to better assess long‑form structured outputs.

By Martin Docekal, Martin Fajcik, Pavel Smrz
arXiv Computer Vision
Oct 12

Scaling Multi-Teacher Distillation for Digital Pathology

The paper introduces a scalable multi-teacher distillation (MTD) framework for digital pathology, addressing the poor performance of standard MTD when many teachers are used. By learning teacher‑specific representations through techniques such as learnable teacher tokens, a tailored attention scheme, mixture‑of‑experts layers, and a contrastive loss, the method scales effectively, enabling compact student encoders that are up to 10× more efficient than larger teacher models while matching or surpassing their performance on 39 tile‑ and slide‑level tasks. The authors release a new collection of efficient foundation models, OMNI, trained from ten state‑of‑the‑art teachers and provide an analysis of the complementary teacher‑specific representations.

By Sofi\`ene Boutaj, Pierre Marza, Varun Belagali, Dimitris Samaras, Maria Vakalopoulou, Stergios Christodoulidis
arXiv Computer Vision
Oct 12

Feature Competition and Its Representational Origins in Vision Language Models: Evidence from the Stroop Paradigm

The paper investigates how vision‑language models (VLMs) resolve conflicts between image features using a Stroop‑style benchmark. By prompting contrastive and generative VLMs on images where text and color disagree, the authors find a consistent bias toward textual cues, with models preferring to respond based on text rather than color. Analysis of the embedding space reveals that textual signals are represented more strongly and separably than color signals, explaining why steering models toward text is easier than toward color.

By Nurb\"uke Teker, Rui Xiao, Zeynep Akata, Shuchen Wu
arXiv Computer Vision
Oct 12

Win-Win Fusion: Multi-Modal Image Fusion with Visual-Semantic Collaboration

The paper introduces W2F, a visual‑semantic mutual‑reinforcing framework for multi‑modal image fusion (MMIF). It enhances feature complementarity with a segmentation‑guided fusion loss and an embedded information restoration prior, and employs interactive gated mixed attention (IGM‑Att) and progressive coupling attention (PC‑Att) to fuse local and global information. A collaborative mechanism with a feature search module and gradient correction further aligns visual and semantic fusion, leading to state‑of‑the‑art visual appeal and decision accuracy.

By Hao Zhang, Xuhui Zuo, Jiayi Ma