Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv AI
Sep 15

Towards a Mechanistic Understanding of Propositional Logical Reasoning in Large Language Models

The paper investigates how large language models perform propositional logical reasoning by conducting a causal mechanistic analysis on the PropLogic-MI benchmark. It identifies four interlocking mechanisms—Staged Computation, Information Transmission, Fact Retrospection, and Specialized Attention Heads—that organize the reasoning process across layers. The study demonstrates that these mechanisms recur across different model families, rule categories, and reasoning hops, indicating a structured, layer‑organized internal process for propositional reasoning.

By Danchun Chen, Qiyao Yan, Chenpeng Wang, Liangming Pan
arXiv Computation and Language
Sep 15

Auditing Generative Audio Calls for Known-Task Audio-LLM Evaluatio

The paper investigates how to evaluate generative audio large language models (Audio‑LLMs) on known closed‑set tasks by separating the decision to call a generative model from the use of acoustic evidence. It introduces a controlled call‑decision framework where a policy can choose between a transcript label, encoder evidence from CLAP, AST, or WavLM, or a generative call to Qwen2‑Audio, Qwen2.5‑Omni, or MOSS‑Audio, and measures the impact of generative calls on accuracy. Results on the VocalSound dataset show that while transcript‑only accuracy is low (0.296), encoder‑based controls achieve high accuracy (≈0.85) without any generative calls, and adding generative calls yields only a marginal improvement (0.925 vs. 0.921).

By Mengzhe Geng
arXiv Computation and Language
Sep 15

Enabling Streaming User Transcription in Full-Duplex Speech-to-Speech Models

The paper introduces a lightweight ASR head that can be added to full‑duplex speech‑to‑speech models, enabling real‑time user transcription without major architectural changes. The method adds only a few parameters and preserves full‑duplex conversational features such as turn‑taking and barge‑in. Experiments show a streaming WER of 10.21% within the duplex framework and 7.73% when trained as a standalone ASR model, matching state‑of‑the‑art performance.

By Ke Hu, Nourchene Ferchichi, Edresson Casanova, Ankita Pasad, Elena Rastorgueva, Chen Chen, Nithin Rao Koluguri, Piotr Zelasko, Yifan Peng, Hainan Xu, Zhehuai Chen, Boris Ginsburg
arXiv Computation and Language
Sep 15

An Empirical Analysis of Factual Errors in Human-Written Text and Its Application to Factual Error Detection

The paper presents an empirical study of factual errors in human-written text, focusing on corrections in newspaper articles to build a taxonomy of common mistakes such as kanji misconversions and unit errors. It evaluates large language models’ ability to detect these errors, finding that even advanced models like GPT‑5.4 achieve only a 52% word‑level F1 score on synthetic data, underscoring the difficulty of the task. The work highlights the gap in research on factual error detection in human writing compared to LLM hallucinations.

By Kazuma Iwamoto, Kazumasa Omura, Shotaro Ishihara
arXiv AI
Sep 15

JaxAHT: A JAX-Based Library for Ad Hoc Teamwork

JaxAHT is a new open‑source library built with JAX that speeds up and standardizes research in Ad Hoc Teamwork (AHT). It offers a unified framework for generating teammates, training ego agents, and evaluating performance against unseen partners, delivering roughly 95× faster wall‑clock times than comparable PyTorch implementations. The library also supplies a diverse set of evaluation teammates for Level‑Based Foraging, Overcooked, and Hanabi, and is used to run a large‑scale benchmark that shows no single algorithm dominates and that agent modeling mainly helps in role‑based, diverse teammate settings.

By Caroline Wang, Rolando Fernandez, Zelal Su Mustafaoglu, Montek Kundan, Jiaxun Cui, Lingyun Xiao, Zhihan Wang, Di Yang Shi, Aditya Madhan, Johnny Liu, Arrasy Rahman, Peter Stone
arXiv Computer Vision
Sep 15

Hyper-LLaVA: Hyperbolic Uncertainty-aware Modality-Balanced Routing for Multimodal Continual Instruction Tuning

Hyper-LLaVA introduces a hyperbolic uncertainty‑aware, modality‑balanced routing mechanism for multimodal continual instruction tuning. It improves intra‑modality task matching by evaluating sample‑to‑task distribution similarity in hyperbolic space and mitigates unreliable modality influence by quantifying task‑matching ambiguity to adaptively balance across modalities. The approach achieves significant performance gains over existing methods.

By Kunlun Xu, Yanqin Zhang, Wenwen Qiang, Jiahuan Zhou
arXiv AI
Sep 15

LLM-Microscope: Uncovering the Hidden Role of Punctuation in Context Memory of Transformers

The paper introduces LLM-Microscope, a toolkit for measuring how Large Language Models encode contextual information at the token level. It shows that seemingly minor tokens—such as determiners, stopwords, and punctuation—carry surprisingly high contextual weight, and removing them degrades performance on benchmarks like MMLU and BABILong-4k. The study also finds a strong link between contextualization and linearity, indicating that the transformation between layers can be approximated by a single linear mapping when tokens are well contextualized.

By Anton Razzhigaev, Matvey Mikhalchuk, Temurbek Rahmatullaev, Elizaveta Goncharova, Polina Druzhinina, Ivan Oseledets, Andrey Kuznetsov
arXiv AI
Sep 15

CoMem: Collective-Individual Memory Synergy for Evolutionary Multi-Agent Systems

CoMem introduces a memory architecture for multi-agent systems that blends private experience with shared knowledge. It includes Private Experience Sedimentation to retain useful individual memories, Collective Wisdom Curation to filter widely proven ideas for sharing, and Parallel Dual-Stream Retrieval to draw from both personal and group memories while maintaining diversity. Experiments on ALFWorld and PDDL benchmarks demonstrate that CoMem improves overall performance and reduces memory pollution.

By Chengxin Yu, Zhaoxin Fan, Faguo Wu, Hongwei Zheng, Yun Zhou, Zhiyu Li
arXiv Machine Learning
Sep 15

MuPlon: Multi-Path Causal Optimization for Claim Verification through Controlling Confounding

MuPlon is a new framework for claim verification that models claims and evidence as a fully connected Claim‑Evidence Graph. It tackles two confounding problems—data noise and data biases—by applying a dual causal intervention strategy: a back‑door path that adjusts node weights to reduce noise and strengthen relevant connections, and a front‑door path that extracts key subgraphs, builds reasoning paths, and uses counterfactual reasoning to remove biases. Experiments show that MuPlon surpasses existing methods and achieves state‑of‑the‑art performance.

By Hanghui Guo, Shimin Di, Pasquale De Meo, Zhangze Chen, Jia Zhu
arXiv Machine Learning
Sep 15

Isolation-based Spherical Ensemble Representations for Tabular Anomaly Detection

The paper introduces ISER, an isolation-based method for unsupervised tabular anomaly detection that uses hypersphere radii to encode local density and maintains linear time and constant space complexity. ISER builds ensemble representations where smaller radii indicate dense regions and larger radii indicate sparse regions, and it employs a similarity-based scoring method that compares these representations to a theoretical anomaly reference pattern. Experiments on 20 real-world datasets show that ISER outperforms 12 state‑of‑the‑art methods, including an enhanced Isolation Forest.

By Yang Cao, Sikun Yang, Hao Tian, Kai He, Lianyong Qi, Ming Liu, Yujiu Yang, Hong-Kun Zhang