Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv Machine Learning
Sep 23

eXplaining to Learn (eX2L): Regularization Using Contrastive Visual Explanation Pairs for Distribution Shifts

The paper introduces eXplaining to Learn (eX2L), an interpretable framework that regularizes a classifier by penalizing similarity between Grad‑CAM maps of the main label classifier and a confounder classifier. This approach decorrelates confounding features from latent representations during training. On the Spawrious Many‑to‑Many Hard Challenge benchmark, eX2L outperforms the current state‑of‑the‑art by 5.49% in average accuracy and 10.90% in worst‑group accuracy, while also demonstrating functional domain invariance through explicit label‑nuisance decoupling.

By Paulo Mario P. Medina, Jose Marie Antonio Mi\~noza, Sebastian C. Iba\~nez
arXiv Machine Learning
Sep 23

Confidence Composition for Multiagent Language Model Systems

The paper addresses the lack of system‑level confidence estimates in multiagent language model systems such as collaborative reasoning and debate. It introduces confidence composition methods, including confidence‑aware routing and log‑odds pooling, to combine agent confidences while maintaining selective utility and probabilistic reliability. Experiments on five benchmarks with diverse model pairs show that gated‑fusion techniques improve AUARC and Brier scores compared to single‑agent and standard debate baselines, and a shared dependence discount further enhances reliability.

By Ali Elahi, Michael J. Curry, Barbara Di Eugenio
arXiv Machine Learning
Sep 23

Density-Ratio Rescoring for Imbalanced Classification Using Raking Duals and Classifier Scores

Density‑Ratio Rescoring (DRR) enhances a classifier trained with the original class prior by adding a survey‑raking dual score that reweights the majority class to match minority feature moments within a tolerance. The method standardizes both the dual and base scores, combines them with equal weight, and uses the fitted dual directly for prediction without resampling or refitting the base model. Experiments on 24 tabular benchmarks and a gene‑expression cohort show that DRR improves average precision over the standardized base on every dataset, with a mean gain of 0.034, and outperforms a shared‑dual raking‑and‑relabeling resampler on most datasets.

By Dongha Kim, Seunghwan Park
arXiv Computation and Language
Sep 23

FrontierMath Erd\H{o}s

arXiv:2609. 25050v1 Announce Type: new Abstract: We introduce FrontierMath Erd\H{o}s (FME), a benchmark of 68 Erd\H{o}s problems that are open as of August 2026.

By Tom Adamczewski (Epoch AI), Thomas F. Bloom (University of Manchester)
arXiv Computation and Language
Sep 23

ChainDoRA: Tensor-Train Factorized Weight-Decomposed Low-Rank Adaptation for Parameter-Efficient LLM Fine-Tuning

ChainDoRA is a new parameter‑efficient fine‑tuning framework for large language models that replaces the dense low‑rank factorization of LoRA with a connected Tensor‑Train (TT) chain. By separating weight magnitude and direction and using a TT rank to control representation capacity, ChainDoRA achieves higher average accuracy on seven commonsense reasoning benchmarks while dramatically reducing trainable parameters—down to 5.35 M versus 56 M for LoRA and DoRA. Ablation studies show that the TT parameterization offers controllable trade‑offs between parameter cost and accuracy.

By Ashfak Yeafi, Mehedi Hasan, Md Khairul Islam
arXiv Computation and Language
Sep 23

TelecomGPT-R1: Unified Post-Training for Reasoning Across Heterogeneous Telecom Tasks

TelecomGPT‑R1 is an open‑source family of unified telecom reasoning models that address the limitations of existing telecom‑specific and general‑purpose LLMs. It is built on a four‑axis framework—protocol, knowledge, modeling, and fault—and trained on a curated corpus of 104,880 verified question‑answer pairs with chain‑of‑thought reasoning. After supervised fine‑tuning, dynamic sampling policy optimization with task‑routed rubric rewards is used to stabilize reinforcement learning across heterogeneous telecom tasks, achieving an 89.64% mean score on seven GSMA Open Telco Leaderboard benchmarks, surpassing leading proprietary models.

By Bohao Wang, Chenwei Wu, Hang Zou, Yu Tian, Lina Bariah, Li Wei, Chongwen Huang, Yongliang Shen, Zhaoyang Zhang, Merouane Debbah
arXiv Computation and Language
Sep 23

Passes Alone, Fails Together: Benchmarking Semantic Coordination in Parallel LLM-Agent Development

Parallel coding agents can produce patches that work individually but fail when merged, a problem arising when one agent alters an interface or rule another agent depends on. The authors introduce *stale*, a benchmark for semantic coordination, evaluating patches both alone and in combination across synthetic tasks, real Django pull‑request pairs, and constructed tasks. While only one of 417 mined Django pairs showed interference after grading corrections, interference appeared in 97% of runs on constructed tasks, and a message describing the concurrent change recovered 82% of those runs.

By Haocheng Xia, Eugene Wu, Yongjoo Park
arXiv Computation and Language
Sep 23

Mining Legal Arguments in U.S. Corporate Case Law

The paper introduces an expert‑annotated dataset of 42 U.S. federal tax opinions on corporate reorganizations under I.R.C. §368, marking the first tree‑structured argument corpus in this domain. Each legal passage is labeled with one of five functional categories—Rule, Analysis, Conclusion, Background Facts, and Procedural History—and can be linked into directed support trees. Experiments demonstrate that functional labels are learnable and that supervised fine‑tuning improves within‑case retrieval, though cross‑case generalization remains weak.

By Luis Brena, William Jurayj, Gregory Deyesu, Zaid Al-Huneidi, Andrew Blair-Stanek, Benjamin Van Durme
arXiv Computation and Language
Sep 23

From Utterances to Networks: Modelling Slang Adoption and Diffusion Across Subreddits

The paper investigates how internet slang spreads across Reddit communities by combining social network analysis with linguistic context. Using large language models as scalable annotators, the authors create a benchmark for detecting slang usage and then model its adoption and diffusion. Findings reveal that users with higher bridging capital promote slang spread, while those with higher bonding capital hinder it, and that broader contextual usage delays new user adoption.

By Xiaoning Wang, Ted Underwood, Zhewei Sun
arXiv Computation and Language
Sep 23

ClusterFewshot: Improving Few-shot Optimization for LLMs workflow

ClusterFewshot is a new strategy for selecting few‑shot demonstrations in large language model workflows. It combines semantic structuring with utility‑aware scoring to build representative demonstration sets, improving accuracy over prior bootstrap‑based methods. In DSPy‑based pipelines, it substantially reduces optimization cost across multiple benchmarks while consistently outperforming earlier approaches in both standalone prompt tuning and hybrid prompt‑weight optimization.

By Omri Bar Haim, Shahar Katz, Lior Wolf
arXiv Computation and Language
Sep 23

Informed Masking: Structure-Aware Perturbation for Reinforcement Learning in Diffusion Large Language Models

Informed Masking (IM) is a new technique for aligning Diffusion Large Language Models (dLLMs) with Reinforcement Learning (RL). It identifies a systematic upstream/downstream token structure in dLLM rollouts and shows that masking downstream tokens creates better subproblems for likelihood estimation. When integrated into three state‑of‑the‑art dLLM RL methods on LLaDA‑8B‑Instruct, IM yields up to 2.01%, 8.68%, and 5.77% relative average gains on math and planning benchmarks while improving training stability.

By Xiaoyi Yu, Enver Sangineto, Pei Fu, Fiorenzo Parascandolo, Wenhui Tan, Ruikang Zhang, Rita Cucchiara, Ruihua Song, Jian Luan
arXiv Computation and Language
Sep 23

Differentiable Fuzzy Inference Layer: A Monotone, Compositional Ordinal Reasoning Head for Large Language Models

The paper introduces the Differentiable Fuzzy Inference Layer (DFIL), a dual‑path prediction head that pairs a standard classifier with a scalar‑bottlenecked branch using ordered membership functions. DFIL enforces monotonicity in the underlying quantity and enables compositional reasoning via t‑norm operations, addressing failures of standard classifier heads that treat ordinal categories as independent labels. The scalar branch also offers an interpretable interface for analyzing residual errors, and the authors demonstrate DFIL’s effectiveness on ordinal natural‑language tasks across various large language model families.

By Zhen Zhang, Amr Alanwar
arXiv Computation and Language
Sep 23

PACE-dLLM: Elastic Block Decoding via Confidence Cliff Estimation for Diffusion Language Models

The paper introduces PACE-dLLM, an acceleration method for diffusion language models (dLLMs) that uses the model’s own per‑step confidence to estimate a ‘confidence cliff’ and determine the optimal look‑ahead horizon for block decoding. By fitting this cliff in closed form at each step, PACE-dLLM sets the horizon to its saturation point and applies an independent confidence threshold for token commitment, thereby avoiding the trade‑offs inherent in fixed‑size block decoding. Experiments on reasoning and code benchmarks show that PACE-dLLM achieves the best average accuracy on open‑source dLLM backbones while delivering significant wall‑clock speedups—up to 5.23× on LLaDA and 3.06× on Dream—improving the quality‑throughput Pareto frontier.

By Xiaocheng Lu, Shuhan Guo, Ziyue Ma, Jie Zhang, Jian Liu, Jingcai Guo, Haoxuan Che, Song Guo
arXiv Computation and Language
Sep 23

Designing and Analysing Argument Mining Pipelines: Towards a Comprehensive Assessment

The paper introduces a meta‑study that reviews state‑of‑the‑art end‑to‑end argument mining (AM) pipelines. It proposes a triple‑perspective framework—linguistic, computational, and domain—to analyze how these pipelines model, compute, and incorporate domain knowledge into argument structures. The authors also outline a general design for the linguistic and computational aspects, aiming to standardize methodology descriptions and enable clearer comparisons among AM approaches.

By Siddharth Bhargava, Sara Tonelli, Patricia Mart\'in-Rodilla
arXiv Computation and Language
Sep 23

Enriching Speech Emotion Representations with Conversational Context

The paper introduces ACERT, a module that incorporates a flexible-length window of conversational context to enhance Speech Emotion Recognition (SER). By capturing emotional evolution across utterances, ACERT outperforms state‑of‑the‑art methods on IEMOCAP, sets a new context‑aware benchmark on SAFE, and achieves strong results on MELD. Ablation studies attribute ACERT’s improvements to emotional and conversational continuity rather than speaker identity or acoustic conditions.

By Arthur Peuvot, Romaric Besan\c{c}on, Ga\"el de Chalendar, Bianca Vieru, Ioana Vasilescu
arXiv Computation and Language
Sep 23

Layout-Guided Masking for GROBID: Lightweight Structural Gains in Large-Scale Scientific PDF Ingestion

The paper introduces a lightweight CPU-based extension to GROBID that uses layout-guided masking to identify figure, table, and paratext regions in scientific PDFs. By routing tokens to specialized GROBID models or discarding them, the method improves structural accuracy on PMC corpora and enhances figure caption recovery. It also achieves competitive table detection and body‑text precision compared to vision‑based GPU parsers while operating entirely on CPU and costing significantly less.

By Luca Foppiano, Sana Khamassi, Vipul Gupta
arXiv Computation and Language
Sep 23

Calibration as a First-Class Criterion in LLM Evaluation

The paper argues that calibration—how well a language model’s confidence aligns with its actual correctness—should be a standard evaluation metric for large language models (LLMs). It notes that while calibration metrics exist, they are rarely applied outside specialized NLP subfields, leading to unverified confidence scores in new models, datasets, and benchmarks. The authors highlight the risks of miscalibration both at deployment (overconfident errors causing harm) and in research workflows (affecting LLM-as-a-judge, synthetic data generation, and active learning). They call for every NLP subfield to pair its primary performance metric with a calibration score, treating calibration as an essential property of every model.

By Mario Sanz-Guerrero, Katharina von der Wense
arXiv Computation and Language
Sep 23

A retrospective analysis on the use of LLMs to study infant syntax learning

The paper reviews how large language models (LLMs) are employed to study infant syntax acquisition, focusing on the BabyLM challenge that aims for human‑level syntactic performance using developmentally realistic corpora. It critically examines dataset construction, model selection, training procedures, and syntactic evaluation methods, highlighting methodological assumptions that limit the theoretical reach of these studies. The authors find that using developmentally realistic corpora has only modest impact on benchmark performance, pointing to fundamental computational differences between LLMs and actual infant syntax learners.

By H\'elie Bazin (SCAI, SND, ISIR), Anouk Barberousse (SND), Fran\c{c}ois Yvon (MLIA)
arXiv Computation and Language
Sep 23

Capable yet Parsimonious: Extracting and Characterizing Hidden Chain-of-Thought in Frontier Models

The paper demonstrates that frontier language models can be prompted to expose their internal chain-of-thought reasoning via a simple custom tool. By comparing these extracted traces to native reasoning on open-source models, the authors confirm that the externalized reasoning aligns with genuine reasoning and outperforms no-reasoning baselines across math, science, and code tasks. They further analyze the structure of the reasoning, noting token-efficient, directed reasoning in models like GPT‑6 Astra, which externalizes only crucial steps while handling elementary ones internally.

By Xiaoyu Luo, Tao Ren, Wenrui Yu, Xiao Li, Qiongxiu Li, Johannes Bjerva