arXiv:2607. 12767v1 Announce Type: new Abstract: Multiple-choice benchmarks that rank candidate completions by conditional log-probability suffer from a length bias: because log-probabilities sum over tokens, longer answers tend to be penalized relative to shorter ones in practice.
arXiv:2608.30372v1 Announce Type: new Abstract: As Large Language Models rapidly advance, performance on standard multiple-choice question answering (MCQA) benchmarks is reaching saturation. While th...
Selecting the best response from multiple small-model samples using a stronger scorer is a simple inference-time strategy, but fails when the small model has already committed to incorrect reasoning paths. PRM guided search avoids this by scoring candidate continuations during generation, but requires a reward model trained with step-level labels.
arXiv:2610.25768v2 Announce Type: replace Abstract: Large language models (LLMs) often generate more reasoning tokens than needed. Limiting the reasoning budget reduces cost, but forcing an answer fr...
The paper introduces Length‑Conditioned Majority Voting (LCMV), a test‑time training method that enhances the standard Majority Voting (MV) technique for Large Language Models (LLMs). LCMV incorporates an empirical bias that longer Chain‑of‑Thoughts (CoTs) are less likely to be correct by modeling a length‑conditioned prior with a logistic function. This prior is adapted on unlabeled test data without fine‑tuning the LLM, and the resulting weighted votes consistently outperform standard MV and other length‑ or confidence‑based aggregation strategies across multiple reasoning benchmarks.
The paper introduces a Bayesian Bradley‑Terry model (BBTM) that employs Polya‑Gamma data augmentation for efficient Gibbs sampling and quantifies uncertainty in latent abilities and ranks. Three active‑learning strategies based on the knowledge‑gradient principle are proposed, using Kendall’s tau distance, posterior rank entropy, and posterior separation as acquisition functions, with approximate one‑step updates to reduce computational cost. The authors also present Posterior Separation Reliability (PSR) as an internal indicator of ranking reliability, and demonstrate through synthetic experiments that BBTM outperforms preference‑based Bayesian comparative judgement, with the entropy‑based strategy excelling at low‑to‑moderate noise and the posterior‑separation strategy at high noise levels.
Majority Voting (MV) is a simple and widely used strategy which improves the reasoning accuracy of Large Language Models (LLMs) by allocating additional inference-time computation. Specifically, in MV...
Comparative judgement uses pairwise comparisons to rank items, but recovering accurate rankings from a limited number of judgements depends on both the inference method and the pairs selected. We deve...
arXiv:2606. 01682v1 Announce Type: cross Abstract: Selecting the best response from multiple small-model samples using a stronger scorer is a simple inference-time strategy, but fails when the small model has already committed to incorrect reasoning paths.
arXiv:2608. 07827v1 Announce Type: new Abstract: Confidence estimation for large language models (LLMs) aims to estimate the probability that a generated answer is correct, while calibration aligns these estimates with empirical accuracy.
The paper compares two common ways of evaluating large language models (LLMs): prompting them to answer questions directly and scoring candidate answers using likelihood-based metrics. The authors introduce a new protocol that ranks declarative statements derived from question–answer pairs, and test it across 95 decoder-only models (0.1B–104B parameters) on 10 multiple-choice QA datasets. They find that while prompted answering accuracy improves sharply with model scale and instruction tuning, statement‑likelihood ranking accuracy stays relatively stable, indicating that the two evaluation methods probe different aspects of model behavior.
arXiv:2607. 27023v1 Announce Type: new Abstract: Evaluating large generative models across benchmarks is time-consuming and computationally expensive.
MAPLE is a framework that characterizes how Knowledge Extraction Enhancement (KEE) techniques influence parametric knowledge extraction in large language models. It introduces diagnostic metrics that separate effects on surface answers, probability distributions, and beyond-sampling recovery, measuring answer recovery, disruption, token rank shifts, and missed answer recovery. The authors evaluate six KEE methods across three model families—OLMo2, Qwen2.5, and Llama3—showing distinct effects and trade‑offs, and argue that MAPLE provides deeper insight than aggregate accuracy alone.