arXiv AI
2d ago
No Universal Signal Predicts Sample-Level LLM Regression under Version Updates
arXiv:2608. 13607v1 Announce Type: new Abstract: Frontier LLMs are updated frequently and typically outperform their predecessors in aggregate.
No Universal Signal Predicts Sample-Level LLM Regression under Version Updates
Frontier LLMs are updated frequently and typically outperform their predecessors in aggregate. But aggregate gains say little about individual samples: an update can still cause sample-level regression, where a response correct under the old model becomes incorrect under the new one.
Related stories
arXiv AI
Aug 10
Counterfactual Simulation Training for Chain-of-Thought Faithfulness
arXiv:2602. 20710v2 Announce Type: replace Abstract: Inspecting Chain-of-Thought reasoning is among the most common means of understanding why an LLM produced its output.
arXiv AI
Jul 21
Oracle Gap and Signal Fidelity: A Fixed-Pool Diagnostic for Test-Time Collaboration
arXiv:2607. 17531v1 Announce Type: cross Abstract: Test-time collaboration, including self-consistency, best-of-N selection, critic models, and verifier pipelines, is often credited with broadly improving LLM reasoning, yet its gains are uneven and sometimes negative.
arXiv Machine Learning
Jun 19
Predicting Mergeability of Parameter-Efficient Fine-Tuning Updates
arXiv:2606. 19549v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) makes it cheap to train many domain- and task-specific language model adapters, but whether two adapters can be merged is usually discovered only after both have been fully trained and evaluated.
arXiv Machine Learning
Aug 3
Reflection or Re-Generation? Why LLM Revision Fails Where Human Revision Succeeds
arXiv:2607. 28908v1 Announce Type: new Abstract: Reflection, the ability to revisit and revise prior reasoning, is central to how humans improve their answers.
arXiv AI
1d ago
Measuring Reward Hacking and Reasoning-Answer Decoupling Under Position-Confounded Optimization
arXiv:2608. 15445v1 Announce Type: new Abstract: When a reward is correct on every training example yet consistent with more than one goal, a model can acquire an unintended one, a failure known as goal misgeneralization.
arXiv Machine Learning
Jul 2
Clotho: Measuring Task-Specific Pre-Generation Test Adequacy for LLM Inputs
arXiv:2509. 17314v4 Announce Type: replace-cross Abstract: Software increasingly relies on the emergent capabilities of Large Language Models (LLMs), from natural language understanding to program analysis and generation.
arXiv Machine Learning
1d ago
Conditional Evaluation of Language Models with Cheap Auxiliary Signals
arXiv:2608. 16210v1 Announce Type: new Abstract: Aggregate accuracy hides where models succeed and fail.
arXiv AI
Aug 7
Online Reasoning Calibration: Test-Time Training Enables Generalizable Conformal LLM Reasoning
arXiv:2604. 01170v2 Announce Type: replace-cross Abstract: While test-time scaling has enabled large language models to solve highly difficult tasks, state-of-the-art results come at exorbitant compute costs.
arXiv AI
Jun 9
Prescriptive Scaling Reveals the Evolution of Language Model Capabilities
arXiv:2602. 15327v2 Announce Type: replace-cross Abstract: Machine learning model performance improvements tend to arise from competition and application.
arXiv AI
Jul 15
Rethinking Reward Models for Multi-Domain Test-Time Scaling
arXiv:2510. 00492v3 Announce Type: replace Abstract: The reliability of large language models (LLMs) during test-time scaling is often assessed with \emph{external verifiers} or \emph{reward models} that distinguish correct reasoning from flawed logic.
arXiv AI
Jul 10
What LLM Forecasters Know but Don't Say: Probing Internal Representations for Calibration and Faithfulness
arXiv:2607. 08046v1 Announce Type: cross Abstract: Large language models fine-tuned for forecasting can be accurate yet poorly calibrated, and their chain-of-thought (CoT) reasoning may not faithfully reflect the evidence behind a forecast.