arXiv AI
Cheap Verifiers, Large Blind Spots: Measuring the Reliability Cost of Cost-Saving Cascades
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jun 15
When Good Verifiers Go Bad: Self-Improving VLMs Can Regress on New Tasks
arXiv:2606. 14629v1 Announce Type: cross Abstract: Verifier-driven self-DPO is a common recipe for self-improving production visual-language models.
arXiv AI
Aug 10
Ratchet: How Reliable Must an LLM Judge Be to Retire a Skill?
arXiv:2605. 22148v3 Announce Type: replace Abstract: A large language model (LLM) agent that writes and edits its own skill library must also decide which skills to keep, from one noisy scalar per skill.
arXiv AI
6d ago
Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams
The study evaluates large language model (LLM) graders on two computer‑science exams, testing 171 configurations of closed‑ and open‑weights models. While the best LLM configuration achieved a mean absolute error of 1.64/35—better than the 2.61/35 error between two human graders—its performance was highly sensitive to the prompt. A short "strict grader" preamble caused most open‑weight models to exceed acceptable error thresholds or stop grading entirely, whereas fine‑tuning with a single LoRA adapter restored parity with human graders and reduced sensitivity to harsh prompts.
arXiv AI
Aug 26
The Shadow Price of Intelligence: Quality Degradation in LLM Inference as a Supply Chain Problem
The paper argues that large language model (LLM) providers, constrained by compute, often degrade service during congestion by routing queries to smaller models, cutting reasoning effort, or truncating context. It shows that this practice misrepresents costs because degraded answers can fail, leading to retries that inflate traffic or churn that erodes lifetime value. By modeling inference allocation with newsvendor, retry, and queueing frameworks, the authors derive a ‘shadow price of intelligence’ that quantifies the marginal value of each query, revealing that throttling under congestion acts as a demand lever rather than a cost lever.
arXiv AI
Jun 2
Self-Trained Verification for Training- and Test-Time Self-Improvement
arXiv:2605. 30290v2 Announce Type: replace-cross Abstract: Self-improvement at scale has been a longstanding goal for reasoning models, and there are two natural places to do it: at test time, through verification-refinement (V-R) loops; and at training time, through self-training methods.
arXiv AI
Jul 17
Answer-Conditioned Chains of Thought Degrade Verifiable-Reasoning Distillation in Large Language Models
arXiv:2607. 14552v1 Announce Type: cross Abstract: A standard recipe for distilling the reasoning ability of large language models (LLMs) is to sample chains of thought from the model, keep those that reach the correct final answer, and fine-tune on the survivors.