arXiv AI
What Does a Harness Repair? A Preregistered Study of Visibility, Baseline Adequacy and Evaluation Defects
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jun 16
Looking Is Not Picking: An Attention-Segment Account of Tool-Selection Failures in LLM Agents
arXiv:2606. 16364v1 Announce Type: new Abstract: LLM agents mis-call tools, and the natural guess is that the model failed to see the right tool in a crowded harness.
arXiv Machine Learning
Jul 1
Falsification, Not Exposure: An Internally Preregistered Placebo-Controlled Decomposition of Self-Repair Feedback in Frozen Small Code Models
arXiv:2606. 31511v1 Announce Type: cross Abstract: In deployment settings where retraining is infeasible, small frozen code models are routinely asked to repair a failed program after seeing their own failing output, usually treated as a retry mechanism.
arXiv AI
Aug 21
Phantom Gains: Auditing Self-Improvement Against a Measured Null
arXiv:2608. 20290v1 Announce Type: new Abstract: Whether a language model has improved itself is increasingly judged not by mean accuracy but by which individual problems it gains and loses.
arXiv AI
Sep 4
It's the Problem, Not the Path: Budget and Difficulty Confounds in LLM Reasoning Trajectories
The paper investigates whether large language models’ reasoning traces truly contain early, informative signals or merely reflect budget and difficulty confounds. Using a restart‑controlled truncation probe, the authors compare continuation success rates against from‑scratch restart curves across 178 problem‑model pairs, finding that only one case shows prefix‑limited success and that continuing a model’s own prefix generally outperforms restarting. A difficulty‑controlled test and two generation‑free analyses reveal that early internal signals do not carry outcome information beyond a problem‑difficulty baseline, underscoring the need for proper counterfactual controls.
arXiv Machine Learning
Sep 30
Losing the name before the box: measuring and repairing what narrow fine-tuning costs a detector outside its deployment vocabulary
arXiv:2609.36426v1 Announce Type: cross Abstract: A detector pretrained on a broad corpus is fine-tuned on a narrow domain, its in-domain accuracy improves, and it ships. We ask what happens meanwhil...
arXiv AI
Aug 26
When Can One Neuron Fix Repetition Loops in LLMs?
The paper investigates whether targeted edits to a few internal components of Gemma 4 instruction‑tuned models can reduce persistent repetition loops that occur during long factual enumeration prompts. By combining per‑layer ablation with per‑neuron attribution, the authors identify specific neurons whose weight edits dramatically lower loop frequency—one sign‑inverted neuron suffices for Gemma 4 E2B. Across all four Gemma variants, loop occurrences drop from 46/384 to 12/384 on held‑out prompts, while general‑purpose benchmarks show no significant regressions. The study also demonstrates that similar sparse edits can mitigate repetition in other families such as Qwen3.5 and LFM2.5, though the effect varies.