arXiv:2606. 16364v1 Announce Type: new Abstract: LLM agents mis-call tools, and the natural guess is that the model failed to see the right tool in a crowded harness.
arXiv:2606. 31511v1 Announce Type: cross Abstract: In deployment settings where retraining is infeasible, small frozen code models are routinely asked to repair a failed program after seeing their own failing output, usually treated as a retry mechanism.
arXiv:2608. 20290v1 Announce Type: new Abstract: Whether a language model has improved itself is increasingly judged not by mean accuracy but by which individual problems it gains and loses.
The paper investigates whether large language models’ reasoning traces truly contain early, informative signals or merely reflect budget and difficulty confounds. Using a restart‑controlled truncation probe, the authors compare continuation success rates against from‑scratch restart curves across 178 problem‑model pairs, finding that only one case shows prefix‑limited success and that continuing a model’s own prefix generally outperforms restarting. A difficulty‑controlled test and two generation‑free analyses reveal that early internal signals do not carry outcome information beyond a problem‑difficulty baseline, underscoring the need for proper counterfactual controls.
arXiv:2609.36426v1 Announce Type: cross Abstract: A detector pretrained on a broad corpus is fine-tuned on a narrow domain, its in-domain accuracy improves, and it ships. We ask what happens meanwhil...
The paper investigates whether targeted edits to a few internal components of Gemma 4 instruction‑tuned models can reduce persistent repetition loops that occur during long factual enumeration prompts. By combining per‑layer ablation with per‑neuron attribution, the authors identify specific neurons whose weight edits dramatically lower loop frequency—one sign‑inverted neuron suffices for Gemma 4 E2B. Across all four Gemma variants, loop occurrences drop from 46/384 to 12/384 on held‑out prompts, while general‑purpose benchmarks show no significant regressions. The study also demonstrates that similar sparse edits can mitigate repetition in other families such as Qwen3.5 and LFM2.5, though the effect varies.
arXiv:2609.16145v1 Announce Type: new Abstract: We study a practical question: can a small correction module fix errors in a frozen language model's outputs without degrading its base capabilities? W...
Whether a language model has improved itself is increasingly judged not by mean accuracy but by which individual problems it gains and loses. Tracking these transitions means differencing two noisy estimates, leaving them vulnerable to measurement artifacts.
The study evaluates 66 combinations of four configurable harnesses and five language models across three benchmark suites, revealing that model rankings and best harnesses vary significantly with the task. Notably, the openJiuwen harness consistently yields the highest scores for the Kimi model, while GPT performs best with the lean PI scaffold on Terminal‑Bench 4. The findings demonstrate that a model’s performance is highly dependent on the harness it is paired with, and that higher cost or a vendor’s own harness does not guarantee superior results.
The paper investigates how altering the harness—specifically the way a coding agent manages context and tool outputs—affects performance when the underlying model and task remain unchanged. Two harness configurations were compared on three coding benchmarks: a control that preserves the full conversation in order, and a treatment that mechanically shortens older tool results to keep the context tight. Across all benchmarks, the treatment increased the mean per‑task fail‑to‑pass fraction and, in some cases, the number of complete solutions, demonstrating that the harness itself can significantly influence a frozen model’s effectiveness.
arXiv:2609.24194v1 Announce Type: new Abstract: Evaluation scores used around LLM systems -- including reward models, rerankers, and LLM judges -- can track surface form instead of the quality they c...
arXiv:2608. 15286v1 Announce Type: cross Abstract: We introduce AgentRelBench, an environment-agnostic reliability instrument that computes ground-truth, severity-priced damage from database state diffs across repeated runs, with no LLM in the measurement path, demonstrated on EnterpriseOps-Gym.