arXiv:2607. 09709v1 Announce Type: new Abstract: Post-training a code generator against a learned judge can optimize proxy features that raise the score without improving the artifact.
By Chenyu Zhou, Qiliang Jiang, Shuning Wu, Xu Zhou
arXiv:2607. 05904v1 Announce Type: new Abstract: Training a language model against its own reference-free judgments (the premise of self-rewarding, self-play, and LLM-as-a-judge pipelines) assumes a model's verdict on a shown answer tracks correctness.
By Chenyu Zhou
The paper introduces SymCE, a dataset of 4,707 false undergraduate‑algebra and real‑analysis conjectures each paired with a deterministic Python verifier that can be executed to check truth. It studies counterexample generation by training a 4B‑parameter language model (Qwen3‑4B) first with supervised fine‑tuning and then with reinforcement learning (GRPO) using the verifier as a reward. The results show that counterexample‑only fine‑tuning can destroy true‑theorem recognition, while reinforcement learning with a sparse outcome‑only reward restores and surpasses baseline performance, achieving higher success rates than larger open‑weight models and competitive performance on other math benchmarks.
"whyItMatters":"The study demonstrates that reinforcement learning guided by a per‑theorem verifier can effectively repair the shortcomings of supervised fine‑tuning in theorem proving, offering a practical approach to improve language models’ mathematical reasoning capabilities."
By Omar Farouk Zouak, Houssam Eddine Boukhalfa, Soumaya Lakehal, Shiv Katiyar, Samia Nefti-Meziani
The paper introduces a method for co-evolving inspectable verifiers alongside self-improving agents. By synthesizing verifiers from clustered failures and selecting them based on agreement with a reference set, the authors demonstrate improved held‑out agreement on MBPP+ and outperform a bare LLM judge. They also show that removing anchor guards collapses the verifier into a vacuous grader, yet the collapsed verifier still trains skills effectively, indicating that downstream task scores cannot certify a self‑evolved verifier.
By Xing Zhang, Guanghui Wang, Yanwei Cui, Ziyuan Li, Wei Qiu, Bing Zhu, Peiyang He
arXiv:2606. 07856v1 Announce Type: new Abstract: When a language model trains on its own verified outputs, does it acquire capability beyond its base, or merely get better at expressing capability the base already had?
By Igor Lima Strozzi
The paper evaluates the true benefit of verifier‑based reinforcement learning (RLVR) over supervised fine‑tuning (SFT) using a deterministic benchmark of 1,877 synthetic GPU and LLM training failure diagnosis tasks. In controlled experiments, SFT alone achieved high performance (0.785 on the hard split), while RLVR added no detectable improvement in‑distribution and only a modest gain under distribution shift (+0.33 points). The study concludes that most performance gains come from SFT, with verifier‑based RL contributing only small additional benefits.
By Amit Sagtani, Saloni Garg
arXiv:2606. 21641v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have been proposed as hyperparameter-optimization (HPO) advisors that "warm-start" search from prior knowledge, proposing strong configurations in very few evaluations.
By Carson Rodrigues, Oysturn Vas, Isaiah Abner DCosta, Nithish Kumar Prabhakaran
PROOF-Gen is a method that improves distillation of tool‑calling models by recovering successful trajectories from teacher failures. It uses per‑scenario prompt optimization to generate corrective guidance that steers the teacher to a passing trajectory, then removes this guidance before training so the student learns from clean demonstrations. On τ2‑bench, PROOF-Gen recovers 93% of failed scenarios, boosting Qwen3‑4B‑Instruct‑2507’s Pass^1 from 0.132 to 0.529 and improving Gemma 4 E4B‑it by 7.2pp on BFCL v4 multi‑turn, while also raising deployed on‑device model performance by up to 5.0pp across response‑quality metrics.
By Anh Ta, Junjie Zhu, Shahin Shayandeh
arXiv:2606. 14629v1 Announce Type: cross Abstract: Verifier-driven self-DPO is a common recipe for self-improving production visual-language models.
By Jianzhe Lin
arXiv:2610.17609v1 Announce Type: new
Abstract: Training-free self-improvement lets an agent use feedback to revise the response it is about to return and to update the harness state that shapes how...
By Tianyi Xu, Yaolun Zhang, Yiran Wu, Joe Nguyen, Shengyu Dai, Saba Moeinizade, Zhenwen Shao, Qingyun Wu, Huazheng Wang
The paper investigates how self‑improving large language models (LLMs) can suffer from the winner’s curse when selecting better candidates based on noisy evaluation sets. By modeling correlated errors and empirically testing Qwen models that rewrite their own instructions, the authors find that most proposals after the first are harmful and that the reported gains on the selection set often overstate true held‑out performance. The study shows that larger selection sets can improve held‑out gains on some tasks (e.g., TREC) but not others (e.g., GSM8K), and that common acceptance rules do not outperform simple greedy selection across entire runs.
By Litao Hu, Yutong Tang
arXiv:2608.29030v1 Announce Type: new
Abstract: In-context watermarking (ICW) prepends an instruction to a query asking the model to embed a statistically detectable signal in its response. It thus e...
By Yepeng Liu, Tianyi Chen, Xuandong Zhao, Dawn Song, Yuheng Bu