arXiv AI
Oct 5

Counterexample Generation via Per-Theorem Symbolic Verifiers: When Imitation Hurts and Reinforcement Repairs

The paper introduces SymCE, a dataset of 4,707 false undergraduate‑algebra and real‑analysis conjectures each paired with a deterministic Python verifier that can be executed to check truth. It studies counterexample generation by training a 4B‑parameter language model (Qwen3‑4B) first with supervised fine‑tuning and then with reinforcement learning (GRPO) using the verifier as a reward. The results show that counterexample‑only fine‑tuning can destroy true‑theorem recognition, while reinforcement learning with a sparse outcome‑only reward restores and surpasses baseline performance, achieving higher success rates than larger open‑weight models and competitive performance on other math benchmarks. "whyItMatters":"The study demonstrates that reinforcement learning guided by a per‑theorem verifier can effectively repair the shortcomings of supervised fine‑tuning in theorem proving, offering a practical approach to improve language models’ mathematical reasoning capabilities."

By Omar Farouk Zouak, Houssam Eddine Boukhalfa, Soumaya Lakehal, Shiv Katiyar, Samia Nefti-Meziani
arXiv AI
Oct 9

Who Verifies the Verifier? Co-Evolving Inspectable Graders with Self-Improving Agents

The paper introduces a method for co-evolving inspectable verifiers alongside self-improving agents. By synthesizing verifiers from clustered failures and selecting them based on agreement with a reference set, the authors demonstrate improved held‑out agreement on MBPP+ and outperform a bare LLM judge. They also show that removing anchor guards collapses the verifier into a vacuous grader, yet the collapsed verifier still trains skills effectively, indicating that downstream task scores cannot certify a self‑evolved verifier.

By Xing Zhang, Guanghui Wang, Yanwei Cui, Ziyuan Li, Wei Qiu, Bing Zhu, Peiyang He
arXiv AI
Nov 6

Measuring the Marginal Value of Verifier-Based RL Beyond Supervised Fine-Tuning

The paper evaluates the true benefit of verifier‑based reinforcement learning (RLVR) over supervised fine‑tuning (SFT) using a deterministic benchmark of 1,877 synthetic GPU and LLM training failure diagnosis tasks. In controlled experiments, SFT alone achieved high performance (0.785 on the hard split), while RLVR added no detectable improvement in‑distribution and only a modest gain under distribution shift (+0.33 points). The study concludes that most performance gains come from SFT, with verifier‑based RL contributing only small additional benefits.

By Amit Sagtani, Saloni Garg