arXiv Computation and Language
What Outcome RL Buys: Finishing, Reliability, Rarely Reach
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv Computation and Language
2d ago
Concise-9B: Removing the Slack from a Reasoning Model's Thinking
The paper introduces Concise-9B, a version of the Qwen3.5-9B reasoning model that reduces unnecessary thinking tokens through concise self‑training. By applying two rounds of on‑policy rejection‑sampling fine‑tuning and a small share of answer‑synthesis targets, Concise-9B cuts token usage to roughly two‑thirds of the original model while maintaining comparable accuracy on a wide range of hard mathematics and reasoning benchmarks. The study also shows that most of the token savings occur after the answer has already formed, and that the approach’s effectiveness depends on the verbosity of the base model.
arXiv Computation and Language
Sep 2
From Base Rollouts to RL Reasoning: A Budgeted Search Perspective
arXiv:2609.01274v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) improves language-model reasoning, but how these gains relate to inference-time decoding and sear...
arXiv Machine Learning
Oct 16
Same Answer, New Box: How Much of RLVR's Out-of-Domain Gain Is Answer Format?
The paper investigates whether the reported out‑of‑domain performance gains of Reinforcement Learning with Verifiable Rewards (RLVR) are due to improved answer quality or simply to changes in answer formatting. By comparing scores from standard extractors with a custom extractor that distinguishes answers inside a oxed{} notation, the authors find that a large portion (about 82%) of the apparent gains on 7B SimpleRL models vanish, indicating that the gains are largely due to formatting rather than better answers. In contrast, a 32B model trained with a different recipe shows minimal format‑related gains, suggesting that the effect depends on the training setup.
arXiv Machine Learning
Aug 19
Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See
The paper evaluates how three large mixture‑of‑experts models (Alibaba, OpenAI, NVIDIA) can be fine‑tuned to reason in a low‑resource language, specifically Greek. Accuracy metrics show little change, but the authors uncover significant qualitative improvements: after supervised fine‑tuning, models reason in Greek on ~98% of items, with better grammaticality and retained general ability. Reinforcement learning with pre‑registered rewards further eliminates reasoning‑channel leaks and format skips, while the Greek‑reasoning habit remains robust to an accuracy‑only gradient.
Hugging Face Trending Papers
Oct 29
There Is No Free Lunch in RLVR: Every pass@1 Gain Has a pass@k Price
Reinforcement learning with verifiable rewards (RLVR) makes a language model solve more problems on the first try and fewer when given many tries: it raises pass@1 while the base model overtakes it at...
arXiv AI
Sep 4
It's the Problem, Not the Path: Budget and Difficulty Confounds in LLM Reasoning Trajectories
The paper investigates whether large language models’ reasoning traces truly contain early, informative signals or merely reflect budget and difficulty confounds. Using a restart‑controlled truncation probe, the authors compare continuation success rates against from‑scratch restart curves across 178 problem‑model pairs, finding that only one case shows prefix‑limited success and that continuing a model’s own prefix generally outperforms restarting. A difficulty‑controlled test and two generation‑free analyses reveal that early internal signals do not carry outcome information beyond a problem‑difficulty baseline, underscoring the need for proper counterfactual controls.