← Back to all news
arXiv Machine Learning By Amit Nautiyal

Which Rollout Taught It That? BehaviorTrace and the Limits of Training-Data Attribution in Online RL

Read the original on arXiv Machine Learning →

The paper introduces BehaviorTrace, an evaluation framework for assessing training‑data attribution in online reinforcement‑learning fine‑tuning of language models. Using a planted behavior on Qwen2.5‑1.5B, the authors show that many attribution signals are confounded by factors such as gradient magnitude and model fluency, and that only the gradient of trigger tokens consistently aligns with the true behavior across seeds. They provide a checklist for evaluating attribution methods and test existing estimators without proposing a new one.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Jul 15

A Learning-Rate-Gated Failure of GRPO in a Small Language and Vision-Language Model Web Agent: A Controlled Null and Its Mechanism

arXiv:2607. 12640v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards, and Group Relative Policy Optimization (GRPO) in particular, is now run routinely on a supervised checkpoint in the hope of producing a stronger agent.

By Chengguang Gan, Zhixi Cai, Yunhao Liang, Hanjun Wei, Shiwen Ni, Qinghao Zhang
llmsragagentsreinforcement-learningmultimodal
More like this →
arXiv Machine Learning
Aug 27

Demystifying Reinforcement Learning Post-Training of Language Models

The paper "Demystifying Reinforcement Learning Post-Training of Language Models" investigates how reinforcement learning (RL) post‑training enhances large language models (LLMs) for tasks such as reasoning, math, and coding. By isolating RL components in a controlled setting, the authors analyze how the base model’s prior distribution, reward granularity, prompt diversity, and model scale influence outcomes, using policy entropy to compare pre‑training, supervised fine‑tuning (SFT), and RL stages. The study clarifies the role of spurious rewards, the importance of the base model’s probability mass on desired behaviors, and how these factors interact to determine post‑training success, offering a practical primer for NLP researchers. "whyItMatters":"The work provides a clearer understanding of RL post‑training mechanics, helping researchers and practitioners effectively apply RL to improve LLM capabilities."

By Donovan Clay, Saket Gollapudi, Sankar Harilal, Min Jang, Jacob Morrison, Sewoong Oh, Natasha Jaques
llmsnlpreinforcement-learningfine-tuning
More like this →