arXiv AI
Jul 7
TRACE: Capability-Targeted Agentic Training
arXiv:2604. 05336v2 Announce Type: replace Abstract: Models often fail to complete agentic tasks because they lack core capabilities required by the target environment.
PG-SFT: Balancing Capability Acquisition and Retention in Offline Agent Fine-Tuning
PG-SFT: Balancing Capability Acquisition and Retention in Offline Agent Fine-Tuning explores how to maintain a model’s existing abilities while teaching it new ones through supervised fine‑tuning on offline agent trajectories. The authors compare standard SFT, KL‑penalty, and update‑magnitude constraints, finding that these methods still degrade non‑target capabilities. They introduce Privilege‑Guided SFT (PG‑SFT), which uses turn‑level information gain to modulate supervision strength, achieving a better trade‑off between acquiring new skills and preserving existing ones, though with a slight drop in target‑task performance.
Related stories
arXiv AI
Sep 10
AgentBrew: Offline Tool-Use Agent Learning from Raw Real-World Trajectories
arXiv:2609.05837v1 Announce Type: new Abstract: LLM-based agents are increasingly deployed in real-world applications through tool-use APIs, yet training them for specific environments remains fundam...
arXiv AI
Jun 16
PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agents
arXiv:2606. 16215v1 Announce Type: cross Abstract: Multi-turn tool-use agents must reason, call tools, and adapt to observations across several interaction turns.
arXiv AI
2d ago
Finetuning with Sampling: SFT Learns Better Than You Think
arXiv:2610.02140v1 Announce Type: cross Abstract: Introducing new capabilities to frontier models has long been the goal of posttraining, which predominantly employs supervised finetuning (SFT) and r...
arXiv AI
Jun 2
SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting
arXiv:2604. 10688v2 Announce Type: replace-cross Abstract: On-policy reinforcement learning has become the dominant paradigm for reasoning alignment in large language models, yet its sparse, outcome-level rewards make token-level credit assignment notoriously difficult.
arXiv Computation and Language
Sep 17
Dependency-Aware Trajectory Refinement for Efficient Multi-Turn Agent Fine-Tuning
arXiv:2609.18417v1 Announce Type: new Abstract: Multi-turn agent trajectories often contain redundant rounds (failed tool calls, parallel sub-queries, verification-only steps) that inflate both train...
arXiv AI
Sep 18
Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL
The paper introduces ActObs, a supervised fine‑tuning method that, unlike standard approaches, also predicts environment observations in agent trajectories. While both ActObs and action‑only training perform similarly after initial fine‑tuning, ActObs diverges during subsequent reinforcement learning, yielding higher pass@k scores on several benchmarks and better cross‑domain task performance. The authors attribute this advantage to ActObs’s joint supervision, which preserves observation gradients and prevents the policy from over‑specializing on actions alone.
arXiv Machine Learning
Sep 11
T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks
The paper introduces T1, a 122‑billion‑parameter Mixture‑of‑Experts model trained with reinforcement learning to perform long‑horizon terminal tasks such as coding and scientific discovery. T1 operates a real shell in a cloud sandbox, making over 300 tool‑call turns per task and receiving rewards from task‑specific verifiers. The authors detail a training recipe that includes aggressive warm‑starting, TITO construction with drift repair, and rollout‑routing replay, achieving significant performance gains on Terminal‑Bench 2.1 and surpassing GPT‑5.4 and GLM‑5.1 on the Long‑Horizon Terminal Bench.
arXiv Machine Learning
Jun 9
PriFT: Prior-Support Guided Supervised Fine-Tuning
arXiv:2606. 09396v1 Announce Type: cross Abstract: Supervised fine-tuning (SFT) is an efficient approach for downstream task adaptation and often serves as the initialization stage for reinforcement learning (RL), but it can show weaker generalization than RL.
arXiv AI
Jun 10
A Unifying Lens on Supervised Fine-Tuning Through Target Distribution Design
arXiv:2606. 11189v1 Announce Type: cross Abstract: Supervised fine-tuning (SFT) typically maximizes the likelihood of every token in a demonstrated trajectory.
arXiv Machine Learning
Sep 2
Online Self-Weighted Fine-Tuning
Online Self-Weighted Fine‑Tuning (OSW‑FT) augments standard supervised fine‑tuning by adding online, trajectory‑level weighting: for each query the model estimates its current success rate from a small number of inference‑only rollouts and rescales the SFT loss accordingly. The method keeps the optimization direction anchored to the expert trajectory while adapting the update magnitude online, and it is shown to be unbiased for any finite rollout count with a convergence analysis. Across Qwen3 models from 0.6B to 4B, OSW‑FT consistently outperforms plain SFT on challenging benchmarks such as AIME, achieving a favorable compute‑performance trade‑off with only two online rollouts.
arXiv Machine Learning
4d ago
On-Policy Parameter Update Direction Underlies Generalization in LLM Post-Training
arXiv:2609.36659v1 Announce Type: new Abstract: The strong generalization performance of on-policy post-training paradigms has motivated studies of their parameter update behaviors. However, these st...