The paper introduces Turn-level Multiscale Density Ratio Estimation (tlm‑DRE), a post‑training technique for large language model agents that assigns varying weights to different turns and employs asymmetric token‑level training based on positive‑negative gaps across multiple turns. Unlike conventional alignment methods such as PPO, DPO, DIL, and GRPO, which mainly target single‑turn tasks, tlm‑DRE is designed for complex multi‑turn reasoning. Experiments on diverse agent benchmarks demonstrate that tlm‑DRE competes with traditional alignment methods and enables LLMs to perform robustly in both in‑domain and out‑of‑domain multi‑turn tasks.
With the rapid development of Large language model (LLM), agent systems enhanced by LLMs show huge potential in being able to deal with complex tasks, especially involving multi-step thinking or inter...
UniToolCall introduces a unified framework for tool-use in large language model agents, standardizing toolset construction, dataset generation, and evaluation. The framework aggregates over 22,000 tools and creates a hybrid training corpus of more than 390,000 instances by combining ten public datasets with synthetically generated, structurally controlled trajectories. It models diverse interaction patterns—single‑hop vs. multi‑hop, single‑turn vs. multi‑turn, serial vs. parallel execution—and adds an Anchor Linkage mechanism to enforce cross‑turn dependencies, while converting seven public benchmarks into a common Query–Action–Observation–Answer format for fine‑grained evaluation.
arXiv:2610.33451v1 Announce Type: new Abstract: The capabilities of large language models (LLMs) and multimodal LLMs (MLLMs) arise from the synergy of the full training pipeline, including data and t...
The paper introduces Align‑LoRA, a unified LoRA framework for multi‑task learning that replaces complex, isolated adapter designs with a single‑adapter model enhanced by a higher rank and an explicit alignment loss. It demonstrates that a router‑free, multi‑head model with high inter‑head redundancy can outperform more elaborate baselines, and that a unified LoRA can achieve competitive performance while enabling weight merging and zero inference latency. Extensive experiments and theoretical analysis confirm that Align‑LoRA surpasses prevailing approaches, offering a simpler, production‑friendly paradigm for parameter‑efficient fine‑tuning of large language models.
PlanPO introduces a group planning-aware policy optimization method for multi-turn agentic large language models, addressing the issue of advantage collapse caused by treating all successful trajectories equally. By incorporating coarse-to-fine advantage signals that reflect differences in trajectory and turn lengths, PlanPO encourages agents to learn generalizable planning and generation behaviors. Experiments show a 27.2% average improvement over GRPO on benchmarks such as ALFWorld, WebShop, and SciWorld, with minimal extra training cost.
arXiv:2510. 08233v3 Announce Type: replace Abstract: Diffusion large language models (dLLMs) are promising alternatives to autoregressive large language models (AR-LLMs), as they potentially allow higher inference throughput.
Reinforcement learning (RL) post-training for large language models (LLMs) follows a efficient paradigm of "rollout then update", which inevitably results in off-policy training data. To resolve this, Importance sampling (IS) is proposed, while the token-level ratios compound over long sequences, causing severe variance exploded.
arXiv:2608. 02391v1 Announce Type: cross Abstract: Tool-using large language model (LLM) agents produce long, multi-turn trajectories, making gradient-based post-training memory-intensive.
The paper introduces SORL, a framework that stabilizes off‑policy reinforcement learning for long‑horizon large language model agents. It identifies two key instability sources—token‑level policy granularity mismatches and high‑variance off‑policy updates—and proposes turn‑level importance sampling and clipping‑triggered normalization to align optimization with multi‑turn interactions. Two instantiations, SO‑PPO and SO‑GRPO, are evaluated on open‑domain, multi‑hop, and medical QA benchmarks, as well as on asynchronous RL for mathematical reasoning, showing improved robustness without the need for early stopping or heuristic tuning.
Self-distillation improves reasoning in large language models by using the model's own rollouts as training signal, typically through implicit logit-level alignment that minimizes KL divergence toward a privileged target distribution. However, because this supervision is generated via uncontrolled sampling, it provides no diagnostic insight into the model's specific errors or corrective guidance for its individual failure patterns.
The paper introduces Multiscale Policy Optimization (MPO) for long‑horizon large language model agents, addressing the mismatch between turn‑level critic values and token‑level action generation. MPO combines Multiscale Advantage Estimation (MAE) with turn‑level importance sampling to better align credit assignment and policy updates at the turn scale. Experiments on search, tool‑use, and automation tasks with 4B and 30B‑A3B models show consistent improvements over strong reinforcement learning baselines.
The paper introduces Cast‑R1, an 8‑billion‑parameter agentic forecasting model that learns to orchestrate analytical and prediction tools through trajectory distillation and multi‑turn reinforcement learning with structured memory. The training pipeline first generates multi‑turn interaction trajectories using a teacher LLM, then fine‑tunes the agent with supervised learning and group relative policy optimization, guided by rewards for accuracy and temporal structure. Experiments across several datasets show that Cast‑R1 effectively equips smaller language models with tool‑augmented, multi‑turn forecasting capabilities within structured workflows.