← Back to all news
arXiv Machine Learning By Debajoy Mukherjee, Jeremy Carleton, Sushil Vemuri, Srinivas Shakkottai, Dileep Kalathil

TreeHCA: Hindsight-Guided Exploration and Credit Assignment for Multi-Turn LLM Agents

Read the original on arXiv Machine Learning →

TreeHCA introduces a method that couples hindsight-guided exploration with hindsight credit assignment for multi-turn large language model agents. By using changes in reference-answer likelihood to revisit intermediate histories and sample alternative continuations, it builds a tree of reasoning and tool interactions, and applies an inverse-hindsight importance correction to propagate terminal outcomes into turn-level advantages. Experiments on seven question-answering benchmarks with three Qwen backbones show that TreeHCA outperforms all baselines in aggregate exact match.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
2d ago

ReAPO: Retry-Aware Policy Optimization for Efficient Exploration in LLM Reasoning

ReAPO: Retry-Aware Policy Optimization for Efficient Exploration in LLM Reasoning proposes a new approach to improve language‑model reasoning by addressing the inefficiencies of independent stochastic rollouts. By framing credit assignment as a Stochastic Shortest Path problem, ReAPO modifies GRPO to allocate more generation to failed trajectories, reinforcing successful retries without rewarding failures. Experiments on Qwen3 and OLMo‑2 models across eight mathematical and scientific benchmarks show that ReAPO outperforms GRPO and other baselines, achieving higher first‑attempt accuracy and reducing generated tokens needed to reach peak performance.

By Nghi Dao, Calder Morton-Ferguson, Noah Golowich, Adam Block
llmsreinforcement-learningbenchmarks
More like this →
arXiv Machine Learning
Sep 4

TIGPO: Temporal Instance-Graph Policy Optimization for Long-Horizon LLM Agents

TIGPO (Temporal Instance-Graph Policy Optimization) extends graph-based credit assignment for long-horizon LLM agents by maintaining a persistent transition graph per task across policy updates. It allocates rollout budgets to both new exploration and revisiting past tasks, pairing current rollouts with earlier ones to create cross‑temporal references that stabilize advantage estimation. Experiments on ALFWorld and WebShop show TIGPO consistently outperforms previous group‑based and graph‑based policy optimization methods.

By Jinwei Gan
llmsagents
More like this →