arXiv AI
Jul 7
AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments
arXiv:2607. 05174v1 Announce Type: new Abstract: Language agents, i.
OdysseyArena: Benchmarking Large Language Models For Long-Horizon, Active and Inductive Interactions
Related stories
arXiv AI
Jun 9
Language-based Trial and Error Falls Behind in the Era of Experience
arXiv:2601. 21754v3 Announce Type: replace Abstract: While Large Language Models (LLMs) excel in language-based agentic tasks, their applicability to unseen, nonlinguistic environments (e.
arXiv AI
Jul 8
Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agents
arXiv:2510. 19771v4 Announce Type: replace Abstract: LLM-based agents are increasingly moving towards proactivity: rather than awaiting instruction, they exercise agency to anticipate user needs and solve them autonomously.
arXiv AI
Sep 25
GRASP: Generating, Revising, and Assessing for Strategic Planning with Agentic AI
GRASP is a multi-stage planning framework that improves the reliability of large language models on complex tasks. It separates planning into three specialized modules—GenPlan for global macro-guidelines, RevPlan for exploring localized strategies, and VerPlan for multi-criteria evaluation—allowing context isolation and strict macro-regularization. Experiments show GRASP outperforms direct LLM planners by significant margins on datasets such as Natural Plan Calendar Scheduling, ZebraLogic, and SciBench Math, and it mitigates performance collapse in multi-task and dual-task settings.
arXiv AI
5d ago
PIVOT: Bridging Planning and Execution in LLM Agents via Trajectory Refinement
PIVOT (Plan-Inspect-eVOlve Trajectories) is a self‑supervised framework that refines large‑language‑model agent plans by treating trajectories as optimizable objects. It operates in four stages—PLAN, INSPECT, EVOLVE, and VERIFY—using structured losses and textual gradients to iteratively improve trajectories and ensure compliance with task constraints. Experiments on DeepPlanning and GAIA show that PIVOT achieves up to 94% relative improvement in constraint satisfaction with human‑in‑the‑loop feedback and still delivers significant gains autonomously, while using 3x to 5x fewer tokens than competing methods.
arXiv AI
Jun 4
AutoLab: Can Frontier Models Solve Long-Horizon Auto Research and Engineering Tasks?
arXiv:2606. 05080v1 Announce Type: new Abstract: Scientific and engineering progress is fundamentally a long-horizon iterative process: proposing changes, running experiments, measuring outcomes, and continuously refining artifacts.
Hugging Face Trending Papers
Aug 10
Evo-Bench: Can Language Models Improve Agent Harness?
Large Language Models (LLMs) have driven rapid progress in autonomous agents, yet standard evaluations remain confined to static task solving. An emerging frontier is harness evolution---the agent's capacity to autonomously optimize its own operating harness.
arXiv AI
Sep 28
Agentick: A Unified Benchmark for General Sequential Decision-Making Agents
Agentick is a unified benchmark for sequential decision‑making agents that evaluates RL, LLM, VLM, hybrid, and human agents on 37 procedurally generated tasks across six capability categories, four difficulty levels, and five observation modalities via a single Gymnasium‑compatible interface. It includes a Coding API, oracle reference policies, pre‑built SFT datasets, a composable agent harness, and a live leaderboard. An evaluation of 27 configurations and over 90,000 episodes shows no single approach dominates, with GPT‑5 mini leading overall, PPO excelling in planning and multi‑agent tasks, and the reasoning harness boosting LLM performance by 3‑10×, while ASCII observations outperform natural language.
arXiv AI
Jun 2
AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents
arXiv:2603. 14465v2 Announce Type: replace Abstract: While Large Language Models (LLMs) have evolved into tool-using agents, they remain brittle in long-horizon interactions.
Hugging Face Trending Papers
Sep 24
GRASP: Generating, Revising, and Assessing for Strategic Planning with Agentic AI
GRASP is a multi-stage planning framework that separates planning into specialized modules: GenPlan for global macro-guidelines, RevPlan for exploring localized strategies, and VerPlan for multi-criteria evaluation. This strategy-aware approach yields state‑of‑the‑art accuracy on diverse datasets, outperforming direct LLM planners by up to 30.8% on ZebraLogic and reducing multi‑task degradation. GRASP’s context isolation and macro‑regularization also give it a 14.5% edge over frontier reasoning models like GPT‑5‑mini.
arXiv AI
Sep 24
Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved Mechanisms
The paper introduces VHD-Play, a pipeline that first samples and solves a mathematical model before generating agentic reinforcement learning environments, ensuring that dynamics and evaluation are aligned from the outset. This approach yields 3,300 diverse environments at a low cost and significantly improves the performance of a large language‑model agent (Qwen3.6‑35B‑A3B) across multiple diagnostic families and external benchmarks. The study demonstrates that stateful interaction is a key factor in learning gains and that scaling the training substrate can further enhance performance.
arXiv Computation and Language
Aug 28
MineExplorer: Evaluating Open-World Exploration of MLLM Agents in Minecraft
MineExplorer is a benchmark designed to assess the open‑world exploration abilities of multimodal large language models (MLLMs) in Minecraft. It filters out tasks that rely heavily on Minecraft‑specific knowledge, organizes tasks into ReAct‑style capabilities, and composes atomic tasks into implicit multi‑hop challenges. A multi‑agent synthesis workflow creates reliable task graphs, sandbox scenes, and rule‑based milestone evaluators, and human evaluation confirms its superiority over a single‑agent baseline. Experiments show that while advanced MLLMs can handle many single‑hop tasks, they struggle with longer trajectories that require coordinating hidden prerequisites, and larger models or different thinking modes do not consistently improve performance.