arXiv AI
Jul 16
Set-shifting Behavioral Test for Harnessed Agents
arXiv:2607. 13396v1 Announce Type: new Abstract: What happens to an LLM agent's tool choice when the reliable tool silently changes within an ongoing session?
Set-shifting Behavioral Test for Harnessed Agents
What happens to an LLM agent's tool choice when the reliable tool silently changes within an ongoing session? We borrow set-shifting from cognitive psychology to study how well agents adapt to hidden reliability shifts.
Related stories
arXiv AI
Oct 26
SWE-Shift: Evaluating Coding Agents in Evolving Workspaces
SWE-Shift is a new benchmark that tests coding agents in environments where tasks and workspaces can change mid-execution. It presents 120 scenarios across 12 change families, each evaluated at nine intervention points, comparing a Clean continuation (no change) with a Shifted continuation (task-relevant update). The benchmark shows that all evaluated agents perform better on Clean than on Shifted, with significant drops in success rates and zero paired success for certain change types.
arXiv AI
Aug 6
OneDayAgent: Towards a Long-Horizon Harness for Autonomous Agents
arXiv:2608. 05013v1 Announce Type: cross Abstract: LLM agents are increasingly applied to open-ended everyday requests that span work, study, and life.
arXiv AI
Oct 15
TRAP: Tool-result Replay Attack for Probing Misbinding in LLM Agents
arXiv:2610.16333v1 Announce Type: new Abstract: As AI agents increasingly act through external tools, tool results are no longer simple outputs but execution artifacts that must remain correctly asso...
arXiv AI
Oct 21
Harness Self-Evolution for Long-Horizon Terminal Agents
The paper introduces RiseHarness, a framework that iteratively refines agent harnesses while keeping model weights fixed. It uses successful and failed runs under the same configuration to analyze failures, combining agent trajectories with runtime monitor traces to understand interactions between model actions and execution components. The framework then searches other harness frameworks for relevant implementations, adapts them, and validates changes through behavioral checks and repeated task evaluation, ultimately integrating mechanisms from multiple frameworks to improve reliability across tasks.
Hugging Face Trending Papers
Jul 7
Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents
Large language model (LLM) agents are increasingly evaluated on their ability to use tools, plan multi-step tasks, coordinate with other agents, and operate over extended horizons. Reported benchmark gains often obscure recurring failure modes documented across otherwise unrelated evaluation efforts.
arXiv AI
Oct 5
Sentry: Learning to Recover from LLM Agent Failures at Test Time
Sentry is a failure‑management layer for large language model agents that learns from failures at test time. It retrieves relevant lessons from an external playbook when a failure occurs, verifies recovery without task rewards, and stores new lessons only if recovery succeeds, keeping the playbook out of the agent’s context. Across multiple benchmarks, Sentry outperforms both runtime‑intervention and context‑evolution baselines, and its lessons transfer to unseen tasks.
arXiv AI
Jul 8
Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents
arXiv:2607. 05775v1 Announce Type: new Abstract: Large language model (LLM) agents are increasingly evaluated on their ability to use tools, plan multi-step tasks, coordinate with other agents, and operate over extended horizons.
Hugging Face Trending Papers
Jun 4
Retrospective Harness Optimization: Improving LLM Agents via Self-Preference over Trajectory Rollouts
AI agents rely on a harness of skills, tools, and workflows to solve complex problems. Continually improving this harness is essential for adapting to new tasks.
arXiv Machine Learning
Jun 5
Retrospective Harness Optimization: Improving LLM Agents via Self-Preference over Trajectory Rollouts
arXiv:2606. 05922v1 Announce Type: cross Abstract: AI agents rely on a harness of skills, tools, and workflows to solve complex problems.
arXiv AI
Jul 8
Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agents
arXiv:2510. 19771v4 Announce Type: replace Abstract: LLM-based agents are increasingly moving towards proactivity: rather than awaiting instruction, they exercise agency to anticipate user needs and solve them autonomously.
Hugging Face Trending Papers
Jul 5
Measuring Harness-Induced Belief Divergence in Multi-Step LLM Agents
Software-agent benchmarks usually report whether an agent solves a task, but the agent reaches that outcome through a harness that controls what it sees, which actions it can take, which failures are repaired, which states are verified, and which evidence is logged. We show that this harness can change the agent's multi-step beliefs even when the task, environment, and base LLM are fixed.