arXiv AI
Instructions Are Evidence, Not Objectives: Measuring Outcome Alignment of Agents
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jun 30
SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions
arXiv:2606. 30573v1 Announce Type: new Abstract: We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks.
arXiv AI
1d ago
AM-Bench: Understanding Agents under Task Underspecification across Models and Harnesses
arXiv:2610.18090v1 Announce Type: new Abstract: Ambiguous or incomplete task specifications complicate both execution and clarification for language-model agents. Yet agents' responses to these infor...
arXiv Computation and Language
1d ago
Interlock: From Local Correctness to Task Completion in Long-Horizon Agents
Interlock is a framework that transforms long‑horizon tasks into typed completion graphs, linking evidence, decisions, and required effects through grounding, coupling, and realization edges. It compiles these graphs into a resettable workspace, a public request, and an evaluation rubric, and uses source‑coverage, public‑input, and agent‑as‑judge reviews to ensure that all mandatory requirements are met. Fine‑tuning Qwen3.6 backbones on Interlock‑generated trajectories improves pass rates on AutomationBench, JobBench, and Harvey LAB, showing that the synthesized experience benefits external task environments.
arXiv AI
Jul 14
AgentAbstain: Do LLM Agents Know When Not to Act?
arXiv:2607. 10059v1 Announce Type: new Abstract: Agent systems based on large language models (LLMs) are increasingly deployed for autonomous tasks, yet existing evaluations mostly focus on task success rather than whether agents know when to abstain.
arXiv AI
Jun 19
ORAgentBench: Can LLM Agents Solve Challenging Operations Research Tasks End to End?
arXiv:2606. 19787v1 Announce Type: new Abstract: Large language models are increasingly deployed as autonomous agents for multi-step tasks in executable environments, yet their ability to perform realistic operations research (OR) work remains unclear.
arXiv Machine Learning
Oct 8
Coding-Agent Benchmarks Should Match Their Users' Task Flows
arXiv:2610.09633v1 Announce Type: new Abstract: The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engine...