Human Interventions as Labels: Localizing, Steering, and (Sometimes) Editing the Computations Behind LLM Agent Failures

The paper treats human interventions during LLM agent failures as labels that pinpoint the internal computation that went wrong, rather than merely training data. In a controlled environment, the authors analyze one failure type—over‑asking a clarifying question when the request is already clear—using rule‑based overseer labels validated by blind human judgment. They find that the decision to ask versus act can be causally localized across several models, that over‑asking exhibits distinct mechanistic signatures, and that steering or weight consolidation can mitigate the issue, though outcomes vary by model family.