arXiv AI
Oct 16

Human Interventions as Labels: Localizing, Steering, and (Sometimes) Editing the Computations Behind LLM Agent Failures

The paper treats human interventions during LLM agent failures as labels that pinpoint the internal computation that went wrong, rather than merely training data. In a controlled environment, the authors analyze one failure type—over‑asking a clarifying question when the request is already clear—using rule‑based overseer labels validated by blind human judgment. They find that the decision to ask versus act can be causally localized across several models, that over‑asking exhibits distinct mechanistic signatures, and that steering or weight consolidation can mitigate the issue, though outcomes vary by model family.

By Cesare Zavattari, Alessandro Tommasi, Giuseppe Prencipe
arXiv AI
3d ago

Imperfectly Imperfect: Evaluating Graded Cognitive Diversity in LLM Persona Simulations

The paper evaluates whether large language model (LLM) agents can exhibit graded cognitive diversity when simulating human-like behaviour. Using a cognition-suite that includes new Cognitive Reflection Test (CRT) and Behavioural Reflection Test (BRT) alongside established measures, the study finds that while agents can replicate trait distributions, they fail to mirror human variation in task performance and downstream behaviour. Claude Sonnet 5 performs best but still shows significant discrepancies compared to human data.

By Sion Weatherhead, Ben Newell, Aaron Belbasis, Flora Salim