arXiv AI
Lessons from Luck: Do LLM-Written Lessons Carry Chance into an Agent's Next Decision?
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Sep 25
Sequential knowledge editing breaks a model's ability to tell good evidence from bad, without costing it accuracy
The paper investigates how sequential knowledge editing can degrade a language model’s ability to discern reliable evidence from unreliable evidence without affecting overall accuracy. Using a conservatively tuned LoRA on Qwen2.5‑7B‑Instruct, the authors show that after 1,000 edits the model’s arbitration score for untouched facts drops by 36%, leading to higher error rates on its most confident decisions, while MMLU accuracy remains unchanged. The study also finds that in some model‑method combinations, sequential edits can reduce MMLU to chance levels even though edit success and locality remain perfect.
arXiv AI
Aug 28
Invocation-Level Reliability of Tool-Using Agents
The paper investigates the reliability of tool‑using agents, focusing on two failure modes: selecting the wrong tool and constructing incorrect arguments. It introduces a correct‑invocation rate metric to distinguish these errors and evaluates five open‑weight models on multi‑step tasks up to depth 8, finding that by depth 6 about 70% of a model’s clean‑context capability is lost due to earlier mistakes. The study reveals that exact‑match scoring against a fixed gold trajectory forces severity and recovery parameters to extreme values, and proposes a conditional‑on‑state scoring remedy that yields more realistic severity estimates.
arXiv AI
Aug 26
Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Fail
The study investigates why small language model agents tend to repeat a tool call that just failed. By recording the failed call and its error message in the transcript, the authors measure a negative corrective gain—agents are more likely to repeat the failed action, with a drop of about 1.03 nats per token. The problem is traced to the harness design rather than the model’s understanding of errors, and the authors show that replacing the verbatim call with a runtime-generated description of the failure can reduce this backfiring effect by 76%.
arXiv AI
Oct 5
Not Until the Evidence Says So: Teaching LLM Investigators When to Close a Case
arXiv:2610.03190v1 Announce Type: cross Abstract: Accident, defect and outage investigations end with a decision that ordinary question answering never faces: whether the evidence gathered so far is...
arXiv Computation and Language
Oct 23
Memories That Outlive Their Reasons: Dependency-Tethered Memory for Tool-Using LLM Agents
The paper introduces Dependency‑Tethered Memory (Tether), a memory system for tool‑using LLM agents that records the exact versions of premises each verdict depends on. By withdrawing any verdict whose support has failed anywhere in the store, Tether reduces unsafe publications to 0.3% and cuts verification calls by 37% compared to naive fact memory, while maintaining similar recall. The study demonstrates that capturing dependencies is crucial for safe memory use, as weaker writers or delayed relation arrivals increase unsafe publication rates.
arXiv AI
Aug 26
How much of a measured AI preference is the model, and how much is the instrument?
The paper investigates how much of an AI model’s expressed preferences are due to the model itself versus the instrument (prompt) used to elicit those preferences. By fixing the set of outcomes and models while varying five different prompting instruments across 15 welfare-related outcomes, the study finds that the ranking of outcomes is only moderately generalizable (coefficient 0.348) and that a single instrument’s preference provides little insight into another instrument’s results. The analysis shows that even after removing any single instrument, model, or a subset of outcomes, the overall preference estimate remains robust, yet the variability across instruments remains significant.