MemToC is a controlled benchmark that tests how large language models resolve conflicts between their internal memory and tool outputs. It contains 6,504 episodes built from 542 factual questions, each paired with a model‑generated closed‑book answer and a tool return whose correctness is known, creating four distinct source‑correctness scenarios. Across five 7‑9B open‑weight models, tool responses overwhelmingly dominate closed‑book answers, and only a minority of instruction‑tuned models correctly retain a verified answer when the tool is wrong, while most follow a correct tool or repeat a wrong tool.
arXiv:2606. 29914v1 Announce Type: cross Abstract: Agent memory systems are increasingly evaluated against RAG and full-context baselines, but reported gains often mix changes in the memory method with changes in the language model, embedding model, or retrieval pipeline, making it unclear what is actually being measured.
ChipMEM introduces a verification‑grounded memory layer for electronic design automation agents that combines cross‑task procedural memory with within‑trajectory statistical guidance. The procedural component stores a skill only after it passes synthesis, simulation, or formal checks, while a Bayesian component ranks recovery strategies based on tool‑call outcomes. Experiments on RTLRewriter‑Bench and CVDP tasks show that ChipMEM improves equivalence‑passing outputs and area metrics compared to agents without memory.
The paper investigates how rewording questions affects the relative performance of language models and whether providing models with tools can stabilize these comparisons. Using a matched protocol that isolates answer readout, reasoning, tool calls, and reporting, the authors evaluate six open models across five domains and five question wordings. They find that tool usage can increase accuracy for programmatic answers but also tends to widen wording sensitivity, especially when call frequency varies with wording and depends on the interface and instructions.
arXiv:2605.09252v3 Announce Type: replace Abstract: Tool-augmented LLM agents tend to call tools indiscriminately, even when the model can answer directly. Each unnecessary call wastes API fees and l...
The paper introduces MERIT, a benchmark that evaluates the marginal benefit of long‑term memory for tool‑using large language model agents while explicitly accounting for cost. MERIT provides episodic tool‑use tasks across three domains, verifies dependence on earlier‑episode facts, and measures memory operations in tokens and dollars. Experiments on GPT‑4.1‑mini, Claude Haiku 4.5, and Claude Sonnet 5 show that memory can significantly improve task success, but its utility varies widely across models and memory implementations, and full replay is rarely cost‑effective.
The paper introduces SMITH, a reinforcement learning framework that jointly trains large language models to create and use tools within a single policy. By alternating between build and use tasks and employing separate reward signals for schema, code, and outcome failures, SMITH enables a 4B Qwen3 model to achieve state‑of‑the‑art accuracy on procedural reasoning benchmarks, outperforming larger untrained models and improving performance on downstream tasks when its tools are applied.
arXiv:2606. 25449v1 Announce Type: cross Abstract: A language model's memory can be worse than having no memory at all.
arXiv:2607. 19257v1 Announce Type: cross Abstract: Practitioners make three prompt-design decisions with almost no controlled evidence behind them: how to format instructions and context (markdown, plain text, prose, or tabular), how many simultaneous instructions a system prompt can carry before compliance degrades, and how much context a model can hold before recall and honesty degrade.
arXiv:2510.13935v3 Announce Type: replace-cross Abstract: The facts a language model stores are tied to its parameter count, so small models that fit on edge devices fail on expert problems, which ne...
The paper introduces Boundary-Aware Skill Memory (BASM), a method that enriches skill memories for large language model agents with explicit boundary fields such as applicability conditions, risk cues, avoidance rules, and recovery notes. This approach transforms retrieved skills from unconditional templates into state‑conditioned guidance, preventing the Skill Imitation Trap where more skills lead to incorrect tool usage. Experiments on three agent benchmarks and four model scales show that BASM improves task success rates, accuracy, and reduces attack success while cutting average steps compared to memory‑free baselines.
The paper introduces DKL, a method for adding new knowledge to instruction‑tuned language models without compromising their instruction‑following abilities. DKL performs extended pre‑training on a base LLM to embed knowledge, then merges these weights into the instruction‑tuned model, avoiding costly instruction fine‑tuning. Experiments show DKL raises RAG accuracy from 54.17% to 79.26% on retrieval failure cases while using far less training data than previous approaches.
The paper presents a study on fine‑tuning open‑source instruction‑tuned language models for a Linear Control Systems course using LoRA. A dataset of 360 system‑user‑assistant conversations was created, and LoRA was applied to Qwen2.5‑3B‑Instruct and Qwen2.5‑7B‑Instruct with ranks r=4, 8, and 16. Evaluation with ROUGE, BERTScore, and structured‑output metrics showed that LoRA improved similarity and stability, with the 7B‑r16 model achieving the highest scores and r=8 offering a good trade‑off between performance and parameter efficiency.