arXiv AI By Arseniy Varlamov, Rishat Zinnatullin, Elisei Rykov, Alexander Panchenko, Ilseyar Alimova

MemToC: Benchmarking Memory-Tool Conflict Resolution in Large Language Models

Read the original on arXiv AI →

MemToC is a controlled benchmark that tests how large language models resolve conflicts between their internal memory and tool outputs. It contains 6,504 episodes built from 542 factual questions, each paired with a model‑generated closed‑book answer and a tool return whose correctness is known, creating four distinct source‑correctness scenarios. Across five 7‑9B open‑weight models, tool responses overwhelmingly dominate closed‑book answers, and only a minority of instruction‑tuned models correctly retain a verified answer when the tool is wrong, while most follow a correct tool or repeat a wrong tool.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning
1d ago

ChipMEM: Verification-Grounded Memory for EDA Agents

ChipMEM introduces a verification‑grounded memory layer for electronic design automation agents that combines cross‑task procedural memory with within‑trajectory statistical guidance. The procedural component stores a skill only after it passes synthesis, simulation, or formal checks, while a Bayesian component ranks recovery strategies based on tool‑call outcomes. Experiments on RTLRewriter‑Bench and CVDP tasks show that ChipMEM improves equivalence‑passing outputs and area metrics compared to agents without memory.

By Abdulrahman AlRabah, Joshua Mabry, Dilek Hakkani-T\"ur, Abdussalam Alawini, Hamid Shojaei, Kartik Hegde, Sandesh Adhikary
arXiv AI
Sep 10

When Does Memory Help? A Cost-Aware Evaluation of Long-Term Memory in Tool-Using LLM Agents

The paper introduces MERIT, a benchmark that evaluates the marginal benefit of long‑term memory for tool‑using large language model agents while explicitly accounting for cost. MERIT provides episodic tool‑use tasks across three domains, verifies dependence on earlier‑episode facts, and measures memory operations in tokens and dollars. Experiments on GPT‑4.1‑mini, Claude Haiku 4.5, and Claude Sonnet 5 show that memory can significantly improve task success, but its utility varies widely across models and memory implementations, and full replay is rarely cost‑effective.

By Shweta Mishra, Shashank Mishra
arXiv AI
Aug 26

Joint Optimization of Tool Creation and Use for Large Language Model Agents

The paper introduces SMITH, a reinforcement learning framework that jointly trains large language models to create and use tools within a single policy. By alternating between build and use tasks and employing separate reward signals for schema, code, and outcome failures, SMITH enables a 4B Qwen3 model to achieve state‑of‑the‑art accuracy on procedural reasoning benchmarks, outperforming larger untrained models and improving performance on downstream tasks when its tools are applied.

By Zhi Rui Tam, Chieh-Yen Lin, Yun-Nung Chen, Shao-Hua Sun, Hung-yi Lee
arXiv AI
Jul 22

Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models

arXiv:2607. 19257v1 Announce Type: cross Abstract: Practitioners make three prompt-design decisions with almost no controlled evidence behind them: how to format instructions and context (markdown, plain text, prose, or tabular), how many simultaneous instructions a system prompt can carry before compliance degrades, and how much context a model can hold before recall and honesty degrade.

By Netanel Eliav