K-Bench: A Benchmark for LLM Unlearning in Agentic Deployments

K-Bench is a new benchmark designed to evaluate large language model (LLM) unlearning when the models are deployed as agents. Unlike previous benchmarks that only inspect the final answer, K-Bench examines all six channels of a ReAct agent—including chain-of-thought, tool calls, tool observations, and elicited summaries—to determine if a secret is leaked. The benchmark measures leakage for secrets placed in the model weights, prompt, or retrieval store, and finds that many existing unlearning methods fail to prevent leaks in deployed agents, especially when secrets reside in the prompt or retrieval store.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• agents
• benchmarks