AI agents

Tool use, function calling, orchestration and the protocols that let models act rather than only answer.

15,733 stories · RSS feed

arXiv Computation and Language
Oct 26

The Schema Coherence Inventory: Measuring Whether an Assigned Belief Organizes a Language Model's Cognition

The paper introduces the Schema Coherence Inventory (SCI), a 96-item forced-choice test designed to determine whether a belief assigned to a persona-conditioned language model actually organizes its behavior or merely reflects affect. SCI evaluates belief organization across eight levels of Beck's cognitive model, using length‑normalized log‑likelihood scores and a target‑specificity criterion to distinguish true belief influence from mood effects. Validation across five model backbones (1.7B–14B parameters) and nine clinicians shows high reliability and discriminant validity, and a 48‑item short form correlates strongly with the full inventory.

By Yuqi Wu, Deyang Li, Katharine Hibbard, Ron Oswald, David Sinclair, Adegboyega Sapara, Kalutota Kuruwe Arachchige Sajana Indula Samarasinghe, Abhinav Pillai, Reham Shalaby, Riffat Alam, Xin-Min Li, Yanbo Zhang, Jie Chen
arXiv AI
Oct 26

MCPHunt: An Evaluation Framework for Cross-Boundary Data Propagation in Multi-Server MCP Agents

MCPHunt is a benchmark framework that isolates non‑adversarial credential propagation across multi‑server MCP trust boundaries. It uses canary‑based taint tracking, a controlled coverage design with risky, benign, and hard‑negative conditions, and completion‑requires‑secret stratification to separate faithful from policy‑violating propagation. Across 3,615 traces from 5 models, policy‑violating propagation rates range from 11.5 % to 41.3 %, with mitigation via prompting reducing rates to 0.3 % while retaining 80.5 % utility.

By Haonan Li, Tianjun Sun, Yongqing Wang, Qisheng Zhang
arXiv AI
Oct 26

What Open-Source Projects Tell Their Coding Agents: A Census of AGENTS.md and CLAUDE.md

The study surveys popular, active GitHub repositories across thirteen languages, treating each repository’s AGENTS.md and CLAUDE.md files as instruction sets for coding agents. It finds that most repositories include such instructions, with CLAUDE.md often referencing AGENTS.md, and that these files contain a mix of project knowledge and workflow rules, including varying stances on AI attribution. The analysis reveals that more popular projects tend to have stricter pull‑request rules and AI contribution guidelines, and that LLM and agent projects are over‑represented among adopters.

By Pawel Zarzycki
arXiv AI
Oct 26

Benchmarks Are Surprisingly Compaction-Friendly

The paper investigates how context compaction—losing information to free up a language model’s context window—affects performance on long‑horizon agentic tasks. Using the OpenCode scaffold and five compaction strategies, the authors evaluate three frontier models on PaperBench, ProgramBench, and PostTrainBench, finding that compaction thresholds largely do not hurt performance for 1M‑context models, but can degrade GPT‑5.3‑Codex on certain benchmarks. The study also shows that compaction can silently drop critical task constraints, especially on ProgramBench and real user‑agent sessions from SWE‑chat, yet current benchmarks remain largely compaction‑friendly because objectives are fixed and progress is stored externally.

By Jeanne Salle, Hardik Bhatnagar, Matthias Bethge, Sahar Abdelnabi, Maksym Andriushchenko
arXiv AI
Oct 26

Interpretable sparse fine-tuning with TopKLoRA enables faithful circuit discovery

The paper introduces TopKLoRA, an interpretable parameter‑efficient fine‑tuning method that activates only the top‑k adapter latents per token. Using this sparse LoRA, the authors fine‑tune large language models on a poisoned dataset, identify and ablate the minimal circuit responsible for a backdoor, and show that the resulting circuits are an order of magnitude smaller than dense LoRA baselines. They also demonstrate how to construct model organisms with localized circuits via gradient‑routing, supporting the development of interpretable‑by‑design architectures.

By Marek Masiak, Andrzej Szablewski, Nicola Cancedda, Constantin Venhoff
arXiv AI
Oct 26

Reuse, Rewrite, or Reject: Evidence-Constrained Trace Reuse for Repeated Mobile GUI Control

The paper introduces a framework that selectively reuses parts of previously executed mobile GUI control traces to reduce LLM inference overhead. It identifies six categories of variation that can affect trace reuse and shows that individual transitions can still be valid when checked against the current task context and UI state. Using graph-text templates, the system achieves roughly 80% fewer LLM tokens and up to 2‑fold speedups on Android and HarmonyOS devices.

By Bingyu Bai, Cheng Zhang, Erhu Feng
arXiv AI
Oct 26

Don't Forget What You Just Learned: Self-Directed Repair Turns Agent Mistakes Into Lasting Lessons

The paper introduces Self‑Direction, a method that lets agents learn from mistakes by reflecting on failed trajectories, distilling lessons into lightweight low‑rank adapters, and storing them for future use. These adapters add only about 0.02% of the base model’s parameters, keeping the original weights frozen and avoiding costly fine‑tuning. Experiments on Qwen‑based agents across research, tool‑use, and terminal tasks show 8.9–17.0 percentage‑point gains in success rates, reduced repeated errors, and transfer of lessons to new tasks.

By Yuval Kansal, Karen Ullrich, Polina Kirichenko, Ilia Kulikov, Niraj K. Jha, Mohammad Pezeshki, Mark Ibrahim
arXiv AI
Oct 26

DMAIC-Bench: From Self-Reflection to Self-Restraint for Agents in Multi-Stage Analytical Tasks

DMAIC-Bench is a new benchmark that tests large‑language‑model agents on their ability to exercise self‑restraint—deciding when to continue, redirect, or stop a multi‑stage analytical task based on intermediate evidence. It contains 97 seeded tasks across 15 industrial cases in four sectors, with paired counterfactual variants that require opposite actions to distinguish warranted restraint from indiscriminate stopping. Evaluation involves a private scorer for execution traces and independent expert review, revealing that while agents often route correctly on GO decisions, they frequently act on invalid evidence at NO‑GO decisions, with a 47% average error rate. whyItMatters":"The benchmark provides a systematic way to measure whether agents can appropriately use evidence to control their own actions, a capability that current benchmarks largely overlook."

By Shan Peng, Shuaiqi Cheng, Miao Peng, Chenyi Zi, Jia Li
arXiv AI
Oct 26

MedInsightBench: Evaluating Medical Analytics Agents Through Multi-Step Insight Discovery in Multimodal Medical Data

MedInsightBench is a new benchmark comprising 332 curated medical cases with annotated insights, designed to evaluate large multimodal models (LMMs) and agent frameworks on their ability to analyze medical image data, ask relevant questions, interpret findings, and synthesize actionable recommendations. The study finds that current LMMs perform poorly on this benchmark due to difficulties in extracting multi-step insights and lacking medical expertise. To address this, the authors propose MedInsightAgent, an automated agent framework with three modules—Visual Root Finder, Analytical Insight Agent, and Follow-up Question Composer—that improves LMM performance on MedInsightBench.

By Zhenghao Zhu, Chuxue Cao, Sirui Han, Yuanfeng Song, Xing Chen, Caleb Chen Cao, Yike Guo
arXiv AI
Oct 26

Train4Merge: RL or SFT---Which Yields Better Teachers for OPD-Based Model Merging?

The study compares supervised fine‑tuning (SFT) and reinforcement learning (RL) as methods for training teacher models that are later merged into a single student via on‑policy distillation (OPD). Across agentic, reasoning, and perception tasks, RL‑trained teachers produce stronger students and better recover teacher performance gains than SFT teachers, both in single‑teacher and multi‑teacher OPD settings. Analysis shows RL teachers deviate less from the shared initialization, leading to faster and higher overlap in teacher–student top‑K predictions, which appears to facilitate more effective OPD learning.

By Jingyuan Huang, Zuming Huang, Yucheng Shi, Zhongzhi Li, Xiaoming Zhai, Wei Chu, Ninghao Liu
arXiv AI
Oct 26

PaperFormalizer: Author-in-the-Loop Autoformalization of Research Papers in Lean

PaperFormalizer is a Lean-based pipeline that assists authors in identifying and closing gaps in their research papers, converting them into Lean-verified versions. Starting from LaTeX, it transcribes the paper into a dependency graph, audits statements for faithfulness and correctness, and formalizes them in Lean using Mathlib and optional domain libraries. The process is author-in-the-loop, prompting the author whenever multiple interpretations exist and allowing them to accept, decline, or redirect proposed repairs, ultimately producing an annotated paper, a change report, and linked Lean code.

By Jiyuan Tan, Vasilis Syrgkanis, Jose Blanchet