arXiv Computation and Language

From Apparent Search Gains to Selection Failures: Auditing Test-Time Inference in Structured Workflow Planning

The paper investigates how different components of test-time inference—prompting, post-processing, and selection—affect performance in structured workflow planning on DisasterBench. By breaking inference into discovery, validation, and selection stages, the authors compare independent sampling, nominal tree-search, and multi-operator search across two model families. Their findings show that while tree-search and independent sampling achieve similar workflow discovery, structural validation rarely discards correct workflows, yet majority voting often fails to select them, revealing significant tool-choice or information-flow errors. A tool-specification-aware verifier can recover some losses but also overturns more correct decisions.

arXiv AI

Oct 26

Mining the Design Space of Multi-Agent Workflows

The paper introduces a two‑phase pipeline that mines reproducible design principles for large language model multi‑agent systems by reusing all intermediate search trajectories from an MCTS optimizer. Using six composable operators and 26 Enigmata tasks, the authors generate a corpus of 28,568 executable workflows and apply an AutoResearch loop to discover and validate orchestration patterns. Key findings include that Star‑first workflows underperform, SharedMemory outperforms debate, and prompt specificity and placeholder‑based routing provide comparable gains to communication‑structure selection.

By Xavier Cadet, Junyan Cheng, Peter Chin, Kyle Richardson

llms agents

arXiv AI

Jun 12

Evoflux: Inference-Time Evolution of Executable Tool Workflows for Compact Agents

arXiv:2606. 12674v1 Announce Type: new Abstract: Compact language models (LMs) reduce cost, latency, and deployment risk for tool agents.

By Kushal Raj Bhandari, Ling Yue, Ching-Yun Ko, Dhaval Patel, Shaowu Pan, Pin-Yu Chen, Jianxi Gao

llms agents reinforcement-learning fine-tuning efficiency

arXiv AI

Jun 10

Regimes: An Auditable, Held-Out-Gated Improvement Loop Demonstrated on LongMemEval with ActiveGraph

arXiv:2606. 10241v1 Announce Type: new Abstract: Autonomous improvement loops are hard to trust because the improvement process is usually external scaffolding bolted onto the agent: failures go unlogged, diagnoses cannot be replayed, and promote-or-discard decisions land in a side database rather than the agent's own history.

By Yohei Nakajima

agents

arXiv AI

Sep 15

DepthBenchCAD: When Does Deeper Auditing Yield More Reliable Conclusions?

arXiv:2609.15122v1 Announce Type: cross Abstract: Generative CAD models are expected to remain behaviorally correct after parameter edits, so increasing the number of edit checks is often treated as...

By Hongye Yang, Zhihao Xie, Shengjun Xiong, Boxiao Huang

arXiv AI

Jun 11

TreeSeeker: Tree-Structured Trial, Error, and Return in Deep Search

arXiv:2606. 11662v1 Announce Type: new Abstract: Deep search requires agents to answer complex questions through multi-step web search, browsing, evidence comparison, and synthesis.

By Zhuofan Shi, Mingzhe Ma, Lu Wang, Fangkai Yang, Pu Zhao, Yiming Guan, Youling Huang, Wei Zhang, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan

agents efficiency

arXiv AI

Aug 6

Diagnosing Tool-Selection Reasoning in LLM Agents with Canary Tools

arXiv:2608. 04719v1 Announce Type: new Abstract: Agent evaluations tell us that a model picked the wrong tool, but rarely why.

By Atul Anand, Sourav Chattaraj

llms agents benchmarks safety

arXiv AI

Oct 15

ResearchFlow-Eval: Tracing the Execution Gap in End-to-End Multi-Claim Autoresearch

arXiv:2610.18088v2 Announce Type: new Abstract: Autoresearch systems increasingly undertake entire research projects. In practice, scientific research requires systems to pursue multiple interdepende...

By Peng Jiang, Dongyang Li, Quanying Liu, Xiaoxuan Jia

agents

arXiv AI

Aug 11

From Trajectories to Evidence: Auditable Experimental Records for Industrial Research Agents

arXiv:2608. 05235v1 Announce Type: cross Abstract: Research agents increasingly conduct multi-round machine-learning experiments in industrial recommendation settings and retain the resulting trajectories to guide later decisions.

By Zijie Zhuang, Changxin Lao, Pengbo Xu, Hanwen Xu, Ruochen Yang, Yingzhi He, Peng Zhang, Jiangxia Cao, Yusheng Huang, Guohong Mu, Jian Liang, Ruiming Tang, Shuang Yang, Zhaojie Liu, Wenwu Ou, Kun Gai

llms agents

Hugging Face Trending Papers

Aug 5

Diagnosing Tool-Selection Reasoning in LLM Agents with Canary Tools

Agent evaluations tell us that a model picked the wrong tool, but rarely why. We introduce canary tools: diagnostic probe tools planted in an agent's Model Context Protocol (MCP) tool set, each engineered to probe one specific tool-selection weakness.

llms agents benchmarks safety

arXiv AI

Sep 16

Assurance Envelopes for Autonomous Coding Agents: Minimum-Cost Evidence for Software Change

arXiv:2609.16302v1 Announce Type: cross Abstract: When a coding agent returns to existing software, it inherits evidence from earlier engineering work: tests, type checks, proofs, static analyses, an...

By Anjan Goswami

agents benchmarks

Hugging Face Trending Papers

Jun 10

TreeSeeker: Tree-Structured Trial, Error, and Return in Deep Search

Deep search requires agents to answer complex questions through multi-step web search, browsing, evidence comparison, and synthesis. A central challenge is deciding how to search when several directions look plausible but only some will later lead to reliable evidence.

agents efficiency

arXiv AI

Jul 28

Confidently Wrong: Exception Chain Collapse in Frontier LLM Rule Evaluation

arXiv:2607. 23386v1 Announce Type: new Abstract: We document a failure class in frontier large language models -- exception chain collapse -- observed in eligibility evaluation under nested conditional rules of the form "A is required UNLESS B applies, UNLESS C overrides B".

By Paul Simpson, John Kozak, Lisa Doake

llms benchmarks safety

Related stories