From Apparent Search Gains to Selection Failures: Auditing Test-Time Inference in Structured Workflow Planning
Read the original on arXiv Computation and Language →The paper investigates how different components of test-time inference—prompting, post-processing, and selection—affect performance in structured workflow planning on DisasterBench. By breaking inference into discovery, validation, and selection stages, the authors compare independent sampling, nominal tree-search, and multi-operator search across two model families. Their findings show that while tree-search and independent sampling achieve similar workflow discovery, structural validation rarely discards correct workflows, yet majority voting often fails to select them, revealing significant tool-choice or information-flow errors. A tool-specification-aware verifier can recover some losses but also overturns more correct decisions.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.