VTOS: Learning to Orchestrate Vision Tools by Co-Searching Solutions and Observers

VTOS (Vision Tools Orchestration Search) is a framework that adaptively orchestrates vision foundation tools—such as open‑vocabulary detectors, segmentation models, and post‑processing operators—by jointly searching for executable solution programs and observer programs that diagnose failures and provide feedback. The observer programs feed observations into a shared VisionThoughts knowledge base, guiding subsequent searches. In two case studies—dense object counting on LVIS‑Count and zero‑shot plant‑disease segmentation on PlantSeg‑OOD—VTOS outperforms static tool pipelines and agentic visual‑programming baselines, especially in complex scenarios like dense, occluded scenes and out‑of‑distribution segmentation.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

• rag
• agents
• computer-vision