VTOS (Vision Tools Orchestration Search) is a framework that adaptively orchestrates vision foundation tools—such as open‑vocabulary detectors, segmentation models, and post‑processing operators—by jointly searching for executable solution programs and observer programs that diagnose failures and provide feedback. The observer programs feed observations into a shared VisionThoughts knowledge base, guiding subsequent searches. In two case studies—dense object counting on LVIS‑Count and zero‑shot plant‑disease segmentation on PlantSeg‑OOD—VTOS outperforms static tool pipelines and agentic visual‑programming baselines, especially in complex scenarios like dense, occluded scenes and out‑of‑distribution segmentation.
arXiv:2605.19410v2 Announce Type: replace Abstract: Segmentation has become easy when the concept is known, requiring retrieval of a learned visual grounding from text. It remains hard for open ad-ho...
The paper introduces the Necessary Tool‑Evidence Path (NTEP) annotation scheme and its associated reward mechanism (NTEP‑R) to better supervise vision‑language models that use external tools. By explicitly specifying which evidence is needed and penalizing redundant tool calls, the authors train an 8B‑parameter model that shows improved accuracy and tool‑use efficiency across seven image‑grounded benchmarks. The approach demonstrates that fine‑grained supervision of tool‑evidence paths is essential for robust agentic VLM performance.
arXiv:2609.24362v1 Announce Type: new Abstract: Sandboxed computer environments support multi-step reasoning with tools, executable programs, and persistent files, yet their extension from language m...
arXiv:2607. 21155v1 Announce Type: cross Abstract: Knowledge-Intensive Visual Question Answering (KI-VQA) benchmarks evaluate Vision-Language Models (VLMs) as multimodal knowledge assistants by requiring external information beyond a provided image to answer questions.
Knowledge-Intensive Visual Question Answering (KI-VQA) benchmarks evaluate Vision-Language Models (VLMs) as multimodal knowledge assistants by requiring external information beyond a provided image to answer questions. KI-VQA involves multiple sub-problems -referring expression understanding, visual grounding, object recognition, knowledge retrieval, and reasoning-yet existing benchmarks typically report only end-task accuracy, obscuring where failures arise.
arXiv:2608. 08219v1 Announce Type: cross Abstract: Video anomaly detection (VAD) is a critical yet challenging task due to the complex and diverse nature of real-world scenarios.
VBVR-Pro is a closed‑loop testbed that enables native visual reasoning through generation, offering 300 procedurally generated tasks that scale training and allow strong transfer to external benchmarks. It supplies verifiable reward scorers based on deterministic, task‑specific rules, outperforming VLM‑as‑a‑judge approaches and providing reliable signals for reinforcement learning. The suite also facilitates controlled modality studies, revealing that video generation excels at persistent spatiotemporal tracking while interleaved generation offers a compute‑efficient alternative, and highlights the importance of vision‑native trajectories for reasoning.
OccludeBench is a manually curated benchmark that tests vision‑language models (VLMs) on their ability to recover fully occluded objects from remaining visual evidence. The benchmark distinguishes direct cases—requiring recovery from a specific physical cue such as a shadow or reflection—from inferential cases, which rely on broader scene context. Across five VLMs, accuracy averages 42.7 % on direct cases versus 62.7 % on inferential cases, a 20‑point gap, while humans achieve 63.4 % and 68.5 % respectively with no significant gap. The study introduces OccludeTrace, a four‑stage diagnostic that isolates the failure point, revealing that models struggle most when recovering the relevant object from the full image projection. Interventions such as inpaint‑then‑confront reduce the gap to a statistically non‑significant difference for models with image‑generation capabilities.
arXiv:2608.29783v1 Announce Type: new Abstract: Industrial anomaly detection is a critical component of modern manufacturing. Most traditional unsupervised methods rely on modelling normal feature di...
arXiv:2609.22588v1 Announce Type: new Abstract: Vision-language models (VLMs) perform strongly on visual question answering benchmarks, yet often make decisions that contradict visual evidence they h...
arXiv:2607. 11818v1 Announce Type: cross Abstract: We introduce MM-ToolSandBox, a benchmark and evaluation framework for visually grounded tool-calling agents.
arXiv:2608.21762v1 Announce Type: cross Abstract: Vision-language models (VLMs) fail many detail-centric questions for a concrete reason: the answer is visible in the image, yet lost after the image...