SurgStreamer is a system that proactively monitors surgical video streams by separating procedure‑specific perception—mapping a 1‑FPS video to a structured state of tools, actions, and phases—from procedure‑agnostic reasoning that evaluates formal monitoring conditions over this state and an event memory. In a sealed test set of 12 laparoscopic cholecystectomies, SurgStreamer achieved a macro event‑F1 of 0.598 over full procedures and 0.677 on 75‑second windows, outperforming streaming vision‑language models. Re‑training only the perception component improved held‑out performance from 0.239 to 0.470, and a controller trained on cholecystectomy data performed comparably to an in‑domain controller on pituitary surgery, highlighting perception under distribution shift as the main limitation.
By Bowen Xiang, Jon S. Heiselman, Michael I. Miga
PROVIA is a system for online mistake detection in egocentric videos that tracks procedure state by maintaining a factual state and a learned summary of performed steps, including mistakes. It uses Bayesian state merging to create an automaton from correct demonstrations and applies a sequential test to convert per‑frame mistake probabilities into alarms. Evaluated on datasets such as CaptainCook4D, IndustReal, HoloAssist, and IMPACT‑ego, PROVIA outperforms baseline methods while operating at 58–70 fps.
By Di Wen, Kailun Yang, Jimmy Weissert, Luc Maria Scherrer, Cedric Z\"ollner, Ruiping Liu, Yufan Chen, Jiale Wei, Junwei Zheng, Kunyu Peng
TRACE (Temporal Audit and Condition-aware Evaluation) is a new benchmark and evaluation framework for streaming video understanding that explicitly records when evidence becomes valid, how visual history is maintained, and how responses are triggered. It combines temporally audited visual tasks, evidence timing, instruction-dependent trigger annotations, a unified causal Core–Adapter protocol, and multidimensional reporting of answer quality, timeliness, response-selection behavior, workload, completion, and reliability. Using 1,240 records from 517 videos, TRACE evaluated eight publicly available models in eight configurations, revealing that similar QA accuracy can hide significant differences in completion, answer validity, generation workload, and proactive performance metrics such as response delay, false alarms, and missed target windows.
By Yibo Ma, Qianqian Zhang, Peng Liu, Tiancheng Zhao
arXiv:2603.29962v4 Announce Type: replace
Abstract: Surgical procedures are inherently complex and risky, requiring extensive expertise and constant focus to navigate evolving intraoperative scenes....
By Shi Li, Vinkle Srivastav, Nicolas Chanel, Saurav Sharma, Nabani Banik, Lorenzo Arboit, Kun Yuan, Pietro Mascagni, Nicolas Padoy
arXiv:2610.34926v1 Announce Type: new
Abstract: An action seen once and the same action seen three times receive the same action-recognition label, so that supervision does not require a model to cou...
By Yihao Hu, Pan Wang, Yitong Lin, Zhihao Wen, Deyuan Liu, Hang Wang, Mingda Wan, Yiwen Zhao, Yutong Li, Yuchen Mao, Tao Lin
arXiv:2608. 14015v1 Announce Type: cross Abstract: Understanding tens-of-minutes surgical videos requires long-horizon temporal reasoning, answering what happens before, after, or across stages of a procedure by grounding the question in visual evidence spread across time.
By Yingying Fan, Penghui Du, Leyan Zhu, Runze He, Zimeng Wu, Yuxuan Zhang, Liang Chen, Jiahao Xie, Jiangtang Wang, Shuai Shao, Anchao Yang, Yutong Bai, Yan Wang
VeriPhy is an auditable physical‑verification system that evaluates generated video by compiling prompts into typed physical obligations and a statically validated execution plan before any frames are observed. During execution, it gates calls to frozen low‑level experts (e.g., segmentation, tracking, counting, depth, OCR, audio‑event detection) and returns provenance‑carrying evidence records, which are mapped to a three‑valued state (supported, contradicted, unknown) with full traceability. On a 1,500‑clip corpus of human‑annotated flaw records, VeriPhy accounts for 228 failures out of 304, outperforming a published question‑decomposition evaluator that accounts for 164, while also providing auditable evidence for each verdict.
An action seen once and the same action seen three times receive the same action-recognition label, so that supervision does not require a model to count. Answers shared across histories that differ i...
The paper introduces SurgHiBench, a hierarchy-aware evaluation suite for surgical video understanding that measures recognition, consistency, and severity across different granularity levels. It also presents HyperSurg, a hyperbolic model that enforces phase-step containment using entailment cones, evaluated on four datasets covering three procedure types. The study shows that models with identical accuracy can differ significantly in error severity, and that hyperbolic geometry improves predictions by aligning them with the procedural hierarchy.
By Ana Manzano Rodr\'iguez, Pascal Mettes, Marlies P. Schijven, Cees G. M. Snoek
arXiv:2606. 26443v1 Announce Type: cross Abstract: A robot working alongside people must reason about what they have done, in what order, and with what intent.
By Baiqi Li, Ce Zhang, Yu Fang, Yue Yang, Shangzhe Li, Mingyu Ding, Gedas Bertasius
ValueFormer introduces stage‑aware, continuous value labels for online failure detection in vision‑language‑action policies. The method uses a 3.5 M‑parameter causal transformer over a frozen DINOv3 backbone to emit per‑frame smooth Monte‑Carlo values for advantage estimation and sharp binary signals for mistake detection. In experiments, reusing teleoperation intervention flags boosts detection AP from 0.38 to 0.82 and flags 95 % of mistakes early; on a bimanual sandwich station, per‑frame post‑training weighting raises task completion from 70 % to 85 % and eliminates a repeat‑pick failure mode.
By Inkyu Sa, Konstantin Stulov, Rajat Bhageria
Surgical procedures follow a phase-to-step hierarchy, yet the video-language models used to recognize them are evaluated with flat per-level metrics that ignore cross-level coherence and error structu...