← Back to all news
arXiv Computation and Language By Olga Loginova, Oleksandr Bezrukov, Frank Keller

Through a PRISM, Clearly: Step-Sequence Reasoning for Mistake Recognition in Procedural Videos

Read the original on arXiv Computation and Language →

The paper introduces PRISM, a method for recognizing procedural mistakes in videos by grounding video segments in language and mapping them to procedural states using a lightweight hidden Markov model trained on correct executions. It aligns reference annotations with four edit-based error types—insertions, deletions, substitutions, and transpositions—to evaluate across heterogeneous benchmarks. Experiments on three datasets show that using language grounding, semantic role labeling, and a trained local decoder consistently improves mistake detection performance compared to nearest-candidate matching.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Machine Learning
Aug 31

Post-Training VLMs for Video Mistake Detection

The paper introduces a new protocol, Mistake Detection Video Question Answering (MD‑VQA), to evaluate whether models can determine if a step in a video follows its description, covering both seen and unseen actions. It proposes a post‑training approach for video‑language models that uses a reward function to highlight discrepancies between instructions and video content. Experiments show this method surpasses zero‑shot, fine‑tuned, and other post‑training baselines, especially on unseen procedures, improving performance by up to 11.6% on EP‑VQA.

By Federico Spurio, Olga Zatsarynna, Lars Doorenbos, Emad Bahrami, Gianpiero Francesca, Juergen Gall
nlpfine-tuningbenchmarks
More like this →
arXiv AI
Aug 19

SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation

SemComp-Bench introduces a new video generation task called Semantic Task Completion, where a model must produce a video that achieves a specified outcome while maintaining semantic alignment with a reference image. The benchmark includes the SemComp-Data dataset, spanning six domains, and a four-stage curation pipeline that transforms raw videos into standardized instances. Evaluation is performed via a vision‑language model that answers structured binary questions, yielding Outcome Achievement (OA) and Generation Reliability (GR) scores.

By Keyu Tu, Zhuowei Chen, Mengqi Huang, Yuxin Wang, Jiahao Zhu, Zhendong Mao, Yongdong Zhang
llmsdiffusionmultimodalbenchmarks
More like this →
arXiv Computer Vision
Sep 23

TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Action

arXiv:2505.01583v2 Announce Type: replace Abstract: Understanding causal event relationships and achieving fine-grained temporal grounding in videos remain challenging for vision-language models (VLM...

By Jen-Hao Cheng, Yi-Hao Peng, Huapeng Zhou, Vivian Wang, Huayu Wang, Hsiang-Wei Huang, Wenhao Chai, Hou-I Liu, Kuang-Ming Chen, Cheng-Yen Yang, Yi-Ling Chen, Vibhav Vineet, Qin Cai, Jenq-Neng Hwang
llmscomputer-visionmultimodalbenchmarks
More like this →