arXiv AI
Jun 26
Confidence-Aware Tool Orchestration for Robust Video Understanding
arXiv:2606. 26904v1 Announce Type: cross Abstract: Video reasoning language models implicitly assume that every input frame is equally reliable.
Accuracy and Cost Claims Do Not Survive Re-Execution in Agentic VideoQA
Related stories
arXiv AI
Oct 9
AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use Tasks
AgentHorizon is a benchmark comprising 1,373 computer‑use tasks with instruction‑trajectory pairs collected from 166 hours of human‑recorded activity across three operating systems. The benchmark uses a paired design that swaps instructions to create negative tasks, testing judges’ ability to distinguish successful trajectories from those that completed a similar but incompatible request. Eleven judges were evaluated, with GPT‑5.5 achieving the highest balanced accuracy of 80.9% on the full trajectory subset, while tool‑use benefits some models but harms others, revealing large variability in judges’ performance.
arXiv Machine Learning
Oct 8
Coding-Agent Benchmarks Should Match Their Users' Task Flows
arXiv:2610.09633v1 Announce Type: new Abstract: The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engine...
arXiv AI
Jul 14
AgentAbstain: Do LLM Agents Know When Not to Act?
arXiv:2607. 10059v1 Announce Type: new Abstract: Agent systems based on large language models (LLMs) are increasingly deployed for autonomous tasks, yet existing evaluations mostly focus on task success rather than whether agents know when to abstain.
Hugging Face Trending Papers
Sep 2
VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement
VeriPhy is an auditable physical‑verification system that evaluates generated video by compiling prompts into typed physical obligations and a statically validated execution plan before any frames are observed. During execution, it gates calls to frozen low‑level experts (e.g., segmentation, tracking, counting, depth, OCR, audio‑event detection) and returns provenance‑carrying evidence records, which are mapped to a three‑valued state (supported, contradicted, unknown) with full traceability. On a 1,500‑clip corpus of human‑annotated flaw records, VeriPhy accounts for 228 failures out of 304, outperforming a published question‑decomposition evaluator that accounts for 164, while also providing auditable evidence for each verdict.
Hugging Face Trending Papers
Oct 7
Coding-Agent Benchmarks Should Match Their Users' Task Flows
The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engineers in JetBrains IDEs, which we call Production S...
arXiv Computer Vision
Sep 28
STORM-Bench: Evaluating Online Video QA under Evolving and Incomplete Evidence
STORM-Bench is a new benchmark for online video question answering that evaluates models’ ability to track state transitions and selectively abstain when visual evidence is insufficient. It contains 5,736 questions across 630 short, change‑dense episodes in five egocentric domains and two simulation subsets, with questions stratified by change intensity and answerability. The benchmark introduces STORM‑BR, a harmonic metric that reveals abstention failures and overconfidence on uncertain queries, showing that traditional accuracy masks gaps in epistemic reliability and state tracking.
arXiv Computer Vision
Sep 4
VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement
VeriPhy is an auditable physical‑verification system that transforms a text prompt into typed physical obligations and a statically validated execution plan before any video frames are generated. During execution, it gates calls to frozen low‑level experts (segmentation, tracking, counting, depth, OCR, audio‑event detection, etc.) and records provenance‑carrying evidence for each action. The system maps these records to a three‑valued state—supported, contradicted, or unknown—providing traceable verdicts that can be used to refine generation models.
arXiv AI
Aug 7
SkillTV-Bench: Benchmarking How Well Judges Perform on Skill-Augmented Agentic Execution
arXiv:2608. 05573v1 Announce Type: new Abstract: LLM agents increasingly execute long-horizon tasks through tool use and environment interaction, shifting evaluation from final-response scoring to verification of complete executions.
arXiv Computation and Language
Oct 9
TRACE: Diagnosing Verifier Brittleness in Agentic Evaluation
The paper introduces TRACE, a protocol designed to diagnose whether changes in verifier scores for large language model agents reflect actual changes in agent behavior or merely alterations in the evaluation process. TRACE works by applying targeted changes to evaluation components, running paired experiments, and re‑scoring unchanged trajectories to isolate the source of score variation. Experiments on synthetic tasks and public benchmark tasks demonstrate that seemingly significant score shifts can often be attributed to evaluation artifacts, while TRACE can also detect genuine behavioral changes.
arXiv Computer Vision
Oct 15
VideoHarness-SRSI: Structured Recursive Self-Improvement for Video Understanding Harness
arXiv:2610.19424v1 Announce Type: new Abstract: LLM system capabilities depend not only on model parameters but also on the executable harnesses that organize information acquisition, tool use, reaso...
arXiv Machine Learning
Aug 28
Finding the Right Evidence: Factor-Guided Coarse-to-Fine Reasoning for Long Videos
The paper introduces PACE, a factor-guided, progressive framework for acquiring evidence in long-video question answering. PACE first indexes clip-level descriptions using question-derived factors, then refines evidence retrieval with contrastive cues derived from candidate answers. On the MMR‑V dataset, PACE achieves 42.6% accuracy and recovers 66.9% of annotated cues, outperforming direct inference and prior agentic baselines, and shows consistent improvements across several long-video benchmarks.