Automated Trajectory Evaluation for Mobile Agents via Step-Level Consequence Reasoning and Aggregation

The paper introduces CRATE, a two‑stage vision‑language model framework that evaluates mobile agents by reasoning about each step’s consequences and aggregating this evidence to assess task completion. It also presents CRATE‑S, an extension that evaluates operational safety. Experiments show CRATE and CRATE‑S outperform existing benchmarks, achieving high F1‑scores on AndroidWorld and MobileRisk datasets.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• agents
• multimodal
• benchmarks
• safety