EngiWorld is a new benchmark that tests autonomous agents across the full engineering design loop, covering 1,301 expert‑curated tasks in six domains (CAD, CAE, CAM, BIM, EDA, and 3D visualization) and 26 professional software platforms with both GUI and CLI interfaces. It introduces an artifact‑centric evaluation method that programmatically verifies geometric validity, physical feasibility, and rule compliance of both final and intermediate artifacts, scoring tasks continuously rather than with binary success. Initial tests of seven frontier models show a large capability gap, with the best model scoring only 44.3 on the EngiScore and just 3.6% of multi‑software attempts succeeding.
Computer-use agents are increasingly evaluated in realistic desktop environments, but existing benchmarks provide limited coverage of professional engineering workflows whose outputs are persistent, s...
arXiv:2609.16251v1 Announce Type: new Abstract: Computer-use agents are increasingly evaluated in realistic desktop environments, but existing benchmarks provide limited coverage of professional engi...
EngiAI introduces a capability-based evaluation framework for tool-connected engineering agents, assessing workflow execution, retrieval-assisted parameter selection, HPC orchestration, and training-code authoring using execution traces and engineering artifacts. The framework was applied to four LLM backends on EngiBench Beams2D and Photonics2D, revealing that proprietary models outperform open-source ones in workflow completion and HPC orchestration, while indexed retrieval improves parameter selection. The study demonstrates that evaluating distinct skills separately provides clearer insight into failure mechanisms than end-to-end success rates alone.
MatToolBench is a new benchmark that evaluates multimodal GUI agents on professional materials science software. It contains 204 tasks across 10 tools in three modalities—GUI operation, OriginPro scripting, and code-based database queries—executed inside a Windows 11 VM. The benchmark offers fine-grained, expert-decomposed scoring and a high-performing multimodal judge for aesthetic assessment, revealing that strong general benchmark performance does not transfer to scientific workflows.
arXiv:2607. 14896v1 Announce Type: cross Abstract: Addressing a structural-engineering request requires more than a single answer; it requires a chain of interdependent artifacts: interpreted requirements, a computable model, validation records, solver outputs, code-check records, and a final report.
arXiv:2607. 05750v1 Announce Type: new Abstract: Computer-aided design (CAD) for industrial components requires long-horizon procedural modeling, robust feature dependencies, editable parametric geometry, and production-grade B-Rep execution.
Computer-aided design (CAD) for industrial components requires long-horizon procedural modeling, robust feature dependencies, editable parametric geometry, and production-grade B-Rep execution. Existing text-to-CAD methods have made promising progress in generating CAD programs from natural-language descriptions, but they still struggle when user prompts are ambiguous, underspecified, or only describe high-level design intent.
The paper introduces KNOWS, a benchmark for evaluating web agents that act as assistants by retrieving, synthesizing, and presenting information across complex, multi-step browser tasks. It outlines a task design rubric, evaluation protocol combining deterministic checks with LLM judgments, and reports that current agents achieve only modest success, with the best performing agent succeeding on less than 3% of tasks. The study highlights significant gaps in agents’ tool use, visual understanding, and long‑horizon reasoning.
VisCAD is a foundation model suite that tackles AI-assisted computer-aided design for industrial products, covering both part-level and assembly-level generation. Its core component, VisCAD‑M1, is a 27B model trained for part-level design generation and outperforms existing models on PubCADBench and RealCADBench, achieving a part-level score of 0.5540 and reaching 0.5797 when used as a test-time verifier. VisCAD also offers a domain-specific harness that improves complex assembly generation compared to general-purpose harnesses, showing quantitative and qualitative advantages.
arXiv:2606. 14397v1 Announce Type: new Abstract: As agentic systems continue to evolve and are widely deployed in real-world scenarios, there is a growing demand to faithfully evaluate their capabilities.
arXiv:2606.14397v4 Announce Type: replace Abstract: As agentic systems continue to evolve and are widely deployed in real-world scenarios, there is a growing demand to faithfully evaluate their capab...
arXiv:2606. 11042v1 Announce Type: new Abstract: Recent years have witnessed the rapid evolution of AI agents toward handling increasingly complex, real-world tasks.