EngiAI: Capability-Based Evaluation of Tool-Connected LLM Agents for Engineering Design

EngiAI introduces a capability-based evaluation framework for tool-connected engineering agents, assessing workflow execution, retrieval-assisted parameter selection, HPC orchestration, and training-code authoring using execution traces and engineering artifacts. The framework was applied to four LLM backends on EngiBench Beams2D and Photonics2D, revealing that proprietary models outperform open-source ones in workflow completion and HPC orchestration, while indexed retrieval improves parameter selection. The study demonstrates that evaluating distinct skills separately provides clearer insight into failure mechanisms than end-to-end success rates alone.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• agents
• safety