Agent4Spec is an LLM-based autonomous multi‑agent system designed to improve API specifications through continuous validation and market analysis. It includes a validation pipeline that detects and self‑corrects OpenAPI syntax errors, and a market analysis pipeline that performs competitive research, identifies functional gaps, and automatically generates missing API‑spec functionalities. In evaluations on curated datasets, the best LLM achieved 100% syntax error detection, over 95% success in implementing corrections, and more than 50% adoption of suggested endpoint improvements.
By Yongjian Tang, Costel Moraru, Doruk Tuncel, Hanya Elhashemy, Steffen Klepke, Thomas Runkler
As Large Language Models (LLMs) evolve into autonomous agents, the need for unified evaluation infrastructure becomes critical. However, current evaluation pipelines remain highly fragmented and tightly coupled, hindering reproducibility and causing redundant engineering.
arXiv:2607. 13705v1 Announce Type: new Abstract: As Large Language Models (LLMs) evolve into autonomous agents, the need for unified evaluation infrastructure becomes critical.
By Zichen Ding, Jiaye Ge, Shufan Jiang, Kai Chen, Mo Li, Qingqiu Li, Zehao Li, Zonglin Li, Tiaohao Liang, Shudong Liu, Zerun Ma, Zixing Shang, Wenhui Tian, Zun Wang, Liwei Wu, Zhenyu Wu, Jun Xu, Bowen Yang, Dingbo Yuan, Qi Zhang, Songyang Zhang, Peiheng Zhou, Dongsheng Zhu
Large language models (LLMs) are increasingly used in agentic coding settings, where they can inspect files, execute commands, run tests, observe failures, and iteratively revise code. This shift raises a central evaluation question: can an agentic LLM generate an end-to-end software artifact that is both deployable and behaviorally correct under execution?
WiseSpec is a requirements‑driven agent framework designed to improve repository‑level code generation. It automatically builds structured, information‑rich requirements, evaluates their quality via execution‑based tests, and iteratively refines them to better guide code generation. Experiments show WiseSpec outperforms all baselines, achieving an average 13.17% improvement in %Resolved.
By Zhao Tian
BabelCoder is an agentic framework for automatic code translation that splits the task into specialized agents for translation, testing, and refinement. Each agent focuses on a specific aspect—generating code, validating correctness, or repairing errors—allowing collaborative improvement of translation quality. Evaluated on four benchmark datasets, BabelCoder outperforms four state‑of‑the‑art baselines, achieving an average accuracy of 94.16% and surpassing existing methods in 94% of cases.
By Fazle Rabbi, Soumit Kanti Saha, Tri Minh Triet Pham, Song Wang, Jinqiu Yang