IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications

IdeaAMBIG is a benchmark that evaluates how well research‑idea specifications are ready for implementation. It contains 660 evidence‑grounded instances, including 163 real‑world gaps from reproducibility reports and GitHub issues and 497 synthetic gaps added to codification‑ready references. The benchmark tests three capabilities—codification‑readiness assessment, defect localization, and clarification action generation—across 13 large language models, revealing that defect localization is the main bottleneck while clarification given the defect performs much better.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

• llms
• agents
• benchmarks