InfraDev: Can Coding Agents Develop AI Infrastructure, and How Can We Tell?
Read the original on arXiv AI →InfraDev is a new benchmark comprising 36 tasks derived from real-world pull requests to Megatron‑LM, vLLM, and verl, designed to test coding agents on infrastructure development for training, inference, and reinforcement learning across many GPUs. Each task requires agents to reimplement multi‑file changes and run correctly, with grading that includes hidden probes and a prompt exposing only the interfaces used. The benchmark reveals that current frontier models resolve only about 39% of tasks, with probes rejecting a third of submissions that pass official tests, and that structural and state errors dominate failures.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.