arXiv AI
Sep 18

BuildBench: Benchmarking LLM Agents on Compiling Real-World Open-Source Software

BuildBench introduces a realistic benchmark for evaluating large language model agents on the task of compiling open‑source software (OSS). It includes diverse OSS projects that lack clear build instructions, have undocumented dependencies, and may require source patching or script modification. The authors also present OSS‑BUILD‑AGENT, a baseline LLM‑based agent that retrieves build instructions effectively and achieves state‑of‑the‑art performance on the benchmark.

By Zehua Zhang, Ati Priya Bajaj, Divij Handa, Siyu Liu, Arvind S Raj, Hongkai Chen, Hulin Wang, Yibo Liu, Zion Leonahenahe Basque, Souradip Nath, Vishal Juneja, Nikhil Chapre, Tiffany Bao, Yan Shoshitaishvili, Adam Doup\'e, Chitta Baral, Ruoyu Wang
arXiv AI
Sep 7

MaxKernel: Agentic Kernel Generation for TPUs

MaxKernel is a multi‑agent system designed to generate high‑performance custom kernels for TPUs. It offers three paradigms: a Human‑in‑the‑Loop agent for collaborative design, an Autonomous agent that runs a fully automated optimization loop, and a Graph‑Based Autonomous Search for global exploration. All paradigms share specialized sub‑agents for planning, implementation, debugging, testing, and profiling, and the system consistently matches expert hand‑tuned baselines on the JaxBench suite and real‑world workloads.

By Shangkun Wang, Nina Cai, Charles Hoong, Julian Walker, Gerson Kroiz, George Vanica, Deepak Patil, Andi Gavrilescu, Hassan Sipra, Sethu Sankaran