Mingbird is a local‑first agent harness designed for small open‑weight language models (2–9 B) that run on ordinary laptops. It introduces ten mechanisms—such as a byte‑level net‑zero prefill budget, a finish gate that re‑reads the task before accepting completion, and signature‑level loop detection—to address common failure modes that arise from the harness rather than the model itself. In controlled experiments on the LRAB benchmark and the $ au^2$‑bench, Mingbird achieves higher overall scores (0.886 and 0.856 respectively) compared to other harnesses, and its ablation studies show that each mechanism contributes measurable performance gains.
By Hao Wang, Ting Huang
arXiv:2609.27490v1 Announce Type: new
Abstract: AI research agents need reliable knowledge of how their experiments change outcomes. We introduce WhatWorkedBench to measure experimental understanding...
By Jingjie Ning, Xueqi Li, Yibo Kong, Dongting Li
The paper reports on training and calibration experiments for three open models on the ARC‑AGI‑2 benchmark, using offline runs on four L4 GPUs for up to 12 hours. For the NVARC Qwen3‑4B grid model, a curated corpus of 12,000 executable rules and 95,993 tasks was built, and fine‑tuning on 1,920 real task records raised the semi‑private score from 28.96 to 33.06, with no synthetic‑task checkpoint outperforming the real‑task model. For the Qwen3.8‑27B reasoning model, fine‑tuning on 1,036 of its own reasoning traces and a specialized inference procedure increased the score from 26.81 to 44.03, while the Qwen3.8‑Flash‑Next mixture‑of‑experts model achieved a best score of 56.67 after expert pruning, routing statistics adaptation, and an FP8 indexer cache.
By Nguyen Minh Thang, Trinh Thi Phuong
EGI‑BENCH is a unified benchmark suite for evaluating large language model agents in enterprise settings, consolidating eight existing benchmarks (CRMARENA, HERB, PERSONABENCH, USERBENCH, MCP‑UNIVERSE, MCPEVAL, BENCHFORCE, LOCOBENCH) under a single command‑line interface with two runtime‑budgeted modes. It introduces the Enterprise General Intelligence (EGI) Score, a six‑dimensional framework (M1‑M6) that aggregates fine‑grained sub‑metrics from each benchmark rather than simple averages, providing a more nuanced view of model capabilities. The paper reports results for fourteen LLMs across GPT, Claude, and Gemini families, showing varied strengths across task outcome, planning, retrieval, tool actioning, safety, and engineering quality.
By Zhiwei Liu, Zhujun Lan, Jielin Qiu, Zixiang Chen, Liangwei Yang, Ming Zhu, Akshara Prabhakar, Wenting Zhao, John Emmons, Jason Wu, Shafiq Rayhan Joty, Junnan Li, Shelby Heinecke, Silvio Savarese, Huan Wang
ToolBottleneckBench is a benchmark that evaluates tool‑agent performance under controlled pressures and fixed harness budgets. It keeps synthetic enterprise tasks constant while varying one pressure—such as the registered tool catalog, visible tool menu, certified workflow, or record encoding—to measure how each affects success. Across 21 model and harness configurations, the visible menu pressure uniquely imposes a large penalty on the model, largely due to truncation at the completion cap, and most failures are driven by harness limits rather than tool‑specific issues.
By Zichuan Xu, Yuyang Dong, Makoto Onizuka, Chuan Xiao
FlavourBench is an automated benchmark that evaluates language models on culinary tasks by providing a versioned, executable ground truth system. Each task presents eight ingredients and asks the model to propose a three‑ingredient portfolio, with all 56 possible portfolios scored by the Epicure system before model execution. The benchmark assesses 27 frontier endpoints across 534 tasks, yielding a FlavourBench Score that averages frozen task scores across families, and includes extensive statistical analysis and reproducible data releases.
By Josef Chen, Erim Hayretci
arXiv:2602.13217v2 Announce Type: replace
Abstract: Reasoning benchmarks need renewal along two axes: freshness and headroom. VeRA makes both executable and auditable by turning each item into a task...
By Zerui Cheng, Jiashuo Liu, Chunjie Wu, Jiayang Sun, Jianzhu Yao, Pramod Viswanath, Ge Zhang, Wenhao Huang
Deep Research Bench II is a new benchmark designed to evaluate Deep Research Agents (DRAs) by requiring them to produce research reports for 132 grounded tasks across 22 domains. Each report is assessed using 9,430 fine‑grained binary rubrics that cover information recall, analysis, and presentation, all derived from expert‑written investigative articles through a rigorous LLM‑plus‑human pipeline. Evaluation of current state‑of‑the‑art DRAs shows that even the best models satisfy fewer than 50% of these rubrics, highlighting a significant gap between automated agents and human experts.
By Ruizhe Li, Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, Zhendong Mao
arXiv:2608. 15089v1 Announce Type: new Abstract: Long-horizon agents can fail even when their underlying models can solve the constituent steps.
By Ziheng Qin, Yaxin Lu, Zhangyang Atlas Wang, Kai Wang
arXiv:2610.27645v1 Announce Type: new
Abstract: As computer-use agents (CUAs) move to long-horizon workflows spanning multiple applications, undetected errors can silently corrupt persistent state an...
By Xu Hu, Haoming Li, Yuxiang Wang, Jinxin Yang, Kangshuo Li, Licheng Wang, Dingyi Kang, Jiayu Song, Tianhao Wang, Zhanhua Hu, Xu Yang, Qiannan Li, Bingzhe Li, Feng Chen
AI research agents need reliable knowledge of how their experiments change outcomes. We introduce WhatWorkedBench to measure experimental understanding, the accuracy of predictions about component cha...
arXiv:2607. 20468v1 Announce Type: new Abstract: AI agents are increasingly used to automate research and development tasks, yet existing benchmarks typically evaluate them on prescribed workflows or narrow action spaces.
By Jehyeok Yeon, Ben Rank, Maksym Andriushchenko