arXiv AI
Qiushi Engine on AstaBench E2E-Bench-Hard
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Oct 2
Mingbird: A Local-First Agent Harness Enabling Small Open Models to Complete Real Tasks
Mingbird is a local‑first agent harness designed for small open‑weight language models (2–9 B) that run on ordinary laptops. It introduces ten mechanisms—such as a byte‑level net‑zero prefill budget, a finish gate that re‑reads the task before accepting completion, and signature‑level loop detection—to address common failure modes that arise from the harness rather than the model itself. In controlled experiments on the LRAB benchmark and the $ au^2$‑bench, Mingbird achieves higher overall scores (0.886 and 0.856 respectively) compared to other harnesses, and its ablation studies show that each mechanism contributes measurable performance gains.
arXiv AI
Sep 24
WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents
arXiv:2609.27490v1 Announce Type: new Abstract: AI research agents need reliable knowledge of how their experiments change outcomes. We introduce WhatWorkedBench to measure experimental understanding...
arXiv AI
4d ago
Rules, Traces and Routing Statistics: Training and Calibration Data for Open Models on ARC-AGI-2
The paper reports on training and calibration experiments for three open models on the ARC‑AGI‑2 benchmark, using offline runs on four L4 GPUs for up to 12 hours. For the NVARC Qwen3‑4B grid model, a curated corpus of 12,000 executable rules and 95,993 tasks was built, and fine‑tuning on 1,920 real task records raised the semi‑private score from 28.96 to 33.06, with no synthetic‑task checkpoint outperforming the real‑task model. For the Qwen3.8‑27B reasoning model, fine‑tuning on 1,036 of its own reasoning traces and a specialized inference procedure increased the score from 26.81 to 44.03, while the Qwen3.8‑Flash‑Next mixture‑of‑experts model achieved a best score of 56.67 after expert pruning, routing statistics adaptation, and an FP8 indexer cache.
arXiv AI
Oct 21
EGI-BENCH: An Enterprise General Intelligence Benchmark Suite
EGI‑BENCH is a unified benchmark suite for evaluating large language model agents in enterprise settings, consolidating eight existing benchmarks (CRMARENA, HERB, PERSONABENCH, USERBENCH, MCP‑UNIVERSE, MCPEVAL, BENCHFORCE, LOCOBENCH) under a single command‑line interface with two runtime‑budgeted modes. It introduces the Enterprise General Intelligence (EGI) Score, a six‑dimensional framework (M1‑M6) that aggregates fine‑grained sub‑metrics from each benchmark rather than simple averages, providing a more nuanced view of model capabilities. The paper reports results for fourteen LLMs across GPT, Claude, and Gemini families, showing varied strengths across task outcome, planning, retrieval, tool actioning, safety, and engineering quality.
arXiv AI
Oct 16
ToolBottleneckBench: Measuring Tool-Agent Performance under Controlled Pressures and Fixed Harness Budgets
ToolBottleneckBench is a benchmark that evaluates tool‑agent performance under controlled pressures and fixed harness budgets. It keeps synthetic enterprise tasks constant while varying one pressure—such as the registered tool catalog, visible tool menu, certified workflow, or record encoding—to measure how each affects success. Across 21 model and harness configurations, the visible menu pressure uniquely imposes a large penalty on the model, largely due to truncation at the completion cap, and most failures are driven by harness limits rather than tool‑specific issues.
arXiv AI
Aug 24
FlavourBench: Ranking Frontier Language Models with Executable Culinary Ground Truth
FlavourBench is an automated benchmark that evaluates language models on culinary tasks by providing a versioned, executable ground truth system. Each task presents eight ingredients and asks the model to propose a three‑ingredient portfolio, with all 56 possible portfolios scored by the Epicure system before model execution. The benchmark assesses 27 frontier endpoints across 534 tasks, yielding a FlavourBench Score that averages frozen task scores across families, and includes extensive statistical analysis and reproducible data releases.