FlavourBench is an automated benchmark that evaluates language models on culinary tasks by providing a versioned, executable ground truth system. Each task presents eight ingredients and asks the model to propose a three‑ingredient portfolio, with all 56 possible portfolios scored by the Epicure system before model execution. The benchmark assesses 27 frontier endpoints across 534 tasks, yielding a FlavourBench Score that averages frozen task scores across families, and includes extensive statistical analysis and reproducible data releases.
By Josef Chen, Erim Hayretci
UpgradeBench is a decision‑centric longitudinal benchmark that evaluates how fine‑tuned language‑model specialists should be handled when new base‑model releases occur. It covers four consecutive Qwen releases, a continuation checkpoint, six tasks, two model sizes, and OLMo checkpoints with known training lineage, and examines whether retraining, adapter transfer, or other recovery strategies improve specialist performance. The benchmark reveals that upgrade gains vary by task and release interval, that direct adapter copying is sensitive to pretraining distance, and that teacher relabeling can recover specialists without new annotations.
"whyItMatters":"The study provides actionable insights into the cost‑effective management of specialist models across model releases, showing how to balance retraining effort with performance gains."
By Ye Chen, Weining Zhang
arXiv:2608. 05466v1 Announce Type: new Abstract: High-quality long-horizon training data for terminal agents is expensive to produce, often costing hundreds to thousands of dollars per task, because each task must keep the instruction, environment, reference solution, and verifier mutually consistent.
By Zhongzhi Li, Yucheng Shi, Zongxia Li, Ruhan Wang, Anhao Li, Zixun Huang, Junyao Yang, Lei Ke, Ninghao Liu, Haitao Mi, Leowei Liang
arXiv:2606. 01080v1 Announce Type: cross Abstract: Large language models often improve on difficult tasks by spending inference-time compute on a reasoning trace before producing the final answer.
By Dhruv Saini, Rohan Pandey
The paper evaluates a manager‑worker scaffold that uses a shared filesystem workspace to orchestrate multi‑agent large language model (LLM) coding tasks without training or tuning. Across nine models—including five open‑weight and four closed‑weight systems—the scaffold yields statistically significant accuracy gains for some models (e.g., Qwen3.8‑27B, GPT‑5.6‑Luna, GPT‑5.6‑Terra, Kimi‑K3, Minimax‑M3) while producing null or negative effects for others (e.g., Qwen3.6‑35B). The study shows that the manager can triple token usage but still achieves higher accuracy at a fraction of the cost compared to larger single‑pass models, with key mechanisms identified as context management and problem decomposition.
By Victor Gao (Sang Won), Vida Khosrowshahi (Sang Won), Ali Khosrowshahi (Sang Won), Xihao Sun (Sang Won), Juhyun Lee (Sang Won), Simon (Sang Won), Lee
The paper introduces SMITH, a reinforcement learning framework that jointly trains large language models to create and use tools within a single policy. By alternating between build and use tasks and employing separate reward signals for schema, code, and outcome failures, SMITH enables a 4B Qwen3 model to achieve state‑of‑the‑art accuracy on procedural reasoning benchmarks, outperforming larger untrained models and improving performance on downstream tasks when its tools are applied.
By Zhi Rui Tam, Chieh-Yen Lin, Yun-Nung Chen, Shao-Hua Sun, Hung-yi Lee