The paper introduces HOA (Humanize Olympiad Agent), a workflow that orchestrates multiple LLM agent sessions via reviewer feedback, a structured wiki, and a verifier to tackle olympiad problems. Using GPT‑5.6 Sol, HOA achieves perfect scores on the IMO, IPhO, and IOI 2026, near‑full marks on the IChO 2026, and solves all 672 PutnamBench problems, while ranking first on the Lean‑Eval leaderboard. With open models like Kimi‑K3, HOA still attains full scores on major olympiads and a gold‑level IChO score, showing that systematic feedback can close gaps between closed‑source and open‑source agents.
By Zhengyang Zhang, Jing Xiong, Hongzhou Lin, Juihui Chung, Zijian Zhang, Song Han, Sihao Liu, Ligeng Zhu
The paper introduces SPLIT, a method that optimally allocates test‑time compute among sampling, model selection, and verification for language models and agents. Using an exact probability formula that accounts for task difficulty, shared blind spots, and verifier correlations, SPLIT is fitted via empirical Bayes on a small labeled pilot and then used to plan budgets for unseen tasks. Across seven diverse benchmarks, 26 models, and six verifier types, SPLIT achieves near‑optimal performance, outperforming rule‑based baselines by up to 2.5 percentage points and providing insights into when additional LLM judges reduce precision.
By Srinivasan Iyengar
arXiv:2607. 20531v1 Announce Type: new Abstract: Large language model (LLM) agents are increasingly deployed over Model Context Protocol (MCP) servers, yet the benchmarks used to evaluate them score the final answer or a fixed "ground-truth" list of tools, both of which are fragile once the underlying data is live and stateful.
By Jerzy Kami\'nski, Ilya Galyukshev, Artem Kuznetsov, Sergey Chuprin, Kirill Redko, Aidar Shumbalov, Anna Kalyuzhnaya
The paper investigates how test‑time computation can enhance language models and at what cost, introducing the SELF‑POT benchmark to evaluate this across competition mathematics, competitive programming, and agentic workflows. SELF‑POT separates candidate coverage from final accuracy, tracks correctness transitions under revision, and measures protocol completion alongside task success. Using a unified budget rule, the study compares direct inference, parallel sampling, and self‑revision across five low‑cost reasoning models, revealing that selection rules and failure handling significantly influence gains and cost savings.
By Bangji Yang, Jingyuan Li, Jiajun Fan, Yi Evie Zhang, Ruihan Guo, Hongba Ma, Neil He, Chumeng Liang, Qinglong Zheng, Zhanghan Ni, Ge Liu
The paper reports on training and calibration experiments for three open models on the ARC‑AGI‑2 benchmark, using offline runs on four L4 GPUs for up to 12 hours. For the NVARC Qwen3‑4B grid model, a curated corpus of 12,000 executable rules and 95,993 tasks was built, and fine‑tuning on 1,920 real task records raised the semi‑private score from 28.96 to 33.06, with no synthetic‑task checkpoint outperforming the real‑task model. For the Qwen3.8‑27B reasoning model, fine‑tuning on 1,036 of its own reasoning traces and a specialized inference procedure increased the score from 26.81 to 44.03, while the Qwen3.8‑Flash‑Next mixture‑of‑experts model achieved a best score of 56.67 after expert pruning, routing statistics adaptation, and an FP8 indexer cache.
By Nguyen Minh Thang, Trinh Thi Phuong
arXiv:2607. 07391v1 Announce Type: new Abstract: Mathematical reasoning benchmarks typically provide all facts needed to solve each problem, while interactive benchmarks often mix reasoning with tools, retrieval, and long-horizon dialogue.
By Charbel Al Bateh, Samer Saab Jr