arXiv:2607. 18260v1 Announce Type: new Abstract: We introduce FindStatBench, an execution benchmark for evaluating large language models on combinatorial code synthesis.
arXiv:2608. 20153v1 Announce Type: new Abstract: Large language models (LLMs) have shown growing potential for automated theoretical computer science (TCS) research, yet existing benchmarks remain far from realistic research settings.
The paper introduces HOA (Humanize Olympiad Agent), a workflow that orchestrates multiple LLM agent sessions via reviewer feedback, a structured wiki, and a verifier to tackle olympiad problems. Using GPT‑5.6 Sol, HOA achieves perfect scores on the IMO, IPhO, and IOI 2026, near‑full marks on the IChO 2026, and solves all 672 PutnamBench problems, while ranking first on the Lean‑Eval leaderboard. With open models like Kimi‑K3, HOA still attains full scores on major olympiads and a gold‑level IChO score, showing that systematic feedback can close gaps between closed‑source and open‑source agents.
arXiv:2610.32441v1 Announce Type: new Abstract: Large language model (LLM) agents are increasingly capable of supporting scientific workflows, but their capabilities across diverse research activitie...
arXiv:2607. 06820v1 Announce Type: new Abstract: Recent advances in AI for Mathematics have focused largely on autoformalization and theorem proving, leaving the role of Computer Algebra Systems (CAS) in agentic LLM workflows underexplored.
The paper introduces h1, a simple agentic loop that interfaces a language model with Lean 4, avoiding complex human-designed workflows. h1 achieves state‑of‑the‑art performance on benchmarks such as PutnamBench, FATE‑X, and FormalConjectures100‑Solved, solving eight open problems and matching or surpassing specialized systems like Goedel‑Architect and AlphaProof Nexus. Experiments show that an efficient Lean interface is crucial for cost and success, suggesting that future evaluations should prioritize execution feedback over elaborate workflows.
arXiv:2605. 28556v2 Announce Type: replace Abstract: As agent capabilities advance, existing benchmarks, such as $\tau^2$-Bench, are becoming increasingly saturated.
arXiv:2608. 11232v1 Announce Type: cross Abstract: Evaluating LLM coding agents in algorithmic trading is difficult because static benchmarks risk data contamination and numerical backtest outputs require ground truth from actual code execution.
arXiv:2606.15872v2 Announce Type: replace Abstract: Frontier scientific reasoning remains a major challenge for large language models (LLMs), where even the strongest commercial systems fall short of...
AgentHop is a diagnostic benchmark comprising 1,011 multiple‑choice questions that test agentic multi‑hop question answering over scientific citation graphs. It provides a controlled seven‑tool sandbox with fixed token, turn, and tool‑call limits, allowing researchers to dissect model performance across four operational axes: retrieval, synthesis, tool‑call, and resource management. Analysis of 19 models shows distinct family‑specific behaviors, such as GPT models committing early, Anthropic and GLM checkpoints verifying before committing, DeepSeek and Kimi over‑searching, and Gemini‑3 Pro maintaining balance, while within‑family differences are revealed in retrieval versus synthesis strategies.
arXiv:2605. 25246v3 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used for optimization modeling and solver-code generation, yet practical operations research and optimization problems often require a harder capability: designing scalable algorithms that exploit problem structure and outperform direct formulation-and-solve baselines.
arXiv:2609.12475v1 Announce Type: new Abstract: Comprehensive benchmark suites are essential for improving large language models (LLMs), but many widely used benchmarks are redundant, making evaluati...