arXiv AI
TCSExam: Evaluating Frontier LLMs on the Full Range of TCS Research
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jul 22
FindStatBench: Evaluating Large Language Models on Combinatorial Code Synthesis
arXiv:2607. 18260v1 Announce Type: new Abstract: We introduce FindStatBench, an execution benchmark for evaluating large language models on combinatorial code synthesis.
arXiv Computation and Language
Aug 21
FormalTCS: Benchmarking End-to-End Frontier Formal Theoretical Computer Science Research of Large Language Models
arXiv:2608. 20153v1 Announce Type: new Abstract: Large language models (LLMs) have shown growing potential for automated theoretical computer science (TCS) research, yet existing benchmarks remain far from realistic research settings.
arXiv AI
Oct 12
Humanize Olympiad Agents: A Multi-Session Workflow for Olympiad Problem Solving with Solve, Review, Wiki, Verify
The paper introduces HOA (Humanize Olympiad Agent), a workflow that orchestrates multiple LLM agent sessions via reviewer feedback, a structured wiki, and a verifier to tackle olympiad problems. Using GPT‑5.6 Sol, HOA achieves perfect scores on the IMO, IPhO, and IOI 2026, near‑full marks on the IChO 2026, and solves all 672 PutnamBench problems, while ranking first on the Lean‑Eval leaderboard. With open models like Kimi‑K3, HOA still attains full scores on major olympiads and a gold‑level IChO score, showing that systematic feedback can close gaps between closed‑source and open‑source agents.
arXiv Computation and Language
Oct 26
SCI-AGENT-EXAM: Benchmarking Agents on Scientific Research Tasks
arXiv:2610.32441v1 Announce Type: new Abstract: Large language model (LLM) agents are increasingly capable of supporting scientific workflows, but their capabilities across diverse research activitie...
arXiv AI
Jul 9
Evaluating SageMath-Augmented LLM Agents for Computational and Experimental Mathematics
arXiv:2607. 06820v1 Announce Type: new Abstract: Recent advances in AI for Mathematics have focused largely on autoformalization and theorem proving, leaving the role of Computer Algebra Systems (CAS) in agentic LLM workflows underexplored.
arXiv Machine Learning
5d ago
h1: A Taste of the Bitter Lesson in Automated Theorem Proving
The paper introduces h1, a simple agentic loop that interfaces a language model with Lean 4, avoiding complex human-designed workflows. h1 achieves state‑of‑the‑art performance on benchmarks such as PutnamBench, FATE‑X, and FormalConjectures100‑Solved, solving eight open problems and matching or surpassing specialized systems like Goedel‑Architect and AlphaProof Nexus. Experiments show that an efficient Lean interface is crucial for cost and success, suggesting that future evaluations should prioritize execution feedback over elaborate workflows.