arXiv:2607. 24063v1 Announce Type: new Abstract: On standard factuality tasks, frontier models now cluster near the top of the scale.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.
On standard factuality tasks, frontier models now cluster near the top of the scale. The question is therefore shifting from how factual a system is toward how much compute that factuality costs.
Agent evaluations increasingly benchmark LLMs, but rankings can be swayed by evaluation conditions such as scaffolds or tasks, making reliability claim‑dependent. A Bayesian variance‑decomposition framework applied to 22 benchmarks shows that reliability varies with the measurement goal: fixed model‑scaffold systems rank reliably, while underlying‑model rankings are less stable. Scaffold choice can alter conclusions, and adding more tasks only modestly improves reliability when scaffold coverage is limited; however, pooling diverse benchmarks can substantially raise cross‑task ranking reliability and reduce cost.
arXiv:2609.07785v1 Announce Type: new Abstract: An LLM-agent leaderboard invites a familiar inference: an agent ranked above another is the better agent. Public evaluation logs may not support that c...
AgentJudgeBench is a new benchmark that evaluates the reliability of large language model (LLM) judges on agentic tool‑calling tasks involving workflow directed acyclic graphs (DAGs). It contains 3,808 instances across six DAG topologies and three difficulty tiers, tested with five generators (3B–70B open‑weight models and GPT‑5.4) and six judges (20B to frontier scale) under both paired‑with‑and‑without‑ground‑truth conditions. The study finds that judge alignment degrades with task difficulty, ground‑truth exposure can sometimes hurt alignment, and structured evaluation rubrics provide modest improvements, revealing a structural ceiling that model capacity alone cannot surpass.
arXiv:2606. 08151v1 Announce Type: new Abstract: Tool-using LLM agents often fail not because relevant text is absent, but because decisive evidence is not selected, compressed, or surfaced at action time.
arXiv:2607. 01211v1 Announce Type: cross Abstract: Repository-level performance-optimization benchmarks such as GSO, SWE-Perf and SWE-fficiency evaluate coding agents by applying patches to real repositories and comparing runtime against unoptimized baselines and official reference patches.