Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,620 stories · RSS feed

arXiv AI
Sep 10

Programmable Cellular Automata

arXiv:2609.06102v2 Announce Type: cross Abstract: Cellular automata is a local computation paradigm where complex behavior can arise from local interactions between simple functions. This paradigm ha...

By Ahmed Khalifa, Muhammad Umair Nasir, Matthew Siper, Steve James, Julian Togelius
arXiv Computer Vision
Sep 10

Integrating Unimodal and Vision-Language Representations in Latent Space for Multi-Label Chest X-Ray Classification

arXiv:2609.09185v1 Announce Type: new Abstract: Multi-label chest X-ray classification has attracted considerable attention in recent years, with the effective use of visual representations and clini...

By Quang-Huy Tran, Duc-Tuan Ngo, Minh-Khoi Nguyen-Bui, Dang-Khoa Bui, Thanh-Trong Tran, Tuan-Khoi Nguyen, Hoang-Anh Ngo
arXiv AI
Sep 10

PAC-CF: Calibrating Irreversible Frontier Pruning in LLM-Guided Search

PAC-CF introduces a method called Probably Approximately Correct Conformal Filtering to calibrate irreversible frontier pruning in large language model (LLM)-guided search. The approach treats tree pruning as a PAC‑guaranteed decision problem, deriving a conformal margin from score deficits on held‑out Native traces and applying a direct score‑gap filtering rule during deployment. Experiments across multiple domains and controllers show that PAC‑CF improves utility at various budgets and reduces workload metrics, notably boosting equal‑domain utility by 4.38 points and cutting physical requests by 18.95% with a 23.76% token reduction on a pruning‑aware ToolTree under a 100‑request budget.

By Tianhao Qian, Jiayu Chen, Lixu Wang
arXiv AI
Sep 10

Kalman Delta Networks: Uncertainty-aware Associative Memory

Kalman Delta Networks (KDNs) extend linear attention models by treating associative memory as a linear–Gaussian state‑space system, enabling the Kalman filter to optimally estimate both memory state and its uncertainty. Two GPU‑friendly approximations—Diagonal KDN and Isotropic KDN—use mean‑field variational inference or a single scalar uncertainty per head, respectively, to maintain tractable uncertainty recurrences during linear‑attention scans. Experiments on 750 M and 1.3 B‑parameter models show that KDN variants consistently lower perplexity and raise downstream accuracy compared to existing linear‑attention baselines.

By Ngoc Bui, Tinglin Huang, Rex Ying
arXiv AI
Sep 10

Learning to Configure Agentic AI Systems

The paper introduces ARC, a lightweight hierarchical policy that learns to configure LLM‑based agent systems on a per‑query basis by treating each configuration as a temporally extended option in a semi‑Markov decision process. Unlike fixed templates or hand‑tuned heuristics, ARC dynamically selects workflows, tools, token budgets, and prompts tailored to the difficulty of each query. Experiments on reasoning, tool‑use, and agentic benchmarks show that ARC outperforms budget‑matched tool‑augmented LLMs, boosting reasoning accuracy by 31.3%, tool‑use accuracy by 13.95%, and doubling success on the τ‑Bench Airline Pass task from 9.0% to 18.0%.

By Aditya Taparia, Som Sagar, Ransalu Senanayake
arXiv Computation and Language
Sep 10

UnitBoost: Managing Compound LLM Systems with a Merge Operator, Not a Model

UnitBoost proposes a non‑generative merge operator to manage compound LLM systems, replacing the opaque higher‑level LLM that traditionally coordinates worker outputs. The operator maps worker outputs to slot‑value proposals, uses a constrained argmax to assemble the final answer, and explicitly tracks unfilled slots as residuals for subsequent rounds, thereby achieving order‑free processing and unit provenance. Across three held‑out benchmarks, UnitBoost outperforms both gold‑label‑selected candidates and input‑matched generative managers, improving compound‑system performance by up to 0.182 points and raising FanOutQA cell F1 from 0.4778 to 0.5524.

By Xing Zhang, Guanghui Wang, Yanwei Cui, Mengdie Flora Wang, Peiyang He