Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv AI
Oct 7

OTel: Open Telco AI Datasets, Benchmarks, and Models

OTel is an open telecom AI resource that provides derived datasets for retrieval, reranking, instruction tuning, and safety/abstention, along with 30 full‑parameter post‑trained baselines covering 10 embedding models, 3 rerankers, and 17 language models. The project has seen significant community engagement, with over 16 million model downloads and more than 157 media mentions by May 2026. Post‑training on OTel data improves performance across all model families, achieving 93.1% NDCG@10 for embeddings, 0.947 MRR@10 for rerankers, and 87.8% correctness for language models.

By Farbod Tavakkoli, Gregory Diamos, Kenneth Church, David Kanter, Mark Austin, Imtiaz Karim, Mirza Masfiqur Rahman, Merouane Abdelkader Debbah, Zeinab Nezami, Ali Maatouk, Leandros Tassiulas, Rex Ying, Nick Sorros, Louis Powell, Nikolaos Vasiloglou, Ashish Vaswani, Somanshu Singla, Adarsh Chaluvaraju
arXiv AI
Oct 7

OOPMAS: Object-Oriented Multi-Agent Systems for Query-Level Workflow Generation

OOPMAS introduces a training‑free framework that generates both the agent set and the coordination workflow at the granularity of individual queries. Agents are defined as object‑oriented class definitions with dedicated roles, tools, and persistent state, while workflows are expressed as executable main functions over these agent objects. A dynamic skill library accumulates structured lessons from execution feedback across optimization rounds, enabling in‑context improvement without any gradient updates or fine‑tuning, and achieves 89.6% accuracy on a mixed‑task benchmark, outperforming the strongest baseline by 18.1 percentage points.

By Qi Cheng, Shengyu Chen, Wei Cheng, Yiqun Xie, Xiaowei Jia, Haoyu Wang, Haifeng Chen
arXiv AI
Oct 7

DHCG: Dynamic Construction of Hierarchical Collaboration Graphs for LLM-Based Multi-Agent Reasoning

The paper introduces DHCG, a framework that dynamically constructs hierarchical collaboration graphs for large language model–based multi‑agent systems. DHCG coordinates Planner, Worker, and Generator modules to adaptively determine the composition and scale of agents during execution, guided by feedback and action‑aware preference optimization. Experiments on code generation, mathematical reasoning, and domain‑specific tasks show DHCG surpasses static and dynamic baselines, improving performance by 2.77–8.02 points and achieving a 13.06‑point gain over single‑agent baselines.

By Jie Ren, Jiakang Yuan, Chenyu Huang, Hezeer Ma, Jiayuan Fan, Tao Chen
arXiv AI
Oct 7

Confidence Reasoning Graphs: Structured Confidence Estimation for LLM Agents

The paper introduces Confidence Reasoning Graphs (CRGs), an inference-time framework that estimates the probability an LLM agent has successfully completed its task using a single trajectory without requiring privileged model access or training data. CRGs decompose the agent’s overall claim of success into contextualized sub-claims, estimate confidence for each terminal claim based on trajectory evidence, and aggregate these into an overall confidence estimate. Experiments across multiple benchmarks, models, and agent frameworks show that CRGs produce better-calibrated confidence and more effective risk-aware decision making than existing verbalized, sampling-based, and white-box surrogate methods, while also providing transparent, auditable evidence for each estimate.

By Brendan King, Farima Fatahi Bayat, Jean-Flavien Bussotti, Pouya Pezeshkpour, Estevam Hruschka
arXiv AI
Oct 7

DAEDALUS: Bootstrapping Agent Memory from Self-Generated Tasks

DAEDALUS is a method that builds reusable memory for large‑language‑model agents by having an explorer agent generate self‑created tasks and a solver agent attempt them. When the solver fails, a heuristic is extracted and only accepted after repeated successful use, then added to a memory bank for future test‑time use. Experiments on AppWorld, τ²‑bench, and AutomationBench show that DAEDALUS raises mean success rates by up to 15.9 points and pass⁵ by up to 2.2× compared to a no‑memory baseline, while also providing a cost‑effective alternative to training‑task or oracle‑verifier approaches.

By Antoine Edy, Max Conti, Victor Xing, Marc-Antoine Allard, Nawfal Benhamdane, Gautier Viaud
arXiv AI
Oct 7

Partially Observable Zero-shot coordination by Predicting Intention of Partner

The paper introduces Predicting Intention of Partner (PIP), a method for zero‑shot coordination in partially observable settings. PIP employs a Joint‑view VAE to create a partner representation from local observations and uses Partner‑State Belief networks to infer hidden partner states from interaction history. Experiments on Burrito‑PO, Overcooked‑PO, and Melting Pot, along with human evaluation, show that PIP outperforms other methods and demonstrates effective coordination with unseen partners under occlusion.

By Jinnyeong Yang, Yuhwan Jeong, Hoyong Kwon, Minseok Kim, Jihun Kim, Kuk-Jin Yoon
arXiv AI
Oct 7

Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?

The paper introduces the concept of "bottling"—the ability of large language model (LLM) agents to transform general capabilities into task‑specific, cost‑effective solutions for large, repetitive workloads. It presents BOTTLED, a benchmark where agents receive an unlabelled workload and must complete it within fixed time, compute, and API budgets, choosing strategies such as training small models or writing reusable programs. Experiments across ten models and three tasks show that strong zero‑shot performance does not guarantee effective bottling, yet bottling can still achieve substantial cost savings and competitive performance compared to specialized cheap inference models.

By Ankit Sonthalia, Haritz Puerto, Alexander Rubinstein, Martin Gubri, Seong Joon Oh
arXiv AI
Oct 7

Neutrosophic Ensemble Classification for Uncertainty-Aware Bearing Fault Detection: Evidence from Laboratory and Variable-Speed Industrial Benchmarks

The paper presents a neutrosophic decomposition of a Random Forest + XGBoost + Logistic Regression ensemble for bearing fault detection, yielding four uncertainty indicators: top‑class evidence (T‑hat), best‑competitor evidence (F‑hat), predictive entropy (I1‑hat), and decision disagreement (I2‑hat). Evaluated on CWRU and JNU benchmarks under a leave‑one‑condition‑out protocol, the ensemble achieves perfect accuracy on most CWRU loads but performs poorly on JNU, where Logistic Regression outperforms the tree ensembles. Additional experiments show that fusing time‑ and frequency‑domain models and using Jensen‑Shannon divergence improves uncertainty estimation, and that envelope‑spectrum features can separate fault classes with 99.57 % accuracy on CWRU. "whyItMatters":"The study demonstrates that refined neutrosophic uncertainty indicators can reveal limitations of conventional confidence scores and guide more reliable fault detection in variable‑speed industrial settings."

By Maikel Leyva-Vazquez, Dayron Rumbaut Rangel, Lorenzo Cevallos-Torres, Alexis Matheu Perez