Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv AI
Oct 31

OPERA: OPamp Design by an Expert-Taught Reasoning Agent, from Specifications to Layout

OPERA is an expert‑taught reasoning agent that uses a large language model to design CMOS opamps from specifications to post‑layout sign‑off with open‑source tools. The agent first learns the gm/ID sizing method, then the gLayout framework, and writes a reusable layout generator that closes the layout‑extraction loop. In the SKY130 process, OPERA designed a shared constant‑gm bias and six different opamps that meet all target specifications across 15 PVT corners, with DRC‑clean and LVS‑matched layouts.

By Vishal Saxena, Soumyajit Mandal
arXiv AI
Oct 31

Aligning Medical Vision Language Models at the Capability Boundary with Entropy Adaptive Rubric Grounding

The paper introduces MedCBR, a two‑stage alignment framework for medical vision‑language models that improves diagnostic reasoning at the capability boundary. First, it uses Difficulty Weighted Boundary Preference Optimization (DBPO) to focus learning on counterfactual boundary cases, and second, it applies Entropy Gated Grounded Policy Optimization (EGPO) to reward anatomical grounding and reasoning fidelity while mitigating rubric noise. Experiments on six medical benchmarks demonstrate consistent accuracy gains, especially on challenging differential diagnosis tasks, with the combined stages delivering super‑additive improvements.

By Hongyu Bai, Jiaqi Zeng, Chunzheng Zhu
arXiv AI
Oct 31

Mind The Gap (MTG-AD): Dataset and Fine-grained Evaluation for End-to-End Inline Audio Descriptions

The paper introduces MTG-AD, a 36‑hour dataset of professionally annotated inline audio descriptions for videos, created by blind and sighted professionals. It also presents MTG‑AD‑Eval, a reference‑free evaluation framework that measures how well generated descriptions fit within a video’s natural silences, providing fine‑grained metrics aligned with expert guidelines. Experiments show that while large multimodal language models can reduce speech overlaps, gaps remain in pacing, and users find the model‑generated descriptions highly beneficial for comprehension.

By Subhashini Venugopalan, Jenny Tan, Taylor Roper, Jimmy Tobin, Anton Kast, Alicia Martin, Sam Sepah, Amy Pavel
arXiv AI
Oct 31

Decision and System One models in rheumatology: accuracy and calibration on specialty-training examination questions and diagnostic case vignettes

The study evaluated the accuracy and calibration of several Decision and System One models on rheumatology specialty‑training exam questions and diagnostic vignettes. On the RheumaMIR exam dataset, Jev achieved 93.0% accuracy in Spanish and 91.9% in English, while GPT‑6 Sol reached 99.4%. On the RheumBench diagnostic vignettes, GPT‑6 Sol again led with 76.9% accuracy, with Jev and other System One models scoring significantly lower (50.0%–59.6%).

By Alfredo Madrid-Garc\'ia, Diego Benavent, Fabian Lechner, Phillip Kremer, Johannes Knitza, Beatriz Merino-Barbancho
arXiv AI
Oct 31

Scaling Mathematical Proof Supervision from Structured Textbooks

The paper introduces TOME-Math, a large-scale corpus that transforms 2,118 university-level mathematics textbooks into about 380,000 structured proof-supervision records, preserving chapter order, proposition-proof links, dependency references, and source provenance. Using these structural signals, the authors explore curriculum learning across multiple subjects and analyze scaling patterns with supervision volume and source breadth. Fine‑tuning on TOME‑Math improves the unweighted mean score on held‑out evaluations and public benchmarks by 16.7 percentage points over the base model, surpassing all three public instruction‑data baselines in aggregate.

By Youqing Fang, Zhouqi Hua, Yinhao Tang, Yuzhe Gu, Lingkai Kong, Bin Liu, Weiming Zhang, Kai Chen, Yanan Sun, Wenwei Zhang
arXiv AI
Oct 31

Agentic AI for Clustering and Relationship Discovery in Prediction Markets

The paper introduces a hierarchical agentic reduction (HAR) pipeline that constructs inspectable cross‑event relation graphs from prediction market contracts. By clustering markets and applying typed LLM reasoning, the method proposes signed relations with rationales between specific markets. Evaluation on eight monthly Kalshi catalogs, using 30.3 million trades, shows that the identified relations exhibit higher signed Pearson and Spearman price‑change correlations than matched within‑topic controls, especially for medium or high‑strength relations.

By Agostino Capponi, Alfio Gliozzo, Brian Zhu
arXiv AI
Oct 31

Anagent For Enhancing Scientific Table & Figure Analysis

The paper introduces Anagent, a multi‑agent framework designed to improve scientific table and figure analysis. It comprises four specialized agents—Planner, Expert, Solver, and Critic—that collaboratively decompose tasks, retrieve information, synthesize analyses, and refine outputs. The authors also present AnaBench, a large benchmark of 63,178 instances across nine domains, and demonstrate that Anagent yields significant performance gains, up to 13.43% without fine‑tuning and 42.12% with fine‑tuning, highlighting the importance of task‑oriented reasoning and context‑aware problem solving.

By Xuehang Guo, Zhiyong Lu, Tom Hope, Qingyun Wang
arXiv AI
Oct 31

Towards Batch-to-Streaming Deep Reinforcement Learning for Continuous Control

The paper introduces two new streaming deep reinforcement learning algorithms—Streaming Soft Actor-Critic (S2AC) and Streaming Deterministic Actor-Critic (SDAC)—designed to work with state‑of‑the‑art batch RL methods and suitable for on‑device fine‑tuning, such as Sim2Real transfer. These algorithms achieve performance comparable to existing streaming baselines on standard benchmarks without the need for extensive per‑environment hyperparameter tuning. The authors also examine the challenges of transitioning from batch to streaming RL, demonstrating that a naive approach can degrade pre‑trained policy performance and propose a principled solution to mitigate this issue.

By Riccardo De Monte, Matteo Cederle, Gian Antonio Susto
arXiv AI
Oct 31

Rocks, Pebbles and Sand: Modality-aware Scheduling for Multimodal Large Language Model Inference

RPS-Serve is a modality-aware scheduling framework for multimodal large language model inference that treats videos, images, and text as rocks, pebbles, and sand, respectively. It estimates GPU admission costs, classifies requests, and applies priority-based scheduling with aging to balance performance across modalities. The approach boosts GPU utilization by 65% and cuts time-to-first-token by up to 9.3× compared to existing disaggregation-based systems.

By Konstantinos Papaioannou, Thaleia Dimitra Doudali
arXiv AI
Oct 31

Adaptive Query Routing: A Tier-Based Framework for Hybrid Retrieval Across Financial, Legal, and Medical Documents

The paper evaluates three retrieval strategies—Vector RAG, Tree Reasoning, and a new Adaptive Hybrid Retrieval (AHR)—on a small corpus of financial, legal, and medical documents, using a four‑tier query complexity benchmark and LLM‑as‑judge evaluation. Tree Reasoning achieves the highest overall score (0.900), but AHR excels on cross‑reference queries (0.850) and all methods tie on multi‑section queries (0.929). Vector RAG shows the lowest latency (1.61 s) and performs best on single multi‑document queries (0.900).

By Afshan Hashmi
arXiv AI
Oct 31

Can I Take Another Dose? DoseBench: Evaluating LLM Decision-Making in OTC Dosing QA

DoseBench is a new benchmark consisting of 230 consumer‑style questions about whether a person can safely take another dose of acetaminophen or ibuprofen. The study evaluates eight large language models, finding that overall accuracy ranges from 44.8 % to 62.2 %, but models with similar accuracy can make vastly different dosing decisions, with some incorrectly approving up to 60.5 % of prohibited doses. The authors show that providing more guidance or drug facts can improve accuracy but may also cause models to overlook missing information, underscoring the difficulty of applying safety rules correctly in health QA.

By Maroof Kousar, Jinsung Han, Mohit Chandra, Munmun De Choudhury, Yibo Hu
arXiv AI
Oct 31

S$^2$-MAS: A Society-Scale Multi-Agent Simulation Framework with a 500-Million-Persona Resource

S$^2$-MAS is a multi‑agent simulation framework that uses 500 million synthetic household personas to model economic responses at society scale. It integrates trainable candidate‑scoring policies, reusable financial scenarios, and explicit state transitions, and has been trained on 2.93 million examples to achieve high accuracy on consumer financial‑choice tasks. Experiments demonstrate the framework’s ability to evaluate millions of households across different purchase‑funding conditions, revealing heterogeneous consumption and repayment patterns driven by policy decisions and accounting rules.

By Yang Li, Yike Wang, Di Wan, Chujun He, Zongsi Xu, Yitao Yu, Shuai Zhong, Minghao Pan, Quanfeng Lu, Xiyu Chen, Sheng Zhong, Deng Binjie, Yichen Guo, Zhenwei Guo, Hou Jun, Kunhan Guo, Weihao Lin, Tuo Zhou, Hai Liu, Chenwei Wu, Jianyuan Zhong, Xilin Dai, Ye Luo
arXiv AI
Oct 31

SkillCT: A Multidimensional Benchmark and Execution-Free Baseline for Agent Skill Assessment

SkillCT is a new benchmark that expands agent skill assessment beyond downstream task execution to a multidimensional evaluation across nine dimensions grouped into Static, Dynamic, and Human-Assisted assessments. It measures specification quality, behavioral robustness, semantic alignment, and responsibility boundaries without requiring repeated executions. The benchmark also introduces an execution-free baseline that predicts skill quality profiles by comparing to similar reference skills, reducing token usage and evaluation time while enabling skill refinement and repository maintenance.

By Mengyu Sun, Zunlong Zhou, Ziyuan Yang, Yi Zhang
arXiv AI
Oct 31

Rubricon: Automated Rubric Generation and Refinement for LLM Evaluation and Training

Rubricon is a pipeline that automatically generates and refines discriminative rubrics for evaluating large language models (LLMs) without requiring human preference labels or manual rubric creation. It samples multiple rubrics, selects the most discriminative ones, adds missing criteria, prunes misaligned or shallow criteria, and uses LLM self‑critique and observed pass rates to guide refinement. Across diverse LLM benchmarks, Rubricon’s rubrics achieve 9% better alignment with expert‑authored rubrics than other synthetic methods, and training a Qwen3.5‑9B model on HealthBench with Rubricon rubrics improves performance by 4.5% over the best alternative synthetic generator.

By Ulyana Tkachenko, Hui Wen Goh, Guram Gogia, Jonas Mueller
arXiv AI
Oct 31

What Does "Now" Mean to Your Computer-Use Agent? Stale Screenshots, Page Loads, and Which Check to Run Before a Click

The paper investigates how stale screenshots can cause computer‑use agents to click on unintended elements. By measuring 31 public web pages, it shows that most stale clicks occur during page loading, with a 17% risk in the first second after a page opens versus only 0.4% after full load. The authors propose that waiting for network activity to stop and performing a pixel‑comparison check before a click can mitigate this risk, and they release a benchmark generator and guidelines for developers.

By Vasundra Srinivasan
arXiv AI
Oct 31

Inner-Learning: From Hand-Coded Beliefs to Declarative Knowledge in Trick-Taking Games

The paper introduces a declarative framework for modeling belief states in trick‑taking card games, converting hand‑coded inference rules from a leading Skat player into a domain‑specific language. By generating and verifying code, the authors correct three bugs in the original engine and expand the rule set through expert strategy annotations and symmetry. Additional cardinality constraints and confidence parameters further refine the model, leading to measurable improvements in winning percentage and score during self‑play tests.

By Stefan Edelkamp
arXiv AI
Oct 31

Evaluating Causal Reasoning in LLMs: A Case Study in the Challenges of Procedural Generation

The paper investigates whether large language models (LLMs) truly perform formal causal reasoning by testing them on datasets generated by the CLadder framework. It finds that LLMs and even simple models like random forests can achieve high accuracy on CLadder data, largely due to superficial, highly predictive patterns embedded by the generator. The authors argue that strong performance on CLadder likely reflects exploitation of these patterns rather than genuine causal reasoning, and that many commercial LLMs show brittle results when critical text is removed.

By Justin Clarke, Isha Karn, Patrick Murray, KJ Hill, David Jensen
arXiv AI
Oct 31

Efficient Agentic Audio-Visual Question Answering through Toolset Pruning and Modality Consolidation

The paper presents a streamlined audio‑visual question answering agent that reduces the number of specialized perception tools by pruning and merging modality‑specific tools into omni‑modal ones. Starting from a specialized‑tool OmniAgent, the authors systematically remove redundant tools and consolidate audio and visual capabilities into just two omni‑modal components—captioning and QA. On three standard AVQA benchmarks, the compact agent matches or surpasses the performance of the original specialized‑tool agent while cutting inference cost by roughly 70%.

By Yi Wang, Tianzi Wang, Peter Bell
arXiv AI
Oct 31

Context Compaction for World Models

The paper "Context Compaction for World Models" introduces a benchmark to evaluate how well world models retrieve context when revisiting regions, and presents an agent harness that compacts context to extend retrieval beyond the model’s native window. It demonstrates that this compaction allows models to recall rooms up to four times farther than their default context size without retraining, while models using sliding‑window attention lose most of their retrieval capability during dreaming. The study compares four world‑model designs, showing that extending context reach via compaction benefits dreaming agents more than switching to sliding‑window attention.

By Hayder Tirmazi
arXiv AI
Oct 31

Reasoning or Pattern Exploitation? Mechanistic Insights from RL-Trained Language Models

The paper investigates whether reinforcement learning (RL) truly endows large language models (LLMs) with reasoning skills or merely tunes them to exploit prompt information. By training small LLMs on the finite logical domain of Tic‑Tac‑Toe and applying Group Relative Policy Optimization (GRPO), the authors show that RL improves accuracy and robustness. However, mechanistic interpretability using sparse autoencoders reveals that the models mainly rely on explicit prompt cues—such as turn order, game progression, and board occupancy—while higher‑level concepts like board symmetries and strategic forks remain weakly represented, suggesting that perceived reasoning may be an artifact of task‑specific adaptations.

By Aishik Nagar, Cheston Tan