Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv Computer Vision
Sep 18

CoRef-GS: Cooperative Referring Gaussian Splatting for Multi-Agent Scene Understanding

CoRef-GS introduces a cooperative referring Gaussian splatting framework for multi‑agent scene understanding, enabling robots to ground object‑ and relation‑centric language queries across independently reconstructed maps. The method builds local open‑vocabulary instance‑aware Gaussian maps, aligns them using a cross‑agent module that enforces geometric and semantic consistency, and grounds queries with a view‑conditioned mask relation graph. Experiments on a new dual‑quadruped benchmark show significant improvements, reducing rotation error from 2.58° to 0.15° and raising real‑world referring mIoU from 52.6% to 68.8%.

By Zhikun Zhou, Kunyu Peng, Runyi Yang, Junhao Cai, Di Wen, Ruiping Liu, Danda Pani Paudel, Yi Zhou, Luc Van Gool, Kailun Yang
arXiv Computer Vision
Sep 18

Semi-LAR: Semi-supervised Contrastive Learning with Linear Attention for Removal of Nighttime Flares

The paper introduces Semi-LAR, a semi‑supervised framework for removing nighttime lens flares. It uses an adaptive pseudo‑label repository that refines supervision through quality assessment, momentum updates, and invalid label filtering. A flare‑aware contrastive loss treats flare‑contaminated inputs as negatives, encouraging representations that distinguish flare patterns while aligning with reliable pseudo targets.

By Xiyu Zhu, Wei Wang, Kui Jiang, Zhengguo Li
arXiv Computer Vision
Sep 18

Performance of Machine Learning Classification in Sonomammogram Images using BI-RADS

This study evaluates the classification accuracy of six modern deep‑learning architectures—VGG19, ResNet50, GoogleNet, ConvNeXt, EfficientNet, and Vision Transformers—on breast ultrasound images categorized by BI‑RADS. Using 2,945 training images and 936 validation images from 1,540 patients, the models were tested in full fine‑tuning, linear evaluation, and training‑from‑scratch settings. The best performance was achieved with full fine‑tuning, yielding 76.39 % accuracy and a 67.94 % F1 score.

By Malitha Gunawardhana, Norbert Zolek
arXiv Machine Learning
Sep 18

Radio-Frequency Convolutional Neural Networks

The paper introduces Radio‑Frequency Convolutional Neural Networks (RF‑CNNs), which repurpose the frequency mixer in wireless radios to perform convolutional neural network inference directly on edge devices. By mapping multi‑channel convolutions onto frequency tones, the passive mixer can execute the entire operation in a single pass, enabling deep CNNs with up to 26.4 million parameters and nine layers to run on smartphones, wearables, and drones. Experimental results show near full‑precision performance while reducing energy consumption to 0.72 fJ per multiply‑accumulate—two orders of magnitude lower than adding a digital processor. "whyItMatters":"The approach leverages existing radio hardware to deliver efficient, state‑of‑the‑art AI inference on billions of devices without increasing size, weight, power, or cost."

By Zhihui Gao, Shi-Yuan Ma, Yiran Chen, Dirk Englund, Tingjun Chen
arXiv Machine Learning
Sep 18

Explaining spatial information flow in short-term traffic forecasting models using a gated graph attention network

The paper introduces a gated graph attention network (GAT) to better explain spatial information flow in short‑term traffic forecasting models. By adding a gate that learns how much of a sensor’s updated state comes from its neighbors versus itself, the authors can progressively withdraw neighbor information and observe its impact on accuracy. Applied to the ST‑MetaNet architecture on data from 498 loop detectors in England, the gated GAT shows that neighbor influence is higher for busy sensors, improves accuracy slightly with mild regularisation, and reveals that the two GAT layers are largely redundant.

By Yue Li, Shujuan Chen, Ying Jin
arXiv Machine Learning
Sep 18

COMPASS: Ordered Clustered Routing at 100K Scale

The paper introduces COMPASS, an algorithm for the Ordered Clustered Traveling Salesman Problem (OCTSP) that combines search with learning-accelerated routing through parallel sub-solvers. COMPASS improves solutions continuously with more compute, exploits clustered structure to achieve exact solutions in time exponential in cluster size, and outperforms existing methods. It works with general distance matrices, not just coordinate inputs, and scales to 100,000 synthetic nodes and 28,500 real e-commerce nodes, representing the largest reported routing solution over asymmetric distances.

By Ido Greenberg, Hugo Linsenmaier, Piotr Sielski, Shie Mannor, Alex Fender, Gal Chechik, Eli Meirom
arXiv Computation and Language
Sep 18

V\={a}kQA: A Benchmark and Evaluation Study for Telugu Spoken Factoid Question Answering

VákQA is a new Telugu spoken factoid question answering benchmark comprising 2,001 question‑answer pairs across six domains, 2.53 hours of speech audio, bilingual transcriptions, and human‑verified reference answers. The study validates automatic evaluation methods against human judgments, finding that Gemini-as-a-judge best approximates human ratings but is unevenly strict, while open‑weight judges tend to penalize correct Telugu answers that differ in surface form. Using this validated setup, the authors benchmark proprietary and open‑weight models, highlighting challenges such as cultural specificity lost in translation, phonetic confusions from speech input, and cascading ASR‑MT errors.

By Bhavana Akkiraju, Ravi Sastry Kolluru, Sri Charan D, Srihari Bandarupalli, Santosh Kesiraju, Anil Vuppala
arXiv AI
Sep 18

A Unified Evaluation Framework for Trustworthy Large Language Models, Agentic AI, and Multimodal Systems

The paper introduces a unified evaluation framework for assessing the trustworthiness of large language models, agentic AI, and multimodal systems. It connects output-level, trajectory-level, and cross-modal assessments across eight dimensions—capability, robustness, safety, fairness, transparency, governance, oversight, and efficiency—while preserving system-specific metrics and providing uncertainty estimates. A meta-evaluation layer checks the validity, reliability, and reproducibility of the evaluation itself, and the framework aligns with governance standards and regulatory requirements.

By Shaina Raza, Ahmed Y. Radwan, Imran Liaquat, Kathryn Hume
arXiv AI
Sep 18

Rethinking Multi-Agent Collaboration: When More Is Less

The paper examines when multi‑agent collaboration is beneficial versus single‑agent approaches. It finds that collaboration yields systematic advantages mainly in long‑horizon tasks with sparse dependencies, while single agents perform better in tightly coupled, sequential workflows. The authors introduce SAIGE, a lightweight multi‑agent mechanism that models collaboration as a dynamically evolving graph, and show that it balances context efficiency and task performance without always improving outcomes as more agents are added.

By Yishuo Yuan, Yibo Wu, Yihan Zhang, Minyuan Sun, Shenliang Li, Xinkai Ma, Yifan Li, Jiaheng Liu
arXiv AI
Sep 18

Customizable and Jointly Optimized Route Planning: A Deep Architecture Enabling Differentiable Shortest-Path Search

The paper introduces a deep architecture that jointly optimizes cost functions and a route-ranking model to accommodate diverse user preferences in route planning. It first generates a complete set of Pareto‑optimal routes using a multi‑objective Dijkstra algorithm, then employs a neural network that emulates shortest‑path search and ranking in an end‑to‑end differentiable framework. A novel loss function treats route preference as a constrained optimization problem, allowing a single objective to be optimized while other attributes remain constrained, and experiments on real‑world data show significant improvements over existing methods.

By Rui Zhao, Chao Chen, Longfei Xu, Chenguang Ji, Hengbin Cui, Kaikui Liu, Xiaolong Li
arXiv AI
Sep 18

Diagnose, Recover, Certify: Task Readiness under Hidden Dynamics Changes

The paper introduces a framework for diagnosing and recovering from hidden dynamics changes in deployed control policies, focusing on the problem of task readiness under dormant dynamics drift. It proposes an intervention-based Bayesian method called Evidence‑Gated Matched‑Pulse Transport that localizes faults and estimates actuator effectiveness, enabling agents to certify readiness for future tasks with limited, task‑agnostic interactions. The approach is evaluated on diverse benchmarks, measuring readiness coverage, selective risk, interaction cost, and return, and identifies regimes where transported evidence is decisive.

By Nguyen Viet Tuan Kiet, Huynh Thi Thanh Binh
arXiv AI
Sep 18

UnifiedPlayers: Enhance Tool-Integrated Reasoning in Agentic Reinforcement Learning

UnifiedPlayers is a cooperative framework that jointly adapts planning, execution, and evaluation for tool-integrated reinforcement learning agents. It consists of a Planning Player that generates tasks, an Execution Player that creates multi-turn trajectories with Python tool calls, and an Evaluation Player that builds executable verifiers, all coordinated by role‑specific rewards under GRPO. The approach outperforms prior baselines on mathematical and general reasoning benchmarks and yields a verifier with high adversarial detection accuracy and more discriminative reward signals.

By Wenjie Liao, Liangjie Zhao, Zehong Cao
arXiv AI
Sep 18

Robust Conformal Intrusion Detection via Traffic-Aware Calibration and Attack-Orbit Invariance

The paper addresses the lack of statistical validity in language‑model‑based network intrusion detection. It introduces traffic‑aware conformal prediction, which calibrates on attacker‑expected traffic to restore coverage guarantees, and further mitigates adaptive attacks by removing attacker‑controllable features, achieving exact pathwise coverage. Experiments on three benchmarks show that this approach maintains coverage while incurring a modest accuracy cost.

By Zhenpeng Li
arXiv AI
Sep 18

A Multi-Modal Generative Model for Tomato Disease Leaves Understanding

The paper introduces SOLAR, a multimodal generative model that jointly interprets visual and textual data to understand tomato leaf diseases across six question‑answering tasks. SOLAR aligns visual features with task‑aware language representations using a Fusion Expert module based on a mixture‑of‑experts, enabling it to generate contextually relevant answers for diverse diagnostic tasks. Evaluated on 41,677 images and 216,209 QA pairs, SOLAR outperforms state‑of‑the‑art vision‑only, vision‑language, and task‑specific models in both closed and open‑ended settings, demonstrating superior accuracy, robustness, and multimodal reasoning.

By Khang Nguyen Quoc, Minh-Phuoc Tran, Gia-Han Truong, Luyl-Da Quach
arXiv AI
Sep 18

Chronicle: Cut-Point Replay for Regression Testing of LLM Agents

Chronicle introduces a method called cut‑point replay to make regression testing of large language model (LLM) agents reproducible. It records an agent’s run at non‑deterministic boundaries as immutable envelopes and then replays selected boundaries while executing the rest live, enabling continuous‑integration tests that detect faulty code changes. Benchmarks show minimal overhead, perfect bit‑stability, and effective detection of unsafe actions in a mutation study.

By Tisha Chawla, Susheem Koul
arXiv AI
Sep 18

oMeBench: Towards Robust Benchmarking of LLMs in Organic Mechanism Elucidation and Reasoning

oMeBench is a large-scale, expert-curated benchmark designed to evaluate large language models (LLMs) on organic mechanism reasoning. It contains over 10,000 annotated mechanistic steps, including reaction type labels, intermediate structures, and difficulty ratings, and introduces the oMeS scoring framework to assess logical consistency and chemical structural similarity. Evaluation shows that while current LLMs display promising chemical intuition, they often fail to produce correct and consistent multi-step reasoning, though prompting and fine-tuning can bring smaller models up to the level of closed‑source frontier models.

By Ruiling Xu, Yifan Zhang
arXiv AI
Sep 18

A Neuropsychologically Grounded Evaluation of LLM Cognitive Abilities

The paper introduces NeuroCognition, a benchmark based on three neuropsychological tests—Raven's Progressive Matrices, Spatial Working Memory, and the Wisconsin Card Sorting Test—to evaluate foundational cognitive abilities in large language models (LLMs). It finds that while LLMs excel on text tasks, their performance drops on image-based and more complex tasks, and they fail different parts of the same tasks compared to humans. NeuroCognition correlates with standard general-capability benchmarks yet measures distinct cognitive skills, highlighting where LLMs align with or diverge from human-like intelligence.

By Faiz Ghifari Haznitrama, Faeyza Rishad Ardi, Alice Oh
arXiv AI
Sep 18

FOCAL: Fine-Grained Optimal-Transport-Driven Contrastive Alignment of Language and ECGs with Waveform Enhancement

FOCAL is a framework that aligns fine-grained ECG waveform segments with specific report tags using Optimal Transport, addressing the lack of localized representation in prior methods. It introduces a semantic similarity matrix to mitigate false negatives when reports share diagnoses, and a coarse‑to‑fine enrichment pipeline that employs Large Language Models to recover missing waveform semantics while filtering hallucinations. Experiments on six datasets show FOCAL achieves state‑of‑the‑art zero‑shot prediction and linear probing performance.

By Haitao Li, Che Liu, Zhengyao Ding, Ziyi Liu, Wenqi Shao, Zhengxing Huang
arXiv Machine Learning
Sep 18

FCx: An algorithm for finding Feasible Counterfactual Explanations

FCx is a new algorithm that generates counterfactual explanations while explicitly enforcing feasibility constraints. It uses a modified Variational Autoencoder with a multi‑factor loss to produce realistic, low‑cost counterfactuals that satisfy both hard constraints supplied by users and soft constraints inferred via causal inference. Experiments on four public datasets demonstrate that FCx matches state‑of‑the‑art performance across multiple metrics while guaranteeing feasibility.

By Kleopatra Markou, Vana Kalogeraki, Dimitrios Gunopulos