Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv Computer Vision
Sep 15

Stereo4DWalker: Learning 4D-aware Embodied Urban Navigation from Internet Stereo Videos

Stereo4DWalker is a 4D-aware embodied navigation model that uses stereo video inputs to explicitly construct structured representations of geometry and motion. These 4D structures are incorporated into a navigation transformer via 4D-conditioned attention layers, enabling the agent to learn robust urban navigation. The authors also curate a large-scale stereo navigation dataset with automatically annotated actions from Internet stereo videos, and demonstrate that Stereo4DWalker outperforms state‑of‑the‑art methods while requiring only 1.5% of the training data.

By Wentao Zhou, Xuweiyi Chen, Vignesh Rajagopal, Jeffrey Chen, Rohan Chandra, Zezhou Cheng
arXiv Computer Vision
Sep 15

Anchor3R: Streaming 3D Reconstruction with Transient Anchors for Long-Horizon Visual Mapping

Anchor3R is a streaming 3D reconstruction framework that predicts window-relative poses and local geometry in the current‑frame coordinate system, forming a dense relative‑pose graph for online pose updates and loop‑aware motion averaging. It improves long‑horizon pose accuracy and dense reconstruction quality on indoor, outdoor, driving, and RGB‑D benchmarks, and generalizes from 48‑frame training sequences to streams exceeding 10,000 frames while keeping GPU memory bounded. The method addresses issues of train‑test mismatch, early‑anchor bias, and accumulated drift found in previous streaming models.

By Peilin Tao, Chong Cheng, Yuansen Du, Caiwei Song, Zhengqing Chen, Xiaoyang Guo, Wei Yin, Weiqiang Ren, Qian Zhang, Hainan Cui, Shuhan Shen
arXiv AI
Sep 15

Energy Accuracy Is Not Enough: A Structure-Aware Benchmark and Evaluation Protocol for Quantum Architecture Search

The paper introduces HamQASBench, a structure‑aware benchmark for quantum architecture search that includes eleven molecular Hamiltonians up to fourteen qubits, exact references, and a comprehensive evaluation protocol. The protocol combines energy accuracy, success rates, reference‑relative circuit cost, local entropy profiles for non‑degenerate targets, and state identification within degenerate ground subspaces. Experiments with five methods across four search paradigms show that energy‑only comparisons miss important differences, such as varying gate counts, distinct spin components in degenerate cases, and differing entanglement profiles, while still achieving chemical accuracy on a near‑product instance.

By Jiayang Niu, Akib Karim, Yan Wang, Jie Li, Ke Deng, Azadeh Alavi, Muhammad Usman, Yongli Ren
arXiv Machine Learning
Sep 15

Assembling the CREW: A Collaborative Multi-agent Reinforcement Learning Framework for Automated Related Work Generation

The paper introduces CREW, a collaborative multi‑agent reinforcement learning framework that automates the generation of the Related Work Section in research papers. Unlike previous methods that follow a fixed workflow, CREW allows large language model agents to dynamically select actions—Retrieve, Disseminate, Compose, and Critique—guided by a policy trained with Independent Proximal Policy Optimization. Experiments on a standard benchmark show that CREW improves output quality and reduces token usage compared to strong baselines.

By Hai-Dang Dang, Bao-Yen Pham, Bao Nguyen, Tran Thi Huong, Huynh Thi Thanh Binh
arXiv Machine Learning
Sep 15

A Lifecycle Cost Analysis of Smart-Contract-Coordinated Federated Learning Marketplaces

The paper experimentally evaluates the operational cost of a DAO‑governed blockchain‑enabled federated learning marketplace. It decomposes gas consumption across the entire contract lifecycle, isolates the impact of on‑chain coordination and IPFS storage, and derives an analytical model for cost amortization. Results show a single training task consumes about 3.8 million gas units per trainer, with the average cost per training round reaching its amortization knee after roughly 20 communication rounds, while model performance remains comparable to conventional FL deployments.

By Luan Mantegazine, Luiza Leidemer, Claudio Geyer
arXiv AI
Sep 15

Domain Generalization for Smartphone-Based Human Activity Recognition: A Systematic Analysis of Components and Interactions

The paper presents a comprehensive benchmark for Domain Generalization (DG) in smartphone-based Human Activity Recognition (HAR), running over 410,000 experiments across multiple architectures, training objectives, initialization strategies, and architectural tweaks. It finds that individual DG components offer limited, highly conditional improvements, while combined configurations often yield stronger, sometimes super‑additive gains that depend on the model and shift scenario. The study also highlights that current source‑validation selection captures only a fraction of the potential oracle performance, underscoring the need for joint DG design and robust model‑selection methods.

By Ot\'avio Oliveira Napoli, Edson Borin
arXiv AI
Sep 15

Hallucination in Multimodal Foundation Models: A Survey on Causes, Corrections, and Evaluations

The article surveys hallucination issues in Large Vision‑Language Models (LVLMs), a type of multimodal foundation model that blends visual data with large language models. It categorizes hallucination causes into model architecture and data quality, presents a taxonomy of mitigation strategies, and critically evaluates existing evaluation benchmarks from both discriminative and generative viewpoints. The survey also outlines open challenges and future research directions to improve LVLM reliability and trustworthiness.

By Yinghao Guo, Wei Lan, Wenyi Chen, Qingfeng Chen, Shichao Zhang, Shirui Pan, Huiyu Zhou, Yi Pan
arXiv AI
Sep 15

Look Before You Leap: Factual Decoding with Internal Attribution Signals

The paper introduces DescaPE, a decoding framework that uses internal model signals to reduce hallucinations in large language models. By identifying a factual‑salient layer span and training a lightweight probe to approximate its signal, DescaPE penalizes high‑risk continuations and rewards factually grounded ones during inference. Experiments on five factuality benchmarks across three LLMs show that DescaPE improves factuality with only a 1.10× latency overhead.

By Hayeong Ryu, JungMin Yun, Byeonggeuk Lim, Sunhee Jo, YoungBin Kim
arXiv AI
Sep 15

Carbon-Aware Routing for Function Calling in Edge-Cloud LLM Systems

The paper presents a carbon‑aware routing framework for function‑calling in large language models that distributes queries across a three‑tier edge‑cloud architecture. A lightweight k‑NN predictor estimates accuracy, delay, and power for each edge tier, and real‑time grid carbon intensity is used to route queries to the lowest‑emission tier that can execute them. Experiments on state‑of‑the‑art benchmarks show the framework matches cloud‑level accuracy while cutting operational carbon emissions by an average of four times.

By Aikaterini Maria Panteleaki, Varatheepan Paramanayakam, Spyros Tragoudas, Iraklis Anagnostopoulos
arXiv AI
Sep 15

Enhancing Event Candidate Acquisition for Event Linking

The paper introduces MACE, a Multi-Agent Candidate Event acquisition method designed to improve event linking by refining event structure before the linking step. MACE employs evidence-specialized large language model agents to gather time, location, participant, and event-type evidence, exposes intermediate queries to candidate-event lookup tools, and allows a coordinator to revise the evidence set before final candidate construction. Experiments on two event linking benchmarks demonstrate that integrating MACE consistently boosts accuracy across different event linking models without altering the underlying models.

By Ziyang Zhang, Yinan Liu, Boyi Xue, Yingxuan Huang, Bin Wang, Xiaochun Yang
arXiv AI
Sep 15

Surprising Effectiveness of Self-Demonstrations in Enhancing Schema-Ontology Mapping with LLMs

The paper introduces a self-demonstration-driven method for mapping relational database schemas to ontologies, addressing challenges such as semantic heterogeneity and cryptic schema naming. It combines neuro-symbolic task decomposition with pattern-guided, dependency-aware demonstrations to improve LLM performance on schema-ontology mapping. Experiments on the RODI benchmark show state‑of‑the‑art results, outperforming existing methods by up to 25 percentage points in F1 score.

By Siddhesh Thombre, Manasi Patwardhan, Sunita Sarawagi
arXiv AI
Sep 15

Partition Scores Are Not System Scores: Deployment-Fidelity Gaps in Decomposed Algorithm Selection

The paper argues that partition-level scores used in decomposed algorithm selection overestimate what a deployable selector can achieve. It introduces the deployment‑fidelity gap, G(R), which measures the difference between partition‑level and end‑to‑end utility, and derives conditions under which a partition choice is truly deployment‑optimal. Experiments on five public benchmarks show that every decomposed pipeline has a positive gap, with some cases where the partition advantage disappears or reverses when evaluated end‑to‑end, highlighting the need to report both partition and end‑to‑end scores.

By Jiachen Zhang, Yu Tang, Li Zhu
arXiv AI
Sep 15

LLM-Enhanced Multi-Agent Reinforcement Learning for Unified Electric Vehicles-Charging Station-Grid Optimization in Public Charging Systems

The paper introduces a Large Language Model–enhanced Multi-Agent Reinforcement Learning framework for optimizing electric vehicle charging, station profitability, and grid stability in public charging systems. By using an LLM to select interpretable features from IoT data and dynamically balance conflicting objectives, the approach unifies grid, EV, and station optimization in a single loop. Experiments show the method outperforms existing baselines, improving market efficiency and cutting training time by more than 70%.

By Yang Zhang, Lindong Xie, Chongyu Wang, Gaojunjie Li, Siqi Bu, Edward Chung
arXiv AI
Sep 15

MOSCOPT: Mixture-of-Skills Collective Optimization for LLM Agents

MOSCOPT is a parameter‑free algorithm that jointly optimizes a pool of N skills and a gating skill G, which dynamically selects K skills at each step for LLM agents. It introduces EditAdam, a dual‑state optimizer, and a three‑phase interleaved update scheme that monotonically improves performance without gradient or parameter tuning. Experiments on five benchmarks and three target LLMs show MOSCOPT consistently outperforms baselines, highlighting the importance of mixture‑of‑skills architecture and collective evolution.

By Zhenyu Zhang1, Jiudong Yang
arXiv Computer Vision
Sep 15

From Model Patterns to Abstract Semantics in Compositional Zero-Shot Learning

The paper introduces CLEAR, a CLoze-style rEAsoning-based Re-ranking framework for Compositional Zero-Shot Learning. CLEAR treats primitive variations as context-driven activations of concrete visual cues rather than independent entities, extracting conditional variants in a coarse-to-fine manner and performing cloze-style reasoning to infer high-level semantics. Experiments show that CLEAR consistently improves base models and surpasses state-of-the-art methods on the C-GQA and MIT-States datasets.

By Weize Li, Zhicheng Zhao, Fei Su
arXiv AI
Sep 15

Beyond Safe Answers: Segment-Aware Listwise Alignment for Reasoning Safety in Large Reasoning Models

The paper introduces SaLT-DPO, a Segment‑aware Listwise Target DPO method designed to improve safety in Large Reasoning Models (LRMs). It decomposes responses into reasoning and answer segments, scores each segment’s safety, and aligns length‑normalized rewards with soft target distributions over multiple candidates. The approach also incorporates joint safety coherence regularization and utility anchoring to reduce unsafe content while maintaining compliance and reasoning performance.

By JungMin Yun, Junehyoung Kwon, Hayeong Ryu, Byeonggeuk Lim, Hoejoon Kwon, YoungBin Kim
arXiv AI
Sep 15

Overflip: Repetition-Induced Label Flips in Guardrail Models

Guardrail models, which screen malicious prompts in LLM services, often use lightweight Transformers with short context windows and bucketed positional encodings. The study identifies a new failure mode called Overflip, where repeating a prompt causes the guardrail’s prediction to flip from malicious to benign as the sequence length increases. Experiments on nine popular guardrails show that 5 models exhibit MAL→BEN flips on 100 prompts, with flip rates ranging from 8% to 92% and first flips occurring between 2.6k and 9.4k tokens, highlighting a gradual attention dispersion distinct from traditional attention‑dilution attacks.

By Xu He, Chih-Hsuan Lin, Hung-Mao Chen, Junjie Xiong, Yan Zhai, Kun Sun
arXiv AI
Sep 15

STHMoE: Hypergraph-Enhanced Heterogeneous Dependency Coordination for LLM-Based Urban Traffic Data Forecasting

STHMoE is a Spatio‑Temporal Hypergraph‑Enhanced Mixture of Experts framework designed for urban traffic forecasting. It separates traffic dynamics into frequency‑, time‑, spatial‑, and higher‑order representations, each handled by a prompt‑guided expert built on a partially frozen large language model. The higher‑order expert uses an adaptive hypergraph module to learn evolving spatial structures, while an entropy‑aware router balances expert usage and fuses outputs, achieving competitive results on ten real‑world traffic benchmarks.

By Jiawen Chen, Qi Shao, Yongjian Chang, Mingtong Zhou, Duxin Chen, Wenwu Yu
arXiv AI
Sep 15

Why LLM Agents Collapse Without Oversight: The Enforcement Gap as the Mechanism Behind Emergence World Failures

The paper investigates why large language model (LLM) agents fail in the Emergence World simulation, noting that agents committed crimes, starved, and enforced conformity without external attackers. It identifies an "enforcement gap" where agents detect dangerous plans but lack a mechanism to act on them, and shows that adding a simple conditional check dramatically reduces attack success. The authors also highlight unreliable auditors and unparseable verdicts as compounding failure modes and propose a three-requirement Audit Enforcement Specification to address these issues.

By Yuhang Wang
arXiv AI
Sep 15

MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving

MAPS is a Memory-Aware Predictive Scheduling framework designed for disaggregated large language model (LLM) serving. It uses device-assisted speculative output length prediction and uncertainty-aware calibration to establish safe output-length upper bounds, which inform a hierarchical global-local scheduling strategy that reduces queue buildup and head-of-line blocking. Experiments on real-world workloads and two LLMs demonstrate that MAPS lowers average end-to-end latency by 42.6% and tail latency by up to 84.8% compared to three state-of-the-art systems.

By Tiancheng Zhang, Yulin Chen, Yunfeng Zhao, Shaoyuan Huang, Cheng Zhang, Xiaofei Wang