Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv AI
Oct 19

RL-Guided Temporal Localization for Dual-Channel Retrieval in Long-Horizon Agent Memory

The paper introduces TIDER, a time‑aware retrieval framework that uses a reinforcement‑learning trained temporal interval locator to identify relevant time windows for evidence retrieval in long‑horizon agent memory. By combining temporally grounded candidates with semantic retrieval and reranking them by proximity to the inferred windows, TIDER mitigates the displacement of target memories caused by temporally mismatched yet semantically similar entries. The lightweight 1.7B locator achieves state‑of‑the‑art accuracy on LifeBench and LoCoMo benchmarks, demonstrating the importance of temporal cues for effective evidence retrieval.

By Dabin Sheng, Zhe Wu, Jinming Zhao, Junliang Xing, Yuanchun Shi
arXiv Machine Learning
Oct 19

Embodiment-Classifier Guidance: Transferring Frozen VLAs Across Robot Dynamics without Demonstrations

The paper introduces Embodiment‑Classifier Guidance, a method that adapts a frozen flow‑based policy π_0 to new robot embodiments without additional demonstrations or parameter updates. By training a lightweight classifier to distinguish successful action chunks between source and target bodies, the policy’s denoising steps are nudged toward the target distribution using the classifier’s gradient. Experiments on the LIBERO simulation benchmark show that this guidance improves average success rates from 72% to 80% across 12 controller–body pairs, reduces failures by 31%, and demonstrates data‑efficient adaptation with only a few target‑body rollouts.

By Devin Han, Hershraj Niranjani
arXiv Computer Vision
Oct 19

PreciseEdit: Strong Yet Stable Semantic Transport for One-Step Image Editing

PreciseEdit is a new one‑step image editing framework that improves semantic transport by applying strong target‑directed changes only along reliable directions and spatial locations. It is model‑agnostic, training‑free, and inversion‑free, and uses a dual‑noise‑scale strategy to balance strong editing with background fidelity and structural consistency. Experiments on PIE‑Bench show that PreciseEdit outperforms existing methods across a range of editing tasks.

By Jingyi Wang, Xiao Lu
arXiv AI
Oct 19

Are Large Reasoning Models Good Everyday Reasoners? A Comparison with Non-Reasoning Models on Logical Reasoning Tasks

The paper evaluates large reasoning models (LRMs) on everyday logical reasoning tasks, introducing a benchmark that captures linguistic and pragmatic reasoning phenomena. It compares LRMs with non-reasoning models under controlled prompting and finds that LRMs often perform worse on everyday reasoning, that some non-reasoning models with chain-of-thought prompting can outperform them, and that additional prompting instructions offer limited control over reasoning behavior. These results highlight that current reasoning models do not reliably capture everyday reasoning and suggest the need for better training objectives and broader evaluation.

By Kentaro Ozeki, Risako Ando, Takanobu Morishita, Hirohiko Abe, Koji Mineshima, Mitsuhiro Okada
arXiv AI
Oct 19

Rethinking User Encoders: Routing Diffusion States for Scenario-Adaptive User Representation

The paper introduces FINDERS, a diffusion‑language‑model framework that replaces static user encoders with a dynamic routing system. FINDERS uses four operations—denoise, factorize, route, and compress—to create scenario‑adaptive user representations by organizing denoising states into transferable cross‑modal semantics and modality‑specific factors. The authors also build BehaviorQA, a large corpus with explicit behavioral rationales, and demonstrate through extensive experiments and online A/B tests that FINDERS outperforms existing methods across ten industrial scenarios.

By Junru Zhang, Jinyong Wen, Bin Dou, Wuliang Huang, Baokun Wang, Yu Cheng, Weiqiang Wang, Cheng Peng
arXiv AI
Oct 19

Sep-Latent: Separator Tokens as Natural Compression Anchors for Implicit Reasoning

Sep‑Latent introduces separator tokens as natural compression anchors for implicit reasoning in large language models, enabling the model to aggregate and compress reasoning logic into these anchors via an Attention‑Decay mechanism. The framework aligns explicit and implicit pathways through multi‑level distillation, ensuring logical consistency, and allows dynamic reasoning depth by terminating the latent thinking phase based on query complexity. On the GSM8k‑Aug benchmark, Sep‑Latent achieves 56.8% accuracy with an average of only 5.4 tokens, outperforming prior fixed‑length implicit models in efficiency while narrowing the gap with explicit Chain‑of‑Thought methods.

By Yinlong Xu, Jianfeng Huang, Zhengdong Zhou, Mingjie Lin, Fuyu Lv, Dan Ou, Hongxia Xu, Jian Wu
arXiv AI
Oct 19

AI-Driven Reproducibility and Analytical Robustness in Scientific Discovery

The paper introduces SEER‑Repro, a benchmark of 20 SEER‑based studies with data and resources for reproduction. An LLM agent reproduces 77.4% of 2,190 atomic claims and provides executable baselines. An agentic analysis pipeline then tests robustness by identifying analytical choices, proposing alternatives, and rerunning analyses, flagging potentially consequential choices in 9.5% of fully reproduced findings for expert review.

By Shicheng Liu, Krista Opsahl-Ong, Andrew Drozdov, Julian Killingback, Arnav Singhvi, Ronak Pradeep, Michael Bendersky, James C. Dickerson, Monica S. Lam
arXiv AI
Oct 19

OwlSight: A Panoramic Monitor for Chain-of-Thought Reasoning

OwlSight is a new chain‑of‑thought (CoT) monitoring system that uses a customizable Factor strategy to adapt to a wide range of tasks and monitoring needs. By incorporating fine‑grained and coarse‑grained factors, OwlSight can detect specific behaviors such as email leakage or broader privacy concerns across multiple dimensions like capability, safety, and trustworthiness. Experiments on 15,971 reasoning trajectories show that OwlSight outperforms general‑purpose LLMs by 1.32‑30.57 points at fine granularity and 11.71‑27.81 points at coarse granularity, achieving state‑of‑the‑art results on OpenAI’s CoT monitorability benchmark and demonstrating robustness to model changes, factor rephrasings, CoT compression, and obfuscation.

By Lewen Yan, Yanxu Zhu, Jie Zhang, Jing Shao
arXiv AI
Oct 19

Recoverability-Controlled Safety Miscalibration in Large Language Models

The paper introduces RiskCueBench, a benchmark designed to separate intent recovery from safety behavior in large language models. It evaluates 25 model groups on 640 prompts covering 160 latent intents across eight risk categories, revealing that higher contextual specificity can reduce refusal rates and safety scores even after intent recovery. The study highlights that safety behavior may be miscalibrated to prompt surface form, underscoring the need for evaluations that condition on semantic recoverability.

By Yuwei Wang, Xingwang Liang, Yu Xie, Haotian Chi, Haijun Geng
arXiv AI
Oct 19

Self-Evolving Coding Agents via Harness-Policy Optimization

The paper introduces Self-HPO, a recursive framework that alternates between optimizing the harness and the policy of large language model agents for coding tasks. By letting the current policy propose new harness candidates based on its failures, the method aligns harness evolution with policy improvement, leading to consistent performance gains across three benchmarks. Experiments show that Self-HPO raises the average resolve rate of Qwen3.5-35B-A3B from 57.8% to 64.8% and improves reliability and capability metrics, with benefits that generalize to out‑of‑distribution settings.

By Zhepei Wei, Wenlin Yao, Baolin Peng, Danqing Wang, Qianhui Wu, Hao Cheng, Yu Meng, Jianfeng Gao
arXiv AI
Oct 19

Competence-Based Evaluation of Logical Reasoning in Large Language Models

The paper proposes a competence‑based evaluation (CBE) framework for assessing logical reasoning in large language models. CBE tests whether models consistently apply the same reasoning competence across a family of problems, using a parameterized generator to vary complexity while preserving the underlying rule. Experiments on GPT and Qwen models show that even strong models often fail to meet the competence condition across the tested family, indicating limitations in generalization.

By Jize Jiang, Chengxiang Zhai
arXiv AI
Oct 19

COMET: Decoupled Distillation, Routing, and Capacity Control for Task-Agnostic Continual Vision--Language Learning

COMET is a task‑agnostic continual learning framework for vision‑language models that addresses modal‑consistency drift, representation‑space interference, and per‑task capacity shortfall. It uses a frozen teacher for feature reference, Tri‑Affinity Distillation to preserve image‑image, text‑text, and image‑text geometry, and Student‑Aware Cross‑Expert Distillation to align new experts with prior ones, while a contextual bandit and reconstruction router manage capacity and inference. Experiments on the X‑TAIL benchmark show that each component improves continual learning performance, demonstrating that separating feature reference, expert shaping, routing, and capacity control enables effective multimodal adaptation.

By Seungyoon Woo, Keonhee Park, Gunhee Kim
arXiv AI
Oct 19

Aligning Language Models Towards Counterfactual Invariance

The paper investigates how small changes in prompt characteristics, such as demographic attributes, can alter the semantics and tone of large language model outputs. It shows that standard alignment methods like reinforcement learning can worsen counterfactual bias by increasing response quality differences across demographics. To mitigate this, the authors propose a policy‑gradient approach with a group‑level reward that operates on matched counterfactual prompt pairs, and they validate its effectiveness on fairness benchmarks while maintaining overall response quality.

By Basar Demir, Emily Shen, Ksheeraj Sai Vepuri, Sourabh Kulkarni, Harihar Subramanyam, Ankit Jain, Nan Jiang, Chirag Nagpal
arXiv AI
Oct 19

SIT-Eval: Situation-Based Interactive Turn-by-Turn Evaluation of Conversational Agents

SIT‑Eval is a framework that evaluates conversational agents by sampling diverse situations at each turn and judging responses against situational rules. It uses an LLM‑interlocutor to generate uniformly sampled contexts and a context‑aware LLM‑judge to assess rule compliance. Benchmarking eight large language models on 1,292 turns uncovered hidden failure modes, showing that models with similar overall scores can differ greatly in rule adherence, especially in ambiguity clarification and Socratic questioning.

By Ravil Mussabayev, Saveliy Zavertyaev, Dmitriy Parpulov, Andrey Savchenko
arXiv AI
Oct 19

Porimon: An LLM-Based Pok\'emon Battle Agent Enhanced by Long/Short-Term Knowledge Augmented Generation

The paper introduces Porimon, an LLM-based agent for Pokémon Battles that uses Long/Short-Term Knowledge Augmented Generation (LSTKAG) to incorporate past states of the current task and retrieve experience summaries from similar previous instances. Porimon also employs an external API for precise damage calculation and detailed game information. Experimental results from 15,000 simulated battles show that Porimon, with hyperparameter optimization, outperforms both the earlier PokéLLMon agent and a rule-based heuristic player, and ablation studies confirm the benefit of the game information retrieval extension, though the impact of Long-Term KAG remains inconclusive.

By Dongyin Zhuo, Fengjunjie Pan, Nenad Petrovic, Alois Knoll
arXiv AI
Oct 19

Audit-First VAPO: Risk-Certified Selective Updates under Imperfect Verification

The paper introduces Audit-First VAPO, a method that separates directional admission from magnitude control for selective updates under imperfect verification. It employs an observation-only accept‑appeal‑abstain policy with a finite secondary‑verification budget, freezing the action trace before clean labels are added, and uses finite‑sample bounds to certify harmful risk, coverage, and verifier‑call rate. Experiments on Qwen3.5‑0.8B and GSM8K show that RC‑VAPO achieves high accuracy while maintaining low selected harmful risk and modest verifier cost compared to baseline methods.

By Miaobo Hu, Shuhao Hu, Xiaobo Guo, Xin Wang, Bokun Wang, Rui Chen, Daren Zha
arXiv AI
Oct 19

VietMed: A Dataset and Benchmark for Automatic Speech Recognition of Vietnamese in the Medical Domain

VietMed is a Vietnamese medical speech recognition dataset featuring 16 h of labeled medical speech, 1 000 h of unlabeled medical speech, and 1 200 h of unlabeled general‑domain speech. It claims to be the largest public medical ASR dataset worldwide in seven dimensions—total duration, speaker count, disease coverage, recording conditions, speaker roles, unique medical terms, and accents—and also the largest Vietnamese speech dataset by duration. The authors release pre‑trained models (w2v2‑Viet and XLSR‑53‑Viet) and fine‑tuned medical ASR models, showing that XLSR‑53‑Viet outperforms the state‑of‑the‑art XLSR‑53 by reducing WER from 51.8 % to 29.6 % on a medical test set. whyItMatters":"The dataset and models provide the first large‑scale, publicly available Vietnamese medical ASR resource, enabling research and development of medical speech technologies in a language and domain that previously lacked sufficient data."

By Khai Le-Duc
arXiv AI
Oct 19

LeanTTA: A Backpropagation-Free and Stateless Approach to Quantized Test-Time Adaptation on Edge Devices

LeanTTA is a backpropagation-free, stateless framework for quantized test-time adaptation designed for edge devices. It updates normalization statistics on the fly, avoiding large batches and historical data, and combines partial adaptation with quantized module fusion to reduce computational costs. Experiments across sensor modalities show a 15.7% error reduction, peak memory usage of only 11.2 MB for ResNet18, and fast adaptation within an order of magnitude of normal inference speeds on-device.

By Cynthia Dong, Hong Jia, Young D. Kwon, Georgios Rizos, Cecilia Mascolo
arXiv AI
Oct 19

HydroShear: Hydroelastic Shear Simulation for Tactile Sim-to-Real Reinforcement Learning

HydroShear is a hydroelastic tactile simulator that models stick‑slip transitions, path‑dependent force and shear buildup, and full SE(3) object‑sensor interactions. It uses Signed Distance Functions to track indenter displacements on a sensor membrane, producing physics‑based force fields from arbitrary watertight geometries while remaining engine‑agnostic. Experiments on GelSight Minis show HydroShear reproduces real tactile shear more faithfully than existing methods, enabling zero‑shot sim‑to‑real reinforcement learning with a 93% success rate across four dexterous tasks.

By An Dang, Jayjun Lee, Mustafa Mukadam, X. Alice Wu, Bernadette Bucher, Manikantan Nambi, Nima Fazeli