Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv AI
Oct 27

Recursive Semantic Divergence for LLM Agent Consistency

The paper introduces Recursive Semantic Divergence (RSD), an unsupervised, trajectory‑level consistency metric for large language model agents that use tools. RSD recursively measures expected cumulative semantic disagreement between two independent rollouts from the same state, using natural language inference to detect logical inconsistency. When applied as a reward signal during fine‑tuning, RSD significantly reduces variability in task progress across benchmarks and model scales while matching supervised fine‑tuning performance without needing ground‑truth labels.

By Harshavardhan Abichandani, Penny Chong, Atin Ghosh, Daniel Dahlmeier
arXiv AI
Oct 27

A Benchmark for Controllable Speaking-Style Captioning in Audio-Language Models

The paper introduces StyleInstructCaps, a benchmark for controllable speaking‑style captioning (SSC) in audio‑language models. SSC generates natural language descriptions of speech delivery, capturing paralinguistic attributes such as vocal timbre, prosody, and expressivity, and requires dimension‑specific style captions rather than a single undifferentiated description. The benchmark provides a dataset and evaluation framework that assesses metadata groundedness, hallucination, instruction‑following ability, generalization to unseen instructions and audio datasets, and speaker‑style consistency, and shows that instruction‑diverse training improves groundedness, controllability, and speaker consistency across nine state‑of‑the‑art models.

By Shreeram Suresh Chandra, Ashwyn Sharma, Liang Lu, Xiutian Zhao, Leda Sari, Bj\"orn Schuller, Berrak Sisman
arXiv AI
Oct 27

KDHNet: A Knowledge-Driven Hybrid Network for Limited-Calibration SSVEP Decoding

KDHNet is a hybrid network that merges knowledge‑driven analytical filtering with data‑driven deep learning to decode steady‑state visual evoked potentials (SSVEP) from limited calibration data. It uses cascaded task‑discriminant and task‑related component analysis to extract discriminative, reproducible components, then fuses sub‑band weights and a transformer‑based decoder for temporal and spectral representation. Extensive experiments on three public datasets show that KDHNet outperforms existing methods in both intra‑ and inter‑subject settings, especially when calibration data are scarce.

By Zhentao He, Ziwei Wang, Dongrui Wu
arXiv AI
Oct 27

GMoE-FTI: A Universal Facial Template Inversion Method Adopting Genuine Mixture-of-Experts Mechanism

The paper introduces GMoE-FTI, a universal facial template inversion framework that uses a genuine mixture‑of‑experts mechanism to reconstruct faces from various template types. It trains five mapping experts and a matching module, then combines their outputs during inference and fine‑tunes all components jointly, achieving superior performance on the LFW benchmark. Evaluations on seven unseen template types show a 40% improvement in true accept rate and reductions in false accept rates, demonstrating its broad applicability.

By Zixuan Shen, Zhihua Xia, Kaikai Gan, Peipeng Yu
arXiv AI
Oct 27

Diagnosing and Mitigating Evidence Misgrounding in Multimodal Knowledge Editing

The paper introduces WaterBird-Edit, a benchmark that separates bird habitat labels from background context to detect evidence misgrounding in multimodal knowledge editing. It proposes VICE, a retrieval‑based framework that stores edits in external memory and uses visual in‑context demonstrations to focus on task‑relevant evidence. Experiments on three benchmarks show VICE improves lifelong editing performance and reduces sensitivity to background changes on WaterBird-Edit.

By Zhiwu Wang, Yichen Wu, Jinqian Yang, Yizhou Yang, Renzhen Wang, Qian Zhao, Deyu Meng
arXiv AI
Oct 27

AnalogPortBench: Benchmarking LLMs on Analog Circuit Porting

AnalogPortBench is a new benchmark comprising 54 tasks that evaluate how well large language models (LLMs) can port ten analog circuits—from current mirrors to telescopic amplifiers—across process nodes, specifically from a 0.18 µm process to the SKY130 PDK. The tasks vary in difficulty, ranging from straightforward ports to reduced supply and stricter specifications, and include infeasible scenarios; each feasible task has a verified reference design at five process corners. Experiments with six leading LLMs show that while models can handle simple circuit ports, they rarely redesign headroom‑sensitive amplifiers, often miss infeasible specifications, and benefit more from simulation feedback than from additional samples or reasoning steps.

By Saeid Rajabi, Aly Moussa, Vishal Saxena, Chengmo Yang, Satwik Patnaik
arXiv AI
Oct 27

EGGPlacer: Energy-Guided Generative Chip Placement with Diffusion Models

EGGPlacer is an energy‑guided diffusion framework for macro placement that integrates analytical placement feedback and differentiable thermal feedback. By treating these signals as composable energies, the method injects them into reverse diffusion without retraining the generative model. Experiments on public benchmarks show that EGGPlacer achieves competitive mixed‑size HPWL and outperforms other methods in routed wirelength and peak temperature.

By Qi Zhou, Guojun Liu, Juanhong Zhao, Guangzhi Qi, Zehao Chen, Ming Lu
arXiv AI
Oct 27

A Deterministic Security Runtime for LLM Agents on MPI and Batch-Scheduled HPC Systems: Consensus-Governed, Hardware-Rooted Supervision of Every Rank

The paper introduces Enclawed, a deterministic security runtime that protects large language model agents running on high‑performance computing (HPC) systems. It provides hardware‑rooted supervision for every rank, preventing compromised hosts from causing catastrophic failures, and enforces launch policies, lease‑based heartbeats, and automatic eviction or killing of tampered jobs. Enclawed operates without administrator privileges, integrates via Secure Shell, and adds minimal startup overhead while successfully blocking a wide range of simulated attacks.

By Alfredo Metere
arXiv AI
Oct 27

Can Agentic Code Review Catch Vulnerabilities Before They Are Merged?

The paper introduces SecPRBench, a benchmark that evaluates large language model agents on real-world bug‑introducing commits from critical software systems such as Firefox, V8, and OpenSSL. It automates harvesting, building, and validating these commits, and uses an LLM‑as‑a‑judge to match findings with ground‑truth vulnerabilities. Across eight state‑of‑the‑art models, recall ranges from 32.5% to 78.1%, with Grok 4.6 and GPT‑6 sol performing best, while security‑focused harnesses like OpenAI’s Codex Security and Anthropic’s Defending Code influence recall differently.

By Yihan Yang, Chenyuan Yang, Zijie Zhao, Anh Nguyen Hoang, Ziqi Zhang, Weidong Wang, Zhiheng Lyu, Lingming Zhang
arXiv AI
Oct 27

Energy Efficient Lung Tumor Segmentation with a modified CBAM ENet Skip Model

The paper presents a lightweight, energy‑efficient lung tumor segmentation model that combines the ENet architecture with UNet‑style skip connections and a lightweight Convolutional Block Attention Module (CBAM). Trained on TCIA data and evaluated on LIDC‑IDRI, the model achieves a Dice score of 0.90812 while using only 430,052 parameters and 0.549 GMAC, making it up to 10 times faster and far less computationally demanding than UNet or SegNet. Grad‑CAM is employed to provide visual interpretability and highlight areas needing improvement.

By Sharan Thummagunti, Aayushi Padia, Abhiram Koppuravuri, Prabhu Prasad B M, G N Girish
arXiv AI
Oct 27

Novel or Not? PatEx-Gym: Agentic Prior-Art Search for Novelty Assessment in Patent Examination

PatEx-Gym is a new framework for developing and evaluating agents that perform patent examination. It includes PatEx-Bench, a benchmark of 1,000 realistic tasks with large retrieval corpora, and PASS, an annotation‑free method for synthesizing search trajectories. A 4B‑parameter PatEx‑Agent trained on these synthetic trajectories provides a lightweight baseline that competes with much larger general‑purpose models, yet prior‑art discovery remains challenging.

By Qiyao Wang, Xinyi Chen, Hongbo Wang, Jiayuan Lu, Jianhang Li, Minghuan Tan, Yuan Lin, Min Yang
arXiv AI
Oct 27

Pacifo: An Efficient Fully Homomorphic Encryption-based Lightweight ResNet for Trustworthy Machine Learning as a Service

Pacifo is a framework that adapts fine‑tuning strategies to efficiently run the ResNet20 model using Fully Homomorphic Encryption (FHE). It introduces a public‑private layer partitioning scheme and an efficient ReLU approximation, reducing private layers and costly FHE operations during inference. Compared to state‑of‑the‑art FHE implementations, Pacifo achieves a 3.7–21.7× speedup while preserving 95–99 % of the original plaintext model accuracy.

By Yang Li, Gang He, Ruichen Zhang, Guangyuan Liu, Yinqiu Liu
arXiv AI
Oct 27

Mycelium: A Meta-Study of Hallucination Detection Across Provenance, Typology, and Granularities

The paper presents a meta‑study of hallucination detection in large language models, framing it as a benchmark‑conditioned measurement problem. It organizes existing benchmarks along three axes—provenance, typology, and granularity—to analyze how construction, annotation, and evaluation choices influence detection tasks. The authors also examine span‑level evaluation, propose a new Span Coverage F1 metric, and discuss the implications for cross‑benchmark comparisons and generalization.

By Julia Belikova, Elisei Rykov
arXiv AI
Oct 27

RSI Arena: Towards Recursive Self-Improvement with Human Feedback

RSI Arena is a two‑stage experiment where ten large language models, each starting from NVIDIA Nemotron 3.5 Lightning 30B-A3B Base, autonomously train and refine themselves using human feedback. In the first six‑day stage, each agent receives 1,000 GPU‑hours and $300 in API credit to select data, design training procedures, and choose checkpoints, producing working assistants that close a median 76% of the base‑to‑Instruct gap on 12 benchmarks. The second stage gives agents access to comparison records and an additional three days of training, resulting in style‑adjusted relative rating improvements for eight of the ten models, though benchmark gains remain modest and some agents exhibit longer responses or increased harmful compliance.

By Yue Huang, Yanbo Wang, Lichi Li, Yuanyuan Chen, Andreas Haupt, Xiaonan Luo, Zheyuan Liu, Jihye Choi, Felix Juefei-Xu, Xiangliang Zhang, Radha Poovendran, Misha Sra, Ying Liu, Ben Burtenshaw, Thomas Wolf, Alex Pentland, Zichen Chen, Zhangchen Xu
arXiv AI
Oct 27

StateWise: Teaching Language Models to Track and Obey Evolving Constraints in Multi-Turn Dialogue

StateWise is a method that trains a single large language model to maintain and update its own constraint state during multi‑turn dialogue. By fine‑tuning the model to summarize conversations into a typed state with tombstones for revoked constraints and then self‑training on trajectories that pass exact round‑trip checks, it reduces relapse and improves conversation success. The approach outperforms a compiled‑ledger baseline on relapse without requiring inference‑time scaffolding, and it is evaluated on two backbones with held‑out constraint families and unseen phrasings.

By Samarth Varshney, Ritu Yadav
arXiv AI
Oct 27

TriageRAG: Failure-Aware Self-Training for Adapting Retrieval-Augmented Language Models to Specialized Domains

TriageRAG is a failure‑aware self‑training method for retrieval‑augmented language models that adapts them to specialized domains using only unlabeled text. It probes each self‑generated question with and without retrieved context, classifies the result into one of four categories (known, solved, context‑failure, retrieval‑failure), and routes each category to a tailored training signal, including a parametric route that teaches the reader to answer when retrieval misses. Experiments on 11 datasets across three domains and on CS‑Bench show that TriageRAG outperforms a budget‑matched SimRAG by 1.2–6.4 points and uniquely recovers closed‑book performance whenever retrieval harms the base model.

By Samarth Varshney, Ritu Yadav
arXiv AI
Oct 27

One Scan is Enough: Demonstration-Free Adaptation for Language-Guided Navigation

The paper presents a method that replaces human demonstrations with a single brief iPhone scan to adapt a language-guided navigation policy to new buildings. Scan2Instruction transforms the scan into a full training environment, automatically generating navigation episodes with grounded instructions. PRPT-KS then adapts the policy using a closed‑form Pareto utility derived from the reconstructed scene, achieving an 85% real‑robot success rate in five real environments, compared to 49% for a baseline trained on human demonstrations.

By Zhuoyuan Yu, Yuxing Long, Du Wang, Junyang Wang, Hongwei Fan, Hao Dong
arXiv AI
Oct 27

Identically Trained Language Models Break on Different Jailbreaks

The study evaluates how identically trained language models respond to jailbreak prompts, finding that models trained with the same pretraining run exhibit highly correlated non-refusal rates (average 0.984), while different pretraining seeds reduce correlation to 0.811 and different data recipes slightly lower it further to 0.779. This indicates that a model’s susceptibility to jailbreaks is largely determined by its specific pretraining run rather than the data used. Additionally, each pretraining run leaves a distinct fingerprint, enabling identification of the run from a model’s responses to a set of prompts.

By Chi-Wei Chang, Mu-Tien Kuo, Kuan-Wei Lee, Meng-Hsun Tsai
arXiv AI
Oct 27

Improving the Chain-of-Thought Compressibility of Pruned Reasoning Models via Long--Short Contrastive Pruning

The paper introduces Contrastive Set-Difference (CSD), a pruning method that targets neurons involved in redundant reasoning by contrasting activations on long versus short correct solutions. CSD is combined with an extended preference-based chain-of-thought compression that also considers problems a pruned model always or never solves. Applied to a 7B model, the approach reduces thinking tokens by 8–22% on math benchmarks while maintaining accuracy within 0.5 points compared to traditional pruning and compression pipelines.

By Tao Jing, Chen Kang, Feifei Zhai, Pengyuan Liu, Dong Yu
arXiv AI
Oct 27

SHIFT-JEV: Certified Abstention and Gain-Gated Continual Adaptation for LLM-Based Decision Models under Distribution Shift

SHIFT-JEV is a procedure for large language model (LLM) decision systems that addresses domain, label‑prior, emerging‑class, and candidate‑set shifts. It combines risk scoring from uncertainty, format, and option‑consistency signals with certified abstention thresholds, drift and label‑shift monitoring, and a gain‑gated model‑update policy. Experiments on ten shifted test environments with Qwen3 and Phi‑4-mini show that fine‑tuning with KL anchors and temperature scaling reduces calibration error, while SHIFT‑JEV’s uncertainty‑based abstention and chi‑squared shift detection outperform baseline methods, achieving higher accuracy in continual adaptation scenarios.

By Muhammad Atif Saeed, Azka Atiq, Adarsh Kumar, Akhtar Jamil, Uzair Iqbal, Roland Aydin