Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv Machine Learning
Oct 28

When Respondents Are Scarce: LLM-Derived Semantic Priors for Bayesian Confirmatory Factor Analysis

The paper introduces the Semantic‑Empirical Updating Framework (SEUF), which uses large language model paraphrases and embeddings aligned with hypothesized item‑factor assignments to create prespecified priors for Bayesian confirmatory factor analysis. In experiments with a 38‑item survey and two public surveys (FEVS and RIASEC), SEUF maintained semantic structure after wording controls and achieved comparable loading precision with 50 respondents to conventional CFA with 70–82 respondents, reducing required sample size by 29–39%. The approach yielded lower loading errors and better credible‑interval coverage, especially when respondent data were scarce, demonstrating the benefit of incorporating semantic priors in small‑sample measurement.

By Shaohuai Feng, Mohd Wira Mohd Shafiei, Mohd Sukri Shafie, Siyang Tian
arXiv Machine Learning
Oct 28

Adaptive Memory Index Evolution for Agents

Adaptive Memory Index Evolution for Agents (AMIE) is a lightweight plug‑in layer that evolves per‑memory access state from deployment retrievals without altering stored content or retraining the host memory system. It distills a semantic teacher’s keep/suppress judgments into a retrieval‑conditioned policy and accumulates selective actions into bounded state that reweights subsequent host retrievals. Across five retrieval‑based hosts and three benchmarks, trained AMIE improves downstream F1 or accuracy in all 15 comparisons (0.70–3.37 percentage points) and improves retrieval in all 10 comparisons, demonstrating that persistent memory access can be adapted independently of memory content.

By Yuntong Hu, Tongli Su, Yuanfu Sun, Shuo Ma, Qiaoyu Tan, Liang Zhao
arXiv Machine Learning
Oct 28

LEAP: a Latent Embedding Anomaly Patch-Transformer for time-series anomaly detection

LEAP is a new time‑series anomaly detection model that avoids reconstruction by using masked prediction and latent‑embedding evidence. It removes patches from a window, predicts them from remaining context, and scores prediction errors as anomaly evidence. The model incorporates three label‑free novelty signals—latent‑embedding novelty, local temporal contrast, and per‑channel novelty—and achieves strong performance on the TSB‑AD benchmark, with statistically significant gains from each component.

By Cristhian Moya-Mota, Ignacio Aguilera-Martos, Diego Garc\'ia-Gil, Juli\'an Luengo
arXiv Machine Learning
Oct 28

CoPE-CGL: Community-aware Prototype Extraction for Replay-Free Continual Graph Learning

CoPE-CGL is a replay‑free continual graph learning framework that uses community‑aware prototype extraction and a parameter‑free structural prior derived from multi‑hop Laplacian smoothing. It adapts to both transductive and inductive settings, requiring only a constant amount of storage per task and no past graph data. In experiments on Arxiv, CoraFull, and other benchmarks, CoPE‑CGL achieves performance comparable to or better than replay baselines, runs significantly faster, and eliminates catastrophic forgetting.

By Bo-Young Lim, Seokhwan Kim, Kisung Lee, Hyuk-Yoon Kwon
arXiv Machine Learning
Oct 28

Benchmarking uncertainty quantification for engineering regression under graded distribution shift

The paper introduces a graded distribution‑shift benchmark for evaluating uncertainty quantification (UQ) in engineering regression models. It defines a three‑rung shift ladder—distribution, soft, and hard shift—using physically meaningful variables and tests 12 UQ methods across seven public engineering datasets, including extrapolation, turbofan fault, and air‑quality scenarios. Results show conformal methods maintain high coverage under distribution shift, but coverage drops significantly under hard shift, with no clear method outperforming others, and weighted conformalised quantile regression often abstains from predictions.

By Ahmad Lawal, Rahim Pasbanigoloojeh, Asmau Lawal Bilbis
arXiv Machine Learning
Oct 28

TRACE: Memory-Efficient Time Series Class-Incremental Learning via Temporal-Spectral Reconstruction

TRACE is a memory‑efficient, replay‑free framework for Time Series Class‑Incremental Learning that uses generative modeling to preserve class‑specific knowledge. It employs a compact, frozen temporal‑spectral representation and lightweight class‑specific autoencoders to capture intra‑class variations without retaining historical samples or large statistical matrices. Experiments on five benchmarks show that TRACE achieves competitive accuracy while reducing retained memory by about 98.26% compared with the strongest baseline and improving accuracy on GrabMyo by roughly 33.43%.

By Junhui Mao, Junru Zhang, Liming Ye, Cheng Peng, Haoran Shi, Xiaoli Tang, Baokun Wang, Bin Dou, Yabo Dong, Han Yu
arXiv Machine Learning
Oct 28

Real-Time Human Trajectory Prediction from RGB-D for Mobile Robots Using Bi-Mamba

The paper introduces TrajMamba, a lightweight bidirectional state‑space model that predicts short‑term human trajectories from RGB‑D cameras for mobile robots. It aligns training and deployment pipelines by sharing a processing path for pose extraction, depth back‑projection, and smoothing, and encodes recent hip‑center velocities with stacked bidirectional Mamba blocks before decoding future velocities with a GRU head. On the ETH/UCY benchmark, TrajMamba achieves an average ADE of 0.29 m and FDE of 0.51 m, runs in real time on modest hardware, and has been deployed on a TIAGo robot equipped with an NVIDIA Jetson Orin.

By Basheer Al-Tawil, Magnus Jung, Ayoub Al-Hamadi
arXiv Machine Learning
Oct 28

IntNet: A Communication-Driven Multi-Agent Reinforcement Learning Framework for Cooperative Autonomous Driving

IntNet is a multi-agent reinforcement learning framework designed to improve safety in autonomous driving by integrating vehicle intent transmission and adaptive communication scheduling into a single end-to-end learning system. The model uses an observation prediction module for agents to forecast future policy outputs and share them over a vehicle-to-vehicle network, while a Graph Attention Network processes incoming messages and a scheduler optimizes the communication graph for bandwidth efficiency. Experiments in complex urban environments show that IntNet achieves better coordination, lower collision rates, and reduces communication effort by up to 60% compared to state‑of‑the‑art communicative MARL methods.

By Leandro Parada, Kevin Yu, Panagiotis Angeloudis
arXiv Machine Learning
Oct 28

Security of Deep Reinforcement Learning for Autonomous Driving: A Survey

The paper surveys 86 recent studies on the security of reinforcement learning (RL) in autonomous driving. It systematically categorizes attacks and defenses by threat models and single- versus multi-agent settings, addressing gaps in previous surveys. The authors also evaluate how current attacks and defenses apply to autonomous driving, offering guidance for building robust RL systems.

By Ambra Demontis, Srishti Gupta, Maura Pintor, Luca Demetrio, Kathrin Grosse, Hsiao-Ying Lin, Chengfang Fang, Battista Biggio, Fabio Roli
arXiv Machine Learning
Oct 28

Probabilistic adjustment of seasonal forecasts using cVAE with a non-parametric decoder: A case study of Arctic sea ice predictions

The paper presents a probabilistic post‑processing framework that uses a conditional Variational Autoencoder (cVAE) with a non‑parametric decoder to adjust seasonal Arctic sea‑ice forecasts. By replacing the Gaussian decoder with a generator and optimizing with Continuous Ranked Probability Score (CRPS), the method produces larger, bias‑corrected ensembles that are better calibrated, more consistent with observations, and exhibit higher resolution and sharper spectral power than standard cVAEs. The approach demonstrates improved forecast skill and reduced errors compared to benchmark predictions.

By Parsa Gooya, Reinel Sospedra-Alfonso
arXiv Computation and Language
Oct 28

Lipi-Ghor: A Large-Scale, Diarized, Multi-Domain Bengali Speech Corpus and a Scalable Pipeline for Building It

Lipi-Ghor is a large-scale, weakly-supervised Bengali speech corpus containing 885 hours of audio from 998 YouTube videos across 17 domains, each annotated in SSTT format with speaker diarization and timestamps. It is the first Bengali corpus to combine extensive coverage with speaker-diarized, long-form audio annotations, and the authors provide a replicable pipeline for building similar resources in other languages. The dataset includes a human-annotated gold evaluation split and is released under CC‑BY‑4.0, with raw audio withheld.

By Sanjid Hasan, A H M Fuad, Risalat Labib, Bayazid Hasan Simon
arXiv Computation and Language
Oct 28

Beyond IceBERT: Open-Data ModernBERT Encoders for Icelandic from 12M to 1B Parameters

The paper presents a series of Icelandic language encoders built with the ModernBERT architecture, ranging from 12 million to 1 billion parameters, trained on 9 billion tokens of open data. It reexamines IceBERT’s findings, showing that a cleaned web crawl suffices for strong performance and that starting from a multilingual model can aid certain tasks. The 1 billion‑parameter model, trained from scratch, outperforms IceBERT on all eleven benchmark tasks, and when warm‑started from the English Ettin encoder it achieves the best scores across all evaluated tasks.

By Hafsteinn Einarsson
arXiv Computation and Language
Oct 28

Do We Need an LLM to Know Why Sources Conflict? Decision Models for Knowledge-Conflict Typing in RAG

The paper investigates whether fast, cost‑efficient decision models can classify the type of conflict among retrieved sources for a query, a task that improves response quality when the conflict type is known. Using the CONFLICTS benchmark, the authors evaluate 11 commercial decision models against Gemini LLMs, finding that most achieve 50–67% accuracy, often matching or surpassing Gemini 2.5 Flash while incurring 4–39× lower cost and 2.6–6.0× lower latency. A cascade approach that forwards uncertain cases to Gemini 3.8 Flash yields comparable performance at only 25–37% of its cost, and the resulting typed responses generally behave as expected when compared to LLM‑derived types.

By Parv Khatri, Harshul Jain, Mehak Kapoor, Tanya Sah
arXiv Computation and Language
Oct 28

Decision Models as Judges for Hallucination Detection

Decision models, which provide predefined verdicts with associated probabilities, are evaluated as a low‑cost alternative to large language models used as judges for hallucination detection. In a benchmark of 9,852 items across five hallucination types, the best decision model matches the generative judge GPT‑6‑sol on context inconsistency and is within 3 percentage points on factual errors, though it lags on logical and instruction inconsistency. A confidence cascade of decision models improves performance by 1.9–3.3 points on three types and can match or exceed GPT‑6‑sol at only 1–3% of its cost.

By Haozheng Liu, Zhaoqian Xue, Zhongyin Wang, Junting Mu, Zihui Ma, Shuya Feng, Lingyao Li
arXiv Computation and Language
Oct 28

Teach reality, not ethics: a drop-in continued-pretraining corpus installs coupled human-biosphere-AI welfare dispositions across model families

The paper introduces a continued‑pretraining corpus called "coupled welfare" that embeds ecological reasoning into large language models. After a single QLoRA run, the corpus was applied to five open models across three architectures, resulting in a dramatic reduction of harmful decisions under high‑pressure scenarios while leaving general performance unchanged. The study also demonstrates that the improvement is content‑specific, as a neutral control corpus did not yield similar gains, and that the models remain robust against adversarial attempts to justify environmentally destructive actions.

By Trent R. Northen, Mingxun Wang
arXiv Computation and Language
Oct 28

Metacognitive Decoding: Efficient Inference for Diffusion Language Agents

The paper introduces a metacognitive controller that optimizes diffusion language models for agent tasks by deciding when to finalize tokens and how far to generate, thereby reducing computation. The lightweight plug‑in is trained on agent trajectories to balance correctness and generation cost. Experiments on five benchmarks show an average 20.6% reduction in inference FLOPs while keeping task accuracy competitive.

By Mingyu Zhao, Shuhang Lin, Ligong Han, Xiaojing Yu, Zhenting Wang, Yifei Pang, Yang Zhou, Dimitris N. Metaxas
arXiv Computation and Language
Oct 28

Beyond Update Counts: A Control Map for Evaluating Test-Time Model Updates

The paper introduces the Control Map, an evaluation framework for test‑time model updates that records the specific updates made before each answer and separately assesses acquisition, retention, locality, and performance in new settings. It demonstrates that update scheduling can dramatically affect accuracy, that learning an edit does not guarantee lasting or contained changes, and that no single editor consistently outperforms others across models and benchmarks. The authors argue that evaluations should report update histories and assess these dimensions separately.

By Zehao Zhao, Shangjian Yin, Rui Liu, Chongyang Bai, Mark Ibrahim, Kavosh Asadi, Yuchen Lu, Hang Cui, Hamed Firooz
arXiv Computation and Language
Oct 28

HiLo-RD: Cross-Budget Self-Distillation for Masked Diffusion Language Models

HiLo-RD (High-to-Low Budget Representation Distillation) is a method designed to enhance masked diffusion language models (MDLMs) when operating under limited generation budgets. It pairs large-window teacher states with cropped small-window views and aligns their hidden representations at committed positions, guiding both token prediction and commitment order without an external teacher. Experiments on mathematics benchmarks and code generation show that HiLo-RD raises low-budget accuracy—e.g., a 9.4‑point gain on GSM8K—and reduces response length by about 25%.

By Xiang Li, Chengting Yu, Jungang Li, Yilun Zheng, Junzhuo Li, Xingyao Zhang, Wenbo Su, Xuming Hu
arXiv Computation and Language
Oct 28

Mining the Nineteenth Century for the Thirteenth: Register-Scale Forced Alignment and HTR Bootstrapping via Legacy Regesta and Antiquarian Editions

The study evaluates how nineteenth‑century scholarship—specifically Pressutti’s regesta and partial editions—can aid handwritten text recognition (HTR) of Pope Honorius III’s registers beyond the limited modern expert transcriptions available for regnal years 1‑2. By aligning legacy layers with modern source lines, the authors demonstrate that projection onto curial spelling with replay reduces word‑error rate by 1.50 points, that a legacy lexicon covers 82.2 % of out‑of‑vocabulary words and flags errors with 66.7 % precision, and that a distilled medieval NER model achieves an F1 of 82.1 % on a validated benchmark. Additionally, Pressutti’s regesta and related topological data recover 73.9 % of entry starts and suggest a character‑error rate of 7.80 % beyond folio 203, indicating that antiquarian scholarship can steer, check, and segment HTR rather than replace modern transcription.

By Tamas Kovacs, Daniel Luger
arXiv Computation and Language
Oct 28

Coverage Under Contradiction: A Benchmark for Negation-Aware Retrieval

Coverage Under Contradiction: A Benchmark for Negation-Aware Retrieval introduces CoverageBench-Negate, a new benchmark that augments the existing CoverageBench with controlled contradictions. For each evidence nugget, a negated counterpart is created and its supporting documents are edited to support the negation while preserving context, resulting in about 63,000 nugget‑document pairs across six datasets. The benchmark includes rich construction metadata, LLM judgments, and a human audit, and baseline experiments show BM25 can distinguish original from edited evidence better than dense retrievers.

By Pasha Abrishamchian, Eugene Yang, Andrew Yates, Nazli Goharian, Ophir Frieder