Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,620 stories · RSS feed

arXiv AI
Sep 11

Hierarchical and Permutation-Invariant Feature Transformation Learning via Policy-Guided Embedding Search

The paper introduces a framework for learning feature transformations on tabular data that addresses three limitations of prior generative methods: neglect of hierarchical feature relationships, bias from order-sensitive embeddings, and reliance on gradient-based search. It combines a permutation‑invariant hierarchical module using self‑attention pooling to capture interactions across features, operations, and abstraction levels, with a policy‑guided multi‑objective reinforcement learning strategy that starts from strong seeds and optimizes both predictive accuracy and transformation efficiency. Experiments on diverse tabular benchmarks show the approach outperforms strong baselines, and the authors provide public code and data.

By Rui Liu, Tao Zhe, Yanyong Huang, Sankha Narayan Guria, Xiao Luo, Wei Fan, Yanjie Fu, Dongjie Wang
arXiv AI
Sep 11

RESCUE-BENCH: Towards Relation-Aware Multi-Party Emotional Support Conversation Systems

The paper introduces RESCUE-BENCH, a benchmark for relation-aware multi‑party emotional support conversation systems. It is built from real couple and family interview data, comprising 191 samples, 7,079 annotated turns, and 1,064.8 minutes of video, and defines six tasks that assess relational understanding and relation‑sensitive support. Experiments with ten large language models show that while they handle local emotional cues reasonably well, they struggle with tasks that require modeling interpersonal relations, such as predicting relation patterns, viewpoints, and support strategies.

By Haichuan Hu, Yang Xiao, Mingni Tang, Jiawen Duan, Quanjun Zhang, Congqing He, Hao Zhang, Jiashuo Wang, Johan F. Hoorn, Wenjie Li
arXiv Machine Learning
Sep 11

How Much Velocity Does Off-Ball Space Value Need? A Broadcast-Viewport Benchmark

The paper investigates how different velocity regimes affect broadcast‑viewport basketball analytics. Using a calibrated off‑screen imputation protocol, it compares four velocity settings—none, viewport‑legal observed, true‑for‑visible, and true‑for‑all—against a velocity‑aware ground truth across three analytical layers. Results show that velocity is largely useless for imputation, modestly useful for the control surface, and minimally useful for final team verdicts, with the visible channel providing the most benefit.

By Seongjin Choi
arXiv Machine Learning
Sep 11

Enabling Knowledge Graph Understanding at Scale with the EXplore Your Graphs ENgine (EXYGEN)

The paper introduces EXYGEN, a framework that enables conversational access to large knowledge graphs by combining VoID descriptions, ShEx schemas, retrieved triples, and example question‑query pairs in a retrieval‑augmented generation pipeline. On the SciQA benchmark, this approach achieves an exact‑match score of 0.419 without fine‑tuning any large language model, and shows that larger general‑purpose LLMs can outperform smaller code‑specialized ones when provided sufficient context. To scale metadata generation for very large KGs, the authors propose a predicate‑coverage‑aware parallel graph sampling strategy that preserves structural diversity, reduces runtime by over 80× on OpenCitations Meta and GESIS, and is the only tractable method for obtaining complete metadata on ORKG.

By Harshdeep Singh, Yurui Zhu, Giovanni Colavizza, Matteo Romanello
arXiv Machine Learning
Sep 11

Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs

The paper presents py‑kvcache, a new KV offload connector for vLLM that uses asynchronous direct I/O, bounded shared staging, and scheduler‑aware preloading to improve external KV caching performance on NVMe SSDs. Experiments across synthetic workloads, long‑context benchmarks, and production traces show that py‑kvcache can load 80k‑token prefixes 2.0× faster than LMCache, with preloading contributing an additional 1.34× speedup, and achieves overall performance within 4% of native vLLM KV Offload. The study highlights that cache effectiveness depends on transfer granularity, intermediate memory use, and scheduling timing rather than just device bandwidth, indicating that external KV caching should be considered a setup‑specific admission decision.

By Joseph Kanichai, Tiziano De Matteis, Animesh Trivedi
arXiv Computation and Language
Sep 11

Larger Context Window, Fewer Overcorrections: Optimizing Prompts and Batching for Minimal-Edit Grammatical Error Correction

The paper presents a prompt-based method for minimal-edit grammatical error correction (GEC) that reduces overcorrection in large language models (LLMs). It introduces taxonomy-based instructions, batch prompting to regularize overcorrection, and LLM-assisted prompt optimization, achieving an $F_{0.5}$ score of 78.32 on BEA-2019 with Gemini 3.1-Pro. This approach narrows the performance gap to fine-tuned models while avoiding their infrastructure demands.

By Kateryna Karpo, Artem Chernodub
arXiv Machine Learning
Sep 11

Sublinear Variational Optimization of Gaussian Mixture Models with Millions to Billions of Parameters

The paper introduces a highly efficient variational approximation for Gaussian Mixture Models (GMMs) with arbitrary covariances, integrated with mixtures of factor analyzers. This method reduces the per‑iteration runtime from ≠O(NCD^2) to a complexity that scales linearly with dimensionality D and sublinearly with the product NC. Experiments demonstrate sublinear scaling across the entire optimization, order‑of‑magnitude speed‑ups on large benchmarks, training of GMMs with over 10 billion parameters in under nine hours on a single CPU, and competitive zero‑shot image denoising performance.

By Sebastian Salwig, Till Kahlke, Florian Hirschberger, Dennis Forster, J\"org L\"ucke
arXiv Machine Learning
Sep 11

Building Supervision into Hebbian Plasticity through Spike Agreement

The paper introduces Supervised Spike Agreement-Dependent Plasticity (Supervised SADP), a gradient‑free Hebbian learning rule that embeds class labels directly into spike‑driven plasticity. SADP trains output neurons with a supervised Hebbian rule and hidden neurons by measuring Cohen’s kappa agreement with the correct‑class output spike train, optionally aggregating over temporal offsets (K‑shift). Across six benchmark and medical imaging datasets, SADP consistently outperforms reward‑modulated STDP, achieving higher accuracy (e.g., 86.46 % on MNIST) and faster training (up to 2.86× speedup).

By Gouri Lakshmi S, Athira Chandrasekharan, Harshit Kumar, Muhammed Sahad E, Bikas C Das, Saptarshi Bej
arXiv Computation and Language
Sep 11

CMNIE: An Information Extraction Benchmark for Chinese Military News

CMNIE is a new benchmark for extracting structured information from Chinese military news, covering event triggers, arguments, named entities, and entity relations under a unified schema. The dataset contains 13,000 manually annotated instances with 7 event types, 10 argument roles, 7 entity types, and 8 relation types. Experiments show that current supervised models, zero‑shot LLMs, and fine‑tuned LLMs struggle with relation extraction and exact span matching, highlighting the challenge of joint structured extraction in this domain.

By Yan Yu, Mengna Zhu, Zhenyu Song, Hao Yang, Haiwen Chen, Mao Wang
arXiv Machine Learning
Sep 11

GEOSTEER: Geodesic Optimization for Activation Steering in Large Language Models

GeoSteer introduces a geometry-aware, optimization-based approach to norm-preserving activation steering in large language models. By formulating steering as a Riemannian optimization problem, it updates activations through a sequence of small geodesic steps guided by a learned nonlinear objective, avoiding fixed steering directions. Experiments on TruthfulQA, RealToxicityPrompts, and UltraFeedback show that GeoSteer consistently outperforms existing activation steering baselines, offering smoother, more stable, and more consistent steering behavior.

By Xuan Cuong Ngo, Hao Vo, Ngan Le
arXiv AI
Sep 11

Improving 5G AI-RAN MCS Selection by Predicting Retransmissions

The paper introduces NOSTRAdAMUS, a predictive link‑adaptation framework for 5G NR that forecasts retransmissions in the next radio frame using recent HARQ history and adjusts the Modulation and Coding Scheme accordingly. Gradient Boosting models achieve 82.9% overall accuracy, with high‑confidence predictions correct 94.2% of the time and a 5.5 µs inference latency. Evaluated OTA on the X5G testbed and various channel emulators, the approach boosts goodput by up to 71.5% and cuts retransmissions by up to 71.8% without retraining across diverse scenarios.

By Tamerlan Aghayev, Maxime Elkael, Michele Polese, Reshma Prasad, Salvatore D'Oro, Yunseong Lee, Koichiro Furueda, Tommaso Melodia
arXiv AI
Sep 11

KernelGenBench: Can LLMs and Agents Write Efficient Kernels Across Operator Sources and Hardware Platforms?

KernelGenBench is a unified benchmark that evaluates large language models and agentic systems for generating efficient Triton kernels across diverse operator sources and hardware platforms. It covers 210 operators from PyTorch ATen, vLLM, and cuBLAS, and tests a 110‑operator subset on six different chips, consuming over 15 billion tokens in evaluation. The study finds that no single method dominates across all sources and platforms, with significant variations in correctness and performance depending on the operator source and hardware, and that agentic approaches require millions of tokens per successful operator.

By Peiyu Zang, Jian Tao, Jialing Zhang, Yichen Yuan, Wentao Zhang, Guang Liu, Yonghua Lin
arXiv AI
Sep 11

Instance-Aware Algorithm Selection for Maximum Clique via a Dual-Channel Graph Neural Architecture

The paper presents a dual‑channel graph neural architecture for selecting the best exact solver for the Maximum Clique Problem (MCP). It combines a Graph Attention Network that captures local neighborhood patterns with a Multilayer Perceptron that models global statistical descriptors, trained on a benchmark of four state‑of‑the‑art solvers evaluated across diverse graph instances. The resulting model achieves 90.43 % test accuracy, outperforming classical baselines and the single‑best solver.

By Xiang Li, Shanshan Wang, Chenglong Xiao
arXiv Machine Learning
Sep 11

RiVaT-Fuse: Reliability-Calibrated Variational Tensor Fusion for Multimodal Prediction under Modality Uncertainty

RiVaT‑Fuse introduces a reliability‑calibrated variational tensor fusion framework for multimodal image‑metadata prediction, treating fusion as a sample‑wise latent‑state estimation rather than simple aggregation. It replaces scalar modality confidence with matrix‑valued trust geometry, decomposes interactions into additive, multiplicative, and relational components, and couples the latent state with conditional robustness and structured multi‑task prediction. On an image‑level benchmark, RiVaT‑Fuse outperforms direct representation‑level baselines and improves probability and label stability under perturbation.

By Yingfan Xu, Tieming Liu, Ye Liang, Taiping Liu
arXiv Machine Learning
Sep 11

HERALD: High-Fidelity Exemplar Retrieval with Adaptive Landmark Distillation for Heterophily-Aware Graph Condensation

HERALD is a new gradient‑free graph condensation framework that adapts node scoring and feature selection to a graph’s heterophily level. It selects features using a joint Fisher‑discriminability and activation‑density criterion, and scores nodes with a weighted combination of prototype representativeness, decision‑boundary proximity, and Local Intrinsic Dimensionality, where the weights depend on the heterophily ratio. The selected nodes are assembled into a condensed subgraph via score‑ordered BFS expansion, Personalized PageRank pruning, and class rebalancing, achieving comparable storage to BONSAI and outperforming state‑of‑the‑art condensers on heterophilic graphs while remaining competitive on homophilic ones across multiple GNN architectures.

By Sujan Chakraborty, Priyanka Saha, Saptarshi Bej
arXiv Computation and Language
Sep 11

Overview of the NLPCC 2026 Shared Task 11: Agent-Based Experiment Reproduction from Scientific Papers

The article introduces AgentActionBench, a benchmark designed to evaluate agent-based experiment reproduction across machine learning and AI4Science papers. It employs an MCP-based Action Recorder to capture agents’ behavior during reproduction and assesses the resulting traces against paper-specific rubrics. The benchmark includes 150 papers, with a human-annotated subset and model-assisted augmentation expanding it to over 10,000 rubric items, revealing that current systems face execution bottlenecks but that model-generated rubrics correlate strongly with human judgments.

By Hanhua Hong, Yizhi Li, Luu Gia Huy, Jian Yang, Ming Zhou, Chenghua Lin
arXiv Computation and Language
Sep 11

TransClean: A Benchmark for Detecting and Extracting Clean Translations from Large Language Model Outputs

TransClean introduces a benchmark for identifying and removing translation noise—unwanted text such as language labels, explanations, or bilingual repetitions—from large language model (LLM) outputs. The authors analyzed 790,000 translations from 12 LLMs across 22 language pairs, cataloguing 12 common noise patterns and creating 9,900 noisy‑clean pairs (8,800 synthetic, 1,100 authentic). They evaluated two extraction methods—a span‑based approach using quality estimation models and an LLM‑prompted method—demonstrating the first systematic framework to assess and improve translation cleanliness.

By Shenbin Qian, Yves Scherrer
arXiv Computation and Language
Sep 11

RAG-Safety-Bench: Reliable Evaluation of Retrieval-Augmented LLM Safety

RAG-Safety-Bench is a benchmark designed to evaluate how retrieval-augmented generation (RAG) affects the safety of large language models (LLMs). It isolates safety impacts by testing four conditions: non-RAG, RAG with an oracle document, RAG with related but non-answer documents, and RAG with random safe documents. Results on five open-source LLMs reveal an inverse relationship between benign and unsafe capabilities, show that baseline safety guardrails do not guarantee safety in RAG, and confirm that even benign documents can trigger unsafe generation.

By Adithiyan Rajan Indira Saravanan, Kathleen C. Fraser
arXiv Computation and Language
Sep 11

What Language is This? Ask Your Tokenizer

The paper introduces UniLID, a lightweight language identification method that uses the UnigramLM tokenization algorithm to predict a string’s language by evaluating which language’s unigram distribution best explains the text. UniLID is data‑ and compute‑efficient, allows incremental addition of new languages without retraining, and can be integrated into existing tokenization pipelines. Experiments show competitive performance against baselines such as fasttext, GlotLID‑M, and CLD3, achieving 69% accuracy with five labeled samples per language and 89% with 25, and delivering significant gains on fine‑grained dialect identification.

By Clara Meister, Ahmetcan Yavuz, Pietro Lesci, Tiago Pimentel