Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv Machine Learning
Sep 22

PRQuant: Permutation Residual Quantization for Low-Overhead Inference

PRQuant introduces a training‑free, low‑overhead method for low‑bit quantization of linear layers by permuting input channels that cause the largest quantization error into contiguous tail blocks and precomputing residual weight sub‑tensors. The approach eliminates the need for online gathering during inference, converting scattered residual compensation into a regular tail‑augmented GEMM and thereby reducing latency. Experiments show that PRQuant lowers down‑projection reconstruction error and outperforms standard MXFP4 and other post‑training quantization baselines on five downstream benchmarks, improving accuracy by up to 1.24 points on Qwen3‑4B‑Instruct‑2507.

By Peiran Wang, Anqi Wang, Jiaying Zhao, Huiwen Yang, Zhenyu Ming, Rongqian Wang, Yiwu Yao, Kun Tian, Xin Yao, Gong Zhang, Fan Yang, Zhongyi Huang
arXiv Machine Learning
Sep 22

Weak Ties, Strong Signals: Efficient Training Data Detection in Diffusion LLMs via Independent Token Sampling

The paper introduces Independent Token Sampling (ITS), a query‑efficient method for detecting memorized training data in diffusion large language models (dLLMs). ITS selects token sets with weak internal dependency by approximating cumulative conditional mutual information using an attention‑derived pairwise dependency proxy and promotes diversity across sampling rounds. Experiments show ITS outperforms existing baselines, improving AUC by 0.18 on the ArXiv dataset while remaining effective under limited query budgets.

By Hongyao Yu, Tianqu Zhuang, Ziyuan Xu, Hao Fang, Jiaxin Hong, Bin Chen, Shu-Tao Xia
arXiv Machine Learning
Sep 22

User-Level Handover Decision Making Based on Machine Learning Approaches

The paper compares various machine‑learning techniques for user‑level handover decisions in challenging propagation environments, using ns‑3 simulations. Classical methods such as KNN, SVM, and neural networks, along with fuzzy logic systems and boosting machines, are evaluated for classifying the optimal handover target and estimating download time. Results show that SVM and MLP perform best for target classification, while LightGBM yields the lowest error for download time estimation, with fuzzy SOFL offering comparable accuracy at lower processing time.

By Jo\~ao Lima, Alvaro Medeiros, Eduardo Aguiar, Vicente Angelo de Sousa Junior, Tarciana Guerra
arXiv Machine Learning
Sep 22

CurvFlow-DTA: dual-graph discrete Ricci curvature flow for drug--target affinity prediction

CurvFlow-DTA introduces a dual-graph discrete Ricci curvature flow framework for drug–target affinity prediction, replacing static curvature with weighted Forman curvature flow on both drug and protein residue–residue contact graphs. The method precomputes a label‑independent flow trajectory for each entity and uses a pair‑conditioned selector to guide a dual‑branch Flow‑GINE, leveraging frozen ESM‑2 residue representations. Experiments on Davis and KIBA datasets show significant improvements over the Ricci‑GraphDTA baseline, with reductions in mean squared error of up to 19.9% in warm‑start and 27.4% in cold‑start settings, and higher concordance indices across benchmarks.

By Jicheng Ma, Yunyan Yang, Juan Zhao, Liang Zhao
arXiv Machine Learning
Sep 22

Simpler Methods Work Better for L1 Penalized Logistic Models and Large Datasets

Linear models with an $L_1$-norm penalty are still the leading approach for high‑dimensional tasks, yet many existing solvers are slow, ineffective, and hard to parallelise, making them unsuitable for large industry‑scale corpora. The paper evaluates several recent state‑of‑the‑art methods and shows that older techniques outperform them in general use. It also demonstrates that a simple baseline—LBFGS applied to a sub‑gradient with minor tweaks—yields strong performance and is easier to support and scale in production.

By Edward Raff, James Holt
arXiv Machine Learning
Sep 22

Multilingual Safety Signals Are Multi-Layered: Filtering Safety-Degrading Data for Safer LLMs

The paper introduces MMSAFE, a multi-layer framework designed to identify safety-degrading data in multilingual large language models. It shows that safety signals are distributed across multiple layers and only partially shared across languages, unlike the single-layer assumption used in monolingual settings. Experiments demonstrate that MMSAFE reduces harmful-response rates by 60% compared to random filtering and outperforms the best single-layer baseline across various models, languages, and safety benchmarks.

By Jiakun Li, Guowei Song, Sijia Li, Xingwei He, Hongzheng Chai, Yuan Yuan
arXiv Machine Learning
Sep 22

Do LLMs Choose Like Humans? Using Cognitive Theory to Evaluate LLM Decision-Making

The paper investigates whether large language models (LLMs) make decisions in ways that mirror human cognition. Using a new 140,000-trial product choice benchmark, the authors test 12 open‑source and commercial LLMs to see if their context sensitivity aligns with a cognitive economic theory that relies on problem categorization and attention allocation. While context prompts human‑like shifts in choice and problem categorization, it does not consistently reweight attention between features such as price and quality, and neither scaling nor chain‑of‑thought reasoning produces human‑like behavior. The findings indicate that LLM decision mechanisms differ from those of humans.

By Johnathan Sun, Andrei Shleifer, Yonatan Belinkov
arXiv Machine Learning
Sep 22

From Trait Vectors to Circuits: Tracing Refusal and Sycophancy Through Language Models

The paper investigates how steering a language model along a trait vector—specifically for refusal and sycophancy—affects its internal computation. By splitting the model’s computation into a reconstruction circuit before the trait vector and a transmission circuit after it, the authors find that for refusal both circuits are compact and faithful, with the transmission circuit alone largely restoring the lost refusal signal. For sycophancy, the transmission circuit remains compact but the reconstruction circuit is broader and only partially faithful, indicating that the circuits used for steering are not necessarily the same as those that generate the behavior.

By Oscar Mir\'o L\'opez-Feliu, Maya Ozbayoglu
arXiv Machine Learning
Sep 22

Causal Localization of the Refusal Direction in Audio Language Models

The study investigates where a large audio language model (LALM) derives its refusal responses to harmful spoken requests. By applying causal interventions at the audio-to-language-model interface and within the language-model residual layers, the authors find that the primary influence on the refusal margin comes from mid-to-late layers of the text language model rather than the audio front end. Ablations of the audio interface have minimal effect, while zeroing the encoder output still reduces the margin, indicating the audio pathway remains active but is not the main source of refusal decisions.

By Leonardo Haw-Yang Foo, Hung-yi Lee
arXiv Machine Learning
Sep 22

Towards Robust Classroom Attendance: A Comprehensive Evaluation of Face Detection and Recognition Models

The paper introduces the Visage Face dataset, containing 16,234 face samples captured in varied classroom conditions such as different angles, lighting, expressions, and partial occlusions. Using a YOLO-based detector, the authors evaluate seven state‑of‑the‑art face recognition models across thirteen configurations, finding that FaceLiVTv2‑M achieves the highest performance with 99.75% Top‑1/Top‑5 accuracy and a 6.459 ms inference time. These findings demonstrate that the Visage Face dataset provides a realistic and challenging benchmark for classroom attendance systems.

By Himani Trivedi, Hiren Patel, Ridham Patel, Krutika Patel, Nancy Patel
arXiv Machine Learning
Sep 22

TriFleetRCA: On-Premise LLM Root Cause Analysis for Kubernetes

TriFleetRCA is an on‑premise pipeline that performs root‑cause analysis for Kubernetes using a single GPU. It gathers evidence at pod, namespace, or cluster scope, deduplicates and ranks it with BM25, filters runbooks through an ingest guard, and returns a root cause with supporting evidence lines. In a live cluster with four injected faults, the system achieved hit rates of 0.85–0.95 across scopes, improved accuracy with deduplication, and demonstrated robust defense against poisoned runbooks.

By Rohit Patel, Susil Kumar Mohanty, Jeenal Chaudhary
arXiv Machine Learning
Sep 22

ORION-CMR: On-scanner Reporting with Integrated Foundation Model for End-to-End Cardiac MRI Analysis and Interpretation

ORION‑CMR is a scanner‑native, end‑to‑end foundation model for cardiac MRI that performs sequence classification, ventricular function assessment, LGE detection, disease classification, and generates reports in about 90 seconds. Trained on 12.9 million images, it outperformed supervised baselines and a prior CMR foundation model, achieving state‑of‑the‑art LGE classification and scar segmentation. In a multi‑vendor clinical cohort, it reached an AUC of 0.96 for normal‑vs‑abnormal detection and 0.88 for multiclass disease classification, with generated reports agreeing 81.4% with expert interpretation.

By Omer Burak Demirel, Kelly K. Horst, Alessio Perazzolo, Elisa Bruno, Kenan Kaya, Rongzhen Ouyang, Enas Ahmed, Jouke Smink, Spencer L. Waddle, Zainudeen Kallumpurath, Tzu Cheng Chao, Dinghui Wang, Steve G. Langer, Timothy L. Kline, Panagiotis Korfiatis, Jacinta Browne, Ivana Isgum, Tim Leiner
arXiv Machine Learning
Sep 22

Tackling Feature-Classifier Mismatch in Federated Learning via Prompt-Driven Feature Transformation

The paper introduces FedPFT, a federated learning framework that tackles the feature‑classifier mismatch problem by using personalized prompts processed through a shared self‑attention transformation module. Unlike prior methods that either degrade the feature extractor or address the mismatch only after training, FedPFT aligns local features with the global classifier during training, improving aggregation and model performance. Experiments demonstrate that FedPFT surpasses state‑of‑the‑art methods by up to 5.07%, and gains up to 7.08% when combined with collaborative contrastive learning.

By Xinghao Wu, Xuefeng Liu, Jianwei Niu, Guogang Zhu, Mingjia Shi, Shaojie Tang, Jing Yuan
arXiv Computation and Language
Sep 22

Used, Mentioned, or Condemned? A Controlled Contrast-Set Diagnostic for the Use-Mention Distinction in Code-Mixed Hinglish Misogyny Detection

The paper introduces a diagnostic tool for distinguishing the use of misogynistic slurs from their mention in counter‑speech within code‑mixed Hinglish. It identifies evaluation artifacts in existing corpora, releases a 416‑item minimal‑pair contrast set that decorrelates slur presence and gendered register from labels, and proposes a pair‑consistency metric to assess model performance. Experiments show that even strong baselines struggle to consistently label counter‑speech pairs, while a large language model achieves perfect scores, indicating the benchmark measures genuine capability rather than exploitation of artifacts.

By Ashanvi Yadav, Shubham Bhardwaj
arXiv Computation and Language
Sep 22

BudgetMem: Training-Free Selective Memory for Cost-Efficient Long-Context Processing in Language Models

arXiv:2511. 04919v3 Announce Type: replace Abstract: Processing long documents with large language models (LLMs) is expensive: a single query over a 100K-token document can cost from tens of cents to over a dollar in API fees, depending on the model, and memory grows linearly with context length.

By Chandra Vamsi Krishna Alla, Harish Naidu Gaddam, Manohar Kommi, Sheikh Nazib Ahmed