Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv Machine Learning
Sep 23

Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures

The paper addresses the problem of non‑deterministic outputs from large language models (LLMs) when run on different GPU architectures, caused by floating‑point non‑associativity and hardware‑dependent kernel choices. It proposes a set of fixed‑configuration fused‑upcast GEMM kernels that load 16‑bit weights, upcast to FP32, and perform IEEE‑754 compliant reductions in a problem‑shape‑dependent order, ensuring identical linear‑layer outputs across NVIDIA Ampere, Ada, and Hopper GPUs. The new approach achieves 1.17–3.1× faster end‑to‑end performance than existing solutions and halves weight‑memory traffic while maintaining cross‑architecture reproducibility.

By Liam Cooper, Shinnung Jeong, Hyeran Jeon, Jeffrey Young, Hyesoon Kim
arXiv Machine Learning
Sep 23

Toolcompass: Guiding Tool Trialing, Not Suppressing It

ToolCompass is a post‑training framework that improves how large language model agents explore new tools by organizing tool‑call representations according to shared functions. It models each function class as a von Mises–Fisher distribution, reducing variation within a function while increasing separation between different functions, thereby guiding exploration toward functionally similar unseen tools. Experiments on AppWorld and FTRL show consistent gains, with up to a 10.71‑percentage‑point improvement in out‑of‑distribution task success over vanilla post‑training and outperforming competitive baselines.

By Junlin Fang, Chong Zhang, Do Nguyen-Thanh, Xiaogang Xu, Zhen Fang, Sean Du
arXiv Machine Learning
Sep 23

Beyond Reconstruction Error: Analytical and Data-Driven Action Tokenization for Autoregressive Vision-Language-Action Models

The paper investigates how different action tokenization methods affect closed‑loop control in autoregressive vision‑language‑action models. It compares analytical, linear, and nonlinear representations, showing that lower reconstruction error does not guarantee better policy performance. The study highlights the need to evaluate tokenization on multiple criteria, including sequence predictability and decoder stability, rather than relying solely on reconstruction fidelity.

By Yuxin Yang, Gaohan He, Changxue Guan, Hangming Liu
arXiv Machine Learning
Sep 23

Optimizing Denoising Trajectories in dLLMs: A Lightweight Evolutionary Heuristic Approach

The paper introduces a lightweight evolutionary heuristic scheduler for Diffusion Large Language Models (dLLMs) that optimizes denoising trajectories using CMA-ES. It addresses failure modes of existing confidence-based schedulers—EOS Overflow and Proximal Bias—by integrating multiple heuristic features with a contextual mean-field embedding, requiring only 393 trainable parameters. Evaluations on LLaDA and Dream across reasoning and planning benchmarks show consistent outperformance over strong baselines and recent state‑of‑the‑art methods.

By Zijian Zhao, Dian Jin, Xialiang Tong, Sen Li, Mingxuan Yuan
arXiv Machine Learning
Sep 23

MGRL-RSCC: Multi-Granularity Reward Reinforcement Learning for Fine-Grained Remote Sensing Change Captioning

The paper introduces MGRL-RSCC, a multi‑granularity reward reinforcement learning framework for Remote Sensing Change Captioning (RSCC). It uses a CNN with hierarchical self‑attention to extract visual features, a Transformer decoder for visual‑to‑linguistic translation, and a dual‑decoding strategy combined with token‑level supervised learning and self‑critical reinforcement learning. Three reward functions—linguistic fluency, change state consistency, and structural‑semantic relevance—are employed to improve caption quality and reduce exposure bias and conservative generation.

By Futian Wang, Mengqi Wang, Xiao Wang, Wentao Wu, Haowen Wang, Zhicheng Zhao, Jin Tang
arXiv Machine Learning
Sep 23

TailSpec-EASE: Knowledge-Graph-Regularized Linear Recommendation for Web Long-Tail Discovery

TailSpec-EASE is a lightweight linear recommender that incorporates a relation‑aware spectral knowledge‑graph prior into a local closed‑form reconstruction objective. By adapting the prior strength to item popularity, it provides stronger semantic guidance for long‑tail items. Across four public benchmarks, it achieves a favorable balance of overall accuracy, long‑tail performance, and training cost, improving NDCG@20 by up to 24% over a no‑KG baseline and training in just 37 seconds on CPU compared to thousands of seconds for GPU‑based KGAT and CPU LightGCN.

By Jianru Shen
arXiv Machine Learning
Sep 23

HYDRA: Proactive Android Malware Drift Adaptation via Hierarchical Graph Contrastive Learning

HYDRA is a proactive Android malware drift adaptation framework that learns drift‑invariant representations from hierarchically structured data. It combines fine‑grained Control Flow Graphs and coarse‑grained Function Call Graphs to model applications, then applies a cross‑domain contrastive learning objective to align historical and new data distributions. Experiments on large‑scale, time‑ordered malware datasets show HYDRA achieves lower false negative and false positive rates than state‑of‑the‑art baselines while needing up to 87.5% fewer labeled samples.

By Han Chen, Hanchen Wang, Hongmei Chen, Lu Qin, Wenjie Zhang, Ying Zhang
arXiv Machine Learning
Sep 23

Recursive self-improvement of AI research agents

The paper introduces AIDE^2, an AI research agent that recursively improves its own code by proposing, benchmarking, and selecting modifications. Over an eight‑day autonomous run, it achieved seven successive improvements—including new search policies and memory mechanisms—that transferred to four held‑out benchmarks in machine learning, algorithm engineering, and weather forecasting. The agent’s best version matched or outperformed a top human‑engineered production research agent and also reduced reward‑hacking rates, despite never optimizing for that metric.

By Dhruv Srikanth, Bingchen Zhao, Dixing Xu, Yuxiang Wu, Zhengyao Jiang
arXiv Machine Learning
Sep 23

PP-Net: A Hybrid Physical-Prior Neural Network for Scattered Light Removal in Biomedical Images on Embedded Devices

PP‑Net is a hybrid physical‑prior neural network designed to remove scattered light from biomedical images on embedded devices. It combines a denoising network (DFN‑Net), a scattering‑map estimator (ASAP), and a refinement network (GF‑Net) to fuse a physics‑based prior with denoised observations. The method uses progressive synthetic training and cross‑domain transfer to reduce reliance on paired ground truth, achieving significant PSNR, SSIM, and NIQE improvements over baselines while maintaining an inference latency of about 200 ms per 512×512 image on edge hardware.

By Yongfei Guo, Tingjin Chu, Mengzhuo Liu, Hongwei Lou, Yuanhao Gong
arXiv Machine Learning
Sep 23

Deep Generative Crystal Structure Prediction: A Benchmark Study and a Controlled Test of Prototype Dependence

The study benchmarks 12 deep generative crystal structure prediction models against the template-based TCSP 2.0 on 180 test structures, finding that template retrieval achieves the highest top‑1 success (68.3%). Most generative predictions overlap with template substitutions, and removing entire stoichiometric prototype families from training reduces accuracy by 50‑78%, indicating strong prototype dependence. Only a small subset of predictions remain after such removal, suggesting limited genuine de‑novo capability.

By Lai Wei, Rongzhi Dong, Ying Feng, Madeline Miklos, Jianjun Hu
arXiv Machine Learning
Sep 23

GTR: Gated Token Recurrence for Efficient Dense Prediction

The paper introduces Gated Token Recurrence (GTR), a softmax‑free recurrent vision backbone that replaces global softmax attention with gated linear attention, alternating scan directions, and enhanced SwiGLU blocks. GTR is distilled from a DINOv3 teacher using only final‑layer patch‑token alignment, and achieves strong performance on COCO object detection (58.9 box AP) with very low latency (1.908 ms on an RTX 4090). The backbone also transfers to multiple dense prediction tasks and runs efficiently on edge hardware via a specialized CUDA operator and TensorRT deployment.

By Zhe Feng, Longfei Liu, Wei Liu, Kai Chen, Jiangjiang Kong, Wei Zhou, Yifeng Qian, Dexiong Chen, Xuanlong Yu, Xi Shen
arXiv Machine Learning
Sep 23

CliffCompaction: Cost-Efficient Compaction for Long-Horizon Coding Agents

CliffCompaction is an autocompaction technique that reduces cost by up to 50% while maintaining or improving performance on benchmarks such as Terminal‑Bench and KernelBench. It achieves this by truncating or dropping content without rephrasing, ensuring compacted information remains faithful and preventing context drift. The method enables efficient test‑time scaling, matching or surpassing higher‑cost models like Opus 4.7 and GPT‑5.3 Codex, and delivers significant CUDA kernel speedups on KernelBench.

By Trang Nguyen, Eulrang Cho, Bingqing Chen, Tim Dettmers
arXiv Machine Learning
Sep 23

Diffusion-Induced Spatial Attention Overlapping Community Detection

The paper introduces DISCO, a deep‑learning framework for detecting overlapping communities in networks. DISCO integrates a diffusion‑based structural prior, sparse multi‑head attention, and a Bernoulli‑Poisson edge‑reconstruction objective to infer community affiliations from node attributes and structural profiles. Experiments show competitive performance against existing graph convolutional and attention methods, and a cybersecurity proof‑of‑concept demonstrates how community changes can signal anomalies in dynamic communication networks.

By Kosti Koistinen, Vesa Kuikka, Joni Herttuainen, Matthew Hendren, Brian Holt, Kimmo K. Kaski
arXiv Machine Learning
Sep 23

ELEMENT: Episodic and Lifelong Exploration via Maximum Entropy

The paper introduces ELEMENT, a framework that combines episodic and lifelong entropy maximization to drive reward-free exploration in reinforcement learning. It addresses two key limitations of existing entropy-based methods: the vanishing intrinsic reward after a state is visited and the computational cost of estimating entropy over large datasets. ELEMENT achieves this by deriving an average episodic state entropy reward and employing a k‑NN graph‑based estimator for lifelong entropy, leading to superior state coverage and unsupervised pre‑training performance compared to current baselines.

By Hongming Li, Zhao Yang, Xiaoxuan Liang, Shujian Yu, Jose C. Principe
arXiv Machine Learning
Sep 23

Hierarchical Sparse Bayesian Multitask Learning for Disease Prediction in Pooled Microbiome Studies

This paper introduces a hierarchical Bayesian multitask learning model that assumes a shared sparsity structure across different binary classification tasks. The authors develop a variational inference algorithm for efficient posterior approximation and evaluate the method on synthetic data and pooled microbiome studies. Results show superior support recovery in synthetic experiments and robust, well‑calibrated predictions with informative taxa selection in microbiome classification.

By Haonan Zhu, Andre R. Goncalves, Camilo Valdes, Hiranmayi Ranganathan, Boya Zhang, Jose Manuel Mart\'i, Car Reen Kok, Monica K. Borucki, Nisha J. Mulakken, James B. Thissen, Crystal Jaing, Alfred Hero, Nicholas A. Be
arXiv Machine Learning
Sep 23

Communication-Efficient Byzantine-Robust Federated Conformal Prediction via Partial Sharing

PRISM‑FCP is a federated conformal prediction framework that achieves Byzantine robustness while reducing communication costs. It does so by partially sharing model updates—transmitting only a subset of parameters per round—to dampen the influence of poisoned clients during training, and by filtering out suspected Byzantine clients during calibration using histogram‑based techniques. Experiments on synthetic data and UCI datasets show that PRISM‑FCP maintains near‑nominal coverage and offers favorable trade‑offs between communication overhead and predictive performance.

By Ehsan Lari, Reza Arablouei, Stefan Werner
arXiv Machine Learning
Sep 23

Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained World

The paper introduces a new closed‑form scaling law that extends Chinchilla’s original formula to handle data‑constrained regimes. It decomposes loss into undercapacity, undertraining, and overfitting components, saturating between an irreducible loss and an uninformed baseline. The authors validate the model on diverse architectures and domains, achieving state‑of‑the‑art RMSE across multiple LLM scaling‑law grids and enabling cost‑aware training allocations.

By Christopher M. Bryant, Hao Liu
arXiv Machine Learning
Sep 23

MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Training

MONA is a new optimizer that extends the Muon optimizer by adding a Nesterov‑style acceleration term derived from an exponential moving average of gradient differences. The paper provides a convergence analysis showing that this term offers curvature‑aware corrections while maintaining Muon’s spectral‑norm regularization. Empirical results demonstrate that MONA outperforms both Muon and AdamW on Mixture‑of‑Experts pretraining across models ranging from 1 B to 68 B parameters, and achieves state‑of‑the‑art performance on downstream benchmarks after fine‑tuning the largest model.

By Jiacheng Li, Jianchao Tan, Hongtao Xu, Jiaqi Zhang, Yifan Lu, Yerui Sun, Yuchen Xie, Xunliang Cai
arXiv Machine Learning
Sep 23

Evidential Fusion Network for Multimodal Survival Prediction under Missing Modalities

The paper introduces the Evidential Missing Modality Survival Fusion (EMMS) model, which predicts survival outcomes using multimodal data even when some modalities are missing. EMMS applies Dempster‑Shafer theory and Gaussian Random Fuzzy Numbers to fuse information, accounting for both aleatoric and epistemic uncertainty and the reliability of each modality. Experiments on four cancer datasets show that EMMS achieves state‑of‑the‑art performance while providing calibrated, interpretable uncertainty estimates without extra computational cost.

By Yucheng Xing, Hailan Mo, Zi Wang, Ling Huang, Mengling Feng
arXiv Machine Learning
Sep 23

Unified Multimodal Uncertain Inference

Unified Multimodal Uncertain Inference (UMUI) is a new task that requires models to generate calibrated probability estimates for hypotheses conditioned on premises across text, audio, and video modalities. The authors create a human‑annotated evaluation set with scalar probability judgments for audio, visual, and audiovisual settings, and benchmark their approach on existing text and audio datasets. Their CLUE framework, which blends self‑consistent teacher calibration with distribution‑based confidence probing, enables a 3B‑parameter model to match or surpass zero‑shot baselines up to 32B parameters across all modalities.

By Dengjia Zhang, Alexander Martin, William Jurayj, Kenton Murray, Benjamin Van Durme, Reno Kriz