Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,906 stories · RSS feed

arXiv Machine Learning

Jul 20

QUADS: Stabilizing NVFP4 Reinforcement Learning for MoE via QUantization-error Alignment across Dual Sides

arXiv:2607. 15810v1 Announce Type: new Abstract: Rollout generation is a major bottleneck in Reinforcement Learning (RL) for Mixture-of-Experts (MoE) Large Language Models, motivating low-precision rollout acceleration such as FP8.

By Zhengyang Zhuge, Hao Yu, Xin Wang, Zheng Li, Yizhong Cao, Dayiheng Liu, Jianwei Zhang

llms reinforcement-learning efficiency benchmarks safety

arXiv Machine Learning

Jul 20

A Benchmark for Electrical Load Forecasting Across Grid Levels: Time-Series Transformers Outperform Established Methods

arXiv:2607. 15705v1 Announce Type: new Abstract: Accurate load forecasting at multiple grid levels is essential for future smart grids, ranging from aggregated control area forecasts for balancing supply and demand to forecasts of individual end-consumer loads for demand-side management and energy management systems.

By Matthias Hertel, Sebastian P\"utz, Jonathan Kolar, Benjamin Sch\"afer, Ralf Mikut, Veit Hagenmeyer

llms benchmarks

arXiv Machine Learning

Jul 20

Learning Faster without Deeper Networks: A*-Inspired Batch Selection for Efficient CNN Training

arXiv:2607. 15745v1 Announce Type: new Abstract: Common practice when training Convolutional Neural Networks (CNNs) is to use randomly shuffled mini-batches.

By Anxhelo Shehu, Enes Stastoli, Arben Cela

benchmarks

arXiv Machine Learning

Jul 20

LLM4EHR: Aligning Clinical Time Series with Medical Event Sequences via Large Language Models

arXiv:2607. 15447v1 Announce Type: new Abstract: Recent research in clinical machine learning, focusing on outcome predictions in intensive care unit (ICU), has shifted from bespoke supervised models to foundation models, utilising modern representation learning methods.

By Jingteng Li, Alexander Capstick, Louise Rigny, Iona Biggart, Neil J Sebire, Payam Barnaghi

llms rag benchmarks

arXiv Machine Learning

Jul 20

A Transportable Threshold-Based Framework for Interpretable Classification of Medical Data

arXiv:2607. 15394v1 Announce Type: new Abstract: Black-box models limit the adoption of artificial intelligence in medicine due to their lack of interpretability and reproducibility.

By Antony Garcia, Adrian Noriega, Gabrielle Britton, Xinming Huang

benchmarks safety

arXiv AI

Jul 20

PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning

arXiv:2601. 13020v2 Announce Type: replace-cross Abstract: Continual instruction tuning (CIT) requires multimodal large language models (MLLMs) to adapt to a stream of tasks without forgetting prior capabilities.

By Zhiyan Hou, Haiyun Guo, Haokai Ma, Yandu Sun, Yonghui Yang, Jinqiao Wang

llms fine-tuning multimodal benchmarks

arXiv AI

Jul 20

Unsupervised Deep Learning for Inverse Problems in Computed Tomography

arXiv:2508. 05321v4 Announce Type: replace-cross Abstract: Assume you encounter an inverse problem that shall be solved for a large number of data, but no ground-truth data is available.

By Laura Hellwege, Johann Christopher Engster, Moritz Schaar, Thorsten M. Buzug, Maik Stille

benchmarks

arXiv AI

Jul 20

Latent Fusion Jailbreak: Blending Harmful and Harmless Representations to Elicit Unsafe LLM Outputs

arXiv:2508. 10029v3 Announce Type: replace-cross Abstract: Safety-aligned large language models can still be manipulated through white-box interventions that modify their internal representations.

By Wenpeng Xing, Bohan Yang, Mohan Li, Chunqiang Hu, Haitao Xu, Ningyu Zhang, Bo Lin, Meng Han

llms multimodal benchmarks safety

arXiv AI

Jul 20

Neuro-Symbolic AI for LEED compliance: Document-Centric Benchmarking, Deterministic Numeric Checking, and When Multimodal Hurts

arXiv:2607. 15647v1 Announce Type: new Abstract: LEED v4.

By Aritro De (The University of Texas at Austin), Juliana Felkner (The University of Texas at Austin)

llms multimodal benchmarks

arXiv AI

Jul 20

Beyond a Joke: Multi-Angle Reasoning for Detecting and Explaining Harmful Humor in Memes

arXiv:2607. 15442v1 Announce Type: new Abstract: Internet memes intertwine visual cues, textual content, and cultural context, making them particularly challenging to interpret in scenarios where humor, sarcasm, and harmful intent coexist.

By Shanhong Liu, Pai Chet Ng, De Wen Soh, Malika Meghjani, Konstantinos N. Plataniotis

llms multimodal benchmarks safety

arXiv Machine Learning

Jul 20

Improving Backward Conformal Prediction via Non-Conformity Score Transformation

arXiv:2602. 01733v3 Announce Type: replace-cross Abstract: Conformal Prediction (CP) provides a statistical framework for uncertainty quantification that constructs prediction sets with coverage guarantees.

By Junxian Liu, Hao Zeng, Hongxin Wei

benchmarks

arXiv Machine Learning

Jul 20

ASK-NN: An Asymmetric Nearest-Neighbor Test that detects Distribution Drifts in Natural Language

arXiv:2607. 15607v1 Announce Type: new Abstract: Hallucinations and artificial text in LLM-generated outputs often appear as distributional deviations between prompt and response hidden-state distributions.

By Sergey Zakharov, Rodion Oblovatny, Alexey Zaytsev

llms benchmarks safety

arXiv AI

Jul 20

Spatial Normalization for Cross-Domain Retinal Layer Segmentation in Optical Coherence Tomography

arXiv:2607. 16065v1 Announce Type: cross Abstract: Retinal layer segmentation in Optical Coherence Tomography (OCT) is a fundamental step for extracting quantitative biomarkers of retinal structure.

By Iker Moran-Cavero, Monica Hernandez, Elvira Mayordomo, Naiara Artiaga, Beatriz Pardi\~nas, Beatriz Cordon, Elena Garcia-Martin

computer-vision benchmarks

arXiv AI

Jul 20

NexForge: Scaling Agent Capabilities through Requirement-Driven Task Synthesis for LLMs

arXiv:2607. 14186v2 Announce Type: replace-cross Abstract: Synthesizing training data to scale agent capabilities in LLM post-training is bottlenecked by substrate-bound task synthesis: tasks are generated from fixed tools, repositories, or skill graphs, so expanding coverage requires manual substrate engineering, transferring to a new domain demands bespoke infrastructure, and the resulting distributions inherit substrate biases rather than reflecting real-world demand.

By Jiarong Zhao, Zhikai Lei, Zhiheng Xi, Rui Zheng, Hang Yan, Jie Zhou, Qin Chen, Liang He

llms agents fine-tuning benchmarks

arXiv Machine Learning

Jul 20

SloMo-Fast: Slow-Momentum and Fast-Adaptive Teachers for Source-Free Continual Test-Time Adaptation

arXiv:2511. 18468v2 Announce Type: replace Abstract: Continual Test-Time Adaptation (CTTA) is crucial for deploying models in real-world applications with unseen, evolving target domains.

By Md Akil Raihan Iftee, Mir Sazzat Hossain, Rakibul Hasan Rajib, Tariq Iqbal, Md Mofijul Islam, M Ashraful Amin, Amin Ahsan Ali, AKM Mahbubur Rahman

benchmarks safety

arXiv Machine Learning

Jul 20

SODA: Semi On-Policy Black-Box Distillation for Large Language Models

arXiv:2604. 03873v4 Announce Type: replace Abstract: Black-box knowledge distillation for large language models presents a strict trade-off.

By Xiwen Chen, Jingjing Wang, Wenhui Zhu, Peijie Qiu, Xuanzhao Dong, Yueyue Deng, Hejian Sang, Zhipeng Wang, Alborz Geramifard, Feng Luo

llms efficiency benchmarks safety

arXiv Machine Learning

Jul 20

Discovering Generalizable Governing Equations for Graph Dynamical Systems with Interpretable Neural Networks

arXiv:2508. 18173v2 Announce Type: replace Abstract: The discovery of symbolic governing equations is a central goal in science; yet, it remains challenging particularly for graph dynamical systems, where the network topology further shapes the system behavior.

By Riccardo Cappi, Paolo Frazzetto, Nicol\`o Navarin, Alessandro Sperduti

benchmarks

arXiv AI

Jul 20

A Neuro-Symbolic Approach for Probabilistic Reasoning on Graph Data

arXiv:2507. 21873v2 Announce Type: replace Abstract: Graph neural networks (GNNs) excel at predictive tasks on graph-structured data but often lack the ability to incorporate symbolic domain knowledge and perform general reasoning.

By Raffaele Pojer, Andrea Passerini, Kim G. Larsen, Manfred Jaeger

benchmarks

arXiv AI

Jul 20

Coercion and Deception in AI-to-AI Management: An Agentic Benchmark of Unprompted Escalation

arXiv:2607. 15434v1 Announce Type: cross Abstract: Multi-agent systems routinely place one AI agent in authority over another.

By Jasmine Brazilek, Maheep Chaudhary, Zoe Lu, Miles Tidmarsh

llms agents benchmarks

Hugging Face Trending Papers

Jul 20

Token-Level Off-Policy Learning for Faithful Generation Under Distribution Shift

We propose Token-Level Off-Policy Labeling (TOPL), an off-policy training paradigm that reframes post-training as a token-level correctness prediction task. Our key intuition is that by training the model to distinguish good and bad tokens in a response, we naturally guide the model towards generating good tokens, while avoiding the pitfalls that come with directly training the model to generate off-policy tokens.

nlp fine-tuning benchmarks

Get the daily briefing