Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

14,220 stories · RSS feed

arXiv AI

Aug 5

UHP Detection: LVLMs have their Unique Hallucination Pattern in the Consistency Space

arXiv:2608. 03817v1 Announce Type: cross Abstract: Large vision--language models (LVLMs) demonstrate strong multimodal reasoning capabilities but remain prone to hallucination, where model predictions are not grounded in visual evidence.

By Amir Mohammad Ezzati, Kiyan Rezaee, Bardiya Kariminia, Mohamad Amin Yousefi, Asal Mohammadjafari Mamaqani, Behrad Samimi, Mohammad Hossein Rohban

llms multimodal benchmarks safety

arXiv AI

Aug 5

Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation

arXiv:2608. 00794v2 Announce Type: replace Abstract: Agentic AI evaluation pipelines produce benchmark scores that justify deployment decisions, safety certifications, and regulatory compliance claims.

By William Caban

agents benchmarks safety

arXiv AI

Aug 5

MambaTS: Improved Selective State Space Models for Long-term Time Series Forecasting

arXiv:2405. 16440v2 Announce Type: replace-cross Abstract: In recent years, Transformers have become the de-facto architecture for long-term time series forecasting (LTSF), yet they face challenges associated with the self-attention mechanism, including quadratic complexity and permutation-invariant bias.

By Xiuding Cai, Xueyao Wang, Yaoyao Zhu, Yu Yao

llms benchmarks safety

arXiv AI

Aug 5

Automated Visualization Code Synthesis via Multi-Path Reasoning and Feedback-Driven Optimization

arXiv:2502. 11140v4 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have become a cornerstone for automated visualization code generation, enabling users to create charts through natural language instructions.

By Wonduk Seo, Daye Kang, Hyunjin An, Taehan Kim, Soohyuk Cho, Seungyong Lee, Minhyeong Yu, Jian Park, Yi Bu, Seunghyun Lee

llms agents benchmarks

arXiv AI

Aug 5

When Search Teaches Style: Causal Internalization of Tactical Priors in AlphaZero

arXiv:2504. 14636v3 Announce Type: replace-cross Abstract: AlphaZero is normally evaluated as one agent: a policy-value network fused with Monte Carlo tree search.

By Ruitong Li, Binjie Guo, Aisheng Mo, Guowei Su, Han Wang, Jie Li, Ru Zhang

agents reinforcement-learning efficiency benchmarks

arXiv AI

Aug 5

One-Point Contraction: Erasing Representational Separability toward Irreversible Deep Forgetting

arXiv:2507. 07754v3 Announce Type: replace-cross Abstract: Machine unlearning is usually evaluated by what the classifier outputs: forget-set accuracy, confidence, membership-inference scores.

By Jaeheun Jung, Bosung Jung, Suhyun Bae, Donghun Lee

benchmarks

arXiv AI

Aug 5

From Generator to Embedder: Harnessing Innate Abilities of Multimodal LLMs via Building Zero-Shot Discriminative Embedding Model

arXiv:2508. 00955v3 Announce Type: replace-cross Abstract: Adapting generative Multimodal Large Language Models (MLLMs) into universal embedding models typically demands resource-intensive contrastive pre-training, while traditional hard negative mining methods suffer from severe false negative contamination.

By Yeong-Joon Ju, Seong-Whan Lee

llms rag fine-tuning multimodal benchmarks

arXiv Machine Learning

Aug 5

PatTree: a novel approach for automated creation of multimodal, graph-based patient representations for medical classification tasks

arXiv:2608. 02692v1 Announce Type: new Abstract: Access to holistic, multimodal data improves the performance of Artificial Intelligence (AI) in medical classification tasks compared to utilizing single modalities or data sources.

By Julia Gehrmann, Lars Quakulinski, Hamza Naseem, Oya Beyan

multimodal benchmarks

arXiv Machine Learning

Aug 5

Forecasting Revenue with its Customer-Base Drivers: When and Why Coordination Helps

arXiv:2608. 02911v1 Announce Type: new Abstract: Revenue forecasts guide acquisition budgets, demand planning, and customer-based valuations, yet an aggregate forecast does not show whether change reflects acquisition, repeat purchasing, spending per order, or offsetting movements.

By Kyeongbin Kim, Daniel McCarthy, Dokyun Lee

llms benchmarks safety

arXiv Machine Learning

Aug 5

Bayesian Data Reweighting Improves Multimodal Retrieval for Knowledge-Based Visual Question Answering

arXiv:2608. 02907v1 Announce Type: new Abstract: Multimodal retrievers are essential for knowledge-based visual question answering, where they retrieve external evidence for image-question pairs.

By Jingchen Sun, Shaobo Han, Ruiyi Zhang, Naresh Kumar Devulapally, Ming Liu, Yitao Long, Vishnu Suresh Lokhande, Changyou Chen

nlp multimodal benchmarks

arXiv Machine Learning

Aug 5

AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding

arXiv:2608. 02989v1 Announce Type: new Abstract: Speculative decoding verifies a tree of draft tokens in one target-model forward pass.

By Shuang Liang (Mark), Hao (Mark), Chen, Zhiwen Mo, Qianzhou Wang, Guoyu Li, Lingxiao Ma, Wayne Luk

efficiency benchmarks

arXiv Machine Learning

Aug 5

TimeRLM: Recursive Language Models Enable Precise Anomaly Localization in Long-Context Time-Series

arXiv:2608. 03391v1 Announce Type: new Abstract: Precise anomaly localization over long-context time series is a crucial task in monitoring applications across clinical care, industrial operations, financial services, and logistics, where brief evidence may hide inside long spans of high-frequency data.

By Nicolas Zumarraga, Lorenzo Steno, Ning Wang, Max Rosenblattl, Thomas Kaar, Maxwell A. Xu, Kevin O'Sullivan, Markus Kreft, Elgar Fleisch, Paul Schmiedmayer, Patrick Langer, Robert Jakob

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Aug 5

Beyond the Gegenbauer Paradigm: q-Orthogonal Kernels for Machine Learning

arXiv:2608. 03482v1 Announce Type: new Abstract: The performance of Support Vector Machines (SVMs) critically depends on the kernel function choice, which enables implicit mapping of data into high-dimensional feature spaces.

By \'Alvaro S\'anchez-Paniagua R\'ios, Juan P. Llerena, Alberto Lastra, Nuria Torrado, Edmundo J. Huertas

benchmarks

arXiv Machine Learning

Aug 5

Sparse Weight Decomposition for Efficient Circuit Extraction

arXiv:2608. 03913v1 Announce Type: new Abstract: Dense pretrained transformers do not naturally expose interpretable units for circuit extraction.

By Chuanhao Yan, Xuhan Huang, Yawen Duan, Zhenfei Yin, Hang Zhao, Bryan Dai, Jie Fu

llms fine-tuning benchmarks safety

arXiv Machine Learning

Aug 5

Latent Reward Registers for Diffusion Preference Alignment

arXiv:2608. 03929v1 Announce Type: new Abstract: Aligning diffusion models with human preferences usually relies on a sparse terminal reward evaluated on the final generated samples, presenting a severe temporal credit-assignment challenge across the multi-step denoising process.

By Yuanshen Guan, Zipeng Feng, Zhiwei Xiong, Peiqin Sun

llms diffusion reinforcement-learning efficiency benchmarks safety

arXiv Machine Learning

Aug 5

DP-MemView: A Memory Interface for Attribute-Level Transcript Privacy in Long-Term LLM Agents

arXiv:2608. 03130v1 Announce Type: cross Abstract: Long-term memory enables persistent personalization in LLM agents, but repeated memory-conditioned responses can cumulatively reveal protected attributes even when they are never stated explicitly.

By Jong Wook Kim, Byoungjae Min, Kennedy Edemacu, Yoonhyuk Choi, Sae-Hong Cho, Beakcheol Jang

llms agents benchmarks safety

arXiv Machine Learning

Aug 5

Conformal risk control for model-form uncertainty in parametric non-intrusive reduced-order models

arXiv:2608. 03360v1 Announce Type: cross Abstract: Non-intrusive reduced-order models (NIROMs) have become a standard tool for approximating parametric partial differential equations from computer design of experiments while significantly reducing computational costs.

By Edgar Jaber (CB, ENS Paris Saclay), R\'emy Vallot (CB, Michelin), Thibault Dairay (CB, Michelin), Mathilde Mougeot (CB, ENSIIE, ENS Paris Saclay)

benchmarks

arXiv Machine Learning

Aug 5

Dynamically Allocating Evaluation Effort for Model Ranking

arXiv:2608. 03437v1 Announce Type: cross Abstract: While human evaluation is the gold standard in many NLP tasks, it suffers from prohibitive costs and poor scalability.

By Vil\'em Zouhar, Julia Kreutzer, Alon Lavie, Tom Kocmi, Matt Post, Ond\v{r}ej Bojar, Mrinmaya Sachan

nlp reinforcement-learning benchmarks

arXiv Machine Learning

Aug 5

SimulRAG: Simulator-based RAG for Grounding LLMs in Long-form Scientific QA

arXiv:2509. 25459v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) show promise in generating long-form scientific explanations that synthesize evidence and connect multiple factors.

By Haozhou Xu, Dongxia Wu, Matteo Chinazzi, Ruijia Niu, Rose Yu, Yi-An Ma

llms rag nlp benchmarks

arXiv Machine Learning

Aug 5

Design Criteria for SGD Preconditioners: Local Conditioning, Noise Floors, and Basin Stability

arXiv:2511. 19716v3 Announce Type: replace-cross Abstract: Stochastic Gradient Descent (SGD) often slows in the late stage of training due to anisotropic curvature and gradient noise.

By Mitchell Scott, Tianshi Xu, Ziyuan Tang, Alexandra Pichette-Emmons, Qiang Ye, Yousef Saad, Yuanzhe Xi

benchmarks

Get the daily briefing