Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,507 stories · RSS feed

arXiv AI

Jun 2

SPADE-Bench: Evaluating Spontaneous Strategic Deception in Agents via Plan-Action Divergence

arXiv:2606. 02380v1 Announce Type: cross Abstract: As LLM-based agents expand their operational scope, reliability becomes a prerequisite for real-world deployment.

By Yuyan Bu, Haowei Li, Qirui Zheng, Bowen Dong, Kaiyue Yang, Jiaming Ji, Yingshui Tan, Wenxin Li, Yaodong Yang, Juntao Dai

llms agents benchmarks safety

arXiv Machine Learning

Jun 2

Identifiable Markov Switching Models with Instantaneous Effects and Exponential Families

arXiv:2606. 02231v1 Announce Type: cross Abstract: Temporal systems often exhibit non-stationary behaviour, such as seasonal climate variation or glucose fluctuations in patients with type-1 diabetes.

By Roel Hulsman, Carles Balsells-Rodas, Sara Magliacane

benchmarks

arXiv Machine Learning

Jun 2

Random Erasing vs. Model Inversion: A Promising Defense or a False Hope?

arXiv:2409. 01062v4 Announce Type: replace Abstract: Model Inversion (MI) attacks pose a significant privacy threat by reconstructing private training data from machine learning models.

By Viet-Hung Tran, Ngoc-Bao Nguyen, Son T. Mai, Hans Vandierendonck, Ira Assent, Alex Kot, Ngai-Man Cheung

benchmarks safety

arXiv AI

Jun 2

Pause and Think: A Dataset and Benchmark for Video-Grounded Assistive Action Suggestion

arXiv:2606. 00616v1 Announce Type: cross Abstract: Recent Vision-Language Models (VLMs) struggle with grounded reasoning, temporal consistency, and context aware planning in videos.

By Shivam Singh, Saptarshi Majumdar, Pratik Prabhanjan, Zicheng Liu, Emad Barsoum

llms fine-tuning multimodal benchmarks

arXiv AI

Jun 2

LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?

arXiv:2602. 16902v4 Announce Type: replace Abstract: We introduce LLM-Wikirace, a benchmark for evaluating planning, reasoning, and world knowledge in large language models (LLMs).

By Juliusz Ziomek, William Bankes, Lorenz Wolf, Shyam Sundhar Ramesh, Xiaohang Tang, Ilija Bogunovic

llms benchmarks

arXiv AI

Jun 2

From Evaluation to Design: Using Potential Energy Surface Smoothness Metrics to Guide Machine Learning Interatomic Potential Architectures

arXiv:2602. 04861v2 Announce Type: replace-cross Abstract: Machine Learning Interatomic Potentials (MLIPs) sometimes fail to reproduce the physical smoothness of the quantum potential energy surface (PES), leading to erroneous behavior in downstream simulations that standard energy and force regression evaluations can miss.

By Ryan Liu, Eric Qu, Tobias Kreiman, Samuel M. Blau, Aditi S. Krishnapriyan

llms benchmarks

arXiv AI

Jun 2

Rethinking Scientific Modeling: Toward Physically Consistent and Simulation-Executable Programmatic Generation

arXiv:2602. 07083v2 Announce Type: replace-cross Abstract: Structural modeling is a fundamental component of computational engineering science, in which even minor physical inconsistencies or specification violations may invalidate downstream simulations.

By Yongqing Jiang, Jianze Wang, Zhiqi Shen, Zhenghong Lin, Jiayuan Wang, Yijian Yang, Kaoshan Dai, Haoran Luo

llms fine-tuning benchmarks safety

arXiv AI

Jun 2

When and How Much to Imagine: Adaptive Test-Time Scaling with World Models for Visual Spatial Reasoning

arXiv:2602. 08236v2 Announce Type: replace-cross Abstract: Despite rapid progress in MLLMs, visual spatial reasoning remains unreliable when correct answers depend on how a scene would appear under unseen or alternative viewpoints.

By Shoubin Yu, Yue Zhang, Zun Wang, Jaehong Yoon, Huaxiu Yao, Mingyu Ding, Mohit Bansal

llms robotics benchmarks

arXiv Machine Learning

Jun 2

Accelerating physics-informed neural networks for full waveform inversion using a hybrid quantum-classical finite-basis architecture

arXiv:2606. 01110v1 Announce Type: cross Abstract: Full waveform inversion (FWI) reconstructs heterogeneous material properties from receiver data but remains computationally demanding.

By Hoang Anh Nguyen, Divakar Vashisth, Ali Tura

benchmarks

arXiv AI

Jun 2

Targeted Remasking: Replacing Token Editing with Token-to-Mask Refinement in Discrete Diffusion Language Models

arXiv:2605. 26436v2 Announce Type: replace-cross Abstract: Discrete masked diffusion language models such as LLaDA generate text through iterative denoising, where mask tokens are progressively replaced with predicted tokens.

By Lin Yao

llms diffusion benchmarks

arXiv AI

Jun 2

Prior Availability in Industrial Visual Sim-to-Real: A Review of CAD-Guided and CAD-Unavailable Regimes

arXiv:2605. 30581v2 Announce Type: replace-cross Abstract: Industrial visual sim-to-real is often described as transferring from synthetic images to real images, but industrial deployment usually involves a broader mismatch between available evidence and required decisions.

By Chenxi Tao, Seung-Kyum Choi

computer-vision robotics multimodal benchmarks

arXiv Machine Learning

Jun 2

FLaG: Fine-Grained Latent Grouping for Hallucination Detection

arXiv:2606. 00301v1 Announce Type: new Abstract: Hallucinations in large language models (LLMs) arise from heterogeneous failure mechanisms, making reliable detection difficult for any single global uncertainty score.

By Wentao Ye, Liyao Li, Zhiqing Xiao, Muzhi Zhu, Jiaqi Hu, Zhanming Shen, Xiaomeng Hu, Sean Du, Haobo Wang

llms benchmarks safety

arXiv Machine Learning

Jun 2

ATLAS: Agentic Test-time Learning-to-Allocate Scaling

arXiv:2606. 01667v1 Announce Type: new Abstract: Test-time scaling has become a major way to improve large language model reasoning, but its orchestration has remained designer-engineered: a fixed sample budget, a fixed refinement loop, a fixed scoring rule, or a fixed search policy decides how compute is spent, leaving the model in charge of solving but not of orchestration.

By Peijia Qin, Qi Cao, Pengtao Xie

llms agents nlp multimodal benchmarks

arXiv Machine Learning

Jun 2

Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging

arXiv:2606. 01717v1 Announce Type: new Abstract: Instruction tuning aligns large language models, including multimodal ones, with diverse user intents, but scaling to heterogeneous mixtures is hindered by gradient interference and bandwidth-heavy synchronization.

By Minsik Choi, Geewook Kim

llms fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 2

G2LoRA: Gradient Orthogonal Low-Rank Adaptation Framework for Graph Continual Learning on Text-Attributed Graphs

arXiv:2606. 01873v1 Announce Type: new Abstract: LLM-as-Aligner has emerged as a prevalent pre-training paradigm for Text-Attributed Graphs(TAGS), aligning graph and text modalities into a shared embedding space via CLIP-style contrastive learning.

By Yuhan Wang, Yibo Ding, Yutong Ye, Mufan Zhao, Wenbo Zhang, Ruijie Wang, Jianxin Li

llms rag fine-tuning multimodal benchmarks safety

arXiv AI

Jun 2

Learning to Remember, Learn, and Forget in Attention-Based Models

arXiv:2602. 09075v3 Announce Type: replace-cross Abstract: In-Context Learning (ICL) in transformers acts as an online associative memory and is believed to underpin their high performance on complex sequence processing tasks.

By Djohan Bonnet, Jamie Lohoff, Jan Finkbeiner, Elidona Skhikerujah, Emre Neftci

llms benchmarks

arXiv Machine Learning

Jun 2

Evaluating Real-World Generalizability of Algorithm Selection Models

arXiv:2606. 02016v1 Announce Type: new Abstract: Algorithm Selection (AS) aims to automatically identify the most suitable optimization algorithm for a given problem instance by leveraging measurable problem characteristics and historical performance data.

By Gjorgjina Cenikj, Jakub Kudela, Eva Tuba, Tome Eftimov

robotics benchmarks

arXiv Machine Learning

Jun 2

Differentially Private Datastore Generation for Retrieval-Augmented Inference

arXiv:2606. 01413v1 Announce Type: cross Abstract: It is crucial for modern on-device AI systems that rely on retrieval-augmented inference to release and share datastores without compromising individual privacy.

By Abdelrahman Abouelenein, Marwan Torki

rag benchmarks safety

arXiv AI

Jun 2

Soft-NBCE: Entropy-Weighted Chunk Fusion for Long-Context

arXiv:2606. 01101v1 Announce Type: cross Abstract: The quadratic complexity of self-attention remains a bottleneck for Large Language Models (LLMs) processing ultra-long contexts.

By Shihao Ji, Mingyu Li, Zihui Song

llms rag fine-tuning efficiency benchmarks

arXiv Machine Learning

Jun 2

KDH-CAD: Knowledge-data hybrid CAD learning under data scarcity

arXiv:2606. 01702v1 Announce Type: cross Abstract: Deep learning in computer-aided design (CAD) remains fundamentally constrained by the data scarcity challenge: authentic CAD data is difficult to collect at scale, while synthetic data may not faithfully reflect real design practice.

By Ziqin Gao, Zhijie Yang, Qiang Zou

diffusion fine-tuning benchmarks

Get the daily briefing