Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,285 stories · RSS feed

arXiv AI

Jul 1

DeXposure-FM: A Time-series, Graph Foundation Model for Credit Exposures and Stability on Decentralized Financial Networks

arXiv:2602. 03981v2 Announce Type: replace-cross Abstract: Credit exposure in Decentralized Finance (DeFi) is often implicit and token-mediated, creating a dense web of inter-protocol dependencies.

By Aijie Shu, Wenbin Wu, Gbenga Ibikunle, Fengxiang He

benchmarks

arXiv AI

Jul 1

Physics-Constrained Fine-Tuning of Flow-Matching Models for Generation and Inverse Problems

arXiv:2508. 09156v3 Announce Type: replace-cross Abstract: We present a framework for fine-tuning flow-matching generative models to enforce physical constraints and solve inverse problems in scientific systems.

By Jan Tauberschmidt, Sophie Fellenz, Sebastian J. Vollmer, Andrew B. Duncan

fine-tuning benchmarks

arXiv Machine Learning

Jul 1

Calibration, Not Compilation: Detecting and Repairing Misspecified Probabilistic Programs Written by Language Models

arXiv:2606. 31630v1 Announce Type: new Abstract: Language models increasingly write probabilistic programs (in NumPyro, Stan, or Pyro), but a program that compiles, runs, and passes every unit test can still be \emph{statistically} wrong -- a Gaussian likelihood for heavy-tailed data, a Poisson for over-dispersed counts, an invalid prior support, or a pathological parameterization.

By Jian Xu, Delu Zeng, John Paisley, Qibin Zhao

llms benchmarks

arXiv AI

Jul 1

Dataset Construction for Training LLM to Learn Analog Circuit Knowledge

arXiv:2508. 10409v3 Announce Type: replace-cross Abstract: This paper constructs a textual dataset for training large language models (LLMs) to learn analog circuit knowledge and customizes LLM training techniques.

By Zihao Chen, Ji Zhuang, Jinyi Shen, Xiaoyue Ke, Xinyi Yang, Mingjie Zhou, Zhuoyao Du, Xu Yan, Zhouyang Wu, Zhenyu Xu, Jiangli Huang, Li Shang, Xuan Zeng, Fan Yang

llms agents fine-tuning benchmarks

arXiv Machine Learning

Jul 1

ElemeNet: Multiscale Molecular Machine Learning with Uncertainty Quantification Across the Periodic Table

arXiv:2606. 30961v1 Announce Type: cross Abstract: Advances in deep learning architectures and representations have enabled ML-driven chemical property prediction, but state-of-the-art (SOTA) models have remained largely confined to independent codebases and lack support for diverse chemical species.

By Jacob W. Toney, Samir Darouich, Yiran Wang, Aaron G. Garrison, Johannes K\"astner, Heather J. Kulik

llms benchmarks

arXiv AI

Jul 1

Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs

arXiv:2606. 31413v1 Announce Type: new Abstract: Composing independently trained LoRA adapters into a single large language model is useful for multi-domain adaptation, especially when the original training data cannot be shared.

By Seyed Alireza Molavi, Zhan Su, Yan Hu, Peyman Sheikholharam Mashhadi, Stefan Byttner, Prayag Tiwari

llms reinforcement-learning fine-tuning benchmarks

arXiv AI

Jul 1

Beyond Compilation: Evaluating Faithful Natural-Language-to-Lean Statement Formalization

arXiv:2606. 31002v1 Announce Type: new Abstract: Theorem-proving benchmarks evaluate proof search against fixed formal statements, but natural-language-to-Lean formalization must generate the formal statement itself.

By Ke Zhang, Patricio Gallardo Candela, Sudhir Murthy, Yi Xie, Zhi Wang, Maziar Raissi

agents fine-tuning benchmarks

arXiv AI

Jul 1

Optimal Self-Consistency for Efficient Reasoning with Large Language Models

arXiv:2511. 12309v2 Announce Type: replace-cross Abstract: Self-consistency (SC) is a widely used test-time inference technique for improving performance in chain-of-thought reasoning.

By Austin Feng, Marius Alonso, Ambroise Odonnat, Vasilii Feofanov, Ievgen Redko

llms benchmarks

arXiv AI

Jul 1

Beyond expert users: agents should help users construct preferences, not just elicit them

arXiv:2606. 30863v1 Announce Type: new Abstract: Agents typically assume an expert user -- one with well-formed preferences about what they want -- and default to clarifying questions whenever the task is underspecified.

By Irena Saracay, Ludwig Schmidt, Carlos Guestrin

agents benchmarks

arXiv AI

Jul 1

SwiftAudio: Data-Efficient Caption-Only Distillation for One-Step Text-to-Audio Diffusion-based Generation

arXiv:2606. 31259v1 Announce Type: cross Abstract: Diffusion-based text-to-audio (TTA) models achieve impressive synthesis quality but suffer from high inference latency due to iterative multi-step denoising.

By Binh Mai, Tran Quoc Bao Le, Hung Dinh, Cong Tran

diffusion efficiency benchmarks

arXiv AI

Jul 1

PPT-Eval: A Benchmark for Computer-Use Agents on PowerPoint Tasks

arXiv:2606. 31154v1 Announce Type: cross Abstract: Creating and editing slides is a rich, multimodal activity that is ubiquitous in professional and educational settings, making it an ideal testbed for real-world computer-use agents.

By Apurva Gandhi, Vishwas Suryanarayanan, Raja Hasnain Anwar, Firoz Shaik, Shubhang Desai, Thong Q. Nguyen, Muhammad Taqi Raza, Vishal Chowdhary, Graham Neubig

llms agents multimodal benchmarks

arXiv AI

Jul 1

From Idea to Prototype in an Afternoon: Scaffolded, AI-Assisted Rapid VA Prototyping

arXiv:2606. 31311v1 Announce Type: cross Abstract: Testing a new visual-analytics idea usually takes months: one needs to find a realistic data set, clean it, and implement an interactive prototype.

By Gennady Andrienko, Natalia Andrienko

benchmarks

arXiv AI

Jul 1

3D HAMSTER: Bridging Planning and Control in Hierarchical Vision Language Action Models through 3D Trajectory Guidance

arXiv:2606. 31329v1 Announce Type: cross Abstract: Hierarchical Vision-Language-Action (VLA) models decouple high-level planning from low-level control to improve generalization in robot manipulation.

By Dongyoon Hwang, Byungkun Lee, Dongjin Kim, Hyojin Jang, Hoiyeong Jin, Jueun Mun, Minho Park, Hojoon Lee, Hyunseung Kim, Jaegul Choo

llms robotics multimodal benchmarks

arXiv AI

Jul 1

Why Solve It Twice? Hierarchical Accumulation of Skills for Transfer-Efficient ML Engineering

arXiv:2606. 30911v1 Announce Type: new Abstract: ML engineering agents waste compute rediscovering known techniques because every competition is a cold start.

By Yongbin Kim, Yashar Talebirad, Osmar R. Zaiane

llms agents benchmarks

arXiv Machine Learning

Jul 1

FedLAB: Traceable Semantic Codebooks for Federated Multimodal Graph Foundation Learning

arXiv:2606. 32016v1 Announce Type: new Abstract: Multimodal graph foundation models aim to learn reusable knowledge from graphs enriched with text, images, attributes, and relational topology, thereby supporting diverse graph-centric and modality-centric tasks.

By Zekai Chen, Kairui Yang, Xuaner Chen, Xunkai Li, Xun Wu, Rong-Hua Li, Guoren Wang

rag multimodal benchmarks safety

arXiv Machine Learning

Jul 1

TAG-DLM: Diffusion Language Models for Text-Attributed Graph Learning

arXiv:2606. 31166v1 Announce Type: cross Abstract: Text-attributed graphs (TAGs), where each node carries a natural language description, require models to jointly reason over text and graph topology.

By Lingjie Chen, Yuanchen Bei, Haobo Xu, Yanjun Zhao, Yuzhong Chen, Hanghang Tong

llms diffusion fine-tuning benchmarks

arXiv AI

Jul 1

Verify when Uncertain: Beyond Self-Consistency in Black Box Hallucination Detection

arXiv:2502. 15845v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) often hallucinate, limiting their reliability in sensitive applications.

By Yihao Xue, Kristjan Greenewald, Youssef Mroueh, Baharan Mirzasoleiman

llms rag benchmarks safety

arXiv Machine Learning

Jul 1

ShardNet: Training Neural Controllers with Hard, Non-Convex Constraints

arXiv:2606. 30935v1 Announce Type: cross Abstract: While neural network control policies are powerful, their deployment on safety critical systems depends on ensuring that they obey strict constraints.

By Long Kiu Chung, Shreyas Kousik

benchmarks safety

arXiv AI

Jul 1

Visual Semantic Entropy: Do Vision Language Models Recognize Visual Ambiguity?

arXiv:2606. 31407v1 Announce Type: cross Abstract: Vision-language models can produce confident answers on visually ambiguous inputs, resulting in biased predictions.

By Ta Duc Huy, Trang Nguyen, Townim Chowdhury, Ankit Yadav, Minh-Son To, Zhibin Liao, Johan W. Verjans, Vu Minh Hieu Phan

llms rag multimodal benchmarks

arXiv AI

Jul 1

Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR

arXiv:2606. 31813v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) and its variants enable parameter-efficient fine-tuning of large language models under the supervised fine-tuning (SFT) paradigm.

By Ruijia Zhang, Jiacheng Zhu, Hanqing Zhu, Laixi Shi

llms reinforcement-learning fine-tuning benchmarks

Get the daily briefing