Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

17,067 stories · RSS feed

arXiv Machine Learning

Jul 8

$\mathbf{\lambda}$-VAE: Variance Equalization for Posterior Collapse

arXiv:2607. 05531v1 Announce Type: new Abstract: Variational Autoencoders (VAEs) frequently suffer from posterior collapse, a failure mode in which the approximate posterior converges to the prior, rendering the latent code uninformative.

By Girum Demisse

benchmarks

arXiv Machine Learning

Jul 8

Level-Crossing Density as a Mesh-Free High-Frequency Auxiliary Loss for Implicit Neural Representations

arXiv:2607. 05815v1 Announce Type: new Abstract: The Minkowski functionals of a field's excursion sets -- area, boundary measure, and Euler characteristic -- describe its level-set morphology; the Euler characteristic is the cheapest handle on topology.

By Gunner Levi Howe

benchmarks

arXiv AI

Jul 8

Industry Classification of GitHub Repositories Using the North American Industry Classification System (NAICS)

arXiv:2607. 06505v1 Announce Type: cross Abstract: GitHub hosts hundreds of millions of public repositories, but the platform exposes no native mapping from repositories to standardized industry sectors.

By Kevin Xu, Alexander Quispe

llms rag diffusion fine-tuning benchmarks

arXiv AI

Jul 8

Integrating knowledge graphs and multilingual scholarly corpora for domain-adaptive LLMs in SSH

arXiv:2607. 05956v1 Announce Type: new Abstract: The integration of Large Language Models (LLMs) into scientific research workflows, particularly for bibliographic discovery and literature synthesis, raises significant methodological, epistemic and regulatory challenges for the Social Sciences and Humanities (SSH), especially with regard to disciplinary diversity, multilingual access to sources and the evaluation of results.

By Adam Faci, Alessio Miaschi, Anne Combe, Pascal Cuxac, Francesca Frontini, Nicolas Larrousse, St\'ephane Pouyllau

llms rag nlp benchmarks safety

arXiv Machine Learning

Jul 8

PGOT: A Physics-Geometry Operator Transformer for Complex PDEs

arXiv:2512. 23192v4 Announce Type: replace Abstract: While Transformers have demonstrated remarkable potential in modeling Partial Differential Equations (PDEs), modeling large-scale unstructured meshes with complex geometries remains a significant challenge.

By Zhuo Zhang, Xi Yang, Ying Miao, Xiaobin Hu, Yifu Gao, Yong Yang, Canqun Yang, Boocheong Khoo

llms benchmarks

arXiv AI

Jul 8

RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications

arXiv:2607. 06411v1 Announce Type: cross Abstract: Developers increasingly delegate real maintenance work to product-grade coding agents, and many state tasks in their native language, in the style of a customer request rather than a curated English issue.

By Evgeny Shilov (Independent Researcher)

llms agents benchmarks

arXiv AI

Jul 8

Prompt-Adapter Context Routing for Parameter-Efficient Multi-Shot Long Video Extrapolation

arXiv:2607. 06481v1 Announce Type: cross Abstract: We present PACR-Video, a parameter-efficient framework for multi-shot long video extrapolation that preserves recurring entities, scene structure, visual style, and causal progression without full generator fine-tuning.

By Anna C\'ordoba, Adam Puente Tercero, Nerea Angulo Hijo, Mar Linares Tercero, Julia Barrientos, Ainhoa Miranda, Jes\'us Olivera

llms diffusion fine-tuning efficiency benchmarks safety

arXiv Machine Learning

Jul 8

Supervised Reward Inference

arXiv:2502. 18447v2 Announce Type: replace Abstract: Existing approaches to reward inference typically assume that humans provide demonstrations according to specific behavior models.

By Will Schwarzer, Jordan Schneider, Philip S. Thomas, Scott Niekum

robotics benchmarks

arXiv AI

Jul 8

Whose fairness? Structural concentration in AI bias research

arXiv:2607. 05574v1 Announce Type: cross Abstract: Artificial intelligence increasingly mediates consequential decisions in healthcare, law, and public services, and the field has responded with an extensive methodology for measuring and mitigating bias.

By Abhash Shrestha, Subigya Gautam, Anu Sapkota, Sanju Tiwari, Tek Raj Chhetri

benchmarks safety

arXiv AI

Jul 8

Harnessing Code Agents for Automatic Software Verification

arXiv:2607. 06341v1 Announce Type: cross Abstract: Formal verification offers the strongest guarantee of software correctness, but it does not scale: the proofs demanded by interactive theorem provers such as Coq require enormous expert effort.

By Shuangxiang Kan, Shuanglong Kan, Sebastian Ertel

llms agents benchmarks

arXiv AI

Jul 8

Self-Supervised Implicit CEST Reconstruction via Physics-Informed Lorentz Encoding

arXiv:2607. 06132v1 Announce Type: cross Abstract: Multi-Pool Chemical Exchange Saturation Transfer (CEST) MRI provides valuable metabolic information but is clinically limited by long acquisition times.

By Dexuan Li, Yupeng Wu, Chenglong Wang, Hanlin Liu, Hui Zhen, Jianqi Li, Guang Yang

diffusion benchmarks

arXiv AI

Jul 8

NegROI: Click-Centric Uncertainty-Guided Refinement with Scene-Conditioned Negative Prompts for Robust Interactive 3D Segmentation

arXiv:2607. 05955v1 Announce Type: cross Abstract: Interactive 3D segmentation aims to extract object masks in point clouds with minimal user clicks.

By Shuheng Zhang, Feng Wu

llms computer-vision benchmarks

arXiv AI

Jul 8

PVCap: Towards Accurate 3D Dense Captioning via PseudoCap and VoxelCapNet

arXiv:2607. 06097v1 Announce Type: cross Abstract: 3D dense captioning, an emerging vision-language task, aims to generate descriptive sentences for each object in the 3D scene.

By Xiaopei Wu, Chenshu Hou, Liang Peng, Dan Xu, Binbin Lin, Xiaoshui Huang, Yuenan Hou, Yu Li, Wenxiao Wang, Haifeng Liu, Deng Cai, Wanli Ouyang

multimodal benchmarks

arXiv AI

Jul 8

Differentially Private Natural Gradient Descent

arXiv:2607. 05866v1 Announce Type: cross Abstract: Under a fixed privacy budget, the utility of differentially private (DP) training is ultimately determined by its optimization efficiency.

By Pan Li, Kai Chen, Shuai Chang, Shengzhi Zhang, Peizhuo Lv, Jinwen He

benchmarks safety

arXiv AI

Jul 8

BaFCo: A Document Understanding Benchmark for Complex Bangla Form Comprehension

arXiv:2607. 05614v1 Announce Type: cross Abstract: Document comprehension is a challenging yet impactful task for Multimodal Large Language Models, especially as these systems see growing adoption in real-world, human-centric applications.

By Abu Tyeb Azad, Ishita Sur Apan, Fahim Ahmed, Sumaiya Karim Katha, Ezharuddin Jubaer, Armun Alam, Pranjal Kumar Nandi, Amin Ahsan Ali, Aman Chadha, Md Mofijul Islam, AKM Mahbubur Rahman

llms multimodal benchmarks

arXiv AI

Jul 8

LEGATO 2: Toward Multimodal Sheet Music Recognition and Understanding

arXiv:2607. 05769v1 Announce Type: cross Abstract: We propose a novel pipeline, Legato 2, for extracting symbolic notation and semantic knowledge from images of sheet music.

By Guang Yang, Brian Siyuan Zheng, Victoria Ebert, Noah A. Smith

llms computer-vision multimodal benchmarks

arXiv AI

Jul 8

An Experimental Design Approach to Evaluating Agentic AI's Autonomous Model Discovery

arXiv:2607. 06413v1 Announce Type: cross Abstract: Large language model coding agents increasingly perform open-ended data modeling and analysis.

By Hao He, Xueying Liu, Chris J. Kuhlman, Xinwei Deng

llms agents benchmarks

arXiv Machine Learning

Jul 8

PerCaM-Health: Personalized Dynamic Causal Graphs for Healthcare Reasoning

arXiv:2605. 07267v2 Announce Type: replace Abstract: Personalized healthcare decisions require reasoning about how physiological and behavioral variables influence an individual patient over time.

By Elahe Khatibi, Ziyu Wang, Saba A. Farahani, Di Huang, Hung Cao, Ramesh Jain, Amir M. Rahmani

benchmarks

arXiv AI

Jul 8

AbICL: In-Context Learning for Antigen-Specific Antibody Affinity Ranking

arXiv:2607. 05846v1 Announce Type: cross Abstract: Accurate ranking of antibody candidates according to their binding affinity is essential for therapeutic antibody discovery.

By Zhiyuan Chen, Jing Hu, Junzhe Wang, Yueyang Huang, Xinyi Yang, Zhaoyang Wang, Feng Zhu

llms benchmarks

arXiv AI

Jul 8

RPAM: A Principled Metric for Evaluating Associations in Language Models with High Predictive Validity in Downstream Outputs

arXiv:2607. 05679v1 Announce Type: cross Abstract: Language models (LMs) exhibit problematic biases, such as stereotypes.

By Damian Hodel, Jevin West, Aylin Caliskan

llms rag benchmarks

Get the daily briefing