Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

15,813 stories · RSS feed

arXiv AI

Jul 28

Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models

arXiv:2601. 21003v3 Announce Type: replace Abstract: Large Language Models usually put more emphasis on accuracy and therefore, will guess even when not certain about the prediction, which is especially severe when fine-tuned on small datasets due to the inherent tendency toward miscalibration.

By Moule Lin, Shuhao Guan, Andrea Patane, David Gregg, Goetz Botterweck

llms fine-tuning benchmarks

arXiv AI

Jul 28

From Pixels to Prompts: Vision-Language Models

arXiv:2605. 07544v3 Announce Type: replace Abstract: When you read a paper about a new Vision-Language Model today, it can be easy to forget how strange this idea would have sounded not so long ago.

By Khang Nhat Hoang Vo

llms multimodal benchmarks

arXiv Machine Learning

Jul 28

When Low CER is Not Enough: An Analysis of Hallucinations in Vision-Language OCR Systems on Historical Uruguayan Documents

arXiv:2607. 24077v1 Announce Type: cross Abstract: Optical Character Recognition (OCR) is a key component in the digitization of historical archives.

By Marina Gardella (CB), Camilo Mari{\~n}o (UDELAR, CB), Diego Belzarena (UDELAR, CB), Ignacio Ram{\'i}rez (UDELAR), Gregory Randall (UDELAR), Jean-Michel Morel (LU - Hong Kong)

llms computer-vision multimodal benchmarks safety

arXiv AI

Jul 28

Nanbeige4.2-3B: Unlocking Agentic Capabilities in a Compact Model

arXiv:2607. 22083v2 Announce Type: replace Abstract: We present Nanbeige4.

By Nanbeige Lab, :, Chen Yang, Chengrui Huang, Fufeng Lan, Hanhui Chen, Hao Zhou, Huatong Song, Jiaqi Cao, Jiaying Zhu, Jinlin Niu, Kai Wang, Lisheng Huang, Qiliang Liang, Ran Le, Ruixiang Feng, Shuang Sun, Tao Gu, Tao Zhang, Tianyu Luo, Yang Song, Yun Xing, Yuntao Wen, Ziyao Xu, Zongchao Chen, Zongqiang Li

llms rag agents reinforcement-learning fine-tuning benchmarks safety

arXiv AI

Jul 28

AutoMat: Enabling Automated Crystal Structure Reconstruction from Microscopy via Agentic Tool Use

arXiv:2505. 12650v2 Announce Type: replace-cross Abstract: Reconstructing atomistic crystal structures from a single noisy STEM projection is an ill-posed inverse problem: multiple lattices can explain similar contrast, and purely feed-forward models cannot verify physical validity.

By Yaotian Yang, Yiwen Tang, Yizhe Chen, Xiao Chen, Jiangjie Qiu, Hao Xiong, Haoyu Yin, Zhiyao Luo, Yifei Zhang, Sijia Tao, Wentao Li, Qinghua Zhang, Yuqiang Li, Wanli Ouyang, Bin Zhao, Xiaonan Wang, Fei Wei

agents multimodal benchmarks

arXiv Machine Learning

Jul 28

On-Device Inference versus Wireless Streaming: Energy-Efficient Multi-Modal Deep Learning for Wearable Cardiovascular Patches

arXiv:2510. 18668v4 Announce Type: replace Abstract: Wearable cardiovascular sensor patches promise continuous, unobtrusive monitoring, but their tight energy, memory, and compute budgets make it unclear whether physiological signals should be analyzed on the device or streamed to the cloud for processing.

By Mustafa Fuad Rifet Ibrahim, Tunc Alkanat, Felix Manthey, Maurice Meijer, Alexander Schlaefer, Peer Stelldinger

multimodal benchmarks

arXiv AI

Jul 28

AIFL: A Global Daily Streamflow Forecasting Model Using a Deterministic LSTM Pre-trained on ERA5-Land and Fine-tuned on IFS

arXiv:2602. 16579v2 Announce Type: replace-cross Abstract: Reliable global streamflow forecasting is essential for flood preparedness and water resource management, yet data-driven models often suffer from a performance gap when transitioning from historical reanalysis to operational forecast products.

By Maria Luisa Taccari, Kenza Tazi, Ois\'in M. Morrison, Andreas Grafberger, Juan Colonese, Corentin Carton de Wiart, Christel Prudhomme, Cinzia Mazzetti, Matthew Chantry, Florian Pappenberger

fine-tuning benchmarks

arXiv AI

Jul 28

Statistical realism is not evidence that LLMs can estimate treatment effects in social science experiments

arXiv:2604. 02458v3 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly used to simulate human responses and estimate treatment effect of interventions when real-world experiments are costly or infeasible.

By Zonghan Li, Feng Ji

llms benchmarks

arXiv Machine Learning

Jul 28

Efficient Online Conformal Selection with Limited Feedback

arXiv:2605. 14953v2 Announce Type: replace Abstract: We address the problem of conformal selection, where an agent must select a minimal subset of options to ensure that at least one ``success'' is identified with a pre-specified target probability $\phi$.

By Sreenivas Gollapudi, Kostas Kollias, Kamesh Munagala, Ali Sinop

agents reinforcement-learning benchmarks

arXiv Machine Learning

Jul 28

LanteRn: Latent Visual Structured Reasoning

arXiv:2603. 25629v2 Announce Type: replace-cross Abstract: While language reasoning models excel in many tasks, visual reasoning remains challenging for current large multimodal models (LMMs).

By Andr\'e G. Viveiros, Nuno Gon\c{c}alves, Matthias Lindemann, Andr\'e Martins

llms rag diffusion reinforcement-learning fine-tuning multimodal benchmarks

arXiv Machine Learning

Jul 28

Distributional Split Criteria for Random Forests: Extensions, Shrinkage, and the Robustness of Mean Splitting

arXiv:2607. 23721v1 Announce Type: cross Abstract: Distributional random forests replace mean-based CART splitting with criteria that compare the full conditional response distribution in candidate children.

By Silas Koemen

benchmarks

arXiv AI

Jul 28

LC-SEPLM: long-range contact-supervised adaptation for sequence-only protein representation learning

arXiv:2607. 22777v1 Announce Type: cross Abstract: Protein language models learn transferable sequence representations.

By Chen Wang, Boming Kang, Qinghua Cui

llms fine-tuning benchmarks

arXiv AI

Jul 28

FilmBench: A Film-Grade Benchmark for Cinematic Video Generation

arXiv:2607. 24241v1 Announce Type: cross Abstract: Progress in video generation keeps narrowing the visual gap between AI-generated and professionally produced footage, yet most benchmarks still draw prompts from web sources or LLM templates and score them with untrained, generic multimodal models.

By Shengyi Wang, Niantong Li, Guangzheng Hu, Hong Qi, Fei Ding, Weixu Qiao, Jinlin Wang, Xiaotong Lv, Peng Han, Zimeng Li, Fanshu Ding, Yushu Wang, Han Wu, Jingjing Chen, Chongxiao Wang, Yanhao Wu, Chenglong Huang, Xiaoqian Zhu, Jie Tian, Hua Li, Jingjing Fan, Mingshuang Tang, Zhong Li, Hengxia Qiang, Weibin Chen, Jinyang Zhen, Bing Zhao, Lin Qu, Jing Li, Hu Wei

llms agents diffusion multimodal benchmarks safety

arXiv AI

Jul 28

Spatial Reasoning in LLM Game Agents: Impact of Causal Context and Multi-Step Planning

arXiv:2607. 22732v1 Announce Type: new Abstract: LLM-based game agents often perform poorly on more complex tasks.

By Mohit Jiwatode, Ronja Fuchs, Robin Schm\"ocker, Bodo Rosenhahn, Alexander Dockhorn

llms agents benchmarks

arXiv AI

Jul 28

Success Is Not Self-Explanatory: Auditing Success Provenance in Agent Evaluation

arXiv:2607. 24054v1 Announce Type: new Abstract: A correct answer can conceal why an agent succeeded.

By Jingkun Luo, Da-Tian Peng

agents benchmarks

arXiv AI

Jul 28

MemChain: Learning Interpretable Memory Traces for Memory-Augmented LLM Agents

arXiv:2607. 24097v1 Announce Type: new Abstract: Memory-augmented LLM agents typically answer queries by retrieving relevant memories and feeding them directly to an answer model.

By Yiwen Ma, Songjun Tu, Qichao Zhang, Dong Li, Linjing Li, Dongbin Zhao

llms agents reinforcement-learning benchmarks

arXiv AI

Jul 28

Choosing a Text Embedding Model: A Practical Benchmarking and Decision Framework

arXiv:2607. 23507v1 Announce Type: cross Abstract: Choosing the right text embedding model is one of the most consequential -- and most frequently under-examined -- decisions in building a retrieval or search system, yet the model that tops a leaderboard is rarely the best choice for a given deployment.

By Madhav S Baidya

rag nlp benchmarks

arXiv AI

Jul 28

Reason Popper-ly: Patching In-Context Reasoning with Inductive Logic Programming

arXiv:2607. 23019v1 Announce Type: new Abstract: Chain-of-thought (CoT) prompting enables large language models (LLMs) to tackle multi-step reasoning tasks, yet the generated intermediate steps are not guaranteed to be logically sound.

By Zirong Chen, Meiyi Ma

llms benchmarks

arXiv AI

Jul 28

CuraWeb: Joint Optimization of Quality, Redundancy, and Diversity for Web-Scale Pretraining Data

arXiv:2607. 22662v1 Announce Type: new Abstract: Open-web corpora curated via highly selective filters, such as FineWeb-Edu and DCLM, constitute the core of LLM pretraining data and have significantly advanced LLM performance.

By Peiguang Li, Yongwei Zhou, Juncheng Diao, Yuchun Fan, Jian Yang, Jianxiao Yang, Zhongda Su, Shuguang Jiao, Xiao Wei, Zhiye Zou, Gan Dong, Zhizhao Zeng, Rongxiang Weng, Jingang Wang, Xunliang Cai

llms efficiency benchmarks

arXiv AI

Jul 28

LLM-SoccerArena: Benchmarking LLMs on Real-World Predictions in Sports

arXiv:2607. 24573v1 Announce Type: new Abstract: Large language models (LLMs) increasingly support decisions about uncertain future events, yet evaluating their ability to forecast real-world outcomes remains difficult.

By Jonas Schr\"oder, Jonas Schweisthal, Oliver M\"uller, Markus Weinmann, Stefan Feuerriegel

llms benchmarks

Get the daily briefing