Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,077 stories · RSS feed

arXiv AI

Jul 21

DocOCR-Eval: A Correction-Based Framework for OCR Tool Selection Without Ground Truth

arXiv:2607. 16203v1 Announce Type: cross Abstract: Document parsing is a foundational step for document understanding tasks such as visual question answering and key information extraction, as it transforms unstructured scanned images into structured representations by extracting textual, visual, and layout information.

By Zihan Xu, Puzhen Wu, Lawrence Chun Man Lau, Wei Liu, Sirui Li, Yifan Peng, Yihao Ding

llms computer-vision nlp multimodal benchmarks safety

arXiv AI

Jul 21

Simple Domain Generalization for Strong Pixel-Level Image Tampering Detection in Modern VLMs

arXiv:2607. 18230v1 Announce Type: cross Abstract: Modern vision-language models (VLMs) have significantly improved image generation and editing capabilities, making pixel-level image tampering detection increasingly important yet challenging under cross-model and out-of-distribution shifts.

By Yi Tang, Xinyi Shang, Jiacheng Cui, Sondos Mahmoud Bsharat, Jiacheng Liu, Xiaohan Zhao, Tran Dinh Tien, Ahmed Elhagry, Salwa K. Al Khatib, Tianjun Yao, Yonina C. Eldar, Jing-Hao Xue, Hao Li, Salman Khan, Zhiqiang Shen

llms diffusion robotics multimodal benchmarks

arXiv AI

Jul 21

WuYu-EnvLE-Bench: A Benchmark for Evaluating Large Language Models in Environmental Law Enforcement

arXiv:2607. 17745v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly considered for environmental enforcement, but their ability to produce traceable enforcement decisions remains unclear.

By Ziliang Yang, Yi Zhang, Kaijun Lin, Jiachao Ke, Haihong Xu, Zongguo Wen

llms benchmarks

arXiv AI

Jul 21

MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphs

arXiv:2507. 20804v3 Announce Type: replace Abstract: Large Language Models (LLMs) suffer from hallucinations due to their static parametric knowledge.

By Xueyao Wan, Hang Yu

llms rag multimodal benchmarks safety

arXiv AI

Jul 21

DeLIVeR: Decomposed Learning for Information-grounded Veracity Recognition via Reinforced Knowledge Graph Exploration

arXiv:2607. 17935v1 Announce Type: cross Abstract: Automated fact-checking remains a challenge for Large Language Models (LLMs) due to "query brittleness" in traditional retrieval systems.

By Cong Hoan Nguyen, Thomas Hoang, Hieu Minh Duong, Long Nguyen

llms benchmarks

arXiv Machine Learning

Jul 21

Semantic Color Naturalness Breaker: Preventing Illegitimate Colorization via Content-Aware Color Priors

arXiv:2607. 17610v1 Announce Type: cross Abstract: Automatic image colorization enables large-scale and low-cost reuse of grayscale media (e.

By Yuki Nii, Futa Waseda, Ching-Chun Chang, Isao Echizen

benchmarks

arXiv Machine Learning

Jul 21

COVAriance-Induced Fairness Gap Penalty for Subgroup-Fair Clustering

arXiv:2607. 18119v1 Announce Type: cross Abstract: Fair clustering aims to make cluster assignments independent of sensitive attributes, but this goal becomes challenging when multiple sensitive attributes jointly define many subgroups.

By Kyungseon Lee, Hankyo Jeong, Kunwoong Kim, Kwanho Lee, Yongdai Kim

benchmarks safety

arXiv AI

Jul 21

Diffusion-corrected Autoregressive Fourier Neural Operator for Droplet Evolution Prediction

arXiv:2607. 16238v1 Announce Type: cross Abstract: Predicting droplet evolution in material jetting, or Inkjet Printing (IJP), is essential for maintaining printing quality.

By Jinghao Cao, Minsung Kang, Hongyue Sun, Chi Zhou, Jihoon Chung, Xubo Yue, Sanchoy Das, Bo Shen

diffusion benchmarks

arXiv AI

Jul 21

Evidence Interfaces Shape How Retrieval-Augmented Readers Use Support

arXiv:2607. 17108v1 Announce Type: new Abstract: In multi-hop RAG evaluation, a top-k answer score can hide two different failures: the retrieval window may drop part of the support chain, or it may contain support in a form the adapted reader does not use well.

By Junchi Liao, Jiawen Deng, Fuji Ren

rag benchmarks

arXiv Machine Learning

Jul 21

LogicIF: Towards Complex Logic Instruction Following

arXiv:2508. 09125v3 Announce Type: replace-cross Abstract: Instruction following has catalyzed the recent era of Large Language Models (LLMs) and is the foundational skill underpinning more advanced capabilities such as reasoning and agentic behaviors.

By Mian Zhang, Shujian Liu, Sixun Dong, Ming Yin, Yebowen Hu, Xun Wang, Simin Ma, Song Wang, Sathish Reddy Indurthi, Haoyun Deng, Zhiyu Zoey Chen, Kaiqiang Song

llms agents benchmarks

arXiv Machine Learning

Jul 21

Program Synthesis for Simulation-Based Inference: Joint Model Selection and Parameter Estimation

arXiv:2607. 17540v1 Announce Type: new Abstract: Neural simulation-based inference enables parameter estimation for complex models, but typically requires the user to specify a simulator encoding a fixed model structure.

By Siddharth Mishra-Sharma

llms benchmarks

arXiv Machine Learning

Jul 21

Explaining and Tuning Transformer-based LLMs in Arithmetic Tasks with Human Strategies

arXiv:2607. 17166v1 Announce Type: new Abstract: Transformer-based large language models (LLMs) continue to achieve state-of-the-art performance across various natural language processing tasks.

By Luyu Qiu, Jianing Li, Hwanhee Kim, Xiaoyong Wei, Yueyuan Zheng, Janet Hsiao, Lei Chen

llms nlp benchmarks safety

arXiv Machine Learning

Jul 21

ANNLib: A Development Framework for Efficient Approximate Nearest Neighbor Search

arXiv:2607. 17582v1 Announce Type: new Abstract: Approximate Nearest Neighbor Search (ANNS) plays a pivotal role in modern deep learning pipelines.

By Zheqi Shen, Jingbo Su, Zijin Wan, Yan Gu, Yihan Sun

benchmarks

arXiv AI

Jul 21

STAR: Skeletal Token Alignment and Rearrangement for Interaction Recognition

arXiv:2607. 17342v1 Announce Type: cross Abstract: Understanding physical human-robot and human-human interactions is a challenging yet emerging topic in 3D vision.

By Yuhang Wen, Mengyuan Liu, Zixuan Tang, Junsong Yuan, Sirui Li, Beichen Ding

diffusion robotics benchmarks safety

arXiv Machine Learning

Jul 21

WINO: A Weak-Form Physics Informed Neural Operator for Hyperelasticity on Variable Domains

arXiv:2605. 24651v2 Announce Type: replace-cross Abstract: We propose a Weak-form Physics-Informed Neural Operator (WINO), a data-free framework that combines the efficiency of neural operators with the geometric flexibility of the $\varphi$-finite element method ($\varphi$-FEM).

By Bokai Zhu, Yizheng Wang, Qinghui Zhang, Timon Rabczuk

benchmarks

arXiv AI

Jul 21

Trustworthy Protein-Ligand Binding Affinity Prediction via Reliability-Aware Multi-Engine Fusion

arXiv:2607. 17601v1 Announce Type: cross Abstract: Accurate protein-ligand binding affinity prediction is central to computational drug discovery, yet modern docking engines frequently disagree without indicating which prediction to trust.

By Yongchan Hong, Defu Cao, Wenjin Liu, Thomas Ku, Jordy Homing Lam, Emily Nguyen, Willie Neiswanger, Vsevolod Katritch, Yan Liu

benchmarks

arXiv AI

Jul 21

ThAME: 3D Memory-Enabled Heterogeneous Accelerator for LLM Mixture of Experts

arXiv:2607. 17074v1 Announce Type: cross Abstract: Mixture of Experts (MoE) architectures have emerged as a dominant paradigm for scaling Large Language Models (LLMs).

By Pratyush Dhingra, Pramit Kumar Pal, Janardhan Rao Doppa, Partha Pratim Pande

llms benchmarks

arXiv Machine Learning

Jul 21

Learning Spatio-Temporal Foundation Models from Pure Synthetic Data

arXiv:2607. 16251v1 Announce Type: new Abstract: Spatio-Temporal Foundation Models (STFMs) aim to learn generalizable representations of complex dynamical systems across space and time.

By Yutong Feng, Shiyuan Piao, Yutong Xia, Xu Liu, Wenqi Fan, Fugee Tsung, See-Kiong Ng, Yuxuan Liang

diffusion benchmarks safety

arXiv Machine Learning

Jul 21

MultiLoReFT: Decoupling Shared and Modality-Specific Subspaces in Multimodal Learning via Low-Rank Representation Fine-Tuning

arXiv:2607. 16789v1 Announce Type: new Abstract: Real-world perception and decision making are inherently multimodal, integrating complementary signals across modalities.

By Sana Tonekaboni, Viktoria Schuster, Caroline Uhler

fine-tuning multimodal benchmarks safety

arXiv AI

Jul 21

Certified-Gap Dual-Price Policies for Real-Time Truckload Bid Acceptance with Relocating, Clock-Constrained Resources

arXiv:2607. 16891v1 Announce Type: cross Abstract: A truckload carrier must accept or reject each load tender within seconds.

By Aswin Chandrasekaran

benchmarks

Get the daily briefing