Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,633 stories · RSS feed

arXiv Machine Learning

Jul 21

ECG-LLM: Foundation Model for ECG-Based Cardiac Reasoning

arXiv:2607. 16323v1 Announce Type: cross Abstract: Electrocardiography (ECG) is an inexpensive, standard-of-care test for cardiac symptoms, but front-line triage often lacks immediate access to definitive imaging such as echocardiography (ECHO) or cardiac magnetic resonance (CMR).

By Alexander Selivanov, Friederike Jungmann, Jan Kehrer, Karl-Ludwig Laugwitz, Eimo Martens, Daniel Rueckert

llms multimodal benchmarks

arXiv Machine Learning

Jul 21

Spectral-Morphological Attention U-Net: An Efficient Network for Active Wildfire Detection

arXiv:2607. 16472v1 Announce Type: cross Abstract: Over the past decades, the frequency of global wildfires has been increasing steadily.

By Yugong Zeng, Jonathan Wu

computer-vision benchmarks

arXiv Machine Learning

Jul 21

Team DACTYL at PAN 2026: Bayesian Data Mixing and Empirical X-risk Minimization for AI-text Detection

arXiv:2607. 17382v1 Announce Type: cross Abstract: Existing research shows that AI-generated text detection classifiers achieve strong in-distribution (ID) performance but do not maintain the same performance on out-of-distribution (OOD) texts, suggesting overfitting to dataset-specific features.

By Shantanu Thorat

fine-tuning benchmarks

arXiv AI

Jul 21

WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football Forecasting

arXiv:2607. 18084v1 Announce Type: new Abstract: Predicting a football match before kickoff requires more than knowing past results: a model must use changing information and make a clear prediction before the answer is available.

By Zhaokai Wang, Tianlin Gui, Jiayuan Rao, Shangzhe Di, Yihong Tang, Dingli Liang

llms agents benchmarks

arXiv AI

Jul 21

LookME: Lookup-Based Multimodal Embeddings for Layer Injection in Vision-Language Models

arXiv:2607. 16305v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have achieved strong progress in multimodal understanding.

By Zeyu Xu, Xingzhong Hou, Pengkai Guo, Siling Lin, Xiao Xu, Menghua Zhai, Haoyu Chen, Yunke Zhang, Fei Huang

llms rag multimodal benchmarks

arXiv AI

Jul 21

A Deep Reinforcement Learning Algorithm for the Vehicle Routing Problem with Stochastic Demands and Outsourcing

arXiv:2607. 16875v1 Announce Type: cross Abstract: We introduce the vehicle routing problem with stochastic demands and outsourcing options (VRP-SDO), in which a logistics service provider partitions customer requests into customers outsourced to a common carrier and customers committed to its fixed fleet.

By Mohsen Dastpak, Fausto Errico, Ola Jabali

reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jul 21

The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric

arXiv:2607. 18237v1 Announce Type: cross Abstract: Human visual similarity judgments are context-dependent.

By Sheng-Yu Wang, Yotam Nitzan, Aaron Hertzmann, Jun-Yan Zhu, Eli Shechtman, Alexei A. Efros, Richard Zhang

llms fine-tuning multimodal benchmarks

arXiv Machine Learning

Jul 21

FlashRT: Agent Harness for Guiding Agents to Deploy Real-Time Multimodal Applications

arXiv:2607. 18171v1 Announce Type: new Abstract: Real-time multimodal applications, including voice agents and interactive video generation, compose heterogeneous models into pipelines whose efficient deployment requires application-specific decisions about placement, streaming, and intra-model parallelism.

By Krish Agarwal, Zhuoming Chen, Yanyuan Qin, Zhenyu Gu, Atri Rudra, Beidi Chen

llms agents diffusion efficiency multimodal benchmarks

arXiv AI

Jul 21

Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits

arXiv:2607. 16646v1 Announce Type: cross Abstract: Large language models now translate natural-language descriptions of decision problems into solver-ready optimization models, but they fail silently.

By Haifeng Li, Mo Hai

llms benchmarks

arXiv AI

Jul 21

Retain or Consolidate? Budget-Dependent Operator Selection for Language Agent Memory

arXiv:2607. 17545v1 Announce Type: new Abstract: Language agents depend on memory across interactions.

By Qingcan Kang, Mingyang Liu, Shixiong Kai, Kaichao Liang, Zhentao Tang, Yuqi Cui, Tao Zhong, Mingxuan Yuan

llms agents benchmarks safety

arXiv Machine Learning

Jul 21

Right Knowledge, Wrong Answer: Characterizing Parametric Temporal Conflict in Open-Weight Language Models

arXiv:2606. 20959v2 Announce Type: replace Abstract: Language models may encode both outdated facts and their newer replacements.

By Elias Hossain, Sourav Saha, Tasfia Nuzhat Ornee, Sanjeda Sara Jennifer, Umesh Chandra Biswas, Shubhashis Roy Dipta, Rajib Rana, Niloofar Yousefi

llms benchmarks

arXiv AI

Jul 21

InertialAR: Autoregressive 3D Molecule Generation with Inertial Frames

arXiv:2510. 27497v2 Announce Type: replace-cross Abstract: Transformer-based autoregressive models have emerged as a unifying paradigm across modalities such as text and images, but their extension to 3D molecule generation remains underexplored.

By Haorui Li, Weitao Du, Yuqiang Li, Hongyu Guo, Shengchao Liu

llms diffusion nlp benchmarks

arXiv Machine Learning

Jul 21

ExMAG: Learning of Maximally Ancestral Graphs

arXiv:2503. 08245v4 Announce Type: replace Abstract: In mixed graphs, there are both directed and bidirected edges.

By Petr Ry\v{s}av\'y, Pavel Ryt\'i\v{r}, Xiaoyu He, Georgios Korpas, Jakub Mare\v{c}ek

benchmarks

arXiv AI

Jul 21

Coordinated Disentanglement with Iterative Mode Discovery Under Hidden Correlations

arXiv:2607. 17264v1 Announce Type: new Abstract: Disentangled representation learning is a powerful paradigm for robust attribute prediction.

By Rong Hu, Ling Chen

benchmarks

arXiv AI

Jul 21

Generative Ontology Induction: Domain-Agnostic Schema Discovery from Document Corpora Using Large Language Models

arXiv:2607. 16201v1 Announce Type: new Abstract: Ontology engineering remains a critical bottleneck in knowledge-intensive AI systems.

By Sergei Sergienko

llms benchmarks

arXiv AI

Jul 21

Do Maps Still Matter for Machines: Revisiting the Role of Choropleth Maps in Foundation Model Spatial Understanding

arXiv:2607. 17999v1 Announce Type: new Abstract: Spatial understanding is crucial for foundation models (FMs), and maps have long helped humans organize and reason about geographic information.

By Zhiwei Wei, Yonghe Sun, Zhenjia Liu, Wenjia Xu, Chao He, Weihua Dong, Chunbo Liu, Hua Liao

llms benchmarks

arXiv AI

Jul 21

Mechanistic Attention Guidance for Agent Memory Refinement

arXiv:2607. 17621v1 Announce Type: new Abstract: Existing self-evolving memory systems mainly improve agent memory based on textual outputs, such as task trajectories and reflections.

By Yechao Hong, Haiquan Qiu, Yaqing Wang, Quanming Yao

agents benchmarks

arXiv AI

Jul 21

A Geometric Perspective on Stabilizing Value Conflict Resolution

arXiv:2607. 17946v1 Announce Type: cross Abstract: Large Language Models (LLMs) often struggle to navigate value conflicts when trained with the compressed scalar rewards of Reinforcement Learning from Human Feedback (RLHF).

By Saket Reddy, Andy Liu

llms reinforcement-learning benchmarks safety

arXiv AI

Jul 21

Beyond Semantic Equivalence: Logical Graphs for LLM Uncertainty Quantification

arXiv:2607. 16868v1 Announce Type: new Abstract: Large Language Models (LLMs) often produce confidently stated yet unreliable outputs, posing critical challenges for deployment in safety-sensitive applications.

By Yanni Dong, Minghua Liu, Meiling Zhu, Xiaowei Huang, Lijun Zhang

llms benchmarks safety

arXiv AI

Jul 21

Depth Estimators Are Implicit Neural Fields for 3D Scene Geometry Inpainting and Reconstruction

arXiv:2607. 16286v1 Announce Type: cross Abstract: The 3D geometry of real-world scene data is often incomplete.

By Yingzhao Jian, Zihao Lin, Hehe Fan

benchmarks

Get the daily briefing