Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,849 stories · RSS feed

arXiv AI

Jul 20

AEGIS: Assay-Aware Protocol Validation and Runtime Monitoring for Open-Source Liquid Handling Robots

arXiv:2607. 15620v1 Announce Type: cross Abstract: Self-driving laboratories increasingly rely on low-cost liquid handlers such as the Opentrons OT-2, which ship without the pressure-based aspiration monitoring of Hamilton or Tecan systems and are typically run open-loop.

By Priyanka V. Setty, Arvind Ramanathan, Ian Foster, Rick Stevens

llms computer-vision robotics multimodal benchmarks

arXiv Machine Learning

Jul 20

MLLM-DataEngine: Closing the Loop of Multimodal Instruction Tuning Data Generation

arXiv:2607. 15299v1 Announce Type: cross Abstract: In this paper, we propose MLLM-DataEngine, a novel closed-loop system that bridges data generation, model training, and evaluation.

By Zhiyuan Zhao, Bin Wang, Linke Ouyang, Yiqi Lin, Pan Zhang, Xiaoyi Dong, Jiaqi Wang, Conghui He

llms fine-tuning multimodal benchmarks

arXiv AI

Jul 20

AI Trading: Evaluating Large Language Models for Technical Market Analysis

arXiv:2607. 15414v1 Announce Type: cross Abstract: Large Language Models (LLMs) have emerged as powerful tools for processing the heterogeneous information environments of modern financial markets.

By Geofrey Ntale

llms fine-tuning benchmarks safety

arXiv Machine Learning

Jul 20

Intentional Electromagnetic Interference Attacks on Facial Recognition

arXiv:2607. 15512v1 Announce Type: cross Abstract: Attacks on general computer vision algorithms are often relegated to the digital domain, with the optimization performed purely in the digital world and then translated to physical mediums for implementation.

By Tyler Fitzsimmons, Adam Czajka

computer-vision benchmarks

arXiv Machine Learning

Jul 20

Adaptive Multi-Step Lookahead Decoding for Diffusion Language Models

arXiv:2607. 15655v1 Announce Type: cross Abstract: Masked diffusion language models (DLMs) enable parallel text generation by iteratively refining masked tokens, offering a promising alternative to autoregressive decoding.

By Yingqian Cui, Wei Deng, Lantao Mei, Hang Li, Charu C. Aggarwal, Hui Liu, Yue Xing

llms diffusion benchmarks

arXiv AI

Jul 20

Why do CNNs excel at feature extraction? A mathematical explanation

arXiv:2307. 00919v2 Announce Type: replace-cross Abstract: Over the past decade deep learning has revolutionized the field of computer vision, with convolutional neural network models proving to be very effective for image classification benchmarks.

By Vinoth Nandakumar, Arush Tagade, Tongliang Liu

computer-vision benchmarks

arXiv Machine Learning

Jul 20

Trainable Spline Representations for Physics-Informed Learning

arXiv:2607. 15751v1 Announce Type: new Abstract: This work introduces Physics-Informed Splines (PI-Splines), a structured spline-based architecture for physics-informed learning.

By Giovanni Canali, Nicola Demo, Gianluigi Rozza

benchmarks

arXiv AI

Jul 20

Ask Twice, Look Twice: Prompt Echoing Resolves the Question-First Paradox in Vision-Language Models

arXiv:2607. 15565v1 Announce Type: cross Abstract: Where should the question go in a vision-language model (VLM) prompt: before the image or after it?

By Rakshanda Hassan Abhinandan, John Galeotti, Deva Ramanan, Gautam Rajendrakumar Gare

llms nlp fine-tuning multimodal benchmarks

arXiv AI

Jul 20

Orbis 2: A Hierarchical World Model for Driving

arXiv:2607. 15898v1 Announce Type: cross Abstract: Current world models operate at a single level of abstraction, with most prioritizing perceptual fidelity while lacking the spatial reasoning and semantic understanding required for real-world downstream tasks.

By Sudhanshu Mittal, Arian Mousakhan, Silvio Galesso, Karim Farid, Jonannes Dienert, Rajat Sahay, Thomas Brox

diffusion benchmarks

arXiv AI

Jul 20

Debiasing Text-to-Image Evaluation via Implicit Cultural Alignment Reward Modeling

arXiv:2607. 15740v1 Announce Type: cross Abstract: As Text-to-Image (T2I) systems rapidly advance, evaluating the cultural authenticity of synthesized content has become increasingly important for fair and trustworthy generative AI.

By Bo-An Chang, Yu-Chih Chen

llms diffusion nlp reinforcement-learning multimodal benchmarks safety

arXiv Machine Learning

Jul 20

Learning Reach-Avoid Task with Reinforcement Learning: Vectorized Simulation and Benchmark

arXiv:2607. 15935v1 Announce Type: cross Abstract: Deep reinforcement learning (DRL) has a longstanding tradition in addressing the reach-avoid task problem, especially for controlling robotic arms.

By Jonas Weihing, Shahram Eivazi

agents reinforcement-learning robotics benchmarks

arXiv Machine Learning

Jul 20

Manifold Dimension Estimation via Local Graph Structure

arXiv:2510. 15141v5 Announce Type: replace-cross Abstract: Most existing manifold dimension estimators rely on the assumption that the underlying manifold is locally flat within the neighborhoods under consideration.

By Zelong Bi, Pierre Lafaye de Micheaux

rag benchmarks

arXiv AI

Jul 20

GeoChrono: Benchmarking and Rethinking Long-Term Temporal Understanding in Remote Sensing

arXiv:2607. 15768v1 Announce Type: cross Abstract: Remote sensing offers an unparalleled vantage point for observing the Earth's long-term surface evolution, yet it demands that a model not only perceive land cover at isolated moments, but also track changes, memorize evolution histories, and reason across time and space.

By Yujie Li, Jiancheng Pan, Zhiwei Wei, Jiuniu Wang, Mugen Peng, Wenjia Xu

benchmarks

arXiv Machine Learning

Jul 20

A Standardized Benchmark for Skeleton-Based Rehabilitation Assessment Using Deep Learning

arXiv:2507. 21018v2 Announce Type: replace-cross Abstract: Automated assessment of human motion plays a vital role in rehabilitation, enabling objective evaluation of patient performance and progress.

By Ali Ismail-Fawaz, Maxime Devanne, Stefano Berretti, Jonathan Weber, Germain Forestier

benchmarks

arXiv AI

Jul 20

Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching

arXiv:2607. 15516v1 Announce Type: cross Abstract: Production LLM deployments combine two cost-reduction primitives: prompt caching (a discounted rate for re-used token prefixes) and prompt compression (fewer tokens sent).

By Yan Song

llms rag benchmarks

arXiv Machine Learning

Jul 20

Analysis of Semi-Supervised Learning on Hypergraphs

arXiv:2510. 25354v3 Announce Type: replace Abstract: Hypergraphs provide a natural framework for modeling multiway interactions.

By Adrien Weihs, Andrea L. Bertozzi, Matthew Thorpe

benchmarks

arXiv AI

Jul 20

ToolVerse: Unlocking Massive Environments and Long-Horizon Tasks for Agentic Reinforcement Learning

arXiv:2607. 15660v1 Announce Type: new Abstract: While LLM agents demonstrate strong reasoning abilities in compact and well-defined scenarios, they struggle to maintain robustness and effectiveness when faced with large-scale, diverse, and dynamic real-world environments that demand seamless tool integration.

By Shuaiyu Zhou, Fengpeng Yue, Zengjie Hu, Yuanzhe Shen, Chenyang Zhang, feng hong, Cao Liu, Ke Zeng

llms agents reinforcement-learning benchmarks

arXiv AI

Jul 20

MGDT: MLLM-Guided Diffusion Transformer with Relation-Adaptive Mixture-of-Experts for Multimodal Knowledge Graph Completion

arXiv:2607. 15592v1 Announce Type: new Abstract: Multimodal Knowledge Graph Completion (MKGC) requires inferring missing entities from structural, textual, and visual cues.

By Xu Hou, Meiyu Liang, Wei Huang, Yawen Li, Zhe Xue, Wu Liu, Guanhua Ye, Lei Shi, Kangkang Lu

llms diffusion multimodal benchmarks safety

arXiv Machine Learning

Jul 20

Multi-marginal temporal Schr\"odinger Bridge Matching from unpaired data

arXiv:2510. 01894v3 Announce Type: replace Abstract: Many natural dynamic processes -- such as in vivo cellular differentiation or disease progression -- can only be observed through the lens of static sample snapshots.

By Thomas Gravier, Thomas Boyer, Auguste Genovesio

diffusion benchmarks

arXiv AI

Jul 20

Loop the Loopies!

arXiv:2607. 16051v1 Announce Type: cross Abstract: We present Loopie, the most powerful looped Transformer to date.

By Zitian Gao, Yilong Chen, Yihao Xiao, Xinyu Yang, Ran Tao, Joey Zhou, Bryan Dai

llms benchmarks

Get the daily briefing