Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,849 stories · RSS feed

arXiv Machine Learning

Jul 20

Rethinking the Global Knowledge of CLIP in Training-Free Open-Vocabulary Semantic Segmentation

arXiv:2502. 06818v4 Announce Type: replace Abstract: Recent works modify CLIP to perform open-vocabulary semantic segmentation in a training-free manner (TF-OVSS).

By Jingyun Wang, Cilin Yan, Guoliang Kang

rag computer-vision benchmarks

arXiv Machine Learning

Jul 20

Cost-efficient generative AI summarization for scalable automated essay scoring in educational assessment

arXiv:2607. 15829v1 Announce Type: cross Abstract: Automated essay scoring (AES) enables scalable assessment and timely feedback but remains challenged by transformer input-length limitations, which can cause information loss when processing long essays.

By Haowei Hua

llms nlp benchmarks safety

arXiv AI

Jul 20

Modularized Dynamic-Granularity Video LLM for Multi-Event Long Video Understanding

arXiv:2607. 15778v1 Announce Type: cross Abstract: Video Large Language Models (Video LLMs) have made significant advancements in various video understanding tasks.

By Wei Feng, Xin Wang, Yu-Wei Zhan, Yuwei Zhou, Wenwu Zhu

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jul 20

An Embarrassingly Simple Way to Optimize Orthogonal Matrices at Scale

arXiv:2602. 14656v2 Announce Type: replace Abstract: Orthogonality constraints are ubiquitous in robust and probabilistic machine learning.

By Adri\'an Javaloy, Antonio Vergari

benchmarks

arXiv AI

Jul 20

S1-Omni: A Unified Multimodal Reasoning Model for Scientific Understanding, Prediction, and Generation

arXiv:2607. 15686v1 Announce Type: new Abstract: We present S1-Omni, a unified multimodal reasoning model for scientific understanding, prediction, and generation.

By Jiahao Zhao, Junyi Liu, Lifeng Xu, Nan Xu, Qingli Wang, Qingxiao Li, Tianle Chen, Xiaoyu Wu, Yawen Zheng, Zikai Wang, Guanming Liu, Hequn Zhou, Jingyi Wang, Jingyuan Shu, Keqi Wang, Li He, Songyang Diao, Wenhui Xu, Xinyu Ren, Yaqin Fan, Yujin Zhou, Zhanao Yao

llms diffusion multimodal benchmarks safety

arXiv AI

Jul 20

A Formally Grounded ODRL Evaluator: Implementation and Comparison

arXiv:2607. 15987v1 Announce Type: new Abstract: The ODRL policy language is emerging as the de-facto standard for policy modelling data access and usage preferences, AI governance policies and data workflows in European dataspaces.

By Jaime Osvaldo Salas, Paolo Pareti, Adeel Aslam, Christopher Maidens, George Konstantinidis

benchmarks

arXiv Machine Learning

Jul 20

Inpainting Insights: Elevating Visual XAI with Photorealistic Perturbations

arXiv:2607. 15482v1 Announce Type: new Abstract: The increasing complexity of state-of-the-art machine learning models has made their behavior progressively harder to interpret, spurring rapid advancements in the field of eXplainable Artificial Intelligence (XAI).

By Josef Lindl, Mariana Chaves, Damien Garreau

benchmarks

arXiv AI

Jul 20

Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching

arXiv:2607. 15516v1 Announce Type: cross Abstract: Production LLM deployments combine two cost-reduction primitives: prompt caching (a discounted rate for re-used token prefixes) and prompt compression (fewer tokens sent).

By Yan Song

llms rag benchmarks

arXiv AI

Jul 20

GeoChrono: Benchmarking and Rethinking Long-Term Temporal Understanding in Remote Sensing

arXiv:2607. 15768v1 Announce Type: cross Abstract: Remote sensing offers an unparalleled vantage point for observing the Earth's long-term surface evolution, yet it demands that a model not only perceive land cover at isolated moments, but also track changes, memorize evolution histories, and reason across time and space.

By Yujie Li, Jiancheng Pan, Zhiwei Wei, Jiuniu Wang, Mugen Peng, Wenjia Xu

benchmarks

arXiv Machine Learning

Jul 20

Constrained Hebbian Learning Supports Efficient Representational Allocation under Structural Constraints

arXiv:2607. 16027v1 Announce Type: new Abstract: Introduction: Biological systems face anatomical and metabolic constraints, including costly synaptic maintenance and limited connectivity.

By Patrick Inoue, Florian R\"ohrbein, Andreas Knoblauch

rag efficiency benchmarks

arXiv AI

Jul 20

Debiasing Text-to-Image Evaluation via Implicit Cultural Alignment Reward Modeling

arXiv:2607. 15740v1 Announce Type: cross Abstract: As Text-to-Image (T2I) systems rapidly advance, evaluating the cultural authenticity of synthesized content has become increasingly important for fair and trustworthy generative AI.

By Bo-An Chang, Yu-Chih Chen

llms diffusion nlp reinforcement-learning multimodal benchmarks safety

arXiv Machine Learning

Jul 20

SC-JEPA: Stabilizing Latent Predictive Learning for Time-Series Anomaly Prediction

arXiv:2602. 04643v2 Announce Type: replace Abstract: Time-series anomaly prediction aims to forecast future system failures before they fully emerge, making latent predictive models such as JEPA a promising framework for capturing precursor dynamics.

By Yanan He, Yunshi Wen, Xin Wang, Tengfei Ma

diffusion efficiency benchmarks

arXiv Machine Learning

Jul 20

Manifold Dimension Estimation via Local Graph Structure

arXiv:2510. 15141v5 Announce Type: replace-cross Abstract: Most existing manifold dimension estimators rely on the assumption that the underlying manifold is locally flat within the neighborhoods under consideration.

By Zelong Bi, Pierre Lafaye de Micheaux

rag benchmarks

arXiv AI

Jul 20

When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models

arXiv:2603. 26556v2 Announce Type: replace-cross Abstract: Converting a pretrained Transformer into a more efficient hybrid model through distillation offers a promising approach to reducing inference costs.

By Juan Gabriel Kostelec, Qinghai Guo

llms efficiency benchmarks

arXiv AI

Jul 20

An Exam for Active Observers

arXiv:2607. 16165v1 Announce Type: cross Abstract: Human vision is a closed loop: gaze is continuously redirected by intermediate hypotheses rather than a single snapshot.

By Jiarui Zhang, Muzi Tao, Shangshang Wang, Ollie Liu, Xuezhe Ma, Willie Neiswanger

llms multimodal benchmarks

arXiv AI

Jul 20

IMBench: A Benchmark for Intuitive Robotic Manipulation

arXiv:2607. 15641v1 Announce Type: cross Abstract: Humans combine reasoning and motor control to solve complex manipulation tasks under diverse constraints.

By Anurag Maurya, Sukhvansh Jain, Prajwal Avhad, Gautham Balachandran, Ziyi Zhou, Atharva Kshirsagar, Satyam Singh, Bowen Li. Rishabh Mukund, Ritul Singh, Jatin Vira, Suvonil Chatterjee, Devesh K. Jha

llms agents robotics multimodal benchmarks

arXiv AI

Jul 20

AI Trading: Evaluating Large Language Models for Technical Market Analysis

arXiv:2607. 15414v1 Announce Type: cross Abstract: Large Language Models (LLMs) have emerged as powerful tools for processing the heterogeneous information environments of modern financial markets.

By Geofrey Ntale

llms fine-tuning benchmarks safety

arXiv AI

Jul 20

Ask Twice, Look Twice: Prompt Echoing Resolves the Question-First Paradox in Vision-Language Models

arXiv:2607. 15565v1 Announce Type: cross Abstract: Where should the question go in a vision-language model (VLM) prompt: before the image or after it?

By Rakshanda Hassan Abhinandan, John Galeotti, Deva Ramanan, Gautam Rajendrakumar Gare

llms nlp fine-tuning multimodal benchmarks

arXiv AI

Jul 20

RTL-Sequencer: Towards Scalable RTL Timing Prediction with the Sequence-based Paradigm

arXiv:2607. 15830v1 Announce Type: cross Abstract: Accurate timing prediction at the register-transfer level (RTL) is a longstanding challenge in design automation.

By Ziyan Guo, Wenji Fang, Wenkai Li, Yuchao Wu, Shang Liu, Zhiyao Xie

benchmarks

arXiv Machine Learning

Jul 20

Trainable Spline Representations for Physics-Informed Learning

arXiv:2607. 15751v1 Announce Type: new Abstract: This work introduces Physics-Informed Splines (PI-Splines), a structured spline-based architecture for physics-informed learning.

By Giovanni Canali, Nicola Demo, Gianluigi Rozza

benchmarks

Get the daily briefing