Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,257 stories · RSS feed

arXiv Machine Learning
Jul 7

The Moving Target: A Longitudinal Audit of Trustworthiness Drift Across Twelve Checkpoints of Open-Source Chat LLMs

arXiv:2607. 02587v1 Announce Type: cross Abstract: Model cards quote trust-benchmark scores without recording when they were measured, and the same number is routinely carried across successive checkpoints of one release line as if the model behind it had not shifted.

By Zhichao Fan, Yanhang Li, Zexin Zhuang, Xian Sun, Yingshuo Wang
arXiv Machine Learning
Jul 7

Qantara: Bridge-Flow Training for Multi-Paradigm JEPA Control

arXiv:2607. 04978v1 Announce Type: new Abstract: Joint-Embedding Predictive Architectures (JEPAs) underpin a growing family of latent world models for control from raw pixels, but every existing JEPA world model commits at training time to a single inference paradigm: either trajectory optimisation in a learned dynamics model, or direct behaviour cloning.

By Ruslan Rakhimov, George Bredis, Yuriy Maksyuta, Daniil Gavrilov
arXiv AI
Jul 7

PosterHarness: Turning Scientific Poster Generation into an Auditable Instruction-Following Benchmark

arXiv:2607. 03006v1 Announce Type: cross Abstract: Text-rich image models can now design poster-scale layouts, but we lack ways to measure whether they honor scientific communication contracts: legible labels, prescribed aspect ratios, and -- above all -- abstaining from fabricated scientific figures.

By Tianyi Yang, Dawei Fu, Youpeng Wu, Zixun Kou, Linrui Chen, Ruobing Jiang, Zijian Wang, Qiang Li
arXiv AI
Jul 7

S-EMBER: A Large-Scale Benchmark for Streaming Egocentric Memory Retrieval

arXiv:2607. 02689v1 Announce Type: cross Abstract: As wearable devices enable continuous first-person recording, AI assistants must reason across long time horizons to recall past experiences-a capability known as episodic memory.

By Xiaodong Wang, Xuanyi Zhao, Pedro Rodriguez, Devendra Singh Sachan, Barlas Oguz, Seungwhan Moon, Shang-Wen Li, Gargi Ghosh, Xin Dong, Wen-Tau Yih
arXiv Machine Learning
Jul 7

Channel-Adaptive Robust Aggregation for Over-the-Air Federated Learning in Heterogeneous Networks

arXiv:2607. 04218v1 Announce Type: new Abstract: The growing demand for privacy-preserving, data-intensive applications such as IoT, augmented reality, and autonomous systems positions Federated Learning (FL) as a key enabler in 6G networks.

By Zubaida Fatima, Zubair Shaban, Yusuf Jamal, Nazreen Shah, Ranjitha Prasad, B. N. Bharath
arXiv AI
Jul 7

STELLA: Efficient Sensor-to-LLM Translation for On-Device Human Activity Recognition

arXiv:2607. 03089v1 Announce Type: cross Abstract: HAR is increasingly expected to run continuously on edge devices, yet recent LLM-based methods remain hard to deploy: raw sensor prompts are long, cloud inference adds latency and privacy risk, and fine-tuned LLM pipelines turn general-purpose models into task-specific classifiers.

By Nirhoshan Sivaroopan, Albert Zomaya, Kanchana Thilakarathna