Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

20,687 stories · RSS feed

arXiv AI

Jun 24

MOCHA: Multi-modal Objects-aware Cross-arcHitecture Alignment

arXiv:2509. 14001v5 Announce Type: replace-cross Abstract: Personalized object detection aims to adapt a general-purpose detector to recognize user-specific instances from only a few examples.

By Elena Camuffo, Francesco Barbato, Mete Ozay, Simone Milani, Umberto Michieli

llms rag computer-vision efficiency multimodal benchmarks safety

arXiv AI

Jun 24

CineCap: Structured Reasoning with Spatio-Temporal Anchors for Cinematographic Video Captioning

arXiv:2606. 24636v1 Announce Type: new Abstract: Cinematographic captioning aims to describe how a video is filmed using professional film-language concepts such as camera movement, shot size, depth of field, composition, and shooting angle.

By Xinyu Mao, Yuhui Zeng, Xiaokun Liu, Wenyu Qin, Meng Wang, Xin Tao, Pengfei Wan, Xiaohan Xing, Max Meng

llms diffusion reinforcement-learning fine-tuning multimodal benchmarks

arXiv AI

Jun 24

Entity Resolution via Batched Oracle Queries

arXiv:2606. 24407v1 Announce Type: cross Abstract: We consider an oracle that processes a limited batch of records at a time and clusters those that refer to the same real-world entity.

By Lorenzo Balzotti, Donatella Firmani, Luca Gagliardelli, Giovanni Simonini

benchmarks

arXiv AI

Jun 24

Metis: Bridging Text and Code Memory for Self-Evolving Agents

arXiv:2606. 24151v1 Announce Type: cross Abstract: Self-evolving agents improve over time by distilling experience from past executions and reusing it in future tasks.

By Zijie Dai, Siuhin He, Hui Li, Qihui Zhou, Jiajun Li, Mingcong Song, Guoping Long, Hongjie Si, Xin Yao, Lin Zhang, James Cheng, Xiao Yan

agents benchmarks

arXiv Machine Learning

Jun 24

DREG: A Layer-Wise Jacobian Regularization as a General-Purpose Penalty

arXiv:2606. 23942v1 Announce Type: new Abstract: We present a large-scale empirical study isolating the contributions of the Derivative Regularization penalty (DREG).

By Rowan Martnishn

llms nlp benchmarks safety

arXiv AI

Jun 24

Tuning without Peeking: Provable Generalization Bounds and Robust LLM Post-Training

arXiv:2507. 01752v4 Announce Type: replace-cross Abstract: Gradient-based optimization is the workhorse of deep learning, offering efficient and scalable training via backpropagation.

By Ismail Labiad, Mathurin Videau, Matthieu Kowalski, Marc Schoenauer, Alessandro Leite, Julia Kempe, Olivier Teytaud

llms benchmarks safety

arXiv AI

Jun 24

Ten Digits on a Train: AI-Assisted Verification of Two Eigenvalue Problems

arXiv:2606. 23821v1 Announce Type: cross Abstract: Accurate numerical eigenvalues are often difficult to certify, especially in singular or non-normal settings.

By Matthew J. Colbrook

benchmarks

arXiv Machine Learning

Jun 24

GRACE: Gated Refinement for Accurate Causal Edge Discovery in High-Dimensional Time Series

arXiv:2606. 23880v1 Announce Type: new Abstract: From climate teleconnections to gene regulation, modern time-series datasets encompass tens or hundreds of interacting variables, making causal discovery increasingly challenging.

By Mohammad Fesanghary, Abhinav Havaldar

benchmarks

arXiv AI

Jun 24

Beyond Trajectory Imitation: Strategy-Guided Policy Optimization for LLM Reasoning

arXiv:2606. 24064v1 Announce Type: new Abstract: Distilling reasoning capabilities from strong to weak language models typically involves imitating specific solution trajectories, effectively transferring what to answer rather than how to reason.

By Tianyuan Shi, Canbin Huang, Bei Li, Xin Chen, Xiaojun Quan, Jingang Wang, Qifan Wang

llms agents reinforcement-learning fine-tuning efficiency benchmarks

arXiv AI

Jun 24

What Does ODRL Mean? A Cross-Level Ontological Grounding of Permissions, Prohibitions, and Duties in UFO-L

arXiv:2606. 24344v1 Announce Type: cross Abstract: ODRL policy evaluators produce verdicts, but say nothing about the normative positions a policy brings into existence, the authority structures those positions presuppose, or who holds the power to declare a norm violated.

By Daham M. Mustafa, Christoph Lange, Giancarlo Guizzardi, Diego Collarana, Christoph Quix, Stefan Decker

benchmarks

arXiv AI

Jun 24

HOLMES: Evaluating Higher-Order Logical Reasoning in LLMs

arXiv:2606. 23238v2 Announce Type: replace Abstract: Logical reasoning is essential for reliable AI, yet existing benchmarks are largely first-order-logic-centric, focusing on object-level deduction over fixed predicates.

By Yucheng Wu, Jundong Xu, Mingzhen Ju, Yue Yu, Chenpeng Wang, Haoxuan Li, Liangming Pan

llms benchmarks

arXiv AI

Jun 24

RetiSEM: Generalising Causal Models for Fragmented Biomedical Data

arXiv:2606. 24488v1 Announce Type: cross Abstract: Learning causal models from fragmented biomedical data is challenging because clinical, molecular, and imaging variables are often incomplete or not jointly observed.

By Inam Ullah, Imran Razzak, Shoaib Jameel

multimodal benchmarks

arXiv AI

Jun 24

SEAL: Searching Expandable Architectures for Incremental Learning

arXiv:2505. 10457v3 Announce Type: replace-cross Abstract: Incremental learning is a machine learning paradigm where a model learns from a sequential stream of tasks.

By Matteo Gambella, Manuel Roveri

efficiency benchmarks

arXiv AI

Jun 24

T2D-Bench: Evidence-Gated Evaluation of LLM Outputs for Type 2 Diabetes Using a Multi-Layer Clinical-Lifestyle Knowledge Graph

arXiv:2606. 24145v1 Announce Type: new Abstract: Large language models (LLMs) can produce clinically fluent recommendations for type 2 diabetes while failing to satisfy guideline constraints or explicitly justify lifestyle-related glycemic claims.

By Saba A. Farahani, Hung Cao, Ramesh Jain, Amir M. Rahmani

llms benchmarks safety

arXiv AI

Jun 24

Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study

arXiv:2606. 21428v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) language models are often described as ideal for resource-constrained inference.

By Alfarizy Alfarizy, Hung Truong Thanh Nguyen, Ren\'e Richard, Roozbeh Razavi-Far, Hung Cao

llms efficiency benchmarks

arXiv Machine Learning

Jun 24

FAIRVAR: Fair Federated Learning via Variance Regularization

arXiv:2508. 12042v3 Announce Type: replace Abstract: Federated learning (FL) allows collaborative training of machine learning models across multiple parties without sharing raw data.

By Zahra Kharaghani, Ali Dadras, Tommy L\"ofstedt

benchmarks safety

arXiv AI

Jun 24

When Retrieval Metrics Mislead: Measuring Policy Signal in Long-Horizon Tool-Use Agents

arXiv:2606. 23937v1 Announce Type: cross Abstract: Exact-match retrieval recall is often used as a proxy for whether a retriever supplies useful policy context to a downstream decision model.

By Tianyu Ding, Juan Pablo De la Cruz Weinstein

agents fine-tuning benchmarks

arXiv Machine Learning

Jun 24

Open-Vocabulary BEV Segmentation with 3D-Aware Geometric Constraints

arXiv:2606. 24353v1 Announce Type: cross Abstract: Bird's-eye view (BEV) perception fuses multi-camera images into a unified top-down representation for autonomous driving.

By Hojun Choi, Seulbin Hwang, Dae Jung Kim, Kisung Kim, Hyunjung Shim, Jinhan Lee

llms agents computer-vision robotics efficiency multimodal benchmarks

arXiv AI

Jun 24

Representation Interventions Enable Lifelong Knowledge Memory Control in LLMs

arXiv:2511. 20892v4 Announce Type: replace Abstract: Large language models (LLMs) often produce incorrect or outdated content after being employed.

By Xuyuan Liu, Shengyu Chen, Xinshuai Dong, Yanchi Liu, Xujiang Zhao, Haoyu Wang, Yujun Yan, Haifeng Chen, Zhengzhang Chen

llms benchmarks

arXiv AI

Jun 24

DramaDirector: Geometry-Guided Short Drama Generation

arXiv:2606. 24107v1 Announce Type: cross Abstract: Short dramas, with their rapid shot rhythms, dialogue-driven focus shifts, and demanding cinematographic grounding, pose challenges that prompt-level or text-only video generation pipelines struggle to meet.

By Hengji Zhou, Sijie Liu, Jianrun Chen, Xingchen Zou, Lianghao Xia, Liqiang Nie

agents diffusion fine-tuning benchmarks safety

Get the daily briefing