Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,849 stories · RSS feed

arXiv AI

Jul 21

HyCoRec: Hypergraph-Enhanced Multi-Preference Learning for Alleviating Matthew Effect in Conversational Recommendation

arXiv:2607. 17461v1 Announce Type: cross Abstract: The Matthew effect is a notorious issue in Recommender Systems (RSs), \emph{i.

By Yongsen Zheng, Ruilin Xu, Ziliang Chen, Guohua Wang, Mingjie Qian, Jinghui Qin, Liang Lin

benchmarks

arXiv AI

Jul 21

LEGO Co-builder: Exploring Fine-Grained Vision-Language Modeling for Multimodal LEGO Assembly Assistants

arXiv:2507. 05515v3 Announce Type: replace Abstract: Vision-language models (VLMs) are facing the challenges of understanding and following multimodal assembly instructions, particularly when fine-grained spatial reasoning and precise object state detection are required.

By Haochen Huang, Yue Su, Xin Sun, Moonisa Ahsan, Mohammad Aliannejadi, Irene Viola, Zhaochun Ren, Chuang Yu, Aneta Lisowska, Artem Belopolsky, Koen Hindriks, Pablo Cesar, Junxiao Wang, Jiahuan Pei

llms computer-vision fine-tuning multimodal benchmarks

arXiv AI

Jul 21

PEARL: Auditable Repair for Scientific Reasoning Graph Extraction

arXiv:2607. 17917v1 Announce Type: new Abstract: Scientific Reasoning Graph Extraction (SRGE) aims to recover explicit links among observations, evidence, intermediate claims, and paper-level conclusions.

By Bohan Su, Pengze Li, Yuchen Lu, Xi Chen

llms agents benchmarks

arXiv AI

Jul 21

OpenMHC: Accelerating the Science of Wearable Foundation Models

arXiv:2607. 16235v1 Announce Type: cross Abstract: Mobile and wearable devices offer an unprecedented opportunity for continuous, passive health monitoring and active health coaching.

By Narayan Schuetz, Yuze Bai, Lianggang Pan, Edgar Eggert, Favour Nerrise, Juan Delgado-SanMartin, Max Rosenblattl, Milana Gurbanova, Mohammad Asadi, Anders Johnson, Paul Schmiedmayer, Dennis Wang, Allan Lawrie, Daniel Seung Kim, Xin Liu, Akshay Paruchuri, Ehsan Adeli, Euan Ashley, Kelly W. Zhang

benchmarks

arXiv AI

Jul 21

Sparse Evidence Can Suffice: Agentic Evidence Seeking for Multimodal Video Misinformation Detection

arXiv:2607. 18080v1 Announce Type: cross Abstract: Multimodal video misinformation detection is commonly formulated as a holistic video-understanding task, where the entire video and its associated content are processed and judged in a single pass.

By Haochen Zhao, Yongxiu Xu, Xinkui Lin, Dong Xie, Jiarui Lu, Yuqi Qian, Yubin Wang, Hongbo Xu, Gaopeng Gou

agents reinforcement-learning multimodal benchmarks

arXiv AI

Jul 21

Empirical Grounding Improves the Realism of LLM Agents Simulating Human Behavior During Disruptions

arXiv:2607. 17437v1 Announce Type: new Abstract: Large language model (LLM) agents offer a generative approach to simulating human behavior under conditions that may have few or no direct historical analogues, a common challenge in disaster and infrastructure-disruption planning.

By Chen Xia, Zexi Kuang, Yuqing Hu

llms agents benchmarks

arXiv AI

Jul 21

UnMaskFork: Test-Time Scaling for Masked Diffusion via Deterministic Action Branching

arXiv:2602. 04344v2 Announce Type: replace-cross Abstract: Test-time scaling strategies have effectively leveraged inference-time compute to enhance the reasoning abilities of Autoregressive Large Language Models.

By Kou Misaki, Takuya Akiba

llms diffusion benchmarks

arXiv AI

Jul 21

Multi-modal cross-domain mixed fusion model with dual disentanglement for fault diagnosis under unseen working conditions

arXiv:2512. 24679v2 Announce Type: replace Abstract: Intelligent fault diagnosis has become an indispensable technique for ensuring machinery reliability.

By Pengcheng Xia, Yixiang Huang, Chengjin Qin, Chengliang Liu

fine-tuning multimodal benchmarks

arXiv Machine Learning

Jul 21

Token-Level Off-Policy Learning for Faithful Generation Under Distribution Shift

arXiv:2607. 17524v1 Announce Type: cross Abstract: We propose Token-Level Off-Policy Labeling (TOPL), an off-policy training paradigm that reframes post-training as a token-level correctness prediction task.

By Zitong Huang, Gustavo Lucas Carvalho, Deqing Fu, Robin Jia

nlp fine-tuning benchmarks

arXiv AI

Jul 21

Measuring and Improving Complex-Atomic Answer Consistency in Endoscopic VQA

arXiv:2607. 17834v1 Announce Type: cross Abstract: Endoscopic visual question answering (VQA) increasingly asks complex questions that combine several endoscopic answer components rather than isolated factual queries.

By Yuhao Liu, Cheng Zhao, Guanghui Yue

nlp multimodal benchmarks

arXiv AI

Jul 21

CoCurve: Cross-Module Co-Pruning Curvature for Training-Free Structured LLM Pruning

arXiv:2607. 17568v1 Announce Type: cross Abstract: Structured pruning compresses large language models (LLMs) by removing whole computational units, such as attention heads and feed-forward (FFN) channel groups.

By Zhiren Gong, Zihao Zeng, Zijie Wang, Tiantong Wang, Chau Yuen, Wei Yang Bryan Lim

llms fine-tuning efficiency benchmarks

arXiv AI

Jul 21

FST.ai 2.5: Explainable and Uncertainty-Aware AI for Olympic and Para-Taekwondo Decision Support, Athlete Digital Twins, and Federation-Scale Analytics

arXiv:2607. 16597v1 Announce Type: new Abstract: The rapid digitalisation of elite sport has created new opportunities for integrating artificial intelligence (AI), performance analytics, and decision-support systems into athlete development and competition management.

By Keivan Shariatmadar, Ahmad Osman, Ramin Rey

benchmarks

arXiv AI

Jul 21

OFD-Net: Teacher-Free Reliable Semi-supervised Medical Image Segmentation with Orthogonal Feature Disentanglement Net of Foreground-Background

arXiv:2607. 16705v1 Announce Type: cross Abstract: Semi-supervised learning (SSL) is an effective solution for medical image segmentation with limited annotations.

By Shao-feng Jiang, Zhe-yang Jing, Qin Lu, Huan-huan Shi, Zhen Chen, Cong-xuan zhang, Chen Yi

computer-vision benchmarks safety

arXiv AI

Jul 21

Integrating High-Level Requirements to Low-Level Tests with Machine-Readable V&V Specifications

arXiv:2607. 17686v1 Announce Type: cross Abstract: Modern software teams have mature tools for low-level testing, such as pytest, JUnit, and Jest, which make it inexpensive to write unit tests and run them on every commit.

By Mansur Arief, Nur Ahmad Khatim, Ali Akarma, Ahmad Alfan Alfian Irfan

agents benchmarks

arXiv AI

Jul 21

Debate-on-Graph: Reliable and Adaptive Reasoning of Large Language Model on Uncertain Knowledge Graph

arXiv:2607. 17266v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated remarkable capabilities in natural language processing.

By Peiji Yu, Xin Chen, Tianxing Wu

llms agents nlp benchmarks safety

arXiv AI

Jul 21

Autonomous Discovery of Wireless Communications Algorithms

arXiv:2607. 17762v1 Announce Type: cross Abstract: Large language model (LLM)-driven evolutionary search is an emerging algorithm-discovery paradigm that has already produced novel results in several scientific fields.

By Fay\c{c}al A\"it Aoudia, Jakob Hoydis, Sebastian Cammerer, Gian Marti, Merlin Nimier-David, Nicolas Roussel, Alexander Keller

llms agents benchmarks

arXiv AI

Jul 21

Uncovering Latent Reasoning Strategies in Language Models

arXiv:2607. 17674v1 Announce Type: cross Abstract: A language model $p_\theta(y \mid x)$ trained on reasoning tasks learns to solve problems via multiple distinct strategies, yet these strategies are implicit and entangled within the model's response distribution.

By Awni Altabaa, John Lafferty

llms benchmarks

arXiv AI

Jul 21

ST-Veto: Spatio-Temporal Token Veto for Diffusion MLLMs via Taylor Prediction and Visual Grounding

arXiv:2607. 17884v1 Announce Type: new Abstract: Vision Language Models (VLMs) achieve strong reasoning with Chain-of-Thought (CoT) prompting but incur high sequential-generation cost, error accumulation, and limited self-correction.

By Keuntae Kim, Beomseok Lee, Hyunwoo Kim, Yong Suk Choi

llms diffusion multimodal benchmarks

arXiv AI

Jul 21

Reducing Per-Sample Harm in Stochastic Optimization

arXiv:2607. 16261v1 Announce Type: cross Abstract: Modern optimizers combine gradients from the current mini-batch with historical optimization state, such as momentum or adaptive moments.

By Apostolos Avranas

computer-vision benchmarks

arXiv AI

Jul 21

Benchmarking Machine Learning Models for Multi-Omics-Based Breast Cancer Prediction

arXiv:2607. 16250v1 Announce Type: cross Abstract: Estrogen Receptor (ER) status is a critical biomarker in breast cancer diagnosis, prognosis, and treatment selection.

By Priyanka Paudel, Madan Baduwal

efficiency benchmarks

Get the daily briefing