Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,077 stories · RSS feed

arXiv AI

Jul 21

OFD-Net: Teacher-Free Reliable Semi-supervised Medical Image Segmentation with Orthogonal Feature Disentanglement Net of Foreground-Background

arXiv:2607. 16705v1 Announce Type: cross Abstract: Semi-supervised learning (SSL) is an effective solution for medical image segmentation with limited annotations.

By Shao-feng Jiang, Zhe-yang Jing, Qin Lu, Huan-huan Shi, Zhen Chen, Cong-xuan zhang, Chen Yi

computer-vision benchmarks safety

arXiv AI

Jul 21

FST.ai 2.5: Explainable and Uncertainty-Aware AI for Olympic and Para-Taekwondo Decision Support, Athlete Digital Twins, and Federation-Scale Analytics

arXiv:2607. 16597v1 Announce Type: new Abstract: The rapid digitalisation of elite sport has created new opportunities for integrating artificial intelligence (AI), performance analytics, and decision-support systems into athlete development and competition management.

By Keivan Shariatmadar, Ahmad Osman, Ramin Rey

benchmarks

arXiv AI

Jul 21

CoCurve: Cross-Module Co-Pruning Curvature for Training-Free Structured LLM Pruning

arXiv:2607. 17568v1 Announce Type: cross Abstract: Structured pruning compresses large language models (LLMs) by removing whole computational units, such as attention heads and feed-forward (FFN) channel groups.

By Zhiren Gong, Zihao Zeng, Zijie Wang, Tiantong Wang, Chau Yuen, Wei Yang Bryan Lim

llms fine-tuning efficiency benchmarks

arXiv AI

Jul 21

Measuring and Improving Complex-Atomic Answer Consistency in Endoscopic VQA

arXiv:2607. 17834v1 Announce Type: cross Abstract: Endoscopic visual question answering (VQA) increasingly asks complex questions that combine several endoscopic answer components rather than isolated factual queries.

By Yuhao Liu, Cheng Zhao, Guanghui Yue

nlp multimodal benchmarks

arXiv Machine Learning

Jul 21

Token-Level Off-Policy Learning for Faithful Generation Under Distribution Shift

arXiv:2607. 17524v1 Announce Type: cross Abstract: We propose Token-Level Off-Policy Labeling (TOPL), an off-policy training paradigm that reframes post-training as a token-level correctness prediction task.

By Zitong Huang, Gustavo Lucas Carvalho, Deqing Fu, Robin Jia

nlp fine-tuning benchmarks

arXiv AI

Jul 21

Multi-modal cross-domain mixed fusion model with dual disentanglement for fault diagnosis under unseen working conditions

arXiv:2512. 24679v2 Announce Type: replace Abstract: Intelligent fault diagnosis has become an indispensable technique for ensuring machinery reliability.

By Pengcheng Xia, Yixiang Huang, Chengjin Qin, Chengliang Liu

fine-tuning multimodal benchmarks

arXiv AI

Jul 21

UnMaskFork: Test-Time Scaling for Masked Diffusion via Deterministic Action Branching

arXiv:2602. 04344v2 Announce Type: replace-cross Abstract: Test-time scaling strategies have effectively leveraged inference-time compute to enhance the reasoning abilities of Autoregressive Large Language Models.

By Kou Misaki, Takuya Akiba

llms diffusion benchmarks

arXiv AI

Jul 21

Empirical Grounding Improves the Realism of LLM Agents Simulating Human Behavior During Disruptions

arXiv:2607. 17437v1 Announce Type: new Abstract: Large language model (LLM) agents offer a generative approach to simulating human behavior under conditions that may have few or no direct historical analogues, a common challenge in disaster and infrastructure-disruption planning.

By Chen Xia, Zexi Kuang, Yuqing Hu

llms agents benchmarks

arXiv AI

Jul 21

Sparse Evidence Can Suffice: Agentic Evidence Seeking for Multimodal Video Misinformation Detection

arXiv:2607. 18080v1 Announce Type: cross Abstract: Multimodal video misinformation detection is commonly formulated as a holistic video-understanding task, where the entire video and its associated content are processed and judged in a single pass.

By Haochen Zhao, Yongxiu Xu, Xinkui Lin, Dong Xie, Jiarui Lu, Yuqi Qian, Yubin Wang, Hongbo Xu, Gaopeng Gou

agents reinforcement-learning multimodal benchmarks

arXiv AI

Jul 21

OpenMHC: Accelerating the Science of Wearable Foundation Models

arXiv:2607. 16235v1 Announce Type: cross Abstract: Mobile and wearable devices offer an unprecedented opportunity for continuous, passive health monitoring and active health coaching.

By Narayan Schuetz, Yuze Bai, Lianggang Pan, Edgar Eggert, Favour Nerrise, Juan Delgado-SanMartin, Max Rosenblattl, Milana Gurbanova, Mohammad Asadi, Anders Johnson, Paul Schmiedmayer, Dennis Wang, Allan Lawrie, Daniel Seung Kim, Xin Liu, Akshay Paruchuri, Ehsan Adeli, Euan Ashley, Kelly W. Zhang

benchmarks

arXiv AI

Jul 21

PEARL: Auditable Repair for Scientific Reasoning Graph Extraction

arXiv:2607. 17917v1 Announce Type: new Abstract: Scientific Reasoning Graph Extraction (SRGE) aims to recover explicit links among observations, evidence, intermediate claims, and paper-level conclusions.

By Bohan Su, Pengze Li, Yuchen Lu, Xi Chen

llms agents benchmarks

arXiv AI

Jul 21

LEGO Co-builder: Exploring Fine-Grained Vision-Language Modeling for Multimodal LEGO Assembly Assistants

arXiv:2507. 05515v3 Announce Type: replace Abstract: Vision-language models (VLMs) are facing the challenges of understanding and following multimodal assembly instructions, particularly when fine-grained spatial reasoning and precise object state detection are required.

By Haochen Huang, Yue Su, Xin Sun, Moonisa Ahsan, Mohammad Aliannejadi, Irene Viola, Zhaochun Ren, Chuang Yu, Aneta Lisowska, Artem Belopolsky, Koen Hindriks, Pablo Cesar, Junxiao Wang, Jiahuan Pei

llms computer-vision fine-tuning multimodal benchmarks

arXiv AI

Jul 21

TRACE: Trajectory-Based Safety Patch Learning for LLM Post-Training Realignment

arXiv:2607. 16242v1 Announce Type: cross Abstract: Fine-Tuning-as-a-Service (FTaaS) platforms let users train large language models (LLMs) on customized tasks, but this pipeline could erode models' safety alignment.

By Changyue Li, Jiaming He, Youliang Yuan, Jialin Wu, Boxi Yu, Zhicong Huang, Pinjia He

llms fine-tuning benchmarks safety

arXiv AI

Jul 21

Normalized Rewards for Preference Optimization

arXiv:2607. 16240v1 Announce Type: cross Abstract: Direct Alignment Algorithms (DAAs) such as DPO have become a common way to post-train and align LLMs with human preferences.

By Shawn Im, Federico Danieli, Skyler Seto, Barry-John Theobald, Katherine Metcalf

llms reinforcement-learning benchmarks safety

arXiv AI

Jul 21

When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMs

arXiv:2510. 22228v2 Announce Type: replace-cross Abstract: Layer pruning has emerged as a widely adopted technique for improving the efficiency of large language models (LLMs).

By Keyu Wang, Tian Lyu, Guinan Su, Lu Yin, Marco Canini, Jonas Geiping, Shiwei Liu

llms fine-tuning efficiency benchmarks

arXiv AI

Jul 21

FormulaCode: Evaluating Agentic Optimization on Large Codebases

arXiv:2603. 16011v3 Announce Type: replace-cross Abstract: Large language model (LLM) coding agents increasingly operate at the repository level, motivating benchmarks that evaluate their ability to optimize entire codebases under realistic constraints.

By Atharva Sehgal, James Hou, Akanksha Sarkar, Ishaan Mantripragada, Swarat Chaudhuri, Jennifer J. Sun, Yisong Yue

llms agents benchmarks

arXiv AI

Jul 21

Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification

arXiv:2603. 26648v3 Announce Type: replace-cross Abstract: Recent advances in large language models have improved the capabilities of coding agents, yet systematic evaluation of complex, end-to-end website development remains limited.

By Zehai He, Wenyi Hong, Zhen Yang, Ziyang Pan, Mingdao Liu, Xiaotao Gu, Jie Tang

llms agents multimodal benchmarks

arXiv AI

Jul 21

HyCoRec: Hypergraph-Enhanced Multi-Preference Learning for Alleviating Matthew Effect in Conversational Recommendation

arXiv:2607. 17461v1 Announce Type: cross Abstract: The Matthew effect is a notorious issue in Recommender Systems (RSs), \emph{i.

By Yongsen Zheng, Ruilin Xu, Ziliang Chen, Guohua Wang, Mingjie Qian, Jinghui Qin, Liang Lin

benchmarks

arXiv AI

Jul 21

WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football Forecasting

arXiv:2607. 18084v1 Announce Type: new Abstract: Predicting a football match before kickoff requires more than knowing past results: a model must use changing information and make a clear prediction before the answer is available.

By Zhaokai Wang, Tianlin Gui, Jiayuan Rao, Shangzhe Di, Yihong Tang, Dingli Liang

llms agents benchmarks

arXiv AI

Jul 21

LookME: Lookup-Based Multimodal Embeddings for Layer Injection in Vision-Language Models

arXiv:2607. 16305v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have achieved strong progress in multimodal understanding.

By Zeyu Xu, Xingzhong Hou, Pengkai Guo, Siling Lin, Xiao Xu, Menghua Zhai, Haoyu Chen, Yunke Zhang, Fei Huang

llms rag multimodal benchmarks

Get the daily briefing