Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

23,273 stories · RSS feed

arXiv AI

Jun 8

Which Anatomy Matters Under Limited Labels? A Data-Efficient Anatomy-Aware Benchmark for Cardiac Pathology Prediction

arXiv:2606. 06509v1 Announce Type: cross Abstract: Numerous medical imaging problems must be solved under limited labels and constrained compute, yet it remains unclear whether performance gains are driven mainly by more expressive models or by better representation of clinically meaningful anatomy.

By Himanshu Singh

computer-vision benchmarks

arXiv AI

Jun 8

Don't Pause: Streaming Video-Language Synchrony for Online Video Understanding

arXiv:2606. 06991v1 Announce Type: cross Abstract: Online Video Large Language Models (Video-LLMs) have advanced toward seamless human-AI interaction through frame-by-frame processing and proactive responding.

By Zhenyu Yang, Kairui Zhang, Shengsheng Qian, Weiming Dong, Changsheng Xu

llms benchmarks

arXiv AI

Jun 8

Standard vs. Modular Sampling: Best Practices for Reliable LLM Unlearning

arXiv:2509. 05316v2 Announce Type: replace-cross Abstract: A conventional LLM Unlearning setting consists of two subsets -"forget" and "retain", with the objectives of removing the undesired knowledge from the forget set while preserving the remaining knowledge from the retain.

By Praveen Bushipaka, Lucia Passaro, Tommaso Cucinotta

llms benchmarks safety

arXiv Machine Learning

Jun 8

Elmes*: Automated Construction of Fine-Grained Evaluation Rubrics for Large Language Models in Long-Tail Educational Scenarios

arXiv:2606. 06546v1 Announce Type: new Abstract: Evaluating large language models (LLMs) for education requires measuring how models teach, not only what they know.

By Tao Liu, Ye Lu, Ruohua Zhang, Siyu Song, Wentao Liu, Aimin Zhou, Hao Hao

llms agents benchmarks safety

arXiv Machine Learning

Jun 8

GlucoFM-Bench: Benchmarking Time-Series Foundation Models for Blood Glucose Forecasting

arXiv:2606. 06881v1 Announce Type: new Abstract: Blood glucose forecasting models are foundational for modern diabetes management systems, as reliable short-term predictions can enable proactive interventions, support automated insulin delivery, and reduce the risk of hypo- and hyperglycemic events.

By Baiying Lu, Zhaohui Liang, Ryan Pontius, Shengpu Tang, Temiloluwa Prioleau

llms benchmarks

arXiv Machine Learning

Jun 8

TALAN: Task-Aligned Latent Adaptation Networks for Targeted Post-Training of Large Language Models

arXiv:2606. 06902v1 Announce Type: new Abstract: Targeted post-training aims to improve reasoning, math, and code without degrading strengths.

By Chengkai Zhang, Ziteng Liu, Junpu Wang, Zeyi Tao, Yang Wang, Sagar Chordia, Qin Huang

llms fine-tuning benchmarks

arXiv Machine Learning

Jun 8

SEAM: Shortcut-Aware Real-Time Detection of Scripted vs. Spontaneous Speech for Interview Guardrails

arXiv:2606. 06837v1 Announce Type: cross Abstract: Scripted vs spontaneous speech detection is appealing for interview guardrails, but benchmark performance can be inflated by shortcuts tied to corpus identity, channel conditions, and recording artifacts rather than speaking style itself.

By Vsevolod (V.), Kovalev, Pranay Manocha

efficiency benchmarks safety

arXiv AI

Jun 8

Auditing Training Data in Domain-adapted LLMs: LoRA-MINT

arXiv:2606. 06946v1 Announce Type: cross Abstract: We present LoRA-MINT, a new methodology for Membership Inference Test (MINT) applied to recent Large Language Models (LLMs) fine-tuned for specific Natural Language Processing (NLP) tasks through Low-Rank Adaptation (LoRA).

By Gonzalo Mancera, Daniel DeAlcala, Aythami Morales, Julian Fierrez, Ruben Tolosana, Francisco Jurado

llms nlp fine-tuning benchmarks

arXiv AI

Jun 8

Attention Consistent Longitudinal Medical Visual Question Answering Guided by Vision Foundation Models

arXiv:2606. 06534v1 Announce Type: cross Abstract: Longitudinal medical visual question answering (VQA) requires reasoning about anatomical differences between an image of a current time point and an image of a referred time point.

By Jialin Wu, Qianru Zhang, Georges El Fakhri, Xiaofeng Liu

llms nlp multimodal benchmarks safety

arXiv AI

Jun 8

TSAQA: Time Series Analysis Question And Answering Benchmark

arXiv:2601. 23204v2 Announce Type: replace Abstract: Time series data are integral to critical applications across domains such as finance, healthcare, transportation, and environmental science.

By Baoyu Jing, Sanhorn Chen, Lecheng Zheng, Boyu Liu, Zihao Li, Jiaru Zou, Tianxin Wei, Zhining Liu, Zhichen Zeng, Ruizhong Qiu, Xiao Lin, Yuchen Yan, Dongqi Fu, Jingchao Ni, Jingrui He, Hanghang Tong

llms nlp fine-tuning benchmarks

arXiv AI

Jun 8

DualGate-Net: A Prior-Gated Dual-Encoder Framework for Histopathology Cell Detection

arXiv:2606. 07222v1 Announce Type: cross Abstract: Cell detection in histopathology images strongly depends on surrounding tissue context, where visually similar cells may belong to different classes under different microenvironments.

By Bahman Jafari Tabaghsar, Son Tran, K. Devaraja, Atul Sajjanhar

benchmarks

arXiv Machine Learning

Jun 8

How Far Can Chord-Symbol Time-Series Adaptation Carry Genre Identity? Capabilities and Boundaries in Multi-Genre Chord-Symbol Modeling

arXiv:2606. 07334v1 Announce Type: cross Abstract: Harmony is a compact symbolic layer where mathematical pitch relations, acoustic consonance, and musical convention meet.

By Jinju Lee

llms fine-tuning benchmarks

arXiv Machine Learning

Jun 8

Dash2Sim: Closed-Loop Driving Simulation from in-the-wild Dashcam Videos

arXiv:2606. 07366v1 Announce Type: cross Abstract: Self-driving simulations typically rely on data collected in a small number of cities or on hand-authored synthetic scenarios.

By Anurag Ghosh, Francesco Pittaluga, Khiem Vuong, Angela Chen, Juan Alvarez-Padilla, Manmohan Chandraker, Srinivasa Narasimhan

robotics benchmarks

arXiv Machine Learning

Jun 8

Skip a Layer or Loop It? Learning Program-of-Layers in LLMs

arXiv:2606. 06574v1 Announce Type: new Abstract: Large language models (LLMs) perform inference by following a fixed depth and order, non-recurrent execution of all layers.

By Ziyue Li, Yang Li, Tianyi Zhou

llms benchmarks

arXiv AI

Jun 8

MVCL-DAF++: Enhancing Multimodal Intent Recognition via Prototype-Aware Contrastive Alignment and Coarse-to-Fine Dynamic Attention Fusion

arXiv:2509. 17446v3 Announce Type: replace-cross Abstract: Multimodal intent recognition (MMIR) suffers from weak semantic grounding and poor robustness under noisy or rare-class conditions.

By Haofeng Huang, Yifei Han, Long Zhang, Bin Li, Yangfan He, Yaxin Xue

multimodal benchmarks safety

arXiv AI

Jun 8

DEFINED: A Data-Efficient Computational Framework for Fine-Grained Creativity Assessment in Debate Scenarios

arXiv:2606. 07226v1 Announce Type: cross Abstract: Human creativity has emerged as a critical competency in the era of large language models.

By Tongzhou Yu, Mingjia Li, Hong Qian, Wenkai Wang, Zongbao Zhang, Yaoyu Jiang, Xiangfeng Wang, Aimin Zhou, Jiajun Guo

llms benchmarks safety

arXiv AI

Jun 8

CrowdMath: A Dataset of Crowdsourced Mathematical Research Discussions

arXiv:2606. 06526v1 Announce Type: new Abstract: Large language models have made substantial progress on mathematical reasoning, but existing benchmarks typically evaluate well-specified problems with final answers, step-by-step solutions, or complete proofs.

By Sherin Muckatira, Jesse Geneson, Slava Gerovitch, Pavel Etingof, Mikhail Gronas, Anna Rumshisky

llms benchmarks

arXiv Machine Learning

Jun 8

Implicit Neural Optimal Transport via Fixed-Point Optimization

arXiv:2605. 10792v2 Announce Type: replace-cross Abstract: We propose an implicit neural formulation of optimal transport that eliminates adversarial min--max optimization and multi-network architectures commonly used in existing approaches.

By Yesom Park, Eric Gelphman, Stanley Osher, Samy Wu Fung

benchmarks safety

arXiv Machine Learning

Jun 8

Generative Molecular Morphing for Flexible-Size Design via Unbalanced Optimal Transport

arXiv:2606. 07239v1 Announce Type: new Abstract: The success of generative molecular design hinges on a model's steerability toward high-reward samples.

By Malte Franke, Stefan P. Schmid, Zarko Ivkovic, Kjell Jorner, Andreas Krause

diffusion benchmarks

arXiv Machine Learning

Jun 8

CF-JEPA: Mask-free forward prediction with asymmetric encoder utilization for time-series representation learning

arXiv:2606. 07031v1 Announce Type: new Abstract: Self-supervised learning (SSL) for time-series representation learning is dominated by two paradigms: contrastive methods, which face challenges in constructing positive or negative pairs, and masking-based methods, which disrupt the temporal continuity of time-series signals.

By Jaehoon Lee, Sunghyun Sim

llms rag benchmarks

Get the daily briefing