Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,906 stories · RSS feed

arXiv Machine Learning

Jul 10

PIT-SUN: A Deployable Empirical Marginal Transform Framework with Expectation-Consistent Recovery for Regression in Recommender Systems

arXiv:2607. 08202v1 Announce Type: new Abstract: Estimating original-space conditional expectations is central to value-driven recommender systems, including dwell time, GMV, and LTV forecasting.

By Mingyu Zhao, Zhaohan Li, Zhenxiong Miao, Xu Zhang, Dewei Leng, Yanan Niu, Kun Gai

multimodal benchmarks

arXiv AI

Jul 10

SHAP-Weighted Cross-Modal Expert Fusion for Emotion and Sentiment Recognition: Evidence and Limits

arXiv:2607. 08573v1 Announce Type: new Abstract: Multimodal emotion and sentiment recognition is commonly addressed by early fusion, which concatenates modalities before classification, or late fusion, which combines independently trained unimodal predictors.

By Adis Alihodzic, Selma Skopljakovic Hubljar

llms multimodal benchmarks

arXiv Machine Learning

Jul 10

What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

arXiv:2607. 08032v1 Announce Type: new Abstract: Large language models, and the agents built on them, spend an ever-growing share of their compute and memory on remembering: caching attention keys and values, carrying long prompts, maintaining recurrent state, and storing what happened in previous turns and sessions.

By Ashwin Gerard Colaco, Nada Lahjouji

llms agents efficiency benchmarks

arXiv AI

Jul 10

A Graph Neural Network Model for Real-Time Gesture Recognition Based on sEMG Signals

arXiv:2607. 07850v1 Announce Type: new Abstract: For seemless control of advanced hand prostheses and augmented reality, accurate and immediate hand gestures recognition is essential.

By Pragatheeswaran Vipulanandan, Kamal Premaratne, Manohar Murthi

benchmarks

arXiv AI

Jul 10

Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench

arXiv:2607. 08284v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated rapidly improving long-context capabilities, prompting a wave of benchmarks designed to evaluate them.

By Siddhartha Jain, Ameya Velingker

llms nlp benchmarks

arXiv AI

Jul 10

WebSwarm: Recursive Multi-Agent Orchestration for Deep-and-Wide Web Search

arXiv:2607. 08662v1 Announce Type: cross Abstract: Large language model (LLM)-based web search agents are transforming information seeking from simple factoid question answering into complex, deep-and-wide search and research-oriented tasks.

By Xiaoshuai Song, Liancheng Zhang, Kangzhi Zhao, Yutao Zhu, Zhongyuan Wang, Guanting Dong, Jinghan Yang, Han Li, Kun Gai, Ji-Rong Wen, Zhicheng Dou

llms agents nlp benchmarks

arXiv AI

Jul 10

Validity of LLMs as data annotators: AMALIA on authority

arXiv:2607. 08731v1 Announce Type: cross Abstract: A national language model offers a linguistic community its own instrument for measuring what its citizens say and value.

By Manuel Pita

llms benchmarks

arXiv Machine Learning

Jul 10

HeaPA: Difficulty-Aware Heap Sampling and On-Policy Query Augmentation for LLM Reinforcement Learning

arXiv:2601. 22448v2 Announce Type: replace Abstract: RLVR has become a standard recipe for training LLMs on reasoning tasks with verifiable outcomes, but when rollout generation dominates the cost, efficiency hinges on which prompts are sampled and when.

By Weiqi Wang, Xin Liu, Binxuan Huang, Hejie Cui, Rongzhi Zhang, Changlong Yu, Shuowei Jin, Jingfeng Yang, Qingyu Yin, Zhengyang Wang, Zheng Li, Yifan Gao, Priyanka Nigam, Bing Yin, Lihong Li, Yangqiu Song

llms reinforcement-learning benchmarks

arXiv AI

Jul 10

OpenCoF: Learning to Reason Through Video Generation

arXiv:2607. 08763v1 Announce Type: cross Abstract: Reasoning has become a core capability for large models, especially when reliable decisions require understanding logical consequences.

By Xinyan Chen, Ziyu Guo, Renrui Zhang, Dongzhi Jiang, Hongsheng Li

diffusion fine-tuning benchmarks

arXiv AI

Jul 10

A Multi-cluster Boundary Learning Method for Out-of-Scope Intent Detection via MiniLM Embedding

arXiv:2607. 07974v1 Announce Type: cross Abstract: Intent detection is a critical task that bridges human intents and system actions in human-machine interaction systems.

By Yihong Xu, Mingyu Kang, Linyuan L\"u

llms rag benchmarks

arXiv Machine Learning

Jul 10

Spectral Stability of Pseudoinverse-Based Extreme Learning Machine

arXiv:2607. 08581v1 Announce Type: new Abstract: Extreme Learning Machine (ELM) computes output weights analytically using the Moore-Penrose pseudoinverse.

By Bich Van Nguyen, Ngoc Anh Khong

benchmarks

arXiv AI

Jul 10

Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE

arXiv:2607. 07740v1 Announce Type: cross Abstract: Modern LLMs are increasingly deployed in long-context applications such as retrieval-augmented generation, repository-level coding, and agentic workflows whose accumulated reasoning and tool traces routinely push the input an order of magnitude past the pretraining window, making zero-shot context extension the dominant deployment path for open-weight checkpoints.

By Haozhan Tang, Zerui Wang, Yuxian Gu, Song Han, Han Cai

llms rag agents efficiency benchmarks

arXiv AI

Jul 10

INTENT: An LSTM Framework for Vehicle Intention Prediction in Intersection Scenarios with Comprehensive Ablation Analysis

arXiv:2607. 08316v1 Announce Type: new Abstract: Vehicle intention prediction is a pivotal aspect in the agility and safety of autonomous vehicles in all driving scenarios; if genuine enhancement of autonomous vehicles are required, we need to make them adopt human interpretation of driver's intention especially in cases that require a lot of human interaction as well as complex driving behaviors like the ones at intersections, roundabouts and emergency cases such as sudden stops where vehicle intention prediction helps in taking the correct evasive action within a real time period where every second of action makes an impact and can prevent a catastrophe from taking place.

By Logine M. Zaki, Catherine M. Elias

agents benchmarks safety

arXiv AI

Jul 10

Towards Precision Therapy in Hepatocellular Carcinoma: A Clinical-Reasoning LLM for Risk Stratification and Treatment Guidance

arXiv:2607. 08602v1 Announce Type: new Abstract: Hepatocellular carcinoma (HCC) is a common malignancy and a leading cause of cancer-related mortality.

By Peng Cui, Jitao Wang, Siyan Xue, Yao Huang, Haoming Xia, Dong Li, Dengxiang Liu, Weilin Wang, Liping Liu, Leida Zhang, Yunfu Cui, Tao Peng, Daolin Ji, Haitao Zhao, Wei Zhang, Xiaojuan Wang, Weijie Ma, Zongren Ding, Jinlong Li, Yuan Ding, Jiajing Zhao, Zhiyu Chen, Chengkun Yang, Ziyue Huang, Jiaqi Liu, Fusheng Liu, Yang Zhou, Xiaojuan Wang, Zhongquan Sun, Shiyun Bao, Xiaojun Wang, Ming Yang, Guangxin Li, Bin Shu, Yong Liao, Hongxuan Li, Yao Tang, Shizhong Yang, Yongyi Zeng, Yufeng Yuan, Yinpeng Dong, Jihui Hao, Jun Zhu, Jiahong Dong

llms benchmarks

arXiv AI

Jul 10

Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks

arXiv:2607. 07907v1 Announce Type: cross Abstract: With the growing adoption of VLMs, DMs, LLMs, and AFMs, these multimodal foundation models can inadvertently encode sensitive, copyrighted, biased, or unsafe cross-modal associations that originate from their training data.

By Nobin Sarwar, Shubhashis Roy Dipta, Zheyuan Liu, Vaidehi Patil

llms multimodal benchmarks

arXiv AI

Jul 10

OmniFood-Bench: Evaluating VLMs for Nutrient Reasoning and Personalized Health Advice

arXiv:2607. 08423v1 Announce Type: new Abstract: The rapid integration of Large Vision-Language Models (VLMs) into critical infrastructure promises to revolutionize personalized healthcare and dietary management.

By Qian Jiang, Zhecheng Shi, Jingpu Yang, Zirui Song, Miao Fang

llms agents multimodal benchmarks safety

arXiv AI

Jul 10

Simulator Ensembles for Trustworthy Autonomous Driving Systems Testing

arXiv:2503. 08936v3 Announce Type: replace-cross Abstract: Scenario-based testing with driving simulators is extensively used to identify failing conditions of automated driving assistance systems (ADAS).

By Lev Sorokin, Matteo Biagiola, Andrea Stocco

agents robotics benchmarks

arXiv AI

Jul 10

Omni-Sleep: A Sleep Foundation Model via Hierarchical Contrastive Learning of CNS--ANS Dynamic

arXiv:2607. 07720v1 Announce Type: cross Abstract: Sleep physiology arises from the coordinated dynamics of the central nervous system (CNS) and autonomic nervous system (ANS), as reflected by multimodal polysomnography signals including EEG, EOG, EMG, ECG, and respiration.

By Zhoujie Hou, Song Wang, Kexin Lou, Mo Wang, Chen Wei, Quanying Liu

multimodal benchmarks

arXiv Machine Learning

Jul 10

Image classification via a quantum-inspired strategy involving a mixture of experts

arXiv:2607. 07754v1 Announce Type: new Abstract: Pattern recognition problems arise in a variety of physical image processing situations, and convolutional neural networks are a popular scheme for the required feature extraction and classification tasks.

By Kumari Jyoti, Rohith Babu, Apoorva D. Patel

diffusion computer-vision benchmarks

arXiv AI

Jul 10

PS4: Proxy-Supervised Joint Training for Real Target Speaker Extraction

arXiv:2607. 08111v1 Announce Type: cross Abstract: Training target speaker extraction (TSE) models for real conversational mixtures remains challenging because large-scale training corpora and clean target speech for supervision are unavailable.

By Wanyi Ning, Wei Zhou, Yingpeng Li, Yinshang Guo, Haitao Qian, Yiming Cheng

fine-tuning multimodal benchmarks

Get the daily briefing