Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

23,288 stories · RSS feed

arXiv AI

Jun 6

ReTreVal: Reasoning Tree with Validation and Cross-Problem Memory for Large Language Models

arXiv:2601. 02880v2 Announce Type: replace Abstract: Every existing inference-time reasoning framework discards all failure context at problem boundaries, leaving a model solving problem 500 no wiser than it was on problem 1.

By Abhishek HS, Pavan C Shekar, Arpit Jain, Ashwanth Krishnan

llms fine-tuning benchmarks

arXiv AI

Jun 6

MAVEN A Multi-Agent Framework for Multicultural Text-to-Video Generation

arXiv:2605. 16716v4 Announce Type: replace-cross Abstract: Text-to-video (T2V) generation has rapidly progressed in visual fidelity, yet its ability to faithfully represent multiple cultures within a single prompt remains underexplored.

By Shuowei Li, Yuming Zhao, Parth Bhalerao, Oana Ignat

agents diffusion multimodal benchmarks

arXiv AI

Jun 6

Beyond Waveform Robustness: Robust Feature-Vocoder Adversarial Attacks on Automatic Speech Recognition

arXiv:2606. 05678v1 Announce Type: cross Abstract: Automatic speech recognition (ASR) systems have become widely used for multilingual speech-to-text transcription.

By Yifan Liao, Zongmin Zhang, Zhen Sun, Yuhui Sun, Xinhu Zheng, Xinlei He

multimodal benchmarks safety

arXiv AI

Jun 6

LongSpace: Exploring Long-Horizon Spatial Memory from Perception to Recall in Video

arXiv:2606. 05677v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have advanced image and video understanding and can increasingly handle longer visual inputs.

By Shiqiang Lang, Jing Liu, Haoyang He, Peiwen Sun, Yuanteng Chen, Tao Liu, Lan Yang, Longteng Guo, Honggang Zhang

llms agents robotics multimodal benchmarks

arXiv AI

Jun 6

MAviS: A Multimodal Conversational Assistant For Avian Species

arXiv:2603. 07294v2 Announce Type: replace-cross Abstract: Fine-grained understanding and species-specific multimodal question answering are vital for advancing biodiversity conservation and ecological monitoring.

By Yevheniia Kryklyvets, Mohammed Irfan Kurpath, Sahal Shaji Mullappilly, Jinxing Zhou, Fahad Shabzan Khan, Rao Anwer, Salman Khan, Hisham Cholakkal

llms nlp multimodal benchmarks

arXiv AI

Jun 6

Emotion-Aware Image Generation from Korean Diary Text via LLM-based Prompt Translation and LoRA Fine-Tuning

arXiv:2606. 05816v1 Announce Type: cross Abstract: T2I models cannot effectively capture sentiment from various types of text, including diaries, as they primarily focus on visual object-related patterns rather than contextual emotional understanding.

By Jihun Cho, Soo-Yeon Jeong, Sun-Young Ihm

llms diffusion fine-tuning benchmarks

arXiv AI

Jun 6

The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs?

arXiv:2504. 10020v4 Announce Type: replace-cross Abstract: Contrastive decoding strategies are widely used to reduce object hallucinations in multimodal large language models (MLLMs).

By Hao Yin, Guangzong Si, Zilei Wang

llms multimodal benchmarks safety

arXiv AI

Jun 6

Learning Geometric Representations from Videos for Spatial Intelligent Multimodal Large Language Models

arXiv:2606. 05833v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) excel at 2D semantic understanding but lack intrinsic 3D awareness, resulting in representations that fail to maintain geometric and spatial consistency across video frames.

By Haibo Wang, Lifu Huang

llms diffusion multimodal benchmarks

arXiv AI

Jun 6

Learning Adaptive Parallel Execution for Efficient Code Localization

arXiv:2601. 19568v2 Announce Type: replace Abstract: Code localization constitutes a key bottleneck in automated software development pipelines.

By Ke Xu, Siyang Xiao, Ming Liang, Yichen Yu, Zhixiang Wang, Jingxuan Xu, Dajun Chen, Wei Jiang, Yong Li

agents reinforcement-learning fine-tuning benchmarks

arXiv AI

Jun 6

ATT-CR: Adaptive Triangular Transformer for Cloud Removal

arXiv:2606. 05999v1 Announce Type: cross Abstract: Cloud removal aims to accurately reconstruct the ground objects obscured by clouds in remote sensing images.

By Yang Wu, Ye Deng, Pengna Li, Wenli Huang, Kangyi Wu, Xiaomeng Xin, Jinjun Wang

llms benchmarks

arXiv AI

Jun 6

InfoDensity: Rewarding Information-Dense Traces for Efficient Reasoning

arXiv:2603. 17310v2 Announce Type: replace Abstract: Large Language Models (LLMs) with extended reasoning capabilities often generate verbose and redundant reasoning traces, incurring unnecessary computational cost.

By Chengwei Wei, Jung-jae Kim, Longyin Zhang, Shengkai Chen, Nancy F. Chen

llms reinforcement-learning benchmarks

arXiv AI

Jun 6

CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from Multiple Perspectives

arXiv:2504. 10823v4 Announce Type: replace-cross Abstract: Navigating dilemmas involving conflicting values is challenging even for humans in high-stakes domains, let alone for AI, yet prior work has been limited to everyday scenarios.

By Ayoung Lee, Ryan Sungmo Kwon, Peter Railton, Lu Wang

llms benchmarks safety

arXiv AI

Jun 6

Humans' ALMANAC: A Human Collaboration Dataset of Action-Level Mental Model Annotations for Agent Collaboration

arXiv:2606. 06388v1 Announce Type: new Abstract: Recent advances in LLM agents have enabled complex cognitive capabilities, such as multi-step reasoning, planning, and tool use, that increasingly position these agents as human collaborators.

By Jiaju Chen, Yuxuan Lu, Jiayi Su, Chaoran Chen, Songlin Xiao, Zheng Zhang, Yun Wang, Yunyao Li, Jian Zhao, Tongshuang Wu, Toby Jia-Jun Li, Dakuo Wang, Bingsheng Yao

llms agents benchmarks

arXiv AI

Jun 6

TokenMizer: Graph-Structured Session Memory for Long-Horizon LLM Context Management

arXiv:2606. 06337v1 Announce Type: new Abstract: Large language model (LLM) deployments for long-horizon tasks face a fundamental constraint: context windows are finite while productive work sessions are not.

By Shweta Mishra

llms benchmarks

arXiv AI

Jun 6

CogManip: Benchmarking Manipulative Behavior in Multi-Turn Interactions with Large Language Model

arXiv:2606. 06099v1 Announce Type: new Abstract: Whether Large Language Models (LLMs) exhibit covert psychological manipulation in complex human-AI interactions has garnered increasing safety concerns.

By Zeyang Yue, Chenfei Yan, Feifei Zhao, Haibo Tong, Mengwen Xu, Xiaozhen Wang, Erliang Lin, Yi Zeng

llms robotics benchmarks safety

arXiv AI

Jun 6

Towards World Models in Biomedical Research

arXiv:2606. 05925v1 Announce Type: new Abstract: A central goal of biomedicine is to understand, predict and ultimately control the dynamic mechanisms by which biological systems respond to perturbations, disease progression and therapeutic intervention.

By Guangyu Wang, Jingkun Yue, Siqi Zhang, Yu Liu, Xiaoyu Wang, Mingyuan Meng, Changwei Ji, Zongbo Han, Yulin Wang, Yang Yue, Frank Fu, Ting Chen, Song Wu, Ziwei Liu, Jiangning Song, Ming Li, Gao Huang, Xiaohong Liu, Athanasios Vasilakos, Xingcai Zhang, Ping Zhang, Yong Li

llms benchmarks safety

arXiv AI

Jun 6

Improving Answer Extraction in Context-based Question Answering Systems Using LLMs

arXiv:2606. 06197v1 Announce Type: cross Abstract: Question answering (QA) systems have achieved notable progress with the advent of large language models (LLMs).

By Hafez Abdelghaffar, Ahmed Alansary, Ali Hamdi

llms nlp fine-tuning benchmarks

arXiv AI

Jun 6

Breaking the Chain: A Causal Analysis of LLM Faithfulness to Intermediate Structures

arXiv:2603. 16475v2 Announce Type: replace Abstract: In schema-guided reasoning (SGR) pipelines, LLMs produce explicit intermediate structures -- rubrics, checklists, or verification queries -- before committing to a final decision.

By Oleg Somov, Mikhail Chaichuk, Gleb Ershov, Karim Vafin, Mikhail Seleznyov, Alexander Panchenko, Elena Tutubalina

llms benchmarks

arXiv AI

Jun 6

SubtleMemory: A Benchmark for Fine-Grained Relational Memory Discrimination in Long-Horizon AI Agents

arXiv:2606. 05761v1 Announce Type: new Abstract: Persistent AI assistants, such as OpenClaw, accumulate large collections of related memories over long-term interactions.

By Wenxuan Wang, Haoyu Sun, Fukuan Hou, Mingyang Song, Weinan Zhang, Yu Cheng, Yang Yang

agents benchmarks

arXiv Machine Learning

Jun 5

Semi-Offline Reinforcement Learning for Optimized Text Generation

arXiv:2306. 09712v2 Announce Type: replace Abstract: In reinforcement learning (RL), there are two major settings for interacting with the environment: online and offline.

By Changyu Chen, Xiting Wang, Yiqiao Jin, Victor Ye Dong, Li Dong, Jie Cao, Yi Liu, Rui Yan

reinforcement-learning benchmarks

Get the daily briefing