Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv AI

Oct 15

AffectWorld: Benchmarking Emotion-Driven Future World Prediction

arXiv:2610.18305v1 Announce Type: cross Abstract: The world encompasses both physical environments and human actors. Predicting its evolution requires understanding both physical laws and how emotion...

By Changhao Song, Yacong Chen, Hui Gao, Yazhou Zhang, Peng Zhang

llms agents multimodal benchmarks

arXiv AI

Oct 15

ShoeFit: A New Dataset and Dual-image-stream DiT Framework for Virtual Footwear Try-On

arXiv:2610.18325v1 Announce Type: cross Abstract: Virtual footwear try-on (VFTON), a critical yet underexplored area in virtual try-on (VTON), aims to synthesize faithful try-on results given diverse...

By Yuhan Li, Zhiyu Jin, Yifan Tong, Wenxiang Shang, Benlei Cui, Xuanhong Chen, Hangcheng Zhu, Bingbing Ni

rag benchmarks

arXiv AI

Oct 15

Agaram 2.5B: The First Tamil-First Language Model Trained from Scratch with Tool Calling. A Playbook for Under-Represented Languages

arXiv:2610.18420v1 Announce Type: cross Abstract: We trained Agaram 2.5B, a 2.5-billion-parameter Tamil-English language model, from scratch on about 50 billion tokens (58% Tamil, the rest English, m...

By Ramchand Kumaresan, Madhusudhanan G, Kaushik Kezendirane

llms agents nlp fine-tuning benchmarks

arXiv AI

Oct 15

Do Factuality Methods Make LLMs More Factual? Reproducibility and Fair Comparison

arXiv:2610.18415v1 Announce Type: cross Abstract: Large Language Models (LLMs) are aligned to human preferences, after pre-training and supervised fine-tuning steps. However, standard RL alignment st...

By Pratuat Amatya, Vinay Setty, V Venktesh, Avishek Anand

llms reinforcement-learning fine-tuning benchmarks safety

arXiv AI

Oct 15

Resource-Efficient 3D Shape and Orientation Estimation of Road Users using Roadside RGB Camera and 3D Radar

arXiv:2610.18417v1 Announce Type: cross Abstract: End-to-end deep learning pipelines for camera-radar 3D object detection are well established in driver assistance systems and autonomous vehicles; ho...

By Shiva Agrawal, Savankumar Bhanderi, Gordon Elger

agents computer-vision benchmarks

arXiv AI

Oct 15

Learning Perturbation-Specific Representations through Conditional Energy Prior for Cellular Response Prediction

arXiv:2610.18456v1 Announce Type: cross Abstract: AI-based cellular perturbation prediction is essential to accelerate drug discovery and therapeutic identification. However, perturbation-specific si...

By Yuchen Yan, Yatao Bian

diffusion benchmarks

arXiv AI

Oct 15

GeoMER: A Joint Euclidean-Hyperbolic Geometry Framework for Multimodal Emotion Recognition

arXiv:2610.18587v1 Announce Type: cross Abstract: Multimodal Emotion Recognition (MER) aims to understand human emotions by integrating information from multiple modalities such as speech and text. P...

By Ting Zhu, Huizhi Liang, Boguslaw Obara, Shufei Duan, Varun Ojha

multimodal benchmarks

arXiv AI

Oct 15

AMemOps: Evaluating Memory Updates in Long Audio Streams

arXiv:2610.18652v1 Announce Type: cross Abstract: Audio memory must recognise recurring sources across acoustic changes while keeping their attributes up to date. Overall accuracy does not reveal how...

By Peiwei Chang, Li Zhang, Mark D. Plumbley, Philip J. B. Jackson, Wenwu Wang

benchmarks

arXiv AI

Oct 15

Jev-ToM: Evaluating Theory of Mind in Typed Decision Models

arXiv:2610.18610v1 Announce Type: cross Abstract: Dialogue and collaborative assistants repeatedly judge what a partner knows, believes, or intends. Typed decision models offer a potential alternativ...

By Kawai Chung, Chunkit Chan, Yuxuan Liu, Yauwai Yim, Xi Yang, Wenbin Hu, Huihao Jing, Chiu Yeung Tang, Changxuan Fan, Ki Sen Hung, Tsun On Kwok, Haoran Li

llms benchmarks

arXiv AI

Oct 15

ST-GRU PredRNN-V2: A GRU-Based Reformulation of PredRNN-V2 for Video Prediction

arXiv:2610.18645v1 Announce Type: cross Abstract: Video prediction requires models to capture complex spatiotemporal dependencies and to retain relevant information over long sequences. PredRNN and i...

By Augusta van Haren, Hanna Hoogen, Luca Pattavina, Quincy Rommen

benchmarks

arXiv AI

Oct 15

RadarGSOcc: Doppler-Aware 3D Semantic Occupancy Prediction Using 4D Radar Tensor with Geometric-Semantic Feature Extraction

arXiv:2610.18686v1 Announce Type: cross Abstract: 3D semantic occupancy prediction is crucial for autonomous driving, but camera-based methods degrade in low-light conditions and LiDAR-based methods...

By Christofel Rio Goenawan, Dong-Hee Paek, Seung-Hyun Kong

agents robotics efficiency benchmarks

arXiv AI

Oct 15

GeoReader: Learning Latent Geometry for Spatial Reasoning in Vision-Language Models

arXiv:2610.18694v1 Announce Type: cross Abstract: Vision-language models (VLMs) struggle with metric and multi-view relationships that remain implicit in RGB observations. Geometry foundation models...

By Rong Li, Shijie Li, Jiangyong Huang, Lingdong Kong, Jiaming Zhou, Dongyue Lu, Zeying Gong, Baoxiong Jia, Junwei Liang, Siyuan Huang

llms nlp fine-tuning multimodal benchmarks safety

arXiv AI

Oct 15

Hybrid Reasoning Agents

arXiv:2610.18861v1 Announce Type: cross Abstract: How can we build an agent that thinks beyond human language, yet still acts through it in the external environment? Modern LLM agents still reason an...

By Jiaru Zou, Yunzhe Qi, Sirui Chen, Federico Bianchi, Shang Zhu, Pan Lu, Jingrui He, James Zou

llms agents diffusion benchmarks

arXiv AI

Oct 15

DoubtLess: Training-Free Contrastive Decoding for Efficient Reasoning

arXiv:2610.18862v1 Announce Type: cross Abstract: Reasoning models achieve strong performance through long chains of thought (CoT), but often spend more tokens than needed to solve a problem. Existin...

By Minghao Luo, Liang Chen

agents benchmarks

arXiv AI

Oct 15

Identification of Entailment and Contradiction Relations for Natural Language Argumentation: A Neurosymbolic Approach

arXiv:2405.01259v2 Announce Type: replace Abstract: Argumentation is an important cognitive activity that involves the scrutiny of arguments and counterarguments. When given an argument, it is desira...

By Xuyao Feng, Anthony Hunter

benchmarks

arXiv AI

Oct 15

Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving

arXiv:2503.21821v2 Announce Type: replace Abstract: We introduce Physics, a benchmark for evaluating foundation models on advanced physics problem solving. It contains 1034 annotated, open-ended prob...

By Kaiyue Feng, Yilun Zhao, Yixin Liu, Tianyu Yang, Chen Zhao, John Sous, Arman Cohan

llms benchmarks

arXiv AI

Oct 15

Decomposability-Guided Cooperative Coevolution for Large-Scale Itinerary Planning

arXiv:2506.06121v3 Announce Type: replace Abstract: Large-scale itinerary planning is a variant of the traveling salesman problem, aiming to determine an optimal path that maximizes the collected poi...

By Ziyu Zhang, Peilan Xu, Yuetong Sun, Yuhui Shi, Wenjian Luo

benchmarks

arXiv AI

Oct 15

Risky-Bench: Evaluating Agentic Safety Risks in Real-World Life-Assist Deployments

arXiv:2602.03100v2 Announce Type: replace Abstract: Large Language Models (LLMs) are increasingly deployed as agents that operate in real-world environments, introducing safety risks beyond linguisti...

By Jingnan Zheng, Yanzhen Luo, Jingjun Xu, Bingnan Liu, Yuxin Chen, Chenhang Cui, Gelei Deng, Chaochao Lu, Xiang Wang, An Zhang, Tat-Seng Chua

llms agents benchmarks safety

arXiv AI

Oct 15

Strategy Executability in Mathematical Reasoning: Leveraging Human-Model Differences for Effective Guidance

arXiv:2602.22583v2 Announce Type: replace Abstract: Example-based guidance is widely used to improve mathematical reasoning at inference time, yet its effectiveness is highly unstable across problems...

By Weida Liang, Yiyou Sun, Shuyuan Nan, Chuang Li, Dawn Song, Kenji Kawaguchi

llms benchmarks

arXiv AI

Oct 15

Logarithmic Scores, Power-Law Discoveries: Disentangling Measurement from Coverage in Agent-Based Evaluation

arXiv:2604.00477v2 Announce Type: replace Abstract: LLM-based agent judges are an emerging approach to evaluating conversational AI, yet a fundamental uncertainty remains: can we trust their assessme...

By HyunJoon Jung, William Na

llms agents benchmarks safety

Get the daily briefing