Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv Computation and Language

Sep 21

QuranicMMLU: A Cognitively-Aware Benchmark for Evaluating Generative AI Solutions on Quranic Linguistic Knowledge

arXiv:2609.22038v1 Announce Type: new Abstract: We introduce QuranicMMLU, a benchmark for evaluating generative AI on Quranic Arabic across multiple dimensions of linguistic complexity. Existing Qura...

By Rawan El Ghali, Umm Kulsoom, Anas Madkoor, Dima Faris Alsaudi, Roaa Abdelmagid, Roaa Ibrahim, Raghad Mousa, Hamza Aljaji, Abdullah Khanafer, Abdallah Alkanani, Salah Feras Alali, Rawan Khaled Mohamed, Ehsaneddin Asgari

llms nlp benchmarks

arXiv Computation and Language

Sep 21

RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents

arXiv:2609.22000v1 Announce Type: new Abstract: Computer-use agents (CUAs) have advanced along two separate lines: graphical interaction and software development through code and the command line. Re...

By Shuai Bai, Jiayong Deng, Yikun Fu, Chang Gao, Xuhao Hu, Mianqiu Huang, Yizhen Jiang, Yuheng Jing, Dehui Kong, Keliang Li, Ning Li, Wanli Li, Dayiheng Liu, Dunjie Lu, Changwei Luo, Que Shen, Zheyuan Wang, Zijian Wang, Jie Wu, Gao Wu, Zhihui Xie, Rui Xie, Haiyang Xu, An Yang, Jiakang Yuan, Yanming Zhang, Jiajun Zhang, Xi Zhang, Zhenru Zhang, Zhuo Zhen, Mingkang Zhu, Bowen Zhou

llms agents benchmarks

arXiv Computation and Language

Sep 21

VQ-Logits: Compressing the Output Bottleneck of Large Language Models via Vector Quantized Logits

arXiv:2505.10202v2 Announce Type: replace Abstract: Large Language Models (LLMs) have achieved remarkable success but face significant computational and memory challenges, particularly due to their e...

By Jintian Shao, Hongyi Huang, Jiayi Wu, YiMing Cheng, ZhiYu Wu, You Shan, MingKai Zheng

llms rag efficiency benchmarks

arXiv Computation and Language

Sep 21

MemAudit: Auditing Long-Term Agent Memory via Hidden User-State Recovery

arXiv:2606.24595v2 Announce Type: replace Abstract: Long-term memory promises LLM agents that grow more capable across sessions, maintaining an accurate, evolving understanding of the user that inter...

By Enze Ma, Yufan Zhou, Wei-Chieh Huang, Jie Yang, Huanhuan Ma, Zixuan Wang, Chengze Li, Chunyu Miao, Philip S. Yu, Zhen Wang

llms agents benchmarks

arXiv Computer Vision

Sep 21

MAGIC: Marginal-Guided Compression with Optimal Transport for Efficient Visual Document Retrieval

arXiv:2609.21018v1 Announce Type: new Abstract: Recent visual document retrieval (VDR) systems such as ColPali use multi-vector page embeddings, in which patch-level vectors enable fine-grained evide...

By Xu Yuan, Hua Liu, Wenqi Fan, Qing Li

rag benchmarks

arXiv Computer Vision

Sep 21

PrismAlign: Prior-Steered Multi-View VLM Alignment for Hallucination-Robust Table OCR

arXiv:2609.21351v1 Announce Type: new Abstract: Table extraction suffers from frequent structural errors and semantic hallucinations. We propose PrismAlign, a multi-VLM framework aligning diverse vis...

By Guangyi Liu, Qianjun Huang, Boyu Hou

computer-vision multimodal benchmarks safety

arXiv Computer Vision

Sep 21

SIRA: Reasoning-Aware Surgical Instrument Segmentation via Query-Anchored Alignment

arXiv:2609.21402v1 Announce Type: new Abstract: Surgical instrument segmentation (SIS) plays a critical role in robotic assistance and surgical workflow analysis. However, most existing SIS methods f...

By Zhibo Zhang, Qijie Wang, Zengqiang Yan

computer-vision robotics multimodal benchmarks safety

arXiv Computer Vision

Sep 21

JEPA Guided Diffusion: Predictive Vision-Language Conditioning for Generative Traffic Forecasting

arXiv:2609.21379v1 Announce Type: new Abstract: Accurate traffic forecasting requires both understanding scene dynamics and synthesizing realistic future observations. Recent diffusion-based video ge...

By Trinh Tra Giang Nguyen, Thanh Nguyen Vo, Nguyen Hoai Thuong Bui, Ha Duc Bui

diffusion multimodal benchmarks safety

arXiv Computer Vision

Sep 21

P$^3$-SAM: SAM with Perceptual Parallel Prompt for Few-Shot Strip Steel Surface Defect Segmentation

arXiv:2609.21424v1 Announce Type: new Abstract: Few-shot semantic segmentation (FSS) of strip steel surface defects (S$^3$D) has posed significant challenges distinct from natural scenes. Unlike natu...

By Qian Xu, Hang Xiong, Anpeng Wang, Sam Kwong, Cong Zhang, Runmin Cong

computer-vision benchmarks

arXiv Computer Vision

Sep 21

DRT: Dense Reasoning Trace for Efficient and Grounded Multimodal Reasoning

arXiv:2609.21675v1 Announce Type: new Abstract: Despite the remarkable progress in Multimodal Large Language Models (MLLMs), prevailing Chain-of-Thought (CoT) paradigms remain confined to the natural...

By Wan Xu, Yuanfan Guo, Kevin Han, LaLa Chen, Wangmeng Zuo

llms reinforcement-learning multimodal benchmarks safety

arXiv Computer Vision

Sep 21

The Role of Radiometric Features in Cross-Site Leaf-Wood Segmentation of LiDAR Point Clouds

arXiv:2609.21903v1 Announce Type: new Abstract: Leaf-wood segmentation of individual trees from LiDAR point clouds is essential for quantitative structure models (QSMs) used in non-destructive biomas...

By Roman Kaharlytskyi, Derek T. Robinson, Roberto Guglielmi

computer-vision benchmarks

arXiv Computer Vision

Sep 21

MintAct: A Unified Visual Agent for Digital Environments

arXiv:2609.22083v1 Announce Type: new Abstract: We present MintAct, a family of vision-language models that unifies UI grounding, multi-step navigation across mobile, desktop, and web, and visual too...

By Mingfei Gao, Rui Tian, Haiming Gang, Bohan Zhai, Le Zhang, Yuanzheng Gong, Di Feng, Ege \"Ozsoy, Kaixin Ma, Vishwesh Kirthivasan, O\u{g}uzhan Fatih Kar, Roman Bachmann, Anders Boesen Lindbo Larsen, Afshin Dehghan

llms agents reinforcement-learning multimodal benchmarks

arXiv Computer Vision

Sep 21

Do Spinning Radar Doppler Velocity Measurements Improve Vehicle Detection and Tracking?

arXiv:2609.21000v1 Announce Type: cross Abstract: Spinning frequency-modulated continuous-wave (FMCW) radars have been gaining popularity in autonomous vehicle perception on account of their robustne...

By Eric Xie, Daniil Lisus, Timothy D. Barfoot

agents fine-tuning benchmarks

arXiv Computer Vision

Sep 21

Adaptive Color Grading

arXiv:2609.21169v1 Announce Type: cross Abstract: Independent control of tonescale regions (e.g., shadows, highlights) is essential for painters, photographers and cinematographers to bring 2D images...

By Trevor D. Canham, Abhijith Punnappurath, Michael S. Brown

robotics benchmarks

arXiv Computer Vision

Sep 21

Refining Ground Truth Poses in Autonomous Driving Datasets via Neural Rendering

arXiv:2504.15776v2 Announce Type: replace Abstract: Public autonomous driving datasets underpin the training and benchmarking of perception, mapping, and localization algorithms, yet residual inaccur...

By Quentin Herau, Nathan Piasco, Moussab Bennehar, Luis Rold\~ao, Dzmitry Tsishkou, Bingbing Liu, Cyrille Migniot, Pascal Vasseur, C\'edric Demonceaux

agents robotics multimodal benchmarks

arXiv Machine Learning

Sep 21

Write on Paper and Get the Online Digital Trace: A New Era for Handwriting

arXiv:2609.12702v2 Announce Type: replace Abstract: Capturing the digital trace of handwriting usually requires a specific stylus and a compatible substrate, be it a capacitive touchscreen, an Electr...

By Florent Imbert, Yann Soullard, Eric Anquetil, Tanja Harbaum, Alexey Serdyuk, Fabian Kress, Tim Hamann, Peter Kampf

benchmarks

arXiv Computation and Language

Sep 21

Reviser: Revision-Capable Text Generation via Autoregressive Cursor Actions

arXiv:2609.20830v1 Announce Type: new Abstract: Revision-capable generation is appealing because it can insert or revise earlier content, but many non-autoregressive and edit-based approaches obtain...

By Sean Diab

llms diffusion benchmarks

arXiv Computation and Language

Sep 21

Boosting Deepresearch and LongContext Ability with Self-Generated Deepresearch Rollouts Traces

arXiv:2609.20844v1 Announce Type: new Abstract: Deepresearch (DR) agents interact with real-world web environments through multi-turn search and visit, causing their contexts to grow rapidly over tim...

By Zihan Wang, Hao Wang, Boyuan Jiang, Yiqun Zhang, Shi Feng, Xiaocui Yang, Yiwen Ye, Jianghang Lin, Xiaozhong Ji, Jinghao Lin, Kai Wu

agents reinforcement-learning benchmarks safety

arXiv Computation and Language

Sep 21

Rethinking Human-Aligned Evaluation: An Analysis of Semantic Metrics Beyond WER

arXiv:2609.21663v1 Announce Type: new Abstract: Word Error Rate (WER), the most commonly used metric for Automatic Speech Recognition (ASR), treats every lexical deviation from the reference as equal...

By Hritika Sharma, Thibault Ba\~neras-Roux, Alessandra Pinto, Petr Motlicek, Hyunggu Jung, Esa\'u Villatoro-Tello, Somang Nam

llms multimodal benchmarks

arXiv Computation and Language

Sep 21

Evaluating In-Context Learning and Retrieval Strategies for Devanagari Post-OCR Correction

arXiv:2609.21595v1 Announce Type: new Abstract: In-context learning using Large Language Models (LLMs) offers a compelling path to training-free post-OCR correction, yet its effectiveness for Devanag...

By Abhishek Bhandari, Gaurav Harit

llms computer-vision fine-tuning benchmarks

Get the daily briefing