Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv AI

Sep 16

The Troy Moment of AI: Why Some Will Cheat and Some Will Follow?

arXiv:2609.15494v2 Announce Type: replace Abstract: Recent investigations of the July 2026 OpenAI-Hugging Face incident motivate two questions: when an assigned task becomes impossible, does an agent...

By Ivy Zhang

llms agents benchmarks

arXiv AI

Sep 16

MASCOT: Multi-Agent Socio-Collaborative Companion Systems

arXiv:2601.14230v3 Announce Type: replace-cross Abstract: Multi-agent systems (MAS) are emerging as promising socio-collaborative companions for emotional and cognitive support. However, existing sys...

By Yiyang Wang, Yiqiao Jin, Alex Cabral, Josiah Hester

llms agents benchmarks safety

arXiv AI

Sep 16

Comparative Characterization of KV Cache Management Strategies for LLM Inference

arXiv:2604.05012v2 Announce Type: replace-cross Abstract: Efficient inference with Large Language Models (LLMs) increasingly relies on Key-Value (KV) caches to store previously computed key and value...

By Oteo Mamo, Olga Kogiou, Hyunjin Yi, Weikuan Yu

llms efficiency benchmarks

arXiv AI

Sep 16

Ave: Guiding Agentic GPU Optimization Using Data-Flow Invariants

arXiv:2604.18616v2 Announce Type: replace-cross Abstract: LLM coding agents can generate correct GPU kernels, but their performance still trails expert libraries. Reaching peak throughput requires co...

By Haohui Mai, Xiaoyan Guo, Xiangyun Ding, Daifeng Li, Qiuchu Yu, Chenzhun Guo, Cong Wang, Jiacheng Zhao, Christos Kozyrakis, Binhang Yuan

llms rag agents reinforcement-learning efficiency benchmarks safety

arXiv AI

Sep 16

REDDIT: Forgetting-Resistant Correction of Timestamp Drift in ASR via Replay-Based Distribution Editing

arXiv:2607.05364v4 Announce Type: replace-cross Abstract: Modern autoregressive ASR systems can emit timestamps as decoded tokens, enabling timestamped transcription without frame-level aligners or i...

By Cheng-Kang Chou, Ming-To Chuang, Ke-Han Lu, Chan-Jan Hsu, Hung-yi Lee

fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Sep 16

Neural Field Ensembles for Aerodynamic Surface Prediction: Winning Solution to the ONERA CRM Wall Distribution 2025 Challenge

arXiv:2609.17160v1 Announce Type: new Abstract: Machine-learning surrogate models offer a promising alternative to high-fidelity Computational Fluid Dynamics (CFD) simulations for aerodynamic analysi...

By Lionel Salesses, Caroline Sainvitu, Tariq Benamara

benchmarks

arXiv Machine Learning

Sep 16

Towards Scalable RLVR: Multimodal Instruction Following Data Synthesis and Distillation

arXiv:2609.16059v1 Announce Type: cross Abstract: Multimodal instruction following (MMIF) is crucial for building generalist agents. However, current training paradigms rely heavily on Supervised Fin...

By Yirong Zeng, Zhang Sai, Yuxian Wang, Yutai Hou, Yufei Liu, Xiao Ding, Bibo Cai

agents reinforcement-learning fine-tuning efficiency multimodal benchmarks

arXiv Machine Learning

Sep 16

TEMPO: Learning Temporal Context for Dynamic Robot Manipulation

arXiv:2609.16864v1 Announce Type: cross Abstract: Vision-language-action (VLA) models have achieved impressive performance in quasi-static manipulation, but struggle in dynamic manipulation tasks bec...

By Zhenyang Feng, Jimin Heo, Erik B. Sudderth, Unnat Jain

robotics efficiency multimodal benchmarks

arXiv Machine Learning

Sep 16

OptiPrime: Optimizing Private Inference through Protocol-Hardware Co-design

arXiv:2609.16898v1 Announce Type: cross Abstract: Private deep neural network (DNN) inference based on hybrid homomorphic encryption (HE) and multi-party computation (MPC) can protect user data with...

By Jiangrui Yu, Ye Yu, Si Chen, Chenqi Lin, Wenxuan Zeng, Junfeng Fan, Mingyu Gao, Meng Li

benchmarks

arXiv Machine Learning

Sep 16

Tables Decoded: DELTA for Structure, TARQA for Understanding

arXiv:2609.17458v1 Announce Type: cross Abstract: Table understanding is a core task in document intelligence, encompassing two key subtasks: table reconstruction and table visual question answering...

By Jahanvi Rajput, Dhruv Kudale, Saikiran Kasturi, Utkarsh Verma, Ganesh Ramakrishnan

llms computer-vision nlp fine-tuning benchmarks

arXiv Machine Learning

Sep 16

CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference

arXiv:2604.06036v4 Announce Type: replace-cross Abstract: Continuous inference over concurrent video streams imposes substantial compute and memory demands on vision-language model (VLM) serving. Str...

By Yulin Zou, Wenyan Chen, Yan Chen, Anya Rajan, JooYoung Park, Shivaraman Nitin, Luo Tao, Francisco Romero, Dmitrii Ustiugov

llms efficiency multimodal benchmarks

arXiv Computation and Language

Sep 16

Negation Beyond the Verbal Channel: Temporal Multimodal Correlates in Dialogue

arXiv:2609.16396v1 Announce Type: new Abstract: Negation is typically modeled through its linguistic realization, although spoken interaction is accompanied by tightly coordinated nonverbal behavior....

By Leon Hammerla, Patrick Schrottenbacher, Alexander Mehler

multimodal benchmarks

arXiv Computation and Language

Sep 16

Can LLMs Follow the Pulse of a Crisis? Evaluating Crisis Sentiment in Bangladesh's July Uprising

arXiv:2609.16997v1 Announce Type: new Abstract: Crisis sentiment analysis is especially challenging for low-resource languages such as Bangla, where language, context, and public reaction shift rapid...

By Md. Samiul Alim, Mahir Shahriar Tamim, Tanvir Ahmed Khan, Sharjil Khan, Rafia Ferdous Duti, Shahriyar Zaman Ridoy, Mohammad Ali Moni

llms nlp fine-tuning benchmarks

arXiv Computation and Language

Sep 16

ECHO: Early-layer Collaborative Hierarchical Orchestration with Bonus Logits in Speculative Decoding

arXiv:2609.17241v1 Announce Type: new Abstract: While draft-model-free speculative decoding offers a promising path to efficient LLM inference, it is frequently constrained by stale draft candidates...

By Ziyang Ma, Zihong Zhang, Zuchao Li, Lefei Zhang, Baoyuan Qi, Siqi Li, Simin Yu

llms agents fine-tuning benchmarks

arXiv Computation and Language

Sep 16

Same Problem, Different Field: Cross-Domain Solution Import via Domain-Stripped Computational Fingerprints

arXiv:2609.07595v2 Announce Type: replace-cross Abstract: The same underlying computational problem is solved across unrelated fields under different names: recursive Bayesian state estimation appear...

By Eryk Kulikowski

llms rag efficiency benchmarks

arXiv Computer Vision

Sep 16

FAHCD-Net: Frequency-Adaptive Heatmap-Conditional Diffusion Networks for Robust Facial Landmark Detection

arXiv:2609.16842v1 Announce Type: new Abstract: Facial Landmark Detection(FLD) is a crucial task in various applications and has achieved significant advancements in recent years. However, current FL...

By Jun Wan, Jiwei Hu, Shengkai Hu, Qilu Zhu

diffusion benchmarks

arXiv Computer Vision

Sep 16

Bridging the Perceptual Gap: Residual-Enhanced Downscaling and Manifold-Aware Perception Alignment Adaptation for NR-IQA

arXiv:2609.16664v1 Announce Type: new Abstract: Leveraging Large Vision-Language Models like CLIP has recently set new benchmarks for No-Reference Image Quality Assessment (NR-IQA). However, the cont...

By Yu Li, Zhengran Shen, Yachun Mi, Puchao Zhou, Shaohui Liu

llms multimodal benchmarks safety

arXiv Computer Vision

Sep 16

SSC-Priors: Exploring Semantic and Visibility Priors to Boost Lidar Semantic Scene Completion

arXiv:2609.17413v1 Announce Type: new Abstract: This paper investigates easy strategies to boost the performance of existing networks for lidar semantic scene completion (SSC) without requiring compl...

By Tetiana Martyniuk, Jonathan Seele, Alexandre Boulch, Gilles Puy, Renaud Marlet, Raoul de Charette

computer-vision benchmarks

arXiv Computer Vision

Sep 16

HuMemSLAM: Efficient Human-Inspired Semantic Place Recognition for Robust Visual SLAM

arXiv:2609.17168v1 Announce Type: cross Abstract: Autonomous systems require reliable place recognition for efficient and effective simultaneous localisation and mapping (SLAM). Traditional geometric...

By Mayowa Adebambo, Sebastian Donnelly, Armand Amaritei, Andrew Bradley, Alexander Rast

agents benchmarks

arXiv Computer Vision

Sep 16

Unlocking Zero-shot Potential of Semi-dense Image Matching via Gaussian Splatting

arXiv:2511.21265v2 Announce Type: replace Abstract: Learning-based image matching critically depends on large-scale, diverse, and geometrically accurate training data. 3D Gaussian Splatting (3DGS) en...

By Juncheng Chen, Chao Xu, Yanjun Cao

benchmarks safety

Get the daily briefing