Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv AI

Sep 30

CineSubBench: Evaluating LLMs on Long-Form Narrative and Cultural Understanding from Multilingual Movie Subtitles

arXiv:2609.36218v1 Announce Type: cross Abstract: Large language models are increasingly evaluated in specialized domains such as law, medicine, software engineering, and cybersecurity, yet film rema...

By Mir Tafseer Nayeem, Susmoy Chakraborty, Davood Rafiei

llms benchmarks safety

arXiv AI

Sep 30

MoRE: Scaling mixture of experts with hardware-aware low-rank routing

arXiv:2609.36301v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) layers are central to frontier language models, and recent architectures push toward more and smaller experts. In this regim...

By Honam Wong, Surbhi Goel, Enric Boix-Adser\`a

llms benchmarks

arXiv AI

Sep 30

LLMs Learn to Evade Latent Monitors from Prior Feedback Alone

arXiv:2609.36490v1 Announce Type: cross Abstract: Latent space monitors aim to detect undesired behaviors in LLM agents by inspecting an agent's internal activations rather than its outputs. However,...

By Hugo Lyons Keenan, Christopher Leckie, Sarah Erfani

llms agents diffusion fine-tuning benchmarks

arXiv AI

Sep 30

FM-ReID: Selective Competitive Token Routing for Object Re-Identification

arXiv:2609.36560v1 Announce Type: cross Abstract: Object re-identification (ReID) faces a recurring challenge: different identities can share highly similar global appearances, while the cues that di...

By Zhiqi Li, Xiaowei Zhou, Zeyuan Sun, Feng Gao, Junyu Dong

benchmarks

arXiv AI

Sep 30

Interactive-Policy Distillation with Bidirectional Propose-and-Verify

arXiv:2609.36546v1 Announce Type: cross Abstract: On-policy distillation (OPD) trains a student model on its self-generated trajectories with dense token-level teacher feedback. However, naive OPD ma...

By Shutong Wu, Xiwen Chen, Brendan Rappazzo, Daiheng Zhang, Anderson Schneider, Yuriy Nevmyvaka, Jiawei Zhang

efficiency benchmarks

arXiv AI

Sep 30

Simple Agentic Memory for Generalist Robot Policies

arXiv:2609.36595v1 Announce Type: cross Abstract: Visual-memory systems commonly retain or compress past observations. Robot control additionally requires interaction-derived state that no individual...

By Yuyou Zhang, Yunbei Zhang, Miao Li, Janet Wang, Zijian Jin, Shilong Liu, Ding Zhao

agents robotics benchmarks

arXiv AI

Sep 30

Frontier Autolab: Organizational Memory, Adversarial Dissent and Temporal Leakage in Multi-Agent LLM Firms Across Fifty Years of Technological Change

arXiv:2609.36739v1 Announce Type: cross Abstract: Multi-agent LLM systems are increasingly structured like organizations, with roles, critics and shared memory, yet they are evaluated on tasks that l...

By Bravish Ghosh

llms agents benchmarks safety

arXiv AI

Sep 30

Dual-Mode Low-Rank Learner with Bridge-Prototype Ensemble for Vision-Language Class-Incremental Learning

arXiv:2609.36759v1 Announce Type: cross Abstract: Benefiting from transferable visual-textual alignment, CLIP has been widely adopted for class-incremental learning (CIL). However, existing learners...

By Chiyuan He, Zihuan Qiu, Fanman Meng, Chao Wang, Liangjiang Chen, Linfeng Xu, Qingbo Wu, Hongliang Li

rag multimodal benchmarks safety

arXiv AI

Sep 30

Visual Parallel Search: Learning to Search High-Resolution Images with Parallel Tile Inspection and Adaptive Zoom

arXiv:2609.37002v1 Announce Type: cross Abstract: High-resolution visual question answering often fails because a multimodal model does not acquire the small, spatially localized evidence needed to a...

By Xijia Tao, Yihua Teng, Xinyu Fu, Cheng Gong, Ziru Liu, Xudong Xie, Rui Liu, Lingpeng Kong

agents nlp reinforcement-learning fine-tuning multimodal benchmarks

arXiv AI

Sep 30

Abductive World Modeling via Causal Representation Learning

arXiv:2609.36985v1 Announce Type: cross Abstract: The central challenge of world modeling is to learn representations that capture how the world evolves. However, existing world models predominantly...

By Ziqi Liu, Songhan Yang, Linfan Zhou, Jiatong Liu, Lijun Peng, Long Wan, Yinqi Bai

benchmarks

arXiv AI

Sep 30

Beam Search as Test-Time Self-Distillation via Counterfactual Contexts

arXiv:2609.37041v1 Announce Type: cross Abstract: Self-Distillation Fine-Tuning (SDFT) enables a language model to act as its own teacher: by conditioning on a demonstration, the model produces an im...

By Su Ee Tan, Xiaotong Ji, Rasul Tutunov, Haitham Bou-Ammar, Matthieu Zimmer

llms fine-tuning efficiency benchmarks

arXiv AI

Sep 30

DatalogBench: Evaluating Large Language Models on Text-to-Datalog Synthesis

arXiv:2609.37233v1 Announce Type: cross Abstract: Datalog underpins reasoning tasks such as program analysis, but its programs are hard to write. Existing synthesizers automate this task but require...

By Yuan Li, Hanyun Jiang, Guowei Tian, Chengpeng Wang, Peisen Yao

llms agents benchmarks

arXiv AI

Sep 30

PowerMarketJax: A JAX Benchmark Suite for Multi-Agent Reinforcement Learning in Power Markets

arXiv:2609.37321v1 Announce Type: cross Abstract: Power markets are a natural testbed for multi-agent reinforcement learning (MARL), where multiple self-interested participants repeatedly submit bids...

By Zhanhua Pan, Xin Qin, Xiao Liu, Zhilong Cao, Jianhong Wang, Dawei Qiu

agents reinforcement-learning benchmarks

arXiv AI

Sep 30

LazySloth: Bounded LLM-based Lazy Tree Search for Fast Long Video Comprehension

arXiv:2609.37426v1 Announce Type: cross Abstract: Modern vision-language models (VLMs) have shown promising results in long-video understanding due to the rich semantic information they can capture....

By Arka Mukherjee, Kaleen Shrestha, Larissa Zhu, Maja Matari\'c

llms rag agents multimodal benchmarks

arXiv AI

Sep 30

Risk-Aware Semantic Grounding for Trustworthy LLM-Based Robot Planning

arXiv:2609.37554v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as high-level planners in robot navigation, but their outputs may become unreliable when instructi...

By {\L}ukasz Sobczak, Nur Kele\c{s}o\u{g}lu, S{\l}awomir Piotr Nowak

llms robotics benchmarks safety

arXiv AI

Sep 30

Semantic Map Sharing and Capability-Aware Coverage Planning for AI-Native 6G Robotic Coordination

arXiv:2609.37666v1 Announce Type: cross Abstract: Search and Rescue (SAR) operations increasingly deploy heterogeneous teams of aerial and ground robots. However, conventional coverage methods typica...

By Abdulqader Dhafer, Qi Wang, Zhou Daniel Hao

robotics benchmarks

arXiv AI

Sep 30

Making Duplicate Reimbursement Unrepresentable: A Verified Ethereum E-Invoice System for Humans and AI Agents

arXiv:2609.37819v1 Announce Type: cross Abstract: Electronic invoices are replacing paper invoices worldwide, but today's centralized architectures leave three problems unsolved on the consumption si...

By Jia Cai

llms agents benchmarks safety

arXiv AI

Sep 30

Breaking the Uniformity Trap: Scaling Video Diffusion Model via SplitMoE

arXiv:2609.38140v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE), popularized by large language models, is a promising paradigm for scaling visual generative models. However, conventional t...

By Yu Xu, Yuxin Zhang, Xiao Yang, Haotian Yang, Yizhi Wang, Xinwei Huang, Minxuan Lin, Angtian Wang, Chongyang Ma, Fan Tang

llms diffusion benchmarks

arXiv AI

Sep 30

Evaluating Test-Time Scaling of General LLM Agents

arXiv:2602.18998v2 Announce Type: replace Abstract: LLM agents are increasingly expected to operate as general-purpose systems that resolve real-world user requests, yet their dynamic scaling behavio...

By Xiaochuan Li, Ryan Ming, Pranav Setlur, Abhijay Paladugu, Andy Tang, Hao Kang, Shuai Shao, Rong Jin, Chenyan Xiong

llms agents benchmarks

arXiv AI

Sep 30

Are Agents Ready to Teach? A Multi-Stage Benchmark for Real-World Teaching Workflows

arXiv:2605.14322v4 Announce Type: replace Abstract: Language agents are increasingly deployed in professional workflows, yet tutoring remains a high-stakes capability that existing evaluations only p...

By Zixin Chen, Peng Liu, Rui Sheng, Haobo Li, Jianhong Tu, Xiaodong Deng, Kashun Shum, Dayiheng Liu, Huamin Qu

agents benchmarks

Get the daily briefing