Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

19,504 stories · RSS feed

arXiv AI

Jun 26

Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planning

arXiv:2606. 27330v1 Announce Type: cross Abstract: Multimodal web agents can assist humans in operating repetitive GUI tasks, where effective task planning is essential for decomposing complex tasks into executable actions.

By Tianyi Men, Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao

agents multimodal benchmarks safety

arXiv AI

Jun 26

Do Safety Guardrails Need to Reason? LeanGuard: A Fast and Light Approach for Robust Moderation

arXiv:2606. 26686v1 Announce Type: new Abstract: In order to screen a prompt or a response, the recent guardrail methods generate a chain-of-thought (CoT) before they issue a verdict.

By Dongbin Na

robotics benchmarks safety

arXiv Machine Learning

Jun 26

RSPC: A Benchmark for Modeling Stress and Psychiatric Conditions in Digitally Mediated Relationships using Psychiatrist Annotations

arXiv:2606. 27247v1 Announce Type: new Abstract: In NLP, mental health conditions are often modeled as isolated phenomena, without interpersonal context.

By Parmitha Vangapandu, Sai Ganesh Mokkapati, Sathwik Narkedimilli, MSVPJ Sathvik, Timothy Liu, Simon See, Johannes C. Eichstaedt

llms nlp fine-tuning benchmarks

arXiv AI

Jun 26

MIRROR: Novelty-Constrained Memory-Guided MCTS Red-Teaming for Agentic RAG

arXiv:2606. 26793v1 Announce Type: cross Abstract: Multimodal agentic retrieval-augmented generation (RAG) systems expand the attack surface beyond prompt injection to include text poisoning, image injection, direct-query attacks, and orchestrator-level tool manipulation.

By Inderjeet Singh, Andr\'es Murillo, Motoyoshi Sekiya, Yuki Unno, Junichi Suga

rag agents robotics multimodal benchmarks safety

arXiv AI

Jun 26

Learning Motion Feasibility from Point Clouds in Cluttered Environments

arXiv:2606. 26700v1 Announce Type: cross Abstract: Motion feasibility prediction plays a central role in robotics, particularly in task and motion planning and manipulation.

By Sajid Ansari, Arthi, Girish Varma, Antony Thomas

llms robotics benchmarks

arXiv AI

Jun 26

Look-Before-Move: Narrative-Grounded World Visual Attention in Dynamic 3D Story Worlds

arXiv:2606. 26964v1 Announce Type: new Abstract: As embodied AI and world models increasingly operate in dynamic 3D environments, visual perception must move beyond passively interpreting given observations toward actively deciding what to observe.

By Jiaming Bian, Bingliang Li, Yuehao Wu, Pichao Wang, Zhi Wang, Hailan Ma, Huadong Mo, Zhenhong Sun

robotics benchmarks

arXiv AI

Jun 26

DMSC: Dynamic Multi-Scale Coordination Framework for Time Series Forecasting

arXiv:2508. 02753v5 Announce Type: replace-cross Abstract: Time Series Forecasting (TSF) faces persistent challenges in modeling intricate temporal dependencies across different scales.

By Haonan Yang, Jianchao Tang, Zhuo Li, Long Lan

llms multimodal benchmarks

arXiv AI

Jun 26

Context Recycling for Long-Horizon LLM Inference

arXiv:2606. 26105v1 Announce Type: cross Abstract: Large language models (LLMs) exhibit strong capabilities in short-context reasoning but degrade in performance over long conversational horizons due to context window limitations and inefficient token usage.

By Derek Thomas

llms agents benchmarks

arXiv AI

Jun 26

The Inattentional Gap: Task-Conditioned Language and Vision Models Omit the Safety-Critical Signals They Can Otherwise Report

arXiv:2606. 26529v1 Announce Type: cross Abstract: AI safety is evaluated by how reliably a model detects the hazards it is told to find, yet accidents often arise from the hazard no one specified.

By Kwan Soo Shin

benchmarks safety

arXiv AI

Jun 26

Autoregressive Boltzmann Generators

arXiv:2606. 27361v1 Announce Type: cross Abstract: Efficient sampling of molecular systems at thermodynamic equilibrium is a hallmark challenge in statistical physics.

By Danyal Rehman, Charlie B. Tan, Yoshua Bengio, Avishek Joey Bose, Alexander Tong

llms benchmarks

arXiv AI

Jun 26

Helpfulness Hurts: Domain-Dependent Degradation of Mid-Trained Compassion Values Under Post-Training

arXiv:2606. 26102v1 Announce Type: cross Abstract: Standard post-training pipelines apply supervised fine-tuning (SFT) and reinforcement learning (RL) to make language models helpful, but these processes may inadvertently degrade values instilled during pre-training.

By Jasmine Brazilek, Juliana Seawell

llms reinforcement-learning fine-tuning benchmarks

arXiv AI

Jun 26

Disco-LoRA: Disentangled Composition of Content, Style, and Motion for Multi-concept Video Customization

arXiv:2606. 26668v1 Announce Type: cross Abstract: Video customization based on Text-to-Video (T2V) models aims to learn specific features from reference data to generate controllable videos.

By Xuancheng Xu, Gengyun Jia, Bing-Kun Bao

diffusion fine-tuning benchmarks

arXiv AI

Jun 26

ResilPhase: Plug-and-Play Phase Mapping and Noise-Resilient Macro-Trajectory Extrapolation for Diffusion Acceleration

arXiv:2606. 26769v1 Announce Type: new Abstract: The adoption of powerful diffusion models is hindered by their significant inference latency.

By Qicheng Zhao, Yu Li, Qi Sun, Zheyu Yan

diffusion efficiency benchmarks

arXiv AI

Jun 26

EVOM: Agentic Meta-Evolution of Actor-Critic Architectures for Reinforcement Learning

arXiv:2606. 26327v1 Announce Type: cross Abstract: In actor-critic reinforcement learning, network architectures are typically manually designed.

By Boyun Zhang, Chao Wang, Kai Wu

llms agents reinforcement-learning benchmarks

arXiv AI

Jun 26

CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs

arXiv:2606. 26650v1 Announce Type: cross Abstract: In this paper, we present CAT-Q, Cost-efficient and Accurate Ternary Quantization, for compressing and accelerating LLMs.

By Shigeng Wang, Chao Li, Yangyuxuan Kang, Jiawei Fan, Anbang Yao

llms efficiency benchmarks

arXiv AI

Jun 26

Confidence-Aware Tool Orchestration for Robust Video Understanding

arXiv:2606. 26904v1 Announce Type: cross Abstract: Video reasoning language models implicitly assume that every input frame is equally reliable.

By Yangfan He, Yujin Choi, Jaehong Yoon

llms agents robotics benchmarks

arXiv AI

Jun 26

Limited Reference, Reliable Generation: A Two-Component Framework for Tabular Data Generation in Low-Data Regimes

arXiv:2509. 09960v2 Announce Type: replace-cross Abstract: Synthetic tabular data generation is increasingly essential in machine learning, supporting downstream applications when real-world, high-quality tabular data is insufficient.

By Mingxuan Jiang, Keyang Chen, Yongxin Wang, Yongsheng Zhao, Ziyue Dai, Yicun Liu, Zeping Li, Qiuyang Zhang, Hongyi Nie, Hongbin Zhu, Sen Liu, Guangnan Ye, Hongfeng Chai

llms diffusion fine-tuning benchmarks safety

arXiv AI

Jun 26

The Best of the Two Worlds: Harmonizing Semantic and Hash IDs for Sequential Recommendation

arXiv:2512. 10388v3 Announce Type: replace-cross Abstract: Conventional Sequential Recommender Systems (SRS) typically assign unique hash IDs (HID) to construct item embeddings, which mainly capture collaborative signals from historical user-item interactions.

By Ziwei Liu, Yejing Wang, Wanyu Wang, Wang Zejian, Qidong Liu, Zijian Zhang, Chong Chen, Wei Huang, Xiangyu Zhao

rag efficiency benchmarks safety

arXiv Machine Learning

Jun 26

Can Large Language Models Reliably Code Qualitative Humanitarian Data? A Benchmark Study Against Human Expert Adjudication

arXiv:2606. 26541v1 Announce Type: new Abstract: Data from affected populations are crucial for informing humanitarian response, but their value depends on timely and consistent interpretation of nuanced accounts of need.

By Jerome Marston, Tino Kreutzer, Salom\'e Garnier, Ella Boone, Phuong N Pham, Patrick Vinck

llms benchmarks safety

arXiv AI

Jun 26

Temporal Validity in Retrieval Memory: Eliminating Stale-Fact Errors for AI Agents over Evolving Knowledge

arXiv:2606. 26511v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) gives agents access to accumulated knowledge, but has no model of time.

By Neeraj Yadav

llms rag agents benchmarks

Get the daily briefing