Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,859 stories · RSS feed

arXiv AI

Jun 16

MMLongEmbed: Benchmarking Multimodal Embedding Models in Long-Context Scenarios

arXiv:2606. 14747v1 Announce Type: cross Abstract: Recent advancements have significantly expanded the theoretical context windows of Multimodal Embedding Models (MEMs).

By Haitian Wang, Ruoxi Sun, Quantong Qiu, Juntao Li, Junhui Li, Hua Chen, Jinxiong Chang, Min Zhang

rag multimodal benchmarks

arXiv AI

Jun 16

When Do LLMs Reason? A Dynamical Systems View via Entropy Phase Transitions

arXiv:2605. 22873v2 Announce Type: replace-cross Abstract: Chain-of-thought (CoT) reasoning has become the default strategy for enhancing LLM capabilities, yet its application raises a fundamental question: when is explicit reasoning actually beneficial?

By Wei Xia, Haoqing Wang, Zhi-Hong Deng, Yehui Tang

llms benchmarks

arXiv AI

Jun 16

MiroBench: Benchmarking Realism in Agentic Simulation of Real-world Discussions

arXiv:2606. 14715v1 Announce Type: cross Abstract: LLM agents are increasingly used to simulate real world interactions, but it remains unclear whether simulated behaviors preserve the content patterns and interaction dynamics of real human behaviors.

By Yaoning Yu, Ye Yu, Haojing Luo, Haohan Wang

llms agents benchmarks safety

arXiv AI

Jun 16

BRIDGE: Biological Evidence Refinement and Heterogeneous Dynamic Gating for Gene Regulatory Networks

arXiv:2606. 14734v1 Announce Type: cross Abstract: Motivation: Gene regulatory network inference from single-cell RNA sequencing (scRNA-seq) data is important for uncovering cell-state-specific transcriptional programs.

By Ziyang Dong, Shanwen Tan, Hengchuang Yin, Wei Liu, Yifan Wang, Siyu Yi, Jiancheng Lv, Wei Ju

benchmarks

arXiv Machine Learning

Jun 16

Distribution Alignment for One-Shot Federated Learning via Optimal Transport

arXiv:2606. 16655v1 Announce Type: new Abstract: One-Shot Federated Learning (OSFL) addresses extreme communication regimes in which clients interact with the server only once, amplifying the impact of heterogeneous client data distributions.

By Daniele Berardini (AI for Good), Vito Paolo Pastore (AI for Good, MaLGa-DIBRIS, University of Genoa, Genoa, Italy), Vittorio Murino (AI for Good, Department of Computer Science, University of Verona, Verona, Italy)

efficiency benchmarks safety

arXiv AI

Jun 16

SkillsVote: Lifecycle Governance of Agent Skills from Collection, Recommendation to Evolution

arXiv:2605. 18401v2 Announce Type: replace-cross Abstract: Long-horizon LLM agents generate traces that could become reusable experience, but raw trajectories are noisy, local, and hard to govern.

By Hongyi Liu, Haoyan Yang, Tao Jiang, Bo Tang, Feiyu Xiong, Yuyu Luo, Zhiyu Li

llms agents benchmarks

arXiv AI

Jun 16

Rethinking Scaffolding in LLM Tutors: The Interactional Mismatch Between Benchmarks and Real-World Deployments

arXiv:2606. 15766v1 Announce Type: new Abstract: A central pedagogical value evaluated in AI tutor benchmarks is scaffolding: guiding students through graduated steps toward a solution.

By Alexandra Neagu, Jeffrey T. H. Wong, Marcus Messer, Rhodri Nelson, Peter B. Johnson

llms rag benchmarks safety

arXiv Machine Learning

Jun 16

Dual-Network PINNs for Optimal Control: A Reproducible Benchmark on the Mass-Spring-Damper System

arXiv:2606. 15271v1 Announce Type: cross Abstract: This work presents a transparent and reproducible benchmark study of a direct dual-network Physics-Informed Neural Network (PINN) formulation for the optimal control of a mass-spring-damper system.

By Abdeladhim Tahimi, Rinaldo Vieira da Silva Junior

benchmarks

arXiv AI

Jun 16

CLoVE: Personalized Federated Learning through Clustering of Loss Vector Embeddings

arXiv:2506. 22427v2 Announce Type: replace-cross Abstract: We propose CLoVE (Clustering of Loss Vector Embeddings), a novel algorithm for Clustered Federated Learning (CFL).

By Randeep Bhatia, Nikos Papadis, Murali Kodialam, TV Lakshman, Sayak Chakrabarty

rag benchmarks

arXiv AI

Jun 16

RetailBench: Benchmarking long horizon reasoning and coherent decision making of LLM agents in realistic retail environments

arXiv:2606. 15862v1 Announce Type: new Abstract: Large language model (LLM) agents have made rapid progress on short-horizon, well-scoped tasks, yet their ability to sustain coherent decisions in dynamic long-horizon environments remains uncertain.

By Linghua Zhang, Jun Wang, Jingtong Wu, Zhisong Zhang

llms agents benchmarks

arXiv AI

Jun 16

FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies

arXiv:2605. 27284v2 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models are increasingly expected to not only complete robot tasks, but also follow human instructions about how those tasks should be executed.

By Xintong Hu, Xuhong Huang, Jinyu Zhang, Yutong Yao, Yuchong Sun, Qiuyue Wang, Mingsheng Li, Sicheng Xie, Yitao Liu, Junhao Chen, Yixuan Chen, Yingming Zheng, Shuai Bai, Tao Yu

robotics multimodal benchmarks safety

arXiv AI

Jun 16

Cascaded Sparse Autoencoders Learn Multi-Level Visual Concepts in Multimodal LLMs

arXiv:2606. 16193v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have demonstrated strong performance on vision-language tasks, yet their internal visual representations remain difficult to interpret.

By Yusong Zhao, Hengyi Wang, Tanuja Ganu, Akshay Nambi, Hao Wang

llms multimodal benchmarks safety

arXiv Machine Learning

Jun 16

A Biased Nonnegative Block Term Tensor Decomposition Model for Dynamic QoS Prediction

arXiv:2605. 04813v2 Announce Type: replace Abstract: With the rapid development of cloud computing and Web services, Quality of Service (QoS) has become a key criterion for service selection and recommendation.

By Wenjing Liu, Yujia Lei, Qu Wang

benchmarks safety

arXiv AI

Jun 16

Driving, Fast or Slow? Neuro-Symbolic Guidance for Motion Prediction in Multi-Modal Ground Mobility

arXiv:2606. 15251v1 Announce Type: cross Abstract: Accurate and interpretable motion prediction for heterogeneous traffic spaces, including pedestrians, bicycles, cars, and trucks, is essential for safe autonomous navigation.

By Simon Kohaut, Felix Divo, Julius Hahnewald, Benedict Flade, Julian Eggert, Kristian Kersting, Devendra Singh Dhami

agents multimodal benchmarks

arXiv Machine Learning

Jun 16

FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policies

arXiv:2603. 27450v2 Announce Type: replace Abstract: Thanks to their remarkable flexibility, diffusion models and flow models have emerged as promising candidates for policy representation.

By Chenxiao Gao, Edward Chen, Tianyi Chen, Bo Dai

diffusion reinforcement-learning robotics efficiency benchmarks

arXiv AI

Jun 16

ACC: Compiling Agent Trajectories for Long-Context Training

arXiv:2605. 21850v2 Announce Type: replace-cross Abstract: Recent development of agents has renewed demand for long-context reasoning capacity of LLMs.

By Qisheng Su, Zhen Fang, Shiting Huang, Yu Zeng, Yiming Zhao, Kou Shi, Ziao Zhang, Lin Chen, Zehui Chen, Lijun Wu, Feng Zhao

llms agents fine-tuning benchmarks

arXiv AI

Jun 16

Double-Helix Vision (DH-V2): A Geometry-Based Visual Sampler for Bandwidth-Constrained Perception

arXiv:2606. 14773v1 Announce Type: cross Abstract: We present Double-Helix Vision (DH), a geometry-based visual sampler that compresses 2D images into compact 1D signals using paired golden-ratio-inspired spiral trajectories.

By Jinwen Wen

robotics benchmarks

arXiv AI

Jun 16

Is Your Trajectory Displacement Safe in Long-tail?

arXiv:2606. 16313v1 Announce Type: cross Abstract: Long-tail scenarios remain a major bottleneck for autonomous driving evaluation, even as datasets grow by orders of magnitude.

By Qiao Sun, Weicheng Zheng, Yixin Huang, Hang Zhao

agents robotics benchmarks safety

arXiv AI

Jun 16

CAP: Towards PPG Universal Representation Learning with Patient-level Supervision

arXiv:2606. 15284v1 Announce Type: cross Abstract: Photoplethysmography (PPG) plays a central role in wearable health monitoring and clinical decision support.

By Chenyang He, Xinyi Shao, Shun Huang, Bosong Huang, Daoqiang Zhang, Ming Jing, Cheng Ding

multimodal benchmarks safety

arXiv AI

Jun 16

EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective

arXiv:2605. 18421v2 Announce Type: replace-cross Abstract: Recent benchmarks for Large Language Model (LLM) agents mainly evaluate reasoning, planning, and execution.

By Yuyao Wang, Zhongjian Zhang, Mo Chi, Kaichi Yu, Yuhan Li, Miao Peng, Bing Tong, Chen Zhang, Yan Zhou, Jia Li

llms agents benchmarks

Get the daily briefing