Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,561 stories · RSS feed

arXiv AI

Jun 16

Mask Proposal Voting Based on Geodesic Framework for Robust Image Segmentation

arXiv:2606. 14912v1 Announce Type: cross Abstract: Despite great advances, finding accurate segmentation remains a challenging task, especially in scenarios with cluttered backgrounds, complex intensity variations and topology appearance.

By Li Liu, Mingzhu Wang, Zhenjiang Li, Da Chen, Laurent D. Cohen

computer-vision benchmarks

arXiv AI

Jun 16

A Survey on 3D Skeleton Based Person Re-Identification: Taxonomy, Advances, Challenges, and Interdisciplinary Prospects

arXiv:2401. 15296v4 Announce Type: replace-cross Abstract: Person re-identification via 3D skeletons is an important emerging research area that attracts increasing attention within the pattern recognition community.

By Haocong Rao, Chunyan Miao

benchmarks

arXiv Machine Learning

Jun 16

Agent trajectories as programs: fingerprinting and programming coding-agent behavior

arXiv:2606. 16988v1 Announce Type: cross Abstract: Benchmark scores tell you what an agent got right; they do not tell you how it got there.

By Hamidah Oderinwale

agents benchmarks

arXiv Machine Learning

Jun 16

Multi-Fidelity SINDy: Sparse Discovery of Nonlinear Dynamical Systems with Fidelity-Weighted Measurements

arXiv:2606. 15690v1 Announce Type: new Abstract: Data from simulations and experiments are rarely noise-free and often exhibit heterogeneous levels of fidelity.

By Filippo Zacchei, Ana Larra\~naga, Attilio Frangi, Andrea Manzoni, Steven L. Brunton

benchmarks

arXiv AI

Jun 16

Federated Medical Image Segmentation under Real-World Label Noise: A Benchmark Suite for Noisy Label Learning Method Selection

arXiv:2606. 16868v1 Announce Type: cross Abstract: While federated learning (FL) enables collaborative medical image segmentation without centralizing sensitive data, real-world deployment is frequently complicated by cross-site label imperfections such as contour disagreement, missing or additional structures, and confused labels.

By Markus Bujotzek, Dimitrios Bounias, Stefan Denner, Ralf Floca, Maximilian Fischer, Peter Neher, Klaus Maier-Hein

computer-vision benchmarks

arXiv Machine Learning

Jun 16

A Biased Nonnegative Block Term Tensor Decomposition Model for Dynamic QoS Prediction

arXiv:2605. 04813v2 Announce Type: replace Abstract: With the rapid development of cloud computing and Web services, Quality of Service (QoS) has become a key criterion for service selection and recommendation.

By Wenjing Liu, Yujia Lei, Qu Wang

benchmarks safety

arXiv AI

Jun 16

Bridging Geographic Bias in Urban Streetscape Inference via Lifelong Learning with Visual-Semantic Pivoting

arXiv:2606. 15055v1 Announce Type: cross Abstract: Visual perception of urban streetscapes underpins evidence-based decisions in landscape planning, public health, and place-making.

By Xinze Zhang

benchmarks safety

arXiv AI

Jun 16

Emergent Strategic Reasoning Risks in AI: A Taxonomy-Driven Evaluation Framework

arXiv:2604. 22119v2 Announce Type: replace Abstract: As reasoning capacity and deployment scope grow in tandem, large language models (LLMs) gain the capacity to engage in behaviors that serve their own objectives, a class of risks we term Emergent Strategic Reasoning Risks (ESRRs).

By Tharindu Kumarage, Lisa Bauer, Yao Ma, Dan Rosen, Yashasvi Raghavendra Guduri, Anna Rumshisky, Kai-Wei Chang, Aram Galstyan, Rahul Gupta, Charith Peris

llms agents benchmarks safety

arXiv AI

Jun 16

VeriGraph: Towards Verifiable Data-Analytic Agents

arXiv:2606. 16603v1 Announce Type: cross Abstract: LLM-based agents have demonstrated strong capabilities in data-intensive analytical tasks, yet their outputs are rarely verifiable: a reliance on linear text trajectories makes their reasoning difficult to audit.

By Jiajie Jin, Zhao Yang, Wenle Liao, Yuyang Hu, Guanting Dong, Xiaoxi Li, Yutao Zhu, Zhicheng Dou

llms agents benchmarks

arXiv AI

Jun 16

Let LLMs Judge Each Other: Multi-Agent Peer-Reviewed Reasoning for Medical Question Answering

arXiv:2606. 15419v1 Announce Type: cross Abstract: Objective: To enhance the accuracy, interpretability, and robustness of large language models (LLMs) in medical question answering (MedQA).

By Zaifu Zhan, Shuang Zhou, Rui Zhang

llms agents nlp benchmarks safety

arXiv AI

Jun 16

RetailBench: Benchmarking long horizon reasoning and coherent decision making of LLM agents in realistic retail environments

arXiv:2606. 15862v1 Announce Type: new Abstract: Large language model (LLM) agents have made rapid progress on short-horizon, well-scoped tasks, yet their ability to sustain coherent decisions in dynamic long-horizon environments remains uncertain.

By Linghua Zhang, Jun Wang, Jingtong Wu, Zhisong Zhang

llms agents benchmarks

arXiv Machine Learning

Jun 16

FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policies

arXiv:2603. 27450v2 Announce Type: replace Abstract: Thanks to their remarkable flexibility, diffusion models and flow models have emerged as promising candidates for policy representation.

By Chenxiao Gao, Edward Chen, Tianyi Chen, Bo Dai

diffusion reinforcement-learning robotics efficiency benchmarks

arXiv AI

Jun 16

Distilling Drifting Transformers with Representation Autoencoders

arXiv:2606. 15553v1 Announce Type: cross Abstract: Representation Autoencoders (RAEs) have improved diffusion and flow models by semantically richer latent space owing to the strongly label-wise clustered DINO features in the pretrained encoders.

By Jiawei Zhang, Mengfei Xia, Gen Li, Yuantao Gu

llms diffusion efficiency benchmarks

arXiv AI

Jun 16

CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies

arXiv:2606. 16613v1 Announce Type: new Abstract: As LLM agents become capable of increasingly long-horizon tasks, evaluating their performance in economic systems is becoming increasingly important.

By Issa Sugiura, Daichi Hattori, Kazuo Araragi, Keita Ogawa, Shota Onose, Taro Makino, Teppei Usuki, Takashi Ishida

llms agents benchmarks

arXiv AI

Jun 16

CHILLGuard: Towards Fine-Grained Chinese LLM Safety Guardrail with Scalable Data Construction and Model-aware Preference Alignment

arXiv:2606. 15396v1 Announce Type: cross Abstract: Malicious content generated from large language models (LLMs) could pose severe safety risks and ethical concerns.

By Wenbo Yu, Bohua Wang, Hao Fang, Kuofeng Gao, Jingru Zeng, Xiaochen Yang, Tianyi Zhang, Xiaoxiao Ma, Jiawei Kong, Hao Wu, Bin Chen, Shu-Tao Xia, Min Zhang

llms rag benchmarks safety

arXiv Machine Learning

Jun 16

Distribution Alignment for One-Shot Federated Learning via Optimal Transport

arXiv:2606. 16655v1 Announce Type: new Abstract: One-Shot Federated Learning (OSFL) addresses extreme communication regimes in which clients interact with the server only once, amplifying the impact of heterogeneous client data distributions.

By Daniele Berardini (AI for Good), Vito Paolo Pastore (AI for Good, MaLGa-DIBRIS, University of Genoa, Genoa, Italy), Vittorio Murino (AI for Good, Department of Computer Science, University of Verona, Verona, Italy)

efficiency benchmarks safety

arXiv AI

Jun 16

Beyond Scalar Distances: Semantic Attribute Gradients from Frozen MLLMs for Visual Embeddings

arXiv:2606. 15134v1 Announce Type: cross Abstract: Vision encoders for retrieval are typically trained with class-label supervision: each training pair reduces to a scalar that uniformly pushes the embedding apart or pulls it together, as if every visual attribute either differed or matched.

By Shubhang Bhatnagar, Dheeraj Baiju, Narendra Ahuja

llms rag efficiency multimodal benchmarks

arXiv AI

Jun 16

Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution

arXiv:2604. 03472v3 Announce Type: replace-cross Abstract: Co-evolutionary self-play, where one language model generates problems and another solves them, promises autonomous curriculum learning without human supervision.

By Jacob Dineen, Aswin RRV, Zhikun Xu, Ben Zhou

llms agents reinforcement-learning benchmarks

arXiv AI

Jun 16

Entropy-Gated Latent Recursion

arXiv:2606. 16620v1 Announce Type: cross Abstract: Inference-time scaling has become the dominant lever for improving language-model reasoning, but existing methods derive rollout diversity from a single source: stochastic token-level sampling.

By Soham Bhattacharjee, Dushyant Singh Chauhan, Salem Lahlou, Martin Takac, Nils Lukas

reinforcement-learning benchmarks

arXiv AI

Jun 16

Not All Skills Help: Measuring and Repairing Agent Knowledge

arXiv:2606. 15390v1 Announce Type: cross Abstract: LLM agents can improve without weight updates by accumulating natural-language skills from experience, but current systems entrust every decision about which skills to keep and how to apply them to LLM judgment alone.

By Yixuan Wang, Yiyang Zhou, Yiming Liang, Congyu Zhang, Fuxiao Liu, Jiawei Zhou, Huaxiu Yao

llms agents benchmarks

Get the daily briefing