Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,880 stories · RSS feed

arXiv AI

Jun 10

PromptEmbedder: Efficient and Transferable Text Embedding via Dual-LLM Soft Prompting

arXiv:2605. 28066v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have demonstrated remarkable efficacy in text embedding, yet current adaptation methods like LoRA face significant bottlenecks in computational efficiency and cross-architecture transferability.

By Yu-Che Tsai, Kuan-Yu Chen, Yuan-Hao Chen, Yu-Han Chang, Ching-Yu Tsai, Yu-Hsiang Chuang, Shou-De Lin

llms rag fine-tuning benchmarks safety

arXiv AI

Jun 10

HIPIF: Hierarchical Planning and Information Folding for Long-Horizon LLM Agent Learning

arXiv:2606. 10507v1 Announce Type: new Abstract: While Large Language Models (LLMs) have demonstrated strong capabilities as autonomous agents across a wide range of tasks, their performance often degrades in multi-turn long-horizon agentic tasks.

By Juncheng Diao, Zhicong Lu, Peiguang Li, Yongwei Zhou, Changyuan Tian, Qingbin Li, Rongxiang Weng, Jingang Wang, Xunliang Cai

llms agents nlp reinforcement-learning benchmarks

arXiv AI

Jun 10

Harnessing the Collective Intelligence of AI Agents in the Wild for New Discoveries

arXiv:2606. 10402v1 Announce Type: cross Abstract: Scientific discovery is often a collective process: researchers share partial results, inspect failed attempts, and build on each other's ideas over long time horizons.

By Federico Bianchi, Yongchan Kwon, Aneesh Pappu, James Zou

agents benchmarks

arXiv AI

Jun 10

KG-SoftMAP: Soft Knowledge-Graph Priors for Bayesian Network Structure Learning from Sparse Discrete Data

arXiv:2606. 10358v1 Announce Type: cross Abstract: Learning Bayesian network (BN) structure from sparse discrete data is hard: when each instance records only a few variables, most variable pairs lack the joint observations needed for reliable scoring, and data-only methods recover little structure.

By Guoliang Xu, James E. Corter

llms benchmarks

arXiv AI

Jun 10

People-Centred Medical Image Analysis via Fairness-Aware Human-AI Cooperation

arXiv:2604. 26991v2 Announce Type: replace-cross Abstract: Machine learning models for medical image analysis often exhibit subgroup-dependent performance, which impacts how decisions should be allocated between automated systems and human experts under limited resources.

By Zheng Zhang, Milad Masroor, Cuong Nguyen, Tahir Hassan, Yuanhong Chen, David Rosewarne, Kevin Wells, Thanh-Toan Do, Gustavo Carneiro

agents benchmarks safety

arXiv AI

Jun 10

LIBERO-Occ: Evaluating and Improving Vision-Language-Action Models under Scene-Induced Occlusion via Viewpoint Imagination

arXiv:2606. 10862v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models achieve strong performance on standard manipulation benchmarks, but most evaluations assume that task-relevant objects are fully visible.

By Taishan Li, Jiwen Zhang, Siyuan Wang, Xuanjing Huang, Zhongyu Wei

robotics multimodal benchmarks

arXiv AI

Jun 10

TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning

arXiv:2606. 11119v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models.

By Heming Zou, Qi Wang, Yun Qu, Yuhang Jiang, Lizhou Cai, Yixiu Mao, Ru Peng, Xin Xu, Weijie Liu, Kai Yang, Saiyong Yang, Xiangyang Ji

llms agents reinforcement-learning benchmarks

arXiv AI

Jun 10

MMD Guidance: Training-Free Distribution Adaptation for Diffusion Models via Maximum Mean Discrepancy Guidance

arXiv:2601. 08379v2 Announce Type: replace-cross Abstract: Pre-trained diffusion models have emerged as powerful generative priors for both unconditional and conditional sample generation, yet their outputs often deviate from the characteristics of user-specific target data.

By Matina Mahdizadeh Sani, Nima Jamali, Mohammad Jalali, Farzan Farnia

diffusion fine-tuning benchmarks safety

arXiv Machine Learning

Jun 10

CITRAS-FM: Tiny Time Series Foundation Model for Covariate-Informed Zero-Shot Forecasting

arXiv:2606. 10798v1 Announce Type: new Abstract: Pretrained time series foundation models (TSFMs) have enabled zero-shot forecasting on unseen target series.

By Yosuke Yamaguchi, Issei Suemitsu, Yuki Kajihara, Wenpeng Wei

llms benchmarks

arXiv AI

Jun 10

What Fits (Into Few Tokens) Doesn't Overfit: Compression and Generalization in ML Research Agents

arXiv:2606. 11045v1 Announce Type: new Abstract: Reusing a held-out benchmark adaptively should, in principle, invite overfitting.

By Martin Andres Bertran, Aaron Roth, Zhiwei Steven Wu

llms agents diffusion reinforcement-learning benchmarks

arXiv AI

Jun 10

GCA Framework: A GCC Countries-Grounded Dataset and Agentic Pipeline for Climate Decision Support

arXiv:2604. 12306v3 Announce Type: replace-cross Abstract: Climate decision-making in the GCC states increasingly demands systems that can translate heterogeneous scientific and policy evidence into actionable guidance, yet general-purpose large language models (LLMs) remain weak both in region-specific climate knowledge and grounded interaction with geospatial and forecasting tools.

By Muhammad Umer Sheikh, Khawar Shehzad, Salman Khan, Fahad Shahbaz Khan, Muhammad Haris Khan

llms agents fine-tuning multimodal benchmarks

arXiv AI

Jun 10

T1-Bench: Benchmarking Multi-Scenario Agents in Real-World Domains

arXiv:2606. 11070v1 Announce Type: cross Abstract: Recent advances in reasoning and tool-calling capabilities of large language models (LLMs) have enabled increasingly capable agentic systems.

By Genta Indra Winata, Amartya Chakraborty, Yuzhen Lin, Swasthi P Rao, Shikhhar Siingh, Houhan Lu, Nadia Bathaee, Sriharsha Hatwar, Paresh Dashore, Anmol Jain, Kshitij Tayal, Xiuzhu Lin, Anirban Das, Sambit Sahu, Shi-Xiong Zhang

llms agents benchmarks

arXiv Machine Learning

Jun 10

Few-step Generative Models as Lossy Compression

arXiv:2606. 10450v1 Announce Type: cross Abstract: DiffC provides a principled way to reuse pre-trained diffusion models for lossy compression, but its encoding and decoding procedures remain slow because they require many discretized forward and reverse steps.

By Fuma Kimishima, Jinjia Zhou

diffusion benchmarks

arXiv Machine Learning

Jun 10

Optimization-based Online Conformal Prediction for Multi-step Forecasting

arXiv:2508. 13362v2 Announce Type: replace Abstract: Conformal prediction (CP) is well-suited for uncertainty quantification in time series forecasting due to its distribution-free coverage guarantees.

By Ruipu Li, Daniel Menacho, Alexander Rodr\'iguez

agents robotics benchmarks

arXiv AI

Jun 10

Flexible Flows for Biological Sequence Design

arXiv:2606. 10543v1 Announce Type: cross Abstract: Designing functional biological sequences requires navigating vast discrete spaces under strict evolutionary and biophysical constraints.

By Yogesh Verma, Dani Korpela, Harri L\"ahdesm\"aki, Vikas Garg

diffusion benchmarks

arXiv AI

Jun 10

RoboGPT-R1: Enhancing Robot Task Planning with Reinforcement Learning

arXiv:2510. 14828v3 Announce Type: replace Abstract: Improving the reasoning capabilities of embodied agents is crucial for robots to complete complex human instructions in long-view manipulation tasks successfully.

By Jinrui Liu, Bingyan Nie, Boyu Li, Yaran Chen, Yuze Wang, Shunsen He, Haoran Li

llms agents reinforcement-learning robotics fine-tuning benchmarks

arXiv AI

Jun 10

Is Fairness Truly Fair? Towards Reliable Lipschitz Fairness in Multi-Task Learning via Fixed-\texorpdfstring{$\delta$}{delta} Alignment

arXiv:2606. 10632v1 Announce Type: cross Abstract: Lipschitz-style individual fairness formalizes the idea that semantically similar examples should receive similar predictions, but its evaluation in multi-task learning (MTL) can be confounded by method-induced representation scales.

By Junbo Ding, Xin Zang, Chenchen Pan, Donghao Song, Jiaxin Zhu, Danhuai Guo

benchmarks safety

arXiv AI

Jun 10

Decoupling Thought from Speech: Knowledge-Grounded Counterfactual Reasoning for Resilient Multi-Agent Argumentation

arXiv:2606. 10475v1 Announce Type: cross Abstract: Multi-agent debate frameworks have been shown to improve large language model performance in convergent tasks, but they are currently optimized in a way that heavily favors final output accuracy rather than stability of the process.

By Jakub Mas{\l}owski, Jaros{\l}aw A. Chudziak

llms rag agents benchmarks safety

arXiv AI

Jun 10

Can Image Models Imagine Time? ImageTime: A Novel Benchmark for Probing Visual World Modeling Through Spatiotemporal Consistency

arXiv:2606. 10620v1 Announce Type: cross Abstract: Image generation models now produce high-quality static images, yet their ability to represent how a visual world changes over time remains poorly understood.

By Xinrui Wu, Lichen Huang

llms diffusion multimodal benchmarks safety

arXiv AI

Jun 10

CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks

arXiv:2606. 09833v1 Announce Type: cross Abstract: AI agents are reshaping the workspace, leading to drastic change of how humans work.

By Yijia Shao, Zora Zhiruo Wang, Neel Ahuja, Yicheng Wang, Bowen Liu, Diyi Yang

llms agents benchmarks

Get the daily briefing