Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,561 stories · RSS feed

arXiv AI

Jun 16

Haiku to Opus in Just 10 bits: LLMs Unlock Large Compression Gains

arXiv:2604. 02343v2 Announce Type: replace-cross Abstract: We study the compression of LLM-generated text across lossless and lossy regimes, characterizing a compression-compute frontier where more compression is possible at the cost of more compute.

By Roy Rinberg, Annabelle Michael Carrell, Simon Henniger, Nicholas Carlini, Keri Warr

llms fine-tuning benchmarks

arXiv AI

Jun 16

Think-at-Hard: Selective Latent Iterations to Improve Reasoning Language Models

arXiv:2511. 08577v3 Announce Type: replace-cross Abstract: Improving the reasoning abilities of Large Language Models (LLMs), especially under parameter constraints, is crucial for real-world applications.

By Tianyu Fu, Yichen You, Zekai Chen, Guohao Dai, Huazhong Yang, Yu Wang

llms fine-tuning benchmarks

arXiv Machine Learning

Jun 16

FlowRL: A Taxonomy and Modular Framework for Reinforcement Learning with Diffusion Policies

arXiv:2603. 27450v2 Announce Type: replace Abstract: Thanks to their remarkable flexibility, diffusion models and flow models have emerged as promising candidates for policy representation.

By Chenxiao Gao, Edward Chen, Tianyi Chen, Bo Dai

diffusion reinforcement-learning robotics efficiency benchmarks

arXiv AI

Jun 16

Bridging the Gap: Enabling Natural Language Queries for NoSQL Databases through Text-to-NoSQL Translation

arXiv:2502. 11201v3 Announce Type: replace-cross Abstract: NoSQL databases are core data infrastructure, yet natural-language access to them remains underdeveloped: correct query generation must recover how a non-relational data model represents entities, nested paths, arrays, missing fields, and dynamic keys.

By Jinwei Lu, Jiawei Lu, Chen Zhang, Zhiqian Qin, Haodi Zhang, Yuanfeng Song, Raymond Chi-Wing Wong

llms benchmarks data-engineering

arXiv Machine Learning

Jun 16

CoIRL-AD: Collaborative-Competitive Imitation-Reinforcement Learning in Latent World Models for Autonomous Driving

arXiv:2510. 12560v2 Announce Type: replace-cross Abstract: End-to-end autonomous driving models trained with imitation learning (IL) often generalize poorly, particularly in long-tail scenarios where expert demonstrations are sparse.

By Xiaoji Zheng, Ziyuan Yang, Yanhao Chen, Yuhang Peng, Yuanrong Tang, Gengyuan Liu, Bokui Chen, Jiangtao Gong

agents reinforcement-learning robotics benchmarks

arXiv AI

Jun 16

A Survey on 3D Skeleton Based Person Re-Identification: Taxonomy, Advances, Challenges, and Interdisciplinary Prospects

arXiv:2401. 15296v4 Announce Type: replace-cross Abstract: Person re-identification via 3D skeletons is an important emerging research area that attracts increasing attention within the pattern recognition community.

By Haocong Rao, Chunyan Miao

benchmarks

arXiv AI

Jun 16

Surprise-Guided MergeSort: Budget-Efficient Human-in-the-Loop Ranking via Adaptive Comparison Scheduling

arXiv:2606. 15623v1 Announce Type: cross Abstract: Pairwise comparison is the gold standard for subjective ranking tasks; however, exhaustive annotation requires a massive number of human comparisons ($O(n^2)$).

By Yujin Park, Haejun Chung, Ikbeom Jang

llms multimodal benchmarks safety

arXiv AI

Jun 16

LiteOdyssey: A Lightweight Reasoning AI Agent for Interpretable Rare-Disease Diagnosis

arXiv:2606. 16149v1 Announce Type: new Abstract: Most medical AI systems improve by scaling additional machinery: more fine-tuning data, more agents, and/or larger retrieval databases.

By Minh-Ha Nguyen, Erica Gray, Chih-Ting Yang, Rizwan Hamid, Lingyao Li, Siyuan Ma, Thomas A. Cassini, Cathy Shyr

llms agents fine-tuning benchmarks

arXiv Machine Learning

Jun 16

Your Privacy My Cloak: Backdoor Attacks on Differentially Private Federated Learning

arXiv:2606. 17035v1 Announce Type: new Abstract: Prior research suggests that differential privacy (DP) inherently enhances the robustness of federated learning (FL) against backdoor attacks.

By Xiaolin Li, Ning Wang, Ninghui Li, Wenhai Sun

benchmarks safety

arXiv Machine Learning

Jun 16

DAL: A Practical Prior-Free Black-Box Framework for Piecewise Stationary Bandits

arXiv:2501. 19401v5 Announce Type: replace Abstract: We introduce a practical, black-box framework termed Detection Augmented Learning (DAL) for the problem of piecewise stationary bandits without knowledge of the underlying non-stationarity.

By Argyrios Gerogiannis, Yu-Han Huang, Subhonmesh Bose, Venugopal V. Veeravalli

reinforcement-learning benchmarks

arXiv AI

Jun 16

CLoVE: Personalized Federated Learning through Clustering of Loss Vector Embeddings

arXiv:2506. 22427v2 Announce Type: replace-cross Abstract: We propose CLoVE (Clustering of Loss Vector Embeddings), a novel algorithm for Clustered Federated Learning (CFL).

By Randeep Bhatia, Nikos Papadis, Murali Kodialam, TV Lakshman, Sayak Chakrabarty

rag benchmarks

arXiv AI

Jun 16

NEXUS: Neural Energy Fields for Physically Consistent Contact-Rich 3D Object Dynamics

arXiv:2606. 15015v1 Announce Type: cross Abstract: Physics-grounded video generation requires controllable 3D object dynamics that remain physically consistent under contact, deformation, and external forcing.

By Qizhen Ying, Guangming Wang, Yangchen Pan, Victor Adrian Prisacariu, Yixiong Jing

diffusion benchmarks

arXiv AI

Jun 16

RoboPIN: Grounded Embodied Reasoning via Pinned Chain-of-Thought

arXiv:2606. 15753v1 Announce Type: new Abstract: Embodied reasoning requires models to perceive task-relevant objects and spaces in physical environments and maintain consistent visual grounding throughout multi-step reasoning.

By Yaoting Huang, Yifu Yuan, Linqi Han, Chengwen Li, Shuoheng Zhang, Xianze Yao, Hongyao Tang, Yan Zheng, Jianye Hao

llms robotics multimodal benchmarks safety

arXiv Machine Learning

Jun 16

Unlocking Latent Dimensions: Exploring Representations of Large-Scale X-ray Scattering Data using Variational Autoencoders

arXiv:2606. 14999v1 Announce Type: new Abstract: Scientific user facilities generate X-ray scattering data faster than traditional workflows can process them.

By Monika Choudhary, Xiaoya Chong, Runbo Jiang, Wiebke Koepp, Petrus H. Zwart, Damon English, Gregory M. Su, Eric Schaible, Chenhui Zhu, Mostafa Nassr, Noah P. Wamble, Kelvin Kam-Yun Li, Jonathan M. Chan, Jose Carlos Diaz, Cameron McKay, Lynn Katz, Benny Freeman, Guillaume Freychet, Yevgen Matviychuk, Eliot Gann, Daniel B. Allan, Benedikt Sochor, Frank Schluenzen, Stephan V. Roth, Ethan Crumlin, Dylan McReynolds, Tanny Chavez, Alexander Hexemer

diffusion benchmarks

arXiv AI

Jun 16

Is Your Trajectory Displacement Safe in Long-tail?

arXiv:2606. 16313v1 Announce Type: cross Abstract: Long-tail scenarios remain a major bottleneck for autonomous driving evaluation, even as datasets grow by orders of magnitude.

By Qiao Sun, Weicheng Zheng, Yixin Huang, Hang Zhao

agents robotics benchmarks safety

arXiv AI

Jun 16

DynaDebate: Breaking Homogeneity in Multi-Agent Debate with Dynamic Path Generation

arXiv:2601. 05746v2 Announce Type: replace Abstract: Recent years have witnessed the rapid development of Large Language Model-based Multi-Agent Systems (MAS), which excel at collaborative decision-making and complex problem-solving.

By Zhenghao Li, Zhi Zheng, Wei Chen, Jielun Zhao, Yong Chen, Tong Xu, Enhong Chen

llms agents benchmarks

arXiv AI

Jun 16

Critically Engaged Pragmatism: Scientific Norm and Social, Pragmatist Epistemology for AI Science Evaluation Tools

arXiv:2601. 09753v2 Announce Type: replace-cross Abstract: AI science evaluation tools aim to assess research credibility.

By Carole J. Lee

benchmarks safety

arXiv AI

Jun 16

When Do LLMs Reason? A Dynamical Systems View via Entropy Phase Transitions

arXiv:2605. 22873v2 Announce Type: replace-cross Abstract: Chain-of-thought (CoT) reasoning has become the default strategy for enhancing LLM capabilities, yet its application raises a fundamental question: when is explicit reasoning actually beneficial?

By Wei Xia, Haoqing Wang, Zhi-Hong Deng, Yehui Tang

llms benchmarks

arXiv AI

Jun 16

An affordable hardware-aware neural architecture search for deploying convolutional neural networks on ultra-low-power computing platforms

arXiv:2606. 16290v1 Announce Type: cross Abstract: Hardware-aware neural architecture search (HW-NAS) allows the integration of Convolutional Neural Networks (CNNs) in microcontrollers devices by automatically designing neural architectures that can fit prearranged hardware constraints.

By Andrea Mattia Garavagno, Edoardo Ragusa, Antonio Frisoli, Paolo Gastaldo

computer-vision benchmarks

arXiv AI

Jun 16

MMLongEmbed: Benchmarking Multimodal Embedding Models in Long-Context Scenarios

arXiv:2606. 14747v1 Announce Type: cross Abstract: Recent advancements have significantly expanded the theoretical context windows of Multimodal Embedding Models (MEMs).

By Haitian Wang, Ruoxi Sun, Quantong Qiu, Juntao Li, Junhui Li, Hua Chen, Jinxiong Chang, Min Zhang

rag multimodal benchmarks

Get the daily briefing