Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,090 stories · RSS feed

arXiv Machine Learning

Jun 4

BBOmix: A Tabular Benchmark for Hyperparameter Optimization of Unsupervised Biological Representation Learning

arXiv:2606. 05139v1 Announce Type: new Abstract: The rapid advancement of high-throughput sequencing has led to large, high-dimensional omics datasets.

By Luca Thale-Bombien, Jan Ewald, Ralf K\"onig, Aaron Klein

fine-tuning efficiency benchmarks

arXiv AI

Jun 4

SePO: Self-Evolving Prompt Agent for System Prompt Optimization

arXiv:2606. 04465v1 Announce Type: cross Abstract: System prompt optimization improves agent behavior without modifying the underlying model, yielding human-readable, model-agnostic instructions.

By Wangcheng Tao, Han Wu, Weng-Fai Wong

agents fine-tuning benchmarks

arXiv AI

Jun 4

SMAC-Talk: A Natural Language Extension of the StarCraft Multi-Agent Challenge for Large Language Models

arXiv:2606. 04202v1 Announce Type: new Abstract: As LLMs become more widely deployed, they are increasingly expected to work alongside other AI agents rather than operating in isolation.

By Joel Sol, Homayoun Najjaran

llms agents benchmarks

arXiv AI

Jun 4

SciIntegrity-Bench: A Benchmark for Evaluating Academic Integrity in AI Scientist Systems

arXiv:2605. 10246v2 Announce Type: replace Abstract: AI scientist systems are increasingly deployed for autonomous research, yet their academic integrity has never been systematically evaluated.

By Zonglin Yang, Xingtong Liu, Xinyan Xu

llms agents benchmarks safety

arXiv Machine Learning

Jun 4

DPDL: Towards Differential Privacy Preservation in Decentralized Stochastic Learning on Non-IID Data

arXiv:2606. 04399v1 Announce Type: new Abstract: In the paradigm of decentralized learning, a group of agents collaborate to train a global model using distributed datasets without a central server.

By Yunsheng Yuan, Xue Xiao, Lina Wang, Feng Li

agents benchmarks safety

arXiv Machine Learning

Jun 4

Optimal Transport Flow Matching by Design

arXiv:2606. 04092v1 Announce Type: cross Abstract: Flow matching models learn to transport samples from a simple prior distribution to a complex data distribution.

By Shimon Malnick, Matan Rusanovsky, Ohad Fried, Shai Avidan

diffusion benchmarks safety

arXiv AI

Jun 4

Rebalancing Reference Frame Dominance to Improve Motion in Image-to-Video Models

arXiv:2605. 19398v3 Announce Type: replace-cross Abstract: Image-to-video models often generate videos that remain overly static, compared to text-to-video models.

By Wooseok Jeon, Seungho Park, Seunghyun Shin, Sangeyl Lee, Hyeonho Jeong, Hae-Gon Jeon

diffusion benchmarks

arXiv AI

Jun 4

Scaling Novel Graph Generation via Lightweight Structure-Guided Autoregressive Models

arXiv:2606. 04287v1 Announce Type: cross Abstract: Generating realistic and diverse graphs is a key problem in machine learning, with applications in molecular discovery, circuit design, cybersecurity, and beyond.

By Alessio Barboni, Massimiliano Lupo Pasini, Bishal Lakha, Edoardo Serra

diffusion benchmarks

arXiv Machine Learning

Jun 4

RAVQ-HoloNet: Rate-Adaptive Vector-Quantized Hologram Compression

arXiv:2511. 21035v2 Announce Type: replace Abstract: Holography offers significant potential for AR/VR applications.

By Shima Rafiei, Zahra Nabizadeh Shahr-Babak, Soroush Khoubyarian, Alexandre Cooper, Shadrokh Samavi, Shahram Shirani

efficiency benchmarks

arXiv AI

Jun 4

SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimization

arXiv:2505. 11166v3 Announce Type: replace-cross Abstract: Despite advances in pretraining with extended context sizes, large language models (LLMs) still face challenges in effectively utilizing real-world long-context information, primarily due to insufficient long-context alignment caused by data quality issues, training inefficiencies, and the lack of well-designed optimization objectives.

By Huashan Sun, Shengyi Liao, Yansen Han, Yu Bai, Yang Gao, Cheng Fu, Weizhou Shen, Fanqi Wan, Ming Yan, Ji Zhang, Fei Huang

llms benchmarks safety data-engineering

arXiv AI

Jun 4

Geometry-Aware Hallucination Detection in Large Language Models

arXiv:2601. 06196v3 Announce Type: replace-cross Abstract: Large language models (LLMs) frequently generate factually incorrect or unsupported content, commonly referred to as hallucinations.

By Bodla Krishna Vamshi, Rohan Bhatnagar, Haizhao Yang

llms rag nlp fine-tuning benchmarks safety

arXiv AI

Jun 4

POLARIS: Guiding Small Models to Write Long Stories

arXiv:2606. 04095v1 Announce Type: cross Abstract: Small open-weight models struggle at long-form creative writing: their generated stories either fall far short of the requested length, or their quality significantly degrades as length increases, especially when compared to frontier models.

By Rishanth Rajendhran, Jenna Russell, Mohit Iyyer, John Frederick Wieting

llms benchmarks

arXiv AI

Jun 4

MIRAGE: Mobile Agents with Implicit Reasoning and Generative World Models

arXiv:2606. 04627v1 Announce Type: new Abstract: Mobile agents are increasingly expected to operate everyday applications from screenshots and language goals, where reliable control requires reasoning over screen affordances, multi-step navigation, and future state changes.

By Zhichao Yang, Yuanze Hu, Haojie Hao, Longkun Hao, Dongshuo Huang, Hongyu Lin, Gen Li, Lanqing Hong, Yihang Lou, Yan Bai

agents diffusion fine-tuning benchmarks

arXiv Machine Learning

Jun 4

LMM-IR: Large-Scale Netlist-Aware Multimodal Framework for Static IR-Drop Prediction

arXiv:2511. 12581v2 Announce Type: replace Abstract: Static IR drop analysis is a fundamental and critical task in the field of chip design.

By Kai Ma, Zhen Wang, Hongquan He, Qi Xu, Tinghuan Chen, Hao Geng

llms diffusion multimodal benchmarks

arXiv AI

Jun 4

Can VLMs Predict Future States? Bootstrapping World Models from Inverse Dynamics

arXiv:2506. 06006v3 Announce Type: replace-cross Abstract: Can unified vision-language models (VLMs) perform forward dynamics prediction (FDP), i.

By Yifu Qiu, Yftah Ziser, Anna Korhonen, Shay B. Cohen, Edoardo M. Ponti

llms fine-tuning multimodal benchmarks

arXiv AI

Jun 4

Caught in the Act(ivation): Toward Pre-Output and Multi-Turn Detection of Credential Exfiltration by LLM Agents

arXiv:2606. 04141v1 Announce Type: cross Abstract: LLM agents often place sensitive credentials in the same context window as untrusted retrieved content, creating a direct path for indirect prompt injection to induce credential exfiltration.

By Kargi Chauhan, Pratibha Revankar

llms agents benchmarks

arXiv AI

Jun 4

SaliMory: Orchestrating Cognitive Memory for Conversational Agents

arXiv:2606. 04120v1 Announce Type: cross Abstract: Conversational agents that serve as lifelong companions must maintain persistent memory across all interactions.

By Kai Zhang, Xinyuan Zhang, Hongda Jiang, Shiun-Zu Kuo, Hyokun Yun, Ejaz Ahmed, Shereen Oraby, Ziyun Li, Sanat Sharma, Ann Lee, Ahmed A Aly, Anuj Kumar, Raffay Hamid, Xin Luna Dong

llms agents reinforcement-learning benchmarks

arXiv AI

Jun 4

Recover-LoRA for Aggressive Quantization: Reclaiming Accuracy in 2-Bit Language Models via Low-Rank Adaptation with Knowledge Distillation on Synthetic Data

arXiv:2606. 04238v1 Announce Type: cross Abstract: Aggressive weight quantization to 2-bit precision offers substantial throughput and memory gains for large language model (LLM) inference, but typically incurs severe accuracy degradation.

By Devleena Das, Rajeev Patwari, Elliott Delaye, Ashish Sirasao

llms fine-tuning efficiency benchmarks

arXiv Machine Learning

Jun 4

GENEB: Why Genomic Models Are Hard to Compare

arXiv:2606. 04525v1 Announce Type: cross Abstract: Progress in genomic foundation models is difficult to assess due to fragmented benchmarks, incompatible evaluation protocols, and task-specific reporting.

By Daria Ledneva, Mikhail Nuridinov, Denis Kuznetsov

nlp benchmarks safety

arXiv AI

Jun 4

The Loss Is Not Enough: Sampling Conditions and Inductive Bias in Contrastive Representation Learning

arXiv:2606. 04280v1 Announce Type: cross Abstract: Contrastive learning has become a leading paradigm for self-supervised representation learning, yet the conditions under which it recovers meaningful latent geometry remain incompletely understood.

By Justinas Zaliaduonis, Patrick Putzky, Till Richter, Sergios Gatidis

diffusion benchmarks safety

Get the daily briefing