Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

22,530 stories · RSS feed

arXiv AI

Jun 9

Multi-Turn Evaluation of Deep Research Agents Under Process-Level Feedback

arXiv:2606. 09748v1 Announce Type: new Abstract: Existing benchmarks for deep research agents (DRAs) assess only single-shot outputs, ignoring a key question: can DRAs improve their reports when guided by feedback?

By Rishabh Sabharwal, Hongru Wang, Amos Storkey, Jeff Z. Pan

agents benchmarks

arXiv AI

Jun 9

DSFNet: Learning Dual-Domain Spectral Operators for Multi-Modality Spatio-Temporal Forecasting in Urban Transportation Systems

arXiv:2606. 07695v1 Announce Type: cross Abstract: Multi-Modality Spatio-Temporal Forecasting (MoSTF) extends traditional spatio-temporal forecasting by incorporating diverse traffic modalities.

By Yongchao Li, Yang Li, Zhuoxuan Li, Jun Chen, Chu Zhang, Jinde Cao, Leszek Rutkowski

benchmarks

arXiv AI

Jun 9

FunctionEvolve: Structure-Guided Symbolic Regression with LLMs

arXiv:2606. 07704v1 Announce Type: cross Abstract: Symbolic regression aims to uncover explicit scientific laws from data.

By Zeyu Xia, Jun Zhu, Dong Yan

llms benchmarks

arXiv AI

Jun 9

MLingualFC: Evaluating Jailbreak Vulnerabilities in Multilingual Vision-Language Models

arXiv:2606. 07706v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have demonstrated strong performance across multimodal tasks, yet their safety robustness remains an open challenge.

By Rishabh Makwana, Mamta, Deeksha Varshney, Oana Cocarascu

llms multimodal benchmarks safety

arXiv AI

Jun 9

SLMJury: Can Small Language Models Judge as Well as Large Ones?

arXiv:2606. 07810v1 Announce Type: cross Abstract: Large language models (LLMs) are widely used as judges for evaluating model outputs, but their high cost, latency, and opacity limit scalability.

By Anish Laddha, Nitesh Pradhan, Gaurav Srivastava

llms agents nlp benchmarks safety

arXiv AI

Jun 9

Beyond Point Estimates: Benchmarking Uncertainty Quantification Methods on the AION-1 Astronomical Foundation Model

arXiv:2606. 07771v1 Announce Type: cross Abstract: Foundation models for astronomical surveys offer powerful learned representations that can be transferred to downstream regression tasks such as galaxy property estimation.

By Karla Tame-Narvaez, Aleksandra \'Ciprijanovi\'c, Shubhendu Trivedi

rag benchmarks

arXiv AI

Jun 9

POISE: Position-Aware Undetectable Skill Injection on LLM Agents

arXiv:2606. 07943v1 Announce Type: cross Abstract: Agent skills provide a lightweight mechanism for extending general-purpose agents, but their open format exposes them to skill-poisoning attacks.

By Haochang Hao, Dehai Min, Zhifang Zhang, Yunbei Zhang, Miao Xu, Yingqiang Ge, Lu Cheng

llms agents multimodal benchmarks

arXiv Machine Learning

Jun 9

In-Context Learning of Temporal Point Processes with Foundation Inference Models

arXiv:2509. 24762v3 Announce Type: replace Abstract: Modeling event sequences of multiple event types with marked temporal point processes (MTPPs) provides a principled way to uncover governing dynamical rules and predict future events.

By David Berghaus, Patrick Seifner, Kostadin Cvejoski, C\'esar Ojeda, Rams\'es J. S\'anchez

llms benchmarks

arXiv Machine Learning

Jun 9

GraphER: An Efficient Graph-Based Enrichment and Reranking Method for Retrieval-Augmented Generation

arXiv:2603. 24925v2 Announce Type: replace Abstract: Retrieval-augmented generation (RAG) systems that rely on semantic search often fail to retrieve the complete set of evidence for complex queries, particularly when information is distributed across multiple sources.

By Ruizhong Miao, Yuying Wang, Rongguang Wang, Chenyang Li, Tao Sheng, Sujith Ravi, Dan Roth

rag agents benchmarks

arXiv AI

Jun 9

vla.cpp: A Unified Inference Runtime for Vision-Language-Action Models

arXiv:2606. 08094v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) policies are typically shipped as Python/PyTorch stacks that assume a workstation-class GPU, a mismatch for the hardware on which robots actually run.

By Khanh D. Nguyen, Hung T. Ho, Chinh T. Nguyen, Thanh Q. Duong, Linh D. Le, Duy M. H. Nguyen, Vien A. Ngo, An T. Le

llms diffusion robotics multimodal benchmarks

arXiv Machine Learning

Jun 9

Quantum feature-map learning with reduced resource overhead

arXiv:2510. 03389v2 Announce Type: replace-cross Abstract: Current quantum computers require algorithms that use limited resources economically.

By Jonas J\"ager, Philipp Els\"asser, Elham Torabian

efficiency benchmarks

arXiv AI

Jun 9

PIPE-Cypher: Automatic Enterprise Benchmark Generation for Text-to-Cypher Systems

arXiv:2606. 08481v1 Announce Type: cross Abstract: Enterprise property graphs vary widely in schema structure, internal terminology, domain assumptions, governance constraints, and user interaction patterns.

By Suraj Ranganath, Anish Raghavendra

llms agents benchmarks

arXiv AI

Jun 9

ActProbe: Action-Space Probe for Early Failure Detection of Generative Robot Policies

arXiv:2606. 08508v1 Announce Type: cross Abstract: Generative robot policies fail unpredictably at deployment: they hesitate at critical moments, drift off-task, or commit to unrecoverable actions.

By Bingjia Huang, Xiangyu Li, Xiang Wang, Liang Mi, Zixu Hao, Weijun Wang, Hao Wu, Kun Li, Yunxin Liu, Ting Cao

reinforcement-learning robotics fine-tuning benchmarks

arXiv AI

Jun 9

STELLAR: Spatio-Temporal Environmental Learning with Latent Alignment and Refinement for Long-Tailed Species Distribution Modeling

arXiv:2606. 08484v1 Announce Type: cross Abstract: Joint Species Distribution Modeling (JSDM) is a key enabler for biodiversity monitoring and conservation planning.

By Shufeng Kong, Tao Yu, Yuanyuan Wei, Caihua Liu, Junwen Bai, Yingheng Wang, Marc Grimson, Daniel Fink, Carla P. Gomes

diffusion benchmarks safety

arXiv AI

Jun 9

Muon Learns More Robust and Transferable Features than Adam

arXiv:2606. 09658v1 Announce Type: cross Abstract: Muon has recently emerged as a state-of-the-art optimizer for pretraining Large Language Models (LLMs) and vision classifiers.

By Tianyu Ruan, Fengzhuo Zhang, Shuche Wang, Shihua Zhang

llms fine-tuning benchmarks

arXiv AI

Jun 9

Data Synthesis and Parameter-Efficient Fine-Tuning for Low-Resource NMT: A Case Study on Q'eqchi' Mayan

arXiv:2606. 09767v1 Announce Type: cross Abstract: Neural machine translation for digitally low-resource Indigenous languages is often hindered by extreme data scarcity, prompting reliance on extractive web-scraping.

By Alexander Chulzhanov, Soeren Eberhardt, Arjun Mukherjee

llms nlp fine-tuning benchmarks

arXiv AI

Jun 9

AHA-WAM:Asynchronous Horizon-Adaptive World-Action Modeling with Observation-Guided Context Routing

arXiv:2606. 09811v1 Announce Type: cross Abstract: World-action models have emerged as a promising paradigm for robot manipulation, jointly modeling visual scene dynamics and actions to inject physical priors into policy learning.

By Jisong Cai, Long Ling, Shiwei Chu, Zhongshan Liu, Jiayue Kang, Zhixuan Liang, Wenjie Xu, Yinan Mao, Weinan Zhang, Xiaokang Yang, Ru Ying, Ran Zheng, Yao Mu

llms diffusion robotics benchmarks

arXiv AI

Jun 9

Shared Latent Structures Enable Unified Backdoor Detection and Mitigation in LLMs

arXiv:2606. 07963v1 Announce Type: new Abstract: Backdoor attacks in large language models (LLMs) are often treated as isolated trigger-response failures, motivating defenses tailored to specific triggers or behaviors.

By Omar Mahmoud, Aly M. Kassem, Thommen George Karimpanal, Buddhika Laknath Semage, Negar Rostamzadeh, Golnoosh Farnadi, Santu Rana

llms robotics fine-tuning benchmarks safety

arXiv AI

Jun 9

Sound and Complete Neurosymbolic Reasoning with LLM-Grounded Interpretations

arXiv:2507. 09751v3 Announce Type: replace Abstract: Large language models (LLMs) have demonstrated impressive capabilities in natural language understanding and generation, but exhibit problems with logical consistency in their output.

By Bradley P. Allen, Prateek Chhikara, Thomas Macaulay Ferguson, Filip Ilievski, Paul Groth

llms rag benchmarks safety

arXiv AI

Jun 9

CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoning

arXiv:2509. 25004v2 Announce Type: replace Abstract: Online reinforcement learning with verifiable rewards (RLVR) has become an effective paradigm for improving the reasoning abilities of large language models, but most methods still optimize reasoning trajectories over the static problem set, wasting rollout budget on solved or overly difficult problems.

By Shijie Zhang, Zheng Xiao, Shiyu Liu, Guohao Sun, Kevin Zhang, Xiang Guo, Rujun Guo, Shaoyu Liu, Wangxiao Zhao, Guanjun Jiang

llms reinforcement-learning benchmarks

Get the daily briefing