Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,930 stories · RSS feed

arXiv Machine Learning

Jun 30

MOSAIC: Orchestrating Collaborative Knowledge Tracing with Hierarchical Semantic Alignment

arXiv:2606. 29049v1 Announce Type: new Abstract: Knowledge Tracing (KT) is important for personalized education but traditionally suffers from two key limitations: a reliance on shallow ID-based representations that neglect semantic depth and a restriction to single-granularity mastery estimation that overlooks hierarchical knowledge dependencies.

By Xinjin Li, Mengyue Wang, Yuzhen Lin, Pengbin Feng, Ziqi Sha, Yeyang Zhou, Yu Ma

llms rag benchmarks safety

arXiv Machine Learning

Jun 30

Freeze, Prompt, and Adapt: A Framework for Source-free Unsupervised GNN Prompting

arXiv:2505. 16903v2 Announce Type: replace Abstract: Prompt tuning has become a key mechanism for adapting pre-trained Graph Neural Networks (GNNs) to new downstream tasks.

By Peyman Baghershahi, Sourav Medya

llms fine-tuning benchmarks

arXiv AI

Jun 30

Beyond 2D Matching: A Unified Single-Stage Framework for Geometry-Aware Cross-View Object Geo-Localization

arXiv:2606. 30576v1 Announce Type: cross Abstract: Cross-view object geo-localization (CVOGL) aims to locate a target object from a query view (e.

By Liyao Wang, Ruipu Wu, Haojun Xu, Lei Shi, Linjiang Huang, Si Liu

computer-vision robotics multimodal benchmarks

arXiv Machine Learning

Jun 30

Bridging the Gap Between Image Restoration and Navigational Safety in Hazy Conditions: A New Visibility Estimation Metric for Maritime Surveillance

arXiv:2606. 30049v1 Announce Type: cross Abstract: Visibility distance is critical to maritime navigational safety because it determines the effective observation range of shipborne and shore-based monitoring systems.

By Wentao Feng, Guobei Peng, Wengang Mao, Ryan Wen Liu

computer-vision benchmarks safety

arXiv AI

Jun 30

Database Context Compression for Text-to-SQL on Real-World Large Databases

arXiv:2606. 28601v1 Announce Type: cross Abstract: Recent progress in Text-to-SQL has been driven by stronger language models and prompting strategies, yet performance on real enterprise benchmarks such as Spider 2.

By Jingwen Liu, Weibin Liao, Xin Gao, Junfeng Zhao, Yasha Wang

llms benchmarks data-engineering

arXiv AI

Jun 30

Steerable Visual Representations

arXiv:2604. 02327v2 Announce Type: replace-cross Abstract: Pretrained Vision Transformers (ViTs) such as DINOv2 and MAE provide generic image features that can be applied to a variety of downstream tasks such as retrieval, classification, and segmentation.

By Jona Ruthardt, Manu Gaur, Deva Ramanan, Makarand Tapaswi, Yuki M. Asano

llms computer-vision multimodal benchmarks

arXiv AI

Jun 30

MESA: Prioritizing Vulnerable Communication Channels for Securing Multi-Agent Systems

arXiv:2606. 30602v1 Announce Type: cross Abstract: Multi-agent systems (MAS) are increasingly used to automate complex, distributed workflows.

By Kunyang Li, Kyle Domico, Jonathan Gregory, Patrick McDaniel

llms agents benchmarks

arXiv AI

Jun 30

KernelSight-LM: A Kernel-Level LLM Inference Simulator

arXiv:2606. 28565v1 Announce Type: cross Abstract: As large language models (LLMs) move into production serving, practitioners must rapidly evaluate inference performance across diverse hardware, models, and serving parameters to meet cost and latency targets.

By Xiteng Yao, Taeho Kim, Hengzhi Pei, Xinle Liu, Kyle Ulrich, Leonard Lausen, Ashish Khetan, Xiang Song, George Karypis, Martin Herbordt

llms efficiency benchmarks

arXiv AI

Jun 30

SVC-Probe: A Framework for Evaluating Perturbation Generalization in Spatial Foundation-Model Embeddings

arXiv:2606. 28465v1 Announce Type: cross Abstract: This work examines perturbation generalization in spatial foundation-model embeddings derived from fluorescence microscopy images.

By Jake Y. Chen, Huu Phong Nguyen, Fuad Al Abir, Ehsan Saghapour

rag benchmarks

arXiv AI

Jun 30

An Agentic AI Pipeline for Appliance-Level Energy Anomaly Detection and LLM-Driven Recommendations

arXiv:2606. 28467v1 Announce Type: cross Abstract: Appliance-level energy monitoring in office buildings produces noisy alerts that non-expert facility managers struggle to use.

By Dihia Falouz, Aida Douaibia, Amine Bechar, Youssef Elmir, Abbes Amira, Adel Oulefki

llms rag agents benchmarks

arXiv AI

Jun 30

Collective cooperation without individual fidelity in LLM agents

arXiv:2606. 30454v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as agents in simulations of social systems, yet it remains unclear when their behavior can be interpreted as a faithful proxy for human decision-making.

By Henrique Ferraz de Arruda, Carlos Gracia L\'azaro, Alberto Aleta, Yamir Moreno

llms agents benchmarks

arXiv AI

Jun 30

Field Order Should Not Matter: Permutation-Invariant Embedding Model Fine-Tuning for Structured Metadata Retrieval

arXiv:2606. 30473v1 Announce Type: cross Abstract: We study retrieval over catalogs of structured metadata, where each record is a small schema whose fields answer different kinds of query.

By Aivin V. Solatorio, Olivier Dupriez, Rafael Macalaba

llms rag agents fine-tuning benchmarks

arXiv Machine Learning

Jun 30

B3O: Scalable Boltzmann Batch Bayesian Optimization

arXiv:2606. 30228v1 Announce Type: new Abstract: Modern engineering workflows increasingly rely on massive parallel simulation, driving the need for scalable, large-batch Bayesian Optimization (BO).

By Maximilian Bloor, Liyuan Xu, Hrvoje Stojic, Victor Picheny

benchmarks

arXiv AI

Jun 30

Beyond IID: How General Are Tabular Foundation Models, Really?

arXiv:2606. 30410v1 Announce Type: cross Abstract: Foundation models for predictive machine learning on tabular data have recently gained significant traction in academia and industry.

By Lennart Purucker, Andrej Tschalzev, Nick Erickson, Gioia Blayer, David Holzm\"uller, Alan Arazi, Alexander Pfefferle, Mustafa Tajjar, Ga\"el Varoquaux, Frank Hutter

benchmarks

arXiv AI

Jun 30

Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions

arXiv:2507. 05257v4 Announce Type: replace-cross Abstract: Recent benchmarks for Large Language Model (LLM) agents primarily focus on evaluating reasoning, planning, and execution capabilities, while another critical component-memory, encompassing how agents memorize, update, and retrieve long-term information-is under-evaluated due to the lack of benchmarks.

By Yuanzhe Hu, Yu Wang, Julian McAuley

llms rag agents benchmarks

arXiv AI

Jun 30

ReactiveBFM: Reactive Closed-Loop Motion Planning Towards Universal Humanoid Whole-Body Control

arXiv:2606. 30362v1 Announce Type: cross Abstract: While current Behavior Foundation Models (BFMs) provide robust control priors for humanoids, they only execute pre-defined reference motions.

By Xiao Chen, Weishuai Zeng, Xiaojie Niu, Zirui Wang, Jianan Li, Huayi Wang, Furui Xu, Jiahe Chen, Weixiang Zhong, Lihe Ding, Kailin Li, Jiangmiao Pang, Tai Wang, Tianfan Xue, Jingbo Wang

rag benchmarks safety

arXiv AI

Jun 30

One Scene, Two Depths: Probing Geometric Ambiguity in Monocular Foundation Models

arXiv:2606. 29600v1 Announce Type: cross Abstract: A faithful 3D world representation should account for layered geometry, where a single camera ray may contain multiple visible and geometrically valid surfaces.

By Xiaohao Xu, Feng Xue, Xiang Li, Haowei Li, Shusheng Yang, Tianyi Zhang, Matthew Johnson-Roberson, Xiaonan Huang

llms computer-vision benchmarks

arXiv AI

Jun 30

RSGPNet: Geometric Prompting for Remote Sensing Open-Vocabulary Semantic Segmentation

arXiv:2606. 28410v1 Announce Type: cross Abstract: Open-vocabulary semantic segmentation (OVSS) enables text-guided segmentation of unseen objects, breaking fixed-class limitations to achieve open-world understanding.

By Shanwen Wang, Xin Sun, Sirui Wang, Xiao Xiang Zhu

llms computer-vision benchmarks safety

arXiv AI

Jun 30

XRAG: eXamining the Core -- Benchmarking Foundational Components in Advanced Retrieval-Augmented Generation

arXiv:2412. 15529v4 Announce Type: replace-cross Abstract: Retrieval-augmented generation (RAG) synergizes the retrieval of pertinent data with the generative capabilities of Large Language Models (LLMs), ensuring that the generated output is not only contextually relevant but also accurate and current.

By Qili Zhang, Qianren Mao, Yangyifei Luo, Yashuo Luo, Hanwen Hao, Zhilong Cao, Weifeng Jiang, Zhijun Chen, Junnan Liu, Feng Yan, Xiaolong Wang, Jinlong Zhang, Zhenting Huang, Zhixing Tan, Jie Sun, Bo Li, Jianxin Li, Philip S. Yu

llms rag benchmarks

arXiv AI

Jun 30

Vividh-ASR: A Complexity-Tiered Benchmark and Optimization Dynamics for Robust Indic Speech Recognition

arXiv:2605. 13087v2 Announce Type: replace-cross Abstract: Fine-tuning multilingual ASR models like Whisper for low-resource languages often improves read speech but degrades spontaneous audio performance.

By Kush Juvekar, Kavya Manohar, Aditya Srinivas Menon, Arghya Bhattacharya, Kumarmanas Nethil

fine-tuning multimodal benchmarks

Get the daily briefing