Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

19,246 stories · RSS feed

arXiv AI

Jun 30

ScAle: Attention Head Scaling as a Minimal Adapter for Spatial Reasoning in Vision Language Models

arXiv:2606. 29579v1 Announce Type: cross Abstract: Spatial reasoning remains a persistent challenge for many vision language models (VLMs), and improving it typically requires fine-tuning with substantial additional parameters.

By Rahul Chowdhury, Timothy A Rupprecht, Xuan Shen, Pu Zhao, Yanzhi Wang

llms fine-tuning benchmarks

arXiv AI

Jun 30

Assessing the Business Process Modeling Competences of Large Language Models

arXiv:2601. 21787v2 Announce Type: replace-cross Abstract: The creation of Business Process Model and Notation (BPMN) models is a complex and time-consuming task requiring both domain knowledge and proficiency in modeling conventions.

By Chantale Lauer, Peter Pfeiffer, Alexander Rombach, Nijat Mehdiyev

llms fine-tuning benchmarks

arXiv AI

Jun 30

Improvement of Robot's Simultaneous Localization and Mapping Using an Effective Transformation to Achieve Linear Model

arXiv:2606. 28475v1 Announce Type: cross Abstract: Nowadays mobile robots have wide engineering applications.

By Seyed Farzad Bahreinian, Maziar Palhang, Mohammad Reza Taban, Hasan Enami Eraghi

robotics benchmarks

arXiv Machine Learning

Jun 30

Efficient 3D Gaussian Splatting with Axis-Shared Rasterization and Order-independent Transmittance

arXiv:2506. 07069v2 Announce Type: replace-cross Abstract: 3D Gaussian Splatting (3DGS) has emerged as a powerful technique for novel view synthesis, combining high-quality reconstruction with efficient rendering.

By Zhican Wang, Guanghui He, Lingjun Gao, Dantong Liu, Shell Xu Hu, Chen Zhang, Zhuoran Song, Nicholas Lane, Hongxiang Fan

agents robotics benchmarks

arXiv AI

Jun 30

When Medical Safety Alignment Fails: A Benchmark for Evaluating LLMs on High-Risk Medical Queries

arXiv:2606. 28332v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used for medical and health-related questions, yet their safety in high-risk medical scenarios remains poorly understood.

By Yige Li, Jun Sun, Wei Zhao, Zhe Li, Yutao Wu, Hanxun Huang, Xiang Zheng, Xingjun Ma

llms fine-tuning benchmarks safety

arXiv AI

Jun 30

HyBIRD: Hyperbolic Bridge Retrieval and Diagnosis for Methodology Inspiration Retrieval

arXiv:2606. 28336v1 Announce Type: cross Abstract: Methodology Inspiration Retrieval (MIR) asks a system to retrieve prior papers whose methods can inspire a new research proposal.

By Yang Yang, Boyun Xu, Hao Fu, Jindong Li, Zining Zhong, Bowen Tian, Jiemin Wu, Menglin Yang, Yutao Yue

llms benchmarks

arXiv AI

Jun 30

Two-Stage Prompt Optimization for Few-Shot Relation Extraction: From Reasoning-Guided Search to Gradient-Guided Refinement

arXiv:2606. 29639v1 Announce Type: cross Abstract: Automatic prompt optimization is still underexplored for episodic few-shot relation extraction with smaller language models.

By Aunabil Chakma, Mihai Surdeanu, Eduardo Blanco

llms benchmarks

arXiv AI

Jun 30

Exploring LLM Agent Designs and Interaction Modalities for Scientific Visualization

arXiv:2604. 27996v3 Announce Type: replace Abstract: This paper examines how large language model (LLM) agents perform on scientific visualization (SciVis) tasks that require generating visualization workflows from natural-language instructions.

By Jackson Vonderhorst, Kuangshi Ai, Haichao Miao, Shusen Liu, Chaoli Wang

llms agents benchmarks

arXiv AI

Jun 30

Animation2Code: Evaluating Temporal Visual Reasoning in Video-to-Code Generation

arXiv:2606. 28593v1 Announce Type: cross Abstract: While recent vision-language models (VLMs) have achieved significant improvements on static visual-to-code tasks such as generating code for webpages, charts, or SVGs, it remains unclear whether they can recover temporal dynamics when motion is present.

By Anya Ji, Abhijith Varma Mudunuri, David M. Chan, Alane Suhr

llms fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Jun 30

CoLR-Det: Collaborative Latent Restoration for Small Object Detection in Low-Resolution Remote Sensing Images

arXiv:2601. 12507v2 Announce Type: replace-cross Abstract: Low-resolution remote sensing small object detection is limited by both missing visual details and the ambiguity of how details serve detection.

By Ruo Qi, Linhui Dai, Yusong Qin, Chaolei Yang, Yanshan Li

computer-vision benchmarks

arXiv Machine Learning

Jun 30

How Far Do On-Prem Open LLMs Get on Text-to-SQL? A Cross-Family Size x Technique Frontier on BIRD

arXiv:2606. 29733v1 Announce Type: cross Abstract: Organizations that cannot send data to a cloud API increasingly ask: how good is Text-to-SQL if the model must run on-premises on open weights, and which popular accuracy "recipes" are worth their compute?

By Vladimir Beskorovainyi

llms rag benchmarks data-engineering

arXiv AI

Jun 30

Customized Generative AI Agent for Transportation Engineering Practice: A Development and Continued Pre-training Guideline

arXiv:2606. 29014v1 Announce Type: new Abstract: Recent advancements in generative artificial intelligence (AI) and large language models (LLMs) have shown significant promise in automating complex reasoning, summarization, and question-answering tasks.

By Dianwei Chen (Terry), Yuan-Zheng Lei (Terry), Zifan Zhang (Terry), Yuchen Liu (Terry), Xianfeng (Terry), Yang

llms agents nlp fine-tuning benchmarks safety

arXiv AI

Jun 30

Inference-Time Diversity in RL-Trained Lean Theorem Provers: A Diagnostic Study

arXiv:2601. 16172v3 Announce Type: replace Abstract: RL-trained Lean theorem provers mode-collapse at inference time: on miniF2F-test with DeepSeek-Prover-V1.

By Zachary Burton

reinforcement-learning fine-tuning benchmarks

arXiv AI

Jun 30

Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Framework

arXiv:2606. 29808v1 Announce Type: cross Abstract: Chart data extraction, which reverse-engineers data tables from chart images, is essential for reproducibility, analysis, retrieval, and redesign.

By Yuchen He, Peizhi Ying, Liqi Cheng, Kuilin Peng, Yuan Tian, Dazhen Deng, Yingcai Wu

llms multimodal benchmarks

arXiv AI

Jun 30

FlatLands: Generative Floormap Completion From a Single Egocentric View

arXiv:2603. 16016v2 Announce Type: replace-cross Abstract: A single egocentric image typically captures only a small portion of the floor, yet a complete metric traversability map of the surroundings would better serve applications such as indoor navigation.

By Subhransu S. Bhattacharjee, Dylan Campbell, Rahul Shome

robotics benchmarks

arXiv AI

Jun 30

Hyper-Network Neural Functional Maps for Unsupervised Robust 3D Shape Matching

arXiv:2606. 30131v1 Announce Type: cross Abstract: Functional maps are the cornerstone of recent non-rigid 3D shape matching methods due to their efficiency and performance.

By Dongliang Cao, Florian Bernard

benchmarks safety

arXiv AI

Jun 30

Query-Aware Spreading Activation for Multi-Hop Retrieval over Knowledge Graphs

arXiv:2606. 30133v1 Announce Type: cross Abstract: Retrieval-augmented generation built on knowledge graphs (Graph RAG) outperforms flat passage retrieval on multi-hop question answering by leveraging graph structure.

By Illia Makarov, Mykola Glybovets

rag diffusion nlp benchmarks

arXiv AI

Jun 30

SUMO: Segment and Track Any Motion with Nonlinear State Space Models

arXiv:2606. 29861v1 Announce Type: cross Abstract: Visual Object Tracking (VOT) and Moving Object Segmentation (MOS) are two fundamental tasks in computer vision that involve both spatial and temporal object dynamics.

By Kexin Tian, Sixu Li, Keshu Wu, Yang Zhou, Zhengzhong Tu

computer-vision robotics benchmarks

arXiv AI

Jun 30

MotionAtlas: Detailed Region Captioning for Motion-Centric Videos

arXiv:2606. 29531v1 Announce Type: cross Abstract: We propose MotionAtlas, a system for detailed captioning of motion-centric videos, comprising (1) a dedicated human-annotated benchmark, (2) a scalable, high-quality pipeline to construct training samples, and (3) a family of powerful Video-MLLMs.

By Weisong Liu, Haochen Wang, Kuan Gao, Yuhao Wang, Yikang Zhou, Zhongwei Ren, Jacky Mai, Anna Wang, Yanwei Li, Jason Li, Zhaoxiang Zhang

benchmarks safety data-engineering

arXiv Machine Learning

Jun 30

Multi-Block Diffusion Language Models

arXiv:2606. 29215v1 Announce Type: new Abstract: Block Diffusion Language Models (BD-LMs) improve diffusion-based text generation with KV caching and flexible-length generation.

By Yijie Jin, Jiajun Xu, Yuxuan Liu, Chenkai Xu, Yi Tu, Jiajun Li, Dandan Tu, Xiaohui Yan, Kai Yu, Pengfei Liu, Zhijie Deng

llms diffusion benchmarks

Get the daily briefing