Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,477 stories · RSS feed

arXiv AI

Jun 30

Evidence-Driven LLM Agent for C-to-Synthesizable-C Conversion and Verification

arXiv:2606. 28409v1 Announce Type: cross Abstract: Software-compilable C programs routinely fail to complete the four-stage pipeline of a high-level synthesis (HLS) toolchain -- compilation, C simulation (CSim), synthesis, and C/RTL co-simulation (CoSim) -- because HLS accepts only a synthesizable subset of C (HLS-C).

By Zhe Zhao, Hongbing Lang, Zhihan Xiao, Luke Ztz Hu, John Imoleayo Adebisi, Songping Mai

llms rag agents benchmarks

arXiv Machine Learning

Jun 30

An Integrated Two-Stage Deep-Learning Tool for Rapid Post-Hurricane Damage Identification and Repair Scheduling

arXiv:2606. 29117v1 Announce Type: cross Abstract: Post-hurricane damage assessment and repair scheduling can require computationally intensive simulation and optimization.

By Hooman Torkaman, Ellis Oti Boateng, Jignesh Solanki, Anurag Srivastava

llms benchmarks

arXiv AI

Jun 30

Fast and Accurate Outlier-Aware LiDAR Super-Resolution for SLAM Applications

arXiv:2606. 28607v1 Announce Type: cross Abstract: This work tackles the challenge of enhancing low-resolution LiDAR sensors for SLAM applications through a novel Deep Unrolling-based Super-Resolution (SR) model.

By Christos Anagnostopoulos, Alexandros Gkillas, Nikos Piperigkos, Aris S. Lalos

computer-vision benchmarks

arXiv AI

Jun 30

FedLAS: Feature-Modulated Bidirectional Label Smoothing for Neural Network Calibration

arXiv:2606. 28654v1 Announce Type: cross Abstract: Deep Neural Network (DNN) classifiers suffer from poor calibration when their softmax outputs (predictive confidence) deviate from the empirical likelihoods.

By Thiru Thillai Nadarasar Bahavan, Sachith Seneviratne, Saman Halgamuge

benchmarks

arXiv Machine Learning

Jun 30

ITSPACE: Monotone Gaussian Optimal Transport Updates

arXiv:2606. 30523v1 Announce Type: new Abstract: Covariance matrices serve as compact descriptors of feature distributions in many machine-learning pipelines, including domain adaptation and Gaussian embeddings.

By Woojoo Na, Jennifer Dy

rag fine-tuning benchmarks safety

arXiv Machine Learning

Jun 30

Zero-Label Driving Scenario Complexity Detection via Joint Embedding Predictive Architecture

arXiv:2606. 28383v1 Announce Type: cross Abstract: Identifying complex and safety-critical driving scenarios in large unlabelled datasets is an important but expensive problem.

By Santosh Jaiswal

rag agents benchmarks safety

arXiv AI

Jun 30

Dynamic Parsing and Updating Natural Language Specification using VLMs for Robust Vision-Language Tracking

arXiv:2606. 29357v1 Announce Type: cross Abstract: Vision-language tracking guided by natural language specifications leverages high-level semantic cues of target objects to substantially boost tracking accuracy and robustness.

By Xiao Wang, Liye Jin, Dan Xu, Yuehang Li, Lan Chen, Yaowei Wang, Yonghong Tian, Jin Tang

llms multimodal benchmarks safety

arXiv AI

Jun 30

Bridging VideoQA and Video-Guided Agentic Tasks via Generalized Keyframe Extraction

arXiv:2606. 29445v1 Announce Type: cross Abstract: Video understanding is a fundamental capability for multimodal intelligence, and recent Multimodal Large Language Models (MLLMs) have achieved remarkable performance on Video Question Answering (VideoQA) benchmarks.

By Sunqi Fan, Qingle Liu, Runqi Yin, Meng-Hao Guo, Shuojin Yang

llms agents nlp multimodal benchmarks

arXiv AI

Jun 30

Closing the Activation-Cone Blind Spot: Response-Time Probing and Unified Defense

arXiv:2606. 29441v1 Announce Type: cross Abstract: Inference-time safety methods for large language models have proliferated, yet no systematic comparison exists.

By Subhadip Mitra

llms benchmarks safety

arXiv AI

Jun 30

Sample-Efficient Learning of Probabilistic Causes for Reachability in Markov Decision Processes with Probabilistic Guarantees

arXiv:2606. 29681v1 Announce Type: new Abstract: Probabilistic model checking for Markov decision processes (MDPs) provides quantitative guarantees, but often offers limited insight into why undesired outcomes occur.

By Ryohei Oura, Georgios Fainekos, Hideki Okamoto, Bardh Hoxha

reinforcement-learning benchmarks

arXiv AI

Jun 30

Robust Multi-Agent LLMs under Byzantine Faults

arXiv:2605. 09076v2 Announce Type: replace-cross Abstract: Large language model (LLM) agents increasingly collaborate over peer-to-peer networks to improve their reliability.

By Haejoon Lee, Vincent-Daniel Yun, Dimitra Panagou, Sai Praneeth Karimireddy

llms agents robotics benchmarks safety

arXiv AI

Jun 30

Towards Continual Motion-Language Agents: LoRA Variants for Incremental Motion Understanding and Generation

arXiv:2606. 30266v1 Announce Type: cross Abstract: Motion-language agents must possess the bidirectional capability to both understand human movement (motion-to-text, M2T) and generate it from natural language (text-to-motion, T2M).

By Bertram Taetz, Hugo Albuquerque Cosme da Silva, Gabriele Bleser-Taetz

llms agents fine-tuning benchmarks

arXiv AI

Jun 30

Toward Secure and Reliable PDDL Formalization of Large Language Models with Planner-in-the-Loop Feedback

arXiv:2606. 29700v1 Announce Type: new Abstract: Planning often requires symbolic specifications that are both executable and verifiable.

By Jiamei Jiang, Jiajing Zhang, Feifei Mo, Linjing Li, Daniel Zeng

llms agents fine-tuning benchmarks safety

arXiv AI

Jun 30

The CRISTAL Method: Neurosymbolic analysis from AI-synthesized world models

arXiv:2606. 29799v1 Announce Type: new Abstract: This project introduces the CRISTAL Method (Coherent Reliable Intentional Synthesis of Truthful Analysis Logic), a neurosymbolic framework for automating complex analysis workflows, with fundamental investment analysis as a primary use case.

By Rafael Kaufmann, Felix Neub\"urger, Michael Walters, Thomas Kopinski, Dimitrije Markovi\'c

llms agents benchmarks

arXiv AI

Jun 30

Little Brains, Big Feats: Exploring Compact Language Models

arXiv:2606. 30062v1 Announce Type: cross Abstract: While large language models have been dominating the research landscape recently, small language models remain highly relevant across various domains; yet, they receive far less attention.

By Dari Baturova, Elena Bruches, Ivan Chernov, Roman Derunets, Arsenii Fomin, Andrey Kostin

llms rag benchmarks

arXiv AI

Jun 30

CLQT: A Closed-Loop, Cost-Aware, Strategy-Consistent Benchmark for Diagnostic Evaluation of LLM Portfolio-Management Agents

arXiv:2606. 29771v1 Announce Type: new Abstract: LLM agents are increasingly cast as autonomous portfolio managers, and benchmarks have moved from financial question-answering to sequential trading.

By Bo Qu, Mingguang Chen

llms agents benchmarks safety

arXiv AI

Jun 30

RoboGaze: Evaluating Robot World Models via Structured Vision-Language Analysis

arXiv:2606. 28385v1 Announce Type: cross Abstract: Recent advances in robot world models enable synthetic video generation for embodied prediction and planning.

By Minh-Loi Nguyen, Nghiem Tuong Diep, Hung Khang Nguyen, Minh Le, Doanh Le Thien, Hoang H. Tran, Dung D. Le, Vu N. Duong, Daniel Sonntag, An Thai Le, Duy Minh Ho Nguyen, Vien Anh Ngo, Tran Van Nhiem

llms agents diffusion robotics multimodal benchmarks safety

arXiv AI

Jun 30

DyGnROLE: Asymmetric Pretraining for Edge Classification on Dynamic Graphs

arXiv:2602. 23135v2 Announce Type: replace-cross Abstract: Edge classification on directed dynamic graphs requires modeling interactions between source and destination nodes exhibiting asymmetrical behavioral patterns and temporal dynamics.

By Tyler Bonnet, Marek Rei

llms rag benchmarks safety

arXiv AI

Jun 30

On the Faithfulness of Post-Hoc Concept Bottleneck Models

arXiv:2606. 30498v1 Announce Type: cross Abstract: Human decision-making interprets the world through high-level concepts, such as recognizing a bird by its belly color.

By Laines Schmalwasser, Jan Blunk, Niklas Penzel, Julia Niebling, Joachim Denzler

llms multimodal benchmarks

arXiv AI

Jun 30

CAPTCHA Solving for Native GUI Agents: Automated Reasoning-Action Data Generation and Self-Corrective Training

arXiv:2603. 23559v2 Announce Type: replace-cross Abstract: GUI agents are rapidly shifting from multi-module pipelines to end-to-end, native vision-language models (VLMs) that perceive raw screenshots and directly interact with digital devices.

By Yuxi Chen, Haoyu Zhai, Chenkai Wang, Rui Yang, Lingming Zhang, Gang Wang, Huan Zhang

llms agents multimodal benchmarks

Get the daily briefing