Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

19,504 stories · RSS feed

arXiv Machine Learning

Jun 30

Transolver-3: Scaling Up Transformer Solvers to Industrial-Scale Geometries

arXiv:2602. 04940v2 Announce Type: replace Abstract: Deep learning has emerged as a transformative tool for the neural surrogate modeling of partial differential equations (PDEs), known as neural PDE solvers.

By Hang Zhou, Haixu Wu, Haonan Shangguan, Yuezhou Ma, Huikun Weng, Jianmin Wang, Mingsheng Long

llms benchmarks

arXiv AI

Jun 30

Evidence-Driven LLM Agent for C-to-Synthesizable-C Conversion and Verification

arXiv:2606. 28409v1 Announce Type: cross Abstract: Software-compilable C programs routinely fail to complete the four-stage pipeline of a high-level synthesis (HLS) toolchain -- compilation, C simulation (CSim), synthesis, and C/RTL co-simulation (CoSim) -- because HLS accepts only a synthesizable subset of C (HLS-C).

By Zhe Zhao, Hongbing Lang, Zhihan Xiao, Luke Ztz Hu, John Imoleayo Adebisi, Songping Mai

llms rag agents benchmarks

arXiv Machine Learning

Jun 30

An Integrated Two-Stage Deep-Learning Tool for Rapid Post-Hurricane Damage Identification and Repair Scheduling

arXiv:2606. 29117v1 Announce Type: cross Abstract: Post-hurricane damage assessment and repair scheduling can require computationally intensive simulation and optimization.

By Hooman Torkaman, Ellis Oti Boateng, Jignesh Solanki, Anurag Srivastava

llms benchmarks

arXiv AI

Jun 30

Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Framework

arXiv:2606. 29808v1 Announce Type: cross Abstract: Chart data extraction, which reverse-engineers data tables from chart images, is essential for reproducibility, analysis, retrieval, and redesign.

By Yuchen He, Peizhi Ying, Liqi Cheng, Kuilin Peng, Yuan Tian, Dazhen Deng, Yingcai Wu

llms multimodal benchmarks

arXiv Machine Learning

Jun 30

Benchmarking Geospatial Foundation Models for Agriculture Applications

arXiv:2606. 29664v1 Announce Type: cross Abstract: Geospatial foundation models pretrained on satellite imagery promise broad generalization across remote sensing tasks and regions, but their geographic transferability has not been systematically tested, especially in agriculture applications.

By Zhuocheng Shang, Sanmay Das, Ahmed Eldawy

computer-vision benchmarks

arXiv AI

Jun 30

Fast and Accurate Outlier-Aware LiDAR Super-Resolution for SLAM Applications

arXiv:2606. 28607v1 Announce Type: cross Abstract: This work tackles the challenge of enhancing low-resolution LiDAR sensors for SLAM applications through a novel Deep Unrolling-based Super-Resolution (SR) model.

By Christos Anagnostopoulos, Alexandros Gkillas, Nikos Piperigkos, Aris S. Lalos

computer-vision benchmarks

arXiv AI

Jun 30

Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions

arXiv:2507. 05257v4 Announce Type: replace-cross Abstract: Recent benchmarks for Large Language Model (LLM) agents primarily focus on evaluating reasoning, planning, and execution capabilities, while another critical component-memory, encompassing how agents memorize, update, and retrieve long-term information-is under-evaluated due to the lack of benchmarks.

By Yuanzhe Hu, Yu Wang, Julian McAuley

llms rag agents benchmarks

arXiv AI

Jun 30

FedLAS: Feature-Modulated Bidirectional Label Smoothing for Neural Network Calibration

arXiv:2606. 28654v1 Announce Type: cross Abstract: Deep Neural Network (DNN) classifiers suffer from poor calibration when their softmax outputs (predictive confidence) deviate from the empirical likelihoods.

By Thiru Thillai Nadarasar Bahavan, Sachith Seneviratne, Saman Halgamuge

benchmarks

arXiv Machine Learning

Jun 30

CoLR-Det: Collaborative Latent Restoration for Small Object Detection in Low-Resolution Remote Sensing Images

arXiv:2601. 12507v2 Announce Type: replace-cross Abstract: Low-resolution remote sensing small object detection is limited by both missing visual details and the ambiguity of how details serve detection.

By Ruo Qi, Linhui Dai, Yusong Qin, Chaolei Yang, Yanshan Li

computer-vision benchmarks

arXiv AI

Jun 30

SEATauBench: Adapting Tool-Agent-User Evaluation Into Low-Resource Southeast Asian Languages

arXiv:2606. 28715v1 Announce Type: cross Abstract: While AI development and evaluation for Southeast Asia (SEA) has grown rapidly, agent capabilities in regional languages are still poorly understood despite its importance to sovereign AI.

By My Chiffon Nguyen, Aulia Adila, Saksorn Ruangtanusak, Kittiphat Leesombatwathana, Vissuta Gunawan Lim, Patomporn Payoungkhamdee, Samuel Cahyawijaya

agents fine-tuning benchmarks

arXiv AI

Jun 30

GeNeRT: A Physics-Informed Approach to Intelligent Wireless Channel Modeling via Generalizable Neural Ray Tracing

arXiv:2506. 18295v2 Announce Type: replace-cross Abstract: Neural ray tracing (RT) has emerged as a promising paradigm for channel modeling by integrating physical propagation principles with neural networks.

By Kejia Bian, Meixia Tao, Shu Sun, Tongjia Zhang, Jun Yu

fine-tuning benchmarks

arXiv AI

Jun 30

Inference-Time Diversity in RL-Trained Lean Theorem Provers: A Diagnostic Study

arXiv:2601. 16172v3 Announce Type: replace Abstract: RL-trained Lean theorem provers mode-collapse at inference time: on miniF2F-test with DeepSeek-Prover-V1.

By Zachary Burton

reinforcement-learning fine-tuning benchmarks

arXiv AI

Jun 30

Sample-Efficient Learning of Probabilistic Causes for Reachability in Markov Decision Processes with Probabilistic Guarantees

arXiv:2606. 29681v1 Announce Type: new Abstract: Probabilistic model checking for Markov decision processes (MDPs) provides quantitative guarantees, but often offers limited insight into why undesired outcomes occur.

By Ryohei Oura, Georgios Fainekos, Hideki Okamoto, Bardh Hoxha

reinforcement-learning benchmarks

arXiv AI

Jun 30

Toward Secure and Reliable PDDL Formalization of Large Language Models with Planner-in-the-Loop Feedback

arXiv:2606. 29700v1 Announce Type: new Abstract: Planning often requires symbolic specifications that are both executable and verifiable.

By Jiamei Jiang, Jiajing Zhang, Feifei Mo, Linjing Li, Daniel Zeng

llms agents fine-tuning benchmarks safety

arXiv AI

Jun 30

FlatLands: Generative Floormap Completion From a Single Egocentric View

arXiv:2603. 16016v2 Announce Type: replace-cross Abstract: A single egocentric image typically captures only a small portion of the floor, yet a complete metric traversability map of the surroundings would better serve applications such as indoor navigation.

By Subhransu S. Bhattacharjee, Dylan Campbell, Rahul Shome

robotics benchmarks

arXiv AI

Jun 30

The CRISTAL Method: Neurosymbolic analysis from AI-synthesized world models

arXiv:2606. 29799v1 Announce Type: new Abstract: This project introduces the CRISTAL Method (Coherent Reliable Intentional Synthesis of Truthful Analysis Logic), a neurosymbolic framework for automating complex analysis workflows, with fundamental investment analysis as a primary use case.

By Rafael Kaufmann, Felix Neub\"urger, Michael Walters, Thomas Kopinski, Dimitrije Markovi\'c

llms agents benchmarks

arXiv AI

Jun 30

CLQT: A Closed-Loop, Cost-Aware, Strategy-Consistent Benchmark for Diagnostic Evaluation of LLM Portfolio-Management Agents

arXiv:2606. 29771v1 Announce Type: new Abstract: LLM agents are increasingly cast as autonomous portfolio managers, and benchmarks have moved from financial question-answering to sequential trading.

By Bo Qu, Mingguang Chen

llms agents benchmarks safety

arXiv Machine Learning

Jun 30

Golden Hour Divide: Trauma Care Accessibility and Resource Vulnerability in Sri Lanka

arXiv:2606. 29889v1 Announce Type: new Abstract: Timely intensive care dictates survival, yet emergency infrastructure remains unevenly distributed across Sri Lanka.

By Sonath Kirindage, Vihanga Nimsara, Sakindu Rajapaksa, Kavyanga Hathurusinghe, Lahiru Dilshan, Subavarshana Arumugam, Nathali Athukorala, Sandareka Wickramanayake, Nisansa de Silva

benchmarks

arXiv AI

Jun 30

Robust Multi-Agent LLMs under Byzantine Faults

arXiv:2605. 09076v2 Announce Type: replace-cross Abstract: Large language model (LLM) agents increasingly collaborate over peer-to-peer networks to improve their reliability.

By Haejoon Lee, Vincent-Daniel Yun, Dimitra Panagou, Sai Praneeth Karimireddy

llms agents robotics benchmarks safety

arXiv AI

Jun 30

Be Faithful When Response: Returning Fluent and Grounded Answers for Vision-Language Models Reinforcement Learning

arXiv:2606. 29984v1 Announce Type: new Abstract: Reinforcement Learning (RL) is an important paradigm for improving the reasoning capabilities of Vision-Language Models (VLMs).

By Peng, Lee, Yin Zhang, Yanglin Zhang, Haonan Wu, Zishan Liu, Ruoxi Zang, Xin Zhu, Jiayin Zheng, Jian Yao, Zefeng Ji, Fei Ma

llms reinforcement-learning multimodal benchmarks

Get the daily briefing