Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,906 stories · RSS feed

arXiv Machine Learning

Jul 8

Improving TabPFN's Synthetic Data Generation by Integrating Causal Structure

arXiv:2603. 10254v2 Announce Type: replace Abstract: Synthetic tabular data generation addresses data scarcity and privacy constraints in a variety of domains.

By Davide Tugnoli, Andrea De Lorenzo, Marco Virgolin, Giovanni Cin\`a

benchmarks safety data-engineering

arXiv AI

Jul 8

MLLM-LLaVA-FL: Multimodal Large Language Model Assisted Federated Learning

arXiv:2409. 06067v3 Announce Type: replace Abstract: Previous studies on federated learning (FL) often encounter performance degradation due to data heterogeneity among different clients.

By Jianyi Zhang, Hao Frank Yang, Ang Li, Xin Guo, Pu Wang, Haiming Wang, Yiran Chen, Hai Li

llms nlp multimodal benchmarks safety

arXiv AI

Jul 8

Do Quantum Transformers Help? A Systematic VQC Architecture Comparison on Tabular Benchmarks

arXiv:2604. 23931v2 Announce Type: replace-cross Abstract: Variational quantum circuits (VQCs) are a leading approach to quantum machine learning on near-term devices, yet it remains unclear which circuit architecture yields the best accuracy-parameter trade-off on classical tabular data.

By Chi-Sheng Chen, En-Jui Kuo

llms benchmarks

arXiv AI

Jul 8

Prompt Robustness Is Task-Dependent: Comparing Objective and Belief-Style Questions in LLM Evaluation

arXiv:2607. 05554v1 Announce Type: cross Abstract: Survey-style evaluations of large language models often treat a prompted response as a measure of a model's values or beliefs.

By Sadia Kamal, Arefa Patwary, Anthony Marchiafava, Atriya Sen, Sagnik Ray Choudhury

llms benchmarks

arXiv Machine Learning

Jul 8

PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint

arXiv:2607. 01646v2 Announce Type: replace Abstract: State-of-the-art large language model (LLM) training takes tens of thousands of graphics processing units (GPUs) for months and encounters failures across the software and hardware stack.

By Haotian Xie, Junlin Chen, Mingkai Zheng, Lishan Yang, Zhao Zhang

llms benchmarks

arXiv AI

Jul 8

ArtisanCAD: An Industrial-Level CAD Agent with Expert-Grounded Knowledge Distillation

arXiv:2607. 05750v1 Announce Type: new Abstract: Computer-aided design (CAD) for industrial components requires long-horizon procedural modeling, robust feature dependencies, editable parametric geometry, and production-grade B-Rep execution.

By Yunhan Xu, Qifeng Wu, Xunjin Li, Yuanwei Bin, Qingsong Yao, Jianghang Gu, Guan Wang, Weihao Lv, Huiyu Yang, Wenfa Luo, Jiao Xiang, Yuntian Chen, Shiyi Chen

agents efficiency benchmarks

arXiv AI

Jul 8

Evaluating calibrated refusal and safe usefulness in dual-use biology settings

arXiv:2607. 05462v1 Announce Type: cross Abstract: As AI agents are incorporated into life science workflows, the capabilities that speed discovery might also enable misuse.

By Edwin H. Wintermute, Harmon Bhasin, Christina M. Agapakis, Dianzhuo Wang, Evan Seeyave, Arjun Banerjee, Daniel Fulop, Matthew C. Watson, Adam J. Meyer, Sandrine Boissel, Jens H. Kuhn, Rishi Jain, Noah D. Taylor, Helena Shomar, Patrick M. Boyle, Kenny Workman

agents benchmarks

arXiv AI

Jul 8

Detoxify: A framework for abusive text transformation using LLMs

arXiv:2507. 10177v2 Announce Type: replace-cross Abstract: Although Large Language Models (LLMs) have demonstrated significant advancements in natural language processing tasks, their effectiveness in the classification and transformation of abusive text into non-abusive versions remains an area for exploration.

By Rohitash Chandra, Jiyong Choi, Jayesh Sonawane

llms nlp benchmarks

arXiv AI

Jul 8

Spider 2.0-AIFunc: Extending Real-World Text-to-SQL to AI-Native SQL Workflows

arXiv:2607. 06229v1 Announce Type: cross Abstract: Major cloud data platforms now expose large language model capabilities as native SQL functions, enabling analysts to perform classification, filtering, sentiment analysis, extraction, similarity search, and aggregation within ordinary SQL queries.

By Tianyang Liu, Canwen Xu, Fangyu Lei, Nikki Lijing Kuang, Jixuan Chen, Tao Yu, Julian McAuley, Zhewei Yao, Yuxiong He

llms agents nlp benchmarks data-engineering

arXiv AI

Jul 8

Universal Algorithm-Implicit Learning

arXiv:2602. 14761v2 Announce Type: replace-cross Abstract: Current meta-learning methods are constrained to narrow task distributions with fixed feature and label spaces, limiting applicability.

By Stefano Woerner, Seong Joon Oh, Christian F. Baumgartner

llms rag nlp multimodal benchmarks

arXiv AI

Jul 8

KAT-Coder-V2.5 Technical Report

arXiv:2607. 05471v1 Announce Type: cross Abstract: We present KAT-Coder-V2.

By Bo Huang, Fengxiang Li, Hao Xu, Haoyang Huang, Hongyi Fu, Jinhua Hao, Kun Yuan, Minglei Zhang, Pengcheng Xu, Shiyang Liu, Wenhao Zhuang, Yuze Shi, Zongxian Feng, Chao Wang, Cheng He, Chongling Rao, Deyu Cao, Fan Yang, Gang Xiong, Haochen Liu, Jiabao Li, Jian Liang, Jinghui Jia, Jingwen Chang, Jun Du, Junyu Shi, Min Li, Mingqi Wu, Qiang Gao, Shangpeng Yan, Shaotong Qi, Shu Xu, Shuo Zhou, Tiankuo Xu, Tong Zheng, Weilun Zhao, Xiancheng Meng, Xianda Sun, Xiaoyu Jiang, Xunhao Jia, Yao Xia, Yimeng Xu, Yinghan Cui, Yingpeng Chen, Yiwen Ning, Yong Wang, Yuxuan Sun, Zhongsheng Liu, Ming Sun, Cheng Luo, Chen Yang, Han Li, Kun Gai

agents reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Jul 8

Classification of Financial Data Using Quantum Support Vector Machine

arXiv:2412. 10860v2 Announce Type: replace-cross Abstract: Quantum Support Vector Machine is a kernel-based approach to classification problems.

By Seemanta Bhattacharjee, MD. Muhtasim Fuad, A. K. M. Fakhrul Hossain

benchmarks

arXiv Machine Learning

Jul 8

Partial Symmetry Detection for 3D Geometry using Contrastive Learning with Geodesic Point Cloud Patches

arXiv:2312. 08230v2 Announce Type: replace-cross Abstract: Detecting partial extrinsic symmetry in 3D geometry is a fundamental yet persistent challenge in computer vision and graphics, critical for tasks ranging from shape completion to procedural generation.

By Gregor Kobsik, Isaak Lim, Leif Kobbelt

diffusion computer-vision benchmarks

arXiv Machine Learning

Jul 8

Learning to Throw Objects Safely in Multi-Obstacle Environments

arXiv:2607. 06388v1 Announce Type: cross Abstract: Robotic throwing enables fast and efficient object placement beyond the robot's immediate workspace, but reliable throwing in cluttered environments remains underexplored.

By Mohammadreza Kasaei, Klemen Voncina, Hamidreza Kasaei

reinforcement-learning robotics benchmarks

arXiv AI

Jul 8

BaFCo: A Document Understanding Benchmark for Complex Bangla Form Comprehension

arXiv:2607. 05614v1 Announce Type: cross Abstract: Document comprehension is a challenging yet impactful task for Multimodal Large Language Models, especially as these systems see growing adoption in real-world, human-centric applications.

By Abu Tyeb Azad, Ishita Sur Apan, Fahim Ahmed, Sumaiya Karim Katha, Ezharuddin Jubaer, Armun Alam, Pranjal Kumar Nandi, Amin Ahsan Ali, Aman Chadha, Md Mofijul Islam, AKM Mahbubur Rahman

llms multimodal benchmarks

arXiv AI

Jul 8

PVCap: Towards Accurate 3D Dense Captioning via PseudoCap and VoxelCapNet

arXiv:2607. 06097v1 Announce Type: cross Abstract: 3D dense captioning, an emerging vision-language task, aims to generate descriptive sentences for each object in the 3D scene.

By Xiaopei Wu, Chenshu Hou, Liang Peng, Dan Xu, Binbin Lin, Xiaoshui Huang, Yuenan Hou, Yu Li, Wenxiao Wang, Haifeng Liu, Deng Cai, Wanli Ouyang

multimodal benchmarks

arXiv AI

Jul 8

UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation

arXiv:2607. 06306v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated growing competence in web page generation.

By Grace Man Chen, Litao Guo, Yifan Wu, Yiyu Chen, Yenchi Tseng, Sicheng Liu, Yuyu Luo, Ying-Cong Chen

llms multimodal benchmarks

arXiv AI

Jul 8

StepShield: When, Not Whether to Intervene on Rogue Agents

arXiv:2601. 22136v2 Announce Type: replace-cross Abstract: Agent safety benchmarks measure whether a monitor detects harm, not when.

By Gloria Felicia (University of Virginia), Zitha Sasindran (Indian Institute of Science Bangalore), Jinfeng He (Cornell University), Michael Eniolade (University of the Cumberlands), Hemant Kumar (University of Arizona), Milan Hussain Angati (California State University Northridge)

agents benchmarks safety

arXiv AI

Jul 8

CSTutorBench: Benchmarking Small Language Models as Tutors for Block-Based Programming

arXiv:2607. 05571v1 Announce Type: new Abstract: Large language models are increasingly explored as AI tutors, yet deploying them in K-12 settings raises concerns around privacy, cost, and reliance on proprietary models.

By H. Chad Lane, Bryson Kageler

llms robotics benchmarks safety

arXiv AI

Jul 8

Information Gain-based Rollout Policy Optimization: An Adaptive Tree-Structured Rollout Approach for Multi-Turn LLM Agents

arXiv:2607. 06223v1 Announce Type: new Abstract: Reinforcement learning has become a promising paradigm for improving large language model (LLM) agents on long-horizon search tasks, where the agent must make a sequence of intermediate decisions before receiving a final outcome.

By Yijun Zhang, Fan Xu, Jiaxin Ding, Yule Xie, Shiqing Gao, Xin Ding, Haoxiang Zhang, Luoyi Fu, Xinbing Wang

llms agents reinforcement-learning benchmarks

Get the daily briefing