Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,452 stories · RSS feed

arXiv Machine Learning

Jun 24

A Hybrid, Multi-Layered Pipeline for Phishing and Threat Classification: Independently Validated URL and NLP Engines with a Calibrated Multi-Channel Fusion Stage

arXiv:2606. 21690v2 Announce Type: replace-cross Abstract: Phishing is a multi-modal threat.

By Saifelden M. Ismail, Aser O. Ibrahim, Omar A. Mahmoud

nlp multimodal benchmarks

arXiv AI

Jun 24

A specialized reasoning large language model for accelerating rare disease diagnosis: a randomized AI physician assistance trial

arXiv:2606. 24510v1 Announce Type: new Abstract: Rare diseases affect millions of individuals worldwide, yet timely diagnosis remains a major public health challenge due to scarcity of specialized clinical expertise.

By Haichao Chen, Songchi Zhou, Zhengyun Zhao, Shikai Hu, Xianghong Jin, Hongwei Ji, Li He, Shuli Li, Yiming Qin, Xin Tan, Runfeng Shi, Yih Chung Tham, Jiaye Zhu, Ye Li, Ye Jin, Longhao Cao, Dawei Li, Honghan Wu, Hongqiu Gu, Guanqiao Li, Tudor Groza, Chunying Li, Dian Zeng, Weihong Yu, Gareth Baynam, Saumya Shekhar Jamuar, Min Shen, Shuyang Zhang, Bin Sheng, Sheng Yu, Tien Yin Wong

llms benchmarks

arXiv AI

Jun 24

Polycepta: Object-Centric Appearance Estimation for Multi-Object Tracking

arXiv:2606. 23604v2 Announce Type: replace-cross Abstract: The tracking-by-detection paradigm in multi-object tracking (MOT) typically relies on static appearance descriptors to complement motion estimation.

By Mohamed Nagy, Naoufel Werghi, Jorge Dias, Majid Khonji

benchmarks

arXiv Machine Learning

Jun 24

Machine Learning Modeling for Real-Time Melt Pool Monitoring in Laser Powder Bed Fusion Additive Manufacturing: A Hybrid Approach

arXiv:2606. 23851v1 Announce Type: new Abstract: This work investigates the implementation of artificial intelligence and machine learning (AI/ML) for real-time monitoring in laser powder bed fusion (LPBF) additive manufacturing.

By Inioluwa Emmanuel, Zhuo Yang, Ho Yeung, Xinyao Zhang

rag computer-vision fine-tuning efficiency benchmarks

arXiv AI

Jun 24

Random Rule Forest (RRF): Interpretable and Manageable Ensembles of LLM-Generated Questions for Predicting Success from Unstructured Data

arXiv:2505. 24622v3 Announce Type: replace Abstract: Many high-stakes screening tasks require predicting rare outcomes from unstructured text, where errors are costly and decisions must be auditable.

By Ben Griffin, Aaron Ontoyin Yin, Diego Vidaurre, Ugur Koyluoglu, Joseph Ternasky, Fuat Alican, Yigit Ihlamur

llms benchmarks

arXiv AI

Jun 24

VeriPilot: An LLM-Powered Verilog Debugging Framework

arXiv:2606. 23759v1 Announce Type: cross Abstract: Verilog debugging remains one of the most time-consuming stages in digital circuit design.

By Yihan Wang, Cheng Liu, Jiazheng Zhang, Lei Zhang, Long Cheng, Xiaowei Li, Huawei Li

llms benchmarks

arXiv AI

Jun 24

Heterogeneous 2D/1D Signal Representation Fusion for Underwater Acoustic Modulation Recognition Under Distribution Shift

arXiv:2606. 23702v1 Announce Type: cross Abstract: Modulation recognition systems rely on heterogeneous signal representations.

By Ronglai Qian, Liang An, Xiaoyan Wang, Qing Fan, Ziwei Huang, Yang Ye

benchmarks

arXiv AI

Jun 24

SemChunk-C: Semantic Segmentation for C Code

arXiv:2606. 23697v1 Announce Type: cross Abstract: Semantic segmentation of code written in a C-family language remains a challenging problem, due to the language's complex syntax, macro expansion, and irregular structural patterns.

By Boris Nazarov, Darya Frolova, Shaked Leibzirer, Pavel Kisilev

llms rag computer-vision benchmarks

arXiv Machine Learning

Jun 24

GRACE: Gated Refinement for Accurate Causal Edge Discovery in High-Dimensional Time Series

arXiv:2606. 23880v1 Announce Type: new Abstract: From climate teleconnections to gene regulation, modern time-series datasets encompass tens or hundreds of interacting variables, making causal discovery increasingly challenging.

By Mohammad Fesanghary, Abhinav Havaldar

benchmarks

arXiv AI

Jun 24

ReMMD: Realistic Multilingual Multi-Image Agentic Verification for Multimodal Misinformation Detection

arXiv:2606. 24112v1 Announce Type: new Abstract: Multimodal misinformation detection is increasingly important because viral posts now combine long multilingual narratives, several images, mixed provenance, and subtle text--image framing errors.

By Chenhao Dang, Dantong Zhu, Jun Yang, Conghui He, Weijia Li

llms agents robotics multimodal benchmarks

arXiv AI

Jun 24

Synergizing Physically Constrained MCMC and Chemical-Informed Gaussian Processes for Reaction Network Discovery

arXiv:2606. 23757v1 Announce Type: cross Abstract: Extracting interpretable governing equations from sparse, noisy chemical time-series data remains difficult because discrete reaction topology and continuous kinetic parameters are tightly coupled.

By Runzhe Liu, Zihao Wang, Wenbo Yang, Shengyang Tao

benchmarks

arXiv AI

Jun 24

Can Language Model Agents be Helpful Circuit Explainers in Mechanistic Interpretability?

arXiv:2606. 24026v1 Announce Type: new Abstract: Mechanistic interpretability has made substantial progress in automatically localizing circuits, but explaining what localized components do remains labor-intensive and difficult to standardize.

By Ayan Antik Khan, Harsh Kohli, Yuekun Yao, Huan Sun, Ziyu Yao

llms agents benchmarks safety

arXiv Machine Learning

Jun 24

LLMs are Bayesian, In Expectation, Not in Realization

arXiv:2507. 11768v3 Announce Type: replace-cross Abstract: Bayesian accounts of in-context learning face a direct objection: exact posterior predictives for exchangeable data are invariant to task-preserving order, yet transformers change next-token probabilities when the same examples are serialized differently.

By Leon Chlon, Fatima Sheaib, Zein Khamis, Maggie Chlon, Mahdi El Zein, MarcAntonio M. Awada

llms benchmarks

arXiv Machine Learning

Jun 24

Macro Graph of Experts for Billion-Scale Multi-Task Recommendation

arXiv:2506. 10520v5 Announce Type: replace-cross Abstract: Graph-based multi-task learning at billion-scale presents a significant challenge, as different tasks correspond to distinct billion-scale graphs.

By Hongyu Yao, Zijin Hong, Hao Chen, Zhiqing Li, Qijie Shen, Zuobin Ying, Qihua Feng, Huan Gong, Feiran Huang

rag benchmarks

arXiv AI

Jun 24

Age of LLM: A Strategic 1v1 Benchmark for Reasoning, Diplomacy and Reliability of Large Language Models under Fog of War

arXiv:2606. 24391v1 Announce Type: new Abstract: We introduce Age of LLM, a turn-based 1v1 benchmark in which two LLMs face off on a 13x7 grid to destroy the enemy base.

By Arnaud Ricci

llms benchmarks safety

arXiv AI

Jun 24

ZONOS2 Technical Report

arXiv:2606. 24320v1 Announce Type: cross Abstract: We present ZONOS2 8B, our latest TTS model, which achieves state-of-the-art naturalness, prosody, and voice cloning fidelity.

By Gabriel Clark, Sofian Mejjoute, Mohamed Osman, George Close, Beren Millidge

efficiency benchmarks

arXiv Machine Learning

Jun 24

Holistic Data Scheduler for LLM Pre-training via Multi-Objective Reinforcement Learning

arXiv:2606. 24133v1 Announce Type: new Abstract: The composition of training data, governed by the diversity of sources and their mixing strategy, is a cornerstone of Large Language Model (LLM) pre-training.

By Chenhao Dang, Jing Ma, Mingjie Liao

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jun 24

NeuroSonic: Conditional Flow Matching for EEG-to-Speech Reconstruction

arXiv:2606. 24087v1 Announce Type: new Abstract: Reconstructing continuous speech from scalp electroencephalography (EEG) remains fundamentally challenging.

By Wenhao Gao, Yifan Wang, Yijia Ma, Carl Yang, Wen Li, Chenyu You

llms diffusion benchmarks

arXiv AI

Jun 24

DTT-BSR+: A Generative-Regression Cascade for Music Source Restoration

arXiv:2606. 24127v1 Announce Type: cross Abstract: Music source restoration (MSR) requires jointly addressing source unmixing and the inversion of non-linear production effects.

By Youran Ni, Shihong Tan, Yuzhu Wang, Gongping Huang

benchmarks

arXiv AI

Jun 24

EG-VQA: Benchmarking Verifiable Video Question Answering with Grounded Temporal Evidence

arXiv:2606. 24797v1 Announce Type: cross Abstract: Recent advances in Video Large Language Models (Video-LLMs) have yielded promising performance on video question answering (VideoQA).

By Linpeng Huang, Weixing Chen, Zexin Chen, Yang Liu, Liang Lin

llms nlp benchmarks safety

Get the daily briefing