← Back to all news

Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv AI
Oct 5

Multimodal reasoning for broadly neutralizing antibody discovery from label-free human B cell repertoires across virus families

The paper introduces ImmuneAgent, a closed‑loop AI system that combines multimodal reasoning, continual meta‑learning, and wet‑lab feedback to identify broadly neutralizing antibodies (bnAbs) from human B cell repertoires. Applied to vaccinated or infected cohorts, ImmuneAgent achieved a 55% neutralization discovery rate and an 11% bnAb yield, outperforming existing sequence‑based predictors and co‑folding models. Five discovered antibodies provided full in vivo protection against lethal influenza, and the system uncovered conserved bnAb reservoirs and structural signatures that enabled cross‑viral antibody discovery without antigen‑specific sorting.

By Hantao Lou, Jianqing Zheng, Can Yue, Meihan Zhang, Yuanchao Bao, Yu Chen, Mengting Huang, Yupeng Yang, Qianyu Pan, Nana Fu, Yansong Shi, Hongli Li, Yangyang Chai, Ruyi Chen, Wansheng Li, Zhu Liang, Rongmei Yao, Yuanhan Mo, Lei Wang, Chunmei Wang, Yun Quan, Qiong Zhang, Xiangxi Wang, Xuetao Cao
multimodalbenchmarks
More like this →
arXiv Machine Learning
Oct 5

RIPPLE in Still Water: Zero-Shot Clustering in Federated Learning with Wavelet Scattering Transform

RIPPLE is a clustered federated learning framework that assigns clients to groups based on an offline spectral characterization of their local data. It uses a variance‑weighted principal‑component prototype embedded via the Wavelet Scattering Transform and decoded by a Gaussian Mixture VAE trained on synthetic client populations. The method achieves the same per‑round communication cost as FedAvg, allows absent clients to receive a personalized model with a single forward pass, and demonstrates superior performance across five heterogeneous benchmarks.

By Alessandro Licciardi
benchmarks
More like this →
arXiv AI
Oct 5

From Benchmarks to Production: A Text-to-SQL System for Complex Financial Data

The paper introduces FLINT, a domain‑specialized Text‑to‑SQL system designed for complex financial databases where data is stored as opaque integer keys. FLINT employs a lookup agent to resolve natural‑language concepts, uses embedding‑based retrieval of query templates from a compact expert‑authored bank, and prunes large schemas by traversing foreign‑key chains. Evaluated on 359 questions across two production financial schemas, FLINT outperforms state‑of‑the‑art baselines and is already deployed in a financial data retrieval service.

By Arijit Sehanobish, Bruno Gomes Coelho, Guillaume Michel, Sophia Zhi, Valerie Faucon-Morin, Kristen Howell
llmsragagentsbenchmarksdata-engineering
More like this →
arXiv AI
Oct 5

HazardWeaver: Scientific Route Selection for Hazard Analysis Agents

HazardWeaver is an AI agent designed to select and execute appropriate scientific methods for hazard analysis by leveraging a Hazard Knowledge Compiler and a Hazard Capability Graph. It dynamically revises its route choices as new evidence and execution results emerge, ensuring that selected methods are both applicable and executable with available data and tools. The system is evaluated on the Hazard Weaver Benchmark, which includes 141 instances across multiple hazard domains, and demonstrates superior performance over existing agent systems, especially when multiple scientific routes are viable.

By Wangshu Zhu, Xueqi Cheng, Liang Wu, Yushun Dong
llmsagentsbenchmarks
More like this →