Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

19,789 stories · RSS feed

arXiv AI

Jun 24

Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions

arXiv:2501. 11790v5 Announce Type: replace-cross Abstract: Recent studies have raised significant concerns regarding the reliability of current mathematics benchmarks, highlighting issues such as simplistic design and potential data contamination.

By Zijin Hong, Hao Wu, Su Dong, Junnan Dong, Yilin Xiao, Yujing Zhang, Zhu Wang, Feiran Huang, Linyi Li, Hongxia Yang, Xiao Huang

llms benchmarks

arXiv AI

Jun 24

ATHENA: Agentic Team for Hierarchical Evolutionary Numerical Algorithms

arXiv:2512. 03476v3 Announce Type: replace-cross Abstract: Progress in computational science depends on complex numerical workflows that must faithfully encode physical laws, yet translating conceptual insight into reliable code remains a major bottleneck.

By Juan Diego Toscano, Daniel T. Chen, George Em Karniadakis

llms agents reinforcement-learning benchmarks

arXiv AI

Jun 24

Entity Resolution via Batched Oracle Queries

arXiv:2606. 24407v1 Announce Type: cross Abstract: We consider an oracle that processes a limited batch of records at a time and clusters those that refer to the same real-world entity.

By Lorenzo Balzotti, Donatella Firmani, Luca Gagliardelli, Giovanni Simonini

benchmarks

arXiv Machine Learning

Jun 24

Experiments with Optimal Model Trees

arXiv:2503. 12902v4 Announce Type: replace Abstract: Model trees provide an appealing way to perform interpretable machine learning for both classification and regression problems.

By Sabino Francesco Roselli, Eibe Frank

benchmarks safety

arXiv AI

Jun 24

Blockwise Policy-Drift Gating for On-Policy Distillation

arXiv:2606. 24084v1 Announce Type: cross Abstract: On-policy distillation (OPD) trains a student policy using teacher signals computed on trajectories sampled by the student itself.

By Liwen Zheng, Haiyun Jiang

efficiency benchmarks

arXiv Machine Learning

Jun 24

Target-Aware Linear Regression Under Distribution Shift

arXiv:2606. 22775v2 Announce Type: replace-cross Abstract: Distribution shift between training and deployment is a pervasive challenge for modern AI systems.

By Zhewen Hou, Tian Zheng

benchmarks safety

arXiv AI

Jun 24

ZONOS2 Technical Report

arXiv:2606. 24320v1 Announce Type: cross Abstract: We present ZONOS2 8B, our latest TTS model, which achieves state-of-the-art naturalness, prosody, and voice cloning fidelity.

By Gabriel Clark, Sofian Mejjoute, Mohamed Osman, George Close, Beren Millidge

efficiency benchmarks

arXiv Machine Learning

Jun 24

3D Masked Autoencoders are Robust Learners of Volumetric and Multimodal Cellular Representations for Microscopy

arXiv:2606. 23964v1 Announce Type: new Abstract: Self-supervised learning in fluorescence microscopy often relies on 2D projections, despite the inherently three-dimensional nature of cells.

By Amirhossein Kardoost, Lion Gleiter, Tingying Peng, Carsten Marr

llms multimodal benchmarks safety

arXiv Machine Learning

Jun 24

Does My Embedding Reflect That $A = B$? Evaluating Mathematical Equivalence in Embedding Models

arXiv:2606. 23959v1 Announce Type: cross Abstract: Because mathematics is highly abstract, a single statement can take very different forms depending on what subfield it is framed in.

By Jiaying Ye, Samarth Rao, Leo Carlin, Kedar Chintalapati, Saharsh Bhargava, Rachit Jaiswal, Michael Zhou, Jared Darlington, Jarod Alper, Vasily Ilin, Henry Kvinge

rag benchmarks

arXiv Machine Learning

Jun 24

Hybrid Sequence Modeling and Reinforced Verification for Controllable Target-Conditioned Decision Making

arXiv:2508. 16420v3 Announce Type: replace Abstract: Target-conditioned sequence models provide a simple interface for controllable offline decision making, but the requested target return can be an unreliable control signal, especially when the target return lies in underrepresented regions of the dataset.

By Yue Pei, Hongming Zhang, Chao Gao, Martin M\"uller, Yingying Zhang, Mengxiao Zhu, Hao Sheng, Ziliang Chen, Liang Lin, Haogang Zhu

llms benchmarks safety

arXiv AI

Jun 24

MOCHA: Multi-modal Objects-aware Cross-arcHitecture Alignment

arXiv:2509. 14001v5 Announce Type: replace-cross Abstract: Personalized object detection aims to adapt a general-purpose detector to recognize user-specific instances from only a few examples.

By Elena Camuffo, Francesco Barbato, Mete Ozay, Simone Milani, Umberto Michieli

llms rag computer-vision efficiency multimodal benchmarks safety

arXiv AI

Jun 24

MMed-Bench-IR: A Heterogeneous Benchmark for Multilingual Medical Information Retrieval

arXiv:2606. 24200v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) in clinical settings increasingly requires multilingual retrieval against predominantly English evidence corpora.

By Junhyeok Lee, Han Jang, Hyeonjin Goh, Kyu Sung Choi

rag benchmarks safety

arXiv Machine Learning

Jun 24

You Don't Need to Run Every Eval

arXiv:2606. 24020v1 Announce Type: new Abstract: A modern model release reports scores on 40+ benchmarks and the same evaluations were run many more times before it: to track training progress, compare design choices, and select the checkpoint for the release.

By Yuchen Zeng, Dimitris Papailiopoulos

benchmarks

arXiv AI

Jun 24

FLUX3D: High-Fidelity 3D Gaussian Generation with Diffusion-Aligned Sparse Representation

arXiv:2606. 24874v1 Announce Type: cross Abstract: Sparse voxel representation has emerged as a scalable foundation for image-to-3D Gaussian Splatting (3DGS) generation, yet current methods struggle to preserve high-frequency visual details of input images due to two structural bottlenecks.

By Haorui Ji, Weizhe Liu, Hongdong Li, Hengkai Guo

llms rag diffusion multimodal benchmarks safety

arXiv AI

Jun 24

Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study

arXiv:2606. 21428v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) language models are often described as ideal for resource-constrained inference.

By Alfarizy Alfarizy, Hung Truong Thanh Nguyen, Ren\'e Richard, Roozbeh Razavi-Far, Hung Cao

llms efficiency benchmarks

arXiv AI

Jun 24

SemChunk-C: Semantic Segmentation for C Code

arXiv:2606. 23697v1 Announce Type: cross Abstract: Semantic segmentation of code written in a C-family language remains a challenging problem, due to the language's complex syntax, macro expansion, and irregular structural patterns.

By Boris Nazarov, Darya Frolova, Shaked Leibzirer, Pavel Kisilev

llms rag computer-vision benchmarks

arXiv AI

Jun 24

Metis: Bridging Text and Code Memory for Self-Evolving Agents

arXiv:2606. 24151v1 Announce Type: cross Abstract: Self-evolving agents improve over time by distilling experience from past executions and reusing it in future tasks.

By Zijie Dai, Siuhin He, Hui Li, Qihui Zhou, Jiajun Li, Mingcong Song, Guoping Long, Hongjie Si, Xin Yao, Lin Zhang, James Cheng, Xiao Yan

agents benchmarks

arXiv AI

Jun 24

RetiSEM: Generalising Causal Models for Fragmented Biomedical Data

arXiv:2606. 24488v1 Announce Type: cross Abstract: Learning causal models from fragmented biomedical data is challenging because clinical, molecular, and imaging variables are often incomplete or not jointly observed.

By Inam Ullah, Imran Razzak, Shoaib Jameel

multimodal benchmarks

arXiv AI

Jun 24

HOLMES: Evaluating Higher-Order Logical Reasoning in LLMs

arXiv:2606. 23238v2 Announce Type: replace Abstract: Logical reasoning is essential for reliable AI, yet existing benchmarks are largely first-order-logic-centric, focusing on object-level deduction over fixed predicates.

By Yucheng Wu, Jundong Xu, Mingzhen Ju, Yue Yu, Chenpeng Wang, Haoxuan Li, Liangming Pan

llms benchmarks

arXiv AI

Jun 24

DeepBD: A Grounded Agentic Workflow for Variant Prioritization and Diagnosis of Genetic Birth Defects

arXiv:2606. 24779v1 Announce Type: cross Abstract: Birth defects are a major cause of fetal loss, neonatal morbidity and long-term disability.

By Shiyu Li, Ziqi Yan, Zhihao Wu, Jielong Lu, Weiran Liao, Jiajun Yu, Genjie Li, Zeyu Chu, Jiajun Bu, Haishuai Wang

llms rag agents benchmarks

Get the daily briefing