Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

17,750 stories · RSS feed

arXiv AI
Jul 7

Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimation

arXiv:2607. 03487v1 Announce Type: cross Abstract: Mutual information (MI) estimation is a central problem in machine learning and statistics; however, existing benchmarks typically evaluate estimators on simplified, low-dimensional distributions, leaving their performance on complex, realistic data largely unexplored.

By Alberto Foresti, Ivan Butakov, Alexander Tolmachev, Giulio Franzese, Alexey Frolov, Pietro Michiardi
arXiv AI
Jul 7

Graph Unitary Message Passing

arXiv:2403. 11199v2 Announce Type: replace-cross Abstract: Unitarity is a useful principle for stabilizing deep neural networks, but in graph neural networks (GNNs) instability is induced not only by learnable parameters but also by the graph propagation operator.

By Haiquan Qiu, Quanming Yao
arXiv AI
Jul 7

Agentic and Generative AI for Open-Source Intelligence and Cyber Investigations: Taxonomy, Evaluation, Challenges, and Future Directions

arXiv:2607. 03233v1 Announce Type: cross Abstract: The rapid growth of publicly available digital information has rendered manual open-source intelligence (OSINT) analysis insufficient for modern intelligence, cybersecurity, and cyber investigation.

By Eduardo Almeida Palmieri, Mohamed Chahine Ghanem, Dipo Dunsin, Zubair Baig, Ed de Quincey, Kim-Kwang Raymond Choo
arXiv AI
Jul 7

LLMoxie: Exploring Agentic AI for Scientific Software Development

arXiv:2607. 02703v1 Announce Type: cross Abstract: In this paper, we describe LLMoxie, an institutional AI platform whose three-tiered architecture supports multi-cloud and on-premise inference, a LiteLLM/MLflow control plane for authentication, budgeting, PII masking, and observability, and an application augmentation layer for AI coding agents.

By Landung Setiawan, Anant Mittal, Cordero Core, Anshul Tambay, Carlos Garcia Jurado Suarez, David A. C. Beck, Andrew J. Connolly, Vani Mandava
arXiv Machine Learning
Jul 7

Functional Bilevel Optimization for Predictive Fairness

arXiv:2607. 05098v1 Announce Type: new Abstract: When sensitive attributes are continuous and high-dimensional $-$ demographic score vectors, posteriors over attributes, age or income profiles $-$ enforcing full statistical independence is often too restrictive, and existing relaxations rely on indirect dependence penalties or adversarial schemes that do not directly target the fairness-accuracy trade-off.

By Ieva Petrulionyte, Julien Mairal, Michael Arbel
arXiv AI
Jul 7

PosterHarness: Turning Scientific Poster Generation into an Auditable Instruction-Following Benchmark

arXiv:2607. 03006v1 Announce Type: cross Abstract: Text-rich image models can now design poster-scale layouts, but we lack ways to measure whether they honor scientific communication contracts: legible labels, prescribed aspect ratios, and -- above all -- abstaining from fabricated scientific figures.

By Tianyi Yang, Dawei Fu, Youpeng Wu, Zixun Kou, Linrui Chen, Ruobing Jiang, Zijian Wang, Qiang Li