Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,065 stories · RSS feed

arXiv AI

Jul 23

Logic-Guided Data Extraction with Answer Set Programming and Large Language Models

arXiv:2607. 19365v1 Announce Type: new Abstract: When Large Language Models (LLMs) are used for semantic data extraction from unstructured text, producing candidate relational facts from natural language, they may remain unreliable for tasks requiring complex combinatorial reasoning and global consistency.

By Mario Alviano, Lorenzo Grillo, Nicola Leone, Fabrizio Lo Scudo

llms benchmarks

arXiv AI

Jul 23

The Maskability Index: Predicting Task-Objective Alignment in Pretrained Language Models

arXiv:2607. 20265v1 Announce Type: cross Abstract: Large-scale pretrained language models such as T5 and BERT have demonstrated strong capabilities for generating structured knowledge.

By Ahmad Pouramini, Mahsa Afsharzadeh

llms rag benchmarks safety

arXiv AI

Jul 23

CrackedPDFs: A Controlled Benchmark for Hidden Prompt Injection in PDFs

arXiv:2607. 19396v1 Announce Type: new Abstract: Document-based LLM systems often flatten a PDF before guardrails inspect it.

By Pukaphol Thienpreecha ("Volk")

llms benchmarks safety

arXiv AI

Jul 23

Leveraging Offline Supervision for Efficient and Generalizable Reinforcement Learning in Large-Scale Vision-Language-Action Models

arXiv:2607. 19399v1 Announce Type: cross Abstract: It is commonly observed that online reinforcement learning (RL) produces better-performing strategies than offline methods across a broad range of performance measures.

By Dmitriy Poyarkov, Aleksei Staroverov, Aleksandr I. Panov

reinforcement-learning fine-tuning multimodal benchmarks

arXiv Machine Learning

Jul 23

Multi-modal transformer for signal classification in nanopore blockade experiments

arXiv:2607. 20323v1 Announce Type: new Abstract: Nanopore devices have emerged as powerful tools for single-molecule sensing, with potential for rapid, portable diagnostics.

By Sandro Kuppel, Julian Ho{\ss}bach, Samuel Tovey, Christian Holm

llms multimodal benchmarks

arXiv AI

Jul 23

Sophisticated Policies from Epistemic Priors

arXiv:2607. 19518v1 Announce Type: new Abstract: Sophisticated Inference is a variant of active inference often associated with recursive belief modeling and tree search.

By Wouter W. L. Nuijten, Bert de Vries

benchmarks

arXiv AI

Jul 23

Reinforcement Learning for Large Language Model Selective Evidence Adoption from Contaminated Retrieval Results

arXiv:2607. 20090v1 Announce Type: cross Abstract: Retrieval-augmented large language models frequently face contexts that interleave useful evidence with misleading statements or instruction-like content.

By Yanyu Chen, Yue Li, Yongyi Cui, Dongsheng Shi, Lichang Dai

llms rag reinforcement-learning benchmarks

arXiv AI

Jul 23

JANUS: Foreseeing Latent Risk for Long-Horizon Agent Safety

arXiv:2607. 19913v1 Announce Type: new Abstract: Agent safety is moving from content moderation toward preventing operational failures before tool-using agents act.

By Yuan Xiong, Linji Hao, Shizhu He, Yequan Wang, Lijun Li

agents reinforcement-learning benchmarks safety

arXiv AI

Jul 23

PRIME-SVR: Physics-infoRmed Implicit Multi-Echo Slice-to-Volume Reconstruction for Fetal T2 mapping

arXiv:2607. 20136v1 Announce Type: cross Abstract: Slice-to-volume reconstruction (SVR) is the standard method for obtaining high-resolution (HR) 3D fetal brain volumes from motion-corrupted 2D MRI slice stacks acquired in multiple orientations.

By Busra Bulut, Maik Dannecker, Thomas Sanchez, Sara Neves Silva, Steven Jia, Jean-Baptiste Ledoux, Leo Pomar, Joanna Sichitiu, Yvan Gomez, Meriam Koob, Vincent Dunet, Maria Deprez, Guillaume Auzias, Francois Rousseau, Jana Hutter, Daniel Rueckert, Meritxell Bach Cuadra

benchmarks

arXiv AI

Jul 23

Language-Specific versus Cross-Lingual Knowledge Graphs for Implicit Aspect Identification in Arabic: A Comparative Study of Reasoning and Adaptation Strategies

arXiv:2607. 20056v1 Announce Type: cross Abstract: Aspect-based sentiment analysis (ABSA) in Arabic must recover both explicitly stated aspects and implicit aspects that are never named in the text.

By Lujain A. Alawwad

llms rag nlp fine-tuning benchmarks

arXiv AI

Jul 23

Pushing the Frontier of Full-Song Generation: Hierarchical Autoregressive Planning Meets Flow-Matching Rendering

arXiv:2607. 20253v1 Announce Type: cross Abstract: In this report, we present a unified song generation framework capable of producing high-quality full-length music from lyrics, text descriptions, and musical attributes.

By Junyu Dai, Xinyue Fan, Weiqin Li, Xiangang Li, Yunjia Li, Bin Ma, Yukun Ma, Chongjia Ni, Yufei Shi, Haoxu Wang, Menglin Wu, Jianwei Yu, Huaicheng Zhang, Han Zhao, Shengkui Zhao, Haina Zhu

diffusion nlp reinforcement-learning benchmarks

arXiv AI

Jul 23

ITPEval: Benchmarking Formal Translation Across Interactive Theorem Provers

arXiv:2607. 19407v1 Announce Type: new Abstract: Formal theorem proving has emerged as a frontier challenge for machine learning, yet the ecosystem is fragmented: proofs remain siloed across incompatible systems, limiting both training data for learning-based provers and the portability of verified results.

By Jiayi Wu, Robert Joseph George, Anima Anandkumar

llms benchmarks

arXiv AI

Jul 23

Structured Latent Space Modeling over Multi-Scale Temporal Patches for Multivariate Time Series Forecasting

arXiv:2607. 19404v1 Announce Type: cross Abstract: Multivariate time series encode structural patterns that unfold across multiple temporal scales, yet most forecasting backbones treat learned representations as transient byproducts of prediction, leaving the organizational geometry of these patterns underexploited.

By Xingsheng Chen, Deyu Yi, Siu-Ming Yiu

diffusion benchmarks safety

arXiv Machine Learning

Jul 23

AlphaRoute: Large Language Models as Semantic Optimizers for Multi-Objective Routing

arXiv:2607. 19768v1 Announce Type: new Abstract: Very Large Scale Integration (VLSI) global routing is an NP-hard combinatorial optimization problem requiring signal net assignment across capacity-constrained 3D grids while minimizing congestion, wirelength, and via transitions.

By Kabir Murjani, Mishri Bhavsar, Manish I. Patel, Jonti Talukdar

llms benchmarks

arXiv AI

Jul 23

FormulaSPIN: Self-Play Fine-Tuning for Natural Language to Spreadsheet Formula Generation

arXiv:2607. 19354v1 Announce Type: new Abstract: Spreadsheet applications are used by hundreds of millions worldwide, yet writing formulas remains a significant barrier.

By Cy Xie

reinforcement-learning fine-tuning benchmarks

arXiv AI

Jul 23

ENTRAP-VL: A Taxonomic Probe for Dual Contextual Entrainment in Vision-Language Models

arXiv:2607. 20092v1 Announce Type: cross Abstract: Contextual entrainment is the tendency of a model to let auxiliary context in its input pull its output, independently of whether that context is relevant, true, or even meaningful.

By Karan Goyal, Afreen Hossain, Debojyoti Das, Vishal Bhutani

llms multimodal benchmarks

arXiv AI

Jul 23

PGTT: Phase-Guided Terrain Traversal for Perceptive Legged Locomotion

arXiv:2510. 18348v2 Announce Type: replace-cross Abstract: State-of-the-art perceptive Reinforcement Learning controllers for legged robots typically either (i) impose oscillator-or IK-based gait priors that constrain the action space, bias policy optimization, and limit adaptability across robot morphologies, or (ii) operate "blind," making them unable to anticipate hind-leg terrain and brittle to observation noise.

By Alexandros Ntagkas, Chairi Kiourt, Konstantinos Chatzilygeroudis

reinforcement-learning robotics benchmarks safety

arXiv Machine Learning

Jul 23

Antigen-specific Antibody Multi-modal Foundation Model for Functional Antibody Design

arXiv:2607. 20057v1 Announce Type: cross Abstract: Antibodies are essential proteins that play a central role in immune recognition by binding specific antigen molecules.

By Xiaoliang Shi, Zichen Wang, Runze Ma, Zhongyue Zhang, Shuangjia Zheng

llms diffusion reinforcement-learning fine-tuning multimodal benchmarks

arXiv AI

Jul 23

Benchmarking Confidential GPU Inference on NVIDIA H100 under Intel TDX

arXiv:2607. 19353v1 Announce Type: new Abstract: Confidential computing is becoming a practical deployment requirement for AI inference workloads that process sensitive inputs or protect proprietary model assets.

By Wei Wang, Abdul Hyee Waqas, Burns Smith

llms efficiency benchmarks

arXiv Machine Learning

Jul 23

Trend strength predicts when generative foundation models win: a power-controlled benchmark, a mechanism, and an actionable selection rule

arXiv:2607. 19383v1 Announce Type: cross Abstract: Pretrained generative foundation models cast forecasting as conditional generation from a learned predictive distribution and forecast unseen series zero-shot.

By Ahmed Cherif

benchmarks

Get the daily briefing