Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,633 stories · RSS feed

arXiv Machine Learning

Jul 21

PsiLogic: Chaos-Aware Active Cancellation for Adam with a Fair Cross-Domain Benchmark

arXiv:2607. 16268v1 Announce Type: new Abstract: Adaptive optimizers such as Adam and AdamW apply the same update rule regardless of whether training is in a chaotic early phase or near convergence.

By Ali Sultonov

llms diffusion nlp benchmarks

arXiv AI

Jul 21

From Scene-Centric to Observer-Centric: Modeling Observer-Aware Relations for 3D Scene Graph Generation

arXiv:2606. 27412v2 Announce Type: replace-cross Abstract: 3D Scene Graph Generation (3DSGG) represents 3D scenes as structured object--relation--object graphs for spatial understanding.

By Jingjun Sun, Chaowei Wang, Zhirui Liu, Jiaxu Tian, Ming Yang, Yaoxing Wang, Yan Di, Shan Gao

benchmarks

arXiv AI

Jul 21

Generative Ontology Induction: Domain-Agnostic Schema Discovery from Document Corpora Using Large Language Models

arXiv:2607. 16201v1 Announce Type: new Abstract: Ontology engineering remains a critical bottleneck in knowledge-intensive AI systems.

By Sergei Sergienko

llms benchmarks

arXiv Machine Learning

Jul 21

Kernel Regression with Tensor Trains and Hadamard Overparameterization

arXiv:2607. 17390v1 Announce Type: cross Abstract: Kernel regression with tensor trains and Hadamard overparameterization (KReTTaH) is introduced as a training-data-free, interpretable, and nonparametric framework for multi-way data imputation.

By Duc Thien Nguyen, Konstantinos Slavakis, Eleftherios Kofidis, Dimitris Pados

efficiency benchmarks

arXiv AI

Jul 21

SynGR: Unleashing the Potential of Cross-Modal Synergy for Generative Recommendation

arXiv:2605. 18920v2 Announce Type: replace-cross Abstract: Generative Recommendation (GR) has emerged as a promising paradigm by formulating item recommendation as a sequence-to-sequence generation task over item identifiers.

By Wei Chen, Xingyu Guo, Shuang Li, Fuwei Zhang, Meng Yuan, Jing Fan, Zhao Zhang, Deqing Wang, Fuzhen Zhuang

multimodal benchmarks safety

arXiv Machine Learning

Jul 21

Team DACTYL at PAN 2026: Bayesian Data Mixing and Empirical X-risk Minimization for AI-text Detection

arXiv:2607. 17382v1 Announce Type: cross Abstract: Existing research shows that AI-generated text detection classifiers achieve strong in-distribution (ID) performance but do not maintain the same performance on out-of-distribution (OOD) texts, suggesting overfitting to dataset-specific features.

By Shantanu Thorat

fine-tuning benchmarks

arXiv Machine Learning

Jul 21

FlashPDE: A Drop-in Fused Triton Operator Library for Neural PDE Solvers

arXiv:2607. 18020v1 Announce Type: new Abstract: Physics-Informed Neural Networks (PINNs) solve PDEs by incorporating physical constraints into neural-network training, but large-scale problems are limited by automatic-differentiation memory overhead and inefficient execution of grid-based PDE operators.

By Peiyu Zang, Bosen Xie, Ruoxiang Xu, Yongqiang Cai

benchmarks

arXiv AI

Jul 21

Boogu-Image-0.1: Boosting Open Agentic Multimodal Generation via Understanding under a Minimal Budget

arXiv:2607. 13125v2 Announce Type: replace-cross Abstract: We introduce Boogu-Image-0.

By Guoxuan Chen, Chufeng Xiao, Haoran Yang, Siyue Xie, Binxiao Huang, Ming Zhang, Cheuk Him Chau, Xinyu Fu, Yingzhao Lian, Tom S. Y. Li, Jintao Lin, Bowen Dong, Zian Qian, Yuhao Liu, Yuxuan Hu, Weikang Shi, Bin Zou, Bowen Zheng, Haoxuan Che, Chang Chen, Yuyang He, Heyang Sun, Tianyu Huang, Chong Hou Choi, Cheng Gong, Han Shi, Haoli Bai, Xihui Liu, Hongsheng Li, Qifeng Chen, Chao Huang, Rui Liu, Chenyang Lei

llms agents diffusion multimodal benchmarks data-engineering

arXiv AI

Jul 21

HiCI: Hierarchical Construction-Integration for Long-Context Attention

arXiv:2603. 20843v3 Announce Type: replace-cross Abstract: Long-context language modeling is commonly framed as a scalability challenge of token-level attention, yet local-to-global information structuring remains largely implicit in existing approaches.

By Xiangyu Zeng, Qi Xu, Yunke Wang, Chang Xu

llms benchmarks safety

arXiv AI

Jul 21

Reward-Driven LLM Agent Workflows: Synthesizing POMDP Routing and Self-Correction for Autonomous Decision-Making

arXiv:2607. 17038v1 Announce Type: new Abstract: This paper addresses key technical challenges in current large language model (LLM) agent applications, including long-horizon planning, sparse reward attribution, and dynamic environmental interaction, by designing and optimizing an intelligent agent workflow.

By Amez Amanj Ali, Kuo-Kun Tseng

llms agents reinforcement-learning robotics multimodal benchmarks safety

arXiv AI

Jul 21

A Systematic Evaluation of Trajectory Data Curation for LoRA Fine-Tuning of Code Agents

arXiv:2607. 17205v1 Announce Type: new Abstract: Supervised fine-tuning (SFT) of open-weight LLMs on expert agent trajectories has emerged as a prominent approach to building capable code agents without reliance on proprietary models.

By Yunze Han

llms agents fine-tuning benchmarks

arXiv AI

Jul 21

Is Your Model Thinking or Just Stagnating? PUMA: Diagnosing Reasoning Pathology via Phase-Momentum Alignment

arXiv:2607. 17188v1 Announce Type: new Abstract: Test-time scaling empowers Large Reasoning Models (LRMs) to tackle complex tasks via extensive Chain-of-Thought (CoT).

By Cheng Yan, Guangyang Ye, Wuyang Zhang, Fan Xu, Zhijun Fan, Xiang Xia, Yanyong Zhang

benchmarks safety

arXiv AI

Jul 21

An Explicit World Model Based on Data-First Ontology: DaoQL Multimodal Storage Validation and Counterfactual Reasoning Evaluation

arXiv:2607. 17269v1 Announce Type: new Abstract: Large language models encode world models implicitly in neural weights, which exposes four structural risks in high-precision domains such as medicine and finance: hallucination, frozen knowledge, poor explainability, and poor modifiability.

By Zhanbo Li, Shifeng Wu, Xiangjin Meng, Wenjie Cai

llms agents multimodal benchmarks safety

arXiv AI

Jul 21

Democratizing AI with Small Language Models: Structured Benchmarking and Parameter-Efficient Fine-Tuning for Local Deployment

arXiv:2607. 16202v1 Announce Type: new Abstract: AI democratization is not primarily a question of matching frontier-scale generality; it is a question of whether capable models can be selected, audited, and specialized under hardware and governance constraints that ordinary institutions can actually satisfy.

By Daniel Cersosimo

llms fine-tuning efficiency benchmarks

arXiv AI

Jul 21

Enhancing Rubric-based RL via Self-Distillation

arXiv:2607. 18082v1 Announce Type: cross Abstract: Rubric-based RL has recently shown promise in improving LLMs on open-ended tasks.

By Mingxuan Xia, Yuhang Yang, Chao Ye, Shuai Zhu, Shenzhi Yang, Guangcheng Zhu, Yuhang Zhang, Cheng Peng, Haobo Wang, Siqing Wang

llms reinforcement-learning efficiency benchmarks

arXiv AI

Jul 21

Judge-dependent safety gains and model-specific helpfulness costs of evidence-sufficiency prompting in clinical LLMs

arXiv:2607. 18086v1 Announce Type: new Abstract: Background: LLM judges increasingly score whether clinical language models give overconfident answers under incomplete evidence, yet whether a measured "safety gain" reflects real behavior change or the judge's calibration is unresolved.

By Koyar Afrasyab

llms benchmarks safety

arXiv AI

Jul 21

When Cultures Move: Measuring and Improving Multicultural Text-to-Video Generation

arXiv:2605. 16716v5 Announce Type: replace-cross Abstract: Text-to-video (T2V) generation has rapidly progressed in visual fidelity, yet its ability to faithfully represent multiple cultures within a single prompt remains underexplored.

By Shuowei Li, Yuming Zhao, Parth Bhalerao, Oana Ignat

agents diffusion multimodal benchmarks

arXiv AI

Jul 21

A Benchmark for Early-stage Parkinson's Disease Detection from Speech

arXiv:2605. 14066v2 Announce Type: replace-cross Abstract: Early-stage Parkinson's disease (EarlyPD) detection from speech is clinically meaningful yet underexplored, and published results are hard to compare because studies differ in datasets, languages, tasks, evaluation protocols, and EarlyPD definitions.

By Terry Yi Zhong, Cristian Tejedor-Garcia, Khiet P. Truong, Janna Maas, Louis ten Bosch, Bastiaan R. Bloem

benchmarks

arXiv AI

Jul 21

Short-Term Electricity Demand Forecasting for New England: A Comprehensive Machine Learning Benchmark with Weather, Calendar, and COVID-19 Indicators

arXiv:2606. 20918v2 Announce Type: replace-cross Abstract: Accurate short-term electricity demand forecasting is critical for reliable power system operation, energy market planning, and infrastructure optimization.

By Reza Ghanavati, Behrooz Mosallaei

llms benchmarks

arXiv AI

Jul 21

Spatiotemporal Knowledge Graphs as Persistent Scene Memory for Embodied Question Answering

arXiv:2510. 01483v3 Announce Type: replace-cross Abstract: Vision-language models (VLMs) demonstrate strong image-level scene understanding, but reasoning over long egocentric video remains costly: because VLMs maintain no persistent memory or explicit spatial representation, all sampled frames must be re-processed for every new query.

By Mohamad Al Mdfaa, Svetlana Lukina, Timur Akhtyamov, Arthur Nigmatzyanov, Dmitrii Nalberskii, Sergey Zagoruyko, Gonzalo Ferrer

llms agents nlp robotics multimodal benchmarks

Get the daily briefing