Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,257 stories · RSS feed

arXiv Machine Learning

Jul 7

Piecewise Dynamic Diffusion Regularization for Reconstruction of Cardiac Cine MRI

arXiv:2607. 03299v1 Announce Type: cross Abstract: Real-time cardiac cine MRI enables visualization of the beating heart during free breathing, but severe undersampling and motion make reconstruction highly challenging.

By Florian F\"urnrohr, Reinhard Heckel

diffusion benchmarks

arXiv Machine Learning

Jul 7

TT-Sparse: Learning Sparse Rule Models with Differentiable Truth Tables

arXiv:2603. 07606v2 Announce Type: replace Abstract: Interpretable machine learning is essential in high-stakes domains where decision-making requires accountability, transparency, and trust.

By Hans Farrell Soegeng, Sarthak Ketanbhai Modi, Thomas Peyrin

benchmarks safety

arXiv Machine Learning

Jul 7

Incentivizing Vision Language Models to Search for Long Video Question Answering

arXiv:2607. 02959v1 Announce Type: cross Abstract: We introduce VSeek, an agentic framework that transforms long-video question answering (LVQA) from a passive, single-pass perception task into a multi-turn retrieval process.

By Harsh Goel, S P Sharan, Sahil Shah, Minkyu Choi, Joungbin An, Kristen Grauman, Sandeep P. Chinchali

llms agents nlp reinforcement-learning benchmarks

arXiv AI

Jul 7

SpectralGCD: Spectral Concept Selection and Cross-modal Representation Learning for Generalized Category Discovery

arXiv:2602. 17395v2 Announce Type: replace-cross Abstract: Generalized Category Discovery (GCD) aims to identify novel categories in unlabeled data while leveraging a small labeled subset of known classes.

By Lorenzo Caselli, Marco Mistretta, Simone Magistri, Andrew D. Bagdanov

efficiency multimodal benchmarks

arXiv AI

Jul 7

APeB: Benchmarking Personalization Ability of Large Language Model Agents

arXiv:2607. 03162v1 Announce Type: new Abstract: LLM-powered agents struggle with personalization when users issue raw, underspecified queries.

By Garry Yang, Zizhe Chen, Xinru Chen, Yongqiang Chen, Jianxiang Wang, Deyu Zou, Linyi Ding, Jialiang Wu, Yunzhong He, Yu Gong, James Cheng, Huaixiao Tou

llms agents benchmarks

arXiv AI

Jul 7

SilvaScenes: Tree Detection and Species Classification from Under-Canopy Images in Natural Forests

arXiv:2510. 09458v2 Announce Type: replace-cross Abstract: Interest in forestry automation is growing alongside rapid advances in deep learning.

By David-Alexandre Duclos, William Guimont-Martin, Gabriel Jeanson, Arthur Larochelle-Tremblay, Martine Lapointe, Th\'eo Defosse, Fr\'ed\'eric Moore, Philippe Nolet, Fran\c{c}ois Pomerleau, Philippe Gigu\`ere

computer-vision benchmarks

arXiv AI

Jul 7

Interpretable Nanoporous Materials Design with Symmetry-Aware Networks

arXiv:2509. 15908v3 Announce Type: replace-cross Abstract: Nanoporous materials hold promise for diverse sustainable applications, yet their vast chemical space poses challenges for efficient design.

By Zhenhao Zhou, Salman Bin Kashif, Jin-Hu Dou, Chris Wolverton, Kaihang Shi, Tao Deng, Zhenpeng Yao

benchmarks safety

arXiv AI

Jul 7

Agent vs. Parametric World Models: Hybrid Planning for Reliable Language Agents

arXiv:2606. 27806v3 Announce Type: replace Abstract: Language agents plan by generating not only actions but also implicit predictions of how the world will change.

By Xinyuan Song, Zekun Cai

llms agents benchmarks

arXiv AI

Jul 7

ICR-RL: Deep Reinforcement Learning via In-Context Regression

arXiv:2509. 11259v2 Announce Type: replace-cross Abstract: Recent advancements in machine learning have largely been driven by foundation models (FMs) trained on large, diverse datasets, enabling them to generalize effectively to new, related tasks.

By David Schiff, Ofir Lindenbaum, Yonathan Efroni

agents reinforcement-learning fine-tuning benchmarks

arXiv AI

Jul 7

Conditional Diffusion Guided Knowledge Transfer for Multi-Domain Knowledge Graph Completion

arXiv:2607. 03154v1 Announce Type: cross Abstract: Multi-domain knowledge graph completion (MKGC) aims to improve missing triple prediction in a target KG by transferring knowledge from other support KGs.

By Jiawei Sheng, Taoyu Su, Xixun Lin, Xiaodong Li, Tingwen Liu

rag diffusion benchmarks

arXiv AI

Jul 7

DynamixSFT: Dynamic Mixture Optimization of Instruction Tuning Collections

arXiv:2508. 12116v2 Announce Type: replace-cross Abstract: As numerous instruction-tuning datasets continue to emerge, dynamically balancing and optimizing their mixtures has become a critical challenge.

By Haebin Shin, Lei Ji, Xiao Liu, Zhiwei Yu, Hyunwoo Yoo, Qi Chen, Yeyun Gong

reinforcement-learning fine-tuning benchmarks

arXiv AI

Jul 7

Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility

arXiv:2501. 10711v5 Announce Type: replace-cross Abstract: Code-related benchmarks play a critical role in evaluating large language models (LLMs), yet their quality fundamentally shapes how the community interprets model capabilities.

By Jialun Cao, Yuk-Kit Chan, Zixuan Ling, Wenxuan Wang, Shuqing Li, Mingwei Liu, Ruixi Qiao, Yuting Han, Chaozheng Wang, Boxi Yu, Pinjia He, Shuai Wang, Zibin Zheng, Michael R. Lyu, Shing-Chi Cheung

llms benchmarks

arXiv AI

Jul 7

Rational Inverse Reasoning: Few-Shot Imitation by Inferring Intent through Planning

arXiv:2508. 08983v2 Announce Type: replace-cross Abstract: Humans can learn a new manipulation task from one or two demonstrations and then perform it in a new room, with new objects, under new constraints.

By Ben Zandonati, Tom\'as Lozano-P\'erez, Leslie Pack Kaelbling

llms robotics multimodal benchmarks

arXiv Machine Learning

Jul 7

SalAngaBhava: A Sinhala Market Dataset for Aspect-based Sentiment Analysis

arXiv:2607. 05259v1 Announce Type: cross Abstract: Sentiment analysis has been a primary domain under Natural Language Processing (NLP) from its inception as it plays a vital role in both real-world and research applications.

By Lakshani Galwatta, Nisansa de Silva, Sarangi Aththanayake, Adithya Galwatta

nlp benchmarks

arXiv Machine Learning

Jul 7

Local Constrained Bayesian Optimization

arXiv:2603. 07965v2 Announce Type: replace-cross Abstract: Bayesian optimization (BO) for high-dimensional constrained problems remains a significant challenge due to the curse of dimensionality.

By Jing Jingzhe, Fan Zheyi, Szu Hui Ng, Qingpei Hu

benchmarks

arXiv AI

Jul 7

Learning in Low-Dimensional Subspaces: Orthogonal Bottlenecks for Reinforcement Learning

arXiv:2605. 26012v2 Announce Type: replace-cross Abstract: Deep reinforcement learning (RL) agents commonly rely on high-dimensional neural representations, despite growing evidence that task-relevant value and policy structure may be intrinsically low-dimensional.

By Aleksandar Todorov, Matthia Sabatelli

agents reinforcement-learning benchmarks

arXiv AI

Jul 7

SwarmResearch: Orchestrating Coding Agents for Open-Ended Discovery

arXiv:2607. 02807v1 Announce Type: new Abstract: Long-running coding agents such as autoresearch can persistently discover optimizations for open-ended problems.

By Yuvraj Virk, Zack Edds, Chunqiu Steven Xia, Lingming Zhang

llms agents benchmarks

arXiv AI

Jul 7

Echoes of Unrest: A Multimodal NLP Framework for Early Warning of Fake News and Violence-Driven Mob Activity

arXiv:2607. 02734v1 Announce Type: cross Abstract: Rapid growth in social media has transformed global communication by enabling fast information exchange, but it has also accelerated the spread of misinformation.

By Md. Maruf Bangabashi, Tahmid Hasan, Golam Mahmud, Md. Mostafijur Rahman, Md. Toufiqur Rahman, Jahanur Biswas

rag nlp multimodal benchmarks

arXiv AI

Jul 7

MentalThink: Shaping Thoughts in Mental SVG World

arXiv:2607. 03530v1 Announce Type: new Abstract: We introduce MentalThink, a visual-symbolic reasoning paradigm that equips Multimodal LLMs (MLLMs) with an executable mechanism for "mental" visualization.

By Kangheng Lin, Jisheng Yin, Dingming Li, En Yu, Yana Wei, Han Zhou, Liang Zhao, Hongyu Zhou, Hongbo Peng, Jianjian Sun, Zheng Ge, Xiangyu Zhang, Daxin Jiang, Jingyu Wang

llms reinforcement-learning fine-tuning multimodal benchmarks safety

arXiv AI

Jul 7

SWIFT: Spatio-temporal Wavelet Integrated Forecasting Framework for Workload Traces

arXiv:2607. 02524v1 Announce Type: cross Abstract: Accurate cloud workload forecasting is pivotal for efficient resource management but remains challenging as workloads are highly volatile and prone to sudden bursts.

By Zeyuan Ding, Lingfeng Zheng, Dian Ding, Guangtao Xue

benchmarks

Get the daily briefing