Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

22,530 stories · RSS feed

arXiv AI

Jun 10

ChartAgent: A Multimodal Agent for Visually Grounded Reasoning in Complex Chart Question Answering

arXiv:2510. 04514v3 Announce Type: replace Abstract: Recent multimodal LLMs have shown promise in chart-based visual question answering, but their performance declines sharply on unannotated charts-those requiring precise visual interpretation rather than relying on textual shortcuts.

By Rachneet Kaur, Nishan Srishankar, Zhen Zeng, Sumitra Ganesh, Manuela Veloso

llms agents nlp multimodal benchmarks

arXiv AI

Jun 10

People-Centred Medical Image Analysis via Fairness-Aware Human-AI Cooperation

arXiv:2604. 26991v2 Announce Type: replace-cross Abstract: Machine learning models for medical image analysis often exhibit subgroup-dependent performance, which impacts how decisions should be allocated between automated systems and human experts under limited resources.

By Zheng Zhang, Milad Masroor, Cuong Nguyen, Tahir Hassan, Yuanhong Chen, David Rosewarne, Kevin Wells, Thanh-Toan Do, Gustavo Carneiro

agents benchmarks safety

arXiv AI

Jun 10

Communication Dynamics Neural Networks: FFT-Diagonalized Layers for Improved Hessian Conditioning at Reduced Parameter Count

arXiv:2605. 08171v2 Announce Type: replace-cross Abstract: Communication Dynamics Neural Networks (CDNNs) apply the circulant-spectral machinery of the Communication Dynamics framework to neural-network layer design.

By Lurong Pan

llms benchmarks

arXiv AI

Jun 10

Role-Agent: Bootstrapping LLM Agents via Dual-Role Evolution

arXiv:2606. 10917v1 Announce Type: new Abstract: Although Large Language Model (LLM) agents have demonstrated strong performance on complex tasks, their learning is often limited by inefficient interaction feedback and static training environments, which hinder broader generalization.

By Xucong Wang, Ziyu Ma, Shidong Yang, Tongwen Huang, Pengkun Wang, Yong Wang, Xiangxiang Chu

llms agents benchmarks safety

arXiv Machine Learning

Jun 10

OncoTraj: a public benchmark for longitudinal resistance prediction in EGFR-mutant non-small-cell lung cancer on osimertinib

arXiv:2606. 11144v1 Announce Type: new Abstract: Resistance to first-line osimertinib in EGFR-mutant non-small-cell lung cancer (NSCLC) is the canonical example of predictable clonal evolution under therapeutic pressure, yet no public benchmark exists for training or evaluating computational models on the corresponding longitudinal patient trajectories.

By Abhijoy Sarkar, Aarchi Singh Thakur

llms benchmarks

arXiv Machine Learning

Jun 10

FreshRetailNet-LT: A Stockout-Annotated Censored Demand Dataset for Latent Demand Recovery and Forecasting in Fresh Retail

arXiv:2505. 16319v3 Announce Type: replace Abstract: Accurate demand estimation is critical for the retail business in guiding the inventory and pricing policies of perishable products.

By Yangyang Wang, Jiawei Gu, Li Long, Xin Li, Li Shen, Zhouyu Fu, Xiangjun Zhou, Xu Jiang

benchmarks safety

arXiv AI

Jun 10

The Model Says Walk: How Surface Heuristics Override Implicit Constraints in LLM Reasoning

arXiv:2603. 29025v3 Announce Type: replace-cross Abstract: Large language models fail when a salient surface cue conflicts with an unstated feasibility constraint.

By Yubo Li, Lu Zhang, Tianchong Jiang, Ramayya Krishnan, Rema Padman

llms benchmarks safety

arXiv Machine Learning

Jun 10

On the Controllability-Fidelity Frontier in Diffusion Editing

arXiv:2606. 09901v1 Announce Type: cross Abstract: Diffusion-based generative models enable powerful image editing capabilities, but achieving precise control while maintaining fidelity and safety remains challenging.

By Yi Hu, Leying Yi, Emily Davis, Finn Carter

diffusion robotics benchmarks safety

arXiv AI

Jun 10

MedFeat: Model-Aware and Explainability-Driven Feature Engineering with LLMs for Clinical Tabular Prediction

arXiv:2603. 02221v2 Announce Type: replace-cross Abstract: In clinical tabular prediction, classical machine learning models with feature engineering often outperform neural methods.

By Zizheng Zhang, Yiming Li, Justin Xu, Jinyu Wang, Rui Wang, Lei Song, Jiang Bian, David W Eyre, Jingjing Fu

llms benchmarks safety

arXiv AI

Jun 10

NOVA: Symbolic Regression Discovery of Interpretable Car-Following and Lane-Change Models with Driver Heterogeneity

arXiv:2606. 10583v1 Announce Type: cross Abstract: We present NOVA, an autonomous symbolic regression framework that identifies interpretable car-following and lane-change structures from raw trajectory data with minimal behavioral priors.

By Ishak Abassi, Nassim Ali Bouazzouni, Farah Ibelaiden, Nadir Farhi

llms agents benchmarks

arXiv Machine Learning

Jun 10

POPSICLE: Benchmark Datasets for Segmentation and Localization in CryoET

arXiv:2606. 10255v1 Announce Type: cross Abstract: Cryo-electron tomography (cryoET) has emerged as a powerful tool in structural and cellular biology by enabling direct visualization of macromolecular structures within intact cells, thereby linking molecular architecture to cellular organization in a native context.

By Jonathan Schwartz, Utz Heinrich Ermel, C. Braxton Owens, Zhuowen Zhao, Ariana Peck, Gus L. W. Hart, Grant J. Jensen, Bridget Carragher, Dari Kimanius

computer-vision benchmarks

arXiv AI

Jun 10

Accounting for AI Inference in Corporate GHG Inventories: A Four-Tier Methodology for Scope 3 Category 1 Reporting

arXiv:2606. 10660v1 Announce Type: cross Abstract: AI inference services -- API subscriptions, enterprise chat tools, and SaaS products with embedded AI features -- fall unambiguously within Scope 3 Category 1 under the Corporate Sustainability Reporting Directive (CSRD), which requires disclosure for fiscal years starting January 2024.

By Guillermo Llopis (SOMA AI, Barcelona)

benchmarks

arXiv AI

Jun 10

Divide and Cooperate: Role-Decomposed Multi-Agent LLM Training with Cross-Agent Learning Signals

arXiv:2606. 10684v1 Announce Type: cross Abstract: Modern language agents which perform multi-step reasoning have shown strong performance in knowledge-intensive question answering.

By Jaewan Park, Solbee Cho, Jay-Yoon Lee

llms agents nlp fine-tuning benchmarks

arXiv AI

Jun 10

Falcon-X: A Time Series Foundation Model for Heterogeneous Multivariate Modeling

arXiv:2605. 27286v2 Announce Type: replace-cross Abstract: Time series foundation models (TSFMs) are transforming the forecasting paradigm through large-scale cross-domain pretraining.

By Yiding Liu, Yifan Hu, Hongjie Xia, Peiyuan Liu, Hongzhou Chen, Xilin Dai, Zewei Dong, Jiang-Ming Yang

benchmarks safety

arXiv AI

Jun 10

Dynamic Linear Attention

arXiv:2606. 10650v1 Announce Type: cross Abstract: The scalability of Large Language Models (LLMs) to long contexts is fundamentally constrained by the quadratic complexity of standard attention, motivating the adoption of linear attention mechanisms with sub-quadratic cost.

By Xin Wang, Hui Shen, Boyuan Zheng, Xueshen Liu, Minkyoung Cho, Zhongwei Wan, Zesen Zhao, Zhuoqing Mao, Shen Yan, Mi Zhang

llms benchmarks

arXiv AI

Jun 10

TaCarla: A comprehensive benchmarking dataset for end-to-end autonomous driving

arXiv:2602. 23499v4 Announce Type: replace-cross Abstract: Collecting a high-quality dataset is a critical task that demands meticulous attention to detail, as overlooking certain aspects can render the entire dataset unusable.

By Tugrul Gorgulu, Atakan Dag, M. Esat Kalfaoglu, Halil Ibrahim Kuru, Baris Can Cam, Halil Ibrahim Ozturk, Ozsel Kilinc

agents computer-vision robotics benchmarks

arXiv AI

Jun 10

Self-Distillation Policy Optimization via Visual Feedback: Bridging Code and Visual Artifacts

arXiv:2606. 10334v1 Announce Type: new Abstract: Code-generating large language models (LLMs) increasingly produce visual artifacts such as charts, web pages, and slides by writing programs that are executed by non-differentiable renderers, committing to code before observing the render.

By Haoyu Dong

llms efficiency benchmarks safety

arXiv AI

Jun 10

Machine Learning Methods for Studying Latent Neural Activity Dynamics

arXiv:2606. 10530v1 Announce Type: cross Abstract: Recent developments in brain recording are driving a demand for machine learning tools capable of decoding the latent structure of large populations of neurons.

By Shufeng Kong, Fumei Deng, Xinyi Dong, Caihua Liu, Weiwei Chen, Yingheng Wang, Daniel Cao, Azahara Oliva, Antonio Fernandez-Ruiz, Carla Gomes

llms diffusion benchmarks

arXiv Machine Learning

Jun 10

CodeAlchemy: Synthetic Code Rewriting at Scale

arXiv:2606. 10087v1 Announce Type: cross Abstract: Pre-training on raw code teaches syntax but provides sparse signal for diverse real-world task formats.

By Ankit Gupta, Aditya Prasad, Rameswar Panda

llms benchmarks

arXiv Machine Learning

Jun 10

Blind denoising diffusion models and the blessings of dimensionality

arXiv:2602. 09639v2 Announce Type: replace Abstract: Denoising diffusion models (DDMs) are state-of-the-art methods for learning densities from data across numerous domains, yet many aspects of the training and sampling pipeline remain poorly understood.

By Zahra Kadkhodaie, Aram-Alexandre Pooladian, Sinho Chewi, Eero Simoncelli

rag diffusion benchmarks

Get the daily briefing