Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,561 stories · RSS feed

arXiv AI

Jun 16

Lost at the End: Primacy Bias in Multimodal Retrieval-Augmented Question Answering

arXiv:2606. 16494v1 Announce Type: cross Abstract: Knowledge-based visual question answering (KB-VQA) lets vision-language systems answer questions that exceed their parametric knowledge by conditioning a reader on passages retrieved from a Wikipedia-scale knowledge base.

By Jieyuan Liu, Jianyang Gu, Shijie Chen, Jefferson Chen, Zhen Wang

llms rag nlp multimodal benchmarks safety

arXiv AI

Jun 16

Propagating Structural Guidance: Synthesizing Fluorescein Angiography from Fundus Images and Sparse OCT Scans

arXiv:2606. 16234v1 Announce Type: cross Abstract: Fundus fluorescein angiography (FFA) is critical for assessing retinal vascular abnormalities, but its acquisition is invasive and not always feasible.

By Tengfei Ma, Ruiqi Wu, Chenran Zhang, Ye Geng, Na Su, Xiangyuan Duanmu, Tao Zhou, Yi Zhou, Wen Fan

multimodal benchmarks safety

arXiv AI

Jun 16

VeriGraph: Towards Verifiable Data-Analytic Agents

arXiv:2606. 16603v1 Announce Type: cross Abstract: LLM-based agents have demonstrated strong capabilities in data-intensive analytical tasks, yet their outputs are rarely verifiable: a reliance on linear text trajectories makes their reasoning difficult to audit.

By Jiajie Jin, Zhao Yang, Wenle Liao, Yuyang Hu, Guanting Dong, Xiaoxi Li, Yutao Zhu, Zhicheng Dou

llms agents benchmarks

arXiv AI

Jun 16

Emergent Strategic Reasoning Risks in AI: A Taxonomy-Driven Evaluation Framework

arXiv:2604. 22119v2 Announce Type: replace Abstract: As reasoning capacity and deployment scope grow in tandem, large language models (LLMs) gain the capacity to engage in behaviors that serve their own objectives, a class of risks we term Emergent Strategic Reasoning Risks (ESRRs).

By Tharindu Kumarage, Lisa Bauer, Yao Ma, Dan Rosen, Yashasvi Raghavendra Guduri, Anna Rumshisky, Kai-Wei Chang, Aram Galstyan, Rahul Gupta, Charith Peris

llms agents benchmarks safety

arXiv AI

Jun 16

Federated Medical Image Segmentation under Real-World Label Noise: A Benchmark Suite for Noisy Label Learning Method Selection

arXiv:2606. 16868v1 Announce Type: cross Abstract: While federated learning (FL) enables collaborative medical image segmentation without centralizing sensitive data, real-world deployment is frequently complicated by cross-site label imperfections such as contour disagreement, missing or additional structures, and confused labels.

By Markus Bujotzek, Dimitrios Bounias, Stefan Denner, Ralf Floca, Maximilian Fischer, Peter Neher, Klaus Maier-Hein

computer-vision benchmarks

arXiv AI

Jun 16

Are Neuro-Inspired Multi-Modal Vision-Language Models Resilient to Membership Inference Privacy Leakage?

arXiv:2511. 20710v2 Announce Type: replace-cross Abstract: In the age of agentic AI, the growing deployment of multi-modal models (MMs) has introduced new attack vectors that can leak sensitive training data in MMs, causing privacy leakage.

By David Amebley, Sayanton Dibbo

llms agents multimodal benchmarks safety

arXiv Machine Learning

Jun 16

Robust Transformer-Based One-Step Stock Index Forecasting via Shifted Data Augmentation

arXiv:2606. 15701v1 Announce Type: new Abstract: Transformers have shown remarkable success in sequence modeling, yet their direct application to financial time series remains challenging due to noisy signals, short-memory dynamics, and distributional shifts.

By Tien Thanh Thach

llms benchmarks

arXiv Machine Learning

Jun 16

EIBench: A Simulator-Based Benchmark and Turn-Credit RL for Emotion Management

arXiv:2606. 15532v1 Announce Type: cross Abstract: Emotional intelligence (EI) in Large Language Models (LLMs) is often evaluated through static understanding tasks or single-response dialogue generation.

By Rongzhi Zhu, Xiang Huang, Yuchuan Wu, Rui Wang, Zequn Sun, Tao Ren, Weiyao Luo, Bingxue Qiu, Jieping Ye, Yongbin Li, Wei Hu

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jun 16

Pushing the Boundaries of Natural Reasoning: Interleaved Bonus from Formal-Logic Verification

arXiv:2601. 22642v2 Announce Type: replace Abstract: Large Language Models (LLMs) show remarkable capabilities, yet their stochastic next-token prediction creates logical inconsistencies and reward hacking that formal symbolic systems avoid.

By Chuxue Cao, Jinluan Yang, Haoran Li, Kunhao Pan, Zijian Zhao, Zhengyu Chen, Yuchen Tian, Lijun Wu, Conghui He, Sirui Han, Yike Guo

llms fine-tuning benchmarks

arXiv Machine Learning

Jun 16

DAL: A Practical Prior-Free Black-Box Framework for Piecewise Stationary Bandits

arXiv:2501. 19401v5 Announce Type: replace Abstract: We introduce a practical, black-box framework termed Detection Augmented Learning (DAL) for the problem of piecewise stationary bandits without knowledge of the underlying non-stationarity.

By Argyrios Gerogiannis, Yu-Han Huang, Subhonmesh Bose, Venugopal V. Veeravalli

reinforcement-learning benchmarks

arXiv Machine Learning

Jun 16

Towards Data-Efficient Cross-Device Generalization of Grad-Shafranov Equilibria via Transfer Learning Neural Operator

arXiv:2606. 15512v1 Announce Type: new Abstract: Real-time reconstruction of magnetohydrodynamic equilibria is essential for plasma shaping, stability assessment and feedback control in magnetic confinement fusion.

By Jay Phil Yoo, William Howes, Yashika Ghai, Kazuma Kobayashi, Souvik Chakraborty, Syed Bahauddin Alam

fine-tuning benchmarks

arXiv AI

Jun 16

Co-Scraper: query-aware DOM Pruning and Reusable Scraper Synthesis for Lightweight Web Data Extraction

arXiv:2606. 14821v1 Announce Type: cross Abstract: The abundant and heterogeneous nature of web content necessitates automated information extraction, and generating scrapers that can be reused across similar web pages offers an effective solution for scalable data extraction.

By Shoupeng Wang, Jiantao Qiu, Wuyang Zhang, Conghui He

fine-tuning efficiency benchmarks

arXiv Machine Learning

Jun 16

Temporal Difference Learning for Diffusion Models

arXiv:2606. 15048v1 Announce Type: new Abstract: Diffusion models are typically trained with objectives that focus on local denoising targets at individual time steps (or adjacent pairs), which do not enforce consistency between predictions along the denoising trajectory.

By Qizhen Ying, Yangchen Pan, Victor Adrian Prisacariu, Junfeng Wen

diffusion reinforcement-learning benchmarks

arXiv AI

Jun 16

Entropy-Gated Latent Recursion

arXiv:2606. 16620v1 Announce Type: cross Abstract: Inference-time scaling has become the dominant lever for improving language-model reasoning, but existing methods derive rollout diversity from a single source: stochastic token-level sampling.

By Soham Bhattacharjee, Dushyant Singh Chauhan, Salem Lahlou, Martin Takac, Nils Lukas

reinforcement-learning benchmarks

arXiv Machine Learning

Jun 16

Elastic ODYN: Differentiable Optimization for Infeasible Control and Learning in Robotics

arXiv:2606. 16564v1 Announce Type: cross Abstract: Robotic systems routinely encounter conflicting objectives, modeling errors, and degenerate contact conditions that render quadratic programs (QPs) infeasible.

By Aristotelis Papatheodorou, Jose Rojas, Ioannis Havoutis, Carlos Mastalli

robotics benchmarks

arXiv AI

Jun 16

AdaMame: A Training Recipe for Adaptive Multilingual Reasoning

arXiv:2606. 15080v1 Announce Type: cross Abstract: While Large Reasoning Models (LRMs) show strong performance in English, they often fail to reason in the language of the query, a phenomenon known as language collapse.

By Dayeon Ki, Kevin Duh, Marine Carpuat

reinforcement-learning fine-tuning benchmarks safety

arXiv AI

Jun 16

Can We Stop Malicious AI? KILLBENCH: A Benchmark for External AI Kill Switch Feasibility

arXiv:2511. 13725v4 Announce Type: replace-cross Abstract: Malicious AI causing harm to humans is not just a Hollywood fantasy.

By Sechan Lee, Hyounghun Kim, Sangdon Park

llms agents benchmarks safety

arXiv AI

Jun 16

RaBiT: Residual-Aware Binarization Training for Accurate and Efficient LLMs

arXiv:2602. 05367v3 Announce Type: replace Abstract: Efficient deployment of large language models (LLMs) requires extreme quantization, forcing a critical trade-off between low-bit efficiency and performance.

By Youngcheon You, Banseok Lee, Minseop Choi, Seonyoung Kim, Hyochan Chong, Changdong Kim, Youngmin Kim, Dongkyu Kim

llms efficiency benchmarks

arXiv AI

Jun 16

Agentic Framework for Deep Learning workload migration via In-Context Learning

arXiv:2606. 15994v1 Announce Type: new Abstract: Translating deep learning models from PyTorch's flexible, object-oriented design to JAX's functional, stateless setup is usually a manual and error-prone task.

By Qiyue Liang, Steven Ingram, George Vanica, Andi Gavrilescu, Newfel Harrat, Hassan Sipra, Sethuraman Sankaran

llms agents multimodal benchmarks safety

arXiv AI

Jun 16

Boosting Knowledge Graph Foundation Models via Enhanced Negative Sampling

arXiv:2605. 27023v2 Announce Type: replace Abstract: Knowledge graphs (KGs) have become the core backbone of numerous downstream tasks such as question answering and recommender systems.

By Yinan Liu, Wenjin Xu, Zhiyuan Zha, Xiaochun Yang, Bin Wang

rag nlp benchmarks

Get the daily briefing