Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

19,504 stories · RSS feed

arXiv AI

Jun 30

How to Train Your Long-Context Visual Document Model

arXiv:2602. 15257v3 Announce Type: replace-cross Abstract: We present the first comprehensive, large-scale study of training long-context vision language models up to 344K context, targeting long-document visual question answering with measured transfer to long-context text.

By Austin Veselka

llms nlp fine-tuning benchmarks

arXiv AI

Jun 30

Mandol: An Agglomerative Agent Memory System for Long-Term Conversations

arXiv:2606. 29778v1 Announce Type: cross Abstract: Long-term conversational agents need to remember and query cross-session, multi-typed information with complex correlations.

By Yuhan Zhang (Institute of Software, Chinese Academy of Sciences), Zhiyuan Guo (Institute of Software, Chinese Academy of Sciences), Ziheng Zeng (Institute of Software, Chinese Academy of Sciences), Wei Wang (Institute of Software, Chinese Academy of Sciences), Wentao Wu (Microsoft Research), Lijie Xu (Institute of Software, Chinese Academy of Sciences)

llms rag agents benchmarks

arXiv Machine Learning

Jun 30

Randomized neural operator for parametric PDEs with fast training and conformal uncertainty quantification

arXiv:2606. 29440v1 Announce Type: new Abstract: Repeatedly solving parametric PDEs is essential for uncertainty quantification, design optimization and inverse problems, but conventional neural operators require expensive non-convex training.

By Zirui Deng, Jingbo Sun, Deyu Meng, Fei Wang

benchmarks

arXiv AI

Jun 30

Evidence-Driven LLM Agent for C-to-Synthesizable-C Conversion and Verification

arXiv:2606. 28409v1 Announce Type: cross Abstract: Software-compilable C programs routinely fail to complete the four-stage pipeline of a high-level synthesis (HLS) toolchain -- compilation, C simulation (CSim), synthesis, and C/RTL co-simulation (CoSim) -- because HLS accepts only a synthesizable subset of C (HLS-C).

By Zhe Zhao, Hongbing Lang, Zhihan Xiao, Luke Ztz Hu, John Imoleayo Adebisi, Songping Mai

llms rag agents benchmarks

arXiv AI

Jun 30

Deterministic Decisions for High-Stakes AI. A Zero-Egress Pipeline with the Deployability of RAG and the Accuracy of Machine Learning

arXiv:2606. 29280v1 Announce Type: cross Abstract: We identify intervention bias as a previously unquantified failure mode of zero-shot large-language-model (LLM) educational advisory agents: without task-specific training, they recommend action when a hindsight-optimal oracle policy mandates inaction.

By Craig Atkinson

llms rag agents efficiency benchmarks safety data-engineering

arXiv AI

Jun 30

MedDiffuseMix: Preserving Diagnostic Evidence with Saliency-Aware Diffusion Medical Image Data Augmentatio

arXiv:2606. 28419v1 Announce Type: cross Abstract: Limited data availability, class imbalance, and domain variability remain major barriers to reliable medical image classification.

By Teerath Kumar, Raja Vavekanand, Muhammad Turab

llms diffusion computer-vision benchmarks

arXiv AI

Jun 30

TAR: Temporal Anchor-Constrained Reasoning for Video Temporal Grounding

arXiv:2508. 07683v2 Announce Type: replace-cross Abstract: Video Temporal Grounding (VTG) aims to localize specific video segments corresponding to natural language queries.

By Chaohong Guo, Xun Mo, Yongwei Nie, Fei Ma, Xuemiao Xu, Chengjiang Long

llms agents reinforcement-learning multimodal benchmarks safety

arXiv Machine Learning

Jun 30

Improved Predictive Performance and Interpretability for Mesomorphic Neural Networks Using Local Fidelity Regularization

arXiv:2606. 29951v1 Announce Type: new Abstract: Interpretable Mesomorphic Neural Networks (IMNs) offer a promising framework that combines the predictive power of deep neural networks with the interpretability of linear models.

By Hugo L. Hammer, Vajira Thambawita, Kristoffer Herland Hellton, P{\aa}l Halvorsen

benchmarks safety

arXiv Machine Learning

Jun 30

ANVIL: Anomaly-based Vulnerability Identification without Labelled Training Data

arXiv:2408. 16028v4 Announce Type: replace-cross Abstract: Supervised-learning-based vulnerability detectors often fall short due to limited labelled training data.

By Weizhou Wang, Eric Liu, Xiangyu Guo, Xiao Hu, Ilya Grishchenko, David Lie

llms benchmarks

arXiv AI

Jun 30

GPTNT: Benchmarking Real-Time Collaboration Between Multimodal Agents on Keep Talking And Nobody Explodes

arXiv:2606. 28514v1 Announce Type: new Abstract: Multimodal models are increasingly deployed to solve tasks collaboratively with humans or other artificial agents.

By Amit Parekh, Sabrina McCallum, Kareem Al-Hasan, Malvina Nikandrou, Alessandro Suglia, Ioannis Konstas

agents multimodal benchmarks

arXiv AI

Jun 30

SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Models

arXiv:2606. 29520v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly used as assistants across the software development lifecycle, yet their ability to reason about software architecture remains largely unmeasured.

By Tiziano Santilli, Francesco Daghero, Mayhar Tourchi Moghaddam

llms benchmarks

arXiv AI

Jun 30

ARKD: Adaptive Reinforcement Learning-Guided Bidirectional KL Divergence Distillation for Text Generation

arXiv:2606. 29869v1 Announce Type: cross Abstract: Knowledge distillation (KD) is a key technique for compressing Large Language Models (LLMs), yet methods relying on a single KL objective often fail to balance primary distribution fitting with long-tail probability modeling, limiting both generation quality and generalization.

By Zilong Liu, Xuewen Zhang, Jinrui Xing, Juyi Qiao, Huiyong Wang, Junming Jiao

llms reinforcement-learning efficiency benchmarks safety

arXiv AI

Jun 30

Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries

arXiv:2606. 28960v1 Announce Type: new Abstract: Physicians now pose millions of clinical questions to AI tools each week, yet these tools are evaluated largely on hypothetical or exam-style questions, not those actually asked in practice.

By Jean Feng, Vishal Patel, Patrick Heagerty, Yifan Mai, Venkatesh Sivaraman, Patrick Vossler, Jialin Ouyang, Anupam B. Jena

llms benchmarks

arXiv AI

Jun 30

Grounding Sim-to-Real Generalization in Robotic Manipulation: An Empirical Study with Vision-Language-Action Models

arXiv:2603. 22876v2 Announce Type: replace-cross Abstract: Learning a generalist control policy for robotic manipulation typically relies on large-scale datasets.

By Ruixing Jin, Zicheng Zhu, Ruixiang Ouyang, Sheng Xu, Bo Yue, Zhizheng Wu, Guiliang Liu

reinforcement-learning robotics multimodal benchmarks

arXiv AI

Jun 30

MedCase-Structured: A Text-to-FHIR Dataset for Benchmarking Diagnostic Reasoning in Clinically Realistic EHR Settings

arXiv:2605. 30295v2 Announce Type: replace-cross Abstract: Large language models (LLMs) show promise for clinical reasoning and decision support, but evaluation in realistic, electronic health record-congruent settings remains limited.

By Valentina Bui Muti, Eug\'enie Dulout, Ziquan Fu

llms benchmarks

arXiv Machine Learning

Jun 30

Spatial Deconfounder: Interference-Aware Deconfounding for Spatial Causal Inference

arXiv:2510. 08762v2 Announce Type: replace Abstract: Causal inference in spatial domains faces two intertwined challenges: (1) unmeasured spatial factors, such as weather, air pollution, or mobility, that confound treatment and outcome, and (2) interference from nearby treatments that violate standard no-interference assumptions.

By Ayush Khot, Miruna Oprescu, Maresa Schr\"oder, Ai Kagawa, Xihaier Luo

benchmarks

arXiv AI

Jun 30

MESA: Prioritizing Vulnerable Communication Channels for Securing Multi-Agent Systems

arXiv:2606. 30602v1 Announce Type: cross Abstract: Multi-agent systems (MAS) are increasingly used to automate complex, distributed workflows.

By Kunyang Li, Kyle Domico, Jonathan Gregory, Patrick McDaniel

llms agents benchmarks

arXiv AI

Jun 30

IMCBench: A benchmark for multimodal LLMs in Image-grounded Medical Conversations

arXiv:2606. 28556v1 Announce Type: new Abstract: Recent advances in large language models and vision-language models have enabled reasoning over multimodal data, offering opportunities for clinical applications such as decision support and triaging.

By Maria Xenochristou, Ashutosh Joshi, Korosh Vatanparvar, Mohammad Abuzar Hashemi, Prasad Kasu, Deepak Bansal, Anchal Nema, Nivedita Wadhwa, Prashams S Jain, Rebecca Abraham, Will Kimbrough, Dilek Hakkani-Tur, Wilko Schulz-Mahlendorf

llms multimodal benchmarks safety

arXiv AI

Jun 30

T3R: Deeper Test-Time Adaptation for Graph Neural Networks via Gradient Rotation

arXiv:2606. 30011v1 Announce Type: cross Abstract: Graph Neural Networks (GNNs) deployed in real-world systems typically have fixed weights, often leading to degraded performance under distribution shifts.

By Huy Truong, Alexander Lazovik, Victoria Degeler

fine-tuning benchmarks

arXiv AI

Jun 30

Ensemble Learning Based Classification Algorithm Recommendation

arXiv:2101. 05993v2 Announce Type: replace-cross Abstract: Selecting an appropriate classification algorithm for a given data set remains a challenging problem in data mining and machine learning.

By Guangtao Wang, Qinbao Song, Xiaoyan Zhu, Jiao Liu

benchmarks

Get the daily briefing