Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

19,246 stories · RSS feed

arXiv AI

Jun 30

Mandol: An Agglomerative Agent Memory System for Long-Term Conversations

arXiv:2606. 29778v1 Announce Type: cross Abstract: Long-term conversational agents need to remember and query cross-session, multi-typed information with complex correlations.

By Yuhan Zhang (Institute of Software, Chinese Academy of Sciences), Zhiyuan Guo (Institute of Software, Chinese Academy of Sciences), Ziheng Zeng (Institute of Software, Chinese Academy of Sciences), Wei Wang (Institute of Software, Chinese Academy of Sciences), Wentao Wu (Microsoft Research), Lijie Xu (Institute of Software, Chinese Academy of Sciences)

llms rag agents benchmarks

arXiv AI

Jun 30

ComMem: Complementary Memory Systems for Test-Time Adaptation of Vision-Language Models

arXiv:2606. 28719v1 Announce Type: new Abstract: Test-time adaptation (TTA) of vision-language models (VLMs) is essential for their robust deployment in dynamic, real-world environments.

By Guanglong Sun, Shuang Cui, Bo Lei, Liyuan Wang, Zihan Zhai, Hongwei Yan, Hang Su, Jun Zhu, Yi Zhong

llms multimodal benchmarks

arXiv AI

Jun 30

AI Trading's Alpha Singularity: Emergent Market Reasoning through Agent-to-Agent Self-Evolution

arXiv:2606. 29194v1 Announce Type: new Abstract: Automated alpha mining holds the scoring function fixed and varies the search algorithm over it.

By Yuqi Li, Siyuan Liu, Bingjun Liu

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Jun 30

HieraMix: A Hierarchical MLP-Mixer for Large-Scale Traffic Forecasting

arXiv:2512. 07854v2 Announce Type: replace Abstract: Traffic forecasting task is significant to modern urban management.

By Yongyao Wang, Xie Yu, Jingyuan Wang, Jiahao Ji, Chao Li

benchmarks

arXiv AI

Jun 30

ARKD: Adaptive Reinforcement Learning-Guided Bidirectional KL Divergence Distillation for Text Generation

arXiv:2606. 29869v1 Announce Type: cross Abstract: Knowledge distillation (KD) is a key technique for compressing Large Language Models (LLMs), yet methods relying on a single KL objective often fail to balance primary distribution fitting with long-tail probability modeling, limiting both generation quality and generalization.

By Zilong Liu, Xuewen Zhang, Jinrui Xing, Juyi Qiao, Huiyong Wang, Junming Jiao

llms reinforcement-learning efficiency benchmarks safety

arXiv AI

Jun 30

T3R: Deeper Test-Time Adaptation for Graph Neural Networks via Gradient Rotation

arXiv:2606. 30011v1 Announce Type: cross Abstract: Graph Neural Networks (GNNs) deployed in real-world systems typically have fixed weights, often leading to degraded performance under distribution shifts.

By Huy Truong, Alexander Lazovik, Victoria Degeler

fine-tuning benchmarks

arXiv AI

Jun 30

Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries

arXiv:2606. 28960v1 Announce Type: new Abstract: Physicians now pose millions of clinical questions to AI tools each week, yet these tools are evaluated largely on hypothetical or exam-style questions, not those actually asked in practice.

By Jean Feng, Vishal Patel, Patrick Heagerty, Yifan Mai, Venkatesh Sivaraman, Patrick Vossler, Jialin Ouyang, Anupam B. Jena

llms benchmarks

arXiv AI

Jun 30

Multimodal Representation Alignment for Cross-modal Information Retrieval

arXiv:2506. 08774v2 Announce Type: replace-cross Abstract: Different machine learning models can represent the same underlying concept in different ways.

By Fan Xu, Luis A. Leiva

llms rag multimodal benchmarks safety

arXiv AI

Jun 30

Search for Truth from Reasoning: A Dynamic Representation Editing Framework for Steering LLM Trajectories

arXiv:2606. 28589v1 Announce Type: new Abstract: Current approaches to enhance Large Language Model (LLM) reasoning, such as Chain-of-Thought and "Wait" prompts, primarily encourage models to think more, yet often fail to guide them toward Truth.

By Tianlong Wang, Yuhang Wang, Weibin Liao, Xin Gao, Xinyu Ma, Yang Lin, Yasha Wang, Liantao Ma

llms benchmarks

arXiv Machine Learning

Jun 30

NeuReasoner: Theory-grounded Mapping of Reasoning Elicitation Boundaries

arXiv:2606. 29971v1 Announce Type: new Abstract: A growing body of work suggests that the reasoning capabilities of large language models are largely latent in their base form, with post-training primarily amplifying rather than introducing them.

By Aydin Javadov, Shyngys Aitkazinov, Tobias Hoesli, Florian von Wangenheim, Bjoern Schuller, Joseph Ollier

llms benchmarks

arXiv AI

Jun 30

EvalSafetyGap: A Hybrid Survey and Conceptual Framework for LLM Evaluation-Safety Failures

arXiv:2606. 30219v1 Announce Type: new Abstract: LLM evaluation and AI safety face a shared measurement problem: benchmark scores, reward-model signals, and reported safety metrics can improve while the latent properties they are meant to represent remain difficult to verify.

By Bu\u{g}ra Alperen Ulu{\i}rmak, Rifat Kurban

llms benchmarks safety

arXiv Machine Learning

Jun 30

CoLR-Det: Collaborative Latent Restoration for Small Object Detection in Low-Resolution Remote Sensing Images

arXiv:2601. 12507v2 Announce Type: replace-cross Abstract: Low-resolution remote sensing small object detection is limited by both missing visual details and the ambiguity of how details serve detection.

By Ruo Qi, Linhui Dai, Yusong Qin, Chaolei Yang, Yanshan Li

computer-vision benchmarks

arXiv Machine Learning

Jun 30

Friend or Foe

arXiv:2509. 00123v2 Announce Type: replace-cross Abstract: A fundamental challenge in microbial ecology is determining whether bacteria compete or cooperate in different environmental conditions.

By Oleksandr Cherednichenko, Josephine Solowiej-Wedderburn, Laura M. Carroll, Eric Libby

benchmarks

arXiv Machine Learning

Jun 30

Synthetic Interaction Data for Scalable Personalization in Large Language Models

arXiv:2602. 12394v2 Announce Type: replace Abstract: Personalized prompting offers large opportunities for deploying large language models (LLMs) to diverse users, yet existing prompt optimization methods primarily focus on task-level optimization while largely overlooking user-specific preferences and latent constraints of individual users.

By Yuchen Ma, Yue Huang, Wenjie Wang, Xiaonan Luo, Xiangliang Zhang, Stefan Feuerriegel

llms agents reinforcement-learning benchmarks safety

arXiv Machine Learning

Jun 30

Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation

arXiv:2602. 09305v2 Announce Type: replace Abstract: Large Language Models (LLMs) demonstrate transformative potential, yet their reasoning remains inconsistent and unreliable.

By Pei-Chi Pan, Yingbin Liang, Sen Lin

llms reinforcement-learning fine-tuning benchmarks safety

arXiv Machine Learning

Jun 30

Spectral Embedding via Chebyshev Bases for Robust DeepONet Approximation

arXiv:2512. 09165v2 Announce Type: replace Abstract: Deep Operator Networks (DeepONets) have emerged as a powerful framework for data-driven operator learning, providing flexible surrogates for nonlinear mappings arising in partial differential equations (PDEs).

By Muhammad Abid, Omer San

rag diffusion benchmarks safety

arXiv Machine Learning

Jun 30

Attention Enhanced Entity Recommendation for Intelligent Monitoring in Cloud Systems

arXiv:2510. 20640v2 Announce Type: replace Abstract: In this paper, we present DiRecGNN, an attention-enhanced entity recommendation framework for monitoring cloud services at Microsoft.

By Fiza Husain, Anson Bastos, Anjaly Parayil, Ayush Choure, Chetan Bansal, Rujia Wang, Saravan Rajmohan

llms efficiency benchmarks

arXiv Machine Learning

Jun 30

ML-Powered LDAP Reconnaissance Detection using Weak Supervision

arXiv:2606. 28917v1 Announce Type: new Abstract: Lightweight Directory Access Protocol (LDAP) is a protocol that allows users to query and modify Active Directory (AD) data.

By Shaefer Drew, Edward Raff, Michael Brautbar, Yaron Zinar, Benjamin Malmberg, Dor Agron, Sagi Sheinfeld, Avraham Kama, Asaf Romano

benchmarks

arXiv Machine Learning

Jun 30

Active Quantum Kernel Acquisition for Gaussian Process Regression

arXiv:2606. 28833v1 Announce Type: new Abstract: Quantum kernel estimation on near-term hardware is shot-budgeted: every entry of the kernel Gram matrix is a Bernoulli expectation that must be sampled with a finite number of circuit executions.

By Jian Xu, Delu Zeng, Qibin Zhao

benchmarks

arXiv Machine Learning

Jun 30

The Contagion Tensor: A Framework for Measuring Output-Distribution Coupling in Multi-Agent LLM Systems -- and Auditing the Claims It Enables

arXiv:2606. 28839v1 Announce Type: new Abstract: We introduce the Contagion Tensor, a measurement framework for quantifying how large language model (LLM) output distributions couple across modalities, agents, and time steps.

By Zewen Liu

llms agents benchmarks

Get the daily briefing