Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

17,521 stories · RSS feed

arXiv AI

Jul 8

DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression

arXiv:2607. 06523v1 Announce Type: new Abstract: Long-context language model inference is increasingly limited by the memory bandwidth and capacity required to store key-value caches, yet existing compression methods often apply uniform budgets across layers or tokens and degrade retrieval when lexical cues and semantic states require different preservation.

By Anna Cordoba, Adam Puente Tercero, Nerea Angulo Hijo, Mar Linares Tercero, Julia Barrientos, Ainhoa Miranda, Jesus Olivera

llms nlp efficiency benchmarks

arXiv AI

Jul 8

An Experimental Design Approach to Evaluating Agentic AI's Autonomous Model Discovery

arXiv:2607. 06413v1 Announce Type: cross Abstract: Large language model coding agents increasingly perform open-ended data modeling and analysis.

By Hao He, Xueying Liu, Chris J. Kuhlman, Xinwei Deng

llms agents benchmarks

arXiv AI

Jul 8

RPAM: A Principled Metric for Evaluating Associations in Language Models with High Predictive Validity in Downstream Outputs

arXiv:2607. 05679v1 Announce Type: cross Abstract: Language models (LMs) exhibit problematic biases, such as stereotypes.

By Damian Hodel, Jevin West, Aylin Caliskan

llms rag benchmarks

arXiv AI

Jul 8

AbICL: In-Context Learning for Antigen-Specific Antibody Affinity Ranking

arXiv:2607. 05846v1 Announce Type: cross Abstract: Accurate ranking of antibody candidates according to their binding affinity is essential for therapeutic antibody discovery.

By Zhiyuan Chen, Jing Hu, Junzhe Wang, Yueyang Huang, Xinyi Yang, Zhaoyang Wang, Feng Zhu

llms benchmarks

arXiv Machine Learning

Jul 8

Level-Crossing Density as a Mesh-Free High-Frequency Auxiliary Loss for Implicit Neural Representations

arXiv:2607. 05815v1 Announce Type: new Abstract: The Minkowski functionals of a field's excursion sets -- area, boundary measure, and Euler characteristic -- describe its level-set morphology; the Euler characteristic is the cheapest handle on topology.

By Gunner Levi Howe

benchmarks

arXiv AI

Jul 8

Integrating knowledge graphs and multilingual scholarly corpora for domain-adaptive LLMs in SSH

arXiv:2607. 05956v1 Announce Type: new Abstract: The integration of Large Language Models (LLMs) into scientific research workflows, particularly for bibliographic discovery and literature synthesis, raises significant methodological, epistemic and regulatory challenges for the Social Sciences and Humanities (SSH), especially with regard to disciplinary diversity, multilingual access to sources and the evaluation of results.

By Adam Faci, Alessio Miaschi, Anne Combe, Pascal Cuxac, Francesca Frontini, Nicolas Larrousse, St\'ephane Pouyllau

llms rag nlp benchmarks safety

arXiv Machine Learning

Jul 8

Mitigating Factual Hallucination in Large Reasoning Models via Mixed-Mode Advantage Regularization

arXiv:2607. 05861v1 Announce Type: cross Abstract: Large reasoning models (LRMs) improve language model capabilities by generating explicit thinking traces before final answers.

By Kaishen Wang, Tong Zheng, Xuehao Cui, Ruibo Chen, Tianyi Xiong, Heng Huang

llms nlp reinforcement-learning benchmarks safety

arXiv Machine Learning

Jul 8

Deep Reinforcement Learning for Dynamic Battery Management of Autonomous Order Pickers

arXiv:2607. 05683v1 Announce Type: new Abstract: Battery charging of Autonomous Mobile Robots (AMRs) in warehouses is a critical operational challenge that heavily impacts both order processing times and throughput.

By Taniya Shaji, Abhay Sobhanan, Christof Defryn

agents reinforcement-learning robotics efficiency benchmarks

arXiv AI

Jul 8

From Passive Observer to Active Critic: Reinforcement Learning Elicits Process Reasoning for Robotic Manipulation

arXiv:2603. 15600v2 Announce Type: replace-cross Abstract: Accurate process supervision remains a critical challenge for long-horizon robotic manipulation.

By Yibin Liu, Yaxing Lyu, Daqi Gao, Zhixuan Liang, Weiliang Tang, Shilong Mu, Xiaokang Yang, Yao Mu

reinforcement-learning robotics fine-tuning benchmarks

arXiv AI

Jul 8

Token-Based Dual-view Fusion and Adaptation of Large Vision Models for Breast Cancer Classification

arXiv:2607. 06309v1 Announce Type: cross Abstract: Accurate breast cancer classification from mammography requires effective integration of complementary information from craniocaudal (CC) and mediolateral oblique (MLO) views, which provide a more complete characterization of breast abnormalities.

By Aysan Ghayouri Pirsoltan, Shima Babakordi, Mohammad Reza Mohammadi

llms computer-vision benchmarks

arXiv Machine Learning

Jul 8

Exogenous Dropout: A Simple, Strong Baseline for Corruption-Robust Time Series Forecasting with Covariates

arXiv:2607. 05452v1 Announce Type: new Abstract: Time series forecasters that use exogenous covariates are fragile in deployment: when those covariates are noised, temporally misaligned, or missing, strong exogenous-fusion and exogenous-adapted models can degrade far above the endogenous-only floor.

By Hao Hu, Xue-shan Ai

benchmarks

arXiv AI

Jul 8

FirstResearch: Auditable Question Formation for LLM Scientific Discovery Agents

arXiv:2607. 05682v1 Announce Type: new Abstract: LLM systems for scientific discovery increasingly assist with ideation, literature synthesis, experiment planning, and report generation, but the first research question they propose can remain difficult to audit: it may sound plausible without exposing the mechanism, falsifier, or assumption that a scientist should inspect.

By Yufeng Wang

llms agents benchmarks

arXiv Machine Learning

Jul 8

LLM-Driven Neural Network Generation with Same-Family Architecture Guidance: Disentangling Transfer and Adaptation

arXiv:2607. 05704v1 Announce Type: new Abstract: Large language models (LLMs) can generate neural-network modifications, but unrestricted generation is often invalid or harmful.

By Kabir Dev Paul Baghel, Radu Timofte, Dmitry Ignatov

llms benchmarks

arXiv AI

Jul 8

The Balkanization of Execution-Security Research for AI Coding Agents: Isolation, Access Control, and Time-of-Check-to-Time-of-Use Vulnerabilities

arXiv:2607. 05743v1 Announce Type: cross Abstract: AI coding agents now read repositories, call tools, and execute shell commands with limited human oversight, and a fast-growing body of work studies whether the execution layer around them is actually safe.

By Mohammadreza Rashidi

llms agents benchmarks safety

arXiv AI

Jul 8

PVCap: Towards Accurate 3D Dense Captioning via PseudoCap and VoxelCapNet

arXiv:2607. 06097v1 Announce Type: cross Abstract: 3D dense captioning, an emerging vision-language task, aims to generate descriptive sentences for each object in the 3D scene.

By Xiaopei Wu, Chenshu Hou, Liang Peng, Dan Xu, Binbin Lin, Xiaoshui Huang, Yuenan Hou, Yu Li, Wenxiao Wang, Haifeng Liu, Deng Cai, Wanli Ouyang

multimodal benchmarks

arXiv AI

Jul 8

CMDR: Contextual Multimodal Document Retrieval

arXiv:2607. 05927v1 Announce Type: cross Abstract: Multimodal document retrieval aims to retrieve relevant pages while preserving both textual and visual content from the original document.

By Ryota Tanaka, Taku Hasegawa, Kyosuke Nishida

rag multimodal benchmarks

arXiv AI

Jul 8

CSTutorBench: Benchmarking Small Language Models as Tutors for Block-Based Programming

arXiv:2607. 05571v1 Announce Type: new Abstract: Large language models are increasingly explored as AI tutors, yet deploying them in K-12 settings raises concerns around privacy, cost, and reliance on proprietary models.

By H. Chad Lane, Bryson Kageler

llms robotics benchmarks safety

arXiv AI

Jul 8

Multi-Task Instruction Tuning via Data Scheduling for Low-Resource Arabic SpeechLLMs

arXiv:2601. 12494v3 Announce Type: replace-cross Abstract: Audio large language models (LLMs) enable unified speech understanding and generation, but adapting them to linguistically complex and dialect-rich settings such as Arabic-English remains challenging.

By Hunzalah Hassan Bhatti, Firoj Alam, Shammur Absar Chowdhury

llms nlp fine-tuning multimodal benchmarks

arXiv Machine Learning

Jul 8

Channel-wise Retrieval for Multivariate Time Series Forecasting

arXiv:2604. 05543v2 Announce Type: replace Abstract: Multivariate time series forecasting often struggles to capture long-range dependencies due to fixed lookback windows.

By Junhyeok Kang, Jun Seo, Soyeon Park, Sangjun Han, Seohui Bae, Hyeokjun Choe, Soonyoung Lee

rag benchmarks

arXiv AI

Jul 8

From Global to Granular: Revealing IQA Model Performance via Correlation Surface

arXiv:2601. 21738v2 Announce Type: replace-cross Abstract: Evaluation of Image Quality Assessment (IQA) models has long been dominated by global correlation metrics, such as Pearson Linear Correlation Coefficient (PLCC) and Spearman Rank-Order Correlation Coefficient (SRCC).

By Baoliang Chen, Danni Huang, Hanwei Zhu, Lingyu Zhu, Wei Zhou, Shiqi Wang, Yuming Fang, Weisi Lin

benchmarks

Get the daily briefing