Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

19,504 stories · RSS feed

arXiv AI

Jun 26

TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference

arXiv:2606. 27161v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have achieved strong multimodal reasoning capabilities, but their efficiency is limited by the large number of visual tokens, which introduces substantial computational overhead.

By Tinghao Wang, Yichen Guo, Rui Huang, Zheng Lu, Qizhe Zhang, Chenxi Li, Yuan Zhang, Jiajun Cao, Zhirong Shen, Yaosong Du, Guangyan Gan, Wenya Wang, Lin William Cong, Shanghang Zhang

llms efficiency multimodal benchmarks safety

arXiv Machine Learning

Jun 26

Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Models

arXiv:2510. 09976v2 Announce Type: replace Abstract: Vision-Language-Action (VLA) models such as OpenVLA, Octo, and $\pi_0$ have shown strong generalization by leveraging large-scale demonstrations, yet their performance is still fundamentally constrained by the quality and coverage of supervised data.

By Mingyang Lyu, Yinqian Sun, Erliang Lin, Huangrui Li, Ruolin Chen, Feifei Zhao, Yi Zeng

diffusion reinforcement-learning fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 26

DualEval: Joint Model-Item Calibration for Unified LLM Evaluation

arXiv:2606. 26429v1 Announce Type: new Abstract: Current LLM evaluation relies on two complementary but often disconnected signals: static benchmarks with objective correctness labels and arena-style preference data that better reflect open-ended user interactions.

By Aaron J. Li, Hao Huang, Youngmin Park, Yitong Ma, Wei-Lin Chiang, Li Chen, Cho-Jui Hsieh, Bin Yu, Ion Stoica

llms benchmarks

arXiv Machine Learning

Jun 26

ProtoKV: Streaming Video Understanding under Delayed Query with Summary-State Memory

arXiv:2606. 26762v1 Announce Type: cross Abstract: Streaming video understanding (SVU) must answer queries that arrive asynchronously while visual tokens stream continuously under strict GPU-memory and query-time latency budgets.

By Le Tu Ngoc Minh (KAIST), Jinyeong Lim (KAIST), Dongsu Han (KAIST)

efficiency benchmarks

arXiv AI

Jun 26

How Do Tool-Augmented LLM Agents Perform on Real-World Energy Analytics Tasks?

arXiv:2606. 26346v1 Announce Type: new Abstract: Agentic benchmarks have emerged across general-purpose and domain-specific settings, including finance, coding, law, and drug discovery, yet energy-domain evaluations remain largely limited to static knowledge recall.

By David Akinpelu, Akintonde Abbas, Rereloluwa Alimi, Ayodeji Lana

llms rag agents benchmarks safety

arXiv AI

Jun 26

Vulnerability of Natural Language Classifiers to Evolutionary Generated Adversarial Text

arXiv:2606. 27215v1 Announce Type: new Abstract: Deep learning models have achieved impressive performance across various fields but remain vulnerable to adversarial inputs, particularly in NLP, where such attacks can have significant real-world consequences.

By Manjinder Singh, Alexander E. I. Brownlee, Mohamed Elawady

llms rag nlp benchmarks safety

arXiv Machine Learning

Jun 26

Reproducibility Study of "AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models"

arXiv:2606. 26783v1 Announce Type: new Abstract: Fang et al.

By Ananth K S, Arya Hariharan

llms benchmarks safety

arXiv AI

Jun 26

ConflictScore: Identifying and Measuring How Language Models Handle Conflicting Evidence

arXiv:2606. 26437v1 Announce Type: cross Abstract: Existing metrics for factuality and faithfulness evaluate whether an answer is supported or contradicted by its grounding documents, but they fail to capture when both supporting and contradicting evidence coexist.

By Siyi Liu, Aaron Halfaker, Dan Roth, Patrick Xia

llms benchmarks

arXiv AI

Jun 26

Hierarchical Fault Detection and Diagnosis for Transformer Architectures

arXiv:2604. 28118v2 Announce Type: replace-cross Abstract: Transformers now underpin critical AI systems across industry and research.

By Sigma Jahan, Saurabh Singh Rajput, Tushar Sharma, Mohammad Masudur Rahman

llms benchmarks

arXiv Machine Learning

Jun 26

Clue-Guided Money Laundering Group Discovery

arXiv:2606. 26189v1 Announce Type: new Abstract: Money Laundering Group Discovery (MLGD) aims to identify hidden criminal groups and recover their complete structures in large-scale financial networks.

By Boyang Wang, Jianing Cao

benchmarks

arXiv AI

Jun 26

NebulaExp-8B: An Empirical Post-Training Pipeline via Full-Scale Ablation Research

arXiv:2606. 26671v1 Announce Type: new Abstract: Post-training alignment determines the reasoning and human preference following capabilities of large language models, yet most existing works withhold detailed data construction, filtering rules and training recipes, which hinders community reproducibility and lightweight model optimization.

By Qiaobo Hao, Yangqian Wu, Shunyi Wang, Zhongjian Zhang, Ziqun Li, Yayin He, Muqing Li, Chen Zhong

llms reinforcement-learning fine-tuning efficiency benchmarks safety

arXiv AI

Jun 26

Speaking Numbers to LLMs: Multi-Wavelet Number Embeddings for Time Series Forecasting

arXiv:2606. 26487v1 Announce Type: cross Abstract: Large language models (LLMs) are attractive for context-aware time series forecasting because they can integrate heterogeneous textual signals, yet their discrete, language-oriented tokenization and embedding interfaces are misaligned with continuous numerical values, often harming numerical ordering and forecasting reliability.

By Defu Cao, Zijie Lei, Muyan Weng, Jiao Sun, Yan Liu

llms rag nlp benchmarks

arXiv Machine Learning

Jun 26

Implementation of reinforcement learning in chemical reaction networks: application to phototaxis as curiosity-driven exploration

arXiv:2606. 26168v1 Announce Type: new Abstract: Living systems navigate environments using noisy and incomplete sensory signals.

By Ruyi Tang (LCQB-AG), Gr\'egoire Sergeant-Perthuis (LCQB-AG), David Colliaux

reinforcement-learning benchmarks safety

arXiv AI

Jun 26

Symbolic Reasoning Frameworks Trigger Memory-Mediated Ecosystem Dynamics in Multi-Agent LLM Systems

arXiv:2606. 07552v2 Announce Type: replace-cross Abstract: Large language models exhibit a risk-averse "turtle" bias as strategic agents.

By Augustin Chan

llms agents benchmarks safety

arXiv Machine Learning

Jun 26

EMA-FS: Accelerating GBDT Training via Gain-Informed Feature Screening

arXiv:2606. 26337v1 Announce Type: new Abstract: Gradient Boosted Decision Trees (GBDT), exemplified by LightGBM, spend a dominant fraction of training time -- typically 65-70% -- constructing per-feature histograms.

By Yan Song

benchmarks

arXiv Machine Learning

Jun 26

Optimizing CUDA like a Human: Micro-Profiling Tools as Expert Surrogates for LLM-Based GPU Kernel Optimization

arXiv:2606. 26453v1 Announce Type: new Abstract: We present KernelPro, a closed-loop multi-agent system that automatically generates, profiles, and iteratively optimizes GPU kernel code by integrating large language model (LLM) code generation with hardware profiler feedback and pluggable bottleneck detection tools.

By Jiading Gai, Shuai Zhang, Kaj Bostrom, Jin Huang, Vihang Patil, Haoyang Fang, Bernie Wang, Huzefa Rangwala, George Karypis

llms agents efficiency benchmarks

arXiv Machine Learning

Jun 26

Theory-Scale Auto-Formalization of Logics for Computer Science

arXiv:2606. 26525v1 Announce Type: new Abstract: Auto-formalization is critical for scalable formal verification, but existing progress largely focuses on isolated statements, while theory-scale auto-formalization, which coherently translates hundreds of interdependent definitions, lemmas, and theorems, remains open due to challenges in consistency, faithfulness, scalability, and correctness.

By Yuming Feng, Frederick Pu, One An, Osbert Bastani, Li Zhang, Jiani Huang, Xujie Si, Ziyang Li

agents benchmarks

arXiv Machine Learning

Jun 26

Can Large Language Models Reliably Code Qualitative Humanitarian Data? A Benchmark Study Against Human Expert Adjudication

arXiv:2606. 26541v1 Announce Type: new Abstract: Data from affected populations are crucial for informing humanitarian response, but their value depends on timely and consistent interpretation of nuanced accounts of need.

By Jerome Marston, Tino Kreutzer, Salom\'e Garnier, Ella Boone, Phuong N Pham, Patrick Vinck

llms benchmarks safety

arXiv Machine Learning

Jun 26

HyperDFlash: MHC-Aligned Block Speculative Decoding with Gated Residual Reduction

arXiv:2606. 26744v1 Announce Type: new Abstract: We present HyperDFlash, a block-parallel speculative decoding framework tailored to the novel multi-hyper-connection (MHC) architecture proposed by DeepSeek-V4.

By Luxi Lin, Shuang Peng, Rui Ma, Junhao Hua, Shuwei Fan, Zhengda Qin, Qiang Wang, Hongjian Sun, Fangmin Chen, Songwei Liu

efficiency benchmarks safety

arXiv Machine Learning

Jun 26

Probabilistic NDVI Forecasting from Sparse Satellite Time Series and Weather Covariates

arXiv:2602. 17683v3 Announce Type: replace Abstract: Short-term forecasting of vegetation dynamics is a key enabler for data-driven decision support in precision agriculture.

By Irene Iele, Giulia Romoli, Daniele Molino, Elena Mulero Ayll\'on, Filippo Ruffini, Paolo Soda, Matteo Tortora

multimodal benchmarks

Get the daily briefing