Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

23,305 stories · RSS feed

arXiv AI

Jun 4

Geometry-Aware Hallucination Detection in Large Language Models

arXiv:2601. 06196v3 Announce Type: replace-cross Abstract: Large language models (LLMs) frequently generate factually incorrect or unsupported content, commonly referred to as hallucinations.

By Bodla Krishna Vamshi, Rohan Bhatnagar, Haizhao Yang

llms rag nlp fine-tuning benchmarks safety

arXiv AI

Jun 4

Can VLMs Predict Future States? Bootstrapping World Models from Inverse Dynamics

arXiv:2506. 06006v3 Announce Type: replace-cross Abstract: Can unified vision-language models (VLMs) perform forward dynamics prediction (FDP), i.

By Yifu Qiu, Yftah Ziser, Anna Korhonen, Shay B. Cohen, Edoardo M. Ponti

llms fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 4

LazyAttention: Efficient Retrieval-Augmented Generation with Deferred Positional Encoding

arXiv:2606. 04302v1 Announce Type: cross Abstract: Key-value (KV) caching accelerates inference of large language models (LLMs) by reusing past computations for generated tokens.

By Haocheng Xia, Mihir Pamnani, Hanxi Fang, Supawit Chockchowwat, Yongjoo Park

llms rag efficiency benchmarks

arXiv AI

Jun 4

SoLoPO: Unlocking Long-Context Capabilities in LLMs via Short-to-Long Preference Optimization

arXiv:2505. 11166v3 Announce Type: replace-cross Abstract: Despite advances in pretraining with extended context sizes, large language models (LLMs) still face challenges in effectively utilizing real-world long-context information, primarily due to insufficient long-context alignment caused by data quality issues, training inefficiencies, and the lack of well-designed optimization objectives.

By Huashan Sun, Shengyi Liao, Yansen Han, Yu Bai, Yang Gao, Cheng Fu, Weizhou Shen, Fanqi Wan, Ming Yan, Ji Zhang, Fei Huang

llms benchmarks safety data-engineering

arXiv AI

Jun 4

Rollout-Level Advantage-Prioritized Experience Replay for GRPO

arXiv:2606. 04560v1 Announce Type: cross Abstract: Reinforcement learning from verifiable rewards with GRPO is a standard approach for post-training reasoning LLMs.

By Gyeongtae Yoo, Sanghyeok Park, Soohyuk Jang, Ik-hwan Kim, Sungroh Yoon

llms reinforcement-learning benchmarks

arXiv AI

Jun 4

The Meta-Agent Challenge: Are Current Agents Capable of Autonomous Agent Development?

arXiv:2606. 04455v1 Announce Type: new Abstract: Current AI benchmarks evaluate agents on task execution within human-designed workflows.

By Xinyu Lu, Tianshu Wang, Pengbo Wang, zujie wen, Zhiqiang Zhang, Jun Zhou, Boxi Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun

agents benchmarks safety

arXiv AI

Jun 4

Characterizing initial human-AI proof formalization workflows

arXiv:2606. 04273v1 Announce Type: new Abstract: For centuries, human mathematicians have written proofs to substantiate their mathematical arguments; yet, the ability to automatically verify the validity of proofs has long been a challenge.

By Katherine M. Collins, Simon Frieder, Jonas Bayer, Jacob Loader, Jeck Lim, Peiyang Song, Fabian Zaiser, Lexin Zhou, Shanda Li, Sam Looi, Joshua B. Tenenbaum, Umang Bhatt, Adrian Weller, Jose Hernandez-Orallo, Cameron E. Freer, Valerie Chen, Ilia Sucholutsky

benchmarks

arXiv AI

Jun 4

Toward Pre-Deployment Assurance for Enterprise AI Agents: Ontology-Grounded Simulation and Trust Certification

arXiv:2606. 04037v1 Announce Type: new Abstract: Pre-deployment verification of enterprise artificial intelligence (AI) agents remains a critical gap between large language model (LLM) capability benchmarking and production deployment.

By Thanh Luong Tuan, Abhijit Sanyal

llms rag agents benchmarks safety

arXiv AI

Jun 4

POLARIS: Guiding Small Models to Write Long Stories

arXiv:2606. 04095v1 Announce Type: cross Abstract: Small open-weight models struggle at long-form creative writing: their generated stories either fall far short of the requested length, or their quality significantly degrades as length increases, especially when compared to frontier models.

By Rishanth Rajendhran, Jenna Russell, Mohit Iyyer, John Frederick Wieting

llms benchmarks

arXiv AI

Jun 4

Ekka: Automated Diagnosis of Silent Errors in LLM Inference

arXiv:2606. 04594v1 Announce Type: cross Abstract: LLM serving frameworks are quickly evolving with a complex software stack and a vast number of optimizations.

By Yile Gu, Zhen Zhang, Shaowei Zhu, Xinwei Fu, Jun Wu, Yida Wang, Baris Kasikci

llms benchmarks

arXiv AI

Jun 4

NoRA: Evaluating Grounded Reasonableness in Visual First-person Normative Action Reasoning

arXiv:2606. 04806v1 Announce Type: cross Abstract: LLMs and agentic systems are increasingly deployed in social environments, making normative competence critical for safe and appropriate behavior.

By Sichao Li, Sai Ma, Daniel Kilov, Secil Yanik Guyot, Zhuang Li, Seth Lazar

llms agents multimodal benchmarks safety

arXiv Machine Learning

Jun 4

RAVQ-HoloNet: Rate-Adaptive Vector-Quantized Hologram Compression

arXiv:2511. 21035v2 Announce Type: replace Abstract: Holography offers significant potential for AR/VR applications.

By Shima Rafiei, Zahra Nabizadeh Shahr-Babak, Soroush Khoubyarian, Alexandre Cooper, Shadrokh Samavi, Shahram Shirani

efficiency benchmarks

arXiv Machine Learning

Jun 4

LMM-IR: Large-Scale Netlist-Aware Multimodal Framework for Static IR-Drop Prediction

arXiv:2511. 12581v2 Announce Type: replace Abstract: Static IR drop analysis is a fundamental and critical task in the field of chip design.

By Kai Ma, Zhen Wang, Hongquan He, Qi Xu, Tinghuan Chen, Hao Geng

llms diffusion multimodal benchmarks

arXiv Machine Learning

Jun 4

Post-Training Corrections for Improved Time-Series Forecasting

arXiv:2505. 15354v3 Announce Type: replace Abstract: Time-series forecasting is a critical task in various business domains, but it remains inherently challenging.

By Hamza Cherkaoui, Malik Tiomoko, Giuseppe Paolo, Zhang Yili, Yu Meng, Zhang Keli, Hafiz Tiomoko Ali

benchmarks

arXiv AI

Jun 4

LoopFM: Learning frOm HistOrical RePresentations of Foundation Model for Recommendation

arXiv:2605. 29280v2 Announce Type: replace-cross Abstract: Knowledge distillation (KD) transfers a single scalar prediction from a large foundation model (FM) to compact vertical models (VMs), suffering from diminishing transfer ratio -- the fraction of FM improvement captured by the VM -- as a single scalar cannot convey the rich intermediate knowledge that larger FMs learn.

By Shali Jiang, Hua Zheng, Boyang Liu, Laming Chen, Kenny Lov, Chuanqi Xu, Lisang Ding, Qinghai Zhou, Can Cui, Xiaolong Liu, Xiaoyi Liu, Yasmine Badr, Xin Xu, Jiyan Yang, Ellie Dingqiao Wen, Gerard Jonathan Mugisha Akkerhuis, Chenxiao Guan, Rong Jin, Ruichao Qiu, Xian Chen, Shifu Xu, Zhehui Zhou, Ping Chen, Rui Yang, Haicheng Chen, Xiangge Meng, Song Zhou, Dharak Kharod, Shuyu Xu, Qiang Jin, Qiao Yang, Wankun Zhu, Qin Huang, Yuzhen Huang, Darren Liu, Parish Aggarwal, Hui Zhou, Erzhuo Wang, Shuo Chang, Xiaorui Gan, Wenlin Chen, Santanu Kolay, Huayu Li

rag efficiency benchmarks

arXiv AI

Jun 4

Metric-Aware Hybrid Forecasting for the CTF4Science Lorenz Challenge

arXiv:2606. 04191v1 Announce Type: cross Abstract: We describe our approach to the CTF4Science Lorenz challenge, a benchmark that mixes short-horizon forecasting, long-time distribution matching, and trajectory reconstruction across nine task pairs.

By Cen Lu

benchmarks

arXiv AI

Jun 4

Value Entanglement: Conflation Between Different Kinds of Good In (Some) Large Language Models

arXiv:2602. 19101v2 Announce Type: replace-cross Abstract: Value alignment of Large Language Models (LLMs) requires us to empirically measure these models' actual, acquired representation of value.

By Seong Hah Cho, Junyi Li, Anna Leshinskaya

llms rag benchmarks safety

arXiv AI

Jun 4

Safety Under Scaffolding: How Evaluation Conditions Shape Measured Safety

arXiv:2603. 10044v2 Announce Type: replace-cross Abstract: A safety score earned on a benchmark need not predict how the same model behaves once it is wrapped in an agentic scaffold the benchmark never tested.

By David Gringras

llms agents benchmarks safety

arXiv Machine Learning

Jun 4

Beyond Symmetric Alignment: Spectral Diagnostics of Modality Imbalance in Vision-Language Models in the Medical Domain

arXiv:2606. 04613v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) struggle when applied to medical image-text data, yet the tools available to diagnose this failure remain limited.

By Alessandro Gambetti, Qiwei Han, Cl\'audia Soares, Hong Shen

llms multimodal benchmarks safety

arXiv Machine Learning

Jun 4

Optimal Transport Flow Matching by Design

arXiv:2606. 04092v1 Announce Type: cross Abstract: Flow matching models learn to transport samples from a simple prior distribution to a complex data distribution.

By Shimon Malnick, Matan Rusanovsky, Ohad Fried, Shai Avidan

diffusion benchmarks safety

Get the daily briefing