Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,257 stories · RSS feed

arXiv AI

Jul 7

When Aggregate Alignment Misleads: Auditing Policy Repair Without Per-State Expert Actions

arXiv:2607. 03386v1 Announce Type: new Abstract: Agentic AI systems are increasingly used to edit, refine, and repair decision policies, but evaluating these edits is difficult when per-state expert action labels are unavailable.

By Peiying Zhu, Sidi Chang

llms agents benchmarks safety

arXiv AI

Jul 7

Evaluating and Understanding Model Editing for Medical Vision Language Models

arXiv:2607. 05310v1 Announce Type: new Abstract: Model editing promises a fast, targeted way to correct post-deployment mistakes in medical vision-language models (VLMs) without costly retraining.

By Guli Zhu, Chenwei Wu, Liyue Shen

llms diffusion multimodal benchmarks

arXiv AI

Jul 7

RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies

arXiv:2607. 04434v1 Announce Type: cross Abstract: Generalist robot manipulation policies have advanced rapidly, yet existing benchmarks remain limited in systematically evaluating their capabilities.

By Tianxing Chen, Yue Chen, Zixuan Li, Junyuan Tang, Kailun Su, Weijie Wan, Baijun Chen, Haoran Lu, Haowen Yan, Honghao Su, Zhiyang Dou, Kaixuan Wang, Dandan Zhang, Yunze Liu, Yan Qin, Qiwei Liang, Qiwei Wu, Zijian Lin, Wenwei Lin, Yuran Wang, Minghua He, Tianshu Wu, Ruihai Wu, Jingquan Zhou, Kai-Chong Lei, Haibao Yu, Yuanfeng Ji, Weiyang Jin, Guanyu Lin, Xiaofan Li, Qi Xiong, Renjing Xu, Zhongyu Li, Wenhao Chai, Enze Xie, Ziwei Wang, Yao Mu, Hao Dong, Wojciech Matusik, Mingyu Ding, Wenbo Ding, Ping Luo, Masayoshi Tomizuka

robotics benchmarks

arXiv AI

Jul 7

Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Models

arXiv:2607. 02914v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated remarkable capabilities across diverse applications, yet ensuring their simultaneous safety, helpfulness, and trustworthiness remains a persistent challenge.

By Jiyang Guan, Yong Xie, Jun Chen, Jiexi Liu, Zipeng Ye, Defeng Li, Jiayu Shen, Jialing Tao, Hui Xue

llms reinforcement-learning fine-tuning benchmarks safety

arXiv AI

Jul 7

Double-Helix Active Geometry: LiDAR-Anchored Multi-View Depth with Selective Abstention

arXiv:2607. 02561v1 Announce Type: cross Abstract: Consumer depth sensors such as the LiDAR scanner on recent iPhones provide metric range, but their useful range is short and their returns are sparse.

By Jinwen Wen

computer-vision benchmarks safety

arXiv AI

Jul 7

Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry

arXiv:2607. 03702v1 Announce Type: new Abstract: Large language model (LLM) agents have shown strong decision-making capabilities in long-horizon interactive tasks, yet they still struggle to effectively leverage failed trajectories: full retries incur high interaction costs, while experience retrieval tends to dilute critical experience signals.

By Weiyang Guo, Zesheng Shi, Longhui Zhang, Zeen Zhu, Min Zhang, Jing Li

llms agents reinforcement-learning benchmarks

arXiv AI

Jul 7

Metronome: Bound the Cache, Keep the Beat for Real-Time Interaction Model Serving

arXiv:2607. 02640v1 Announce Type: cross Abstract: Real-time interaction models -- Moshi, MiniCPM-o, Qwen-Omni -- turn serving into a periodic real-time task: on every frame a session ingests streaming audio and must respond by a recurring wall-clock deadline, while its KV cache grows monotonically and stays pinned for the whole conversation.

By Jiaying Meng, Bojie Li

efficiency benchmarks

arXiv AI

Jul 7

PedestrianDiffusion: Multimodal Generative Denoising and Dense State Estimation for Inertial Navigation

arXiv:2607. 03349v1 Announce Type: cross Abstract: The accuracy of consumer-grade inertial navigation is bottlenecked by the stochastic noise of Micro-Electro-Mechanical Systems (MEMS).

By I-Hao Lu, Dongsoo Han

rag diffusion multimodal benchmarks

arXiv AI

Jul 7

SwarmResearch: Orchestrating Coding Agents for Open-Ended Discovery

arXiv:2607. 02807v1 Announce Type: new Abstract: Long-running coding agents such as autoresearch can persistently discover optimizations for open-ended problems.

By Yuvraj Virk, Zack Edds, Chunqiu Steven Xia, Lingming Zhang

llms agents benchmarks

arXiv AI

Jul 7

Brand-as-Memory: Vision-Language Models Encode Causal, Mechanistically Localizable Credibility Priors for News Sources

arXiv:2607. 03365v1 Announce Type: cross Abstract: Vision-language models (VLMs) increasingly read news and web content as images, where the publisher's identity is visually present.

By Chih-Ting Liao, Xin Cao

llms multimodal benchmarks safety

arXiv AI

Jul 7

Post-Generation Curation of Synthetic Images via Homogeneous-Heterogeneous Splitting

arXiv:2607. 02637v1 Announce Type: cross Abstract: Recent generative models can produce high-quality synthetic images, offering scalable training training data for data-hungry models.

By Disheng Liu, Tuo Liang, Chaoda Song, Yu Yin

llms computer-vision fine-tuning benchmarks safety

arXiv AI

Jul 7

The S-ICDF Dataset: Sionna-Simulated Dynamic Interference Characterization and Direction Finding

arXiv:2607. 03411v1 Announce Type: cross Abstract: Jamming and spoofing threaten wireless and satellite navigation by disrupting or manipulating radio frequency (RF) signals, undermining availability, integrity, and trust.

By Christian Wielenberg, Lucas Heublein, Jonathan Ott, Alexander Mattick, Nisha L. Raichur, Jonas Pirkl, Lukas Schelenz, Tobias Feigl, George Yammine, Christopher Mutschler, Felix Ott

benchmarks

arXiv AI

Jul 7

Folding, Reasoning, and Scaling with Open-source Drug Discovery Engine

arXiv:2607. 03787v1 Announce Type: new Abstract: Accurately modeling biomolecular interactions is a central bottleneck in biology and therapeutic discovery.

By Aureka AI OpenDDE project

benchmarks

arXiv AI

Jul 7

Explainable AI for Screening Abuse-Related Trauma in Bangladeshi Children: A Training-Free Multimodal Framework Evaluated on Noise-Aware Synthetic Data

arXiv:2607. 04010v1 Announce Type: new Abstract: Bangladesh has an estimated 1.

By Salma Hoque Talukdar Koli, Fahima Haque Talukder Jely

multimodal benchmarks

arXiv AI

Jul 7

Unified Audio Intelligence Without Regressing on Text Intelligence

arXiv:2607. 05196v1 Announce Type: cross Abstract: Audio intelligence involves understanding, reasoning about, and generating both audio and speech.

By Zhifeng Kong, Sang-gil Lee, Jaehyeon Kim, Boxin Wang, Zihan Liu, Sungwon Kim, Yang Chen, Arushi Goel, Rajarshi Roy, Wenliang Dai, Zhuolin Yang, Yangyi Chen, Dongfu Jiang, Sreyan Ghosh, Tuomas Rintamaki, Andrew Tao, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping

llms rag agents reinforcement-learning efficiency multimodal benchmarks safety

arXiv AI

Jul 7

EPRA U-Net: An Efficient Pyramid Residual Attention Framework for Accurate Infarct Segmentation in Diffusion-Weighted MRI

arXiv:2607. 03568v1 Announce Type: cross Abstract: Objective: Accurate identification of acute ischemic infarcts on diffusion-weighted magnetic resonance imaging (DWI) is a critical prerequisite for reliable lesion quantification and effective clinical decision support in the management of cerebrovascular events.

By Hasan Ulutas, Muhammet Emin Sahin, Mustafa Fatih Erkoc, Esra Yuce, Turker Tuncer, Sengul Dogan, Serkan Kiranyaz

diffusion computer-vision benchmarks

arXiv AI

Jul 7

CritiqueDriveVLM: From Verifier-Guided Reinforcement Learning to Latent Thought Distillation for Autonomous Driving

arXiv:2607. 04179v1 Announce Type: cross Abstract: End-to-end Vision-Language Models (VLMs) show immense potential in autonomous driving.

By Zhaohong Liu, Hao Ye, Xianlin Zhang, Mengshi Qi

llms agents reinforcement-learning robotics fine-tuning efficiency multimodal benchmarks safety

arXiv AI

Jul 7

Detecting Hallucinations in Retrieval-Augmented Generation through Grounding-Aware Sensitivity by Perturbation (GASP)

arXiv:2607. 04223v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) reduces but does not eliminate hallucination, and existing detectors return a single answer-level score that does not indicate which sentence is unsupported, or why.

By Mohamed Aly Bouke

rag nlp benchmarks safety

arXiv AI

Jul 7

SoftVTBench: A Safety-Aware Visuo-Tactile Benchmark for Physically Constrained Robotic Manipulation of Deformable Objects

arXiv:2607. 04234v1 Announce Type: cross Abstract: Deformable object manipulation poses challenges beyond task completion: successful execution must also maintain safe physical interaction, holding the object stably without slip or drop while avoiding excessive deformation.

By Bowen Jing, Mingxin Wang, Ruiyang Hao, Chenchen Ge, Hanwen Shen, Junjie He, Yang Cui, Yiming Hou, Weitao Zhou, Jiawei Wang, Minglei Li, Dandan Zhang, Ding Zhao, Houde Liu, Xiaofan Li, Si Liu, Ping Luo, Haibao Yu

robotics benchmarks safety

arXiv AI

Jul 7

HiFA4: Training-Free 4-bit FlashAttention on Ascend HIF4 NPUs for LLM Inference

arXiv:2607. 04302v1 Announce Type: cross Abstract: We present HiFA4, a post-training operator-level design that executes both QK^T and PV in FlashAttention as 4-bit HIF4 Cube GEMMs for LLM inference on Ascend NPUs, while maintaining the online softmax state in FP16.

By Hui Dong, Yanzhao Li, Jie Gao, Chunlu Li, Zhiyuan Zhang, Yupeng Sun, Zhenyuan Chen, Zhiqiang Zou

llms nlp efficiency benchmarks

Get the daily briefing