Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,620 stories · RSS feed

arXiv AI

Sep 10

CIPHER: Benchmarking Cross-record Inference over Privacy-Hardened Evidence Records

arXiv:2609.07022v1 Announce Type: cross Abstract: Reasoning over privacy-constrained records requires combining structured attributes with evidence from free-text narratives. We introduce CIPHER (Cro...

By Suparno Roy Chowdhury, Manan Roy Choudhury, Dhruv Madhwal, Vivek Gupta

llms benchmarks safety data-engineering

arXiv AI

Sep 10

MEMOBench: A Process Level Memory Benchmark for Robotic Manipulation

arXiv:2609.07047v1 Announce Type: cross Abstract: Robotic manipulation often requires acting on information that is no longer visible, yet Vision-Language-Action policies are usually evaluated when t...

By Haiyang Sun, Haoxiao Wang, Junming Chen, Weicheng Fang, Zihao Su, Jingkun Yi, Wenyou Yi, Hao Chen, Zhou Zhao

robotics multimodal benchmarks safety

arXiv AI

Sep 10

In-Place Instruction Following in Diffusion Language Models

arXiv:2609.07160v1 Announce Type: cross Abstract: Diffusion Large Language Models (dLLMs) generate text via bidirectional iterative denoising, naturally supporting user-specified constraints anchored...

By Zheng Nie, Zherui Li, Jiaming Zhang, Kun Wang, Zhenhong Zhou, Yufei Guo

llms diffusion fine-tuning benchmarks safety

arXiv AI

Sep 10

Deep Learning for Biopsy-Free Subtyping of Basal Cell Carcinoma from Dermatoscopic Images

arXiv:2609.07180v1 Announce Type: cross Abstract: Basal Cell Carcinoma (BCC) is the most common type of skin cancer, accounting for nearly 80% of skin cancer di- agnoses. Its optimal clinical managem...

By Alexandros Papadopoulos, Chrysa Episkopou, Ioannis Sarafis, Aimilios Lallas, Anastasios Delopoulos

llms benchmarks

arXiv AI

Sep 10

LANTERN: Language Model Assessment on Noisy and Transformed Tasks for Understanding Error and Robustness Nuances

arXiv:2609.07309v1 Announce Type: cross Abstract: Robustness evaluation of large language models (LLMs) remains a critical challenge, particularly in assessing their sensitivity to perturbations in i...

By Vamsi Krishna Kodavali, Rituraj Singh

llms benchmarks

arXiv AI

Sep 10

Harnessing CLIP and DINO: An Uncertainty-Aware Cascaded Fusion Network for Generalizable Deepfake Image Detection

arXiv:2609.07670v1 Announce Type: cross Abstract: The growing realism and accessibility of manipulated and generated faces threaten the trustworthiness of digital media. To detect such forgeries, dee...

By Xuechao Zou, Yi Zhou, Kai Li, Shun Zhang, Yuhui Chen, Congyan Lang, Junliang Xing

llms benchmarks

arXiv AI

Sep 10

Rethinking Sign Language Translation: The Impact of Signer Dependence on Model Evaluation

arXiv:2609.07965v1 Announce Type: cross Abstract: Sign Language Translation has advanced with deep learning, yet evaluations remain largely signer-dependent, with overlapping signers across train/dev...

By Keren Artiaga, Sabyasachi Kamila, Haithem Afli, Conor Lynch, Mohammed Hasanuzzaman

benchmarks

arXiv AI

Sep 10

What Eviction Destroys: A Restore-Counterfactual Audit of Forgetting in Agent Memory

arXiv:2609.08279v1 Announce Type: cross Abstract: Agent memory systems must discard stored information when their history exceeds a fixed token budget. Existing budget-accuracy frontiers quantify the...

By Chen Shen

llms agents benchmarks

arXiv AI

Sep 10

A Measurement Study of LLM Inference Trade-offs Across Edge Continuum Hardware

arXiv:2609.08307v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as backends for intelligent web services, but serving them across the edge continuum requires bala...

By Maysam Khatib, Moysis Symeonides, Demetris Trihinas, George Pallis, Marios D. Dikaiakos

llms efficiency benchmarks

arXiv AI

Sep 10

Towards Embodied Air-Ground Cooperative Object Search: Benchmark, Dataset and Agentic Method

arXiv:2609.08402v1 Announce Type: cross Abstract: Air-Ground Object Search (AGOS) in urban environments is a challenging embodied task, which requires an Unmanned Aerial Vehicle (UAV) and an Unmanned...

By Boao Yu, Zimo Chen, Junreng Rao, Yue Hu, Zhengqiu Zhu, Yong Zhao, Rusheng Ju

llms agents robotics multimodal benchmarks

arXiv AI

Sep 10

DeCAL: Towards Physically-Grounded Dexterous Vision-Language-Action Models via Contact-Aware Latent Co-Imagination

arXiv:2609.09119v1 Announce Type: cross Abstract: Dexterous manipulation involves contact-rich and fine-grained interactions with the physical world, posing significant challenges for existing vision...

By Yankai Fu, Ning Chen, Junkai Zhao, Heng Zhang, Guocai Yao, Pengwei Wang, Zhongyuan Wang, Shanghang Zhang

llms robotics multimodal benchmarks

arXiv AI

Sep 10

Empowering VLMs for Few-Shot Multimodal Time Series Classification via Tailored Agentic Reasoning

arXiv:2605.09395v3 Announce Type: replace Abstract: In this paper, we propose the first VL\underline{\textbf{M}} \underline{\textbf{a}}gentic \underline{\textbf{r}}easoning framework for few-\underli...

By Lin Li, Jiawei Huang, Qihao Quan, Dan Li, Boxin Li, Xiao Zhang, Erli Meng, Wenjie Feng, Jian Lou, See-Kiong Ng

agents multimodal benchmarks safety

arXiv AI

Sep 10

TRACE: Trajectory Correction from Cross-layer Evidence for Hallucination Reduction

arXiv:2605.18163v2 Announce Type: replace Abstract: Hallucination correction is not a one-direction problem. We show that intermediate layers are neither uniformly more truthful than final layers nor...

By Tej Sanibh Ranade

llms fine-tuning benchmarks safety

arXiv AI

Sep 10

Teaching agentic AI to generalize expert diagnostic reasoning in rare diseases

arXiv:2606.16149v5 Announce Type: replace Abstract: Rare disease diagnosis depends on expert reasoning that is scarce and difficult to transfer. Large language models rank the correct disease first i...

By Minh-Ha Nguyen, Erica Gray, Bryce A. Schuler, Kevin W. Byram, Chih-Ting Yang, Fan Ma, Hua Xu, Wu-Chen Su, Chao Yan, Wei-Qi Wei, Adam Wright, Lisa Bastarache, Josh F. Peterson, Lingyao Li, Siyuan Ma, Undiagnosed Diseases Network, Rizwan Hamid, Thomas A. Cassini, Cathy Shyr

llms agents benchmarks

arXiv AI

Sep 10

FATS: A Prompt Injection Attack Utilizing Feign Security Agents with Deceptive Few-shots Learning

arXiv:2410.08776v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) face significant security risks despite their advanced capabilities. While techniques like Reinforcement Learnin...

By Yupeng Ren, Jiangtao Chen, Rui Zhang

llms agents reinforcement-learning benchmarks safety

arXiv AI

Sep 10

When Tools Hurt LLM Reasoning: State-Dependent Belief Revision under External Evidence

arXiv:2508.15754v2 Announce Type: replace-cross Abstract: Tool use is often assumed to monotonically improve reasoning, where external evidence is expected to help when relevant and be ignored when i...

By Yufeng Zhao, Junnan Liu, Hongwei Liu, Dongsheng Zhu, Yuan Shen, Songyang Zhang, Kai Chen

llms agents benchmarks

arXiv AI

Sep 10

ConsistencyAI: A Benchmark to Assess LLMs' Factual Consistency When Responding to Different Demographic Groups

arXiv:2510.13852v3 Announce Type: replace-cross Abstract: Is an LLM telling you different facts than it's telling me? This paper introduces ConsistencyAI, an independent benchmark for measuring the f...

By Peter Banyas, Shristi Sharma, Alistair Simmons, Atharva Vispute

llms rag benchmarks

arXiv AI

Sep 10

Can One-Shot Test-Time Data Augmentation Help with Generalization?

arXiv:2602.00114v5 Announce Type: replace-cross Abstract: Data augmentation is crucial for model generalization, but existing methods are mostly centered on the training stage. Test-time augmentation...

By Yunwei Bai, Yao Shu, Ying Kiat Tan, Tsuhan Chen

diffusion fine-tuning benchmarks

arXiv AI

Sep 10

CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation

arXiv:2603.00039v2 Announce Type: replace-cross Abstract: LLM-as-a-judge ensembles are the standard paradigm for scalable evaluation, but their aggregation mechanisms suffer from a fundamental flaw:...

By Jitian Zhao, Changho Shin, Tzu-Heng Huang, Satya Sai Srinath Namburi GNVV, Frederic Sala

llms benchmarks safety

arXiv AI

Sep 10

Asking the Right Questions: Ontology-Grounded Interpretable Embeddings for Biomedical Text

arXiv:2603.01690v3 Announce Type: replace-cross Abstract: While dense biomedical embeddings achieve strong performance, their opaque dimensions limit transparency in biomedical NLP. Recent question-b...

By Yixuan Tang, Zhenghong Lin, Yandong Sun, Wynne Hsu, Mong Li Lee, Anthony K. H. Tung

rag nlp benchmarks

Get the daily briefing