Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv AI

Sep 15

Rolling Day-Wise Mortality Prediction in Critically Ill Patients With AKI on CRRT Utilizing Machine Pressure Waveforms

This study develops a rolling day‑wise mortality prediction model for critically ill patients with acute kidney injury on continuous renal replacement therapy (CRRT) by incorporating minute‑level machine pressure waveforms alongside electronic health record (EHR) data. After cleaning the raw machine data—aligning it with therapy intervals, removing priming and downtime minutes, and denoising artifacts—the authors trained a transformer‑based stacked ensemble that fuses a sequence transformer with classical models using circuit‑instability features and clinical variables. In a multi‑center benchmark of 976 patients, the combined model achieved a one‑day mortality AUROC of 0.766, outperforming machine‑only (0.625) and EHR‑only (0.717) models, and SHAP analysis highlighted key pressure‑related features such as filter pressure, transmembrane pressure, and access‑to‑return difference. whyItMatters":"By integrating previously discarded CRRT machine data, the study provides a continuous, real‑time risk signal that could enable earlier detection of patient deterioration and improve mortality prediction in a high‑risk population."

By Shehan Irteza Pranto, Joanna Yang, Joshua Lambert, Stuart L. Goldstein, Lili Chan, Girish N. Nadkarni, Tiago K. Colicchio, Javier A. Neyra, Jin Chen

llms benchmarks

arXiv AI

Sep 15

Same Patient, Different Order: Action-Level Reliability of Clinical LLM Agents Under Repeated Runs

The paper introduces a new evaluation method called "same-input rerun" to assess the consistency of clinical language‑model agents across repeated runs. By replaying 1,000 MedAgentBench tasks with identical inputs, the authors find that action‑level outputs—such as test orders, medication requests, and referrals—vary significantly, even when benchmark scores remain unchanged. The study demonstrates that current benchmarks, which typically evaluate only a single run per task, can miss substantial behavioral divergence.

By Rohith Reddy Bellibatlu, Manpreet Singh, Zhoutian Han, Wenbin Zhang

llms agents benchmarks

arXiv AI

Sep 15

FaithfulBench: Does AI Counsel Uphold or Undermine the User's Professed Faith?

arXiv:2609.13634v1 Announce Type: cross Abstract: Do AI assistants help believers reason about moral dilemmas consistently with their faith? We present FaithfulBench, the first benchmark to score AI...

By M Waleed Kadous, Benjamin Olsen, Walter Scheirer, Daniel D. Slate, Alexander Arnold, DZ Kalman

benchmarks

arXiv AI

Sep 15

Leakage-Safe and Scheduler-Aware Machine Learning for Grid Job Runtime Prediction

arXiv:2609.13701v1 Announce Type: cross Abstract: Accurate job runtime prediction can improve scheduling-aware resource management in grid and distributed computing environments, but prediction model...

By Ashfaq Ali Shafin, Khandaker Mamun Ahmed

benchmarks

arXiv AI

Sep 15

HarnessBandit: Joint Learnability-Transferability Scheduling for Multi-Harness Agentic Reinforcement Learning

arXiv:2609.13739v1 Announce Type: cross Abstract: Language-model agents are increasingly deployed through diverse harnesses that differ in system prompts, tool schemas, control loops, and trajectory...

By Hongliang Wei (Harbin Institute of Technology, Alibaba Cloud), Xiaobing Tu (Alibaba Cloud), Yinggui Wang (Alibaba Cloud), Zhengxi Liu (Alibaba Cloud), Rongkun Xue (Alibaba Cloud), Jinkui Ren (Alibaba Cloud), Xiantao Zhang (Alibaba Cloud), Debin Zhao (Harbin Institute of Technology), Xiaopeng Fan (Harbin Institute of Technology)

agents reinforcement-learning benchmarks

arXiv AI

Sep 15

GEAR: From Dynamic Encoding to Dynamic Activation in Social Trajectory Prediction

arXiv:2609.13778v1 Announce Type: cross Abstract: Human trajectory prediction requires modeling both individual motion patterns and social interactions among agents. Existing methods have made substa...

By Jiaheng Chen, Jiaxing Li, Leixia Wang, Jianan Ju, Tinghe Zhang

agents benchmarks safety

arXiv AI

Sep 15

Bangla Sentence Function Classification: Corpus Development, Model Benchmarking, and Interpretability

arXiv:2609.13869v1 Announce Type: cross Abstract: Automatic sentence function identification is important for many downstream natural language processing (NLP) applications such as dialogue systems,...

By Swapnil Kundu Argha, Abdullah Al Shafi, Rowzatul Zannat, Shoumik Barman Polok, Abdul Muntakim, Jannatul Ferdousi, M. A. Moyeen

nlp multimodal benchmarks safety

arXiv AI

Sep 15

Phorecaster365: A Human-Supervised Reference Architecture for Hybrid Pharmaceutical Sales Forecasting and Planning Decision Support

arXiv:2609.13907v1 Announce Type: cross Abstract: Pharmaceutical sales forecasts inform planning across products, regions, and distribution channels, yet their interpretation depends on inventory ava...

By Houman Kazemzadeh, Kamyar Naderi

benchmarks

arXiv AI

Sep 15

Mizan: A National Benchmark for Evaluating Large Language Models on Iraqi Arabic and the Iraqi Civic Context

arXiv:2609.13980v1 Announce Type: cross Abstract: Arabic large-language-model (LLM) evaluation has matured around Modern Standard Arabic (MSA): aggregated leaderboards such as the Open Arabic LLM Lea...

By Nawar S. Alseelawi, Mustafa S. Aljumaily

llms benchmarks safety

arXiv AI

Sep 15

Confuse the Model, Control the Flow: Understanding and Mitigating Privacy Leakage from LLM Agents with Information Flow Control

arXiv:2609.14003v1 Announce Type: cross Abstract: Personal AI agents built on large language models (LLMs) are increasingly given access to a user's private data and communications in order to provid...

By Minsun Shim, Ramisha Raida Karim, Ruthwik Jakkula, Kaiwen Zhou, Xin Liu, Xin Eric Wang, Zhou Li

llms agents benchmarks safety

arXiv AI

Sep 15

Graph-Transformer Fraud Detection with Self-Supervised Pretraining and Conformal Risk Control

arXiv:2609.14234v1 Announce Type: cross Abstract: Financial fraud in corporate transaction networks has grown more coordinated and harder to detect with rule-based engines and with classical learning...

By Sergei, Komarov

llms multimodal benchmarks safety

arXiv AI

Sep 15

SpermYOLO: A Coordinated YOLO-Based Detector for Accurate and Efficient Sperm and Impurity Detection in Microscopic Images

arXiv:2609.14278v1 Announce Type: cross Abstract: Accurate sperm detection is essential for computer-assisted semen analysis, yet it remains challenging in microscopic images due to dense distributio...

By Shengqi Chen, Zilin Wang, Xingyu Pan, Wenting Yu, Pengchao Deng, Guohua Wu

computer-vision benchmarks

arXiv AI

Sep 15

Biquaternionic Space with Complex-valued Attention for Temporal Knowledge Graph Completion

arXiv:2609.14279v1 Announce Type: cross Abstract: Temporal knowledge graph embedding (TKGE) models infer missing facts in knowledge graphs that evolve over time. Many existing models use a single geo...

By Rushan Geng, Cuicui Luo

rag benchmarks

arXiv AI

Sep 15

Surrogate-Assisted Genetic Programming with Phenotypic Characterisation in Dynamic Multi-Mode Project Scheduling

arXiv:2609.14418v1 Announce Type: cross Abstract: Dynamic multi-mode resource-constrained project scheduling requires decisions to be made under precedence constraints, limited resources, multiple ex...

By Yuan Tian, Yi Mei, Mengjie Zhang

benchmarks

arXiv AI

Sep 15

A Generative AI Integrated Multimodal Framework for Low-Latency Multi-Camera Person Re-Identification

arXiv:2609.14419v1 Announce Type: cross Abstract: Person re-identification (ReID) is essential for multi-camera surveillance and tracking, yet remains difficult due to viewpoint and illumination chan...

By Leon Fernando, C Dombawala, P. Hettigoda, Vanodhya G. Warnasooriya, Ishara Neranjana, Rashmika Nawaratne

llms rag multimodal benchmarks

arXiv AI

Sep 15

Lightweight Generalized DeepFake Face Detection with WAVIE: Wavelet Augmented Vision Intermediate Embeddings

arXiv:2609.14437v1 Announce Type: cross Abstract: Deepfake detection systems often exhibit significant performance degradation when deployed on unseen manipulation methods, limiting their reliability...

By Arya Pulkit, Aditya Ruhela, Akarshan Kapoor, Arnav Bhavsar

llms rag robotics benchmarks

arXiv AI

Sep 15

SENTINEL: A Multi-Pathway Architecture for Detecting Living-Off-the-Land APT Attacks on Windows Command Lines

arXiv:2609.14593v1 Announce Type: cross Abstract: Living-Off-the-Land (LOTL) is the dominant evasion technique of Advanced Persistent Threat (APT) actors, exploiting legitimate Windows utilities to c...

By Ahad Bin Islam Shoeb, Kamrul Hasan, Jamal Uddin Tanvin, Liang Hong, Imtiaz Ahmed, Md Arif Billah, Al Amin

benchmarks safety

arXiv AI

Sep 15

Natural Language Knowledge Graph Query Execution: Leveraging Controlled Semantics in the LLM Context Window

arXiv:2609.14652v1 Announce Type: cross Abstract: Large Language Model (LLM) applications often transfer domain concepts into the model's context informally, through prompt prose, schema dumps, and e...

By Blake G. Fitch

llms benchmarks

arXiv AI

Sep 15

Compositional SVG Generation via VLM-Driven Hierarchical Semantic Parsing

arXiv:2609.14657v1 Announce Type: cross Abstract: While Vision-Language Models (VLMs) excel at visual reasoning, generating structured, editable Scalable Vector Graphics (SVG) remains a fundamental c...

By Sehwan Park, Taehoon Kim, Geonhee Han, Dohyun Kim, Seung Wook Kim, Paul Hongsuck Seo

llms agents multimodal benchmarks

arXiv AI

Sep 15

PU classification under Non-SCAR: clustering-assisted logistic model with oversampling enhancement

arXiv:2609.14675v1 Announce Type: cross Abstract: This study addresses the PU classification problem under violations of the SCAR assumption. We investigate logistic regression-based approaches, name...

By Konrad Furma\'nczyk, Kacper Paczutkowski

benchmarks

Get the daily briefing