Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

12,136 stories · RSS feed

arXiv AI
2d ago

Beyond Thresholds: A Quality-Aware Decision Intelligence Framework for Cold Chain IoT Systems

arXiv:2608. 15082v1 Announce Type: new Abstract: Cold chain logistics has advanced technologically, yet most deployed systems remain reactive monitors, not decision-making agents: thresholds trigger alerts, but nothing relates violations to cumulative product degradation or converts degradation signals into logistics decisions.

By Aashna Sofat, Balwinder Sodhi
arXiv AI
2d ago

LLMs Can Predict Failure Risk, But Struggle to Predict Which Collaboration Protocol Pays Off: Cost-Aware Protocol Routing Across Reasoning Tasks

arXiv:2608. 14927v1 Announce Type: new Abstract: Multi-agent large language model (LLM) systems can improve reasoning by spending more computation, but deployment requires deciding when extra collaboration is worth its cost.

By Chih-Hsuan Yang, Jingyan Jiang, Cheng-Hau Yang, Vikram Vasudevan, Huihuo Zheng, Venkatram Vishwanath, Rajeev Thakur
arXiv AI
2d ago

Evaluating Multimodal LLMs across Text and Audio Modalities for Accessible Disaster Assistance

arXiv:2608. 14651v1 Announce Type: new Abstract: Effective disaster risk communication is a foundational humanitarian challenge, yet current emergency infrastructure fails to meet the needs of individuals with access and functional needs, including hard-of-hearing individuals, pregnant women, mothers with toddlers, and elderly individuals with dementia.

By Anuridhi Gupta, Samara Mansoor, Hemant Purohit
arXiv AI
2d ago

When Agentic Executions Fail: Detecting and Localizing Runtime Faults from Telemetry

arXiv:2608. 14680v1 Announce Type: new Abstract: Reliability in LLM-based agentic systems is a property of the whole execution (its tool calls, model calls, guardrails, and inter-agent messages), not of the final answer alone, yet evaluating only task outcomes reveals little about how or why a run fails.

By Chenkai Zhang, Yiran Li, Yifang Tian, Michalis Bachras, Hans-Arno Jacobsen
arXiv AI
2d ago

CEDAR-GRPO: Process-Aware Reinforcement Learning for General Abductive Reasoning in LLMs

arXiv:2608. 14791v1 Announce Type: new Abstract: Abductive reasoning, often characterized as inference to the best explanation, is central to explanation under uncertainty, from everyday sense-making and investigation to scientific discovery.

By Moein Salimi, Danial Parnian, Shaygan Adim, Amirmohammad Ebrahiminasab, Nima Alighardashi, Parsa Gholami, Sahand Akramipour, Mahdi Jafari Siavoshani, Mohammad Hossein Rohban
arXiv AI
2d ago

ACTS-SQL: Agentic and Critic-Oriented Tree-Structured SQL Correctness with Large Language Models

arXiv:2608. 15145v1 Announce Type: new Abstract: Large Language Models (LLMs) have been increasingly adopted in Text-to-SQL systems, yet SQL errors remain a major obstacle in real-world Text-to-SQL inference pipelines.

By Xinmei Huang, Jie Song, Peng Li, Fuxin Jiang, Jing Zhang, Tieying Zhang, Jianjun Chen, Chenming Liu, Tao Yang, Maoyin Liu, Wenda Li, Hong Chen, Cuiping Li
arXiv Machine Learning
2d ago

Learning Varying Physical Therapist-Patient Interactions for Robot-mediated Upper Limb Task-Specific Training

arXiv:2608. 15995v1 Announce Type: cross Abstract: Upper extremity motor function recovery is positively linked to Task-Specific Training (TST) and sufficient therapy dosage.

By Jia Quan Loh (Human Robotics Laboratory, Department of Mechanical Engineering, The University of Melbourne), Vincent Crocher (Human Robotics Laboratory, Department of Mechanical Engineering, The University of Melbourne), Marlena Klaic (Melbourne School of Health Sciences, The University of Melbourne), Denny Oetomo (Human Robotics Laboratory, Department of Mechanical Engineering, The University of Melbourne), Ying Tan (Human Robotics Laboratory, Department of Mechanical Engineering, The University of Melbourne)