arXiv AI

MoveBench: A Rubric-Based Benchmark for Evaluating Large Language Models in Sports and Health Coaching

arXiv Computation and Language
4d ago

Keepace.ai: A Technical Report on a Large Language Model for Sports and Health

Keepace.ai is a domain‑specialized large language model for sports and health, built on Qwen3.5‑27B through post‑training. It is optimized for Chinese‑language exercise guidance across three tasks: workout plan generation, sports‑and‑health knowledge QA, and personal sports data analysis. Internal rubric‑based evaluations with domain experts show that Keepace outperforms its base model and larger general‑purpose baselines, especially in exercise safety, boundary compliance, structured output completeness, and scientific accuracy.

By ai Team, Bowen Wu, Heng Wang, Jufeng Mi, Kang Liu, Kun Li, Ling Man, Long He, Suli Fan, Xiaosen Jiang, Yi Wang, Ying He, Yu Tang, Yunhan Li, Zifang Liu
arXiv AI
Jul 28

OpenAIs HealthBench in Action: Evaluating an LLM-Based Medical Assistant on Realistic Clinical Queries

arXiv:2509. 02594v3 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on their ability to generate high-quality, accurate, situationally aware answers to clinical questions requires going beyond conventional benchmarks to assess how these systems behave in complex, high-stakes clinical scenarios.

By Sandhanakrishnan Ravichandran, Shivesh Kumar, Rogerio Corga Da Silva, Miguel Romano, Reinhard Berkels, Michiel van der Heijden, Olivier Fail, Valentine Emmanuel Gnanapragasam
arXiv AI
Jun 17

RubricsTree: Scalable and Evolving Open-Ended Evaluation of Personal Health Agents across Health Memory and Medical Skills

arXiv:2606. 18203v1 Announce Type: cross Abstract: The LLM-empowered personal health agents with user health (sensor) metrics have offered a promising pathway to alleviate global disparities in healthcare access.

By Weizhi Zhang, Zechen Li, Hamid Palangi, Ben Graef, A. Ali Heydari, Simon A. Lee, Salman Rahman, Ray Luo, Zeinab Esmaeilpour, Erik Schenck, Chloe Zhang, Yamin Li, Menglian Zhou, Philip S. Yu, Daniel McDuff, Lindsey Sunden, Mark Malhotra, Shwetak Patel, Ahmed A. Metwally
arXiv Machine Learning
Jul 17

Addressing Benchmarking Gaps in Large Language Models for Health and Medicine with Dynamic Red-Teaming

arXiv:2508. 00923v3 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used to answer health-related questions and support healthcare workflows, yet evidence for their safety still relies heavily on static benchmarks that can rapidly become obsolete or be optimized against.

By Jiazhen Pan (Cherise), Bailiang Jian (Cherise), Paul Hager (Cherise), Yundi Zhang (Cherise), Che Liu (Cherise), Friederike Jungmann (Cherise), Hongwei Bran Li (Cherise), Julian Canisius (Cherise), Chenyu You (Cherise), Junde Wu (Cherise), Jiayuan Zhu (Cherise), Fenglin Liu (Cherise), Yuyuan Liu (Cherise), Niklas Bubeck (Cherise), Moritz Knolle (Cherise), Chen (Cherise), Chen (Cherise), Christian Wachinger, Zhenyu Gong, Cheng Ouyang, Georgios Kaissis, Benedikt Wiestler, Daniel Rueckert
arXiv AI
Oct 27

Health-SCORE: Towards Scalable Rubrics for Improving Health-LLMs

Health‑SCORE is a scalable rubric‑based framework designed to evaluate and train health‑focused large language models (LLMs). It reduces the need for extensive human expertise and development cost while maintaining evaluation quality comparable to human‑created rubrics. The framework also serves as a structured reward signal for reinforcement learning and can be embedded in prompts to enhance response quality via in‑context learning.

By Zhichao Yang, Sepehr Janghorbani, Dongxu Zhang, Jun Han, Qian Qian, Andrew Ressler II, Gregory D. Lyng, Sanjit Singh Batra, Robert E. Tillman
arXiv AI
Oct 9

\$OneMillion-Bench: How Far are Language Agents from Human Experts?

The paper introduces OneMillion-Bench (OMB), a benchmark comprising 400 expert-curated tasks across Law, Finance, Industry, Healthcare, and Natural Science. OMB evaluates language agents on complex, real-world scenarios that require authoritative source retrieval, conflict resolution, domain-specific rule application, and constraint decision-making. The benchmark uses a rubric-based scoring system assessing factual accuracy, logical coherence, practical feasibility, and professional compliance to differentiate agent performance on expert-level problems.

By Yang Liu, Jiaqi Li, Jun Bai, Qianyu Yang, Xiaobo Hu, Tao Peng, Zaiyuan Wang, Ran Tian, Jiayun Dong, Chun Zhang, Zixia Jia, Kaiyuan Chen, Yixin Ren, Yang Liu, Yanglihong Xiao, Lingyue Yin, Tiliang Duan, Ge Zhang, Gang Yao, Hao Chen, Yuan Gong, Jianpeng Jiao, Zilong Zheng