HerHealthEval is a controlled evaluation framework that tests multilingual understanding of women's-health communication across English, French, and Modern Standard Arabic. It provides six communicative forms—canonical, clinical, layperson, indirect or hedged, emotionally concerned, and deliberately under-specified—each conveying the same clinical concern except the under-specified form omits details to assess clarification needs. The study evaluates multilingual instruction models and QLoRA-adapted variants on tasks such as concern classification, risk calibration, clarification behavior, parse compliance, and cross-form consistency, revealing that high aggregate accuracy can mask safety-relevant failures and that language-invariant risk labels improve performance.
By Hassan Saeed Hassan Albattra, Mazen Mohammed Bahgat, Rahatara Ferdousi, Hana Essam Sayed Ahmed Amrya, Mariam Mousa
The article investigates whether commercial machine translation (MT) tools—Google Translate, Microsoft Translator, GPT, Gemini, and Claude—are ready to support communication during crises or pandemics. Using the TICO-19 dataset, which contains pandemic-related sentences in high-priority low-resource languages, the study evaluates the usability of translations produced by these systems. The goal is to determine how effectively these MT tools can bridge language gaps between governments, aid providers, medical professionals, and affected communities.
By Vipasha Bansal, William D. Lewis, Nicole Mihaylova, Elizabeth Brown, Benjamin Pong
arXiv:2503.22735v2 Announce Type: replace-cross
Abstract: This article reports on the third iteration of a survey of computerized tools and technologies taught as part of postgraduate translation tra...
By Andrew Rothwell, Joss Moorkens, Tomas Svoboda
arXiv:2608. 19981v1 Announce Type: new Abstract: We present HealMed, an expert-reviewed benchmark for multilingual evaluation of large language models in medicine.
By Yingjian Chen (Drew), Fan Gao (Drew), Sherry T. Tong (Drew), Haoyu Zhang (Drew), Aosong Feng (Drew), Kevin W. Jin (Drew), Xing Wu (Drew), Jinghui Lu (Drew), Abdul Samad (Drew), Akbar Faruqi (Drew), Cesar Caraballo (Drew), Cibele Brand\~ao (Drew), Dhruva (Drew), Gupta, Eunji Jeon, Gabriel Madera-Santiago, Geon Lee, Hugo Toshio Itikawa, Insook Cho, Isabelli Martins, Isarar Siddique, Israr Ahmed, Jihyo Kwak, Kanyakorn Veerakanjana, Luis Guilherme Cardoso, Minjin Kim, Piyalitt Ittichaiwong, Renee Dua, Santiago Gudi\~no-Rosales, Xiujie Chen, Zeo Lapalus, Zixin Xu, Michihiro Yasunaga, Rex Ying, Heuiseok Lim, Jaewoo Kang, Chanjun Park, Hang Jiang, Ethan Goh, Hyunjae Kim, Edison Marrese-Taylor, Yusuke Iwasawa, Yutaka Matsuo, Qingyu Chen, Irene Li
arXiv:2610.32842v1 Announce Type: new
Abstract: It is essential that we communicate with the entirety of an affected population during a crisis, including those who do not speak a region's dominant l...
By Jen Wilson, Vipasha Bansal, William D. Lewis
arXiv:2609.17544v1 Announce Type: new
Abstract: Large language models (LLMs) are increasingly being explored for clinical applications, yet their assessment for real-world traditional Chinese medicin...
By Jiacheng Xie, Xiaoting Tang, Yang Yu, Jinpu Li, Shouli Li, Congcong Jing, Yantao Yang, Zhiyong Zhao, Ziyang Zhang, Qilin Song, Guanghui An, Dong Xu
arXiv:2606. 00019v1 Announce Type: cross Abstract: Ambient artificial intelligence (AI) documentation tools are increasingly deployed to reduce clinician documentation burden, but their implications for biased language in clinical notes remain unclear.
By Yiliang Zhou, Yawen Guo, Sairam Sutari, Jasmine Dhillon, Alexandra L. Beck, Emilie Chow, Steven Tam, Danielle Perret, Deepti Pandita, Gelareh Sadigh, Archana J. McEligot, Kai Zheng
arXiv:2610.33749v1 Announce Type: new
Abstract: As machine translation systems approach human-level quality, traditional evaluation methodologies struggle to detect subtle translation errors. We crit...
By Dmitry Popov, Vladislav Negodin, Ekaterina Enikeeva, Iana Matrosova, Nikolay Karpachev, Max Ryabinin
arXiv:2609.14819v1 Announce Type: cross
Abstract: Large language models (LLMs) have a growing range of applications in medicine, and their evaluation is critical for ensuring they provide benefit and...
By Suzannah E McKinney, Phuc Vu, Samuel A Justice, Christopher Humphries, Alyssa Pradhan, Timothy J Keyes, Bernardo C Bizzo, Keith J Dreyer, Sarah F Mercaldo, James M Hillis
The paper introduces MSafeTemp, a large multilingual safety alignment corpus covering 60 languages, created via a template-based augmentation pipeline that extracts safe templates from harmful prompts, translates them, and fills slots with culturally authentic entities. The authors demonstrate that evaluating safety solely in English can misjudge multilingual performance, as attacks that fail in English may succeed when translated, and culturally localized variants reveal additional failures. They also show that augmenting safety training data with these templates improves safety performance beyond English.
By Max Neuwinger, Hanna Yukhymenko, Imanol Schlag
arXiv:2606. 00018v1 Announce Type: cross Abstract: Ambient AI documentation systems generate clinical note drafts that clinicians frequently revise before signing off into electronic health records, yet how these edits alter hedging language remains unclear.
By Yiliang Zhou, Yawen Guo, Di Hu, Sairam Sutari, Emilie Chow, Steven Tam, Danielle Perret, Deepti Pandita, Kai Zheng
arXiv:2601.09717v2 Announce Type: replace-cross
Abstract: Online medical consultations contain sensitive health information whose privacy implications depend not only on the entities mentioned but al...
By Yiwei Yan, Guanfeng Liu