CoBe is a benchmark that evaluates a model’s ability to perform conversational counterfactual text editing, requiring the model to revise a given scenario based on a hypothetical change while preserving unaffected facts. The dataset includes 2,000 questions across domains such as healthcare, science, engineering, business, and everyday life. Experiments show that current models often default to associational generation rather than maintaining causal structure, and performance does not reliably improve with larger model size or longer outputs.
By Arvind Raghavan, Inwoo Hwang, Shreyas Havaldar, Kai-Zhan Lee, Aurghya Maiti, Yushu Pan, Jeffrey Wu, Mingxuan Li, Elias Bareinboim
The study evaluates counterfactual bias in ten open‑source large language models (LLMs) for pediatric Emergency Severity Index (ESI) prediction. By creating paired clinical vignettes that differ only in demographic or socioeconomic variables, the authors measure shifts in acuity assignment, finding that counterfactual sensitivity varies widely across model families and sizes. A fine‑tuned Qwen2.5‑7B model exhibited the lowest sensitivity, while larger or medical‑domain models sometimes showed greater shifts, highlighting the need for fairness assessment before clinical deployment.
By Manar Aljohani, Brandon Ho, Kenneth McKinley, Dennis Ren, Xuan Wang
arXiv:2606. 17165v1 Announce Type: cross Abstract: Organizations and researchers show increasing interest in using large language models (LLMs) in place of human participants in A/B tests, in the hope of experimenting faster and at lower cost.
By Joel Persson, M{\aa}rten Schultzberg, Sebastian Ankargren
arXiv:2608. 15254v1 Announce Type: new Abstract: Clinical-AI guidance increasingly recommends prompting language models to reason with attention to diversity, equity, and inclusion (DEI).
By Diego Mardian, Frank Liu
arXiv:2606. 16723v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly take actions (screening applicants, recommending credit, triaging patients), yet fairness for LLMs is still measured by grading answers.
By Triveni Morla, Rohith Reddy Bellibaltu, Manpreet Singh, Manmeet Singh Kapoor
arXiv:2609.22090v1 Announce Type: new
Abstract: An LLM producing the response pattern associated with a human psychological effect is not the same claim as the LLM possessing that bias. We present Ps...
By Joy Bose
arXiv:2610.22618v1 Announce Type: new
Abstract: Large language models rely on spurious features in clinical classification: swapping a single gender token in clinical notes alters gender-irrelevant d...
By Fardin Ahsan Sakib, Ziwei Zhu, Ozlem Uzuner
arXiv:2609.38036v2 Announce Type: replace-cross
Abstract: Understanding gender biases in large language models (LLMs) is increasingly important as these systems become embedded in decision-support to...
By Edoardo Bolzoni, Valerio Capraro
The study examines how different editorial framings in prompts influence large language models’ statistical analysis reports. Using a 4×4 factorial design, researchers found that certain framings—particularly brutally critical prompts on genuine effects and significance-seeking prompts on underpowered nulls—led to factual misrepresentations. Tone shifts were more widespread, with critical framing inducing defensive language across all data patterns, while a confound in the data largely prevented both factual and tonal distortions.
By Paras Balani, Subhrakanta Panda
arXiv:2609.32807v2 Announce Type: replace
Abstract: Clinical LLM evaluation often emphasizes answer accuracy; however, accuracy alone does not test counterfactual consistency or demographic robustnes...
By Chaitali Deb Purkayastha, Bharath Kumar Bolla, Vishnu Surya Reddy Nandi
arXiv:2609.14124v1 Announce Type: cross
Abstract: Medical image analysis is often hindered by biased datasets, which can lead to biased models and limited clinical applicability. A promising strategy...
By Yasin Ibrahim, Robin J. Evans, Konstantinos Kamnitsas
arXiv:2410. 01227v2 Announce Type: replace-cross Abstract: In the context of medical records, patients often experience testimonial injustice, where the textual account undermines the validity of their experiences.
By Kenya S. Andrews, Mesrob I. Ohannessian, Elena Zheleva