arXiv Machine Learning
Compared to What? Baselines and Metrics for Counterfactual Prompting
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Oct 12
CoBe: A Benchmark For Conversational Counterfactual Text Editing
CoBe is a benchmark that evaluates a model’s ability to perform conversational counterfactual text editing, requiring the model to revise a given scenario based on a hypothetical change while preserving unaffected facts. The dataset includes 2,000 questions across domains such as healthcare, science, engineering, business, and everyday life. Experiments show that current models often default to associational generation rather than maintaining causal structure, and performance does not reliably improve with larger model size or longer outputs.
arXiv AI
Oct 2
Counterfactual Auditing of Bias in Open-Source Large Language Models for Clinical Triage
The study evaluates counterfactual bias in ten open‑source large language models (LLMs) for pediatric Emergency Severity Index (ESI) prediction. By creating paired clinical vignettes that differ only in demographic or socioeconomic variables, the authors measure shifts in acuity assignment, finding that counterfactual sensitivity varies widely across model families and sizes. A fine‑tuned Qwen2.5‑7B model exhibited the lowest sensitivity, while larger or medical‑domain models sometimes showed greater shifts, highlighting the need for fairness assessment before clinical deployment.
arXiv AI
Jun 17
Statistical Foundations of LLM-based A/B Testing: A Surrogacy Framework for Human Causal Inference
arXiv:2606. 17165v1 Announce Type: cross Abstract: Organizations and researchers show increasing interest in using large language models (LLMs) in place of human participants in A/B tests, in the hope of experimenting faster and at lower cost.
arXiv AI
Aug 18
Demographic Injection in Medical Language Models under Diversity, Equity, and Inclusion Prompts
arXiv:2608. 15254v1 Announce Type: new Abstract: Clinical-AI guidance increasingly recommends prompting language models to reason with attention to diversity, equity, and inclusion (DEI).
arXiv AI
Jun 16
AgentFairBench: Do LLM Agents Discriminate When They Act?
arXiv:2606. 16723v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly take actions (screening applicants, recommending credit, triaging patients), yet fairness for LLMs is still measured by grading answers.
arXiv Computation and Language
Sep 22
Recognition, Simulation, and Refusal: A Contamination-Aware Study of Classic Psychological Effects in LLM Agents
arXiv:2609.22090v1 Announce Type: new Abstract: An LLM producing the response pattern associated with a human psychological effect is not the same claim as the LLM possessing that bias. We present Ps...