arXiv AI June 17, 2026 By Joel Persson, M{\aa}rten Schultzberg, Sebastian Ankargren

Statistical Foundations of LLM-based A/B Testing: A Surrogacy Framework for Human Causal Inference

arXiv:2606. 17165v1 Announce Type: cross Abstract: Organizations and researchers show increasing interest in using large language models (LLMs) in place of human participants in A/B tests, in the hope of experimenting faster and at lower cost.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning

Aug 24

Compared to What? Baselines and Metrics for Counterfactual Prompting

arXiv:2605.01048v2 Announce Type: replace-cross Abstract: Counterfactual prompting (i.e., perturbing a single factor and measuring output change) is widely used to evaluate things like LLM bias and C...

By Zihao Yang, Mosh Levy, Yoav Goldberg, Byron C. Wallace

llms safety

arXiv AI

Jun 24

When Preferences Fail to Become Incentives: A Utility-Behavior Gap in Large Language Models

arXiv:2606. 22974v2 Announce Type: replace Abstract: Recent work on preference elicitation in large language models (LLMs) has demonstrated that, when given a series of choices between two outcomes, LLMs reveal a coherent, model-specific utility structure.

By Yujun Zhou, Christopher M. Ackerman

llms safety

arXiv Computation and Language

Sep 22

Recognition, Simulation, and Refusal: A Contamination-Aware Study of Classic Psychological Effects in LLM Agents

arXiv:2609.22090v1 Announce Type: new Abstract: An LLM producing the response pattern associated with a human psychological effect is not the same claim as the LLM possessing that bias. We present Ps...

By Joy Bose

llms agents robotics benchmarks safety

arXiv Machine Learning

Sep 14

Aligning Language Models with Observational Data: Opportunities and Risks from a Causal Perspective

The paper discusses how large language models (LLMs) can be fine‑tuned with observational data to improve alignment with human preferences and business goals. It highlights that directly using such data can cause models to learn spurious correlations, and introduces DeconfoundLM, a method that removes known confounders from reward signals. Experiments show that DeconfoundLM better recovers causal relationships and outperforms baseline methods by over 16% in objective score when confounding is present.

By Erfan Loghmani

llms fine-tuning

arXiv Machine Learning

Sep 1

Item-Mean Surrogates: Why Richer Persona Data Fail to Improve LLMs as Human Surrogates

arXiv:2608.29455v1 Announce Type: cross Abstract: LLMs are increasingly used as human surrogates, often on the premise that richer persona data could make them substitutes or exploratory tools for sp...

By Daehwan Ahn, Chengfeng Mao, Dokyun Lee

llms fine-tuning benchmarks

arXiv Machine Learning

Jul 17

Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap

arXiv:2607. 14604v1 Announce Type: new Abstract: Online controlled experiments are the gold standard for hypothesis testing in online platforms.

By Olivier Jeunen

llms

One email a morning, machine-written

Related stories

Compared to What? Baselines and Metrics for Counterfactual Prompting

When Preferences Fail to Become Incentives: A Utility-Behavior Gap in Large Language Models

Recognition, Simulation, and Refusal: A Contamination-Aware Study of Classic Psychological Effects in LLM Agents

Aligning Language Models with Observational Data: Opportunities and Risks from a Causal Perspective

Item-Mean Surrogates: Why Richer Persona Data Fail to Improve LLMs as Human Surrogates

Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap