arXiv AI October 27, 2026 By Qi Xu, Zijie Lei, Zhifei Deng, Zhigang Hua, Luke Simon

When Does Pre-Trained Capacity Help Causal Learning? From Scalar Prediction to Counterfactual Post-Training

The paper investigates how the capacity of pre‑trained language models influences performance on causal learning tasks, using two systems built on the Qwen 2.5 family: CausalLLM for scalar potential outcome estimation and CausalGen for generating treatment‑conditioned text or code. Experiments show that increasing backbone size from 0.5B to 1.5B yields mixed results for scalar estimation on IHDP but significantly improves CodeGen pass@1 and treatment‑effect scores after full generative training. Within the 0.5B CausalLLM, narrative inputs slightly reduce mean sqrt(PEHE) compared to tabular inputs, and overall improvements over a no‑LLM baseline combine representation and input format changes, suggesting a distinction between available and convertible capacity rather than a simple scaling law.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jul 2

CausalMix: Data Mixture as Causal Inference for Language Model Training

arXiv:2607. 01104v1 Announce Type: cross Abstract: In Large Language Model (LLM) training, data mixing plays a pivotal role in determining model performance.

By Zinan Tang, Yukun Zhang, Shaomian Zheng, Zhuoshi Pan, Qizhi Pei, Dingnan Jin, Jun Zhou, Yujun Wang, Biqing Huang

llms

arXiv Machine Learning

Aug 3

Evidence-Type Competition: When Can Interventional Data Teach Language Models Causal Direction?

arXiv:2607. 29484v1 Announce Type: cross Abstract: Interventional data is widely regarded as the gold standard for teaching models causal reasoning.

By Xining Xun

llms robotics

arXiv AI

Aug 6

Can Post-Training Transform LLMs into Causal Reasoners?

arXiv:2602. 06337v2 Announce Type: replace-cross Abstract: Causal inference is essential for decision-making but remains challenging for non-experts.

By Junqi Chen, Sirui Chen, Chaochao Lu

llms reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Aug 18

Architecture-Dependent Causal Transfer of Activation States Across Large Language Models

arXiv:2608. 16347v1 Announce Type: cross Abstract: Direct communication between AI systems relies on natural language as an intermediate layer, incurring encoding/decoding overhead, token cost, and latency.

By Fernando Cardenas Piepereit

llms safety

arXiv Machine Learning

Aug 11

Early Data Exposure Improves Robustness to Subsequent Fine-Tuning

arXiv:2605. 12705v2 Announce Type: replace Abstract: How can we train models whose post-trained capabilities survive subsequent fine-tuning?

By Lawrence Feng, Gaurav R. Ghosal, Jacob Mitchell Springer, Ziqian Zhong, Aditi Raghunathan

fine-tuning

arXiv AI

Jul 17

Generative Synthetic Data for Causal Inference: Pitfalls, Remedies, and Opportunities

arXiv:2604. 23904v3 Announce Type: replace-cross Abstract: Synthetic tabular data are often evaluated by distributional similarity, privacy distance, or train-on-synthetic-test-on-real predictive performance, but these criteria do not ensure validity for causal inference.

By Yichen Xu

llms diffusion benchmarks safety

One email a morning, machine-written

Related stories

CausalMix: Data Mixture as Causal Inference for Language Model Training

Evidence-Type Competition: When Can Interventional Data Teach Language Models Causal Direction?

Can Post-Training Transform LLMs into Causal Reasoners?

Architecture-Dependent Causal Transfer of Activation States Across Large Language Models

Early Data Exposure Improves Robustness to Subsequent Fine-Tuning

Generative Synthetic Data for Causal Inference: Pitfalls, Remedies, and Opportunities