arXiv AI
Jul 2
CausalMix: Data Mixture as Causal Inference for Language Model Training
arXiv:2607. 01104v1 Announce Type: cross Abstract: In Large Language Model (LLM) training, data mixing plays a pivotal role in determining model performance.
When Does Pre-Trained Capacity Help Causal Learning? From Scalar Prediction to Counterfactual Post-Training
The paper investigates how the capacity of pre‑trained language models influences performance on causal learning tasks, using two systems built on the Qwen 2.5 family: CausalLLM for scalar potential outcome estimation and CausalGen for generating treatment‑conditioned text or code. Experiments show that increasing backbone size from 0.5B to 1.5B yields mixed results for scalar estimation on IHDP but significantly improves CodeGen pass@1 and treatment‑effect scores after full generative training. Within the 0.5B CausalLLM, narrative inputs slightly reduce mean sqrt(PEHE) compared to tabular inputs, and overall improvements over a no‑LLM baseline combine representation and input format changes, suggesting a distinction between available and convertible capacity rather than a simple scaling law.
Related stories
arXiv Machine Learning
Aug 3
Evidence-Type Competition: When Can Interventional Data Teach Language Models Causal Direction?
arXiv:2607. 29484v1 Announce Type: cross Abstract: Interventional data is widely regarded as the gold standard for teaching models causal reasoning.
arXiv AI
Aug 6
Can Post-Training Transform LLMs into Causal Reasoners?
arXiv:2602. 06337v2 Announce Type: replace-cross Abstract: Causal inference is essential for decision-making but remains challenging for non-experts.
arXiv Machine Learning
Aug 18
Architecture-Dependent Causal Transfer of Activation States Across Large Language Models
arXiv:2608. 16347v1 Announce Type: cross Abstract: Direct communication between AI systems relies on natural language as an intermediate layer, incurring encoding/decoding overhead, token cost, and latency.
arXiv Machine Learning
Aug 11
Early Data Exposure Improves Robustness to Subsequent Fine-Tuning
arXiv:2605. 12705v2 Announce Type: replace Abstract: How can we train models whose post-trained capabilities survive subsequent fine-tuning?
arXiv AI
Jul 17
Generative Synthetic Data for Causal Inference: Pitfalls, Remedies, and Opportunities
arXiv:2604. 23904v3 Announce Type: replace-cross Abstract: Synthetic tabular data are often evaluated by distributional similarity, privacy distance, or train-on-synthetic-test-on-real predictive performance, but these criteria do not ensure validity for causal inference.
arXiv Machine Learning
Sep 30
What You Observe Determines How You Identify Causal Effects: Evaluating Causal Models across Observational Views
arXiv:2609.36881v1 Announce Type: new Abstract: Causal foundation models (CFMs) pre-trained on data generated from various structural causal models (SCMs) have been proposed for estimating causal eff...
arXiv Machine Learning
Sep 10
CausalBN-Bench: A Comprehensive Benchmark for Causal Learning Capability of LLMs
arXiv:2404.06349v3 Announce Type: replace Abstract: The ability to understand causality significantly impacts the competence of large language models (LLMs) in output explanation and counterfactual r...
arXiv Machine Learning
Aug 19
TabCausal: Pretraining Across Causal Environments for Tabular Causal Discovery
TabCausal is a causal discovery foundation model that learns to map datasets directly to causal graphs by pretraining across diverse causal environments. It uses a dynamic task construction strategy to expose the model to varied graph priors, mechanisms, noise models, dimensions, sample sizes, and intervention regimes, improving transferability from observational and mixed‑interventional data. On large synthetic benchmarks and a new protocol‑guided semantic benchmark, TabCausal outperforms many classical baselines and shows robust structure recovery, especially when interventional evidence is available.
arXiv AI
Sep 10
When and Why LLM Causal Priors Help: Closed-Loop Prior Selection for Amortized Causal Inference
arXiv:2609.06941v1 Announce Type: new Abstract: Causal effect estimation asks how an outcome would change under an intervention, and medicine, economics, and public policy all treat it as a foundatio...
arXiv AI
Sep 28
Causal Retention in Interactive Agents: Interface Factorization and Selective Adaptation
The paper introduces the concept of causal retention in interactive agents, examining whether a frozen learned state can correctly answer a mechanism‑probe map that is fixed independently of training. It shows that for finite structural causal models the optimal probe error is a Bayes decision risk, vanishing only when each learning‑interface fiber lies within a single probe‑answer fiber, and provides theoretical results such as a posterior‑coverage theorem and an exact edit decomposition. Experiments on finite causal systems, continuous simulators, TD‑MPC2, and Qwen2.5‑7B‑Instruct demonstrate that causal retention can be achieved with high accuracy, outperforming task‑performance‑based approaches.
arXiv Machine Learning
Oct 23
ACTIVA: Amortized Causal Effect Estimation via Variational Autoencoders
ACTIVA is a transformer‑based conditional variational autoencoder designed to estimate full interventional distributions from observational data without requiring interventional data, a causal graph, or per‑task fitting. It learns a conditional latent prior that amortizes causal knowledge across diverse training tasks, enabling a single forward pass to handle new causal models drawn from the training distribution. Empirical results on synthetic and biologically realistic gene‑expression simulations show that ACTIVA outperforms correlational baselines, reduces spurious non‑descendant effects, and matches strong amortized baselines, while also validating its theoretical mixture‑of‑distributions property on observationally equivalent models.