arXiv AI October 31, 2026 By Justin Clarke, Isha Karn, Patrick Murray, KJ Hill, David Jensen

Evaluating Causal Reasoning in LLMs: A Case Study in the Challenges of Procedural Generation

The paper investigates whether large language models (LLMs) truly perform formal causal reasoning by testing them on datasets generated by the CLadder framework. It finds that LLMs and even simple models like random forests can achieve high accuracy on CLadder data, largely due to superficial, highly predictive patterns embedded by the generator. The authors argue that strong performance on CLadder likely reflects exploitation of these patterns rather than genuine causal reasoning, and that many commercial LLMs show brittle results when critical text is removed.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Aug 6

Can Post-Training Transform LLMs into Causal Reasoners?

arXiv:2602. 06337v2 Announce Type: replace-cross Abstract: Causal inference is essential for decision-making but remains challenging for non-experts.

By Junqi Chen, Sirui Chen, Chaochao Lu

llms reinforcement-learning fine-tuning benchmarks

arXiv AI

Aug 11

TempoBench: Reasoning Execution Without Causal Attribution Is Just Simulation

arXiv:2510. 27544v3 Announce Type: replace Abstract: Current training paradigms, optimized for long-horizon reasoning trace execution, have made Large Language Models (LLMs) excel at pattern matching and forward simulation of reasoning, but underperform at counterfactual causal understanding and reasoning.

By Nikolaus Holzer, William Fishell, Baishakhi Ray, Mark Santolucito

llms agents benchmarks

arXiv Machine Learning

Sep 10

CausalBN-Bench: A Comprehensive Benchmark for Causal Learning Capability of LLMs

arXiv:2404.06349v3 Announce Type: replace Abstract: The ability to understand causality significantly impacts the competence of large language models (LLMs) in output explanation and counterfactual r...

By Yu Zhou, Xingyu Wu, Jibin Wu, Liang Feng, Kay Chen Tan

llms benchmarks

arXiv AI

Jun 9

TempoBench: Evaluating Temporal Causal Reasoning in Large Language Models

arXiv:2510. 27544v2 Announce Type: replace Abstract: Temporal reasoning involves understanding how systems evolve over time through input-driven state transitions.

By Nikolaus Holzer, William Fishell, Baishakhi Ray, Mark Santolucito

llms fine-tuning benchmarks

Hugging Face Trending Papers

Jul 9

CausalDS: Benchmarking Causal Reasoning in Data-Science Agents

Large language models (LLMs) increasingly act as integrated data-science agents, combining abstract reasoning with advanced tool use. Yet the relevant benchmark landscape largely divides into symbolic causal reasoning benchmarks without realistic data analysis or data analysis benchmarks without a principled causal data-generating structure.

llms agents benchmarks

arXiv AI

Jul 10

CausalDS: Benchmarking Causal Reasoning in Data-Science Agents

arXiv:2607. 08093v1 Announce Type: new Abstract: Large language models (LLMs) increasingly act as integrated data-science agents, combining abstract reasoning with advanced tool use.

By Andrej Leban, Yuekai Sun

llms agents benchmarks

One email a morning, machine-written

Related stories

Can Post-Training Transform LLMs into Causal Reasoners?

TempoBench: Reasoning Execution Without Causal Attribution Is Just Simulation

CausalBN-Bench: A Comprehensive Benchmark for Causal Learning Capability of LLMs

TempoBench: Evaluating Temporal Causal Reasoning in Large Language Models

CausalDS: Benchmarking Causal Reasoning in Data-Science Agents

CausalDS: Benchmarking Causal Reasoning in Data-Science Agents