← Back to all news
arXiv Computation and Language

EconCausal: A Context-Aware Economic Reasoning Benchmark for Large Language Models

Related stories

arXiv AI
Aug 26

From Causal Plausibility to Causal Reliability: Evaluating LLMs as Calibrated Direct Causal-Edge Classifiers

The study evaluates 12 instruction‑tuned open‑weight LLMs on six causal‑graph benchmarks, testing five prompting strategies and four confidence sources. Findings show that LLMs tend to over‑predict edges, misclassify indirect or reversed edges as direct, and exhibit high over‑confidence, while conventional confidence estimates are unreliable and agreement signals offer limited improvement. The results suggest LLMs should be used as externally validated soft causal priors rather than definitive causal‑structure evidence.

By Amit Kumar, Elnur Adl Zarabi, Suranjana Trivedy, Zhiqian Chen, Lei Zhang, Kaiqun Fu, Taoran Ji
llmsbenchmarks
More like this →
arXiv Machine Learning
Sep 2

How Do Language Models Choose Between Context and Memory?

The paper investigates how language models decide between contextual information and their internal memory when the two conflict. By estimating "authority directions" from agreement prompts and swapping these directions between matched prompts, the authors show that such interventions can reproduce 30–68% of the shift in source choice across Qwen, Llama, and OLMo models. Cross‑task experiments reveal that authority directions learned on one task transfer only modestly (≈9%) to another, indicating that authority computations are largely task‑specific.

By Benjamin Shih, John Winnicki, Arianna Cao
llms
More like this →
arXiv Computation and Language
Sep 28

Evidence-Grounded Auditing of Identification Assumptions in Climate-Policy Causal Evaluations

The paper introduces ARGUS, a language‑model pipeline that audits evidence for identification assumptions in difference‑in‑differences studies of climate policy. ARGUS evaluates reported evidence against an eleven‑dimension rubric, abstaining when evidence cannot be retrieved. In tests, ARGUS detects 73% of injected flaws versus 18% for a keyword approach, abstains on about 40% of assessments in 26 economics papers, and often assigns higher risk than human labels in a five‑paper pilot.

By Yonghong Zhang, Yong Xie, Isabel M. Parra, Ricardo Correia
benchmarks
More like this →
arXiv AI
Oct 5

Counterfactual Evidence Audits Predict LLM-Agent Susceptibility to Ranked Context

The paper introduces a counterfactual evidence audit to assess how large language model agents respond to different sets of documents. By exposing agents to two mirrored sets of five documents and measuring changes in six downstream decisions, the authors predict how the agents will behave when presented with a full 45‑document context. Across multiple open‑weight model families and tasks, the five‑document audit accurately forecasts full‑context effects, reducing prediction error by 62% and correctly identifying the direction of most material effects.

By Rana Muhammad Usman
llmsragagents
More like this →
arXiv AI
Sep 17

Bypassing the Rationale: Causal Auditing of Implicit Reasoning in Language Models

The paper introduces a causal, layerwise audit method called the CoT Mediation Index (CMI) to evaluate whether chain-of-thought (CoT) prompting truly influences a language model’s internal computation. By comparing performance degradation from patching CoT-token hidden states against matched control patches, the authors find that CoT influence is often confined to narrow reasoning windows and can be nearly absent even when the model produces fluent rationales. The study shows that models explicitly tuned for reasoning exhibit stronger mediation, while Mixture-of-Experts models display more distributed mediation, indicating that CoT faithfulness varies across models and tasks.

By Anish Sathyanarayanan, Aditya Nagarsekar, Aarush Rathore
llms
More like this →