REVIEW 6 cited by
Can Large Language Models Infer Causation from Correlation?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Causal inference is one of the hallmarks of human intelligence. While the field of CausalNLP has attracted much interest in the recent years, existing causal inference datasets in NLP primarily rely on discovering causality from empirical knowledge (e.g., commonsense knowledge). In this work, we propose the first benchmark dataset to test the pure causal inference skills of large language models (LLMs). Specifically, we formulate a novel task Corr2Cause, which takes a set of correlational statements and determines the causal relationship between the variables. We curate a large-scale dataset of more than 200K samples, on which we evaluate seventeen existing LLMs. Through our experiments, we identify a key shortcoming of LLMs in terms of their causal inference skills, and show that these models achieve almost close to random performance on the task. This shortcoming is somewhat mitigated when we try to re-purpose LLMs for this skill via finetuning, but we find that these models still fail to generalize -- they can only perform causal inference in in-distribution settings when variable names and textual expressions used in the queries are similar to those in the training set, but fail in out-of-distribution settings generated by perturbing these queries. Corr2Cause is a challenging task for LLMs, and would be helpful in guiding future research on improving LLMs' pure reasoning skills and generalizability. Our data is at https://huggingface.co/datasets/causalnlp/corr2cause. Our code is at https://github.com/causalNLP/corr2cause.
Forward citations
Cited by 6 Pith papers
-
CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matching
A new benchmark and training strategy show LLMs trained to internalize causal reasoning steps are less fooled by semantically similar, label-flipped questions than models using explicit chain-of-thought.
-
CausalT5k: Diagnosing Refusal and Failure Modes in Trustworthy Causal Reasoning Across Causal Rungs
CausalT5k provides a 5,147-case diagnostic benchmark with trap taxonomy, pressure variants, and Utility/Safety metrics for causal reasoning in LLMs.
-
CausalAbstain: Enhancing Multilingual LLMs with Causal Reasoning for Trustworthy Abstention
CausalAbstain filters multilingual self-feedback by comparing how much it changes the model's abstention decision, improving abstention accuracy over baselines on two benchmarks.
-
Retrieval Augmented Generation based Large Language Models for Causality Mining
Dynamic retrieval of few-shot examples by causal connective and semantic similarity improves LLM causality detection and extraction over static prompting in most tested settings.
-
Causal Distillation: Transferring Structured Explanations from Large to Compact Language Models
Small language models fine-tuned on GPT-4 causal explanations score high on a new teacher-similarity metric, but the paper provides no independent evidence that causal reasoning was transferred.
-
A comprehensive taxonomy of hallucinations in Large Language Models
A survey that organizes LLM hallucination types, causes, benchmarks, and mitigations, and restates the theorem that hallucination is inevitable for computable LLMs.
Discussion (0). Sign in to comment.