Pith. sign in

REVIEW 1 major objections 3 minor 23 references

Causality for Natural Language Processing

T0 review · 1 major / 3 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The thesis argues that current large language models lack robust formal causal reasoning: they perform near chance on correlation-to-causation inference, stay below 62% on multi-step causal questions, and their fine-tuned gains collapse…

desk verdict A thesis compilation of important causality-in-NLP benchmarks; the Corr2Cause MEC construction may have a subtle labeling flaw worth checking. read the letter →

arxiv 2504.14530 v1 pith:73F5BZZX submitted 2025-04-20 cs.CL cs.AIcs.CYcs.LG

Zhijing Jin This is my paper · ORCID
classification cs.CLcs.AIcs.CYcs.LG MSC 68T5062D2068T07
keywords causalreasoninglargelanguagemodelsdiscoveryeffectestimationbenchmarkdatasetschain-of-thoughtpromptingandanticausallearningmechanisticinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This thesis sets out to determine whether large language models can perform formal causal reasoning rather than recite causal facts memorized from training text. Using newly constructed, symbolically grounded benchmarks, it reports that current models perform close to chance when asked to infer causation from correlation and stay below 62% accuracy on multi-step causal questions, with fine-tuned gains collapsing under paraphrasing or variable renaming. It then uses causal graphs and interventions as an interpretive lens to show where these failures originate inside the models and how the causal direction of data collection shapes task difficulty. A sympathetic reader would take the central claim to be that LLM causal competence is largely memorization-like, and that formal causal machinery is needed to measure, locate, and eventually repair it.

What carries the argument

The load-bearing machinery is the directed acyclic graph treated as a causal model, paired with d-separation and Markov equivalence to generate correlation statements and validity labels, and with the ladder of causation and do-calculus to generate query-answer pairs for interventions and counterfactuals. The benchmarks convert these graph-theoretic conditions into natural-language verbalizations, and CausalCoT forces a language model through six formal steps: extract the graph, classify the query, formalize it, gather data, derive the estimand, and calculate. In the interpretability chapters the same graph idea is inverted: interventions on inputs, on attention entries, and on textual surface forms quantify direct versus total causal effects, separating desired sensitivity to the ground-truth result from brittleness to spurious factors.

What would settle it

A single transformer whose fine-tuned Corr2Cause F1 stays above roughly 90 after full variable refactorization, and whose CLadder accuracy on anti-commonsense and nonsensical story variants rises well above chance, would refute the paper's central diagnosis, because the current results collapse specifically on those perturbations.

Watch

Extended reading notes

Core claim

The paper's central claim is that today's LLMs do not yet possess robust formal causal reasoning. On Corr2Cause, a benchmark built from all legal directed graphs of two to six variables, a causal hypothesis is labeled valid only when it holds across the whole Markov equivalence class of the stated correlations; seventeen off-the-shelf LLMs score near random, with the best F1 at 33.38 for BART MNLI and GPT-4 at 29.08. On CLadder, which generates 10K questions spanning the three rungs of the ladder of causation, GPT-4 reaches 62.03% accuracy, and a purpose-built chain-of-thought prompt, CausalCoT, raises that to 70.40%. Fine-tuning on Corr2Cause can reach 94.74% F1 in-distribution, but this apparent mastery drops by up to 62.30 F1 points under paraphrasing or variable renaming. The thesis also argues, from the independent causal mechanisms principle, that prediction tasks aligned with the causal direction of data collection behave differently from anticausal ones, and it applies causal-effect reasoning to locate where factual and counterfactual mechanisms compete inside transformer layers.

Load-bearing premise

The load-bearing premise is that the benchmark instruments measure formal causal reasoning and not template artifacts: the paper itself flags that Corr2Cause assumes a closed system with no hidden confounders and faithfulness, that CLadder assumes its verbalizations faithfully encode the symbolic causal query, and that the independent-mechanisms principle may not transfer to text data.

Editorial extensions

If this is right

  • Corr2Cause and CLadder provide out-of-distribution tests of formal causal skills: because their questions are generated symbolically, high scores cannot be explained by memorized verbatim examples.
  • Fine-tuned causal performance should be reported alongside paraphrased and variable-renamed test sets; under those perturbations the thesis observes F1 drops of up to 62 points, so i.i.d. gains alone overstate learning.
  • Structured causal prompting (CausalCoT) raises GPT-4's CLadder accuracy from 62.03% to 70.40%, with the largest gains on associational questions and diminishing returns toward counterfactual reasoning.
  • The causal direction of data collection predicts how much semi-supervised learning and domain adaptation help; practitioners can use the causal-versus-anticausal categorization to anticipate when unlabeled data will be effective.
  • Editing only two or three attention-map entries in specific heads can shift whether a model follows a counterfactual redefinition or recalls a memorized fact, indicating that mechanism-level control is possible without full retraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not draw this conclusion, but its perturbation results imply that causal benchmark reports should include a variable-renaming audit as standard practice; any accuracy that does not survive renaming is template matching, not causal reasoning.
  • A practical corollary the paper leaves implicit: if current LLMs are causal parrots on formal questions, applications that surface causal claims from text should either stay at the level of correlation or route the query through an explicitly represented causal model before asserting an effect.
  • The causal-versus-anticausal taxonomy could be turned into a data-collection rule: record the order in which X and Y were generated, and choose which side gets annotation effort based on whether semi-supervised learning or domain adaptation is the target use.
  • The competition-of-mechanisms method suggests a testable extension to safety: apply the same attention modification to prompts that redefine a harmful concept in-context and check whether a few localized entries can suppress memorized associations.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 3 minor

Summary. This manuscript is a PhD thesis that compiles peer-reviewed papers on causality for natural language processing. It argues that large language models (LLMs) lack robust formal causal reasoning, based on two new benchmarks: Corr2Cause (Chapter 2), which tests causal discovery from correlational statements, and CLadder (Chapter 3), which tests causal effect reasoning across the three rungs of Pearl's ladder. It then presents studies on mechanistic interpretability of LLMs (Chapter 4), a causal framework for quantifying robustness of mathematical reasoning (Chapter 5), the implications of causal and anticausal learning directions in NLP tasks (Chapters 6-7), and two applications to text-based computational social science (Chapters 8-9). The thesis concludes that current LLMs perform near random on formal causal inference and that their apparent causal ability is partly memorization rather than reasoning.

Significance. If the central claims hold, the thesis provides valuable benchmark resources and a comprehensive framework for evaluating causal reasoning in LLMs. The manuscript's strengths include open-sourced data and code for each chapter, formal grounding of dataset generation in causal inference theory, and explicit robustness checks such as paraphrase and variable-refactorization perturbations (Chapter 2), anti-commonsensical and nonsensical verbalizations (Chapter 3), and adversarial interventions on input factors (Chapter 5). The constituent chapters have been peer-reviewed at major NLP/ML venues, lending credibility to the individual results. The thesis also offers a useful synthesis of causality-for-NLP research directions and applications.

major comments (1)
  1. [§2.3.3-§2.3.5, Table 2.1] The construction of Corr2Cause conflates graph isomorphism with Markov equivalence, which corrupts the ground-truth labels of the benchmark. In §2.3.3, DAGs are generated by enumerating upper-triangular adjacency matrices (edges only from smaller to larger alphabetical index) and then reduced to one representative per isomorphism class before §2.3.4 computes MECs from the d-separation sets of these representatives. However, Markov equivalence is not preserved by isomorphism-based deduplication: two DAGs in the same MEC need not be isomorphic, and isomorphic DAGs can induce different d-separation statements on a fixed label set. Concretely, for N=3 the true MEC of the chain A→B→C contains the fork B→A,B→C and the reverse chain C→B→A. The fork is isomorphic to the source A→B,A→C and the reverse chain to the chain, so both are deduplicated away. As a result, the correlation set {A independent of C given B; A correlates with B; B correlates with C} is mapped to an 'MEC' containing only the chain, and the hypothesis 'A directly causes B' is labeled valid. In the true MEC over labeled variables, this hypothesis is not entailed, because it is false in the fork and in the reverse chain. Thus the labels in Corr2Cause are not the formal causal-discovery answer advertised in §2.3.5, and the headline near-random results in Table 2.4 may reflect disagreement with an incorrect labeling scheme rather than absence of causal reasoning. The statistics in Table 2.1 (6 DAGs → 5 MECs for N=3, whereas the 25 labeled DAGs on three variables form 9 MECs) signal that the deduplication has collapsed distinct MECs. This is a load-bearing issue for Chapter 2 and for the thesis's central claim; it requires recomputing MECs over the full set of labeled DAGs (or at least before isomorphism deduplication) and regenerating the benchmark and experiments.
minor comments (3)
  1. [§3.3.4] The human sanity check reports 82% accuracy on 50 questions from one expert evaluator; this is a low ceiling and raises the possibility that some questions are ambiguous or that the verbalization does not uniquely encode the symbolic query. Please report inter-annotator agreement and a larger human evaluation to strengthen the benchmark's validity.
  2. [Tables 2.4, 3.2, 5.3] Many LLM evaluations are reported as point estimates without confidence intervals, and some GPT-3 metrics in Chapter 5 are approximate due to API constraints (Appendix A.4.3, asterisks in figures). Please report variance or significance tests for the key comparisons, such as the CausalCoT improvement over vanilla GPT-4 in Table 3.2, so readers can assess whether differences are meaningful.
  3. [§6.2, Table 6.1] The categorization of NLP tasks into causal, anticausal, and mixed is central to the meta-analysis in Chapter 6, but the assignment relies on the authors' judgment. Please provide more explicit decision rules or inter-annotator agreement for the categorization, and discuss how sensitive the meta-analysis results are to borderline cases (e.g., question answering, machine translation).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the thesis benchmarks and empirical claims rest on independently generated formal ground truth and external model evaluations.

full rationale

The core benchmarks (Corr2Cause and CLadder) are constructed algorithmically from formal causal definitions: d-separation sets, Markov equivalence classes, and an oracle causal-inference engine. Ground-truth labels are computed from these formal objects before any model is consulted; the reported LLM results are then measured against that independently generated ground truth, so there is no fitted input being renamed as a prediction. The CLadder ground truth is identifiable by construction, and the anti-commonsensical/nonsensical verbalizations are added specifically to block memorization as an alternative explanation. Part II uses public datasets (CounterFact, MWP benchmarks) and measures causal effects via logit inspection and interventions; no claim reduces to its own definition. Part III's causal/anticausal categorization is grounded in the temporal order of the data collection/annotation process (Figure 6.1) rather than in the SSL/DA outcomes being explained, and the ICM-based expectations are checked against a meta-analysis of over 100 published SSL and 30 DA studies, not against a fitted version of those results. The MDL assay in Section 6.4 is a separate empirical probe of ICM on NLP data. Part IV is standard applied causal inference with confounder control. The thesis does cite the author's own published papers, but those citations are to the peer-reviewed versions of the same chapters and are not invoked as unverified premises or as an imported uniqueness theorem; the load-bearing evidence is the open-sourced data generation code, the external model evaluations, and the published empirical literature. Therefore no circularity is present.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The thesis introduces no new physical or statistical entities; its inventions are datasets, benchmarks, and methods (Corr2Cause, CLadder, CausalCoT, TextMatch). The central claims depend on dataset construction assumptions (closed world, faithfulness, identifiability) and on transferring the ICM principle to NLP.

free parameters (2)
  • Attention multiplier alpha = alpha=5
    In Chapter 4 (Eq. 4.5), alpha is chosen via grid search over [2,5,10,100] to maximize the factual recall rate, and the intervention result depends on this choice.
  • Numerical space cap C = C=300
    In Chapter 5 (Section 5.4.2), the range of integer values considered is capped at 300 to control costs and keep numbers as single tokens; this constrains the intervention space.
assumptions (4)
  • domain assumption Closed system, no hidden confounders
    Corr2Cause assumes a closed system of the given variables and no unobserved causes (Section 2.3.6). This makes the task tractable but potentially unrepresentative of real causal discovery.
  • standard math Faithfulness
    Corr2Cause relies on faithfulness (Section 2.2.2) to equate d-separation with conditional independence, enabling MEC-based labeling.
  • standard math Identifiability by construction in CLadder
    CLadder constructs graphs, queries, and data so the query is always identifiable, then uses do-calculus to derive the estimand (Section 3.3.1).
  • domain assumption ICM principle applies to NLP data
    Chapter 6 transfers the independent causal mechanisms principle from causal inference to NLP text, and relies on the causal/anticausal categorization of tasks (Table 6.1). This is tested only partially via MDL on MT data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Causality for Natural Language Processing." pith.science (2026). https://pith.science/paper/73F5BZZX

@misc{pith2026250414530,
  author       = {Pith},
  title        = {Pith review of: Causality for Natural Language Processing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/73F5BZZX}},
  note         = {Machine review of arXiv:2504.14530}
}
read the original abstract

Causal reasoning is a cornerstone of human intelligence and a critical capability for artificial systems aiming to achieve advanced understanding and decision-making. This thesis delves into various dimensions of causal reasoning and understanding in large language models (LLMs). It encompasses a series of studies that explore the causal inference skills of LLMs, the mechanisms behind their performance, and the implications of causal and anticausal learning for natural language processing (NLP) tasks. Additionally, it investigates the application of causal reasoning in text-based computational social science, specifically focusing on political decision-making and the evaluation of scientific impact through citations. Through novel datasets, benchmark tasks, and methodological frameworks, this work identifies key challenges and opportunities to improve the causal capabilities of LLMs, providing a comprehensive foundation for future research in this evolving field.

Figures

Figures reproduced from arXiv: 2504.14530 by the authors.

Figure 2.1
Figure 2.1. Illustration of the motivation behind our task and dataset. [PITH_FULL_IMAGE:figures/full_fig_p018_2_1.png] view at source ↗
Figure 2.2
Figure 2.2. Pipeline of the data construction process. [PITH_FULL_IMAGE:figures/full_fig_p021_2_2.png] view at source ↗
Figure 3.1
Figure 3.1. Example question in our CLadder dataset featuring an instance of Simpson’s paradox (Pearl, 2022). We generate the following (symbolic) triple: (i) the causal query; (ii) the ground-truth answer, derived through a causal inference engine (Pearl and Mackenzie, 2018); and (iii) a step-by-step explanation. We then verbalize these questions by turning them into stories, inspired by examples from the causality literature,… view at source ↗
Figures from the paper (32 more)
Figure 3.2
Figure 3.2. Figure 3.2: The data-generating process of the CLadder dataset. The upper part of the figure describes the formal part of the question generation, which samples inputs for the CI Engine and derives a ground truth answer. The bottom part describes the natural lan￾guage part of th…
Figure 3.3
Figure 3.3. Figure 3.3: Distribu￾tions of query types in our 10K data. words (i.e., 98.74% correctness), which shows that most of the language in our dataset follows English grammar. For human readability, we checked how comprehensible the questions are to students who have taken causality …
Figure 3.4
Figure 3.4. Figure 3.4: Illustration of our CausalCoT prompting strategy, which designs a chain of subquestions inspired by the idea of a CI engine (Pearl and Mackenzie, 2018). We build our CausalCoT prompting strategy using GPT-4 (OpenAI, 2023), a recent au￾toregressive LLM that achieves s…
Figure 3.5
Figure 3.5. Figure 3.5: Heatmap showing the how helpful each query type is to solving subsequent query types. As an additional analysis, we look into the effect of in-context learning (ICL) by providing an example solution before ask￾ing the question. The interesting question to us is wheth…
Figure 4.1
Figure 4.1. Figure 4.1: Top: An example showing that LLMs can fail to recognize the correct mech [PITH_FULL_IMAGE:figures/full_fig_p047_4_1.png]
Figure 4.2
Figure 4.2. Figure 4.2: Logits of the factual token tfact and counterfactual token tcofa across different positions and layers in GPT-2. The logit of tfact is higher in the subject position in the initial layers and in the last position of the premise and second sentence in the final layers…
Figure 4.3
Figure 4.3. Figure 4.3: Contributions of the attention and MLP blocks to the competition of the mech [PITH_FULL_IMAGE:figures/full_fig_p052_4_3.png]
Figure 4.4
Figure 4.4. Figure 4.4: Attention pattern for relevant attention heads. [PITH_FULL_IMAGE:figures/full_fig_p054_4_4.png]
Figure 4.5
Figure 4.5. Figure 4.5: The factual recall mechanism increases substantially across GPT-2 and Pythia [PITH_FULL_IMAGE:figures/full_fig_p056_4_5.png]
Figure 4.6
Figure 4.6. Figure 4.6: Prediction frequency of factual token by similarity level. We show the percent [PITH_FULL_IMAGE:figures/full_fig_p057_4_6.png]
Figure 5.1
Figure 5.1. Figure 5.1: Through our framework, we conduct do-interventions on the input and eval [PITH_FULL_IMAGE:figures/full_fig_p061_5_1.png]
Figure 5.2
Figure 5.2. Figure 5.2: Causal graph of model predictions on math questions. We highlight the dif [PITH_FULL_IMAGE:figures/full_fig_p062_5_2.png]
Figure 5.3
Figure 5.3. Figure 5.3: Comparison of DCE(N → R) and TCE(N on R). ∗approx values, see Ap￾pendix A.4.3 [PITH_FULL_IMAGE:figures/full_fig_p069_5_3.png]
Figure 5.4
Figure 5.4. Figure 5.4: Heatmaps displaying P(g)for Distil-GPT-2 (left), GPT-J-6B (center), and GPT-3 Davinci-002 (right). g is the ground-truth result g = n1 +n2 (n1 and n2 are represented by the x and y axes, respectively. The probability values for each combination of ((n1, n2), g) are a…
Figure 5.5
Figure 5.5. Figure 5.5: Comparison of DCE(S → R) and TCE(T on R). We use ∗ to denote approxi￾mated values, explained in Appendix A.4.3 [PITH_FULL_IMAGE:figures/full_fig_p075_5_5.png]
Figure 5.6
Figure 5.6. Figure 5.6: Comparison of direct and total effects of [PITH_FULL_IMAGE:figures/full_fig_p076_5_6.png]
Figure 5.7
Figure 5.7. Figure 5.7: Comparison of direct and total effects of [PITH_FULL_IMAGE:figures/full_fig_p076_5_7.png]
Figure 6.1
Figure 6.1. Figure 6.1: Annotation process for NLP data: the random variable that exists first is typ [PITH_FULL_IMAGE:figures/full_fig_p080_6_1.png]
Figure 6.2
Figure 6.2. Figure 6.2: (Top) A causal graph C → E, where C is the cause and E is the effect. The function f(·, NE) denotes the causal process, or mechanism, PE|C by which the effect E is generated from C and unobserved noise NE. (Bottom) Based on whether the direction of prediction aligns …
Figure 6.3
Figure 6.3. Figure 6.3: The ICM principle assumes that the generative process PC of the cause C is indepen￾dent of the causal mechanism PE|C: the two distributions share no information and each may be changed or manipulated without affecting the other. In the anticausal direction, on the ot…
Figure 7.1
Figure 7.1. Figure 7.1: An overview of the paper structure, where we first investigate the causal dis [PITH_FULL_IMAGE:figures/full_fig_p094_7_1.png]
Figure 7.1
Figure 7.1. Figure 7.1: Contextualizing it in the task of SA, we formally define an emotion arc of the [PITH_FULL_IMAGE:figures/full_fig_p098_7_1.png]
Figure 7.2
Figure 7.2. Figure 7.2: Causal attribution in LLaMa-7B and Alpaca-7B, showing how much each sen [PITH_FULL_IMAGE:figures/full_fig_p103_7_2.png]
Figure 7
Figure 7. Figure 7: in the row of “Prompt C1”. On the other hand, Prompt C2 should trigger [PITH_FULL_IMAGE:figures/full_fig_p103_7.png]
Figure 8.1
Figure 8.1. Figure 8.1: The data collection pipeline and architecture of our system to predict the state [PITH_FULL_IMAGE:figures/full_fig_p109_8_1.png]
Figure 8.2
Figure 8.2. Figure 8.2: Backdoor Adjustment. To formulate the causal impact, Pearl (1995) defines a language for causality called do￾calculus, by which the causal impact of X on Y is formulated as the interventional distri￾bution: P(Y|do(X)) , (8.1) where do(X) refers to an intervention on …
Figure 8
Figure 8. Figure 8: , there are two ways how [PITH_FULL_IMAGE:figures/full_fig_p117_8.png]
Figure 8.3
Figure 8.3. Figure 8.3: Comparative study of states. MS and GA is a pair of states with the most [PITH_FULL_IMAGE:figures/full_fig_p120_8_3.png]
Figure 9.1
Figure 9.1. Figure 9.1: An overview of our research question. framework (Hernán and Robins, 2010) to quantify a paper’s impact by how much of the academic success in the follow-up papers should be causally attributed to this paper. We introduce CausalCite, an enhanced citation based metric …
Figure 9.2
Figure 9.2. Figure 9.2: The causal graph of our study. colliders (e.g., both the influence from a and the citations of b influencing later awards received by b). 9.2.1 CausalCite Indices In this section, we introduce various indices that measure the causal impact of a paper. Two-Paper Inter…
Figure 9.3
Figure 9.3. Figure 9.3: Distributions of ToT (mean: 142) and non￾ToT papers (mean: 1,623). Random Features for Large-Scale Kernel Machines (NeurIPS 2017) BLEU Metric (NAACL 2018) ImageNet Dataset (CVPR 2019) 10 1 10 2 10 3 10 4 CausalCite Non-ToT ToT [PITH_FULL_IMAGE:figures/full_fig_p134_…
Figure 9.5
Figure 9.5. Figure 9.5: The distribution of TCI values by percentile of 100 random papers, which [PITH_FULL_IMAGE:figures/full_fig_p135_9_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 7 canonical work pages

  1. [4]

    How to Make Causal Inferences Using Texts

    Religion and reactance to covid-19 mitigation guidelines. American Psychologist. 99 Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 24...

    work page Pith review arXiv 2009

  2. [11]

    Global Sentiment Analysis Of COVID-19 Tweets Over Time

    Global sentiment analysis of COVID-19 tweets over time. CoRR, abs/2010.14234. 102 Micol Marchetti-Bowick and Nathanael Chambers. 2012. Learning for microblogs with distant supervision: Political forecasting with Twitter. In Proceedings of the 13th Confer- ence of the European Chapter of the Association for Computational Linguistics, pages 603–612, Avignon...

    work page Pith review arXiv 2010

  3. [12]

    CoRR, abs/2310.04625

    Copy suppression: Comprehensively understanding an attention head. CoRR, abs/2310.04625. 45, 47 Matt McGue, Merete Osler, and Kaare Christensen. 2010. Causal inference and observa- tional research: The utility of twins. Perspectives on psychological science , 5(5):546–556. 118 Brendan D. McKay and Adolfo Piperno. 2014. Practical graph isomorphism, II. J. ...

    arXiv 2010

  4. [13]

    CoRR, abs/2306.09479

    Inverse scaling: When bigger isn’t better. CoRR, abs/2306.09479. 47 Sergio Hernan Garrido Mejia, Elke Kirschbaum, and Dominik Janzing. 2022. Obtaining causal information by merging datasets with MAXENT. In International Conference on Artificial Intelligence and Statistics, AISTATS 2022, 28-30 March 2022, Virtual Event , vol- ume 151 of Proceedings of Mach...

    arXiv 2022

  5. [15]

    38, 39 Tetsuya Nasukawa and Jeonghee Yi

    OpenReview.net. 38, 39 Tetsuya Nasukawa and Jeonghee Yi. 2003. Sentiment analysis: Capturing favorability using natural language processing. In Proceedings of the 2nd international conference on Knowledge capture, pages 70–77. ACM. 83, 93 Brady Neal. 2020. Introduction to causal inference. 135 Jingwei Ni, Zhijing Jin, Markus Freitag, Mrinmaya Sachan, and ...

    work page 2003

  6. [16]

    CoRR, abs/2303.13375

    Capabilities of GPT-4 on medical challenge problems. CoRR, abs/2303.13375. 29, 32 Nostalgebraist. 2020. interpreting gpt: the logit lens. Accessed: Nov 2023. 37, 40 Maxwell I. Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Oden...

    arXiv 2020

  7. [20]

    114, 121 Paolo Manghi, Andrea Mannocci, Francesco Osborne, Dimitris Sacharidis, Angelo Salatino, and Thanasis Vergoulis. 2021. New trends in scientific knowledge graphs and research impact assessment. 127 Kamaran H Manguri, Rebaz N Ramadhan, and Pshko R Mohammed Amin. 2020. Twit- ter sentiment analysis on worldwide covid-19 outbreaks. Kurdistan Journal of...

    work page 2021

  8. [21]

    ArXiv, abs/1905.06316

    What do you learn from context? probing for sentence structure in contextualized word representations. ArXiv, abs/1905.06316. 38 Avijit Thawani, Jay Pujara, Filip Ilievski, and Pedro Szekely. 2021. Representing numbers in NLP: a survey and a vision. In Proceedings of the 2021 Conference of the North Ameri- can Chapter of the Association for Computational ...

    arXiv 1905

Show all 23 references
  1. [499]

    101 Martin M Monti, Lawrence M Parsons, and Daniel N Osherson. 2012. Thought beyond language: Neural dissociation of algebra and natural language. Psychological science, 23(8):914–922. 53 Joris M. Mooij, Jonas Peters, Dominik Janzing, Jakob Zscheischler, and Bernhard Schölkopf...

    2012 arXiv

  2. [672]

    22, 23 Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2022. Roscoe: A suite of metrics for scoring step- by-step reasoning. 31, 143 Charles J Gomez, Andrew C Herman, and Paolo Parigi. 2022. Leading countr...

    2022 arXiv

  3. [731]

    94 Bruce Schneier. 1996. Applied cryptography. John Willey and Sons Inc.,. 77 Bernhard Schölkopf. 2022. Causality for machine learning. In Hector Geffner, Rina Dechter, and Joseph Y. Halpern, editors, Probabilistic and Causal Inference: The Works of Judea Pearl, volume 36 of A...

    1996 arXiv

  4. [1103]

    78 Elena Voita and Ivan Titov

    ACM. 78 Elena Voita and Ivan Titov. 2020. Information-theoretic probing with minimum descrip- tion length. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020 , pages 183–196. Association for Compu...

    2020 arXiv

  5. [1993]

    Psychological science, 4(6):401–405

    When more pain is preferred to less: Adding a better end. Psychological science, 4(6):401–405. 84, 88, 94 Immanuel Kant. 1781. Critique of Pure Reason. Cambridge University Press. 7 BIBLIOGRAPHY • 197 Karim S Kassam and Wendy Berry Mendes. 2013. The effects of measuring emotio...

    2013 arXiv

  6. [2008]

    Nonlinear causal discovery with additive noise models. In Advances in Neural Information Processing Systems 21, Proceedings of the Twenty-Second Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 8-11, 2008, volume 21, pag...

    2008 arXiv

  7. [2010]

    In 26th Conference on Uncertainty in Arti- ficial Intelligence, pages 143–150, Corvallis, OR

    Inferring deterministic causal relations. In 26th Conference on Uncertainty in Arti- ficial Intelligence, pages 143–150, Corvallis, OR. AUAI Press. Best student paper award. 73 Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2023. Analyzing transformers in embedding space...

    2023 arXiv

  8. [2013]

    20 Janyce M Wiebe

    Morbidity and Mortality Weekly Report, 63(16):352. 20 Janyce M Wiebe. 1994. Tracking point of view in narrative. Computational Linguistics, 20(2):233–287. 83, 93 Moritz Willig, Matej Zečević, Devendra Singh Dhami, and Kristian Kersting. 2023. Prob- ing for correlations of caus...

    1994 arXiv

  9. [2015]

    In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 134–140

    Montreal neural machine translation systems for wmt’15. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 134–140. 81 Chaker Jebari, Enrique Herrera-Viedma, and Manuel Jesus Cobo. 2021. The use of citation context to detect the evolution of researc...

    2021 arXiv

  10. [2018]

    Advances in Neural Information Processing Systems, 31

    Learning pipelines with limited data and domain knowledge: A study in parsing physics problems. Advances in Neural Information Processing Systems, 31. 50 Mrinmaya Sachan and Eric Xing. 2017. Learning to solve geometry problems from natural language demonstrations in textbooks....

    2017 arXiv

  11. [2019]

    multilingual

    Language models are unsupervised multitask learners. OpenAI Blog, 1(8). 1, 7, 14, 20, 29, 32, 41, 57, 76, 91, 160, 176 Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of tr...

    2020

  12. [2020]

    InAdvances in Neural Information Process- ing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual

    Language models are few-shot learners. InAdvances in Neural Information Process- ing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. 1, 14, 20, 29, 30, 32, 36, 50, 58, 91, 176 BIBLIOGRAPHY • 185 Sébastien...

    2020 arXiv

  13. [2022]

    CoRR, abs/2205.11916

    Large language models are zero-shot reasoners. CoRR, abs/2205.11916. 62 Andrei N Kolmogorov. 1965. Three approaches to the quantitative definition of informa- tion. Problems of information transmission, 1(1):1–7. 74 Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, ...

    1965 arXiv

  14. [2023]

    In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430

    Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR. 41 Felix Bittmann...

    2023 arXiv

  15. [2024]

    Linguistics, 50(1):237–291

    Can large language models transform computational social science? Comput. Linguistics, 50(1):237–291. 29, 32 Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexan- der Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel...

    2023 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.