Pith. sign in

REVIEW 4 major objections 6 minor 41 references

CausalFlip: A benchmark that forces LLM causal judgments to be grounded in causal structure, not semantic similarity.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 21:23 UTC pith:MBTXUPVP

load-bearing objection Solid benchmark design, but the 'implicit causal reasoning' method does not actually internalize reasoning; the empirical claims are thinner than the title suggests. the 4 major comments →

arxiv 2602.20094 v2 pith:MBTXUPVP submitted 2026-02-23 cs.AI

CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matching

Yuzhe Wang , Yaochen Zhu , Jundong Li This is my paper
classification cs.AI
keywords causal reasoninglarge language modelssemantic matchingspurious correlationchain-of-thoughtimplicit causal reasoningbenchmarkconfounder chain collider
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that large language models can score well on causal-judgment tasks by matching semantic patterns rather than reasoning about causal structure, and that this shortcut can be deliberately exposed and mitigated. It introduces CausalFlip, a benchmark of confounder, chain, and collider questions built from event triples, where each question has a semantically near-twin with the opposite label; a pairwise train-test split means a model that leans on semantic similarity will be systematically wrong. The paper also claims that explicit chain-of-thought supervision, while improving accuracy, still relies on spurious semantics: adding a causally irrelevant fixed prefix before the reasoning steps drops explicit-CoT accuracy much more than it drops accuracy for implicit causal reasoning, a training strategy that progressively masks reasoning-step tokens. A sympathetic reader would care because deploying LLMs in high-stakes settings requires causal grounding, and the benchmark offers a way to measure and train for it.

Core claim

The paper's core discovery is that semantic matching, not causal structure, can drive LLM performance on causal questions, and that this reliance can be measured and reduced. CausalFlip pairs semantically similar questions over the same event triple so that the correct answer flips across the pair; with one question in train and its twin in test, any model answering by lexical or template similarity is pushed toward the wrong label. Across all three causal structures (confounder, chain, collider), answer-only fine-tuning stays near chance, explicit CoT supervision reaches higher accuracy, and the proposed implicit causal reasoning—which progressively removes the earliest reasoning tokens fro

What carries the argument

The load-bearing object is CausalFlip's paired, label-flipped question construction: for each event triple (X,Y,Z) that realizes one of three structures—confounder, chain, collider—two templates ask a direct-effect question and a structure question, and Base versus Opposite structures flip the labels. A pairwise train-test split guarantees every test question has a same-semantics training twin with the opposite answer, so semantic matching is penalized. The second mechanism is the noisy-prefix evaluation: a fixed, causally irrelevant prefix inserted before the CoT steps, used to compare explicit CoT with implicit causal reasoning. The third is the training strategy itself: a progressive mask

Load-bearing premise

The load-bearing premise is that prepending the same fixed, causally irrelevant sentence before the reasoning steps is causally inert, so that any accuracy loss measures reliance on spurious semantics—but that prefix is also an input distribution the model never saw during training, so part of the loss could be ordinary distribution shift.

What would settle it

A direct check would use several different causally irrelevant prefixes at evaluation, not just one fixed prefix. If explicit CoT's accuracy drop nearly disappears when the prefix is varied or made shorter, while the fixed-prefix drop remains large, then the reported degradation is tied to the specific repeated prefix rather than to generic semantic noise; conversely, if every semantically irrelevant perturbation produces a similar larger drop for explicit CoT, the paper's interpretation is supported.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Answer-only fine-tuning stays near chance on CausalFlip (0.52–0.66 across datasets), so high accuracy on standard causal benchmarks is not evidence of causal grounding.
  • Supervision on intermediate causal reasoning steps is the main driver of improved accuracy; explicit CoT and implicit reasoning both outperform no-CoT.
  • Because explicit CoT drops more under a causally irrelevant prefix, even CoT-trained models can be sensitive to semantic noise rather than anchored to the underlying causal graph.
  • Progressively masking reasoning tokens during training reduces that sensitivity and maintains higher accuracy under the noisy prefix.
  • CausalFlip can serve as a diagnostic separating semantic matching from causal structure in LLM training schemes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension would be to vary the noisy prefix randomly per example; if the accuracy drop for explicit CoT persists only for a fixed repeated prefix, part of the effect could be memorization of that specific prefix's correlation with the answer rather than semantic reliance generally.
  • The progressive-masking idea need not be confined to causal questions; it could be tested on other multi-step reasoning benchmarks where internalization might trade away interpretability for robustness.
  • The benchmark's template balance (Default/Alternative × Base/Opposite) suggests a practical recipe for other domains: adversarially pair training and test instances that share surface form but require opposite decisions to force structure-based reasoning.
  • The noisy-prefix drop for explicit CoT could partly reflect ordinary out-of-distribution perturbation in addition to spurious-semantic reliance; a control that fine-tunes with varied prefixes would disentangle these.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents CausalFlip, a benchmark for evaluating causal judgment in LLMs. It builds binary causal questions over event triples under confounder, chain, and collider structures, with Base/Opposite structure variants and Default/Alternative templates. For each question pair, one item goes to training and the semantically similar counterpart with the opposite label goes to testing, so that semantic matching is penalized. The paper also proposes an 'implicit causal reasoning' training strategy that progressively masks the loss on early CoT tokens, and compares it with a pretrained baseline, answer-only fine-tuning, and explicit-CoT fine-tuning on Llama-3.2-3B-Instruct. A noisy-prefix evaluation is introduced to measure reliance on spurious semantic correlations. The authors report that explicit CoT degrades more than implicit causal reasoning under the noisy prefix, and conclude that internalizing reasoning steps improves causal grounding and robustness.

Significance. CausalFlip is a well-motivated and potentially useful benchmark: the label-flipped paired questions, the Base/Opposite balance, and the two template families are thoughtful design choices that go beyond existing causal benchmarks. The code and data release is a strength, and the pairwise split is a legitimate stress test for shortcut learning. However, the paper's central methodological claim about 'internalized causal reasoning' is not supported by the implemented loss in Eq. (10), and the noisy-prefix evidence is confounded with ordinary distribution shift. Given the benchmark's value, the issues are fixable, but the empirical and methodological support for the headline conclusions is currently insufficient.

major comments (4)
  1. [§4.2.2, Eq. (10)] The proposed loss masks only the target tokens of the first r(t) reasoning tokens, but those tokens remain in the conditioning context for all later tokens and for the final-answer term log p(y|x,s). The model is therefore never trained to make the causal judgment without the full explicit chain as input; at inference (§5.3.2) it still generates the chain. This does not implement 'internalized causal reasoning' as claimed in §4.1 and §7. To support the claim, the method must either drop masked tokens from the context (as in the cited implicit-CoT work) or include an answer-only training stage; otherwise the observed robustness may reflect learned insensitivity to unreliable early tokens, not internalization.
  2. [§5.3.4, §5.5] The noisy-prefix evaluation prepends a fixed, never-seen prefix to the prompt. Any accuracy drop could be due to ordinary out-of-distribution sensitivity to a novel input pattern, not specifically to reliance on spurious semantic correlations. The paper does not include control perturbations (e.g., a non-semantic token sequence, multiple different prefixes, or a prefix with no semantic content), nor does it provide evidence that the effect is semantically mediated. The prefix text is also not shown, which hampers reproducibility. The RQ2 conclusion that implicit causal reasoning 'reduces reliance on spurious semantic correlations' is therefore not identified by this experiment.
  3. [Tables 1–3, Figure 3] All results are single runs with no error bars, confidence intervals, or significance tests. Several differences are within sampling noise: on Confounder clean, explicit-CoT 0.892 vs. implicit 0.900 is 8 answers out of 1000; on Collider clean, explicit-CoT is actually higher (0.856 vs. 0.849). The blanket statement that implicit causal reasoning is 'consistently more robust' and 'substantially improved' overstates what the data show. At least three seeds with variance reporting, or a significance test, are needed before drawing these conclusions.
  4. [§4.2.2, §A.2] The method's only novel component is the progressive masking schedule r(t), but the paper never specifies it. The reader cannot know whether r(t) is linear, stepwise, what fraction of tokens is eventually masked, or how it interacts with the 3-epoch fine-tuning. This is a reproducibility issue for the central method, and the schedule should be explicitly reported.
minor comments (6)
  1. [§5.3.4] The text says 'we inject the noisy prefix before the intermediate causal reasoning steps in the training samples' but the evaluation is presumably on test samples at inference. Please clarify where and when the prefix is applied.
  2. [§3.2.2] There is a typo: an extra parenthesis in '(to keep this question semantically close to Question(ii)))'.
  3. [§5.3.1] Only one example of a causal reasoning step is given. The full set of CoT templates used for chain, collider, and Alternative-phrasing questions should be included in the appendix for reproducibility.
  4. [§1, §3] Please report basic dataset statistics: number of event triples, number of train/test questions per sub-dataset, and counts per category (BD/BA/OD/OA). This is needed to interpret the 1000-example test tables.
  5. [§5.3.4] The noisy-prefix text is not disclosed. Please include it verbatim so that the robustness experiment can be reproduced exactly.
  6. [§2.2] Minor typo: 'a singlereasoning steps' rollout' should be 'a single reasoning step rollout'.

Circularity Check

0 steps flagged

No significant circularity: CausalFlip is an intentionally constructed benchmark and the training recipes are empirical; the main concerns are validity and generalization, not circularity.

full rationale

CausalFlip's labels are prescribed from the causal graphs in Section 3.2; no label or parameter is fitted from a model and then reported as a prediction. The pairwise train-test split is an intentional benchmark construction—every test question has a training counterpart with the same events and opposite label—so the claim that semantic matching fails is a design property, not a circular derivation. Accuracy is computed by Eq. (11) on held-out test pairs after LoRA fine-tuning (Section A.2), and the losses in Eqs. (8)-(10) are standard supervision objectives. There are no self-citation loops: the directly related prior method [5] is external prior work, and no uniqueness theorem or load-bearing self-citation is invoked. The skeptical concern about Eq. (10)—the final-answer term log p(y | x, s) still conditions on the full explicit chain s even when early reasoning tokens are masked—is a real correctness/validity concern about whether the method actually implements 'internalization,' but it is a mismatch between the claimed mechanism and the implemented objective, not a reduction of a prediction to its own input. Likewise, the noisy-prefix evaluation in Sections 5.3.4 and 5.5 assumes that a fixed, causally irrelevant prefix is a clean semantic probe rather than an out-of-distribution perturbation; that is a validity risk, not circularity. The paper itself flags the central uncertainty in Section 1: 'it remains unclear whether such internalization better encourages the model to internalize causal reasoning model weights during the forward propagation,' which further supports treating the issue as an open empirical question rather than a circular claim. Overall, the derivation chain is self-contained and no circular step is exhibited.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The paper introduces no new physical or mathematical entities. It relies on the standard probabilistic LLM factorization and on domain assumptions about how natural-language event triples realize causal graphs and how the noisy-prefix perturbation should be interpreted. The main hand-chosen components are the unspecified masking schedule and the unshown noisy-prefix text.

free parameters (3)
  • masking schedule r(t)
    Controls how many leading reasoning tokens are dropped from supervision at each training step. No functional form or schedule is given in §4.2.2; it is central to the proposed implicit causal reasoning method and is a hand-chosen hyperparameter.
  • fine-tuning hyperparameters (LoRA r=4, alpha=8, dropout=0.05, 3 epochs, lr=1e-4, batch 4) = r=4, alpha=8, dropout=0.05, 3 epochs, lr=1e-4, batch 4
    These are standard chosen hyperparameters, reported in Appendix A.2, but no sensitivity analysis is given; they affect the reported accuracy comparisons.
  • noisy-prefix text = fixed text not shown
    The fixed chunk of causally irrelevant text is a design choice central to the noisy-prefix evaluation. Its exact content is not given, so the measured degradation cannot be independently reproduced or checked for unintended semantic cues.
axioms (5)
  • standard math Autoregressive factorization p_theta(x)=prod_t p_theta(x_t | x_<t) and next-token softmax
    Used in §2.1 to frame LLM behavior and to argue that next-token training can exploit spurious correlations; standard probabilistic modeling assumption.
  • domain assumption Correct causal labels are fully determined by the graph structures Z->X,Z->Y; X->Y->Z; X->Z,Y->Z and their opposites
    Benchmark construction in §3.2 assumes natural-language event triples unambiguously realize these structures and that the generated questions' labels follow from them.
  • ad hoc to paper A fixed noisy prefix is causally irrelevant and does not alter the underlying causal relations or logic of reasoning
    Invoked in §5.3.4 to interpret accuracy drop as semantic reliance; not independently verified and not controlled against general distribution shift.
  • domain assumption Manually authored CoT strings (e.g., §5.3.1) are the correct intermediate reasoning for all paired questions
    The supervision signal for explicit/implicit training is a single canonical reasoning string per dataset; its sufficiency and validity are assumed.
  • domain assumption Semantic matching under the pairwise split is systematically penalized
    The design rationale in §3.1 assumes lexical overlap between paired questions is the dominant shortcut and that template and structure balance prevents template-label correlation.

pith-pipeline@v1.3.0-alltime-deepseek · 15056 in / 12581 out tokens · 112681 ms · 2026-08-02T21:23:48.955511+00:00 · methodology

0 comments
read the original abstract

As large language models (LLMs) witness increasing deployment in complex, high-stakes decision-making scenarios, it becomes imperative to ground their reasoning in causality rather than spurious correlations. However, strong performance on traditional reasoning benchmarks does not guarantee true causal reasoning ability of LLMs, as high accuracy may still arise from memorizing semantic patterns instead of analyzing the underlying true causal structures. To bridge this critical gap, we propose a new causal reasoning benchmark, CausalFlip, designed to encourage the development of new LLM paradigm or training algorithms that ground LLM reasoning in causality rather than semantic correlation. CausalFlip consists of causal judgment questions built over event triples that could form different confounder, chain, and collider relations. Based on this, for each event triple, we construct pairs of semantically similar questions that reuse the same events but yield opposite causal answers, where models that rely heavily on semantic matching are systematically driven toward incorrect predictions. To further probe models' reliance on semantic patterns, we introduce a noisy-prefix evaluation that prepends causally irrelevant text before intermediate causal reasoning steps without altering the underlying causal relations or the logic of the reasoning process. We evaluate LLMs under multiple training paradigms, including answer-only training, explicit Chain-of-Thought (CoT) supervision, and a proposed internalized causal reasoning approach that aims to mitigate explicit reliance on correlation in the reasoning process. Our results show that explicit CoT can still be misled by spurious semantic correlations, where internalizing reasoning steps yields substantially improved causal grounding, suggesting that it is promising to better elicit the latent causal reasoning capabilities of base LLMs.

Figures

Figures reproduced from arXiv: 2602.20094 by Jundong Li, Yaochen Zhu, Yuzhe Wang.

Figure 1
Figure 1. Figure 1: One representative example where training samples [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the causal structures used in our benchmark and an example of base, opposite pairs. The top row shows [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Accuracy of explicit-CoT versus implicit causal reasoning on CausalFlip across the three sub-datasets under clean [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 29 linked inside Pith

  1. [1]

    Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model.J. Mach. Learn. Res.3, null (March 2003), 1137–1155

    2003

  2. [2]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    Pith/arXiv arXiv 2020

  3. [3]

    Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training Deep Nets with Sublinear Memory Cost. arXiv:1604.06174 [cs.LG] https://arxiv. org/abs/1604.06174

    Pith/arXiv arXiv 2016

  4. [4]

    Jeffrey Cheng and Benjamin Van Durme. 2024. Compressed Chain of Thought: Efficient Reasoning Through Dense Representations. arXiv:2412.13171 [cs.CL] CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matching https://arxiv.org/abs/2412.13171

    Pith/arXiv arXiv 2024

  5. [5]

    Yuntian Deng, Yejin Choi, and Stuart Shieber. 2024. From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step. arXiv:2405.14838 [cs.CL] https: //arxiv.org/abs/2405.14838

    Pith/arXiv arXiv 2024

  6. [6]

    Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi

    Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. 2023. Faith and Fate: Limits of Transformers on Compositionality. arXiv:2305.18654 [cs.CL] https://arxiv.org/abs/2305.18654

    Pith/arXiv arXiv 2023

  7. [7]

    Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical Neural Story Generation. arXiv:1805.04833 [cs.CL] https://arxiv.org/abs/1805.04833

    Pith/arXiv arXiv 2018

  8. [8]

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The Curious Case of Neural Text Degeneration. arXiv:1904.09751 [cs.CL] https: //arxiv.org/abs/1904.09751

    Pith/arXiv arXiv 2020

  9. [9]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685 [cs.CL] https://arxiv.org/abs/2106.09685

    Pith/arXiv arXiv 2021

  10. [10]

    Zhijing Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojasv Kamal, Zhiheng Lyu, Kevin Blin, Fernando Gonzalez Adauto, Max Kleiman-Weiner, Mrinmaya Sachan, and Bernhard Schölkopf. 2024. CLadder: Assessing Causal Reasoning in Language Models. arXiv:2312.04350 [cs.CL] https://arxiv.org/abs/2312.04350

    Pith/arXiv arXiv 2024

  11. [11]

    Zhijing Jin, Jiarui Liu, Zhiheng Lyu, Spencer Poff, Mrinmaya Sachan, Rada Mihalcea, Mona Diab, and Bernhard Schölkopf. 2024. Can Large Language Models Infer Causation from Correlation? arXiv:2306.05836 [cs.CL] https: //arxiv.org/abs/2306.05836

    Pith/arXiv arXiv 2024

  12. [12]

    Vempala, and Edwin Zhang

    Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang

  13. [13]

    Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, et al. 2019. A Study of BFLOAT16 for Deep Learning Training. arXiv:1905.12322 [cs.LG] https: //arxiv.org/abs/1905.12322

    Pith/arXiv arXiv 2019

  14. [14]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023. Large Language Models are Zero-Shot Reasoners. arXiv:2205.11916 [cs.CL] https://arxiv.org/abs/2205.11916

    Pith/arXiv arXiv 2023

  15. [15]

    Emre Kıcıman, Robert Ness, Amit Sharma, and Chenhao Tan. 2024. Causal Reasoning and Large Language Models: Opening a New Frontier for Causality. arXiv:2305.00050 [cs.AI] https://arxiv.org/abs/2305.00050

    Pith/arXiv arXiv 2024

  16. [16]

    Jinqi Lai, Wensheng Gan, Jiayang Wu, Zhenlian Qi, and Philip S. Yu. 2024. Large language models in law: A survey.AI Open5 (2024), 181–196. doi:10.1016/j. aiopen.2024.09.002

    doi:10.1016/j 2024

  17. [17]

    Yuxuan Lai, Chen Zhang, Yansong Feng, Quzhe Huang, and Dongyan Zhao

  18. [18]

    Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner. 2019. Reasoning Over Paragraph Effects in Situations. arXiv:1908.05852 [cs.CL] https://arxiv.org/abs/ 1908.05852

    Pith/arXiv arXiv 2019

  19. [19]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring How Models Mimic Human Falsehoods. arXiv:2109.07958 [cs.CL] https://arxiv. org/abs/2109.07958

    Pith/arXiv arXiv 2022

  20. [20]

    Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. arXiv:1711.05101 [cs.LG] https://arxiv.org/abs/1711.05101

    Pith/arXiv arXiv 2019

  21. [21]

    Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald

  22. [22]

    Thomas McCoy, Ellie Pavlick, and Tal Linzen

    R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference. arXiv:1902.01007 [cs.CL] https://arxiv.org/abs/1902.01007

    Pith/arXiv arXiv 2019

  23. [23]

    Daniel McDuff, Mike Schaekermann, Tao Tu, Anil Palepu, Amy Wang, Jake Garrison, Karan Singhal, Yash Sharma, Shekoofeh Azizi, Kavita Kulkarni, Le Hou, Yong Cheng, Yun Liu, S Sara Mahdavi, Sushant Prakash, Anupam Pathak, Christopher Semturs, Shwetak Patel, Dale R Webster, Ewa Dominowska, Juraj Gottweis, Joelle Barral, Katherine Chou, Greg S Corrado, Yossi M...

    Pith/arXiv arXiv 2023

  24. [24]

    Alec Radford and Karthik Narasimhan. 2018. Improving Language Understanding by Generative Pre-Training. https://api.semanticscholar.org/CorpusID:49313245

    2018

  25. [25]

    Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon

  26. [26]

    Junhan Shi, Yijia Zhu, Zhenning Shi, Dan Zhao, Qing Li, and Yong Jiang. 2025. SpecCoT: Accelerating Chain-of-Thought Reasoning through Speculative Ex- ploration. InFindings of the Association for Computational Linguistics: EMNLP 2025, Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (Eds.). Association for Computational Lingu...

    doi:10.18653/v1/2025.findings-emnlp.1326 2025

  27. [27]

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Nathaneal Scharli, Aakanksha Chowdhery, Philip Mansfield, Blaise Aguera y Arcas, Dale Webster, Greg S. Corrado, Yossi Matias, Katherine Chou, Juraj G...

    Pith/arXiv arXiv 2022

  28. [28]

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, et al . 2023. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. arXiv:2206.04615 [cs.CL] https://arxiv.org/abs/2206.04615

    Pith/arXiv arXiv 2023

  29. [29]

    Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks. arXiv:1409.3215 [cs.CL] https://arxiv.org/abs/ 1409.3215

    Pith/arXiv arXiv 2014

  30. [30]

    Le, Ed H

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. 2022. Challenging BIG-Bench Tasks and Whether Chain-of- Thought Can Solve Them. arXiv:2210.09261 [cs.CL] https://arxiv.org/abs/2210. 09261

    Pith/arXiv arXiv 2022

  31. [31]

    Ruixiang Tang, Dehan Kong, Longtao Huang, and Hui Xue. 2023. Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learning. In Findings of the Association for Computational Linguistics: ACL 2023. Association for Computational Linguistics, 4645–4657. doi:10.18653/v1/2023.findings-acl.284

    doi:10.18653/v1/2023.findings-acl.284 2023

  32. [32]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention Is All You Need. arXiv:1706.03762 [cs.CL] https://arxiv.org/abs/1706.03762

    Pith/arXiv arXiv 2023

  33. [33]

    Zeyu Wang. 2024. CausalBench: A Comprehensive Benchmark for Evaluating Causal Reasoning Capabilities of Large Language Models. InProceedings of the 10th SIGHAN Workshop on Chinese Language Processing (SIGHAN-10), Kam- Fai Wong, Min Zhang, Ruifeng Xu, Jing Li, Zhongyu Wei, Lin Gui, Bin Liang, and Runcong Zhao (Eds.). Association for Computational Linguisti...

    2024

  34. [34]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903 [cs.CL] https: //arxiv.org/abs/2201.11903

    Pith/arXiv arXiv 2023

  35. [35]

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. BloombergGPT: A Large Language Model for Finance. arXiv:2303.17564 [cs.LG] https://arxiv.org/abs/2303.17564

    Pith/arXiv arXiv 2023

  36. [36]

    Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang. 2025. FinGPT: Open- Source Financial Large Language Models. arXiv:2306.06031 [q-fin.ST] https: //arxiv.org/abs/2306.06031

    arXiv 2025

  37. [37]

    skewness

    Dun Zhang, Jiacheng Li, Ziyang Zeng, and Fulong Wang. 2025. Jasper and Stella: distillation of SOTA embedding models. arXiv:2412.19048 [cs.IR] https: //arxiv.org/abs/2412.19048 A Appendix A.1 Reducing Data Skewness Although the pairwise causal questions with opposite causal an- swers are specifically designed to be semantically close to each other, the da...

    Pith/arXiv arXiv 2025

  38. [2011]

    InAAAI Spring Symposium on Log- ical Formalizations of Commonsense Reasoning

    Choice of Plausible Alternatives: An Evaluation of Com- monsense Causal Reasoning. InAAAI Spring Symposium on Log- ical Formalizations of Commonsense Reasoning. Stanford Univer- sity. http://ict.usc.edu/pubs/Choice%20of%20Plausible%20Alternatives- %20An%20Evaluation%20of%20Commonsense%20Causal%20Reasoning.pdf

  39. [2020]

    arXiv:2005.00661 [cs.CL] https://arxiv.org/abs/2005.00661

    On Faithfulness and Factuality in Abstractive Summarization. arXiv:2005.00661 [cs.CL] https://arxiv.org/abs/2005.00661

    Pith/arXiv arXiv 2005

  40. [2021]

    Why Machine Reading Comprehension Models Learn Shortcuts? arXiv:2106.01024 [cs.CL] https://arxiv.org/abs/2106.01024

    Pith/arXiv arXiv

  41. [2025]

    arXiv:2509.04664 [cs.CL] https: //arxiv.org/abs/2509.04664

    Why Language Models Hallucinate. arXiv:2509.04664 [cs.CL] https: //arxiv.org/abs/2509.04664

    Pith/arXiv arXiv