Pith. sign in

REVIEW 5 major objections 6 minor 60 references

Counterfactual Samples Constructing and Training for Commonsense Statements Estimation

T0 review · 5 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Counterfactual training that swaps high-contribution words with embedding-similar alternatives makes a 5B T5 encoder a better commonsense verifier, beating the prior best by 3.07 absolute points.

desk verdict A plausible debiasing/augmentation recipe whose headline 3.07% gain is currently not reproducible from the paper's own tables. read the letter →

arxiv 2412.20563 v1 pith:EBHZOLH6 submitted 2024-12-29 cs.CL

Chong Liu , Zaiwen Feng , Lin Liu , Zhenyun Deng , Jiuyong Li , Ruifang Zhai , Debo Cheng , Li Qin This is my paper · ORCID
classification cs.CL
keywords plausibilityestimationcommonsensereasoningcounterfactualaugmentationcontrastivelearninggradient-basedattributionbiasknowledgefilteringT5encoder
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to improve plausibility estimation — the task of judging whether a commonsense statement is true — by training models on counterfactual examples it constructs automatically. It proposes CCSG, which finds the words a verifier relies on most, swaps those words for embedding-similar alternatives to create 'anti-commonsense' negative sentences, and adds dropout-based positives. These samples feed a supervised contrastive loss that pushes representations of true and false statements apart. The paper reports that a 5-billion-parameter T5 encoder trained this way outperforms the previous state of the art, VERA+T5, by 3.07 percentage points on average accuracy across nine datasets, and that using CCSG to filter GPT-3-generated knowledge improves downstream question-answering accuracy by 4.01 percentage points. If the result holds, it offers a self-contained, knowledge-free training recipe for making commonsense verifiers both more accurate and more sensitive to minimal edits.

What carries the argument

The engine of the method is a counterfactual sample constructor plus a supervised contrastive training objective. The constructor selects candidate words by part-of-speech tagging and GloVe similarity, scores each word's contribution to the predicted probability of the correct label with a modified Grad-CAM gradient, and replaces the top-K contributing words with embedding-similar alternatives to form negative statements whose plausibility label is assumed flipped. Low-level dropout applied to the original statement generates the positive sample. These pairs are then fed to a sentence-level supervised contrastive loss, together with a binary cross-entropy term, which pulls the encoder's representation of a statement toward representations of same-label counterfactuals and pushes it away from opposite-label counterfactuals. That combination is what carries the argument: the contrastive objective is the channel through which commonsense sensitivity is supposed to enter the encoder.

What would settle it

Sample the automatically generated negative sentences and have annotators (or a verified judge) label whether each is genuinely implausible; if more than a small fraction are still plausible, or are ungrammatical, the contrastive loss is being trained on wrong labels and the reported accuracy gains cannot be attributed to commonsense sensitivity.

Watch

Extended reading notes

Core claim

The paper's central claim is that a plausibility estimator can be made substantially better at commonsense judgment by training it to tell each statement apart from automatically generated counterfactual variants of itself. In the CCSG pipeline, the model first assigns each candidate word a contribution score via gradient-based attribution to the correct label; the top-scoring words are then replaced, using a vector knowledge base such as GloVe, by words with highly similar embeddings to produce negatives, while low-level dropout produces label-preserving positives. The model is trained with a sentence-level supervised contrastive loss plus a binary classification loss. Empirically, the paper reports an average accuracy of 87.49 across nine commonsense benchmarks, an absolute gain of 3.07 over VERA+T5 and 6.38 over its own encoder without CCSG, and a 4.01 point accuracy gain when CCSG filters GPT-3-generated knowledge before question answering. It also reports lower commonsense-bias rates on sampled statement fields and a shift in word-contribution scores toward content words, which it interprets as improved language-explainability.

Load-bearing premise

The whole pipeline assumes that swapping a high-contribution word for one with a similar GloVe embedding turns a true statement into a false one, and that assumed flip is used as a training label without any check that the new sentence is actually implausible or even grammatical.

Editorial extensions

If this is right

  • Plausibility estimators can be improved from the training data itself, without external knowledge bases or human-curated perturbations.
  • Filtering an LLM's generated knowledge with the counterfactually trained verifier yields a concrete accuracy gain in downstream QA, 4.01 points in the paper's setting.
  • The contrastive objective should make verifier decisions more traceable, because attribution concentrates on content words such as 'noodles' and 'restaurant' rather than on function words.
  • Small, minimal edits to a statement should produce large prediction changes, which is exactly the commonsense-sensitivity the paper argues standard models lack.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the construction only needs word embeddings and gradient scores, the same recipe could transfer to other label-dependent NLP tasks such as natural language inference or fact verification, but the paper does not test those settings.
  • A direct test of the method's core assumption would be to annotate a sample of generated negatives: if many are still plausible or ungrammatical, the contrastive labels are noisy and the reported gains would need to be explained by the dropout positives rather than by the substitution signal.
  • The causal SCM framing suggests an inference-time intervention on the keyword variable, yet the implementation only augments training data; estimating the average causal effect of keyword substitution directly would clarify whether the debiasing claim is causal or correlational.
  • Since CCSG is model-agnostic, applying it to decoder-only or smaller encoders is a natural next step, but the paper provides no evidence that the 3.07-point gain will transfer to other architectures.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes CCSG, a training-time augmentation for commonsense plausibility estimation (PE). The method identifies high-contribution words via gradient-based contribution scores, replaces them with GloVe-similar words to create negative counterfactual samples, and uses dropout to create positive samples. These samples are then used in a sentence-level supervised contrastive loss on top of a binary classification loss, with a Flan-T5 encoder backbone. The central empirical claim, stated in Section 6.1, is that CCSG+T5 reaches 87.49% average accuracy and outperforms VERA+T5 by 3.07% absolute, and that CCSG filtering improves GPT-3 generated-knowledge QA by 4.01% (Section 6.2). The paper also presents a causal-graph analysis in Section 3.2 and a bias-rate analysis in Section 6.3.

Significance. If the reported results are correct, the paper would show that a simple plug-in augmentation, based on gradient-guided word replacement and contrastive training, can substantially improve a strong 5B-parameter PE model and outperform previous state-of-the-art systems. The experimental scope is broad, covering nine PE datasets and an LLM-knowledge filtering setting. However, the significance is currently limited by three factors: the headline accuracy number is not reproducible from the paper's own tables; the counterfactual negative labels are assumed, not verified; and no code, seeds, or error bars are provided. The causal SCM in Section 3.2 is also decorative rather than used for any causal-effect estimation. These issues prevent the reader from assessing whether the claimed improvements reflect a genuine commonsense signal or are artifacts of training noise and evaluation inconsistencies.

major comments (5)
  1. [Section 6.1, Tables 6 and 7] The headline result is not reproducible from the numbers printed in the paper. The CCSG+T5 row in Table 6 lists 89.99, 93.51, 91.03, 90.08, 91.14, 91.01, 92.77, 70.37, 82.95; these nine values average to 88.09, not the reported 87.49. The value 87.49 matches Table 7, but Table 7 gives different ComVE and C2S scores (88.86 and 81.43) than Table 6 (92.77 and 82.95). The w/o CCSG row in Table 7 is also internally inconsistent: the listed values average to 79.14, not the reported 81.11. Since the claimed 3.07% margin over VERA+T5 is a difference of averages, this arithmetic inconsistency changes the claimed result by up to 0.6 points and makes the central empirical claim unverifiable as printed. Please correct both tables and explain which numbers correspond to the actual runs, and make the raw per-dataset outputs or code available so the calculation can be independently checked.
  2. [Section 4.1, Eq. (7)-(8)] The method assumes that replacing a top-contribution word with a highly cosine-similar GloVe word flips the plausibility of the sentence, and the resulting sentence is then used as a negative example in the supervised contrastive loss. This label flip is never verified. If many replacements preserve the original truth value or produce nonsensical strings, the contrastive loss is trained on incorrect label assignments, and the reported gains could come from fitting noise rather than from a genuine commonsense signal. The paper should measure the flip rate (e.g., by manual evaluation or by using a held-out verifier), filter low-quality replacements, and report how many generated samples are actually used. Without this, the main methodological justification remains an unverified assumption.
  3. [Section 3.2, Eq. (3)] The SCM section defines an Average Causal Effect, ACE_C, but the paper never estimates this or any other causal quantity. No do-calculus, adjustment formula, or causal identification is used anywhere in the algorithm or experiments. The abstract and the contribution list claim CCSG 'mitigates this bias through causal inference techniques,' but the connection between the causal graph and the counterfactual construction is purely rhetorical. Please either remove the causal-inference claims or provide an explicit derivation showing how the counterfactual constructor implements an intervention on the SCM, and ideally report the corresponding causal-effect estimates.
  4. [Section 4.1 and Section 5.5, Table 5] The counterfactual constructor depends on two hyperparameters that are not specified: the number of replaced keywords, top-K, and the size of the initial candidate set, |I|. Table 5 lists learning rate, loss weights, temperature, and batch sizes, but not these parameters. Without them the method is under-specified and not reproducible. Please report the values used and include them in the hyperparameter table.
  5. [Section 6.1 and Section 6.2] All experimental results are reported as single numbers, with no random seeds, no multiple runs, and no error bars or significance tests. Given that the claimed margins are 3.07% and 1.94%, and that the paper's own tables contain substantial arithmetic inconsistencies, the reported improvements cannot be distinguished from run-to-run variance. Please provide standard deviations across at least three seeds, or explain why variance is negligible. In the absence of released code or raw logs, this is necessary to support the central claims.
minor comments (6)
  1. [Section 3.1 and Section 6.1] The threshold definition is inconsistent: Section 3.1 says the model outputs a confidence score s in [0,1] with a 0.5 threshold, while Section 6.1 says a logit threshold of z=0 is used. Since s = sigmoid(z), z=0 corresponds to s=0.5, but the notation should be unified to avoid ambiguity.
  2. [Eq. (6)] The binary classification loss uses the symbol a_i for the accurate label, whereas the task definition and later equations use y_i for the binary label. Please use a single consistent notation.
  3. [Eq. (9)] The accuracy formula is not written correctly: it uses S(z(x_i)) = y_i, but S is called a 'sign function' while y_i is 0/1. A sign function typically returns {-1, 1}, so the indicator condition needs to be defined explicitly (e.g., [sigmoid(z(x_i)) > 0.5] = y_i).
  4. [Table 8] There is a typo in the 'Our' row: '67.8 6' should presumably be '67.86' or '67.8'. Please correct the table.
  5. [Section 6.3.1, Table 9] The manual bias-rate evaluation reports percentages from 150 selected sentences but gives no details about the selection procedure, the annotation instructions, the number of annotators, or inter-annotator agreement. Adding this information would make the claim that CCSG 'alleviates commonsense bias' more credible.
  6. [Table 3] The paper acknowledges that most evaluation datasets are included in Flan-T5's pretraining data, but it does not discuss what this implies for the comparison. Some of the baseline models are also exposed to the same data, but the paper should still state whether the reported gains are robust to this overlap, or at least discuss the potential confound.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CCSG's accuracy claims rest on external benchmarks, not on its own construction.

full rationale

CCSG is an empirical training-time data-augmentation method, and its central claims are test-set accuracies reported against external baselines. No equation in the paper defines the reported accuracy in terms of the method's own inputs, so there is no derivation that reduces to its assumptions. The word-contribution scores in Eq. (4) select replacement words, but the evaluation labels and baseline scores come from independent benchmarks such as VERA, so the 3.07% margin is not a fitted quantity renamed a prediction. The SCM in Section 3.2 is decorative: ACE in Eq. (3) is never computed, no do-calculus step feeds into the loss, and no uniqueness or identifiability result is imported to force the method; the causal framing is rhetorical rather than load-bearing. The negative samples' label flips are assumed rather than verified in Section 4.1, which is a validity and robustness concern, but not circularity, because test accuracy is computed on real labels rather than on the generated labels. There are self-citations to Cheng et al. (2023, 2024) for causal DAGs and causal effects, but those citations only support the rhetorical causal framing and are not load-bearing for the empirical conclusions. One non-circular correctness problem should be noted separately: the printed averages in Tables 6 and 7 are internally inconsistent, since the Table 6 CCSG+T5 row (89.99, 93.51, 91.03, 90.08, 91.14, 91.01, 92.77, 70.37, 82.95) averages to 88.09 rather than the printed 87.49; this is a reproducibility issue, not a circularity issue. Overall, no circular step was identified.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The central claim rests on eight manually chosen hyperparameters, only five of which are given concrete values, plus five domain assumptions. No invented entities are needed. The heaviest burden is the unverified flip of negative-sample labels.

free parameters (8)
  • alpha (binary loss weight) = 1.0
    Manual weighting in total loss L = alpha * L_bin + beta * L_cot; no sensitivity analysis reported (Table 5).
  • beta (contrastive loss weight) = 0.25
    Manual weighting of the contrastive term; no ablation for this value (Table 5).
  • tau (contrastive temperature) = 0.05
    Temperature in Equation 7; set by hand, no sensitivity study (Table 5).
  • R_drop (dropout ratio for positive samples) = 0.05
    Low-level dropout ratio used to generate positive counterfactual samples; no analysis of other ratios (Table 5).
  • top-K (number of replaced keywords)
    Section 4.1 says "pick out the top-K words" but the value of K is not reported anywhere.
  • |I| (initial entity candidate set size)
    Section 4.1 "distill the initial word set I by picking the top |I| words"; size not specified.
  • learning rate = 1e-5
    Optimizer setting for T5 backbone (Table 5); not varied.
  • decision threshold = logit 0
    Section 6.1 says they forgo estimating a model-specific threshold and use z=0 after observing it aligns with the optimal threshold; this is a post hoc choice.
assumptions (5)
  • domain assumption Replacing a high-contribution word with a similar embedding produces a statement whose truth value flips.
    Section 4.1, Critical Entities Substitution and Equation 8: generated samples are treated as negative examples without checking whether the replacement actually makes the statement implausible.
  • domain assumption Low-level dropout preserves the statement's truth value.
    Section 4.1, Positive Samples Generation: dropout-altered sentences are used as positives with the original label, following SimCSE; no validation on PE data.
  • domain assumption Gradient contribution scores identify the words that carry commonsense meaning.
    Equation 4 uses the derivative of the correct-label probability with respect to word embeddings; the paper cites prior work but provides no evaluation that these are the right words.
  • ad hoc to paper The causal DAG and do-operator framework describes the PE data generation process.
    Section 3.2 introduces D, K, C, X, Y and ACE, but no ACE is estimated and no intervention is actually performed in the experiments; the SCM is used as motivation rather than as a computational tool.
  • domain assumption Multi-choice tasks can be converted to binary statements using the VERA statement-group format without loss.
    Section 6.1: "we transform these tasks into the statement group format as described by (Liu et al., 2023)"; correctness depends on that conversion.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Counterfactual Samples Constructing and Training for Commonsense Statements Estimation." pith.science (2026). https://pith.science/paper/EBHZOLH6

@misc{pith2026241220563,
  author       = {Pith},
  title        = {Pith review of: Counterfactual Samples Constructing and Training for Commonsense Statements Estimation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EBHZOLH6}},
  note         = {Machine review of arXiv:2412.20563}
}
read the original abstract

Plausibility Estimation (PE) plays a crucial role for enabling language models to objectively comprehend the real world. While large language models (LLMs) demonstrate remarkable capabilities in PE tasks but sometimes produce trivial commonsense errors due to the complexity of commonsense knowledge. They lack two key traits of an ideal PE model: a) Language-explainable: relying on critical word segments for decisions, and b) Commonsense-sensitive: detecting subtle linguistic variations in commonsense. To address these issues, we propose a novel model-agnostic method, referred to as Commonsense Counterfactual Samples Generating (CCSG). By training PE models with CCSG, we encourage them to focus on critical words, thereby enhancing both their language-explainable and commonsense-sensitive capabilities. Specifically, CCSG generates counterfactual samples by strategically replacing key words and introducing low-level dropout within sentences. These counterfactual samples are then incorporated into a sentence-level contrastive training framework to further enhance the model's learning process. Experimental results across nine diverse datasets demonstrate the effectiveness of CCSG in addressing commonsense reasoning challenges, with our CCSG method showing 3.07% improvement against the SOTA methods.

Figures

Figures reproduced from arXiv: 2412.20563 by the authors.

Figure 1
Figure 1. An ideal PE model should exhibit two indispensable characteristics: (a) Language-Explainable ability: The PE model should not only make a correct prediction but also base its prediction on the appropriate linguistic reference regions. (b) Commonsense-Sensitive ability: The PE model must be enough sensitive to commonsense variations in texts. For example, replacing the critical word “swim” with “run”, in a statement … view at source ↗
Figure 2
Figure 2. The SCM depicts the inferential mechanism of the PE model without any interventions. (b) An intervention on the variable 𝐶 is assigned the value 𝑐0 , represented as do(𝐶 = 𝑐0 ). Similarly, an intervention on the variable 𝐾 is assigned the value 𝑘0 , represented as do(𝐾 = 𝑘0 ) For analytical clarity, we decompose the sentence into two constituent variables: the keyword (𝐾) and the context (𝐶). The proposed causal DAG… view at source ↗
Figure 3
Figure 3. The overview of our CCSG framework is as follows: 1) We prepare appropriate keywords based on the word contributions from the previous training output. 2) We replace the keywords using a vector knowledge base (VKB), such as Glove, to generate counterfactual negative samples, and generate counterfactual positive samples by applying dropout (Gao et al., 2021) to the original samples. 3) We send the counterfactual samp… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Visualization of the contribution distribution of each token. efficacy of CCSG. Utilizing CCSG, our refined model exhibits a significant enhancement of 6.38% in terms of Average Precision when contrasted with the original T5 model. The findings from the ablation study …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 29 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in ":" * " " * FUNCTION f...

  2. [2]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  3. [3]

    Bender, E. M. and Koller, A. (2020). Climbing towards nlu: On meaning, form, and understanding in the age of data. In Proceedings of the 58th annual meeting of the association for computational linguistics , pages 5185--5198

    work page 2020

  4. [4]

    D., Downey, D., Bras, R

    Bhagavatula, C., Hwang, J. D., Downey, D., Bras, R. L., Lu, X., Qin, L., Sakaguchi, K., Swayamdipta, S., West, P., and Choi, Y. (2022). I2d2: Inductive knowledge distillation with neurologic and self-imitation. arXiv preprint arXiv:2212.09246

    arXiv 2022

  5. [5]

    H., Ong, C

    Brodersen, K. H., Ong, C. S., Stephan, K. E., and Buhmann, J. M. (2010). The balanced accuracy and its posterior distribution. In 2010 20th international conference on pattern recognition , pages 3121--3124. IEEE

    2010

  6. [6]

    Brown, T. B. (2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165

    arXiv 2020

  7. [7]

    Chen, H., Feng, S., Ganhotra, J., Wan, H., Gunasekara, C., Joshi, S., and Ji, Y. (2021). Explaining neural network predictions on sentence pairs via learning word-group masks. arXiv preprint arXiv:2104.04488

    work page Pith review arXiv 2021

  8. [8]

    Chen, L., Zheng, Y., Niu, Y., Zhang, H., and Xiao, J. (2023a). Counterfactual samples synthesizing and training for robust visual question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(11):13218--13234

    work page 2023

Show all 60 references
  1. [9]

    Chen, Z., Hu, L., Li, W., Shao, Y., and Nie, L. (2023b). Causal intervention and counterfactual reasoning for multi-modal fake news detection. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 627--638

    2023

  2. [10]

    Cheng, D., Li, J., Liu, L., Liu, J., and Le, T. D. (2024). Data-driven causal effect estimation based on graphical causal modelling: A survey. ACM Computing Surveys , 56(5):1--37

    2024

  3. [11]

    Cheng, D., Xu, Z., Li, J., Liu, L., Liu, J., and Le, T. D. (2023). Causal inference with conditional instruments using deep generative models. In Proceedings of the AAAI conference on artificial intelligence , volume 37, pages 7122--7130

    2023

  4. [12]

    Eisenstein, J. (2022). Informativeness and invariance: Two perspectives on spurious correlations in natural language. arXiv preprint arXiv:2204.04487

    2022 arXiv

  5. [13]

    Feng, T., Qu, L., and Haffari, G. (2023). Less is more: Mitigate spurious correlations for open-domain dialogue response generation models by causal discovery. Transactions of the Association for Computational Linguistics , 11:511--530

    2023

  6. [14]

    Gao, T., Yao, X., and Chen, D. (2021). Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821

    2021 arXiv

  7. [15]

    Goyal, N., Paneri, R., Agarwal, A., Kalani, U., Sancheti, A., and Chhaya, N. (2020). Cam-gen: Causally-aware metric-guided text generation. arXiv preprint arXiv:2010.12795

    2020 arXiv

  8. [16]

    Guan, B., Zhu, X., and Yuan, S. (2024). A t5-based interpretable reading comprehension model with more accurate evidence training. Inf. Process. Manag. , 61(2):103584

    2024

  9. [17]

    and Montani, I

    Honnibal, M. and Montani, I. (2017). spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing. To appear , 7(1):411--420

    2017

  10. [18]

    and Li, L

    Hu, Z. and Li, L. E. (2021). A causal lens for controllable text generation. Advances in Neural Information Processing Systems , 34:24941--24955

    2021

  11. [19]

    and Wallace, B

    Jain, S. and Wallace, B. C. (2019). Attention is not explanation. arXiv preprint arXiv:1902.10186

    2019 arXiv

  12. [20]

    Jia, R., Raghunathan, A., G \"o ksel, K., and Liang, P. (2019). Certified robustness to adversarial word substitutions. arXiv preprint arXiv:1909.00986

    2019 arXiv

  13. [21]

    L., and Choi, Y

    Jung, J., Qin, L., Welleck, S., Brahman, F., Bhagavatula, C., Bras, R. L., and Choi, Y. (2022). Maieutic prompting: Logically consistent reasoning with recursive explanations. arXiv preprint arXiv:2205.11822

    2022 arXiv

  14. [22]

    Kadavath, S., Conerly, T., Askell, et al. (2022). Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221

    2022 arXiv

  15. [23]

    A., Jensen, D., and O'Connor, B

    Keith, K. A., Jensen, D., and O'Connor, B. (2020). Text and causal inference: A review of using text to remove confounding from causal estimates. arXiv preprint arXiv:2005.00649

    2020 arXiv

  16. [24]

    Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., and Krishnan, D. (2020). Supervised contrastive learning. Advances in neural information processing systems , 33:18661--18673

    2020

  17. [25]

    and Mohammad, S

    Kiritchenko, S. and Mohammad, S. M. (2018). Examining gender and race bias in two hundred sentiment analysis systems. arXiv preprint arXiv:1805.04508

    2018 arXiv

  18. [26]

    Ling, C., Zhang, X., Zhao, X., Wu, Y., Liu, Y., Cheng, W., Chen, H., and Zhao, L. (2023). Knowledge-enhanced prompt for open-domain commonsense reasoning. In 1st AAAI Workshop on Uncertainty Reasoning and Quantification in Decision Making

    2023

  19. [27]

    Liu, J., Hallinan, S., Lu, X., He, P., Welleck, S., Hajishirzi, H., and Choi, Y. (2022). Rainier: Reinforced knowledge introspector for commonsense question answering. arXiv preprint arXiv:2210.03078

    2022 arXiv

  20. [28]

    L., Choi, Y., and Hajishirzi, H

    Liu, J., Liu, A., Lu, X., Welleck, S., West, P., Bras, R. L., Choi, Y., and Hajishirzi, H. (2021). Generated knowledge prompting for commonsense reasoning. arXiv preprint arXiv:2110.08387

    2021 arXiv

  21. [29]

    A., Choi, Y., and Hajishirzi, H

    Liu, J., Wang, W., Wang, D., Smith, N. A., Choi, Y., and Hajishirzi, H. (2023). Vera: A general-purpose plausibility estimation model for commonsense statements. arXiv preprint arXiv:2305.03695

    2023 arXiv

  22. [30]

    Liu, Y. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692

    2019 arXiv

  23. [31]

    Madaan, N., Padhi, I., Panwar, N., and Saha, D. (2021). Generate your counterfactuals: Towards controlled counterfactual generation for text. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 13516--13524

    2021

  24. [32]

    and Davis, E

    Marcus, G. and Davis, E. (2023). Chatgpt/llm errors. (public)

    2023

  25. [33]

    and Li, W

    Mu, F. and Li, W. (2023). Enhancing event causality identification with counterfactual reasoning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages 967--975

    2023

  26. [34]

    Introducing chatgpt

    OpenAI (2022a). Introducing chatgpt. (public)

    2022

  27. [35]

    Moddels - overview - gpt3.5

    OpenAI (2022b). Moddels - overview - gpt3.5. (public)

    2022

  28. [36]

    Pearl, J. (2009a). Causal inference in statistics: An overview

    2009

  29. [37]

    Pearl, J. (2009b). Causality . Cambridge university press

    2009

  30. [38]

    Pearl, J. et al. (2000). Models, reasoning and inference. Cambridge, UK: CambridgeUniversityPress , 19(2):3

    2000

  31. [39]

    Pennington, J., Socher, R., and Manning, C. D. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , pages 1532--1543

    2014

  32. [40]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research , 21(140):1--67

    2020

  33. [41]

    T., Wu, T., Guestrin, C., and Singh, S

    Ribeiro, M. T., Wu, T., Guestrin, C., and Singh, S. (2020). Beyond accuracy: Behavioral testing of nlp models with checklist. arXiv preprint arXiv:2005.04118

    2020 arXiv

  34. [42]

    E., Stewart, B

    Roberts, M. E., Stewart, B. M., and Nielsen, R. A. (2020). Adjusting for confounding with text matching. American Journal of Political Science , 64(4):887--903

    2020

  35. [43]

    A., and Gordon, A

    Roemmele, M., Bejan, C. A., and Gordon, A. S. (2011). Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI spring symposium series

    2011

  36. [44]

    S., Hughes, M

    Ross, A. S., Hughes, M. C., and Doshi-Velez, F. (2017). Right for the right reasons: Training differentiable models by constraining their explanations. arXiv preprint arXiv:1703.03717

    2017 arXiv

  37. [45]

    R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D

    Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2020). Grad-cam: visual explanations from deep networks via gradient-based localization. International journal of computer vision , 128:336--359

    2020

  38. [46]

    R., Lee, S., Shen, Y., Jin, H., Ghosh, S., Heck, L., Batra, D., and Parikh, D

    Selvaraju, R. R., Lee, S., Shen, Y., Jin, H., Ghosh, S., Heck, L., Batra, D., and Parikh, D. (2019). Taking a hint: Leveraging explanations to make vision and language models more grounded. In Proceedings of the IEEE/CVF international conference on computer vision , pages 2591--2600

    2019

  39. [47]

    D., and Clark, P

    Tafjord, O., Mishra, B. D., and Clark, P. (2022). Entailer: Answering questions with faithful and truthful chains of reasoning. arXiv preprint arXiv:2210.12217

    2022 arXiv

  40. [48]

    Talmor, A., Herzig, J., Lourie, N., and Berant, J. (2018). Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937

    2018 arXiv

  41. [49]

    Thorne, J., Vlachos, A., Christodoulopoulos, C., and Mittal, A. (2018). Fever: a large-scale dataset for fact extraction and verification. arXiv preprint arXiv:1803.05355

    2018 arXiv

  42. [50]

    Tokpo, E. K. and Calders, T. (2024). Fairflow: An automated approach to model-based counterfactual data augmentation for nlp. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 160--176. Springer

    2024

  43. [51]

    Udomcharoenchaikit, C., Ponwitayarat, W., Payoungkhamdee, P., Masuk, K., Buaphet, W., Chuangsuwanich, E., and Nutanong, S. (2022). Mitigating spurious correlation in natural language understanding with counterfactual inference. In Proceedings of the 2022 Conference on Empirica...

    2022

  44. [52]

    Wang, A., Song, L., Min, Z., Xu, G., Wang, X., Yao, J., and Su, J. (2025). Mitigating the negative impact of over-association for conversational query production. Inf. Process. Manag. , 62(1):103907

    2025

  45. [53]

    Wang, F., Mo, W., Wang, Y., Zhou, W., and Chen, M. (2023). A causal view of entity bias in (large) language models. arXiv preprint arXiv:2305.14695

    2023 arXiv

  46. [54]

    and Culotta, A

    Wang, Z. and Culotta, A. (2020). Identifying spurious correlations for robust text classification. arXiv preprint arXiv:2010.02458

    2020 arXiv

  47. [55]

    and Culotta, A

    Wang, Z. and Culotta, A. (2021). Robustness to spurious correlations in text classification via automatically generated counterfactuals. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 14024--14031

    2021

  48. [56]

    D., Jiang, L., Bras, R

    West, P., Bhagavatula, C., Hessel, J., Hwang, J. D., Jiang, L., Bras, R. L., Lu, X., Welleck, S., and Choi, Y. (2021). Symbolic knowledge distillation: from general language models to commonsense models. arXiv preprint arXiv:2110.07178

    2021 arXiv

  49. [57]

    Wood-Doughty, Z., Shpitser, I., and Dredze, M. (2018). Challenges of using text classifiers for causal inference. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing , volume 2018,...

    2018

  50. [58]

    S., and Kersting, K

    Ze c evi \'c , M., Willig, M., Dhami, D. S., and Kersting, K. (2023). Causal parrots: Large language models may talk causality but are not causal. arXiv preprint arXiv:2308.13067

    2023 arXiv

  51. [59]

    Zeng, X., Li, Y., Zhai, Y., and Zhang, Y. (2020). Counterfactual generator: A weakly-supervised method for named entity recognition. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 7270--7280

    2020

  52. [60]

    Zhao, Y., Xia, T., Jiang, Y., and Tian, Y. (2024). Enhancing inter-sentence attention for semantic textual similarity. Inf. Process. Manag. , 61(1):103535

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.