Pith. sign in

REVIEW 4 major objections 5 minor 56 references

CodeSCM: Causal Analysis for Multi-Modal Code Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A causal model shows that input-output examples steer code LLMs as strongly as natural-language instructions.

desk verdict Useful ablation study with an over-claimed causal wrapper; the I/O finding is likely real, but direct effects are not identified as causal. read the letter →

arxiv 2502.05150 v1 pith:MKP5GM25 submitted 2025-02-07 cs.CL

Mukur Gupta , Noopur Bhatt , Suman Jana This is my paper · ORCID
classification cs.CL
keywords causalmediationanalysisstructuralmodelcodegenerationlargelanguagemodelsmulti-modalpromptsdirecteffectsspuriouscorrelationsbenchmarkmemorization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to explain which parts of a multi-modal prompt actually cause a code-generation LLM to produce correct code, rather than merely correlate with it. It builds CodeSCM, a structural causal model whose latent mediators separate code semantics from natural-language semantics, and intervenes on each prompt modality. Its central finding is that input-output example pairs are not scaffolding: alongside natural-language instructions, they are first-order drivers of generation, and semantics-preserving rewrites of them measurably reduce accuracy. The paper interprets that accuracy drop as a direct effect that bypasses code and language semantics, that is, as evidence of the model's spurious reliance on surface form.

What carries the argument

The central object is CodeSCM, a structural causal model in which each prompt modality (natural language, algorithmic code, the natural-language channel of code, and input-output pairs) is a variable affecting a binary correctness response, mediated by two latent variables: $M_{\mathrm{Code}}$ for code semantics and $M_{\mathrm{NL}}$ for natural-language semantics. The argument runs on path effects: the Direct Effect of a modality is the accuracy change along a path that bypasses both mediators, and the paper realizes that path by applying transformations claimed to preserve mediator values, for example inserting dead code into the algorithmic channel, adding a dead name prefix to the function header, adding a dead string to the natural language, and replacing each assertion equality by two inequalities. These transformations are the machinery that converts a model's accuracy drop into a causal claim about spurious correlations.

What would settle it

Probe the model's hidden representations before and after each semantics-preserving transformation: if the distance between the post-transformation and pre-transformation representations of code or natural-language semantics is non-negligible, then the mediator invariance fails and the reported direct effect is an accuracy change under prompt perturbation rather than a causal direct effect. A simpler behavioral check is to use a different set of semantics-preserving transformations for the same modality and see whether the direct-effect estimates reproduce.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the effect of a modal prompt component on code correctness can be decomposed through two latent mediators, $M_{\mathrm{Code}}$ and $M_{\mathrm{NL}}$, and that the component bypassing those mediators is measurable by semantics-preserving interventions. Applying this decomposition to GPT-4 Turbo, WizardCoder-15B, and Llama-3-8B across HumanEval+, mMBPP+, and CoderEval, the paper reports that I/O pairs have the largest direct effect on HumanEval+ and mMBPP+ and a total effect comparable to that of natural-language instructions; the natural-language channel of code, such as descriptive function names, is also a major total-effect driver, especially for GPT-4T on mMBPP+. The paper further finds evidence of benchmark memorization: models retain non-zero accuracy when natural language is removed and regenerate original function names even after standardization. It also reports that code-specific pretraining, as in CodeLLaMa-13B versus LLaMa-2-13B, brings prompt modalities closer in the embedding space and associates prompts with ground-truth solutions.

Load-bearing premise

The load-bearing premise is that the semantics-preserving interventions leave the model's internal code-semantics and language-semantics mediators unchanged (and, more broadly, that no unobserved confounders are at work), since those invariants are asserted rather than measured.

Editorial extensions

If this is right

  • Input-output example pairs should be treated as a causal ingredient of code generation, not as mere formatting; changing them changes what models produce.
  • A prompt can be considered robust to surface-form variation to the degree its direct effect is small, giving a per-modality robustness score.
  • Semantics-preserving changes can serve as cheap probes for dataset memorization and, in one observed case, for hallucination control when dead code reduced hallucinated Java class names.
  • Leaderboard scores on benchmarks like HumanEval+ may overstate reasoning ability because memorization inflates accuracy.
  • Code-specific pretraining aligns prompt modalities in the embedding space, which is consistent with its downstream generation advantage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the direct-effect reading is right, adversarial or accidental surface variations in I/O examples are a practical attack surface for code assistants, not just a benchmark artifact.
  • The same mediator logic could be extended to other modalities, such as images or error traces, by supplying transformations that provably leave the mediator invariant.
  • A testable extension: replacing the paper's single transformation per modality with several independent semantics-preserving transformations should yield similar direct-effect values; divergence would indicate the transformation, not the modality, drives the drop.
  • The embedding-space result suggests a concrete training intervention, not just a diagnosis: aligning prompt-modality representations during pretraining should reduce direct effects.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes CodeSCM, a structural causal model for multi-modal code generation prompts. Four modal variables (NL, Code_AL, Code_NL, I/O) are connected to two latent mediators M_Code and M_NL, and the authors define total and direct effects via interventions that remove a modality or apply 'semantics-preserving' transformations (dead code, dead string, dead name, inequality rewrites). Experiments on HumanEval+, mMBPP+, and CoderEval with GPT-4T, WizardCoder-15B, and LLaMa-3-8B report accuracy drops (TE/DE) per modality, error-type decompositions, and embedding-space analyses comparing CodeLLaMa and LLaMa-2. The paper's headline findings are that NL and I/O pairs significantly influence code generation and that DEs quantify spurious model leanings.

Significance. The empirical core—simple accuracy-difference ablations across three benchmarks and three models, plus error-type decompositions—is a useful contribution to understanding prompt-modality sensitivity, and it is backed by released code and an additional transformation in Appendix D. If the causal identification were valid, CodeSCM would be a valuable interpretability tool for code LLMs. However, the causal interpretation currently rests on unverified assumptions about latent mediators, and several headline statements ('direct effects representing spurious leanings', 'significantly influence') go beyond what the measurements establish. The paper's value is therefore closer to a well-structured sensitivity study than to a validated causal analysis.

major comments (4)
  1. [Section 3.2] In the DE derivation for Code_AL, equality (i) is the load-bearing step: the claim that MC(Code_AL(X=0)) = MC(Code_AL(X=1)) because dead-code insertion keeps M_Code unchanged is asserted without any estimate, measurement, or validation of the latent mediators. M_Code and M_NL are never observed, and no evidence is provided that insertions of dead code, dead strings, dead names, or inequality rewrites preserve the model's internal code/NL semantics. Without this invariance, the reported DE values are simply accuracy differences between two prompt variants, not path-specific direct effects under Definition 2.2. Section 7 acknowledges only the no-confounder assumption and does not flag this untested mediator-invariance premise.
  2. [Appendix A.2, Eq. (5)] The I/O transformation as written is not semantics-preserving. Replacing `assert I_l == I_r` with `(I_l <= I_r) + (I_r >= I_r)` makes the second inequality a tautology, so the transformed assertion does not assert equality. If the intended transformation was `(I_l <= I_r) + (I_r >= I_l)`, it should be stated correctly; as written, the I/O DE estimates in Table 2 and Section 4.3 rest on a transformation that changes the meaning of the input-output examples.
  3. [Table 2 and Abstract] All TE/DE results are point estimates of accuracy differences without confidence intervals or significance tests, despite the abstract claiming that I/O pairs 'significantly influence' code generation. Several entries are based on very small subsets (CoderEval-SCP has 35 problems and CoderEval-SCJ has 55), so a 2.86% or 18.18% delta can be a one-example change; some entries are marked with '*' as accuracy increases and are then discussed as if they were stable effects. The significance claim needs either statistical support or a qualified phrasing.
  4. [Section 4.2] The inference that non-zero accuracy after removing NL indicates benchmark memorization is not supported. Removing NL leaves Code_AL, Code_NL, and I/O components intact, which can provide substantial information, and non-zero accuracy can arise from the remaining modalities. Similarly, the abstract's interpretation of DEs as 'direct effects representing the model's spurious leanings' is an interpretive label, not a consequence of the measurements. These claims should be reframed as sensitivity/ablation results unless the causal model is identified.
minor comments (5)
  1. [Section 3.1, Eq. (7)] The 'extended Backus–Naur form' in Equation 7 is a simple prompt template with placeholders, not a grammar; either provide a real BNF or remove the term.
  2. [Section 4.2] The paragraph header 'CodeN LT E.' contains a typo, and the notation 'Code N L' is used inconsistently as 'CodeN L' in several places throughout the text and tables.
  3. [Appendix D] The transformation `assert not I_l! = I_r)` contains an extra closing parenthesis and inconsistent spacing around `!=`; as written, it is not a syntactically valid Python statement.
  4. [Table 1] The table caption should define the difference between MBPP+ and mMBPP+; the text says the original MBPP+ lacks Code_AL and mMBPP+ adds a code function header, but the caption does not make this clear.
  5. [Section 5] The sentence claiming 'we are the first to apply causal inference to study modal effects on code generation task' is a strong novelty claim and should be softened or qualified given the identification concerns raised above.

Circularity Check

2 steps flagged · score 6.0 of 10

Direct-effect estimates are defined as perturbation accuracy deltas; the 'spurious leanings' conclusion is true by construction, and mediator invariance is assumed rather than established.

  1. self definitional [Section 4.3, first paragraph (also Abstract)]
    "We define direct effects (DE) by noting the drop in pass@1 accuracy of the model under the semantics-preserving transformations of modalities where the latent mediators remain unchanged (Section 3.2). These effects also represent the spurious correlations, as any non-spurious learning process must be mediated through MNL and MCode."

    The paper defines 'spurious correlations' as effects that bypass the latent mediators, and simultaneously defines DE as the accuracy drop under transformations assumed to leave those mediators fixed. The later conclusion that large DEs reveal the model's 'spurious leanings' is therefore true by definition: the construct (spuriousness) has been equated with the observed accuracy delta under the transformation. Since MNL and MCode are never measured or independently estimated, the mediation analysis does not derive a separate causal quantity; it relabels the perturbation accuracy difference.

  2. self definitional [Section 3.2, 'Causal effects of CodeAL' derivation]
    "where equality (i) follows from the fact that MC(Code_AL(X = 0)) is equal to MC(Code_AL(X = 1)), because the dead code insertion in Equation 3 keeps the code semantics MCode unchanged."

    The path-specific direct effect in Definition 2.2 requires intervening to hold the latent mediator MCode fixed at its X=0 value while Code_AL is varied. The paper instead equates this counterfactual to the observed accuracy difference A(D) - A(D; PCodeAL = CAL + CDC), justifying the equality solely by the assertion that dead-code insertion keeps MCode unchanged. MCode is never measured or intervened upon, so the equality is not established; the 'direct effect' reduces by construction to the pass@1 accuracy delta between two prompt strings. The mediator-invariance premise is an input assumption, not a derived result.

full rationale

The empirical accuracy deltas are real measurements on external benchmarks, and no parameter is fitted to the evaluation data, so the total-effect observations (e.g., removing I/O pairs or natural language changes accuracy) are independent empirical findings. The circularity is confined to the causal-mediation interpretation: DE is equated, by assumption, to the accuracy drop under transformations called 'semantics-preserving,' and 'spurious' is defined as whatever bypasses the unmeasured latent mediators. A further validity concern is that the Appendix A.2 I/O transformation as written replaces the equality I_l == I_r with (I_l <= I_r) plus the tautology (I_r >= I_r), so it is not equality-preserving on its own terms; this strengthens the view that the reported DE values are prompt-sensitivity numbers rather than identified direct effects. The limitations section acknowledges only the no-confounder assumption and does not flag the untested mediator-invariance premise. Because the central 'spurious leanings' claim reduces by definition to the measured perturbation delta, the circularity score is 6; the underlying accuracy measurements keep the paper from being a pure tautology.

Assumptions & free parameters 2 free parameters · 4 assumptions · 2 invented entities

No numerical coefficients are fitted, but the hand-chosen transformations and manual modality decomposition shape all reported effects. The causal interpretation rests on three assumptions (no confounders, mediator mediation, mediator invariance under transformations) that are stated but not tested. The two latent mediators are invented entities without independent falsifiable handles.

free parameters (2)
  • Semantics-preserving transformation per modality = dead code, dead string, func_ prefix, assert to inequalities
    The choice of transformation determines the DE values, as shown by the alternative transformation set in Appendix D (e.g., NL DE changes from 0.50 to 1.23 on WizardCoder/mMBPP+). This is a hand-chosen modeling decision, not a derived quantity.
  • Manual modality decomposition = hand-labeled assignment of prompt tokens to NL, Code_AL, Code_NL, I/O
    The split of components such as function headers into Code_AL and Code_NL is a judgment call that shapes the TE and DE numbers.
assumptions (4)
  • domain assumption No confounders exist in the code-generation process.
    Stated in Section 3.1 ('We do not account for confounding variables in this analysis') and Section 7. Without this, the estimated effects may be biased.
  • domain assumption All modality effects are mediated through latent variables M_Code and M_NL.
    Figure 1 encodes this mediation structure, but it is a structural assumption imposed without testing.
  • ad hoc to paper Semantics-preserving transformations keep the latent mediator values fixed.
    Section 3.2 equates MC(Code_AL(X=0)) with MC(Code_AL(X=1)) because dead code is 'semantics-preserving', but the mediator is a model-internal quantity that is never measured.
  • domain assumption Pass@1 accuracy equals the Bernoulli expectation of the response variable Y.
    The paper equates E[Y] with pass@1 accuracy, ignoring sampling variance and temperature effects.
invented entities (2)
  • M_Code latent mediator
    purpose: Separates the code semantics of a prompt from its surface form.
    Never observed or estimated; its invariance under interventions is assumed.
  • M_NL latent mediator
    purpose: Separates the natural language semantics of a prompt from its surface form.
    Never observed or estimated; its role in mediating effects is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CodeSCM: Causal Analysis for Multi-Modal Code Generation." pith.science (2026). https://pith.science/paper/MKP5GM25

@misc{pith2026250205150,
  author       = {Pith},
  title        = {Pith review of: CodeSCM: Causal Analysis for Multi-Modal Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MKP5GM25}},
  note         = {Machine review of arXiv:2502.05150}
}
read the original abstract

In this paper, we propose CodeSCM, a Structural Causal Model (SCM) for analyzing multi-modal code generation using large language models (LLMs). By applying interventions to CodeSCM, we measure the causal effects of different prompt modalities, such as natural language, code, and input-output examples, on the model. CodeSCM introduces latent mediator variables to separate the code and natural language semantics of a multi-modal code generation prompt. Using the principles of Causal Mediation Analysis on these mediators we quantify direct effects representing the model's spurious leanings. We find that, in addition to natural language instructions, input-output examples significantly influence code generation.

Figures

Figures reproduced from arXiv: 2502.05150 by the authors.

Figure 1
Figure 1. CodeSCM causal graph representing the total and direct effects of the modal variable nodes on the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (a) Modalities in an example from mMBPP+ dataset, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The original HumanEval+ prompt (top) in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: (a) and (b) Embedding PCA projections of modalities in input prompt by CodeLLaMa and LLaMa-2. (c) [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Left figure shows a CoderEval-SCJ prompt where dead code insertion corrects the original prompt’s error [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 19 canonical work pages

  1. [1]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  2. [2]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. https://arxiv.org/abs/2108.07732 Program synthesis with large language models . Preprint, arXiv:2108.07732

    arXiv 2021

  3. [3]

    Chen Avin, Ilya Shpitser, and Judea Pearl. 2005. Identifiability of path-specific effects

    work page 2005

  4. [4]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv 2023

  5. [5]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    arXiv 2020

  6. [6]

    Xingyu Cai, Jiaji Huang, Yuchen Bian, and Kenneth Church. 2020. Isotropy in the contextual embedding space: Clusters and manifolds. In International Conference on Learning Representations

    work page 2020

  7. [7]

    Barr, Santanu Kumar Dash, Prem Devanbu, and Emily Morgan

    Casey Casalnuovo, Earl T. Barr, Santanu Kumar Dash, Prem Devanbu, and Emily Morgan. 2020. https://doi.org/10.1145/3377816.3381720 A theory of dual channel constraints . In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: New Ideas and Emerging Results, ICSE-NIER '20, page 25–28, New York, NY, USA. Association for Computin...

    work page arXiv 2020

  8. [8]

    Pappas, and Eric Wong

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. 2023. https://arxiv.org/abs/2310.08419 Jailbreaking black box large language models in twenty queries . Preprint, arXiv:2310.08419

    arXiv 2023

Show all 56 references
  1. [10]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021 arXiv

  2. [11]

    Jürgen Cito, Isil Dillig, Vijayaraghavan Murali, and Satish Chandra. 2021. https://arxiv.org/abs/2111.05711 Counterfactual explanations for models of code . Preprint, arXiv:2111.05711

    2021 arXiv

  3. [12]

    Paul Denny, Viraj Kumar, and Nasser Giacaman. 2022. https://arxiv.org/abs/2210.15157 Conversing with copilot: Exploring prompt engineering for solving cs1 problems using natural language . Preprint, arXiv:2210.15157

    2022 arXiv

  4. [13]

    Aditya Desai, Sumit Gulwani, Vineet Hingorani, Nidhi Jain, Amey Karkare, Mark Marron, Sailesh R, and Subhajit Roy. 2015. https://arxiv.org/abs/1509.00413 Program synthesis using natural language . Preprint, arXiv:1509.00413

    2015 arXiv

  5. [14]

    Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E

    Amir Feder, Katherine A. Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E. Roberts, Brandon M. Stewart, Victor Veitch, and Diyi Yang. 2022. https://arxiv.org/abs/2109.00725 Causal inference in nat...

    2022 arXiv

  6. [15]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. https://arxiv.org/abs/2002.08155 Codebert: A pre-trained model for programming and natural languages . Preprint, arXiv:2002.08155

    2020 arXiv

  7. [16]

    Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. 2021. Causal analysis of syntactic agreement mechanisms in neural language models. arXiv preprint arXiv:2106.06087

    2021 arXiv

  8. [17]

    Sumit Gulwani, Oleksandr Polozov, Rishabh Singh, et al. 2017. Program synthesis. Foundations and Trends in Programming Languages , 4(1-2):1--119

    2017

  9. [18]

    Jingzhu He, Yuhang Lin, Xiaohui Gu, Chin-Chia Michael Yeh, and Zhongfang Zhuang. 2022. https://doi.org/10.1145/3510003.3510110 Perfsig: Extracting performance bug signatures via multi-modality causal analysis . In 2022 IEEE/ACM 44th International Conference on Software Enginee...

    2022

  10. [19]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring coding challenge competence with apps. NeurIPS

    2021

  11. [20]

    Zhijing Jin, Amir Feder, and Kun Zhang. 2022. https://doi.org/10.18653/v1/2022.emnlp-tutorials.4 C ausal NLP tutorial: An introduction to causality for natural language processing . In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Tuto...

    2022 doi

  12. [21]

    Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2022. Ds-1000: A natural and reliable benchmark for data science code generation. ArXiv, abs/2211.11501

    2022 arXiv

  13. [22]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, ...

    2023 arXiv

  14. [23]

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R \'e mi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal,...

    2022 arXiv

  15. [24]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023 a . https://openreview.net/forum?id=1qvx610Cu7 Is your code generated by chat GPT really correct? rigorous evaluation of large language models for code generation . In Thirty-seventh Conference on Neural Info...

    2023

  16. [25]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023 b . https://arxiv.org/abs/2305.01210 Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation . Preprint, arXiv:2305.01210

    2023 arXiv

  17. [26]

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. https://arxiv.org/abs/2107.13586 Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing . Preprint, arXiv:2107.13586

    2021 arXiv

  18. [27]

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shuj...

    2021 arXiv

  19. [28]

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. https://arxiv.org/abs/2306.08568 Wizardcoder: Empowering code large language models with evol-instruct . Preprint, arXiv:2306.08568

    2023 arXiv

  20. [29]

    Ankan Mullick, Mukur Gupta, and Pawan Goyal. 2024. https://arxiv.org/abs/2404.03598 Intent detection and entity extraction from biomedical literature . Preprint, arXiv:2404.03598

    2024 arXiv

  21. [30]

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. https://arxiv.org/abs/2203.13474 Codegen: An open large language model for code with multi-turn program synthesis . Preprint, arXiv:2203.13474

    2023 arXiv

  22. [31]

    OpenAI. 2024. Gpt-4 turbo. https://help.openai.com/en/articles/8555510-gpt-4-turbo

    2024

  23. [32]

    Judea Pearl. 2000. Causality: Models, Reasoning, and Inference. Cambridge University Press

    2000

  24. [33]

    Judea Pearl. 2022. Direct and indirect effects. In Probabilistic and causal inference: the works of Judea Pearl, pages 373--392

    2022

  25. [34]

    Judea Pearl et al. 2000. Models, reasoning and inference. Cambridge, UK: CambridgeUniversityPress, 19(2):3

    2000

  26. [35]

    Md Mahbubur Rahman, Ira Ceka, Chengzhi Mao, Saikat Chakraborty, Baishakhi Ray, and Wei Le. 2024. Towards causal deep learning for vulnerability detection. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1--11

    2024

  27. [36]

    Sara Rajaee and Mohammad Taher Pilehvar. 2021. An isotropy analysis in the multilingual bert embedding space. arXiv preprint arXiv:2110.04504

    2021 arXiv

  28. [37]

    James M Robins. 2003. Semantics of causal dag models and the identification of direct and indirect effects. Highly structured stochastic systems, pages 70--82

    2003

  29. [38]

    James M Robins and Sander Greenland. 1992. Identifiability and exchangeability for direct and indirect effects. Epidemiology, 3(2):143--155

    1992

  30. [39]

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défoss...

    2023 arXiv

  31. [40]

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2023. https://arxiv.org/abs/2310.11324 Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting . Preprint, arXiv:2310.11324

    2023 arXiv

  32. [41]

    Logan IV au2, Eric Wallace, and Sameer Singh

    Taylor Shin, Yasaman Razeghi, Robert L. Logan IV au2, Eric Wallace, and Sameer Singh. 2020. https://arxiv.org/abs/2010.15980 Autoprompt: Eliciting knowledge from language models with automatically generated prompts . Preprint, arXiv:2010.15980

    2020 arXiv

  33. [42]

    Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bernhard Sch \"o lkopf, and Mrinmaya Sachan. 2022. A causal framework to quantify the robustness of mathematical reasoning with language models. arXiv preprint arXiv:2210.12023

    2022 arXiv

  34. [43]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  35. [44]

    Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 33:12388--12401

    2020

  36. [45]

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven C. H. Hoi. 2021. https://arxiv.org/abs/2109.00859 Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation . Preprint, arXiv:2109.00859

    2021 arXiv

  37. [46]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. https://arxiv.org/abs/2201.11903 Chain-of-thought prompting elicits reasoning in large language models . Preprint, arXiv:2201.11903

    2023 arXiv

  38. [47]

    Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2023. https://arxiv.org/abs/2302.03668 Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery . Preprint, arXiv:2302.03668

    2023 arXiv

  39. [48]

    Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. https://arxiv.org/abs/2302.11382 A prompt pattern catalog to enhance prompt engineering with chatgpt . Preprint, arXiv:2302.11382

    2023 arXiv

  40. [49]

    Qi Wu, Yuyao Zhang, and Marawan Elbatel. 2023. Self-prompting large vision models for few-shot medical image segmentation. In MICCAI workshop on domain adaptation and representation transfer, pages 156--167. Springer

    2023

  41. [50]

    Yongkai Wu, Lu Zhang, Xintao Wu, and Hanghang Tong. 2019. https://arxiv.org/abs/1910.12586 Pc-fairness: A unified framework for measuring causality-based fairness . Preprint, arXiv:1910.12586

    2019 arXiv

  42. [51]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. https://arxiv.org/abs/2305.10601 Tree of thoughts: Deliberate problem solving with large language models . Preprint, arXiv:2305.10601

    2023 arXiv

  43. [52]

    Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. 2024. https://doi.org/10.1145/3597503.3623316 Codereval: A benchmark of pragmatic code generation with generative pre-trained models . In Proceedings of the IEEE...

    2024

  44. [53]

    Wenqing Zheng, S P Sharan, Ajay Kumar Jaiswal, Kevin Wang, Yihan Xi, Dejia Xu, and Zhangyang Wang. 2023. https://arxiv.org/abs/2305.00909 Outline, then details: Syntactically guided coarse-to-fine code generation . Preprint, arXiv:2305.00909

    2023 arXiv

  45. [54]

    Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Zhenqiang Gong, and Xing Xie. 2023. https://arxiv.org/abs/2306.04528 Promptbench: Towards evaluating the robustness of large language models on adversarial prompts ....

    2023 arXiv

  46. [55]

    Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. 2024. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931

    2024 arXiv

  47. [56]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  48. [57]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.