Pith. sign in

REVIEW 3 major objections 4 minor 33 references

ECLAIR is a causally grounded AI framework that embeds large language models as scientific agents in a human-checked workflow for empirical software engineering, demonstrated on a prompt-engineering case study with small negative causal eff

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 09:17 UTC pith:GXPGRJ73

load-bearing objection ECLAIR is a sensible framework paper saddled with a case study whose causal estimates are invalidated by post-treatment adjustment, so it needs major revision before the demonstrations can be trusted. the 3 major comments →

arxiv 2608.02323 v1 pith:GXPGRJ73 submitted 2026-08-03 cs.SE

ECLAIR: A Causally-Grounded AI Framework for Scientific Discovery in Empirical Software Engineering

Alejandro Velasco , Daniel Rodriguez-Cardenas , Dipin Khati , David N. Palacio , Denys Poshyvanyk This is my paper
classification cs.SE
keywords causal inferencelarge language modelssoftware engineeringscientific methodstructural causal modelsprompt engineeringcode generationhuman-in-the-loop
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to establish that large language models can be embedded as 'scientific agents' throughout the empirical software engineering (SE) research process—from literature mapping and hypothesis generation to experimental design, causal analysis, and interpretation—provided that human researchers approve each phase. The framework, ECLAIR, operationalizes structural causal model theory as a step-by-step method, grounding hypotheses in SE-specific artifacts like code and abstract syntax trees. The demonstration case study asks whether prompt design causally affects code generation accuracy; across two 7-billion-parameter models, the paper finds small negative average treatment effects for instruction-style, longer few-shot, and signature-augmented prompts. A sympathetic reader cares because this is a concrete proposal for making LLM-assisted research rigorous and causally explanatory, rather than merely correlational.

Core claim

The paper's central claim is that LLMs, when constrained by causal inference principles and human oversight, can contribute to every stage of the scientific method in software engineering. In the case study, the framework constructs a structural causal model (SCM) for each of three hypotheses, defines treatments, outcomes, and a confounder set of code-complexity and lexical features, and estimates average treatment effects on the CodeBLEU metric. The estimated effects are consistently small and negative for both evaluated models: instruction-style prompts reduce accuracy relative to code-delimited prompts; increasing the number of few-shot examples slightly decreases accuracy; and adding fun

What carries the argument

The load-bearing component is the per-hypothesis structural causal model (SCM) generated by the LLM-based Scientific Agent and vetted by the human researcher. Each SCM names the treatment, outcome, and a confounder set (for example, AST complexity metrics, token counts, whitespace, and prompt length), allowing the framework to apply back-door adjustment and compute the Average Treatment Effect (ATE) from experimental data. The SCM converts an otherwise associational comparison into a causal estimate, and the framework's phase-by-phase checkpoints are what make that estimate defensible as scientific evidence.

Load-bearing premise

The central premise is that the listed confounders—complexity, AST structure, token counts, and related lexical features—fully account for all alternative explanations, so that no unobserved variable drives both prompt choice and accuracy.

What would settle it

A direct test would be a randomized experiment on the same two models and dataset, where prompts are assigned randomly to code snippets and the difference in CodeBLEU is compared with ECLAIR's back-door-adjusted estimates; if the randomized and adjusted estimates disagree in sign or magnitude, the confounder set or the LLM-generated structural model is incomplete. Alternatively, adding a plausible unobserved confounder (e.g., problem difficulty) to the adjustment set and observing a meaningful change in the ATE would refute the claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the framework's causal estimates are sound, SE researchers can move from reporting correlations to explaining why a practice works, using interventions and counterfactuals rather than association.
  • The case study's negative effects imply that prompt engineering gains are not automatic: adding instructions, examples, or signatures can backfire for certain models and tasks.
  • Because ECLAIR documents every assumption, treatment, and confounder, it makes causal claims in SE research more transparent and reproducible than ad hoc experiments.
  • The framework's success on this focused phenomenon opens the door to studying more complex SE phenomena (e.g., bug prediction, code review) through the same causal lens.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The framework's reliance on LLM-generated SCMs means its causal conclusions inherit the LLM's understanding of SE; an independent check against expert-built SCMs would clarify whether the negative prompt effects are real or artifacts of a misspecified graph.
  • The small ATEs are estimated on a single dataset and two 7B models; extending the same causal design to larger models or other languages could reveal whether the direction of the effect changes with scale or domain.
  • The framework's design treats the phenomenon, dataset, and models as fixed in the first phase; a more ambitious version would let the Scientific Agent propose datasets and models, which would test whether the LLM's choices bias the resulting causal findings.
  • If the pattern of small negative effects replicates, it would motivate a search for the mechanism: perhaps extra tokens in a prompt dilute the model's attention, a hypothesis the framework's causal graphs could be extended to test.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes ECLAIR, an eight-phase, human-in-the-loop framework that uses LLMs as scientific agents for empirical SE research, with each phase explicitly aligned to Pearl's causal hierarchy and with mandatory researcher checkpoints. The framework is demonstrated through a case study of prompt-design effects on code generation accuracy for two 7B models, using the CodeText-Galeras dataset and DoWhy for causal estimation. The paper claims that instruction-style, longer few-shot, and signature-augmented prompts yield small negative causal effects on CodeBLEU for both models, and positions ECLAIR as the first causally grounded, executable instantiation of the scientific method for SE.

Significance. The methodological ambition is valuable: ECLAIR makes the scientific method operational for SE by pairing LLM-assisted hypothesis generation with causal identification, explicit human oversight, and a public replication package. If the framework works as described, it would address a real gap in SE research, where most studies report associations rather than causal effects. The strengths include a concrete phase-by-phase protocol, causal vocabulary (treatments, outcomes, confounders, SCMs), the use of SE-specific artifacts such as AST structure, and the availability of all scripts, prompts, and data. However, the paper's central empirical demonstration is compromised by a textbook adjustment-set error in Table 1: post-treatment variables are included as confounders, so the reported ATEs do not identify the claimed total causal effects. Since the case study is the evidence that ECLAIR delivers valid causal inference, this issue is load-bearing and must be fixed before the framework's validation can be accepted.

major comments (3)
  1. [Section 3, Table 1 (H1/H11/H13)] The adjustment set C includes prompt_length and #prompt_tokens as confounders for every hypothesis. For H1, the treatment is adding an instruction; for H11, it is adding few-shot samples; for H13, it is adding a signature. In all three cases, prompt_length and #prompt_tokens are deterministic or near-deterministic functions of the treatment (and of the code sample), i.e., post-treatment descendants of T, not pre-treatment common causes. Conditioning on them in a back-door adjustment blocks part of the causal path from T to CodeBLEU and can also open collider paths, so the resulting estimates do not represent the total ATE of prompt design. The paper never shows the SCM edges or the DoWhy-identified estimand; Table 1 only labels these variables as confounders. This is the standard bad-control mistake and it directly undermines the RQ1 answer and the abstract's causal claims. Please re-est
  2. [Section 3, Table 1 and 'Causal Inference Analysis'] No confidence intervals, standard errors, or p-values are reported for any ATE. The H11 estimate for M1 is -0.001, which is effectively zero, yet the abstract and RQ1 answer state that 'for both models ... instruction-style, longer, and signature-augmented prompts yield small negative causal effects.' The data do not support a negative effect for H11/M1 without uncertainty quantification. The robustness tests are mentioned but not reported: there are no test statistics, figures, or refutation results for the placebo, random common cause, data subset, or unobserved-confounder checks. Since the case study is the demonstration of ECLAIR's validity, the paper should report uncertainty intervals for all rows and show the robustness results, not merely assert their existence.
  3. [Section 3, Knowledge Mapping and Experimentation] The case study's causal target should be clarified. The text states that CodeText-Galeras does not directly pair prompts with accuracy scores, and that the Experiment Design phase resolved this by 'applying prompt treatments synthetically.' This means the treatment is the artificial insertion of instruction text, few-shot samples, or signatures into prompts, not prompt design as it occurs in real usage. That is a legitimate controlled manipulation, but it is an additional assumption that is never stated as a limitation in the RQ1 answer. Moreover, the SCMs were generated by GPT-5 and only superficially described in Table 1; no graph or identifying assumptions are given, and no evidence is provided that the human checkpoint would have caught the post-treatment control problem. Please state the estimand, show at least one example SCM, and discuss the synthetic-treatment assumption explicit
minor comments (4)
  1. [Section 3, Hypothesis H11] The treatment is described as 'continuous' but is the number of few-shot samples, which is inherently discrete and likely ordinal. Please clarify the coding and the number of levels used.
  2. [Section 3, Table 1] The table uses bold and purple color coding for correlation and causal effect. These visual distinctions may not survive monochrome printing or accessibility conversion; consider adding explicit symbols or a dedicated column.
  3. [Section 3, Knowledge Mapping] Sixteen hypotheses were generated but only three were tested, and the selection criterion is not described beyond 'representative.' Please state how the three were chosen and whether the untested hypotheses would change the conclusions if included.
  4. [Section 6 and References] The replication package is listed as a GitHub repository without a commit hash or archived version. For a reproducibility-oriented paper, please provide a versioned artifact identifier (e.g., Zenodo DOI or release tag).

Circularity Check

0 steps flagged

No circular derivation: the case-study ATEs are measured from experimental data, not fitted or presupposed; the only caveats are minor self-citations and a validity concern in the confounder set.

full rationale

The paper's central result, RQ1, is an empirical ATE estimate obtained by running DoWhy on experimentally generated model completions. There is no place where a parameter is fitted to a subset and then 'predicted'; the treatments (adding instructions, few-shot samples, signatures) are actual interventions, and the outcome (CodeBLEU) is independently measured. The ECLAIR framework is a procedural methodology; its novelty claims are not derived mathematically, so there is no derivation chain that could collapse onto its inputs. The self-citations ([15] causal pipeline, [21] CodeText-Galeras dataset, [23] replication repo) are real external artifacts and are not used to prove the ATE values; the estimation itself relies on the external DoWhy library and measured data. The Discussion explicitly limits generalizability ('generalizability across more complex SE phenomena, larger datasets, and additional models remains an open direction'), and Table 1's inclusion of prompt_length and #prompt_tokens as confounders (when they are deterministic functions of the treatment) is a substantive causal-identification concern, not a circularity: it threatens the validity of the ATEs, but the ATEs are still measured rather than derived from the assumption. Robustness checks are asserted ('robustness tests confirmed that the effects were stable and not vulnerable to confounding bias') but not detailed, another validity gap. None of these issues makes the result equivalent to its inputs, so no circular step is present.

Axiom & Free-Parameter Ledger

0 free parameters · 6 axioms · 0 invented entities

The central claim rests on the validity of Pearl's causal framework, the sufficiency of the chosen confounder set, the soundness of GPT-5's generated artifacts, and the representativeness of the dataset and models. No fitted free parameters are reported; the ATEs are estimates, not fitted constants. No new entities are introduced beyond the ECLAIR process itself.

axioms (6)
  • standard math Pearl's SCM framework and do-calculus are applicable to SE research questions and estimable from observational/experimental data
    The entire framework rests on Pearl's causal formalism (Ref [18]); this is a well-established mathematical theory, though its applicability to SE constructs is assumed rather than proven.
  • domain assumption The confounder set C = {complexity, #ast_levels, #ast_nodes, #ast_errors, #whitespaces, #tokens, vocab_size, #words, #identifiers, #loc, prompt_length, #prompt_tokens} suffices for back-door adjustment in all three hypotheses (no unobserved confounding)
    Invoked in Phase 6; the paper states robustness tests were run but does not report them, so the sufficiency of C is assumed. Table 1 lists C.
  • domain assumption The LLM (GPT-5) used as Scientific Agent generates sufficiently valid literature syntheses, hypotheses, and SCM structures that human vetting is adequate to maintain scientific rigor
    Section 2, Phase 2-4; the paper acknowledges hallucination risk and adds human checkpoints, but assumes the LLM outputs are salvageable by review.
  • domain assumption CodeBLEU is an adequate measure of code generation accuracy for the purpose of estimating prompt-design effects
    The outcome for all hypotheses; accuracy is operationalized solely as CodeBLEU, ignoring functional correctness or Execution Match.
  • domain assumption The CodeText-Galeras dataset and the two 7B models (Qwen2.5-Coder-7B, CodeLlama-7b-Python-hf) are representative enough to support the reported causal estimates
    Section 3, Model and Dataset paragraphs; the paper fixes these upfront and acknowledges generalizability is limited in Discussion.
  • ad hoc to paper Synthetic application of prompt treatments to the dataset captures the real intervention of prompt design
    The dataset does not pair prompts with accuracy scores; the paper applies treatments synthetically, assuming this reproduces genuine prompt-design effects. Section 3, Knowledge Mapping.

pith-pipeline@v1.3.0-daily-deepseek · 9602 in / 11431 out tokens · 94289 ms · 2026-08-04T09:17:01.461173+00:00 · methodology

0 comments
read the original abstract

The scientific method has long guided empirical research in Software Engineering (SE), but the complexity of modern software systems often hinders its systematic application. This paper introduces _ECLAIR_, a causally grounded AI framework that integrates Large Language Models (_LLMs_) into every stage of the scientific process, from hypothesis generation to analysis and interpretation. _ECLAIR_ treats _LLMs_ as active **scientific agents** operating under the principles of causal inference, within a human-in-the-loop design that safeguards against the risks of unsound automated reasoning. We demonstrate the framework through a case study examining how prompt design influences code generation accuracy in two _LLMs_. Results show that, for both models, instruction-style, longer few-shot, and signature-augmented prompts yield small negative causal effects on accuracy, illustrating how causal reasoning provides a principled foundation for explaining _why_ software phenomena occur. This study presents the first causally grounded structured methodology for embedding _LLMs_ within the scientific method in SE, designed around the epistemological demands of empirical SE research, establishing a basis for rigorous AI-assisted research.

Figures

Figures reproduced from arXiv: 2608.02323 by Alejandro Velasco, Daniel Rodriguez-Cardenas, David N. Palacio, Denys Poshyvanyk, Dipin Khati.

Figure 1
Figure 1. Figure 1: Our ECLAIR framework (left) contrasted with the canonical scientific method (right) causal effects (e.g., the Average Treatment Effect (ATE)) for each tested hypothesis while controlling for confounding bias, output: ATE estimates and robustness/refutation results per method, fol￾lowing the principles of valid, transparent, and logically consistent inference [4]. (7) Interpretation. In this phase, causal r… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 13 linked inside Pith

  1. [1]

    Introducing GPT-5

    2025. Introducing GPT-5. https://openai.com/index/introducing-gpt-5/

    2025

  2. [2]

    Collins, Elizabeth Bourne, Gareth W

    Abbi Abdel-Rehim, Hector Zenil, Oghenejokpeme Orhobor, Marie Fisher, Ross J. Collins, Elizabeth Bourne, Gareth W. Fearnley, Emma Tate, Holly X. Smith, Lar- isa N. Soldatova, and Ross King. 2025. Scientific hypothesis generation by large language models: laboratory validation in breast cancer treatment.Journal of The Royal Society Interface22, 227 (June 20...

    arXiv 2025

  3. [3]

    Toufique Ahmed, Premkumar Devanbu, Christoph Treude, and Michael Pradel

  4. [4]

    Scott Armstrong and Kesten C

    J. Scott Armstrong and Kesten C. Green. 2022.The Scientific Method: A Guide to Finding Useful Knowledge. Cambridge University Press. doi:10.1017/ 9781009092265

    2022

  5. [5]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    Pith/arXiv arXiv 2023

  6. [6]

    Sebastian Baltes, Florian Angermeir, Chetan Arora, Marvin Muñoz Barón, Chun- yang Chen, Lukas Böhme, Fabio Calefato, Neil Ernst, Davide Falessi, Brian Fitzgerald, Davide Fucci, Marcos Kalinowski, Stefano Lambiase, Daniel Russo, Mircea Lungu, Lutz Prechelt, Paul Ralph, Rijnard van Tonder, Christoph Treude, and Stefan Wagner. 2025. Guidelines for Empirical ...

    Pith/arXiv arXiv 2025

  7. [7]

    Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes

    Daniil A. Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. 2023. Au- tonomous chemical research with large language models.Nature624, 7992 (Dec. 2023), 570–578. Publisher: Nature Publishing Group. doi:10.1038/s41586-023- 06792-0

    doi:10.1038/s41586-023- 2023

  8. [8]

    Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching Large Language Models to Self-Debug. arXiv:2304.05128 [cs.CL] https://arxiv. org/abs/2304.05128

    Pith/arXiv arXiv 2023

  9. [9]

    Giuseppe Crupi, Rosalia Tufano, Alejandro Velasco, Antonio Mastropaolo, Denys Poshyvanyk, and Gabriele Bavota. 2025. On the Effectiveness of LLM-as-a- Judge for Code Generation and Summarization.IEEE Transactions on Software Engineering51, 8 (2025), 2329–2345. doi:10.1109/TSE.2025.3586082

    arXiv 2025

  10. [10]

    Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen tau Yih, Luke Zettlemoyer, and Mike Lewis. 2023. InCoder: A Generative Model for Code Infilling and Synthesis. arXiv:2204.05999 [cs.SE] https://arxiv.org/abs/2204.05999

    Pith/arXiv arXiv 2023

  11. [11]

    Fengqing Jiang, Yichen Feng, Yuetai Li, Luyao Niu, Basel Alomair, and Radha Poovendran. 2025. BadScientist: Can a Research Agent Write Convincing but Unsound Papers that Fool LLM Reviewers? arXiv:2510.18003 [cs]. doi:10.48550/ arXiv.2510.18003

    Pith/arXiv arXiv doi:10.48550/arxiv.2510.18003 2025

  12. [12]

    Polak, So Yeon Kim, MD Al Amin Shuvo, Hrishikesh Shridhar Deodhar, Jeongsoo Han, Dane Morgan, and Hyunseok Oh

    Quanliang Liu, Maciej P. Polak, So Yeon Kim, MD Al Amin Shuvo, Hrishikesh Shridhar Deodhar, Jeongsoo Han, Dane Morgan, and Hyunseok Oh. 2025. Beyond designer’s knowledge: Generating materials design hy- potheses via a large language model.Acta Materialia297 (Sept. 2025), 121307. doi:10.1016/j.actamat.2025.121307

    arXiv 2025

  13. [13]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-Refine: Iterative Refinement with Self-Feedback. arXiv:2303.17651 [cs.CL] https://arxiv.or...

    Pith/arXiv arXiv 2023

  14. [14]

    Landsness, Daniel L

    Ludovico Mitchener, Angela Yiu, Benjamin Chang, Mathieu Bourdenx, Tyler Nadolski, Arvis Sulovari, Eric C. Landsness, Daniel L. Barabasi, Siddharth Narayanan, Nicky Evans, Shriya Reddy, Martha Foiani, Aizad Kamal, Leah P. Shriver, Fang Cao, Asmamaw T. Wassie, Jon M. Laurent, Edwin Melville-Green, Mayk Caldas, Albert Bou, Kaleigh F. Roberts, Sladjana Zagora...

    Pith/arXiv arXiv doi:10.48550/arxiv.2511.02824 2025

  15. [15]

    David Nader Palacio, Alejandro Velasco, Nathan Cooper, Alvaro Rodriguez, Kevin Moran, and Denys Poshyvanyk. 2024. Toward a Theory of Causation for Inter- preting Neural Code Models.IEEE Transactions on Software Engineering50, 5 (May 2024), 1215–1243. doi:10.1109/tse.2024.3379943

    arXiv 2024

  16. [16]

    Alexander Novikov, Ngân V ˜u, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, and Matej Balog. 2025. AlphaEvolve: A coding agent for scientific an...

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.13131 2025

  17. [17]

    Yang Jeong Park, Daniel Kaplan, Zhichu Ren, Chia-Wei Hsu, Changhao Li, Haowei Xu, Sipei Li, and Ju Li. 2024. Can ChatGPT be used to generate scientific hypotheses?Journal of Materiomics10, 3 (May 2024), 578–584. doi:10.1016/j.jmat.2023.08.007

    doi:10.1016/j.jmat.2023.08.007 2024

  18. [18]

    2009.Causality: models, reasoning, and inference

    Judea Pearl. 2009.Causality: models, reasoning, and inference. doi:10.1017/ CBO9780511803161

    2009

  19. [19]

    Judea Pearl. 2018. Theoretical Impediments to Machine Learning With Seven Sparks from the Causal Revolution. arXiv:1801.04016 [cs]. doi:10.48550/arXiv. 1801.04016

    Pith/arXiv arXiv doi:10.48550/arxiv.1801.04016 2018

  20. [20]

    Reddy and Parshin Shojaee

    Chandan K. Reddy and Parshin Shojaee. 2025. Towards scientific discovery with generative AI: progress, opportunities, and challenges. InProceedings of the Thirty- Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence and Fifteenth Symposium on Educational Advances in Artificial...

    doi:10.1609/aaai.v39i27.35084 2025

  21. [21]

    Palacio, Dipin Khati, Henry Burke, and Denys Poshyvanyk

    Daniel Rodriguez-Cardenas, David N. Palacio, Dipin Khati, Henry Burke, and Denys Poshyvanyk. 2023. Benchmarking Causal Study to Interpret Large Lan- guage Models for Source Code. arXiv:2308.12415 [cs.SE] https://arxiv.org/abs/ 2308.12415

    Pith/arXiv arXiv 2023

  22. [22]

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiao- qing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cris- tian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, ...

    Pith/arXiv arXiv 2024

  23. [23]

    2025.ci4sesci: Causal Inference for Software Science

    SEMERU Lab. 2025.ci4sesci: Causal Inference for Software Science. GitHub repository. https://github.com/WM-SEMERU/ci4sesci

    2025

  24. [24]

    Amit Sharma and Emre Kiciman. 2020. DoWhy: An End-to-End Library for Causal Inference.arXiv preprint arXiv:2011.04216(2020). doi:10.48550/arXiv.2011.04216

    Pith/arXiv arXiv doi:10.48550/arxiv.2011.04216 2020

  25. [25]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. InProceedings of the 37th International Conference on Neural Information Processing Systems(New Orleans, LA, USA)(NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, Article 377, 19 pages. doi:10.52202/0...

    doi:10.52202/075280-0377 2023

  26. [26]

    Song Tong, Kai Mao, Zhen Huang, Yukun Zhao, and Kaiping Peng. 2024. Au- tomating psychological hypothesis generation with AI: when large language models meet causal graph.Humanities and Social Sciences Communications11, 1 (July 2024). doi:10.1057/s41599-024-03407-5

    doi:10.1057/s41599-024-03407-5 2024

  27. [27]

    Bayer, Aidan Acquah, Chetana Amancharla, Almog Barzilay-Siegal, Pablo Bermejo, Camille Bilodeau, Pablo Cárdenas Ramírez, Miles Cranmer, Ur- bano L

    Francisco Villaescusa-Navarro, Boris Bolliet, Pablo Villanueva-Domingo, Adrian E. Bayer, Aidan Acquah, Chetana Amancharla, Almog Barzilay-Siegal, Pablo Bermejo, Camille Bilodeau, Pablo Cárdenas Ramírez, Miles Cranmer, Ur- bano L. França, ChangHoon Hahn, Yan-Fei Jiang, Raul Jimenez, Jun-Young Lee, Antonio Lerario, Osman Mamun, Thomas Meier, Anupam A. Ojha,...

  28. [28]

    Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, Anima Anandkumar, Karianne Bergen, Carla P. Gomes, Shirley Ho, Pushmeet Kohli, Joan Lasenby, Jure Leskovec, Tie-Yan Liu, Arjun Manrai, Debora Marks, Bharath Ramsundar, Le Song, Jimeng Sun, Jian Tang, Petar Veličković, Max...

    2023

  29. [29]

    arXiv:2510.26887 [cs.AI] https://arxiv.org/abs/2510.26887

    The Denario project: Deep knowledge AI agents for scientific discovery. arXiv:2510.26887 [cs.AI] https://arxiv.org/abs/2510.26887

    arXiv

  30. [30]

    Khan, Adnan Mahmud, Huck Yang, Alexander Lavin, Michael Levin, Jeremy Frey, Jared Dunnmon, James Evans, Alan Bundy, Saso Dzeroski, Jesper Tegner, and Hector Zenil

    Yanbo Zhang, Sumeer A. Khan, Adnan Mahmud, Huck Yang, Alexander Lavin, Michael Levin, Jeremy Frey, Jared Dunnmon, James Evans, Alan Bundy, Saso Dzeroski, Jesper Tegner, and Hector Zenil. 2025. Exploring the role of large language models in the scientific method: from hypothesis to discovery.npj Artificial Intelligence1, 1 (Aug. 2025), 14. doi:10.1038/s443...

    doi:10.1038/s44387-025-00019-5 2025

  31. [31]

    Ohlsson, Bj ¨rn Regnell, and Anders Wesslén

    Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, Bj ¨rn Regnell, and Anders Wesslén. 2012.Experimentation in Software Engineering. Springer. doi:10.1007/978-3-642-29044-2

    doi:10.1007/978-3-642-29044-2 2012

  32. [33]

    Yangqiaoyu Zhou, Haokun Liu, Tejes Srivastava, Hongyuan Mei, and Chenhao Tan. 2024. Hypothesis Generation with Large Language Models. InProceedings of the 1st Workshop on NLP for Science (NLP4Science), Lotem Peled-Cohen, Nitay ASE ’26, October 12–16, 2026, Munich, Germany Velasco et al. Calderon, Shir Lissak, and Roi Reichart (Eds.). Association for Compu...

    doi:10.18653/v1/2024.nlp4science-1.10 2024

  33. [2025]

    Can LLMs Replace Manual Annotation of Software Engineering Artifacts? arXiv:2408.05534 [cs.SE] https://arxiv.org/abs/2408.05534

    Pith/arXiv arXiv