Pith. sign in

REVIEW 4 major objections 3 minor 53 references

The paper argues that foundation models can detect, explain, and often repair compilation errors that surface only under specific feature combinations in configurable C code, complementing—not replacing—compiler-based and variability-aware

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 08:26 UTC pith:DZQLH32I

load-bearing objection Useful new benchmark and a reasonable empirical study, but the reporting is sloppy and the headline claim of 'localized' detection isn't actually measured. the 4 major comments →

arxiv 2601.16755 v2 pith:DZQLH32I submitted 2026-01-23 cs.SE

An Empirical Study of Foundation Models for Variability-Induced Compilation Errors in Configurable C Code

Rohit Gheyi , Lucas Albuquerque , M\'arcio Ribeiro , Eduardo Almeida , Danyllo Albuquerque , Mirko Perkusich This is my paper
classification cs.SE
keywords configurable systemsconditional compilation#ifdefcompilation errorsfoundation modelslarge language modelsprogram repairvariability bugs
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that foundation models can detect, explain, and often repair compilation errors that appear only under certain combinations of #ifdef feature macros in configurable C code—errors that ordinary compilers miss because they compile one configuration at a time. The authors built a benchmark of 5,000 small C snippets (4–25 lines, 1–5 Boolean macros), validated every snippet by exhaustively compiling all configurations with Clang, and then probed GPT-OSS-20B and Gemini 3 Pro with a standardized prompt asking for failing configurations, an explanation, and a fixed file. In the body's primary evaluation, the open model flagged whether a snippet had any failing configuration with 0.97 precision and 0.90 recall, and produced fixes that compiled under every configuration in 70.2% of faulty snippets. A complementary change-based analysis of real-world Git commit diffs found the models could localize injected faults and a plausible real Linux bug, though the abstract and body report different per-configuration detection numbers. The practical payoff, if the claims hold, is a low-setup way to catch configuration-dependent compile errors early.

Core claim

The paper's central claim is that foundation models, when prompted with a software-product-line view of #ifdef code, can localize and explain configuration-dependent compilation errors and can often repair them without removing variability. In the body's primary evaluation, GPT-OSS-20B decided whether a configurable system had any failing configuration with 0.97 precision and 0.90 recall, and produced code that compiled under every Boolean feature assignment in 1,870 of 2,665 faulty snippets (70.2%). The abstract states different, stricter per-configuration numbers—84.7% micro-precision and 52.1% micro-recall for affected configurations—and reports that coverage depends on whether the model

What carries the argument

The unit of analysis is the configurable C file (or git diff) mapped to a software product line: each #if/#ifdef macro is a feature and each Boolean assignment is a product. A Meta-Prompt-generated instruction asks the model to return JSON with the failing products, an explanation, and a corrected version, with constraints that forbid adding or removing macros; correctness is established by exhaustively compiling all products with Clang. This combination—LLM reasoning plus a compiler oracle—carries the argument.

Load-bearing premise

The load-bearing premise is that the 5,000 synthetic snippets—4–25 lines with 1–5 Boolean macros, generated by ChatGPT-5.2 from 30 seed bugs—represent the distribution of variability-induced compilation errors in real configurable systems; Section 5.7 concedes that small systems may inflate performance.

What would settle it

Compile every Boolean feature assignment of the 5,000 released snippets and of a sample of real file-level subjects with Clang, re-run the models with the paper's exact prompt, and compare the reported failing configurations to the compiler's list; if agreement is low—or if swapping compiler versions changes the baseline—the central generalization fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A practical workflow could run a cheap open-weight model on every commit diff and escalate to a larger model only for large diffs or low-confidence cases, a tiered usage pattern the paper explicitly recommends.
  • Foundation-model analysis and variability-aware parsers are complementary: in the paper's comparison, TypeChef caught every faulty snippet but produced many false positives, while GPT-OSS-20B had far fewer false positives, so combining them could raise precision without losing recall.
  • Repair constrained to preserve variability succeeded in about 70% of faulty small systems, making semi-automated repair of configurable units a realistic near-term target.
  • Context-window limits make full-kernel analysis impractical, but the paper's file- and diff-level granularity still caught a plausible real issue in a 1,197-line Linux commit.
  • The publicly released 5,000-snippet dataset becomes a benchmark for future variability-aware analysis and for fine-tuning smaller models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The abstract's reporting-style result—99.4% coverage with presence conditions versus 29.5% with explicit enumeration—suggests that prompt output format may matter as much as model choice; an immediate experiment would vary only that instruction on the released dataset.
  • The nondeterminism results (pass@k reaching 1.0 for detection at k=4 on the 357-snippet sample) imply that a deployed tool should sample multiple responses and vote on a consensus rather than trust a single call, an operational improvement the paper does not push forward.
  • The gap between system-level figures (0.97/0.90) and configuration-level figures in the abstract (84.7% micro-precision, 52.1% micro-recall) means the claim is much stronger for 'is there a problem somewhere' than for 'here is every failing configuration'; a reader should decide which claim they need.
  • The same protocol could transfer to other conditional-compilation ecosystems (Rust cfg, C++ #if, Java preprocessors) and to non-compile failures such as undefined behavior or security weaknesses; that extension remains untested.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper reports an empirical study of foundation models for detecting and repairing compilation errors that appear only under specific C preprocessor configurations. Study I uses 5,000 compiler-validated synthetic snippets generated by ChatGPT-5.2, evaluates GPT-OSS-20B on the full set and Gemini on a stratified sample, and compares against TypeChef. Study II is a change-based analysis of real-world Git commits plus author-generated mutations. The abstract claims high configuration-level micro-precision/recall, high repair rates, and success on real-world subjects, concluding that foundation models can support localized detection, explanation, and triage. The body, however, measures detection as a system-level existence check, reports different headline numbers, and contains no configuration-level analysis of the sort promised in the abstract.

Significance. The topic is timely, and several design choices are commendable: the ground truth is obtained by exhaustively compiling all Boolean configurations with Clang, TypeChef provides a relevant variability-aware baseline, and the repeated-run stability metrics (pass@k, tar@k, cons@k) are a thoughtful addition. If the claims were supported, the paper would be a useful data point for LLM-based variability analysis. In its current form, however, the central claim of 'localized detection' is not backed by the measured metric, and the abstract and full text describe what appear to be different studies. The paper cannot be accepted without re-analysis or substantial weakening of the claims.

major comments (4)
  1. [§3.5, §4.1, Abstract] The detection metric is system-level. A response is considered correct if the model reports at least one non-compiling configuration and the baseline confirms at least one compilation error. This checks only whether the snippet has any error, not whether the model localized the error to the correct configurations. The abstract's headline numbers (84.7% micro-precision, 52.1% micro-recall for affected configurations) do not appear in the body, and the body's 0.97/0.90 precision/recall are based on the existence check. Because the central claim is 'localized detection, explanation, and triage', the paper needs a configuration-level evaluation, or the claim must be weakened to system-level error-presence detection.
  2. [Abstract vs. body] The abstract and full text report different numbers and model names. Repair: the abstract says 1,930/2,665 (72.4%); §4.2 and the introduction say 1,870/2,665 (70.2%). Study II: the abstract says 100 faulty file-level subjects, 94 target-fault-aligned, five projects, and models Codex-GPT5.5 and Gemini 3.6 Flash; the body reports 14 real commits plus 42 mutations (56 subjects) from seven projects, with ChatGPT-5.2 and GPT-OSS-20B, and states that Gemini fix evaluation was impossible due to malformed JSON. The abstract's 99.4%/29.5% coverage numbers and the counterfactual audit are not present in the body. The authors must reconcile these discrepancies before any claims can be evaluated.
  3. [§5.2] The Gemini evaluation is based only on the parseable subset: 45.4% of responses were unparseable or malformed and were excluded. The reported precision 1.00 and recall 0.88 are therefore conditional on parseability and do not support the abstract's claim that Gemini restored compilability in 95.8% of cases; the body explicitly says the fix task could not be evaluated for Gemini. The authors should report the true denominator, describe how missing responses were handled, and either supply a fix evaluation for Gemini or remove the claim.
  4. [§5.5, §5.5.5] The real-world study lacks independent ground truth. For 13 of the 14 original commits, the conclusion that no error exists rests essentially on the model not reporting one; only the Gnuplot case is checked by manual source inspection, and the Linux 'possible bug' is not confirmed by compilation or by a reproducer. The 42 mutation scenarios are constructed by the authors and have no external benchmark. The abstract's statement that 94 of 100 subjects had target-fault-aligned problems is not in the body. This evidence is too weak to support the generalization to real-world triage.
minor comments (3)
  1. [§3.2 / §7] The paper says the dataset is publicly available, but no URL, DOI, or artifact repository is provided; reference [18] points only to 'supplementary material.' Please add a working link for the dataset and any analysis scripts.
  2. [Throughout] Model names are inconsistent: the abstract uses 'Gemini 3.6 Flash' and 'Codex-GPT5.5', while the body uses 'GEMINI3 PRO' and 'CHATGPT-5.2'. Please standardize the names and ensure the version identifiers match the experiments actually run.
  3. [§5.1.2] The stability analysis is a strength, but the reported pass@1 of 0.978 on the 357-instance sample differs numerically from the 0.90/0.97 precision-recall in §4.1. Clarify whether these are separate metrics on different subsets or two versions of the same result.

Circularity Check

0 steps flagged

No constructional circularity; evaluation is grounded in external compiler ground truth, though abstract/body metric mismatches are validity concerns.

full rationale

The study is an empirical evaluation, not a derivation, and its load-bearing ground truth is external: every configuration is compiled with CLANG-1600.0.26.6 (Sections 3.2.2, 3.5, 4.2), and fixes are checked by recompiling all derived products. Detection is scored against that compiler baseline, not against a quantity derived from the model or from a fitted parameter. No parameter is fitted from the evaluation data and then re-predicted, no uniqueness theorem is imported from the authors, and no ansatz is smuggled in via citation. Self-citations such as [28] supply seed bugs and prior context, but the dataset construction also cites external sources [5], [6], and the empirical claims do not reduce to those citations. The main weaknesses are validity issues rather than circularity: Section 3.5 defines a detection response as correct 'if the model reports at least one non-compiling configuration and the baseline confirms the existence of at least one compilation error,' which is an existence-level check and does not verify configuration-level localization; Section 5.7 explicitly acknowledges that small systems may inflate performance; and the abstract's configuration-level 'micro-precision'/'micro-recall' and the 72.4% repair rate are not derivable from the body's reported 0.97/0.90 precision/recall and 70.2% repair rate (Section 4.2). These are gaps between evidence and claims, not instances where a prediction is equivalent to its input by construction, so the circularity score is 0.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claims rest on the representativeness of a synthetic benchmark, on compiler output as ground truth, and on compilation success as the repair criterion—none of which is independently verified outside the paper.

free parameters (3)
  • temperature (GPT-OSS-20B) = 0.4
    Set by hand in Section 3.4; affects output diversity and thus all reported metrics, but no sensitivity analysis is provided.
  • temperature (GEMINI3 PRO) = 0.2
    Set in Section 5.2 with retries/backoff; again a hand-chosen inference configuration.
  • dataset seed count = 30
    Section 3.2.1: 30 real bugs from prior literature seed the ChatGPT-5.2 synthesis; the choice determines benchmark diversity.
axioms (4)
  • domain assumption CLANG-1600.0.26.6 on macOS Tahoe 26.1 provides correct ground truth for compilation errors
    Section 3.2.2/3.5. Compiler diagnostics define the label; compiler bugs or platform-specific behavior would shift all metrics.
  • ad hoc to paper The 5,000 ChatGPT-5.2-generated snippets are representative of real-world variability-induced compilation errors
    Section 3.2.1 and Section 5.7. The benchmark is synthetic and small (4-25 LOC); generalizability to large systems is assumed rather than demonstrated.
  • domain assumption C99 semantics are the appropriate standard for judging compilation
    Section 3.3 prompt. Many real projects use GNU extensions or newer standards, so C99-only assessment may mislabel valid code.
  • ad hoc to paper Fixes are correct if all configurations compile
    Section 3.5 and abstract. The paper explicitly acknowledges compiler acceptance does not establish semantic correctness, so this is a stated, load-bearing simplification.

pith-pipeline@v1.3.0-alltime-deepseek · 21818 in / 9958 out tokens · 90441 ms · 2026-08-03T08:26:57.314860+00:00 · methodology

0 comments
read the original abstract

In configurable systems, conditional compilation can hide compilation errors under untested feature combinations. We investigate foundation models for detecting such errors and, in a controlled setting, restoring compilability in configurable C code. Study I evaluates GPT-OSS-20B on 5,000 synthetic snippets generated by ChatGPT-5.2 from 30 curated seeds and exhaustively compiled under all Boolean feature assignments; it also compares TypeChef and evaluates Gemini 3.6 Flash on a stratified sample. GPT-OSS-20B achieved 84.7% micro-precision and 52.1% micro-recall for affected configurations. Coverage depended on reporting style: presence conditions covered 99.4% of failing configurations, whereas explicit enumerations covered 29.5% under a prompt requesting only a minimal justifiable set. GPT-OSS-20B restored compilability for 1,930 of 2,665 faulty snippets (72.4%), while Gemini 3.6 Flash did so for 182 of 190 sampled faulty snippets (95.8%). A paired counterfactual audit found no evidence that an identified label-correlated #define property materially influenced GPT-OSS-20B's predictions. Study II evaluates Codex-GPT5.5 on 100 faulty file-level subjects from five mature configurable systems and reports target-fault-aligned problems in 94 subjects, including four of five historical bugs. Overall, foundation models can support localized detection, explanation, and triage, but should complement compiler-based and variability-aware analyses; compiler acceptance does not establish semantic correctness

Figures

Figures reproduced from arXiv: 2601.16755 by Danyllo Albuquerque, Eduardo Almeida, Lucas Albuquerque, M\'arcio Ribeiro, Mirko Perkusich, Rohit Gheyi.

Figure 1
Figure 1. Figure 1: Methodology for evaluating models to detect and fix compilation errors (CEs) in configurable systems (CSs). RQ1 To what extent can GPT-OSS-20B correctly detect the presence of compilation errors across configurations of configurable systems? RQ2 To what extent can GPT-OSS-20B correctly fix compila￾tion errors in configurable systems, producing code that compiles successfully across affected configurations?… view at source ↗
Figure 2
Figure 2. Figure 2: Accuracy and stability metrics for compilation-error detection by GPT-OSS-20B across k repeated runs on the 357-instance sample. meaning that more than 85% of the generated fixes compiled successfully on the first attempt. As additional attempts are allowed, the pass@k metric increases steadily, reaching 0.905 for k = 2 and 0.952 for k = 5. This trend shows that repeated querying increases the likelihood t… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 1 canonical work pages

  1. [1]

    An analysis of the variability in forty preprocessor-based software prod- uct lines,

    J. Liebig, S. Apel, C. Lengauer, C. Kästner, and M. Schulze, “An analysis of the variability in forty preprocessor-based software prod- uct lines,” in Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering. ACM, 2010, pp. 105–114

    2010

  2. [2]

    The discipline of preprocessor-based annotations does #ifdef tag n’t #endif matter,

    R. Malaquias, M. Ribeiro, R. Bonifácio, E. Monteiro, F. Medeiros, A. Garcia, and R. Gheyi, “The discipline of preprocessor-based annotations does #ifdef tag n’t #endif matter,” in International Conference on Program Comprehension, 2017, pp. 297–307

    2017

  3. [3]

    Preprocessor conditional removal by simple partial evaluation,

    I. D. Baxter and M. Mehlich, “Preprocessor conditional removal by simple partial evaluation,” in Proceedings of the Eighth Working Conference on Reverse Engineering. IEEE Computer Society, 2001, pp. 281–290

    2001

  4. [4]

    The love/hate relationship with the C preprocessor: An interview study,

    F. Medeiros, C. Kästner, M. Ribeiro, S. Nadi, and R. Gheyi, “The love/hate relationship with the C preprocessor: An interview study,” in European Conference on Object-Oriented Programming, 2015, pp. 495–518

    2015

  5. [5]

    42 variability bugs in the Linux kernel: a qualitative analysis,

    I. Abal, C. Brabrand, and A. Wasowski, “42 variability bugs in the Linux kernel: a qualitative analysis,” in ACM/IEEE International Conference on Automated Software Engineering. ACM, 2014, pp. 421–432

    2014

  6. [6]

    Variability bugs in highly configurable systems: A qualitative analysis,

    I. Abal, J. Melo, S. St˘ anciulescu, C. Brabrand, M. Ribeiro, and A. Wasowski, “Variability bugs in highly configurable systems: A qualitative analysis,” Transactions on Software Engineering and Methodology, vol. 26, no. 3, pp. 10:1–10:34, 2018

    2018

  7. [7]

    A comparison of 10 sampling algorithms for configurable systems,

    F. Medeiros, C. Kastner, M. Ribeiro, R. Gheyi, and S. Apel, “A comparison of 10 sampling algorithms for configurable systems,” in International Conference on Software Engineering, 2016, pp. 643– 654

    2016

  8. [8]

    Investigating preprocessor- based syntax errors,

    F. Medeiros, M. Ribeiro, and R. Gheyi, “Investigating preprocessor- based syntax errors,” in Generative Programming: Concepts and Experiences. ACM, 2013, pp. 75–84

    2013

  9. [9]

    An empirical study on configuration-related issues: investigating un- declared and unused identifiers,

    F. Medeiros, I. Rodrigues, M. Ribeiro, L. Teixeira, and R. Gheyi, “An empirical study on configuration-related issues: investigating un- declared and unused identifiers,” in Proceedings of the Generative Programming: Concepts and Experiences, ser. GPCE, 2015, pp. 35– 44

    2015

  10. [10]

    Variability-aware parsing in the presence of lexical macros and conditional compilation,

    C. Kästner, P . G. Giarrusso, T. Rendel, S. Erdweg, K. Ostermann, and T. Berger, “Variability-aware parsing in the presence of lexical macros and conditional compilation,” in Proceedings of the 26th Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications. ACM, 2011, pp. 805–824

    2011

  11. [11]

    SuperC: parsing all of C by taming the preprocessor,

    P . Gazzillo and R. Grimm, “SuperC: parsing all of C by taming the preprocessor,” inACM SIGPLAN Conference on Programming Language Design and Implementation. ACM, 2012, pp. 323–334

    2012

  12. [12]

    Goodfellow, Y

    I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. MIT Press, 2016

    2016

  13. [13]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, 2017, pp. 5998–6008

    2017

  14. [14]

    Aicodereview: Advancing code quality with ai-enhanced reviews,

    Y. Almeida, D. Albuquerque, E. Dantas Filho, F. Muniz, K. de Farias Santos, M. Perkusich, H. Almeida, and A. Perkusich, “Aicodereview: Advancing code quality with ai-enhanced reviews,” SoftwareX, vol. 26, p. 101677, 2024

    2024

  15. [15]

    Large language models for software engineering: A systematic literature review,

    X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review,” ACM Transactions on Software Engineering and Methodology, vol. 33, no. 8, 2024

    2024

  16. [16]

    Software testing with large language models: Survey, landscape, and vision,

    J. Wang, Y. Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, “Software testing with large language models: Survey, landscape, and vision,” IEEE Transactions on Software Engineering, vol. 50, pp. 911–936, 2024

    2024

  17. [17]

    The new compiler stack: a survey on the synergy of LLMs and compilers,

    S. Zhang, J. Zhao, Q. Yu, C. Xia, Z. Wang, X. Feng, and H. Cui, “The new compiler stack: a survey on the synergy of LLMs and compilers,” CCF Transactions on High Performance Computing, Jan. 2026

    2026

  18. [18]

    Artifacts,

    R. Gheyi, L. Albuquerque, M. Ribeiro, E. Almeida, D. Albuquerque, and M. Perkusich, “Artifacts,” See supplementary material., 2026

    2026

  19. [19]

    Hints on test data selection: Help for the practicing programmer,

    R. A. DeMillo, R. J. Lipton, and F. G. Sayward, “Hints on test data selection: Help for the practicing programmer,” IEEE Computer, vol. 11, no. 4, pp. 34–41, Apr. 1978

    1978

  20. [20]

    An analysis and survey of the development of mutation testing,

    Y. Jia and M. Harman, “An analysis and survey of the development of mutation testing,” IEEE Transactions on Software Engineering, vol. 37, no. 5, pp. 649–678, 2011

    2011

  21. [21]

    P . C. Clements and L. M. Northrop, Software Product Lines: Practices and Patterns. Boston, MA: Addison-Wesley, 2001

    2001

  22. [22]

    Discipline matters: Refactoring of preprocessor directives in the #ifdef hell,

    F. Medeiros, M. Ribeiro, R. Gheyi, S. Apel, C. Kästner, B. Ferreira, L. Carvalho, and B. Fonseca, “Discipline matters: Refactoring of preprocessor directives in the #ifdef hell,” IEEE Transactions on Software Engineering, vol. 44, no. 5, pp. 453–469, 2018

    2018

  23. [23]

    Evaluating refactorings for disciplining #ifdef annotations: An eye tracking study with novices,

    J. A. S. da Costa, R. Gheyi, M. Ribeiro, S. Apel, V . Alves, B. Fonseca, F. Medeiros, and A. Garcia, “Evaluating refactorings for disciplining #ifdef annotations: An eye tracking study with novices,” Empirical Software Engineering (EMSE), vol. 26, no. 5, p. 92, 2021

    2021

  24. [24]

    Feature interaction in software product line engineering: A systematic mapping study,

    L. R. Soares, P . Schobbens, I. do Carmo Machado, and E. S. de Almeida, “Feature interaction in software product line engineering: A systematic mapping study,” Inf. Softw. Technol., vol. 98, pp. 44–58, 2018. [Online]. Available: https: //doi.org/10.1016/j.infsof.2018.01.016

    work page doi:10.1016/j.infsof.2018.01.016 2018

  25. [25]

    Assessing fine-grained feature dependencies,

    I. Rodrigues, M. Ribeiro, F. Medeiros, P . Borba, B. Fonseca, and R. Gheyi, “Assessing fine-grained feature dependencies,” Information and Software Technology (IST), vol. 78, pp. 27–52, 2016

    2016

  26. [26]

    How configurable is the linux kernel? analyzing two decades of feature-model history,

    E. Kuiter, C. Sundermann, T. Thüm, T. Heß, S. Krieter, and G. Saake, “How configurable is the linux kernel? analyzing two decades of feature-model history,” ACM Transactions on Software Engineering and Methodology, vol. 35, no. 1, Dec. 2025

    2025

  27. [27]

    The Goal Question Metric Approach,

    V . Basili, G. Caldiera, and H. Rombach, “The Goal Question Metric Approach,” Encyclopedia of software engineering, pp. 528–532, 1994

    1994

  28. [28]

    Evaluating the ca- pability of llms in identifying compilation errors in configurable systems,

    L. Albuquerque, R. Gheyi, and M. Ribeiro, “Evaluating the ca- pability of llms in identifying compilation errors in configurable systems,” in Brazilian Symposium on Software Engineering, 2024, pp. 574–580

    2024

  29. [29]

    Metaprompting: Learning to learn better prompts,

    Y. Hou, H. Dong, X. Wang, B. Li, and W. Che, “Metaprompting: Learning to learn better prompts,” arXiv preprint arXiv:2209.11486, 2022

    Pith/arXiv arXiv 2022

  30. [30]

    Pre- train, prompt, and predict: A systematic survey of prompting meth- ods in natural language processing,

    P . Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre- train, prompt, and predict: A systematic survey of prompting meth- ods in natural language processing,” ACM Computing Surveys (CSUR), vol. 55, no. 9, pp. 1–35, 2023

    2023

  31. [31]

    Prompt Engineering Guide,

    DAIR.AI, “Prompt Engineering Guide,” https://www. promptingguide.ai/techniques, 2024

    2024

  32. [32]

    OllamaLLM,

    LangChain API, “OllamaLLM,” https://api.python.langchain. com/en/latest/ollama/llms/langchain_ollama.llms.OllamaLLM. html, 2026

    2026

  33. [33]

    Code generation with small language models: A codeforces-based study,

    D. Souza, R. Gheyi, L. Albuquerque, M. Ribeiro, and G. Soares, “Code generation with small language models: A codeforces-based study,” in International Conference on Machine Learning and Applications, 2025

    2025

  34. [34]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P . de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P . Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P . Tillet, F. P . Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. ...

    Pith/arXiv arXiv 2021

  35. [35]

    Non-determinism of

    B. Atil, S. Aykent, A. Chittams, L. Fu, R. J. Passonneau, E. Radcliffe, G. R. Rajagopal, A. Sloan, T. Tudrej, F. Ture, Z. Wu, L. Xu, and B. Baldwin, “Non-determinism of "deterministic" LLM settings,”

  36. [36]

    Chatbot arena: an open platform for evaluating LLMs by human preference,

    W.-L. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, B. Zhu, H. Zhang, M. I. Jordan, J. E. Gonzalez, and I. Stoica, “Chatbot arena: an open platform for evaluating LLMs by human preference,” in International Conference on Machine Learning, ser. ICML’24. JMLR.org, 2024

    2024

  37. [37]

    TypeChef,

    TypeChef, “TypeChef,” https://github.com/ckaestne/TypeChef, 2026

    2026

  38. [38]

    Note on the sampling error of the difference between correlated proportions or percentages,

    Q. McNemar, “Note on the sampling error of the difference between correlated proportions or percentages,” Psychometrika, vol. 12, no. 2, pp. 153–157, 1947

    1947

  39. [39]

    Language models are few-shot learners,

    T. B. Brown et al., “Language models are few-shot learners,” in Advances in Neural Information Processing Systems, 2020

    2020

  40. [40]

    A change-centric approach to compile configurable systems with #ifdefs,

    L. Braz, R. Gheyi, M. Mongiovi, M. Ribeiro, F. Medeiros, and L. Teixeira, “A change-centric approach to compile configurable systems with #ifdefs,” in Generative Programming: Concepts & Experiences, 2016, pp. 109–119

    2016

  41. [41]

    A change-aware per-file analysis to compile con- figurable systems with #ifdefs,

    L. Braz, R. Gheyi, M. Mongiovi, M. Ribeiro, F. Medeiros, L. Teixeira, and S. Souto, “A change-aware per-file analysis to compile con- figurable systems with #ifdefs,” Computer Languages, Systems & Structures, vol. 54, pp. 427–450, 2018

    2018

  42. [42]

    The AI Code Editor,

    Cursor, “The AI Code Editor,” https://www.cursor.com, 2026

    2026

  43. [43]

    Antigravity,

    Google, “Antigravity,” https://antigravity.google, 2026

    2026

  44. [44]

    Windsurf AI IDE,

    Windsurf, “Windsurf AI IDE,” https://windsurf.com, 2026

    2026

  45. [45]

    Breaking the silence: the threats of using llms in software engineering,

    J. Sallou, T. Durieux, and A. Panichella, “Breaking the silence: the threats of using llms in software engineering,” in International Conference on Software Engineering - New Ideas and Emerging Results. ACM/IEEE, 2024

    2024

  46. [46]

    Finding and under- standing bugs in C compilers,

    X. Yang, Y. Chen, E. Eide, and J. Regehr, “Finding and under- standing bugs in C compilers,” in Programming Language Design and Implementation, M. W. Hall and D. A. Padua, Eds., 2011, pp. 283–294

    2011

  47. [47]

    An empir- ical study of real-world variability bugs detected by variability- oblivious tools,

    A. Mordahl, J. Oh, U. Koc, S. Wei, and P . Gazzillo, “An empir- ical study of real-world variability bugs detected by variability- oblivious tools,” in Foundations of Software Engineering. ACM, 2019, pp. 50–61

    2019

  48. [48]

    A qualitative analysis of variability weaknesses in configurable systems with #ifdefs,

    R. Muniz, L. Braz, R. Gheyi, W. Andrade, B. Fonseca, and M. Ribeiro, “A qualitative analysis of variability weaknesses in configurable systems with #ifdefs,” in Variability Modelling of Software-Intensive Systems, 2018, pp. 51–58

    2018

  49. [49]

    An empirical study on configuration-related code weaknesses,

    F. Medeiros, M. Ribeiro, R. Gheyi, L. Braz, C. Kästner, S. Apel, and K. Santos, “An empirical study on configuration-related code weaknesses,” in Brazilian Symposium on Software Engineering. ACM, 2020, pp. 193–202

    2020

  50. [50]

    Automated program repair,

    C. Le Goues, M. Pradel, and A. Roychoudhury, “Automated program repair,” Communications of the ACM, vol. 62, no. 12, p. 56–65, Nov. 2019

    2019

  51. [51]

    Automatic software repair: A bibliography,

    M. Monperrus, “Automatic software repair: A bibliography,” ACM Computing Surveys, vol. 51, no. 1, Jan. 2018

    2018

  52. [52]

    RustAssistant: Using llms to fix compilation errors in rust code,

    P . Deligiannis, A. Lal, N. Mehrotra, R. Poddar, and A. Rastogi, “RustAssistant: Using llms to fix compilation errors in rust code,” in International Conference on Software Engineering, 2025, p. 3097–3109

    2025

  53. [2025]

    Available: https://arxiv.org/abs/2408.04667

    [Online]. Available: https://arxiv.org/abs/2408.04667

    Pith/arXiv arXiv