Pith. sign in

REVIEW 3 major objections 5 minor 59 references

Doc2CI: A Multi-Service Study of CI Configuration Generation Using Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLMs rarely reproduce CI configurations, and similarity scores hide invalid YAML.

desk verdict A substantial, methodologically careful multi-service benchmark and evaluation whose headline numbers are partly artifacts of fragment-heavy documentation references; worth sending to referees with a request for a sensitivity analysis. read the letter →

arxiv 2608.01451 v1 pith:PGBBREHH submitted 2026-08-02 cs.SE

Taher A. Ghaleb This is my paper · ORCID
classification cs.SE
keywords continuousintegrationCIconfigurationgenerationlargelanguagemodelsYAMLvalidationschemavalidityDOC2CIbenchmarkfailuretaxonomycodespecialization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that evaluating LLM-generated CI configurations by how closely they match a reference is misleading: the binding constraint is structural validity, not similarity. Across 16 models and 53,808 outputs on a new four-service benchmark, exact reference reproduction never exceeds 3.1%, 97.1% of outputs parse as YAML, yet only 70.8% satisfy the service schema. The gap is driven by output completeness: some models emit the minimal documented fragment while others emit a full workflow, and both can be defensible. A failure taxonomy shows most errors are semantic (misinterpretation and omission) rather than syntactic, and a training-free schema-guided repair pass raises schema validity to about 94%.

What carries the argument

The central object is DOC2CI, a benchmark pairing natural-language task descriptions with reference YAML snippets curated from official documentation of GitHub Actions, CircleCI, GitLab CI, and Travis CI. The evaluation machinery is the separation of similarity (eight lexical, semantic, and structural metrics) from validity (YAML parsing plus per-service JSON-schema conformance), supplemented by open coding of 385 outputs into a failure taxonomy. The load-bearing distinction is output completeness: matching a minimal fragment reference versus producing a complete workflow, which the paper shows drives cross-model disagreement and explains why fine-tuning on documentation improves similarity

What would settle it

Take the 3,363 DOC2CI references and run each through its own service schema; if a large fraction of references fail schema validation or are non-workflow files, then the 70.8% schema-validity figure is inflated as a measure of model capability and the fine-tuning validity loss is partly a benchmark artifact. A second check: re-run the 16 models on a filtered DOC2CI containing only complete, schema-valid, executable workflow references; if exact match remains at or below 3.1% and schema validity stays near 71%, the central claim survives, whereas a large jump in schema validity would show the

Watch

Extended reading notes

Core claim

The paper's central claim is that similarity and validity are decoupled objectives for LLM-generated CI configuration, and that validity is the harder constraint. The evidence: DOC2CI, 3,363 description-to-YAML pairs from four CI services; 14 open-weight and 2 proprietary models produce 53,808 configurations; mean cosine similarity for the best models sits near 0.64-0.67 while schema validity averages 70.8% and exact match is at most 3.1%. GPT-4o on GitHub Actions illustrates the decoupling: it has the highest similarity yet only 17.2% schema-valid output because it literally generates minimal fragments. Manual coding of 385 configurations puts only 41.3% as usable as-is, with misinterpretat

Load-bearing premise

The load-bearing premise is that documentation-derived snippets are a fair reference for scoring generated CI configurations; the paper itself finds 13.4% of coded issues are reference-quality artifacts (fragments, non-workflow files, wrong-language snippets), so if those references dominate, the headline numbers partly measure output granularity rather than LLM capability.

Editorial extensions

If this is right

  • Similarity-only evaluation overstates how close LLMs are to usable CI automation; schema checks are cheap and should accompany any CI-generation benchmark.
  • Scale helps structural validity, but code-specialized models give no consistent advantage over general models, so model selection should weigh scale and output discipline rather than the code-versus-general label.
  • Target service matters: CircleCI is substantially harder (56.3% schema-valid) than Travis CI (83.0%), so single-service evaluations can mislead about general performance.
  • Fine-tuning on documentation-derived fragments improves resemblance to references but degrades standalone validity; schema-guided repair is the more reliable validity lever and can be combined with fine-tuning for the best balance.
  • Most inter-model disagreements reflect output-completeness differences rather than genuine capability gaps; ranking models on fragment-heavy references partly rewards generating small snippets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: re-run the same 16 models after filtering DOC2CI to complete, schema-valid workflow references; the paper's own reference-quality statistics imply schema-validity rates would rise and the fine-tuning validity drop would shrink, but the paper does not run this.
  • The monotone repair result suggests a practical floor for CI tooling: feeding validator error messages back to the model before commit can recover roughly two-thirds of invalid outputs with zero regressions; the paper demonstrates this on two models, not as a product.
  • Because completeness mismatch dominates disagreement, similarity rankings on documentation-derived benchmarks are partly an artifact of output granularity; downstream users should compare models by validity and usable rate, and the paper's usable-rate data by model tier is the more actionable signal.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces DOC2CI, a benchmark of 3,363 description-to-YAML pairs from the official documentation of GitHub Actions, CircleCI, GitLab CI, and Travis CI, and evaluates 16 LLMs (14 open-weight models plus GPT-4o and GPT-4.1) generating 53,808 configurations. The study measures reference alignment via eight similarity metrics and validity via YAML parsing and service-schema conformance, complemented by open coding of 385 configurations and an analysis of 86 high-dispersion model disagreements. The main findings are that exact reference reproduction rarely exceeds 3.1%, that parse validity is high (97.1%) while schema validity is only 70.8%, that larger models improve structural validity but code specialization gives no consistent advantage, that failures are dominated by misinterpretation and omission, that model disagreement mostly reflects output completeness rather than capability, and that schema-guided repair raises validity while fine-tuning raises similarity at a substantial validity cost. The paper argues that similarity and validity are decoupled and that schema-aware evaluation is essential for CI configuration generation.

Significance. If the quantitative conclusions hold, the paper is a valuable contribution to empirical software engineering and LLM evaluation. Its strengths are considerable: a multi-service benchmark built from official documentation, a large and reproducible evaluation setup, paired non-parametric statistical tests with Holm correction, bootstrap confidence intervals, effect sizes, and a manual coding protocol with substantial inter-rater reliability (kappa 0.78-0.81). The combination of similarity, schema validity, and a grounded failure taxonomy is a principled advance over single-metric evaluations. The paper also releases a replication package, which supports verification and reuse. The main caveat is that the benchmark's ground truth is partly composed of minimal documentation fragments and non-workflow artifacts, and the paper's own analyses show that this design choice materially affects several headline numbers. A revised version that quantifies this effect and demonstrates that the central claims survive on complete, standalone references would significantly strengthen the contribution.

major comments (3)
  1. [Sections III-A, IV-A, and IV-F] The central claim that similarity and validity are decoupled and that validity is the binding constraint rests on DOC2CI references being an appropriate ground truth. The manuscript itself reports that "many references are minimal fragments" and that some are non-workflow artifacts (Section III-A), that reference-quality issues account for 13.4% of coded issues (Table IV), and that 79% of high-dispersion disagreements are driven by completeness mismatch or reference quality (Section IV-E, Table V). Yet no count is given for how many of the 3,363 pairs have fragment or non-workflow references, and no RQ1 or RQ6 analysis is restricted to complete, standalone references. Because exact match is measured against these references and schema validation requires a standalone file, the headline figures (3.1% exact match, 70.8% schema validity, and the RQ6 validity drop) are partly determined by r
  2. [Section IV-A, RQ1] The GPT-4o/GitHub Actions case is presented as the key evidence for the similarity-validity decoupling: highest similarity, 17.2% schema validity, and 80% fragment outputs. However, the same paragraph reports that on the other three services GPT-4o produces near-0% fragments and 84-92% schema validity. The decoupling in this example is therefore a within-service artifact of generating fragments that match fragment references, not a general capability pattern across services. The recommendation that schema validation is essential is sound, but the illustrative example should be framed as a benchmark-granularity effect, and the text should avoid implying that GPT-4o cannot produce valid GitHub Actions workflows.
  3. [Section IV-F, Table VI] The RQ6 result that fine-tuning improves similarity and lowers schema validity is, as the text acknowledges, a direct consequence of training targets being documentation fragments. This makes the conclusion that "similarity and validity are distinct objectives" partly circular: on any fragment-heavy training set the two objectives are negatively coupled by construction because matching a fragment reference is incompatible with producing a standalone schema-valid file. To support the general claim, the authors need either a control condition in which fine-tuning targets complete standalone configurations, or a per-granularity breakdown (fragment vs complete references) for the fine-tuned vs zero-shot comparison. Without such an analysis, the RQ6 finding is better described as a consequence of benchmark composition than as independent evidence of decoupling.
minor comments (5)
  1. [Table I] The table contains rendering artifacts in the "Type" column (e.g., "/commen◎-do◎s") and uses "Parse%" in a way that is easily confused with the schema-validity column in Table II. Please clean the artifact characters and clarify the column label.
  2. [Section IV-A] The text reports "the two GPT models vs. the 14 open models, Cliff's delta = 0.18 per-configuration" without a p-value or confidence interval. Given the emphasis on effect sizes, please report the uncertainty or at least state the significance threshold.
  3. [Section IV-C] The notation "Holm = 0.94" is unclear; it likely denotes a Holm-corrected p-value or an effect size for the GitHub Actions vs Travis CI contrast. Please make the statistic explicit.
  4. [Section IV-E] In the high-dispersion sampling description, "median cosine range of = 0.79" contains a typographical error (stray "="). Also, the high-dispersion thresholds (best cosine >= 0.72, worst <= 0.35, at least 14 models) are arbitrary; a brief justification or sensitivity check would be helpful.
  5. [Section V-A] The practical recommendations for CI engineers contain a duplicated sentence: "prefer a larger model, weighting output discipline over the code-versus-general label" appears almost verbatim in two consecutive sentences. Remove the repetition.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: claims are empirical measurements; the only minor self-citation ([13]) is used for replication consistency and is not load-bearing. The acknowledged fragment-heavy reference construction is a benchmark-validity threat, not a circular derivation.

full rationale

This paper does not contain a derivation chain: it reports measurements on a constructed benchmark (DOC2CI) and manual coding. No parameter is fitted to a subset and then renamed as a prediction; no uniqueness theorem or ansatz from prior work is invoked as a load-bearing premise; no known result is repackaged under new coordinates. The central claim that similarity and validity are distinct is supported by the near-zero correlation across models (Fig. 3) and by the within-service GPT-4o/GPT-4.1 contrast, which are independent of the specific benchmark references. The paper itself flags the main limitation in Section III-A ('many references are minimal fragments...') and in Section V-B1 ('valid fragments matching fragment references may still be marked schema-invalid when a service schema requires a complete file'), acknowledging that some headline numbers (e.g., 3.1% exact match, 70.8% schema validity, and the fine-tuning validity drop) are partly affected by reference granularity. This is a measurement-design caveat rather than a circular step, because the conclusions do not reduce to the definitions: the exact-match rate is not forced by the fragment references (fragments are easier to match), the schema-validity rate is an external service-schema check, and the fine-tuning drop is a consequence of the training target distribution, which the paper explicitly explains. The self-citation [13] is used only as a replication baseline for GPT-4o on GitHub Actions and for model selection; it does not carry the argument. Score 2 reflects the minor, non-load-bearing self-citation and the acknowledged benchmark-construction caveat, not a circular derivation.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

No fitted parameters or invented physical entities; the ledger lists analysis choices, thresholds, and domain assumptions that shape the quantitative findings. The most consequential is the documentation-as-ground-truth assumption, which the paper itself partially undermines through the reference-quality category.

free parameters (5)
  • High-dispersion inclusion thresholds (best cosine >= 0.72, worst <= 0.35, at least 14 models) = 0.72 / 0.35 / 14
    Chosen by hand to select sharp-split cases for RQ5; the resulting category distribution (59% completeness mismatch) is frame-dependent.
  • Stratified sample sizes (385 RQ4, 77 reliability, 86 RQ5) = 385, 77, 86
    Chosen to meet stated 95% confidence with +/-5% or +/-10% margins; taxonomy frequencies inherit sampling uncertainty.
  • LoRA fine-tuning settings (rank 16, three epochs, cosine schedule, 4-bit quantized) = rank=16, epochs=3
    Fixed without hyperparameter sweep; the paper notes the magnitude of the validity drop could shift with training budget.
  • Model size tier boundaries (small 7-8B, medium 9-15B, large >15B) = 7-8 / 9-15 / >15
    Follows available checkpoints; affects RQ2 scale analysis and the tier-level usable rate findings.
  • Exact-match threshold (canonicalized Levenshtein similarity = 1) = 1.0
    Allows trivial formatting differences but depends on the canonicalization choice.
assumptions (5)
  • domain assumption Service JSON schemas are a valid proxy for service acceptance.
    Used for the strict validity check (Section III-D); the paper acknowledges schemas can lag features, making estimates conservative (Section V-B1).
  • domain assumption Documentation sections and example YAML are a valid proxy for developer intent and correct configurations.
    DOC2CI is built from official docs (Section III-A); the paper's own RQ4/RQ5 find 13.4% reference-quality issues, so this assumption is partially violated.
  • domain assumption all-MiniLM-L6-v2 cosine similarity captures semantic agreement for CI YAML.
    Primary semantic metric used in most analyses (Section III-D); the paper verifies ranking consistency with seven other metrics, which supports but does not fully validate the metric.
  • domain assumption Zero-shot prompting with a single standardized prompt is a fair comparison for all 16 models.
    Used for all generations (Section III-C); per-model prompt tuning could change rankings, as noted in the threats to validity.
  • domain assumption 4-bit quantization and greedy decoding do not change relative model quality.
    Open models run through Ollama with default 4-bit quantization and temperature 0 (Section III-B); quantization could compress instruction-following differences.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Doc2CI: A Multi-Service Study of CI Configuration Generation Using Large Language Models." pith.science (2026). https://pith.science/paper/PGBBREHH

@misc{pith2026260801451,
  author       = {Pith},
  title        = {Pith review of: Doc2CI: A Multi-Service Study of CI Configuration Generation Using Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PGBBREHH}},
  note         = {Machine review of arXiv:2608.01451}
}
read the original abstract

Adopting Continuous Integration (CI) often requires writing YAML configurations that are error-prone and challenging to maintain. Despite increasing LLM use in software engineering, their ability to generate CI configurations from natural language across services and model families remains unclear. This paper presents a large empirical study on using LLMs to generate CI configurations. We introduce DOC2CI, a benchmark of 3,363 description-to-YAML pairs collected from the official documentation of four CI services, and evaluate 14 open-weight models from 7B-34B parameters together with GPT-4o and GPT-4.1, producing over 53,000 configurations. We assess both reference alignment and schema validity to determine whether the generated configurations are structurally valid. We further develop a failure taxonomy from a manual analysis of 385 configurations and examine why LLMs disagree. Across models and services, exact reference reproduction never exceeds 3.1%, and while 97% of outputs parse as YAML, only 71% satisfy service schemas. Larger models improve structural validity, but code specialization provides no consistent advantage over comparable general models. Model differences are driven largely by output completeness: for the same request, some models generate the expected fragment while others produce a full workflow. Finally, a training-free schema-guided repair method improves schema validity to 94%, while fine-tuning improves similarity to documentation but reduces standalone validity. This suggests that similarity and validity are distinct objectives for CI generation and motivate schema-aware evaluation and tooling for LLM-based configuration generation.

Figures

Figures reproduced from arXiv: 2608.01451 by the authors.

Figure 1
Figure 1. Overview of our study. Cylinders denote datasets, boxes denote processing steps, and stacked shapes group research questions. We curate description [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Example of a DOC2CI entry from the GitHub Actions documentation. The section title and explanatory prose form the description, while each YAML block is a reference configuration. Here, one description maps to two valid references: one inherits all secrets, and one forwards a single secret. B. Selected Models We retain the six models evaluated in the prior single￾service study [13]: two proprietary models (GPT-4o [25… view at source ↗
Figure 3
Figure 3. Similarity is not validity: each model’s mean cosine (x) versus schema [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Quality by size tier for code-specialized and general open models (tier means): similarity (left) and schema validity (right). The two types converge [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: RQ6 interventions in the similarity-validity plane. Fine [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 43 canonical work pages

  1. [1]

    Continuous Integration,

    M. Fowler, “Continuous Integration,” https://martinfowler.com/articles/ originalContinuousIntegration.html, accessed May 2025

    work page 2025

  2. [2]

    Developers’ perception of GitHub Actions: A survey analysis,

    S. G. Saroar and M. Nayebi, “Developers’ perception of GitHub Actions: A survey analysis,” inProceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering. ACM, 2023, pp. 121–130

    work page 2023

  3. [3]

    Workflow syntax for GitHub Actions,

    “Workflow syntax for GitHub Actions,” https://docs.github.com/en/ actions/writing-workflows, Accessed May 2025

    work page 2025

  4. [4]

    An empirical study of the long duration of continuous integration builds,

    T. A. Ghaleb, D. A. Da Costa, and Y . Zou, “An empirical study of the long duration of continuous integration builds,”Empirical Software Engineering, vol. 24, no. 4, pp. 2102–2139, 2019

    work page 2019

  5. [5]

    Studying the interplay between the durations and breakages of continuous integration builds,

    T. A. Ghaleb, S. Hassan, and Y . Zou, “Studying the interplay between the durations and breakages of continuous integration builds,”IEEE Transactions on Software Engineering, vol. 49, no. 4, pp. 2476–2497, 2022

    work page 2022

  6. [6]

    Automated code generation for information technology tasks in Y AML through large language models,

    S. Pujar, L. Buratti, X. Guo, N. Dupuis, B. Lewis, S. Suneja, A. Sood, G. Nalawade, M. Jones, A. Morariet al., “Automated code generation for information technology tasks in Y AML through large language models,” inProceedings of the 60th ACM/IEEE Design Automation Conference. IEEE, 2023, pp. 1–4

    work page 2023

  7. [7]

    Usage, costs, and benefits of continuous integration in open-source projects,

    M. Hilton, T. Tunnell, K. Huang, D. Marinov, and D. Dig, “Usage, costs, and benefits of continuous integration in open-source projects,” inProceedings of the 31st IEEE/ACM international conference on automated software engineering, 2016, pp. 426–437

    work page 2016

  8. [8]

    A conceptual replication of continuous integration pain points in the context of Travis CI,

    D. G. Widder, M. Hilton, C. K ¨astner, and B. Vasilescu, “A conceptual replication of continuous integration pain points in the context of Travis CI,” inProceedings of the 2019 27th acm joint meeting on european software engineering conference and symposium on the foundations of software engineering, 2019, pp. 647–658

    work page 2019

Show all 59 references
  1. [9]

    An empirical characterization of bad practices in con- tinuous integration,

    F. Zampetti, C. Vassallo, S. Panichella, G. Canfora, H. Gall, and M. Di Penta, “An empirical characterization of bad practices in con- tinuous integration,”Empirical Software Engineering, vol. 25, no. 2, pp. 1095–1135, 2020

    2020

  2. [10]

    Configuration smells in continuous delivery pipelines: a linter and a six- month study on GitLab,

    C. Vassallo, S. Proksch, A. Jancso, H. C. Gall, and M. Di Penta, “Configuration smells in continuous delivery pipelines: a linter and a six- month study on GitLab,” inProceedings of the 28th ACM joint meeting on european software engineering conference and symposium on the fou...

    2020

  3. [11]

    Toward automatically completing GitHub workflows,

    A. Mastropaolo, F. Zampetti, G. Bavota, and M. Di Penta, “Toward automatically completing GitHub workflows,” inProceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1–12

    2024

  4. [12]

    On the effectiveness of large language models for GitHub workflows,

    X. Zhang, S. Muralee, S. Cherupattamoolayil, and A. Machiry, “On the effectiveness of large language models for GitHub workflows,” inProceedings of the 19th International Conference on Availability, Reliability and Security, 2024, pp. 1–14

    2024

  5. [13]

    Can LLMs write CI? a study on automatic generation of GitHub Actions configurations,

    T. A. Ghaleb and D. Rathnayake, “Can LLMs write CI? a study on automatic generation of GitHub Actions configurations,” inProceedings of the IEEE International Conference on Software Maintenance and Evolution, 2025

    2025

  6. [14]

    Large lan- guage models are zero-shot reasoners,

    T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa, “Large lan- guage models are zero-shot reasoners,”Advances in neural information processing systems, vol. 35, pp. 22 199–22 213, 2022

    2022

  7. [15]

    Strauss and J

    A. Strauss and J. Corbin,Basics of Qualitative Research: Grounded Theory Procedures and Techniques. Sage Publications, 1990

    1990

  8. [16]

    Automatically generating dockerfiles via deep learning: Challenges and promises,

    G. Rosa, A. Mastropaolo, S. Scalabrino, G. Bavota, and R. Oliveto, “Automatically generating dockerfiles via deep learning: Challenges and promises,” inProceedings of the IEEE/ACM International Conference on Software and System Processes. IEEE, 2023, pp. 1–12

    2023

  9. [17]

    Automated DevOps pipeline generation for code repositories using large language models,

    D. Mehta, K. Rawool, S. Gujar, and B. Xu, “Automated DevOps pipeline generation for code repositories using large language models,”arXiv preprint arXiv:2312.13225, 2023

    2023 arXiv

  10. [18]

    BLEU: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: a method for automatic evaluation of machine translation,” inProceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311–318

    2002

  11. [19]

    CrystalBLEU: precisely and efficiently measuring the similarity of code,

    A. Eghbali and M. Pradel, “CrystalBLEU: precisely and efficiently measuring the similarity of code,” inProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022, pp. 1–12

    2022

  12. [20]

    ROUGE: A package for automatic evaluation of summaries,

    C.-Y . Lin, “ROUGE: A package for automatic evaluation of summaries,” inText summarization branches out, 2004, pp. 74–81

    2004

  13. [21]

    chrF: character n-gram F-score for automatic MT eval- uation,

    M. Popovi ´c, “chrF: character n-gram F-score for automatic MT eval- uation,” inProceedings of the tenth workshop on statistical machine translation, 2015, pp. 392–395

    2015

  14. [22]

    Sentence-BERT: Sentence embeddings using siamese bert-networks,

    N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using siamese bert-networks,” inProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing, 2019, pp. 3982–3992

    2019

  15. [23]

    Simple fast algorithms for the editing distance between trees and related problems,

    K. Zhang and D. Shasha, “Simple fast algorithms for the editing distance between trees and related problems,”SIAM Journal on Computing, vol. 18, no. 6, pp. 1245–1262, 1989

    1989

  16. [24]

    Doc2CI: A multi-service study of CI configuration generation using large language models (replication package),

    T. A. Ghaleb, “Doc2CI: A multi-service study of CI configuration generation using large language models (replication package),” https: //figshare.com/s/c52fa9b2e6471e61eb40, 2026

    2026

  17. [25]

    GPT-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkatet al., “GPT-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  18. [26]

    Introducing GPT-4.1 in the API,

    OpenAI, “Introducing GPT-4.1 in the API,” https://openai.com/index/gpt-4-1, 2025, accessed June 2025

    2025

  19. [27]

    LLaMA: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “LLaMA: Open and efficient foundation language models,”arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  20. [28]

    Code llama: Open foundation models for code,

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remezet al., “Code llama: Open foundation models for code,”arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  21. [29]

    Gemma 3 technical report,

    G. Team, A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ram ´e, M. Rivi `ereet al., “Gemma 3 technical report,”arXiv preprint arXiv:2503.19786, 2025

    2025 arXiv

  22. [30]

    Codegemma: Open code models based on gemma,

    H. Zhao, J. Hui, J. Howland, N. Nguyen, S. Zuo, A. Hu, C. A. Choquette-Choo, J. Shen, J. Kelley, K. Bansalet al., “Codegemma: Open code models based on gemma,”arXiv preprint arXiv:2406.11409, 2024

    2024 arXiv

  23. [31]

    MTEB: Massive text embedding benchmark,

    N. Muennighoff, N. Tazi, L. Magne, and N. Reimers, “MTEB: Massive text embedding benchmark,” inProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, 2023, pp. 2014–2037

    2023

  24. [32]

    A little help goes a long way: Tutoring LLMs in solving competitive programming through hints,

    Y . Zhang, W. Dong, J. Liu, S. Wang, D. Wang, T. Ma, Y . Li, and K. Yang, “A little help goes a long way: Tutoring LLMs in solving competitive programming through hints,”IEEE Transactions on Software Engineer- ing, 2025

    2025

  25. [33]

    An empirical study on the code refactoring capability of large language models,

    J. Cordeiro, S. Noei, and Y . Zou, “An empirical study on the code refactoring capability of large language models,”ACM Transactions on Software Engineering and Methodology, 2024

    2024

  26. [34]

    GitHub Actions JSON Schema,

    SchemaStore, “GitHub Actions JSON Schema,” https://json. schemastore.org/github-workflow.json, accessed June 2026

    2026

  27. [35]

    Travis CI JSON Schema,

    ——, “Travis CI JSON Schema,” https://json.schemastore.org/travis. json, accessed June 2026

    2026

  28. [36]

    circleci-yaml-language-server: JSON schema for CircleCI configura- tion,

    “circleci-yaml-language-server: JSON schema for CircleCI configura- tion,” https://github.com/CircleCI-Public/circleci-yaml-language-server, accessed June 2026

    2026

  29. [37]

    GitLab CI/CD configuration JSON schema,

    “GitLab CI/CD configuration JSON schema,” https://gitlab.com/ gitlab-org/gitlab/-/blob/master/app/assets/javascripts/editor/schema/ci. json, accessed June 2026

    2026

  30. [38]

    actionlint: Static checker for GitHub Actions workflow files,

    “actionlint: Static checker for GitHub Actions workflow files,” https: //github.com/rhysd/actionlint, 2024

    2024

  31. [39]

    How to validate your CircleCI config- uration,

    Fernando, “How to validate your CircleCI config- uration,” https://support.circleci.com/hc/en-us/articles/ 360006735753-How-to-validate-your-CircleCI-configuration, accessed June 2026

    2026

  32. [40]

    CI lint: Validate GitLab CI/CD configuration,

    “CI lint: Validate GitLab CI/CD configuration,” https://docs.gitlab.com/ ci/yaml/lint, accessed June 2026

    2026

  33. [41]

    travis-yml: Travis CI configuration processing and validation,

    “travis-yml: Travis CI configuration processing and validation,” https: //github.com/travis-ci/travis-yml, accessed June 2026

    2026

  34. [42]

    A coefficient of agreement for nominal scales,

    J. Cohen, “A coefficient of agreement for nominal scales,”Educational and psychological measurement, vol. 20, no. 1, pp. 37–46, 1960

    1960

  35. [43]

    The measurement of observer agreement for categorical data,

    J. R. Landis and G. G. Koch, “The measurement of observer agreement for categorical data,”Biometrics, pp. 159–174, 1977

    1977

  36. [44]

    The use of ranks to avoid the assumption of normality implicit in the analysis of variance,

    M. Friedman, “The use of ranks to avoid the assumption of normality implicit in the analysis of variance,”Journal of the american statistical association, vol. 32, no. 200, pp. 675–701, 1937

    1937

  37. [45]

    Statistical comparisons of classifiers over multiple data sets,

    J. Dem ˇsar, “Statistical comparisons of classifiers over multiple data sets,” Journal of Machine Learning Research, vol. 7, no. Jan, pp. 1–30, 2006

    2006

  38. [46]

    A practical guide for using statistical tests to assess randomized algorithms in software engineering,

    A. Arcuri and L. Briand, “A practical guide for using statistical tests to assess randomized algorithms in software engineering,” inProceedings of the 33rd international conference on software engineering, 2011, pp. 1–10

    2011

  39. [47]

    Individual comparisons by ranking methods,

    F. Wilcoxon, “Individual comparisons by ranking methods,”Biometrics bulletin, vol. 1, no. 6, pp. 80–83, 1945

    1945

  40. [48]

    On a test of whether one of two random variables is stochastically larger than the other,

    H. B. Mann and D. R. Whitney, “On a test of whether one of two random variables is stochastically larger than the other,”The Annals of Mathematical Statistics, pp. 50–60, 1947

    1947

  41. [49]

    A simple sequentially rejective multiple test procedure,

    S. Holm, “A simple sequentially rejective multiple test procedure,” Scandinavian journal of statistics, pp. 65–70, 1979

    1979

  42. [50]

    The proof and measurement of association between two things,

    C. Spearman, “The proof and measurement of association between two things,”The American journal of psychology, vol. 100, no. 3/4, pp. 441– 471, 1987

    1987

  43. [51]

    Dominance statistics: Ordinal analyses to answer ordinal questions

    N. Cliff, “Dominance statistics: Ordinal analyses to answer ordinal questions.”Psychological bulletin, vol. 114, no. 3, p. 494, 1993

    1993

  44. [52]

    Appropriate statistics for ordinal level data: Should we really be using t-test and Cohen’s d for evaluating group differences on the nsse and other surveys,

    J. Romano, J. D. Kromrey, J. Coraggio, and J. Skowronek, “Appropriate statistics for ordinal level data: Should we really be using t-test and Cohen’s d for evaluating group differences on the nsse and other surveys,” inAnnual meeting of the Florida Association of Institutional...

    2006

  45. [53]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, W. Chenet al., “LoRA: Low-rank adaptation of large language models,” inProceedings of the Tenth International Conference on Learning Representations, 2022

    2022

  46. [54]

    QLoRA: Efficient finetuning of quantized LLMs,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “QLoRA: Efficient finetuning of quantized LLMs,”Advances in neural information processing systems, vol. 36, pp. 10 088–10 115, 2023

    2023

  47. [55]

    Note on the sampling error of the difference between correlated proportions or percentages,

    Q. McNemar, “Note on the sampling error of the difference between correlated proportions or percentages,”Psychometrika, vol. 12, no. 2, pp. 153–157, 1947

    1947

  48. [56]

    Retrieval- augmented generation for knowledge-intensive NLP tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt ¨aschelet al., “Retrieval- augmented generation for knowledge-intensive NLP tasks,”Advances in neural information processing systems, vol. 33, pp. 9459–9474, 2020

    2020

  49. [57]

    Studying the impact of noises in build breakage data,

    T. A. Ghaleb, D. A. Da Costa, Y . Zou, and A. E. Hassan, “Studying the impact of noises in build breakage data,”IEEE Transactions on Software Engineering, vol. 47, no. 9, pp. 1998–2011, 2019

    1998

  50. [58]

    CI/CD configuration practices in open-source Android apps: An empirical study,

    T. Ghaleb, O. Abduljalil, and S. Hassan, “CI/CD configuration practices in open-source Android apps: An empirical study,”ACM Transactions on Software Engineering and Methodology, 2025

    2025

  51. [59]

    On the usage, co-usage and migration of CI/CD tools: A qualitative analysis,

    P. Rostami Mazrae, T. Mens, M. Golzadeh, and A. Decan, “On the usage, co-usage and migration of CI/CD tools: A qualitative analysis,” Empirical Software Engineering, vol. 28, no. 2, p. 52, 2023

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.