Pith. sign in

REVIEW 3 major objections 5 minor 70 references

Explaining GitHub Actions Failures with Large Language Models: Challenges, Insights, and Limitations

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A mixed-methods study of 31 developers finds that LLM-generated explanations of GitHub Actions failures are accepted as correct, clear, concise, and actionable for simpler logs, supporting LLM assistance for common CI/CD errors while…

desk verdict The abstract's 'simpler/small logs' claim is not supported by the RQ1 analysis, but the paper is a legitimate first feasibility study with an honest limitations section. read the letter →

arxiv 2501.16495 v1 pith:OWL4QTJN submitted 2025-01-27 cs.SE

Pablo Valenzuela-Toledo , Chuyue Wu , Sandro Hernandez , Alexander Boll , Roman Machacek , Sebastiano Panichella , Timo Kehrer This is my paper · ORCID
classification cs.SE
keywords GitHubActionsCI/CDlargelanguagemodelsfailureexplanationdevelopersurveyloganalysispromptengineeringfeasibilitystudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports a feasibility study asking whether large language models can explain GitHub Actions workflow failures well enough for developers to act on. Thirty-one developers who maintain and troubleshoot GitHub Actions workflows rated LLM-generated explanations of ten real failure logs on correctness, conciseness, clarity, and actionability. More than 80% of ratings agreed that the explanations were correct and clear for smaller or simpler logs, and the authors conclude that LLMs can feasibly assist with common CI/CD failures. For longer, more complex logs, ratings were more variable and the explanations were seen as less reliable, so the paper argues that improving LLM reasoning and preprocessing logs are the next steps.

What carries the argument

The evaluation machinery is a custom web survey tool that shows a GitHub Actions failure log beside a statically generated LLM explanation and asks developers to rate it on four constructs—correctness, conciseness, clarity, and actionability—using Likert items and open questions. The prompt machinery that carries the result is one-shot prompting: a single worked example of a failure explanation given to the model before the target log. The log-selection pipeline filters JavaScript repositories, applies a 45-word length threshold, and yields ten representative failure cases; the qualitative analysis uses card sorting with reported inter-rater agreement to derive the five actionability attributes.

What would settle it

Take a labeled set of GitHub Actions failure logs with independently confirmed root causes, run the same one-shot LLM explanation pipeline, and measure whether the explanations match the confirmed causes. If fewer than 80% of explanations for shorter logs are objectively correct—or if simple-log accuracy is not clearly higher than complex-log accuracy—the paper's central feasibility claim would not survive as a statement about actual correctness.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLM-generated contextual summaries of GitHub Actions run failures are perceived as largely correct, clear, concise, and actionable for common, simpler failures, with over 80% positive agreement from the surveyed developers, and that this makes LLM-based explanation a feasible support tool for CI/CD troubleshooting. The same evidence shows the limits: when logs are verbose, unstructured, or involve intricate failure sequences, the explanations lose accuracy and consistency, and developers with more experience prefer shorter output. The authors frame this as a feasibility result about perception, not a benchmark of optimal model choice; they identify one-shot prompting as the most effective prompt strategy in their pilot and describe five attributes developers value in actionable explanations: clarity, actionable guidance, specificity, contextual relevance, and conciseness.

Load-bearing premise

The central claim rests on 31 volunteer developers' Likert-scale agreement standing in for actual correctness and usefulness; there is no objective verification that the LLM explanations match the true root causes of the failures.

Editorial extensions

If this is right

  • CI/CD tools could embed LLM-generated explanations for common, simpler failures, reducing the time developers spend reading long logs by hand.
  • Long or unstructured logs need preprocessing that filters out irrelevant entries before the LLM is asked to explain the failure.
  • Explanation depth should adapt to the developer's experience, giving seasoned developers concise summaries and giving newer developers more contextual guidance.
  • Future research on LLM diagnostics should focus on reasoning over complex, interwoven failure sequences rather than on basic summarization of simple errors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A production tool built on this result would need a confidence gate: when an explanation is likely to be unreliable, the tool should defer to a human rather than present the guess as a diagnosis.
  • The 80% figure is a measure of perceived correctness, so an objective benchmark with labeled root causes could produce a different number; the gap between perception and ground truth is untested here.
  • Generalization beyond JavaScript projects, open-weights models, and the ten selected failure cases is untested, and other ecosystems or proprietary models may behave differently.
  • Personalizing explanation depth by expertise could improve satisfaction, but it may also hide critical detail from novices precisely when they need it most.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports a mixed-methods feasibility study in which 31 developers (purposively sampled from JavaScript projects on GitHub, 3.82% response rate) rated LLM-generated explanations of ten GitHub Actions failure logs on correctness, conciseness, clarity, and actionability. The authors claim that over 80% of developers rated the explanations as correct and clear for simpler or smaller logs, that one-shot prompting with Llama3 was the most effective configuration in a pilot, and that qualitative analysis of open-ended answers identifies five attributes of actionable explanations. The paper includes a replication package with raw data and survey materials, and it explicitly discusses threats to validity including the non-representative sample and the probabilistic nature of LLM outputs.

Significance. If the claims are adequately supported, the study provides timely exploratory evidence on a practically important question: whether LLM-generated explanations can help developers diagnose CI/CD failures. The mixed-methods design, the use of a custom evaluation tool (LogExp), the pilot-based prompt selection, the Cohen's kappa inter-rater agreement (0.74) for the qualitative analysis, and the publicly promised replication package are concrete strengths. However, the headline quantitative claim ('over 80% for simpler/small logs') is not directly supported by the reported aggregate data, and the inferential basis is thin given the small, purposive sample. The paper is therefore a useful exploratory contribution whose central claim needs to be re-analyzed and re-stated before archival publication.

major comments (3)
  1. [Abstract and Section III (Answer to RQ1)] The abstract's claim that 'over 80% of developers rated LLM explanations positively in terms of correctness for simpler/small logs' is not supported by the evidence presented. Section III reports only aggregate agreement across all ten failure cases (Fig. 3): St. (1) 82.6%, St. (3) 78.2%, St. (4) 78.4%, St. (5) 83.5%, St. (6) 87.1%. Two correctness statements fall below 80%, and no table or figure stratifies these percentages by log length or complexity. The only place log length is discussed is Section VI, where the authors state that 'shorter logs received higher and more consistent ratings' without reporting the supporting numbers or a statistical test. Please add a stratified analysis (per-case and per log-length/complexity group) with confidence intervals, and re-state the abstract and RQ1 answer to match what the data actually show.
  2. [Sections II and III (quantitative analysis)] The reported percentages are item-level agreement rates (e.g., 82.6% of responses to St. (1) were 'fully agree' or 'agree'), not 'over 80% of developers' as a person-level statistic. Please clarify this wording throughout, and report the number of valid responses per item. In addition, the study provides no confidence intervals, significance tests, or effect sizes for any of the quantitative claims; this is particularly important for the Section VI comparison between CI/CD users (median 4.5, SD 0.5) and non-users (median 3.2, SD 1.2), which is reported without a test. Given the 3.82% response rate and purposive sampling acknowledged in Section VII, the conclusions need uncertainty quantification and more cautious generalization language.
  3. [Section VI (Discussion) and Abstract] The paper's central conditional claim is that LLM explanations are perceived as correct and clear specifically for 'simpler/small logs,' but the supporting analysis appears only as a qualitative remark in Section VI ('shorter logs received higher and more consistent ratings') with no table, figure, or test. Since this claim is load-bearing for the abstract and for the proposed implications (e.g., tailoring explanations to log complexity and developer experience), it must be moved into the results sections with explicit numbers, a defined complexity or length split, and appropriate statistical support. Without this, the headline result may be an artifact of pooling simple and complex cases.
minor comments (5)
  1. [Section VI (RQ1 paragraph)] The word 'concice' should be 'concise' in the sentence about structured and concise logs.
  2. [Section IV (St. 7 discussion and Answer to RQ2)] The phrase 'emphasis that the explanations were communicated clearly' should be 'emphasizing that...'; additionally, the Answer to RQ2 contains a stray comma-period ('easy to understand,.').
  3. [Section V.B] The phrase 'an specific error' should be 'a specific error'.
  4. [Section VI (RQ3 paragraph)] There is a typo with a double period in 'Sallou et al.. discuss'; it should be 'Sallou et al. discuss'.
  5. [Section II (LLM configuration) and Section VII] The paper states in Section VII that 'our study does not aim to optimize LLM performance or determine the best model or prompting strategy,' but the contributions in Section I list 'one-shot prompt tuning as the most effective approach'; please make these statements consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: this is an empirical survey whose outcome is directly measured developer perception; the unsupported 'simpler logs' claim is a reporting gap, not a circular reduction.

full rationale

This paper is an empirical feasibility study, not a derivation chain. It does not fit parameters to a subset of data and then present a closely related quantity as a prediction; the outcome variables (correctness, conciseness, clarity, actionability) are measured directly from Likert-scale and open-ended survey responses provided by external developers. The LLM-generated explanations are the intervention under study, and the ratings are independent judgments elicited from recruited participants, so the evaluation is not equivalent to the authors' own construction by definition. The paper adapts its attribute definitions from prior work, including some publications by the authors themselves, but those citations supply measurement constructs, not load-bearing results that force the study's conclusions. The most salient weakness is that the abstract's 'simpler/small logs' claim is not directly supported by the aggregate RQ1 percentages, and the RQ1 answer states 'over 80% agreement across statements' even though two correctness items fall below 80%; this is a validity and reporting concern, not circularity. Likewise, the acknowledged low response rate and purposive sampling limit generalizability but do not make any claim reduce to its own inputs. No equation equates an output to an input, no fitted value is renamed as a prediction, and no self-citation chain is invoked to forbid alternatives. The paper is therefore self-contained with respect to circularity, and the honest finding is a score of 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

This is an empirical survey study, so the ledger entries are design choices and domain assumptions rather than mathematical free parameters. The main burden on the central claim is the representativeness of the 31 respondents and the validity of perception-based Likert metrics.

free parameters (4)
  • Minimum log length threshold = 45 words
    Used to filter out logs too short for meaningful explanation; chosen empirically via knee method pilot study.
  • Number of failure cases = 10
    Selected by hand for diversity across CI, deployment, and testing; the central survey ratings depend on these specific cases.
  • Prompting technique = one-shot
    Selected from pilot comparison of zero-, one-, and few-shot prompting; the generated explanations depend on this choice.
  • LLM selection = Llama3, Llama2, Mixtral
    Chosen for customizability and availability over proprietary models; results may not generalize to other models.
assumptions (4)
  • domain assumption The four attributes (correctness, conciseness, clarity, actionability) are valid and sufficient constructs for evaluating explanation quality.
    Adopted from prior summarization literature [17], [18]; if these do not capture usefulness, the survey measures the wrong thing.
  • domain assumption Developers who responded to the survey are representative of developers who troubleshoot GitHub Actions failures.
    Purposive sampling of contributors to selected JavaScript repositories; the paper acknowledges low response rate and non-representativeness.
  • domain assumption Likert agreement scores can be aggregated into a percentage agreement metric.
    The paper sums 'agree' and 'fully agree' responses; this treats ordinal scales as interval and ignores neutral and disagree distribution.
  • ad hoc to paper The 45-word threshold leaves enough context for an actionable explanation.
    Determined by the authors' pilot knee method; different thresholds would change the log sample and potentially the ratings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Explaining GitHub Actions Failures with Large Language Models: Challenges, Insights, and Limitations." pith.science (2026). https://pith.science/paper/OWL4QTJN

@misc{pith2026250116495,
  author       = {Pith},
  title        = {Pith review of: Explaining GitHub Actions Failures with Large Language Models: Challenges, Insights, and Limitations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OWL4QTJN}},
  note         = {Machine review of arXiv:2501.16495}
}
read the original abstract

GitHub Actions (GA) has become the de facto tool that developers use to automate software workflows, seamlessly building, testing, and deploying code. Yet when GA fails, it disrupts development, causing delays and driving up costs. Diagnosing failures becomes especially challenging because error logs are often long, complex and unstructured. Given these difficulties, this study explores the potential of large language models (LLMs) to generate correct, clear, concise, and actionable contextual descriptions (or summaries) for GA failures, focusing on developers' perceptions of their feasibility and usefulness. Our results show that over 80\% of developers rated LLM explanations positively in terms of correctness for simpler/small logs. Overall, our findings suggest that LLMs can feasibly assist developers in understanding common GA errors, thus, potentially reducing manual analysis. However, we also found that improved reasoning abilities are needed to support more complex CI/CD scenarios. For instance, less experienced developers tend to be more positive on the described context, while seasoned developers prefer concise summaries. Overall, our work offers key insights for researchers enhancing LLM reasoning, particularly in adapting explanations to user expertise.

Figures

Figures reproduced from arXiv: 2501.16495 by the authors.

Figure 1
Figure 1. Survey definition: Statements 1 through 10 are closed-ended, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Partial view of the LogExp tool’s interface. The log is displayed on the left, allowing participants to choose between viewing a summary or the full log. On the right, the corresponding textual explanation generated by the LLM is presented. Below these sections, participants encountered statements and questions specific to each case. for our survey, which represented diverse failure cases from GA workflows, includin… view at source ↗
Figure 3
Figure 3. The stacked bar chart shows the levels of agreement of [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The stacked bar chart shows the levels of agreement of [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 59 canonical work pages

  1. [1]

    On the rise and fall of CI services in GitHub,

    M. Golzadeh, A. Decan, and T. Mens, “On the rise and fall of CI services in GitHub,” in 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering . IEEE, 2022, pp. 662–672

    work page 2022

  2. [2]

    Developers’ perception of github actions: A survey analysis,

    S. G. Saroar and M. Nayebi, “Developers’ perception of github actions: A survey analysis,” in Proceedings of the 27th Inter- national Conference on Evaluation and Assessment in Software Engineering, 2023, pp. 121–130

    work page 2023

  3. [3]

    How do developers talk about GitHub actions? evidence from online software development community,

    Y . Zhang, Y . Wu, T. Chen, T. Wang, H. Liu, and H. Wang, “How do developers talk about GitHub actions? evidence from online software development community,” in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering , 2024, pp. 1–13

    work page 2024

  4. [4]

    Actionsremaker: Reproducing github actions,

    H.-N. Zhu, K. Z. Guan, R. M. Furth, and C. Rubio- Gonzalez, “Actionsremaker: Reproducing github actions,” in 2023 IEEE/ACM 45th International Conference on Software Engineering: Companion Proceedings. IEEE, 2023, pp. 11–15

    work page 2023

  5. [5]

    A tale of ci build failures: An open source and a financial organization perspective,

    C. Vassallo, G. Schermann, F. Zampetti, D. Romano, P. Leitner, A. Zaidman, M. Di Penta, and S. Panichella, “A tale of ci build failures: An open source and a financial organization perspective,” in 2017 IEEE international conference on software maintenance and evolution . IEEE, 2017, pp. 183–193

    work page 2017

  6. [6]

    A hundred days of continuous integration,

    A. Miller, “A hundred days of continuous integration,” in Agile 2008 conference. IEEE, 2008, pp. 289–293

    work page 2008

  7. [7]

    A framework for automating the measurement of devops research and assess- ment (dora) metrics,

    B. Wilkes, A. M. P. Milani, and M.-A. Storey, “A framework for automating the measurement of devops research and assess- ment (dora) metrics,” in 2023 IEEE International Conference on Software Maintenance and Evolution . IEEE, 2023, pp. 62–72

    work page 2023

  8. [8]

    How trustworthy is your ci accelerator? a comparison of the trustwor- thiness of ci acceleration products,

    Z. Zeng, T. Xiao, M. Lamothe, H. Hata, and S. McIntosh, “How trustworthy is your ci accelerator? a comparison of the trustwor- thiness of ci acceleration products,” IEEE Software, 2024

    work page 2024

Show all 70 references
  1. [9]

    Tools and benchmarks for automated log parsing,

    J. Zhu, S. He, J. Liu, P. He, Q. Xie, Z. Zheng, and M. R. Lyu, “Tools and benchmarks for automated log parsing,” in 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice . IEEE, 2019, pp. 121–130

    2019

  2. [10]

    Experience report: Deep learning-based system log analysis for anomaly detection,

    Z. Chen, J. Liu, W. Gu, Y . Su, and M. R. Lyu, “Experience report: Deep learning-based system log analysis for anomaly detection,” arXiv preprint arXiv:2107.05908 , 2021

    2021 arXiv

  3. [11]

    Detecting ci/cd workflow errors through visual inspection of logs,

    S. Alfaro, A. Bergel, and J. Simmonds, “Detecting ci/cd workflow errors through visual inspection of logs,” Authorea Preprints , 2024

    2024

  4. [12]

    mu printgen: Supporting workflow logs analysis through visual microprint,

    ——, “mu printgen: Supporting workflow logs analysis through visual microprint,” in 2023 IEEE Working Conference on Soft- ware Visualization. IEEE, 2023, pp. 45–49

    2023

  5. [13]

    Evaluating code summarization techniques: A new metric and an empirical characterization,

    A. Mastropaolo, M. Ciniselli, M. Di Penta, and G. Bavota, “Evaluating code summarization techniques: A new metric and an empirical characterization,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , 2024, pp. 1–13

    2024

  6. [14]

    Toward automatically completing GitHub workflows,

    A. Mastropaolo, F. Zampetti, G. Bavota, and M. Di Penta, “Toward automatically completing GitHub workflows,” in Pro- ceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1–12

    2024

  7. [15]

    Code review automation: strengths and weaknesses of the state of the art,

    R. Tufano, O. Dabi ´c, A. Mastropaolo, M. Ciniselli, and G. Bavota, “Code review automation: strengths and weaknesses of the state of the art,” IEEE Transactions on Software Engineer- ing, 2024

    2024

  8. [16]

    Using an llm to help with code understanding,

    D. Nam, A. Macvean, V . Hellendoorn, B. Vasilescu, and B. My- ers, “Using an llm to help with code understanding,” in Proceed- ings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1–13

    2024

  9. [17]

    The impact of test case summaries on bug fixing perfor- mance: An empirical investigation,

    S. Panichella, A. Panichella, M. Beller, A. Zaidman, and H. C. Gall, “The impact of test case summaries on bug fixing perfor- mance: An empirical investigation,” in Proceedings of the 38th international conference on software engineering, 2016, pp. 547– 558

    2016

  10. [18]

    What would users change in my app? summarizing app reviews for recommending software changes,

    A. Di Sorbo, S. Panichella, C. V . Alexandru, J. Shimagaki, C. A. Visaggio, G. Canfora, and H. C. Gall, “What would users change in my app? summarizing app reviews for recommending software changes,” in Proceedings of the 2016 24th ACM SIGSOFT international symposium on founda...

    2016

  11. [19]

    Explaining github actions failures with large language models: Challenges, insights, and limitations,

    P. Valenzuela-Toledo, C. Wu, S. Hern ´andez, A. Boll, R. Machacek, S. Panichella, and T. Kehrer, “Explaining github actions failures with large language models: Challenges, insights, and limitations,” Jan. 2025. [Online]. Available: https://doi.org/10.5281/zenodo.14750197

    2025 doi

  12. [20]

    Grounded copilot: How programmers interact with code-generating models,

    S. Barke, M. B. James, and N. Polikarpova, “Grounded copilot: How programmers interact with code-generating models,” Pro- ceedings of the ACM on Programming Languages , vol. 7, no. OOPSLA1, pp. 85–111, 2023

    2023

  13. [21]

    “it would work for me too

    R. Cheng, R. Wang, T. Zimmermann, and D. Ford, ““it would work for me too”: How online communities shape software developers’ trust in ai-powered code generation tools,” ACM Transactions on Interactive Intelligent Systems , vol. 14, no. 2, pp. 1–39, 2024

    2024

  14. [22]

    Github copilot ai pair program- mer: Asset or liability?

    A. M. Dakhel, V . Majdinasab, A. Nikanjam, F. Khomh, M. C. Desmarais, and Z. M. J. Jiang, “Github copilot ai pair program- mer: Asset or liability?” Journal of Systems and Software , vol. 203, p. 111734, 2023

    2023

  15. [23]

    Conversing with copilot: Exploring prompt engineering for solving cs1 problems using natural language,

    P. Denny, V . Kumar, and N. Giacaman, “Conversing with copilot: Exploring prompt engineering for solving cs1 problems using natural language,” in Proceedings of the 54th ACM Technical Symposium on Computer Science Education V . 1, 2023, pp. 1136– 1142

    2023

  16. [24]

    Is github copilot a substitute for human pair- programming? an empirical study,

    S. Imai, “Is github copilot a substitute for human pair- programming? an empirical study,” in Proceedings of the ACM/IEEE 44th International Conference on Software Engineer- ing: Companion Proceedings , 2022, pp. 319–321

    2022

  17. [25]

    Exploring the learnability of program synthesizers by novice programmers,

    D. Jayagopal, J. Lubin, and S. E. Chasins, “Exploring the learnability of program synthesizers by novice programmers,” in Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology , 2022, pp. 1–15

    2022

  18. [26]

    Discovering the syntax and strategies of natural language programming with generative language models,

    E. Jiang, E. Toh, A. Molina, K. Olson, C. Kayacik, A. Donsbach, C. J. Cai, and M. Terry, “Discovering the syntax and strategies of natural language programming with generative language models,” in Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems , 2...

    2022

  19. [27]

    Top programming languages of 2022 - github octoverse,

    GitHub, “Top programming languages of 2022 - github octoverse,” 2022, accessed: 2024-11-07. [Online]. Available: https://octoverse.github.com/2022/top-programming-languages

    2022

  20. [28]

    Sampling projects in github for msr studies,

    O. Dabic, E. Aghajani, and G. Bavota, “Sampling projects in github for msr studies,” in 2021 IEEE/ACM 18th International Conference on Mining Software Repositories . IEEE, 2021, pp. 560–564

    2021

  21. [29]

    Storing and sharing data from a workflow,

    GitHub, Inc., “Storing and sharing data from a workflow,” https://docs.github.com/en/actions/ writing-workflows/choosing-what-your-workflow-does/ storing-and-sharing-data-from-a-workflow, 2024, accessed: 2024-08-26

    2024

  22. [30]

    An Empirical Comparison of Ethnic and Gender Diver- sity of DevOps and non-DevOps Contributions to Open-Source Projects,

    N. R. Weeraddana, X. Xu, M. Alfadel, S. McIntosh, and M. Na- gappan, “An Empirical Comparison of Ethnic and Gender Diver- sity of DevOps and non-DevOps Contributions to Open-Source Projects,” Empirical Software Engineering , vol. 28, no. 150, p. 1–37, 2023

    2023

  23. [31]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix et al., “Llama: Open and efficient foundation language models,” in arXiv preprint arXiv:2302.13971 , 2023

    2023 arXiv

  24. [32]

    Language models are few-shot learners,

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 1877– 1901

    2020

  25. [33]

    Comparative analysis of pre- trained language models for natural language understanding,

    H. Zhang, Y . Sun, and Y . Qi, “Comparative analysis of pre- trained language models for natural language understanding,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021

    2021

  26. [34]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of Ma- chine Learning Research , vol. 21, no. 140, pp. 1–67, 2020

    2020

  27. [35]

    Language models are unsupervised multitask learn- ers,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learn- ers,” in OpenAI Technical Report, 2019

    2019

  28. [36]

    Diversity in software engineering research,

    M. Nagappan, T. Zimmermann, and C. Bird, “Diversity in software engineering research,” in Proceedings of the 2013 9th joint meeting on foundations of software engineering , 2013, pp. 466–476

    2013

  29. [37]

    Sampling in software engineering research: A critical review and guidelines,

    S. Baltes and P. Ralph, “Sampling in software engineering research: A critical review and guidelines,” Empirical Software Engineering, vol. 27, no. 4, p. 94, 2022

    2022

  30. [38]

    Improving developer participation rates in surveys,

    E. Smith, R. Loftin, E. Murphy-Hill, C. Bird, and T. Zimmer- mann, “Improving developer participation rates in surveys,” in 2013 6th International workshop on cooperative and human aspects of software engineering . IEEE, 2013, pp. 89–92

    2013

  31. [39]

    Personal opinion surveys,

    B. A. Kitchenham and S. L. Pfleeger, “Personal opinion surveys,” in Guide to advanced empirical software engineering . Springer, 2008, pp. 63–92

    2008

  32. [40]

    Empirical standards for software engineering research,

    P. Ralph, N. b. Ali, S. Baltes, D. Bianculli, J. Diaz, Y . Dittrich, N. Ernst, M. Felderer, R. Feldt, A. Filieri et al. , “Empirical standards for software engineering research,” arXiv preprint arXiv:2010.03525, 2020

    2010

  33. [41]

    Card-sorting: From text to themes,

    T. Zimmermann, “Card-sorting: From text to themes,” in Per- spectives on data science for software engineering . Elsevier, 2016, pp. 137–141

    2016

  34. [42]

    Breaking the silence: the threats of using llms in software engineering,

    J. Sallou, T. Durieux, and A. Panichella, “Breaking the silence: the threats of using llms in software engineering,” in Proceed- ings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results , 2024, pp. 102–106

    2024

  35. [43]

    Chatgpt vs llama: Impact, reliability, and challenges in stack overflow discussions,

    L. Da Silva, J. Samhi, and F. Khomh, “Chatgpt vs llama: Impact, reliability, and challenges in stack overflow discussions,” arXiv preprint arXiv:2402.08801, 2024

    2024 arXiv

  36. [44]

    Automated program repair in the era of large pre-trained language models,

    C. S. Xia, Y . Wei, and L. Zhang, “Automated program repair in the era of large pre-trained language models,” in 2023 IEEE/ACM 45th International Conference on Software Engineering . IEEE, 2023, pp. 1482–1494

    2023

  37. [45]

    How is chatgpt’s behavior changing over time?

    L. Chen, M. Zaharia, and J. Zou, “How is chatgpt’s behavior changing over time?” arXiv preprint arXiv:2307.09009 , 2023

    2023 arXiv

  38. [46]

    Assessing hidden risks of llms: an empirical study on robustness, consistency, and credibility,

    W. Ye, M. Ou, T. Li, X. Ma, Y . Yanggong, S. Wu, J. Fu, G. Chen, H. Wang, J. Zhao et al. , “Assessing hidden risks of llms: an empirical study on robustness, consistency, and credibility,”arXiv preprint arXiv:2305.10235, 2023

    2023 arXiv

  39. [47]

    Fine-tuning language models to find agreement among humans with diverse preferences,

    M. A. Bakker, M. J. Chadwick, H. R. Sheahan, M. H. Tessler, L. Campbell-Gillingham, J. Balaguer, N. McAleese, A. Glaese, J. Aslanides, M. M. Botvinick, and C. Summerfield, “Fine-tuning language models to find agreement among humans with diverse preferences,” in Proceedings of ...

    2024

  40. [48]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V . Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” in Proceedings of the 36th International Conference on Neural Information Processing Systems, ser. NIPS ’22....

    2024

  41. [49]

    Expertprompting: Instructing large language models to be distinguished experts,

    B. Xu, A. Yang, J. Lin, Q. Wang, C. Zhou, Y . Zhang, and Z. Mao, “Expertprompting: Instructing large language models to be distinguished experts,” CoRR, vol. abs/2305.14688, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2305.14688

    2023 doi arXiv

  42. [50]

    Responsible AI considerations in text summa- rization research: A review of current practices,

    Y . L. Liu et al., “Responsible AI considerations in text summa- rization research: A review of current practices,” in Findings of the Association for Computational Linguistics . Association for Computational Linguistics, 2023, p. 413

    2023

  43. [51]

    Every build you break: developer-oriented assistance for build failure resolution,

    C. Vassallo, S. Proksch, T. Zemp, and H. C. Gall, “Every build you break: developer-oriented assistance for build failure resolution,” Empirical Software Engineering , vol. 25, pp. 2218– 2257, 2020

    2020

  44. [52]

    An empirical analysis of build failures in the continuous integration workflows of java-based open-source software,

    T. Rausch, W. Hummer, P. Leitner, and S. Schulte, “An empirical analysis of build failures in the continuous integration workflows of java-based open-source software,” in 2017 IEEE/ACM 14th In- ternational Conference on Mining Software Repositories. IEEE, 2017, pp. 345–355

    2017

  45. [53]

    A dataset for studying the evolu- tion of source code summarization,

    A. LeClair and C. McMillan, “A dataset for studying the evolu- tion of source code summarization,” in Proceedings of the 2019 IEEE/ACM 16th International Conference on Mining Software Repositories. IEEE, 2019, pp. 377–388

    2019

  46. [54]

    Automatic software summarization: A system- atic literature review,

    L. Moreno et al., “Automatic software summarization: A system- atic literature review,” Journal of Systems and Software, vol. 140, pp. 62–85, 2018

    2018

  47. [55]

    An empirical study on bug report summa- rization,

    A. Tarar et al. , “An empirical study on bug report summa- rization,” in Proceedings of the 2019 35th IEEE International Conference on Software Maintenance and Evolution . IEEE, 2019, pp. 103–113

    2019

  48. [56]

    Rencos: Improving code summarization with retrieved similar codes,

    L. Zhang et al. , “Rencos: Improving code summarization with retrieved similar codes,” in Proceedings of the 2020 ACM/IEEE 42nd International Conference on Software Engineering . ACM, 2020, pp. 90–100

    2020

  49. [57]

    Human vs. machine-generated summaries: A comprehension study,

    A. Stapleton et al. , “Human vs. machine-generated summaries: A comprehension study,” in Proceedings of the 2020 ACM/IEEE International Conference on Software Engineering. ACM, 2020, pp. 232–242

    2020

  50. [58]

    Reassessing the use of bleu and meteor in source code summarization tasks,

    S. Roy et al., “Reassessing the use of bleu and meteor in source code summarization tasks,” Empirical Software Engineering , vol. 26, no. 1, pp. 1–23, 2021

    2021

  51. [59]

    Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,” in Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summa- rization, 2005, pp. 65–72

    2005

  52. [60]

    Semantic similarity metrics for evaluating code summarization techniques,

    M. Haque et al. , “Semantic similarity metrics for evaluating code summarization techniques,” in Proceedings of the 2022 IEEE International Conference on Software Maintenance and Evolution. IEEE, 2022, pp. 320–330

    2022

  53. [61]

    Side: A contrastive learning metric for code summarization evaluation,

    G. Mastropaolo et al. , “Side: A contrastive learning metric for code summarization evaluation,” in Proceedings of the 2024 ACM/IEEE International Conference on Software Engineering , 2024

    2024

  54. [62]

    Summarizing source code using neural attention models,

    S. Iyer et al. , “Summarizing source code using neural attention models,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics , 2016, pp. 207–215

    2016

  55. [63]

    Fedllm: Federated learning-based large lan- guage models for code summarization,

    V . Kumar et al. , “Fedllm: Federated learning-based large lan- guage models for code summarization,” Journal of Software Engineering, 2024

    2024

  56. [64]

    Automatic generation of natural language sum- maries for java classes,

    L. Moreno et al., “Automatic generation of natural language sum- maries for java classes,” in Proceedings of the 2013 IEEE/ACM 28th International Conference on Automated Software Engineer- ing, 2013, pp. 230–240

    2013

  57. [65]

    Automatically summarizing java methods: A context-based approach,

    P. McBurney and C. McMillan, “Automatically summarizing java methods: A context-based approach,” in Proceedings of the 2016 IEEE/ACM 38th International Conference on Software Engineering, 2016, pp. 499–510

    2016

  58. [66]

    Summarizing software artifacts: A bug report case study,

    S. Rastkar et al., “Summarizing software artifacts: A bug report case study,” in Proceedings of the 2014 ACM SIGSOFT Inter- national Symposium on Foundations of Software Engineering (FSE), 2014, pp. 110–120

    2014

  59. [67]

    On the use of automated text summarization techniques for summarizing source code,

    S. Haiduc et al. , “On the use of automated text summarization techniques for summarizing source code,” in Proceedings of the 2010 ACM/IEEE 32nd International Conference on Software Engineering, 2010, pp. 223–233

    2010

  60. [68]

    A systematic review of app review analysis in software engineering,

    L. Dabrowski et al., “A systematic review of app review analysis in software engineering,” Journal of Systems and Software , vol. 190, p. 110789, 2022

    2022

  61. [69]

    Summarizing software artifacts using machine learning: a comprehensive review,

    L. Nazar et al. , “Summarizing software artifacts using machine learning: a comprehensive review,” Journal of Software: Evolu- tion and Process , vol. 28, pp. 170–188, 2016

    2016

  62. [70]

    Summarization techniques for software artifacts: a comprehensive review,

    S. Panichella et al. , “Summarization techniques for software artifacts: a comprehensive review,” ACM Computing Surveys , vol. 50, no. 2, pp. 1–34, 2018

    2018

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.