Pith. sign in

REVIEW 3 major objections 4 minor 151 references

A survey of 85 studies claims that language-model use in mining software repositories collapses into eight application types and that the field is shifting from fine-tuned BERT-style models to prompted GPT-style models.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Across 85 papers, language models in MSR mostly perform classification and generation on issues, code reviews, and commits, with the field shifting from fine-tuned BERT-size models to large instruction-tuned LLMs used via prompting.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection A real contribution to the MSR-survey literature, with a few data-hygiene problems that need fixing before the numbers are trusted. the 3 major comments →

arxiv 2604.00787 v2 pith:FXYI4S4A submitted 2026-04-01 cs.SE

The Rise of Language Models in Mining Software Repositories: A Survey

Miguel Romero-Arjona , Saman Barakat , Ana B. S\'anchez , Sergio Segura This is my paper
classification cs.SE
keywords language modelsmining software repositoriessystematic surveytaxonomy of applicationslarge language modelsreproducibilitymodel evolutionsoftware engineering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is a systematic survey of the young, fast-growing research area that applies language models to mining software repositories—analyzing commits, issues, code reviews, and other traces of software development. The authors' central claim is that all of this work can be organized into a taxonomy of eight application types: classification, generation, extraction, detection, assessment, enhancement, retrieval, and prediction. They also claim a clear historical transition: before 2023 the field ran almost entirely on small, openly available encoder models (BERT and relatives) fine-tuned for specific tasks; by 2025, large, instruction-tuned generative models used through prompting had become the most common choice. Along the way they quantify the field's data habits—GitHub and a handful of reused datasets dominate—and its reproducibility problems: most studies share supplementary materials, but more than a fifth depend exclusively on closed, proprietary models and few release maintained tools. A reader should care because the survey offers the first broad map of this area and a set of concrete challenges for making the work more reproducible and more useful.

Core claim

The survey's discovery is that LM-based MSR is not a scattered set of point solutions but a field with a stable task geometry. Across 85 primary studies, the authors identify 131 instances of LM use, which reduce to eight application types with classification (32.8%) and generation (22.9%) at the top. LMs play two roles: as main contributions that directly produce artifacts or insights, and as enabling components that convert noisy repository traces into structured inputs for downstream analysis—extraction and detection dominate the enabling role. On models, 235 uses of 94 distinct models break down into a clear temporal pattern: BERT-family encoders under half a billion parameters, usually

What carries the argument

The organizing device is an eight-category taxonomy of LM applications—classification, generation, extraction, detection, assessment, enhancement, retrieval, prediction—derived deductively from the primary studies and applied twice, once for main contributions and once for enabling contributions. The second load-bearing piece is the LM usage-mode distinction (fine-tuning vs. prompting-as-is) combined with model-family, parameter-size, and weight-availability categorizations; this is what lets the survey turn a list of 94 models into a trend narrative. The third piece is a small set of data-source and artifact categories (issue reports, code reviews, source code, and so on) used to compare wh

Load-bearing premise

The load-bearing premise is that the 85 primary studies—found by anchoring a search to the field's flagship conference proceedings and then snowballing—are representative enough of the whole field to support the survey's percentages and trend lines; the paper's own abstract states 177 papers while the body reports 85, so the corpus itself is not pinned down.

What would settle it

Count the primary studies listed on the survey's companion website: if the list contains 177 entries rather than 85, the abstract/body mismatch is resolved against the body; more decisively, a full-text search of the same digital libraries using LM-plus-repository-mining keywords, without any venue anchor, would produce a candidate set whose application-type distribution and model timeline could be compared to the survey's—a materially different distribution would falsify the survey's characterization.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Classification and generation are the field's two pillars, together accounting for over half of all LM uses; anyone planning an LM-based MSR contribution can expect the community to compare against these task types.
  • Small open encoder models are still the workhorses of the field, but the momentum is with large prompted models; evaluation and reproducibility expectations should be adjusted accordingly.
  • Dataset reuse is concentrated in a handful of resources; progress will amplify the importance of those datasets' quality.
  • Because 24.7% of studies depend exclusively on proprietary models and costs are rarely reported, many published results cannot be re-run by independent researchers.
  • The field's reproducibility is stronger for raw materials than for tools: 74 of 85 studies share materials, but only a minority of the 21 proposed tools are maintained.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • One consequence the authors leave implicit is that if the taxonomy becomes a shared vocabulary, future work gets a stable baseline for measuring the field's evolution; repeating the survey in three to five years with the same categories would be a natural test.
  • An extension of the cost finding: requiring token-level and monetary cost reporting for any study that uses API-based models would directly address the 6-in-37 reporting gap the survey documents.
  • The survey's observation that requirements and other non-code artifacts are rarely mined suggests, as an editorial extrapolation, that new datasets for those artifact types may be disproportionately valuable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper reports a systematic literature review of language-model (LM) use in mining software repositories (MSR). Based on 85 primary studies collected from MSR conference proceedings (2017–2025) plus backward snowballing, it proposes an eight-type taxonomy of LM applications and answers four research questions concerning applications, data sources/artifacts, model families and temporal trends, and reproducibility/reuse. Headline findings include the distribution of application types (classification 32.8%, generation 22.9%, extraction 15.3%, detection 13%, assessment 5.3%, enhancement 4.6%, retrieval 3.8%, prediction 2.3%), a claimed transition from BERT-style encoder-only models to larger instruction-tuned decoder models, and the observation that reproducibility is only partially supported (24.7% exclusive proprietary-model reliance; 21 tools identified). The protocol is transparent: pre-declared inclusion/exclusion criteria, two-phase screening with reported inter-rater agreement, backward snowballing, and author-confirmed extraction.

Significance. If the findings hold, this is a timely and useful map of an emerging research area. The contribution is primarily empirical: it provides a reproducible corpus, an eight-category taxonomy, and a set of trends that future LM4MSR research can build on. The author-confirmed extraction and the high inter-rater agreement are notable methodological strengths. However, the quantitative claims are only as strong as the corpus, and the current text contains internal inconsistencies in the reported study counts. The survey would be a solid reference if these issues are resolved.

major comments (3)
  1. [Abstract vs. §3.4] The abstract states that the survey is based on an analysis of 177 papers, whereas the full text and all analyses use 85 primary studies (774 initial papers → 100 after Phase 1 → 47 after Phase 2 + 38 snowballed = 85). This is not a trivial typo: every percentage in the paper is computed over the 85-paper corpus. The 177 figure is unexplained and must be corrected or justified.
  2. [§3.4, §3.6.4, Table 1] The search protocol says Phase 2 yielded 47 papers from MSR proceedings and backward snowballing added 38, for a total of 85. However, §3.6.4 and Table 1 report only 38 papers from the MSR conference, described as 45% of all publications. Since 47 of the 85 primary studies must have come from the MSR proceedings under the stated protocol, this is internally inconsistent. The authors should reconcile these counts or explain any later recategorization.
  3. [§3.4 and answers to RQ1–RQ3] Corpus representativeness is asserted but not validated. The search was anchored on MSR conference proceedings, with snowballing only from the references of selected papers, and the authors acknowledge this 'may not have captured all relevant studies.' Yet the headline percentages for application types, model families, and temporal trends are all computed over this corpus. No recall estimate, full multi-database search, or sensitivity analysis is provided. Given that MSR-venue papers account for 45% of the corpus and journal papers only 12%, a systematic under-sampling of non-MSR venues, industry studies, or arXiv-first work could materially shift the reported distribution. I ask for a validation step, such as a comparison against a full IEEE/ACM/Springer search or an independent query, before the quantitative characterization is presented as representative.
minor comments (4)
  1. [References] Reference [1] gives the year 2026 for the first International Workshop on Mining Software Repositories, but the workshop occurred in 2004 (as the URL indicates). Please correct the year.
  2. [§3.6.1, Figure 2b] The quadratic fit to cumulative publication counts yields R² = 0.9933, but cumulative counts almost always follow a smooth curve; this is weak evidence of 'polynomial growth.' Consider reporting annual counts or residual diagnostics instead.
  3. [§4, Answer to RQ1] The text reports study-level counts (45 main, 30 enabling, 10 both) and then application-level counts (85 main, 46 enabling) without explicitly distinguishing the two. Please clarify so readers do not interpret these as contradictory.
  4. [§3.4] The companion website [135] is cited but its contents are not described. Adding a sentence about what is available there would help readers verify the primary-study list and extracted data.

Circularity Check

0 steps flagged

No circularity: the survey's findings are descriptive summaries of an external 85-paper corpus; author self-citations are only methodological precedents and are not load-bearing.

full rationale

The survey's central claims—the eight-type application taxonomy, the percentages (classification 32.8%, generation 22.9%, etc.), and the BERT-to-GPT shift—are all arithmetic summarizations of the 85 primary studies, which are external to the present paper. The taxonomy is induced from those studies (§4: 'We derived this taxonomy through a deductive approach... inspecting each study and assigning an application label... consolidating... into the final taxonomy of eight application types'), so it is not defined in terms of the survey's own conclusions. No parameter is fitted to a subset and then 'predicted' elsewhere; no uniqueness theorem is invoked from the authors' prior work; and no ansatz is smuggled in via self-citation. The author self-citations ([43], [136], [142]) appear only as methodological precedents in §3 and as ordinary background references, and none of the survey's quantitative findings depends on them. The only significant limitation is corpus representativeness, explicitly acknowledged in §3.4: 'Our search strategy may not have captured all relevant studies due to its focus on the MSR conference proceedings. However... we are confident that the resulting corpus is sufficiently robust to characterise current trends.' This is a sampling-validity threat, not a circularity: the findings reduce to the external corpus by design, which is the normal evidentiary basis of a survey, not a self-referential derivation. There is also an apparent inconsistency between the abstract's '177 papers' and the full text's '85 papers,' but that is a correctness/consistency issue, not circularity. Accordingly, the derivation chain is self-contained with respect to the external primary studies, and no circular step is present.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 0 invented entities

This is a literature survey, so the axiom ledger is light: no fitted parameters carry the conclusions and no entities are postulated. The only fitted numbers are cosmetic trend-fit coefficients on cumulative publication counts. The load-bearing assumptions are methodological: corpus representativeness (explicitly acknowledged as a limitation in §3.4) and the reliability of the screening/validation procedure.

free parameters (1)
  • Quadratic trend coefficients for cumulative publication counts = not reported (R² = 0.9933)
    §3.6.1 fits a quadratic to cumulative paper counts and reports R² as evidence of 'strong polynomial growth'; cumulative series are autocorrelated, so the statistic is near-meaningless. Cosmetic; not load-bearing.
axioms (3)
  • domain assumption The MSR conference proceedings plus backward snowballing yield a representative corpus of LM-for-MSR research
    §3.4: the search is anchored on the MSR conference; the authors acknowledge relevant studies may be missed but assert the corpus is 'sufficiently robust to characterise current trends'.
  • domain assumption Two-phase independent screening with high inter-rater agreement yields reliable study inclusion
    §3.4 reports Cohen's κ = 0.965 (Phase 1) and 0.94 (Phase 2); reliability is measured against the raters' shared judgment, not against an external gold standard.
  • domain assumption Author-confirmed extractions are accurate
    §3.5: corresponding authors were contacted to confirm extracted data; this assumes self-reporting by primary-study authors is reliable.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of The Rise of Language Models in Mining Software Repositories: A Survey." pith.science (2026). https://pith.science/paper/FXYI4S4A

@misc{pith2026260400787,
  author       = {Pith},
  title        = {Pith review of: The Rise of Language Models in Mining Software Repositories: A Survey},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FXYI4S4A}},
  note         = {Machine review of arXiv:2604.00787}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The Mining Software Repositories (MSR) field focuses on analyzing the rich data contained in software repositories to derive actionable insights into software processes and products. Mining repositories at scale requires techniques capable of handling large volumes of heterogeneous data, a challenge for which language models (LMs) are increasingly well-suited. Since the advent of Transformer-based architectures, LMs have been rapidly adopted across a wide range of MSR tasks. This article presents a comprehensive survey of the use of LMs in MSR, based on an analysis of 177 papers. We examine how LMs are applied, the types of artifacts analyzed, which models are used, how their adoption has evolved over time, and the availability of supplementary materials and tools supporting reproducibility and reuse. Building on this analysis, we propose a taxonomy of LM applications in MSR, identify key trends shaping the field, and highlight open challenges alongside actionable directions for future research.

Figures

Figures reproduced from arXiv: 2604.00787 by Ana B. S\'anchez, Miguel Romero-Arjona, Saman Barakat, Sergio Segura.

Figure 1
Figure 1. Figure 1: Search process for identifying primary studies. [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: LMs for MSR papers published. 3.6.2 Organisations. We associated each primary study with the affiliation of its first author. Overall, the studies span more than 50 institutions, most of which contributed only one or two papers. Out of the 85 primary studies, 81 were led by authors affiliated with academic institutions, while only three were led by companies (i.e., Google, Microsoft, and JetBrains), with o… view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of MSR applications by LM role: main vs enabling contribution. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Applications and artefacts across contribution types. [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of type of content across source data and generated datasets. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Distribution of LMs by family and type. example, T5-based models account for 36 uses (23 text- and 13 code-oriented), while Llama-based models contribute 24 uses (17 text- and 7 code-oriented). Beyond these categories, embedding-focused models, such as Sentence-BERT, represent 18 uses (8%) and are typically used to encode repository artefacts into vector representations that enable downstream tasks such as… view at source ↗
Figure 7
Figure 7. Figure 7: Distribution of LMs by parameter size range and usage mode. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: LM usage by family over the years. Model names in [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Distribution of platforms hosting supplementary materials. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

151 extracted references · 15 canonical work pages · 1 internal anchor

  1. [1]

    1st International Workshop on Mining Software Repositories

    2026. 1st International Workshop on Mining Software Repositories. http://2004.msrconf.org/. Accessed March 2026

    2026

  2. [2]

    ArduinoProg repository

    2026. ArduinoProg repository. https://github.com/imamnurby/ArduProg. Accessed March 2026

    2026

  3. [3]

    Code review civility enhancement tool repository

    2026. Code review civility enhancement tool repository. https://github.com/Oyakiolo052/ATUC_Artifacts. Accessed March 2026

    2026

  4. [4]

    CuREV repository

    2026. CuREV repository. https://github.com/OussamaSghaier/CuREV. Accessed February 2026

    2026

  5. [5]

    DARA repository

    2026. DARA repository. https://github.com/PurdueDualityLab/PTM-Naming/tree/main/Naming_anomaly_detection. Accessed March 2026

    2026

  6. [6]

    DocWarn repository

    2026. DocWarn repository. https://github.com/awsm-research/docwarn-replication/tree/main/code. Accessed March 2026

    2026

  7. [7]

    2026. GitHub. https://github.com/. Accessed February 2026

    2026

  8. [8]

    GitHub Spec kit

    2026. GitHub Spec kit. https://github.github.com/spec-kit/index.html. Accessed March 2026

    2026

  9. [9]

    GiveMeLabeledIssues API repository

    2026. GiveMeLabeledIssues API repository. https://github.com/JoeyV55/GiveMeLabeledIssuesAPI. Accessed February 2026

    2026

  10. [10]

    GiveMeLabeledIssues user interface repository

    2026. GiveMeLabeledIssues user interface repository. https://github.com/JoeyV55/GiveMeLabeledIssuesUI. Accessed February 2026

    2026

  11. [11]

    Google Play Store

    2026. Google Play Store. https://play.google.com/. Accessed February 2026

    2026

  12. [12]

    Hugging Face

    2026. Hugging Face. https://huggingface.co/. Accessed February 2026

    2026

  13. [13]

    Issue-Labeler repository

    2026. Issue-Labeler repository. https://github.com/issue-labeler/issue-labeler-model. Accessed February 2026

    2026

  14. [14]

    iTiger repository

    2026. iTiger repository. https://github.com/soarsmu/iTiger. Accessed February 2026

    2026

  15. [15]

    2026. Jira. https://www.atlassian.com/software/jira. Accessed March 2026

    2026

  16. [16]

    Jira Topic Extractor repository

    2026. Jira Topic Extractor repository. https://github.com/AuthEceSoftEng/jira-topic-extractor. Accessed March 2026

    2026

  17. [17]

    LLMSecEval repository

    2026. LLMSecEval repository. https://github.com/tuhh-softsec/LLMSecEval/tree/main/Code%20Generation. Accessed February 2026

    2026

  18. [18]

    Matplotlib gallery

    2026. Matplotlib gallery. https://matplotlib.org/stable/gallery/index.html. Accessed February 2026

    2026

  19. [19]

    MicroRec repository

    2026. MicroRec repository. https://github.com/MicroRec/MicroRec-Replication/tree/main/tool-code. Accessed February 2026

    2026

  20. [20]

    Moltbook

    2026. Moltbook. https://www.moltbook.com/. Accessed March 2026

    2026

  21. [21]

    One billionth repository on GitHub

    2026. One billionth repository on GitHub. https://github.com/Red-Killer/shit/issues/1. Accessed March 2026

    2026

  22. [22]

    PonziSleuth repository

    2026. PonziSleuth repository. https://github.com/tasteking/PonziSleuth-ASE24. Accessed March 2026

    2026

  23. [23]

    Pre-trained model metadata extractor tool repository

    2026. Pre-trained model metadata extractor tool repository. https://github.com/PurdueDualityLab/PeaTMOSS-Artifact/tree/main/LLM-Pipeline. Accessed March 2026

    2026

  24. [24]

    Prompt quality improvement tool repository

    2026. Prompt quality improvement tool repository. https://github.com/SOAR-Lab/prompt-knowledge-gap/tree/main/chrome_extension. Accessed March 2026

    2026

  25. [25]

    PSFinder repository

    2026. PSFinder repository. https://github.com/soarsmu/PSFinder. Accessed March 2026

    2026

  26. [26]

    RepoChat repository

    2026. RepoChat repository. https://github.com/sabedu/repositoryChat. Accessed February 2026

    2026

  27. [27]

    Stack Exchange Data Dump

    2026. Stack Exchange Data Dump. https://archive.org/details/stackexchange. Accessed February 2026

    2026

  28. [28]

    Stack Overflow

    2026. Stack Overflow. https://stackoverflow.com/. Accessed March 2026. Manuscript submitted to ACM 24 Romero-Arjona et al

    2026

  29. [29]

    2026. YouTube. https://www.youtube.com/. Accessed February 2026

    2026

  30. [30]

    Samuel Abedu, Ahmad Abdellatif, and Emad Shihab. 2024. LLM-Based Chatbots for Mining Software Repositories: Challenges and Opportunities. InProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering (EASE ’24). 201–210. doi:10.1145/3661167. 3661218

    doi:10.1145/3661167 2024

  31. [31]

    Samuel Abedu, SayedHassan Khatoonabadi, and Emad Shihab. 2026. Synergizing LLMs and Knowledge Graphs: A Novel Approach to Software Repository-Related Question Answering.ACM Trans. Softw. Eng. Methodol.(2026). doi:10.1145/3796510

    work page doi:10.1145/3796510 2026

  32. [33]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 Technical Report.arXiv preprint arXiv:2303.08774(2023). doi:10.48550/arXiv.2303.08774

    Pith/arXiv arXiv doi:10.48550/arxiv.2303.08774 2023

  33. [35]

    Waleed Alhindi, Abdulrahman Aleid, Ilyes Jenhani, and Mohamed Wiem Mkaouer. 2023. Issue-Labeler: an ALBERT-based Jira Plugin for Issue Classification. In2023 IEEE/ACM 10th International Conference on Mobile Software Engineering and Systems (MOBILESoft). 40–43. doi:10.1109/ MOBILSoft59058.2023.00012

    arXiv 2023

  34. [40]

    Florian Angermeir, Maximilian Amougou, Mark Kreitz, Andreas Bauer, Matthias Linhuber, Davide Fucci, Daniel Mendez, Tony Gorschek, et al. 2025. Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies.arXiv preprint arXiv:2510.25506 (2025). doi:10.48550/arXiv.2510.25506

    doi:10.48550/arxiv.2510.25506 2025

  35. [41]

    Experiment

    Claudia Ayala, Burak Turhan, Xavier Franch, and Natalia Juristo. 2022. Use and Misuse of the Term “Experiment” in Mining Software Repositories Research.IEEE Transactions on Software Engineering48, 11 (2022), 4229–4248. doi:10.1109/TSE.2021.3113558

    arXiv 2022

  36. [42]

    Imam Nur Bani Yusuf, Diyanah Binte Abdul Jamal, and Lingxiao Jiang. 2023. Automating Arduino Programming: From Hardware Setups to Sample Source Code Generation. In2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR). 453–464. doi:10.1109/MSR59073. 2023.00069

    arXiv 2023

  37. [43]

    David Benavides, Sergio Segura, and Antonio Ruiz-Cortés. 2010. Automated analysis of feature models 20 years later: A literature review.Information Systems35, 6 (2010), 615–636. doi:10.1016/j.is.2010.01.001

    doi:10.1016/j.is.2010.01.001 2010

  38. [45]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language Models are Few-Shot Learners. InAdvances in Neural Information Processing Systems, Vol. 33. 1877–1901

    2020

  39. [49]

    Chaturvedi, V.B

    K.K. Chaturvedi, V.B. Sing, and Prashast Singh. 2013. Tools in Mining Software Repositories. In2013 13th International Conference on Computational Science and Its Applications. 89–98. doi:10.1109/ICCSA.2013.22

    work page doi:10.1109/iccsa.2013.22 2013

  40. [52]

    Giuseppe Colavito, Filippo Lanubile, and Nicole Novielli. 2023. Few-Shot Learning for Issue Report Classification. In2023 IEEE/ACM 2nd International Workshop on Natural Language-Based Software Engineering (NLBSE). 16–19. doi:10.1109/NLBSE59153.2023.00011

    arXiv 2023

  41. [53]

    Giuseppe Colavito, Filippo Lanubile, and Nicole Novielli. 2025. Benchmarking large language models for automated labeling: The case of issue report classification.Information and Software Technology184 (2025), 107758. doi:10.1016/j.infsof.2025.107758 Manuscript submitted to ACM The Rise of Language Models in Mining Software Repositories: A Survey 25

    arXiv 2025

  42. [55]

    Abdelhalim Hafedh Dahou and Brigitte Mathiak. 2023. Automatic Categorization of Software Repository Domains with Minimal Resources. In International Joint Conference on Knowledge Discovery, Knowledge Engineering, and Knowledge Management. 33–47. doi:10.1007/978-3-031-87569-4_2

    work page doi:10.1007/978-3-031-87569-4_2 2023

  43. [56]

    Farias, Renato Novais, Methanias Colaço Júnior, Luís Paulo da Silva Carvalho, Manoel Mendonça, and Rodrigo Oliveira Spínola

    Mário André de F. Farias, Renato Novais, Methanias Colaço Júnior, Luís Paulo da Silva Carvalho, Manoel Mendonça, and Rodrigo Oliveira Spínola

  44. [57]

    Vincenzo De Martino, Joel Castaño, Fabio Palomba, Xavier Franch, and Silverio Martínez-Fernández. 2025. A Methodological Framework for LLM-Based Mining of Software Repositories.arXiv preprint arXiv:2508.02233(2025). doi:10.48550/arXiv.2508.02233

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2508.02233 2025

  45. [58]

    Vincenzo De Martino, Silverio Martínez-Fernández, and Fabio Palomba. 2025. Do Developers Adopt Green Architectural Tactics for ML-Enabled Systems? A Mining Software Repository Study. In2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS). 135–139. doi:10.1109/ICSE-SEIS66351.2025.00019

    arXiv 2025

  46. [59]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 4171–4186. doi:10.18653...

    doi:10.18653/v1/n19-1423 2019

  47. [61]

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In9th International Conference on Learning Representations

    2021

  48. [64]

    Saad Ezzini, Sallam Abualhaija, Chetan Arora, and Mehrdad Sabetzadeh. 2023. AI-based Question Answering Assistance for Analyzing Natural- language Requirements. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 1277–1289. doi:10.1109/ICSE48619.2023. 00113

    arXiv 2023

  49. [65]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou

  50. [66]

    Enrico Fregnan, Fernando Petrulio, and Alberto Bacchelli. 2022. The evolution of the code during review: an investigation on review changes. Empirical Software Engineering27, 7 (2022), 177

    2022

  51. [68]

    Gonzalez-Barahona and Gregorio Robles

    Jesus M. Gonzalez-Barahona and Gregorio Robles. 2023. Revisiting the reproducibility of empirical software engineering studies based on data retrieved from development repositories.Information and Software Technology164 (2023), 107318. doi:10.1016/j.infsof.2023.107318

    arXiv 2023

  52. [69]

    Marlo Haering, Christoph Stanik, and Walid Maalej. 2021. Automatically Matching Bug Reports With Related App Reviews. In2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). 970–981. doi:10.1109/ICSE43902.2021.00092

    arXiv 2021

  53. [70]

    Ahmed E. Hassan. 2008. The road ahead for Mining Software Repositories. In2008 Frontiers of Software Maintenance. 48–57. doi:10.1109/FOSM. 2008.4659248

    arXiv 2008

  54. [71]

    Junda He, Christoph Treude, and David Lo. 2025. LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision, and the Road Ahead.ACM Trans. Softw. Eng. Methodol.34, 5, Article 124 (2025), 30 pages. doi:10.1145/3712003

    doi:10.1145/3712003 2025

  55. [72]

    Thong Hoang, Hoa Khanh Dam, Yasutaka Kamei, David Lo, and Naoyasu Ubayashi. 2019. DeepJIT: An End-to-End Deep Learning Framework for Just- in-Time Defect Prediction. In2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR). 34–45. doi:10.1109/MSR.2019.00016

    arXiv 2019

  56. [73]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large Language Models for Software Engineering: A Systematic Literature Review.ACM Trans. Softw. Eng. Methodol.33, 8, Article 220 (Dec. 2024), 79 pages. doi:10.1145/3695988

    doi:10.1145/3695988 2024

  57. [74]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. InInternational Conference on Learning Representations

    2022

  58. [75]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions.ACM Trans. Inf. Syst.43, 2, Article 42 (2025), 55 pages. doi:10.1145/3703155

    doi:10.1145/3703155 2025

  59. [76]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. GPT-4o System Card.arXiv preprint arXiv:2410.21276(2024). doi:10.48550/arXiv.2410.21276

    Pith/arXiv arXiv doi:10.48550/arxiv.2410.21276 2024

  60. [77]

    Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. CodeSearchNet Challenge: Evaluating the State of Semantic Code Search.arXiv preprint arXiv:1909.09436(2019). doi:10.48550/arXiv.1909.09436 Manuscript submitted to ACM 26 Romero-Arjona et al

    Pith/arXiv arXiv doi:10.48550/arxiv.1909.09436 2019

  61. [79]

    Mia Mohammad Imran, Preetha Chatterjee, and Kostadin Damevski. 2024. Shedding Light on Software Engineering-Specific Metaphors and Idioms. In2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE). 2555–2567. doi:10.1145/3597503.3639585

    arXiv 2024

  62. [80]

    Mia Mohammad Imran, Preetha Chatterjee, and Kostadin Damevski. 2024. Uncovering the Causes of Emotions in Software Developer Communication Using Zero-shot LLMs. In2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE). 2244–2256. doi:10.1145/3597503.3639223

    arXiv 2024

  63. [81]

    Mia Mohammad Imran, Yashasvi Jain, Preetha Chatterjee, and Kostadin Damevski. 2023. Data Augmentation for Improving Emotion Recognition in Software Engineering Communication. InProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (ASE ’22). Article 29, 13 pages. doi:10.1145/3551349.3556925

    arXiv 2023

  64. [83]

    Maliheh Izadi, Kiana Akbari, and Abbas Heydarnoori. 2022. Predicting the objective and priority of issue reports in software repositories.Empirical Software Engineering27, 2 (2022), 50. doi:10.1007/s10664-021-10085-3

    work page doi:10.1007/s10664-021-10085-3 2022

  65. [85]

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2026. A Survey on Large Language Models for Code Generation. 35, 2, Article 58 (2026), 72 pages. doi:10.1145/3747588

    doi:10.1145/3747588 2026

  66. [86]

    I see models being a whole other thing

    Wenxin Jiang, Mingyu Kim, Chingwo Cheung, Heesoo Kim, George K Thiruvathukal, and James C Davis. 2025. “I see models being a whole other thing”: an empirical study of pre-trained model naming conventions and a tool for enhancing naming consistency.Empirical Software Engineering 30, 6 (2025), 155. doi:10.1007/s10664-025-10711-4

    work page doi:10.1007/s10664-025-10711-4 2025

  67. [88]

    Sathvik Joel, Jie Wu, and Fatemeh Fard. 2025. A Survey on LLM-based Code Generation for Low-Resource and Domain-Specific Programming Languages.ACM Trans. Softw. Eng. Methodol.(2025). doi:10.1145/3770084

    doi:10.1145/3770084 2025

  68. [90]

    Yuning Kang, Zan Wang, Hongyu Zhang, Junjie Chen, and Hanmo You. 2021. APIRecX: Cross-Library API Recommendation via Pre-Trained Language Model. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 3425–3436. doi:10.18653/v1/2021.emnlp- main.275

    doi:10.18653/v1/2021.emnlp- 2021

  69. [91]

    Obie, Pragya Agrawal, and John Grundy

    Hourieh Khalajzadeh, Mojtaba Shahin, Humphrey O. Obie, Pragya Agrawal, and John Grundy. 2023. Supporting Developers in Addressing Human-Centric Issues in Mobile Apps.IEEE Transactions on Software Engineering49, 4 (2023), 2149–2168. doi:10.1109/TSE.2022.3212329

    arXiv 2023

  70. [92]

    Siwon Kim, Sangdoo Yun, Hwaran Lee, Martin Gubri, Sungroh Yoon, and Seong Joon Oh. 2023. ProPILE: probing privacy leakage in large language models. InProceedings of the 37th International Conference on Neural Information Processing Systems (NIPS ’23). Article 911, 13 pages

    2023

  71. [93]

    2004.Procedures for Performing Systematic Reviews

    Barbara Kitchenham. 2004.Procedures for Performing Systematic Reviews. Joint Technical Report, Keele University TR/SE-0401 and NICTA 0400011T.1

    2004

  72. [94]

    Elvan Kula, Arie van Deursen, and Georgios Gousios. 2021. Modeling Team Dynamics for the Characterization and Prediction of Delays in User Stories. In2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). 991–1002. doi:10.1109/ASE51524.2021.9678939

    arXiv 2021

  73. [95]

    Andrew Lampinen, Ishita Dasgupta, Stephanie Chan, Kory Mathewson, Mh Tessler, Antonia Creswell, James McClelland, Jane Wang, and Felix Hill

  74. [96]

    Stanislav Levin and Amiram Yehudai. 2017. Boosting Automatic Commit Classification Into Maintenance Activities By Utilizing Source Code Changes. InProceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering (PROMISE). 97–106. doi:10.1145/3127005.3127016

    arXiv 2017

  75. [97]

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 7871–7880. doi...

    doi:10.18653/v1/2020.acl-main.703 2020

  76. [98]

    Jiawei Li, David Faragó, Christian Petrov, and Iftekhar Ahmed. 2024. Only diff Is Not Enough: Generating Commit Messages Leveraging Reasoning and Action of Large Language Model.Proc. ACM Softw. Eng.1, FSE, Article 34 (2024), 22 pages. doi:10.1145/3643760

    doi:10.1145/3643760 2024

  77. [100]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. StarCoder: may the source be with you!arXiv preprint arXiv:2305.06161(2023). doi:10.48550/arXiv.2305.06161 Manuscript submitted to ACM The Rise of Language Models in Mining Software Repositories: A Survey 27

    Pith/arXiv arXiv doi:10.48550/arxiv.2305.06161 2023

  78. [101]

    Yikun Li, Mohamed Soliman, and Paris Avgeriou. 2023. Automatic identification of self-admitted technical debt from four different sources. Empirical Software Engineering28, 3 (2023), 65. doi:10.1007/s10664-023-10297-9

    doi:10.1007/s10664-023-10297-9 2023

  79. [102]

    Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, and Neel Sundaresan. 2022. Automating Code Review Activities by Large-Scale Pre-training. InProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ES...

    arXiv 2022

  80. [103]

    Bo Lin, Shangwen Wang, Zhongxin Liu, Yepang Liu, Xin Xia, and Xiaoguang Mao. 2023. CCT5: A Code-Change-Oriented Pre-trained Model. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2023). 1509–1521. doi:10.1145/3611643.3616339

    arXiv 2023

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.