REVIEW 4 major objections 5 minor 26 references
A Framework for Using LLMs for Repository Mining Studies in Empirical Software Engineering
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read This paper proposes PRIMES, a four-stage checklist for using LLMs in repository mining, and claims that standardizing prompt engineering improves the reliability and reproducibility of such studies.
desk verdict A useful, honestly written checklist for LLM-based repository mining, but its reliability claim outruns the evidence because validation only covers known classes. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the PRIMES checklist itself, a four-stage workflow: (1) creation of prompts for piloting, (2) an iterative prompt pilot test on a single LLM with dual human/LLM annotation and a Cohen's kappa threshold, (3) evaluation among multiple LLMs using an oracle and metrics like accuracy and cost, and (4) output validation with automated checks for format, duplication, hallucination, and provenance tracking. The central statistical gate is the agreement coefficient: the prompt is only used on the full dataset once it passes a pre-specified threshold (in the authors' case, kappa above 0.9) on a representative sample.
What would settle it
Run the PRIMES validation loop on a new repository mining task, then apply the accepted prompt to the full dataset and have human experts annotate a second random sample. If the observed agreement on the second sample falls materially below the 0.9 threshold, the framework's assumption that sample agreement generalizes would be refuted.
Extended reading notes
Core claim
The core discovery is a practical, experience-derived framework—PRIMES—that packages the authors' lessons as a checklist for conducting repository mining studies with LLMs. The framework's central assertion is that reliable LLM-based data collection depends on an iterative loop: define objectives, craft a prompt, test it on a small sample against human annotation using an agreement coefficient (e.g., Cohen's kappa above 0.9), refine the prompt until the threshold is met, compare multiple LLMs on the same benchmark using an oracle, and finally validate outputs for formatting errors, duplications, and hallucinations while tracking provenance. The authors report that applying this process in their two studies led to high classification accuracy (95.58% and 97.91% for the two models compared) and enabled them to select the model best suited to the task.
Load-bearing premise
The framework assumes that once the LLM agrees with human annotations on a representative sample (kappa above 0.9), the same prompt produces equally reliable outputs on the entire dataset, which requires the sample to be representative, the human oracle to be correct, and 0.9 to be a sufficient threshold.
Editorial extensions
If this is right
- Following PRIMES should make LLM-based data collection more reproducible, because the prompt refinement and validation steps are documented rather than ad hoc.
- Comparing multiple LLMs before committing to one should reduce model-specific bias and let researchers choose based on accuracy, explanation quality, and cost.
- The agreement-threshold gate should prevent low-quality prompts from being applied to an entire dataset, which reduces the risk of large-scale annotation errors.
- Incorporating PRIMES into reporting standards would make LLM mining studies easier to compare and to build upon.
Reading between the lines
- Beyond repository mining, the same validate-then-scale loop could apply to any LLM-assisted annotation task in empirical research, such as classifying survey responses or issue reports, though the paper does not test this.
- The specific threshold of Cohen's kappa above 0.9 is an experience-based default, not a statistically derived guarantee; testing how classification error on the full dataset relates to sample kappa would be a natural next step.
- The framework's cost-comparison step implies a trade-off: using multiple LLMs improves reliability but increases cost, so a future extension could formalize when the added reliability outweighs the expense.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This experience report proposes PRIMES, a four-stage framework for using large language models (LLMs) in repository mining studies: (1) creation of prompts for piloting, (2) pilot testing with statistical agreement and iterative refinement, (3) evaluation among multiple LLMs against a human oracle, and (4) output validation including deduplication, formatting, hallucination checks, tracking, and automation. The framework is derived from the authors' two prior studies on green architectural tactics in ML-enabled systems and on evolution of ML models on Hugging Face. The paper's central claim is that standardizing prompt engineering with PRIMES can enhance the reliability and reproducibility of LLM-based data collection.
Significance. If validated, the PRIMES framework would offer a practical, actionable checklist for empirical software engineering researchers, and the paper's emphasis on reproducibility through output tracking, automated validation, and cost-aware model comparison is a genuine strength. The paper is also honest about being a preliminary framework and about the broader limitations of LLMs, such as hallucinations and biases. However, the evidence for the central claim is anecdotal and self-reported, and the quantitative validation is limited to agreement on an oracle restricted to documented categories, leaving the novel outputs that the pipeline aims to discover without empirical support. The framework is reasonable as a set of heuristics from two case studies, but the reliability claim is not yet established at the strength the conclusions suggest.
major comments (4)
- [Section III-B and III-C] The acceptance gate for full-data collection is agreement on a pilot sample (e.g., Cohen's kappa > 0.9 in Section III-B), but the oracle in the exemplar study [9] is restricted to documented green tactics: Section III-C states that all new or undocumented tactics were set aside for later LLM-assisted discovery. High agreement on the documented subset therefore establishes reliability only for known categories; it says nothing about the accuracy of the undocumented or novel tactics that the pipeline is designed to surface. The paper itself reports that both GPT-4o and Claude 3 Haiku failed to identify genuinely new tactics, with Claude only occasionally repeating known ones, so no quantitative evidence supports the reliability of the novel portion of the final dataset. The conclusion that PRIMES 'can enhance the reliability' of LLM-based repository mining is consequently stronger than the evidence presented.
- [Section IV and Conclusion] The central claim is supported only by retrospective self-reports from the authors' two studies; there is no baseline comparison, no measurement of error rates before versus after applying PRIMES, and no independent replication. Phrases such as 'Our findings indicate' overstate what an experience report with two anecdotal applications can establish. The paper should either explicitly restrict the claim to 'the authors found these practices useful in two studies' or provide a small evaluation, for example a re-analysis of one study with and without the framework's validation steps.
- [Section III-B] The threshold of Cohen's kappa > 0.9 is introduced without justification or discussion of its dependence on category prevalence and base rates. In settings with rare categories, a high overall kappa can coexist with high error rates on the rare class, so the gate does not guarantee reliable extraction of the least frequent, most interesting items. The paper should justify the threshold or use class-wise metrics (precision and recall per category) in addition to the aggregate coefficient.
- [Section III-C] The oracle construction paragraph cites a sample size calculator and states that a 'sufficiently large, representative data subset' is needed, but it does not report the sample size, sampling strategy, or representativeness checks used in the exemplar studies. Without this information, a reader cannot assess whether the validation sample supports the claimed generalization to the full dataset. Reporting these details would also align with the paper's own reproducibility goals.
minor comments (5)
- [Section I and II] Minor language issues: 'based empirical software engineering' in Section I should be 'based on empirical software engineering', and 'For the sake of our knowledge' in Section II should be 'To the best of our knowledge'.
- [Figure 2] Figure 2 is dense and the text is very small; consider enlarging the figure or providing a table version of the checklist so that the steps are readable.
- [References] Reference [26] appears to be incomplete (missing venue or publisher), and reference [22] lacks an access date for the URL.
- [Section IV] The phrase 'in the acknowledged the limitations' is ungrammatical; consider 'in addressing the acknowledged limitations'.
- [Section III-C] The three example metrics are listed clearly, but the connection between those metrics and the preceding 'Benchmark' paragraph is not explicit; a sentence linking the benchmark design to the metrics would improve readability.
Circularity Check
No significant circularity found; PRIMES is a self-report experience framework with independent statistical validation gates.
full rationale
The paper is an experience report that packages lessons learned from the authors' two prior repository-mining studies into a checklist (PRIMES). The central claim that following PRIMES can enhance reliability and reproducibility is a proposal supported by those experiences, not a result derived from the framework by construction. The validation procedure (Cohen's kappa > 0.9 against a human oracle on a sample) is an external statistical check: the agreement coefficient is computed independently of the checklist and is not a fitted parameter renamed as a prediction. The oracle is explicitly limited to previously documented green tactics, and the paper openly reports that both LLMs 'fail to identify new ones'; this is a validity/generalizability limitation, not a circular reduction, because the framework does not define 'reliability' as agreement on the known subset only, nor does it claim to have validated novel-tactic discovery. The self-citations [9], [10] are the evidentiary base for an experience report, which is normal and not load-bearing in the sense of an unverified uniqueness theorem or an imported ansatz. No equation, definition, or claim in the paper reduces to its own inputs; therefore no circularity step is exhibited.
Assumptions & free parameters
free parameters (2)
- Agreement threshold for prompt acceptance =
Cohen's kappa > 0.9
- Oracle sample size =
not specified, uses a sample size calculator
assumptions (3)
- domain assumption Human annotations on a representative sample are a valid oracle for LLM output quality.
- domain assumption Cohen's kappa above 0.9 between LLM and human annotators is sufficient to ensure reliable data extraction.
- domain assumption Experiences from the two underlying studies generalize to other repository mining contexts.
invented entities (1)
-
PRIMES framework
Cite this review
Pith. "Pith review of A Framework for Using LLMs for Repository Mining Studies in Empirical Software Engineering." pith.science (2026). https://pith.science/paper/HYGSALTR
@misc{pith2026241109974,
author = {Pith},
title = {Pith review of: A Framework for Using LLMs for Repository Mining Studies in Empirical Software Engineering},
year = {2026},
howpublished = {\url{https://pith.science/paper/HYGSALTR}},
note = {Machine review of arXiv:2411.09974}
}
read the original abstract
Context: The emergence of Large Language Models (LLMs) has significantly transformed Software Engineering (SE) by providing innovative methods for analyzing software repositories. Objectives: Our objective is to establish a practical framework for future SE researchers needing to enhance the data collection and dataset while conducting software repository mining studies using LLMs. Method: This experience report shares insights from two previous repository mining studies, focusing on the methodologies used for creating, refining, and validating prompts that enhance the output of LLMs, particularly in the context of data collection in empirical studies. Results: Our research packages a framework, coined Prompt Refinement and Insights for Mining Empirical Software repositories (PRIMES), consisting of a checklist that can improve LLM usage performance, enhance output quality, and minimize errors through iterative processes and comparisons among different LLMs. We also emphasize the significance of reproducibility by implementing mechanisms for tracking model results. Conclusion: Our findings indicate that standardizing prompt engineering and using PRIMES can enhance the reliability and reproducibility of studies utilizing LLMs. Ultimately, this work calls for further research to address challenges like hallucinations, model biases, and cost-effectiveness in integrating LLMs into workflows.
Figures
Reference graph
Works this paper leans on
-
[9]
V . De Martino, S. Mart ´ınez-Fern´andez, and F. Palomba, “Do developers adopt green architectural tactics for ml-enabled systems? a mining software repository study,” arXiv preprint arXiv:2410.06708 , 2024
arXiv 2024
-
[1]
Using an llm to help with code understanding,
D. Nam, A. Macvean, V . Hellendoorn, B. Vasilescu, and B. Myers, “Using an llm to help with code understanding,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , 2024, pp. 1–13
2024
-
[2]
Uncovering the causes of emotions in software developer communication using zero-shot llms,
M. M. Imran, P. Chatterjee, and K. Damevski, “Uncovering the causes of emotions in software developer communication using zero-shot llms,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1–13
2024
-
[3]
Large language models for software engi- neering: A systematic literature review,
X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engi- neering: A systematic literature review,” ACM Transactions on Software Engineering and Methodology , 2023
2023
-
[4]
J. White, S. Hays, Q. Fu, J. Spencer-Smith, and D. C. Schmidt, “Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design,” in Generative AI for Effective Software Development. Springer, 2024, pp. 71–108
work page 2024
-
[5]
Is chatgpt the ultimate programming assistant–how far is it?
H. Tian, W. Lu, T. O. Li, X. Tang, S.-C. Cheung, J. Klein, and T. F. Bissyand´e, “Is chatgpt the ultimate programming assistant–how far is it?” arXiv preprint arXiv:2304.11938 , 2023
arXiv 2023
-
[6]
A. Della Porta, V . De Martino, G. Recupito, C. Iemmino, G. Catolino, D. Di Nucci, and F. Palomba, “Using large language models to support software engineering documentation in waterfall life cycles: Are we there yet?” 2024
work page 2024
-
[7]
Unveiling chatgpt’s usage in open source projects: A mining-based study,
R. Tufano, A. Mastropaolo, F. Pepe, O. Dabi ´c, M. Di Penta, and G. Bavota, “Unveiling chatgpt’s usage in open source projects: A mining-based study,” in 2024 IEEE/ACM 21st International Conference on Mining Software Repositories (MSR) . IEEE, 2024, pp. 571–583
work page 2024
Show all 26 references
-
[8]
Large language models for software engineering: Sur- vey and open problems,
A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Sur- vey and open problems,” in 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). IEEE...
2023
-
[10]
How do machine learning models change?
J. Casta ˜no, R. Caba˜nas, A. Salmer´on, D. Lo, and S. Mart´ınez-Fern´andez, “How do machine learning models change?” 2024
2024
-
[11]
Detecting code smells using chatgpt: Initial insights,
L. L. Silva, J. Silva, J. E. Montandon, M. Andrade, and M. T. Valente, “Detecting code smells using chatgpt: Initial insights,” in Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement , 2024, pp. 400–406
2024
-
[12]
Prompt engineering or fine tuning: An empirical assessment of large language models in automated software engineering tasks,
J. Shin, C. Tang, T. Mohati, M. Nayebi, S. Wang, and H. Hem- mati, “Prompt engineering or fine tuning: An empirical assessment of large language models in automated software engineering tasks,” arXiv preprint arXiv:2310.10508, 2023
-
[13]
Detecting code comment inconsistencies using llm and program analysis,
Y . Zhang, “Detecting code comment inconsistencies using llm and program analysis,” in Companion Proceedings of the 32nd ACM Inter- national Conference on the Foundations of Software Engineering , 2024, pp. 683–685
2024
-
[14]
A systematic survey of prompt engineering in large language models: Techniques and applications,
P. Sahoo, A. K. Singh, S. Saha, V . Jain, S. Mondal, and A. Chadha, “A systematic survey of prompt engineering in large language models: Techniques and applications,” arXiv preprint arXiv:2402.07927 , 2024
2024 arXiv
-
[15]
Finetuned language models are zero-shot learners,
J. Wei, M. Bosma, V . Y . Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V . Le, “Finetuned language models are zero-shot learners,” arXiv preprint arXiv:2109.01652 , 2021
2021 arXiv
-
[16]
Do advanced language models eliminate the need for prompt engineering in software engineering?
G. Wang, Z. Sun, Z. Gong, S. Ye, Y . Chen, Y . Zhao, Q. Liang, and D. Hao, “Do advanced language models eliminate the need for prompt engineering in software engineering?” arXiv preprint arXiv:2411.02093, 2024
2024 arXiv
-
[17]
Language models are few-shot learners,
T. B. Brown, “Language models are few-shot learners,” arXiv preprint arXiv:2005.14165, 2020
2005 arXiv
-
[18]
Chain-of-thought prompting elicits reasoning in large language models,
J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24 824–24 837, 2022
2022
-
[19]
A coefficient of agreement for nominal scales,
J. Cohen, “A coefficient of agreement for nominal scales,” Educational and psychological measurement , vol. 20, no. 1, pp. 37–46, 1960
1960
-
[20]
A survey of large language models,
W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y . Hou, Y . Min, B. Zhang, J. Zhang, Z. Dong et al. , “A survey of large language models,” arXiv preprint arXiv:2303.18223, 2023
2023 arXiv
-
[21]
Sampling in software engineering research: A critical review and guidelines,
S. Baltes and P. Ralph, “Sampling in software engineering research: A critical review and guidelines,” Empirical Software Engineering, vol. 27, no. 4, p. 94, 2022
2022
-
[22]
Chatgpt api keys,
“Chatgpt api keys,” https://openai.com/index/openai-api/, 2024
2024
-
[23]
Getting started - anthropic,
“Getting started - anthropic,” https://docs.anthropic.com/en/api/ getting-started, 2024
2024
-
[24]
A synthesis of green architectural tactics for ml-enabled systems,
H. J ¨arvenp¨a¨a, P. Lago, J. Bogner, G. Lewis, H. Muccini, and I. Ozkaya, “A synthesis of green architectural tactics for ml-enabled systems,” in Proceedings of the 46th International Conference on Software Engineer- ing: Software Engineering in Society , 2024, pp. 130–141
2024
-
[25]
Home — Great Expectations — docs.greatexpectations.io,
“Home — Great Expectations — docs.greatexpectations.io,” https:// docs.greatexpectations.io/docs/home/, [Accessed 29-10-2024]
2024
-
[26]
Teaching mining software repositories
Z. Codabux, F. Fard, R. Verdecchia, F. Palomba, D. Di Nucci, and G. Recupito, “Teaching mining software repositories.”
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.