{"id":"00f9d858-d690-4bcf-b5b3-e827bd1f3d52","arxiv_id":"2411.09974","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"PRIMES is a proposed four-stage checklist for prompt creation, pilot testing, multi-LLM evaluation, and output validation in LLM-based repository mining.","lead":"This paper packages lessons from two repository mining studies into a four-stage checklist, PRIMES, for creating, refining, and validating LLM prompts. It provides practical guidance for researchers using LLMs to label software repository data, but the framework's benefits are shown through the authors' own experience rather than a controlled evaluation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validation gate targets only documented categories, so the PRIMES claim of reliable LLM output is unsupported for the novel 'undocumented' findings the pipeline is meant to produce.","rationale":"The reader's conditional verdict is correct for the right general reason: the benefits of PRIMES are not independently validated. My stress-test sharpens this into a specific logical gap. For the central reliability claim to hold, the statistical agreement gate in Section III-B must certify the outputs that enter the LLM-enhanced dataset. In the described use cases, the LLM is asked to identify both documented and undocumented items, yet the oracle construction in Section III-C explicitly excludes undocumented items from manual validation. A kappa above 0.9 on the documented subset is therefore compatible with arbitrary error rates on the undocumented subset, which is precisely where hallucinations and false discoveries are most likely. This is not a disagreement with external consensus; it is an internal mismatch between the framework's validation procedure and its data-collection objective. The proposed concrete test would settle whether the gap is real or merely theoretical. I do not recommend rejecting the paper: it is explicitly a preliminary experience report and candidly lists limitations, including hallucinations and model biases. The conditional verdict already demands evidence before treating the framework's claimed benefits as established, which is the appropriate stance. I mark partial agreement with the reader because the reader identified a broader trust-in-oracle and sample-representativeness assumption; I isolate the more specific and, in my reading, more load-bearing failure mode of oracle incompleteness for undocumented outputs.","tokens_in":8665,"tokens_out":3826,"duration_ms":45121,"concrete_test":"Re-run the validation protocol of study [9] on a held-out stratified sample that includes files with known green tactics and files likely to contain undocumented tactics. Have two independent domain experts annotate both subsets. Apply PRIMES by refining the prompt until kappa > 0.9 on the known-tactic subset only, then compute precision, recall, and kappa separately on the undocumented subset. If performance on the undocumented subset falls substantially below the documented subset (e.g., precision below 0.8 or kappa below 0.9), the kappa gate does not generalize to novel findings, and the reliability claim fails for exactly the outputs the framework is designed to enable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The PRIMES gate that licenses full-data collection is agreement on a pilot sample (e.g., Cohen's kappa > 0.9, Section III-B). But the oracle used in the exemplar study [9] is deliberately limited to green tactics already documented in the literature: Section III-C states 'we limited manual classification to green tactics already documented in the literature... All new or undocumented tactics were set aside for later analysis when the LLM could help us discover them.' This makes the validation distribution conditional on a predefined label set. High kappa on that subset establishes only that the LLM agrees with a human on known classes; it does not validate the outputs of greatest scientific interest and highest hallucination risk, namely undocumented or new tactics. The paper itself reports that both GPT-4o and Claude 3 Haiku 'fail to identify new ones,' while Claude merely 'occasionally repeated known ones' when asked to go beyond the catalog. If the final dataset is enriched with such novel items, no numerical evidence in the paper shows their accuracy. The conclusion that PRIMES 'can enhance the reliability' of LLM-based repository mining is therefore stronger than the evidence: the only quantitative support is conditional on an incomplete oracle.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This experience report proposes PRIMES, a four-stage framework for using large language models (LLMs) in repository mining studies: (1) creation of prompts for piloting, (2) pilot testing with statistical agreement and iterative refinement, (3) evaluation among multiple LLMs against a human oracle, and (4) output validation including deduplication, formatting, hallucination checks, tracking, and automation. The framework is derived from the authors' two prior studies on green architectural tactics in ML-enabled systems and on evolution of ML models on Hugging Face. The paper's central claim is that standardizing prompt engineering with PRIMES can enhance the reliability and reproducibility of LLM-based data collection.","tokens_in":8894,"tokens_out":4238,"duration_ms":44984,"significance":"If validated, the PRIMES framework would offer a practical, actionable checklist for empirical software engineering researchers, and the paper's emphasis on reproducibility through output tracking, automated validation, and cost-aware model comparison is a genuine strength. The paper is also honest about being a preliminary framework and about the broader limitations of LLMs, such as hallucinations and biases. However, the evidence for the central claim is anecdotal and self-reported, and the quantitative validation is limited to agreement on an oracle restricted to documented categories, leaving the novel outputs that the pipeline aims to discover without empirical support. The framework is reasonable as a set of heuristics from two case studies, but the reliability claim is not yet established at the strength the conclusions suggest.","major_comments":[{"comment":"The acceptance gate for full-data collection is agreement on a pilot sample (e.g., Cohen's kappa > 0.9 in Section III-B), but the oracle in the exemplar study [9] is restricted to documented green tactics: Section III-C states that all new or undocumented tactics were set aside for later LLM-assisted discovery. High agreement on the documented subset therefore establishes reliability only for known categories; it says nothing about the accuracy of the undocumented or novel tactics that the pipeline is designed to surface. The paper itself reports that both GPT-4o and Claude 3 Haiku failed to identify genuinely new tactics, with Claude only occasionally repeating known ones, so no quantitative evidence supports the reliability of the novel portion of the final dataset. The conclusion that PRIMES 'can enhance the reliability' of LLM-based repository mining is consequently stronger than the evidence presented.","section":"Section III-B and III-C"},{"comment":"The central claim is supported only by retrospective self-reports from the authors' two studies; there is no baseline comparison, no measurement of error rates before versus after applying PRIMES, and no independent replication. Phrases such as 'Our findings indicate' overstate what an experience report with two anecdotal applications can establish. The paper should either explicitly restrict the claim to 'the authors found these practices useful in two studies' or provide a small evaluation, for example a re-analysis of one study with and without the framework's validation steps.","section":"Section IV and Conclusion"},{"comment":"The threshold of Cohen's kappa > 0.9 is introduced without justification or discussion of its dependence on category prevalence and base rates. In settings with rare categories, a high overall kappa can coexist with high error rates on the rare class, so the gate does not guarantee reliable extraction of the least frequent, most interesting items. The paper should justify the threshold or use class-wise metrics (precision and recall per category) in addition to the aggregate coefficient.","section":"Section III-B"},{"comment":"The oracle construction paragraph cites a sample size calculator and states that a 'sufficiently large, representative data subset' is needed, but it does not report the sample size, sampling strategy, or representativeness checks used in the exemplar studies. Without this information, a reader cannot assess whether the validation sample supports the claimed generalization to the full dataset. Reporting these details would also align with the paper's own reproducibility goals.","section":"Section III-C"}],"minor_comments":[{"comment":"Minor language issues: 'based empirical software engineering' in Section I should be 'based on empirical software engineering', and 'For the sake of our knowledge' in Section II should be 'To the best of our knowledge'.","section":"Section I and II"},{"comment":"Figure 2 is dense and the text is very small; consider enlarging the figure or providing a table version of the checklist so that the steps are readable.","section":"Figure 2"},{"comment":"Reference [26] appears to be incomplete (missing venue or publisher), and reference [22] lacks an access date for the URL.","section":"References"},{"comment":"The phrase 'in the acknowledged the limitations' is ungrammatical; consider 'in addressing the acknowledged limitations'.","section":"Section IV"},{"comment":"The three example metrics are listed clearly, but the connection between those metrics and the preceding 'Benchmark' paragraph is not explicit; a sentence linking the benchmark design to the metrics would improve readability.","section":"Section III-C"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is better positioned as a lessons-learned experience report than as a validated framework; the requested revision should reframe the central claim accordingly. The paper fits venues that accept experience reports, and it would benefit from a companion artifact containing the checklist in a machine-readable form to support adoption and future validation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"PRIMES is a four-stage checklist for using LLMs in repository mining: prompt creation, pilot testing against a human oracle, cross-LLM comparison, and output validation. That is the real contribution. It is not a breakthrough, but it is a sane, structured procedure that researchers in this subfield can adopt or adapt. The paper is an experience report, and it reads like one: the authors are candid about what they did, what failed, and what they would advise others to do. The emphasis on iterating prompts against an agreement threshold, comparing multiple LLMs before committing, and tracking outputs for reproducibility is concrete and actionable. The two example applications help show how the steps play out.\n\nThe main soft spot is exactly where the stress-test lands. The oracle used to validate the LLM in the green-tactics study is built from documented tactics only; the paper says new or undocumented tactics were set aside. So a Cohen's kappa above 0.9 on that subset says the LLM agrees with a human on known classes. It does not validate the outputs you most want from an LLM—the novel or undocumented findings. In fact, the paper reports both models failed to identify genuinely new tactics, with Claude occasionally repeating known ones. That makes the conclusion that PRIMES 'can enhance the reliability' of LLM-based mining stronger than the evidence. The claim is not false, but it is conditional on an incomplete oracle, and the paper does not flag that condition in its conclusion. The agreement threshold and oracle sample size are also asserted rather than justified; they are free parameters that could change the outcome. And the 'first framework' claim is plausible but not established—the related work section is thin relative to the prompt-engineering literature.\n\nThere are minor issues: the writing has some grammatical wobbles, and the hint boxes add little. Those are not blockers.\n\nWho is this for? Anyone setting up an LLM-based repository mining study who wants a head start on prompt validation, multi-model comparison, and reporting. It is a useful checklist, not a validated methodology. A serious editor should send it to peer review, because the subfield needs shared practice, but the authors should be pushed to soften the reliability conclusion, report the validation distribution explicitly, and add a limitation that the gate does not cover novel-label discovery. I would cite this if I were doing MSR with LLMs, mostly as a citable checklist. Bring it to reading group if you are working in that area; otherwise it is a quick skim.","headline":"A useful, honestly written checklist for LLM-based repository mining, but its reliability claim outruns the evidence because validation only covers known classes.","tokens_in":9411,"tokens_out":2712,"would_cite":true,"duration_ms":26586,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes PRIMES, a four-stage checklist for using LLMs in repository mining, and claims that standardizing prompt engineering improves the reliability and reproducibility of such studies.","keywords":["large language models","mining software repositories","prompt engineering","PRIMES","empirical software engineering","reproducibility","Cohen's kappa","LLM validation"],"falsifier":"Run the PRIMES validation loop on a new repository mining task, then apply the accepted prompt to the full dataset and have human experts annotate a second random sample. If the observed agreement on the second sample falls materially below the 0.9 threshold, the framework's assumption that sample agreement generalizes would be refuted.","tokens_in":8494,"feed_emoji":"✅","tokens_out":5307,"duration_ms":46523,"temperature":0.7,"pith_summary":"The paper is an experience report from two prior LLM-based repository mining studies, one on green architectural tactics in ML projects and one on the evolution of ML models. Its central claim is that poor prompt design and lack of validation are the main sources of error when LLMs are used to extract data from software repositories, and that a structured four-stage checklist can reduce those errors. That checklist, called PRIMES, covers prompt creation, pilot testing against human annotations with an agreement threshold, comparison of multiple LLMs against an oracle, and output validation. If the claim holds, researchers can treat LLM mining as a more trustworthy data-collection method rather than an ad hoc exercise.","feed_headline":"LLM mining needs a checklist: PRIMES makes it reproducible","feed_subtitle":"PRIMES validates prompts against human labels, compares models, and tracks outputs before you trust the data.","key_machinery":"The load-bearing mechanism is the PRIMES checklist itself, a four-stage workflow: (1) creation of prompts for piloting, (2) an iterative prompt pilot test on a single LLM with dual human/LLM annotation and a Cohen's kappa threshold, (3) evaluation among multiple LLMs using an oracle and metrics like accuracy and cost, and (4) output validation with automated checks for format, duplication, hallucination, and provenance tracking. The central statistical gate is the agreement coefficient: the prompt is only used on the full dataset once it passes a pre-specified threshold (in the authors' case, kappa above 0.9) on a representative sample.","core_discovery":"The core discovery is a practical, experience-derived framework—PRIMES—that packages the authors' lessons as a checklist for conducting repository mining studies with LLMs. The framework's central assertion is that reliable LLM-based data collection depends on an iterative loop: define objectives, craft a prompt, test it on a small sample against human annotation using an agreement coefficient (e.g., Cohen's kappa above 0.9), refine the prompt until the threshold is met, compare multiple LLMs on the same benchmark using an oracle, and finally validate outputs for formatting errors, duplications, and hallucinations while tracking provenance. The authors report that applying this process in their two studies led to high classification accuracy (95.58% and 97.91% for the two models compared) and enabled them to select the model best suited to the task.","pith_inferences":["Beyond repository mining, the same validate-then-scale loop could apply to any LLM-assisted annotation task in empirical research, such as classifying survey responses or issue reports, though the paper does not test this.","The specific threshold of Cohen's kappa above 0.9 is an experience-based default, not a statistically derived guarantee; testing how classification error on the full dataset relates to sample kappa would be a natural next step.","The framework's cost-comparison step implies a trade-off: using multiple LLMs improves reliability but increases cost, so a future extension could formalize when the added reliability outweighs the expense."],"forward_implications":["Following PRIMES should make LLM-based data collection more reproducible, because the prompt refinement and validation steps are documented rather than ad hoc.","Comparing multiple LLMs before committing to one should reduce model-specific bias and let researchers choose based on accuracy, explanation quality, and cost.","The agreement-threshold gate should prevent low-quality prompts from being applied to an entire dataset, which reduces the risk of large-scale annotation errors.","Incorporating PRIMES into reporting standards would make LLM mining studies easier to compare and to build upon."],"supporting_citations":[{"why":"The first experience study on green architectural tactics in ML projects; supplies the multi-LLM comparison and oracle construction.","marker":"[9]"},{"why":"The second experience study on ML model evolution; supplies the iterative prompt refinement and kappa validation procedure.","marker":"[10]"},{"why":"Defines Cohen's kappa, the agreement coefficient used as the validation gate.","marker":"[19]"},{"why":"Provides sampling guidelines for software engineering research, used to justify representative oracle construction.","marker":"[21]"},{"why":"An earlier LLM mining study that the authors contrast with PRIMES, showing prompt detail affects code smell detection.","marker":"[11]"},{"why":"An empirical comparison of prompt engineering and fine-tuning, used as related work on prompting strategies.","marker":"[12]"}],"fun_headline_variants":["PRIMES: the checklist for reliable LLM repo mining","Iterate and validate prompts with PRIMES for better LLM mining","PRIMES framework boosts LLM mining accuracy to 98%","A practical checklist for reproducible LLM repository mining"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that once the LLM agrees with human annotations on a representative sample (kappa above 0.9), the same prompt produces equally reliable outputs on the entire dataset, which requires the sample to be representative, the human oracle to be correct, and 0.9 to be a sufficient threshold.","fun_headline_variants_meta":{"raw":{"variants":["PRIMES: the checklist for reliable LLM repo mining","Iterate and validate prompts with PRIMES for better LLM mining","PRIMES framework boosts LLM mining accuracy to 98%","A practical checklist for reproducible LLM repository mining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1338,"prompt_tokens":911,"completion_tokens":427,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":368}},"tokens_in":527,"tokens_out":427,"duration_ms":4812,"temperature":1.0,"reasoning_tokens":368,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:05:03.575633+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the PRIMES validation loop on a new repository mining task, then apply the accepted prompt to the full dataset and have human experts annotate a second random sample. If the observed agreement on the second sample falls materially below the 0.9 threshold, the framework's assumption that sample agreement generalizes would be refuted.","supporting_citations":[{"cited_title":"How do machine learning models change?","cited_arxiv_id":null,"evidence_quote":"The second experience study on ML model evolution; supplies the iterative prompt refinement and kappa validation procedure."},{"cited_title":"Detecting code smells using chatgpt: Initial insights,","cited_arxiv_id":null,"evidence_quote":"An earlier LLM mining study that the authors contrast with PRIMES, showing prompt detail affects code smell detection."}],"review_version":1}