{"id":"6e59d68f-9c93-4ed6-9b96-0b1246f81747","arxiv_id":"2604.00787","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Across 85 papers, language models in MSR mostly perform classification and generation on issues, code reviews, and commits, with the field shifting from fine-tuned BERT-size models to large instruction-tuned LLMs used via prompting.","lead":"This survey maps 85 studies that apply language models to mine data from software repositories, grouping their tasks into eight categories such as classification, generation, and extraction. It is a practical reference for tracking how AI is reshaping software-engineering research, including which models dominate and how often studies share reusable tools.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Corpus representativeness is the load-bearing risk: all survey percentages are counts over an 85-paper corpus anchored only on MSR conference proceedings, with no recall validation.","rationale":"The reader's weakest_assumption identifies corpus representativeness as the load-bearing point. I agree. The survey is a systematic, carefully executed review, and the internal mechanical issues (abstract/body paper count discrepancy, the R²=0.9933 interpretation, the joke citation) are real but peripheral. None of them invalidates the qualitative portrait; they are data-hygiene flaws. The central scientific claim is that eight application types and their relative frequencies accurately characterize LM-based MSR research. Every quantitative finding is a summary statistic over the 85 selected papers. If the selection is skewed, the percentages and trends are skewed with it. The protocol (§3.4) offers a clear rationale for anchoring at MSR, but it does not establish representativeness: there is no recall check, no forward snowballing, and no triangulation with a full-text database search. The authors explicitly concede the limitation, which is honest, but that does not reduce its severity. The proposed concrete test—an independent database search with inclusion/exclusion applied—would directly assess recall and distributional stability. If the test shows the non-captured studies are similar in application mix, the concern is resolved. If not, the survey's conclusions should be explicitly conditional on the MSR-conference-centered corpus. Since the reader already rated the paper CONDITIONAL and identified this same risk, my critique does not change their verdict; it strengthens it. No significant additional concern outweighs this one, and the non-finding option is not appropriate because the representativeness assumption is genuinely load-bearing and unvalidated.","tokens_in":31722,"tokens_out":4655,"duration_ms":54188,"concrete_test":"Run an independent validation search in Scopus or DBLP for 2017–2025 using a broad query combining LM terms (e.g., 'language model', 'large language model', 'BERT', 'GPT') with MSR terms (e.g., 'mining software repository', 'repository mining', 'commit message', 'issue report', 'code review'). Apply the survey's inclusion/exclusion criteria to the retrieved set. Compute: (1) recall—the fraction of the 85 primary studies that appear in the retrieved set; (2) the application-type distribution among the non-overlapping studies that pass inclusion. If recall is below ~80%, or if any application-type share in the non-overlapping set differs from the reported share (e.g., classification 32.8%, generation 22.9%) by more than 5 percentage points, the survey's percentages are corpus-dependent and the central characterization must be revised or explicitly scoped to the MSR-conference-centric corpu","verdict_should_be":"UNCHANGED","load_bearing_attack":"The survey's central claim is a quantitative characterization of LM use in MSR: the eight-category distribution (classification 32.8%, generation 22.9%, ...), the model-family shift from BERT to GPT, and the 24.7% proprietary-only reliance. Every one of these percentages is an arithmetic mean over the 85 primary studies. The corpus was built by screening MSR conference proceedings (2017–2025) and then backward snowballing only the references of those papers (§3.4). This protocol inherits the selection biases of one conference: it overweights MSR-venue work (45% of the corpus), short papers, and studies that happen to cite or be cited by the seed set. It systematically undersamples journal articles (12% of the corpus), industry-led work, and any LM-for-MSR research published in venues not referenced by the seed papers. The authors acknowledge the limitation ('may not have captured all relevant studies', §3.4) but provide no validation: no recall estimate, no comparison against a full bibliographic search, no sensitivity analysis. If the non-captured studies have a different mix of application types or model families, the headline percentages and the 'clear transition' narrative (RQ3) shift. Thus the load-bearing condition is that the MSR-anchored corpus is representative of the broader field; this is asserted, not demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a systematic literature review of language-model (LM) use in mining software repositories (MSR). Based on 85 primary studies collected from MSR conference proceedings (2017–2025) plus backward snowballing, it proposes an eight-type taxonomy of LM applications and answers four research questions concerning applications, data sources/artifacts, model families and temporal trends, and reproducibility/reuse. Headline findings include the distribution of application types (classification 32.8%, generation 22.9%, extraction 15.3%, detection 13%, assessment 5.3%, enhancement 4.6%, retrieval 3.8%, prediction 2.3%), a claimed transition from BERT-style encoder-only models to larger instruction-tuned decoder models, and the observation that reproducibility is only partially supported (24.7% exclusive proprietary-model reliance; 21 tools identified). The protocol is transparent: pre-declared inclusion/exclusion criteria, two-phase screening with reported inter-rater agreement, backward snowballing, and author-confirmed extraction.","tokens_in":31949,"tokens_out":7047,"duration_ms":75672,"significance":"If the findings hold, this is a timely and useful map of an emerging research area. The contribution is primarily empirical: it provides a reproducible corpus, an eight-category taxonomy, and a set of trends that future LM4MSR research can build on. The author-confirmed extraction and the high inter-rater agreement are notable methodological strengths. However, the quantitative claims are only as strong as the corpus, and the current text contains internal inconsistencies in the reported study counts. The survey would be a solid reference if these issues are resolved.","major_comments":[{"comment":"The abstract states that the survey is based on an analysis of 177 papers, whereas the full text and all analyses use 85 primary studies (774 initial papers → 100 after Phase 1 → 47 after Phase 2 + 38 snowballed = 85). This is not a trivial typo: every percentage in the paper is computed over the 85-paper corpus. The 177 figure is unexplained and must be corrected or justified.","section":"Abstract vs. §3.4"},{"comment":"The search protocol says Phase 2 yielded 47 papers from MSR proceedings and backward snowballing added 38, for a total of 85. However, §3.6.4 and Table 1 report only 38 papers from the MSR conference, described as 45% of all publications. Since 47 of the 85 primary studies must have come from the MSR proceedings under the stated protocol, this is internally inconsistent. The authors should reconcile these counts or explain any later recategorization.","section":"§3.4, §3.6.4, Table 1"},{"comment":"Corpus representativeness is asserted but not validated. The search was anchored on MSR conference proceedings, with snowballing only from the references of selected papers, and the authors acknowledge this 'may not have captured all relevant studies.' Yet the headline percentages for application types, model families, and temporal trends are all computed over this corpus. No recall estimate, full multi-database search, or sensitivity analysis is provided. Given that MSR-venue papers account for 45% of the corpus and journal papers only 12%, a systematic under-sampling of non-MSR venues, industry studies, or arXiv-first work could materially shift the reported distribution. I ask for a validation step, such as a comparison against a full IEEE/ACM/Springer search or an independent query, before the quantitative characterization is presented as representative.","section":"§3.4 and answers to RQ1–RQ3"}],"minor_comments":[{"comment":"Reference [1] gives the year 2026 for the first International Workshop on Mining Software Repositories, but the workshop occurred in 2004 (as the URL indicates). Please correct the year.","section":"References"},{"comment":"The quadratic fit to cumulative publication counts yields R² = 0.9933, but cumulative counts almost always follow a smooth curve; this is weak evidence of 'polynomial growth.' Consider reporting annual counts or residual diagnostics instead.","section":"§3.6.1, Figure 2b"},{"comment":"The text reports study-level counts (45 main, 30 enabling, 10 both) and then application-level counts (85 main, 46 enabling) without explicitly distinguishing the two. Please clarify so readers do not interpret these as contradictory.","section":"§4, Answer to RQ1"},{"comment":"The companion website [135] is cited but its contents are not described. Adding a sentence about what is available there would help readers verify the primary-study list and extracted data.","section":"§3.4"}],"recommendation":"major_revision","confidential_remarks":"The core material is useful and the protocol is transparent, but the paper currently contains two concrete count inconsistencies (177 vs. 85 in the abstract; 47 vs. 38 MSR-conference papers) and a load-bearing representativeness claim that is not validated. These are fixable within the scope of a revision: correct the counts and add a recall/sensitivity analysis against a broader search. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a capable, genuinely useful survey of LM use in MSR. It is not a paper dump; the eight-type taxonomy with the main-vs-enabling distinction, and the documented shift from BERT-era encoders to GPT-era instruction-tuned models, are real contributions. The protocol is above the usual bar for SE surveys: pre-declared criteria, two-phase screening with inter-rater kappa around 0.95, author-confirmed extraction. If you work in MSR or on LMs for repository mining, you will want this on your desk.\n\nThe soft spots are real but not disqualifying. The abstract says 177 papers while the body consistently says 85. That is a glaring inconsistency and it undercuts confidence in the data hygiene. The citation for \"GitHub exceeds 1B repositories\" points to a joke issue in a repo called 'shit'; that should be replaced. And the R² = 0.9933 fit to cumulative publication counts is presented as \"strong polynomial growth,\" but almost any cumulative curve fits a quadratic well, so it is weak evidence.\n\nThe substantive concern is corpus representativeness. Searching MSR conference proceedings plus backward snowballing is a reasonable pragmatic strategy, and the authors acknowledge it may miss studies. But they do not validate it: no recall estimate, no comparison against a full database search, no sensitivity analysis. Every headline percentage is a statistic over those 85 papers. If the missing studies have a different mix of application types or model families, the numbers shift. That said, the qualitative direction — LMs diversify after 2023, encoder models still dominate small-scale use, tool maintenance is poor — is robust to most plausible selection bias. I would treat the taxonomy and trends as trustworthy at the qualitative level, and the percentages as estimates with an unknown error bar.\n\nWho this is for: new researchers entering the area, experienced people writing related-work sections, and anyone needing a baseline on reproducibility practices. It deserves serious peer review — the gap is real and the method is careful. I would send it out, but require fixing the abstract/body mismatch, the citation, and ideally adding a validation step or softening the quantitative claims.","headline":"A real contribution to the MSR-survey literature, with a few data-hygiene problems that need fixing before the numbers are trusted.","tokens_in":32540,"tokens_out":2724,"would_cite":true,"duration_ms":31418,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A survey of 85 studies claims that language-model use in mining software repositories collapses into eight application types and that the field is shifting from fine-tuned BERT-style models to prompted GPT-style models.","keywords":["language models","mining software repositories","systematic survey","taxonomy of applications","large language models","reproducibility","model evolution","software engineering"],"falsifier":"Count the primary studies listed on the survey's companion website: if the list contains 177 entries rather than 85, the abstract/body mismatch is resolved against the body; more decisively, a full-text search of the same digital libraries using LM-plus-repository-mining keywords, without any venue anchor, would produce a candidate set whose application-type distribution and model timeline could be compared to the survey's—a materially different distribution would falsify the survey's characterization.","tokens_in":31524,"feed_emoji":"🤖","tokens_out":6581,"duration_ms":71313,"temperature":0.7,"pith_summary":"This paper is a systematic survey of the young, fast-growing research area that applies language models to mining software repositories—analyzing commits, issues, code reviews, and other traces of software development. The authors' central claim is that all of this work can be organized into a taxonomy of eight application types: classification, generation, extraction, detection, assessment, enhancement, retrieval, and prediction. They also claim a clear historical transition: before 2023 the field ran almost entirely on small, openly available encoder models (BERT and relatives) fine-tuned for specific tasks; by 2025, large, instruction-tuned generative models used through prompting had become the most common choice. Along the way they quantify the field's data habits—GitHub and a handful of reused datasets dominate—and its reproducibility problems: most studies share supplementary materials, but more than a fifth depend exclusively on closed, proprietary models and few release maintained tools. A reader should care because the survey offers the first broad map of this area and a set of concrete challenges for making the work more reproducible and more useful.","feed_headline":"Eight task types cover language-model use in repo mining","feed_subtitle":"Classification leads the field, small fine-tuned models are giving way to large prompted ones, and reproducibility lags.","key_machinery":"The organizing device is an eight-category taxonomy of LM applications—classification, generation, extraction, detection, assessment, enhancement, retrieval, prediction—derived deductively from the primary studies and applied twice, once for main contributions and once for enabling contributions. The second load-bearing piece is the LM usage-mode distinction (fine-tuning vs. prompting-as-is) combined with model-family, parameter-size, and weight-availability categorizations; this is what lets the survey turn a list of 94 models into a trend narrative. The third piece is a small set of data-source and artifact categories (issue reports, code reviews, source code, and so on) used to compare wh","core_discovery":"The survey's discovery is that LM-based MSR is not a scattered set of point solutions but a field with a stable task geometry. Across 85 primary studies, the authors identify 131 instances of LM use, which reduce to eight application types with classification (32.8%) and generation (22.9%) at the top. LMs play two roles: as main contributions that directly produce artifacts or insights, and as enabling components that convert noisy repository traces into structured inputs for downstream analysis—extraction and detection dominate the enabling role. On models, 235 uses of 94 distinct models break down into a clear temporal pattern: BERT-family encoders under half a billion parameters, usually","pith_inferences":["One consequence the authors leave implicit is that if the taxonomy becomes a shared vocabulary, future work gets a stable baseline for measuring the field's evolution; repeating the survey in three to five years with the same categories would be a natural test.","An extension of the cost finding: requiring token-level and monetary cost reporting for any study that uses API-based models would directly address the 6-in-37 reporting gap the survey documents.","The survey's observation that requirements and other non-code artifacts are rarely mined suggests, as an editorial extrapolation, that new datasets for those artifact types may be disproportionately valuable."],"forward_implications":["Classification and generation are the field's two pillars, together accounting for over half of all LM uses; anyone planning an LM-based MSR contribution can expect the community to compare against these task types.","Small open encoder models are still the workhorses of the field, but the momentum is with large prompted models; evaluation and reproducibility expectations should be adjusted accordingly.","Dataset reuse is concentrated in a handful of resources; progress will amplify the importance of those datasets' quality.","Because 24.7% of studies depend exclusively on proprietary models and costs are rarely reported, many published results cannot be re-run by independent researchers.","The field's reproducibility is stronger for raw materials than for tools: 74 of 85 studies share materials, but only a minority of the 21 proposed tools are maintained."],"fun_headline_variants":["Eight task types map language-model use in repo mining","Survey: 8 LM tasks in software repo mining","Classification and generation lead LM repo-mining tasks","Large prompted models overtake small fine-tuned in repo mining","Reproducibility lags in LM-based repo mining, survey finds"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the 85 primary studies—found by anchoring a search to the field's flagship conference proceedings and then snowballing—are representative enough of the whole field to support the survey's percentages and trend lines; the paper's own abstract states 177 papers while the body reports 85, so the corpus itself is not pinned down.","fun_headline_variants_meta":{"raw":{"variants":["Eight task types map language-model use in repo mining","Survey: 8 LM tasks in software repo mining","Classification and generation lead LM repo-mining tasks","Large prompted models overtake small fine-tuned in repo mining","Reproducibility lags in LM-based repo mining, survey finds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000889,"raw_usage":{"total_tokens":3638,"prompt_tokens":678,"completion_tokens":2960,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":422,"completion_tokens_details":{"reasoning_tokens":2879}},"tokens_in":422,"tokens_out":2960,"duration_ms":18320,"temperature":1.0,"reasoning_tokens":2879,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T05:33:41.060519+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Count the primary studies listed on the survey's companion website: if the list contains 177 entries rather than 85, the abstract/body mismatch is resolved against the body; more decisively, a full-text search of the same digital libraries using LM-plus-repository-mining keywords, without any venue anchor, would produce a candidate set whose application-type distribution and model timeline could be compared to the survey's—a materially different distribution would falsify the survey's characterization.","supporting_citations":[],"review_version":1}