{"id":"fcf0da48-b7c7-4770-8519-ce271d1b23f9","arxiv_id":"2607.06471","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":5,"one_line_summary":"A large-scale mining study identifies and characterizes 2,502 verified DDD repositories on GitHub, revealing a 2017 adoption inflection point, C#/TypeScript language dominance, and sustained professional-grade engineering activity.","lead":"This paper mines and validates 2,502 GitHub repositories that genuinely implement Domain-Driven Design, using a GPT-4o pipeline to filter out false positives from keyword-based search. It finds that DDD adoption surged after 2017, projects are long-lived, and C# and TypeScript—not Java—dominate practical use.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The 50-repository validation sample may not represent the hardest classification cases; the LLM-vs-A2 agreement (κ=0.54) hints the pipeline's reliability is more fragile than the headline κ=0.77 suggests.","rationale":"The paper provides a useful first large-scale empirical baseline for DDD on GitHub, and the LLM-based validation pipeline is methodologically interesting. The triplicate majority-vote strategy and the internal consistency metrics (κ≥0.96 across runs) are reassuring for reproducibility of the LLM's own outputs. However, internal consistency measures self-agreement, not correctness; the external validity rests entirely on the 50-sample benchmark. The discrepancy between LLM-vs-A1 (κ=0.77) and LLM-vs-A2 (κ=0.54) is the most concrete signal that the pipeline's accuracy may be more assessor-dependent than the headline suggests. Combined with the unstratified validation sample and the two known structural asymmetries (language coverage and file cap), there is a real risk that the κ=0.77 point estimate does not generalize uniformly across the full candidate pool. This does not invalidate the study — the broad qualitative findings (DDD adoption accelerating after 2017, C#/TypeScript prominence, Layered/Clean Architecture dominance) are likely robust to moderate classification error. But the specific quantitative percentages (C# at 34.17%, TypeScript at 17.71%, Layered at 28.90%, etc.) should be interpreted with wider uncertainty than the paper conveys. The reader's CONDITIONAL verdict is appropriate; I would not change it, but I would sharpen the concern from 'small sample' to 'unstratified sample with known structural asymmetries and assessor-dependent agreement.'","tokens_in":24856,"tokens_out":3315,"duration_ms":255847,"concrete_test":"Draw a stratified random sample of 50 additional repositories from the 4,206 candidates, deliberately over-sampling (a) repositories whose primary language is NOT among the top 7 inspected extensions and (b) repositories with >300 filtered source files. Have both human assessors independently label them using the same protocol. Recompute LLM-vs-human kappa. If kappa drops below 0.61 (the 'substantial' threshold), the generalization of κ=0.77 to the full pool is not supported, and the downstream distributional findings (especially language percentages and architectural style breakdowns) should be treated as approximate rather than precise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader correctly identifies the 50-sample validation as the load-bearing assumption, but the concern is more specific than sample size alone. Two issues compound it. First, the paper reports two different LLM-vs-human agreement rates depending on which assessor is the reference: LLM vs A1 yields κ=0.77 (45/50), but LLM vs A2 yields only κ=0.54 (40/50) — 'moderate' rather than 'substantial' agreement (Table 3, Section 3.7.2). The paper treats A1 as the reference and headlines κ=0.77, but if A2's labels are closer to ground truth, the pipeline's reliability drops meaningfully. Second, the 50-sample validation is not reported as stratified by language, repository size, or discovery source. The pipeline has two known structural asymmetries: (a) source files were only inspected for 7 language extensions (.java, .cs, .ts, .js, .py, .php, .go), with the remaining ~4.7% of repos assessed from metadata alone (Table 2 note); and (b) for 16.7% of repositories exceeding the 300-file cap, only the first 300 paths in database storage order were visible to the model (Section 3.6), meaning architecturally relevant files in larger repos may have been systematically missed. If the 50-sample validation underrepresents these harder cases (non-top-7 languages and large repos), the κ=0.77 overestimates true performance on the full 4,206-pool. Since all downstream findings — language distribution, architectural style breakdowns, longevity statistics — depend on the 2,502-repository dataset being correctly classified, a systematic misclassification in these strata would skew the headline results.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"This paper presents a large-scale Mining Software Repositories (MSR) study characterising Domain-Driven Design (DDD) adoption on GitHub. The authors mine 11,742 candidate repositories using a hybrid topic- and README-keyword strategy, apply inclusion/exclusion criteria to obtain 4,206 candidates, and then use a GPT-4o-based agentic pipeline with triplicate majority voting to semantically validate 2,502 repositories as genuine DDD implementations. The pipeline is validated against a manually labelled sample of 50 repositories (reported κ = 0.77). The paper then characterises the verified dataset along six dimensions: temporal evolution, architectural styles, exemplary projects, ownership, technology/business ecosystems, and community engagement. Key findings include a 2017 inflection point in adoption, C# dominance (34%), Layered/Clean Architecture prevalence, and median project longevity of 340 days.","tokens_in":25129,"tokens_out":2038,"duration_ms":184710,"significance":"The paper addresses a genuine gap: prior DDD research is overwhelmingly theoretical, and no large-scale empirical characterisation exists. The use of an agentic LLM pipeline for semantic validation at scale is a methodologically interesting contribution that goes beyond keyword matching. The noise rate benchmarks per discovery query (Table 4) are a useful, transferable contribution for future MSR studies. The dataset and code are stated to be available, and the triplicate majority-vote design with internal consistency reporting (κ ≥ 0.96) is a reasonable approach to mitigating LLM non-determinism. The finding that C# and TypeScript, not Java, dominate practical DDD adoption is a concrete, falsifiable claim that challenges the academic status quo.","major_comments":[{"comment":"§3.7.2, Table 3: The paper headlines κ = 0.77 as the agreement between the LLM pipeline and human experts, but Table 3 reveals that this value holds only when Assessor 1 (A1) is the reference. Against Assessor 2 (A2), the LLM achieves only κ = 0.54 (moderate agreement), with F1 dropping from 92.5% to 85.3%. The paper does not justify why A1 is treated as the ground-truth reference rather than A2, or rather than a reconciled label. Since the two human raters themselves agree at only κ = 0.77 (with 5 disagreements on 50 cases), the choice of reference is not innocuous. All downstream findings — language distribution, architectural style breakdowns, longevity statistics — depend on the 2,502-repository dataset produced by this pipeline. The paper should either (a) justify the choice of A1 as reference with explicit criteria, or (b) report results against the reconciled ground-truth labels (","section":null},{"comment":"§3.7.2: The 50-repository validation sample is drawn randomly from the 4,206 candidate pool, but the paper does not report whether the sample was stratified by language, repository size, or discovery source. Two known structural asymmetries in the pipeline create risk that unstratified validation overestimates performance: (a) source files were inspected only for 7 language extensions (.java, .cs, .ts, .js, .py, .php, .go), with the remaining ~4.7% of repositories assessed from metadata and directory structure alone (Table 2 note); and (b) for 16.7% of repositories exceeding the 300-file path cap, only the first 300 paths in database storage order were visible to the model (§3.6). If the validation sample underrepresents these harder cases (non-top-7 languages and large repositories), the κ = 0.77 may not generalise. The paper should report the composition of the 50-sample by language, (","section":null},{"comment":"§3.6, §4.2: The architectural style classification is produced by the same GPT-4o pipeline and validated only indirectly. The paper reports a unanimous agreement rate of 88.05% for architectural labels across the three LLM runs (§3.7.1) but does not report LLM-vs-human agreement for the architectural style label specifically — only for the binary isDDD label. Since RQ2's central finding (Layered Architecture at 28.9%, Clean Architecture at 22.78%) is load-bearing for the paper's architectural characterisation, the absence of human validation for the multi-class architectural label is a gap. The paper should either add human-validated architectural labels for the 50-sample or explicitly downgrade the RQ2 claims to ","section":null}],"minor_comments":[{"comment":"§1, Introduction: The phrase 'first large-scale characterisation' is used multiple times. Consider softening to 'first large-scale MSR characterisation of DDD on GitHub' to be precise about scope.","section":null},{"comment":"§3.6: The temperature is fixed at 0 and random seed at 42. It would help to note whether the Azure OpenAI API guarantees deterministic output at temperature 0, as some providers do not.","section":null},{"comment":"Table 4: The 'Other specific topics' row aggregates three low-frequency queries. Consider listing them individually in a footnote for full reproducibility, as is partially done in the caption.","section":null},{"comment":"§4.1.3: The minimum longevity of -1,697.07 days is reported without explanation in the results section (it is explained later in §7.4). Consider adding a brief parenthetical note at first mention.","section":null},{"comment":"§4.5.2: The business domain classification uses a 'rule-based keyword classification' (§3.8) but the exact keyword-to-domain mapping is not provided. Consider including it as supplementary material.","section":null},{"comment":"§3.7.1: The Disagreement Rate is reported as 5.20%, but the Majority Agreement Rate is 99.95%. These should sum to 100% if they are complementary; clarify whether the 5.20% refers to the rate of 2-1 splits among all repositories or among the subset that did not reach unanimous agreement.","section":null},{"comment":"Figure 2: The caption mentions 'up to eight conversation turns' but the text does not specify what happens if the model does not reach a verdict within eight turns. Clarify.","section":null},{"comment":"§4.6.1: The comparison to automotive software medians (24 stars, 9 forks) cites [18, 19], but [19] is Cosentino et al., a systematic mapping study, not a landscape study reporting star/fork medians. Verify the citation.","section":null}],"recommendation":"major_revision","confidential_remarks":"The two-kappa issue (Table 3) is the most important concern. The paper presents κ = 0.77 as the headline validation metric in the abstract, highlights, and conclusion, but this is cherry-picked from two available values (0.77 vs A1, 0.54 vs A2). The inter-rater agreement between the two humans themselves is also only 0.77, which means the LLM's agreement with A1 is no better than A2's agreement with A1 — this is worth noting to the authors as it contextualises the pipeline's reliability. The paper is otherwise a solid MSR study with a useful methodological contribution; the issues are fixable with additional analysis and more careful framing of the validation results."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive review. All three major comments identify legitimate methodological gaps that we will address in the revision. Specifically: (1) we will report results against reconciled ground-truth labels rather than a single assessor, and justify the reference choice; (2) we will report the composition of the 50-repository validation sample by language, repository size, and discovery source, and discuss stratification implications; (3) we will add human-validated architectural labels for the 50-sample and, where LLM-vs-human agreement for the multi-class architectural label is insufficient, explicitly downgrade RQ2 claims to indicative rather than definitive.","responses":[{"response":"The referee is correct. The choice of A1 as the sole reference was not justified, and the discrepancy between κ=0.77 (vs. A1) and κ=0.54 (vs. A2) is material. We will revise the paper to report LLM performance against the reconciled ground-truth labels (the 50 cases resolved through joint discussion between A1 and A2) as the primary reference, rather than a single assessor. We will retain the per-assessor breakdown in Table 3 for transparency. This is a straightforward revision: the reconciled labels already exist in our data, and we will recompute agreement, precision, recall, and F1 against them. We expect the reconciled-reference metrics to fall between the two assessor-specific values, and we will report them honestly regardless of where they land. We will also add an explicit sentence acknowledging that the two human raters themselves agree at only κ=0.77, which bounds the ceiling of any LLM-vs-human validation on this sample.","revision_made":"yes","referee_comment":"§3.7.2, Table 3: κ=0.77 holds only against Assessor 1; against Assessor 2, κ=0.54. The paper does not justify why A1 is the ground-truth reference. Should either justify A1 or report against reconciled labels."},{"response":"The referee raises a valid concern. We did not stratify the 50-repository validation sample, and we did not report its composition. We will address this in two ways. First, we will report the composition of the 50-sample by primary language, repository size (file count), and discovery source (topic-based vs. README-based), so readers can assess representativeness. Second, we will add an explicit discussion in Section 7 (Threats to Validity) acknowledging that if the sample underrepresents the harder cases — non-top-7-language repositories (assessed from metadata and directory structure alone) and large repositories exceeding the 300-file path cap — the reported κ may overestimate pipeline performance on those subpopulations. We cannot retroactively stratify the existing sample without relabelling a new stratified set, which would require additional manual annotation beyond the scope of this revision. We will therefore be transparent about the limitation and frame the κ=0.77 as a best-case estimate for the harder cases, while noting that the top-7 languages cover 95.29% of the candidate pool, limiting the practical impact of this concern for the majority of the dataset.","revision_made":"partial","referee_comment":"§3.7.2: The 50-repository validation sample was drawn randomly without reported stratification by language, size, or discovery source. Known structural asymmetries (non-top-7 languages assessed from metadata alone; 16.7% of repositories exceeding the 300-file cap) create risk that unstratified validation overestimates performance."},{"response":"This is a fair and important point. The binary isDDD label was validated against human judgement, but the multi-class architectural style label was not. We will add human-validated architectural labels for the 50-repository sample. Both assessors already inspected these repositories for the binary DDD judgement; we will extend the existing labelling to include the dominant architectural style for each repository, compute LLM-vs-human agreement for the multi-class label, and report the results in a revised Section 3.7.2. If the agreement is substantially lower than for the binary label — which is plausible given the greater subjectivity of multi-class architectural classification — we will explicitly downgrade the RQ2 claims from definitive distributional findings to indicative of dominant structural tendencies, consistent with the hedging language already present in Section 3.7.1. We note that the manuscript already states: 'We therefore treat the resulting distribution as indicative of dominant structural tendencies within the ecosystem rather than definitive architectural ground truth' (Section 3.7.1). We will strengthen this framing in the RQ2 results section itself, not only in the reliability discussion.","revision_made":"yes","referee_comment":"§3.6, §4.2: Architectural style classification is produced by the same GPT-4o pipeline and validated only indirectly (88.05% unanimous agreement across LLM runs). No LLM-vs-human agreement is reported for the multi-class architectural label, only for the binary isDDD label. RQ2 findings (Layered 28.9%, Clean 22.78%) are load-bearing and lack human validation."}],"tokens_in":24911,"tokens_out":1109,"duration_ms":178517,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"Short version: this is the first large-scale empirical characterization of DDD on GitHub, and the agentic GPT-4o validation pipeline is a genuine methodological contribution to MSR landscape studies. The central soft spot is that the 50-repository human validation sample is small and possibly unrepresentative of the harder classification cases, and the paper headlines the more favorable kappa (0.77 vs. A1) while burying the less favorable one (0.54 vs. A2). I think it deserves a serious referee who pushes on validation rigor, but the core contribution is real and useful regardless of whether the exact quantitative breakdowns survive scrutiny.","headline":"First large-scale empirical DDD landscape on GitHub; LLM validation pipeline is the methodological novelty, but the 50-sample human validation is the load-bearing weak point.","tokens_in":25897,"tokens_out":204,"would_cite":false,"duration_ms":96158,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"2,502 GitHub repos reveal DDD as mature practice, not theory","keywords":[],"falsifier":"If the GPT-4o pipeline has systematic language-specific or size-specific biases not captured in the 50-repository benchmark, the composition of the 2,502-repository dataset could be skewed, undermining downstream claims about language distribution, architectural styles, and longevity.","tokens_in":24975,"feed_emoji":"🏗️","tokens_out":836,"duration_ms":124670,"temperature":0.7,"pith_summary":"This paper argues that Domain-Driven Design (DDD), a software architecture paradigm introduced in 2004 to align code structure with business domain logic, has crossed the threshold from academic concept to stable industrial practice in the open-source world. The authors mined 11,742 candidate GitHub repositories using keyword and topic searches, then built an automated GPT-4o pipeline that inspected source code and directory structures to verify which projects genuinely implement DDD patterns, yielding 2,502 confirmed repositories. The central empirical claims are that DDD adoption accelerated sharply after a 2017 inflection point, that verified projects live far longer than typical GitHub repositories (median 340 days versus 9.9 days), that C# and TypeScript rather than Java dominate implementation, and that Layered and Clean Architecture are the prevailing structural patterns while CQRS and Event Sourcing recur in distributed, data-intensive systems. The paper also reports that 25.3% of verified projects record no explicit business domain in their metadata, which the authors interpret as a persistent gap between architectural intent and what version control systems capture.","feed_headline":"DDD grows up: 2,502 verified repos show mature engineering practice","feed_subtitle":"C# and TypeScript lead adoption, projects live 34x longer than typical GitHub repos, and a 2017 inflection point marks the shift from theory","key_machinery":"GPT-4o agentic semantic validation pipeline with triplicate majority-vote (kappa=0.77 vs. human experts; 94.75% unanimous agreement across runs)","core_discovery":"The paper's central object is the 2,502-repository validated dataset, produced by a triplicate GPT-4o semantic validation pipeline that achieved Cohen's kappa = 0.77 against two human expert raters on a 50-repository benchmark. Using this dataset, the authors establish that DDD in open source is characterized by sustained engineering maturity (long-lived projects, organizational ownership at 23%, 11.15 average contributors), a 2017 inflection point in adoption, C# dominance at 34.17% challenging the Java-centric academic assumption, and a concentration of Layered (28.9%) and Clean (22.78%) architectural styles. The pipeline itself is presented as a transferable methodological contribution: a","pith_inferences":[],"forward_implications":["Researchers studying software evolution, technical debt, or architectural decay now have a curated set of 2,502 verified DDD repositories as a benchmark corpus of engineered, long-lived projects.","The 78.7% overall noise rate in keyword-based repository discovery provides a concrete calibration point for future domain-specific mining studies, especially for ambiguous abbreviations like 'ddd'.","The dominance of C# and TypeScript over Java in verified DDD repositories suggests academic curricula and textbooks may be misaligned with industrial adoption patterns.","The 25.3% of projects with no identifiable business domain in metadata exposes a structural limitation of Git-based version control for capturing architectural intent, motivating lightweight traceability standards."],"fun_headline_variants":["2,502 verified repos reveal DDD's shift to mature engineering","C# and TypeScript lead DDD adoption across 2,502 GitHub repos","GPT-4o validates 2,502 DDD repos, revealing long-lived projects","Post-2017 DDD projects live 34x longer than typical GitHub repos","DDD in practice: 2,502 verified repos show C# and TS dominance"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The GPT-4o pipeline's agreement with human experts, measured on a 50-repository sample, is assumed to hold across the full 4,206-repository candidate pool spanning all languages, sizes, and architectural styles, even though source-file inspection was limited to seven language extensions covering 95.29% of repositories and the validation sample is small relative to the candidate set.","fun_headline_variants_meta":{"raw":{"variants":["2,502 verified repos reveal DDD's shift to mature engineering","C# and TypeScript lead DDD adoption across 2,502 GitHub repos","GPT-4o validates 2,502 DDD repos, revealing long-lived projects","Post-2017 DDD projects live 34x longer than typical GitHub repos","DDD in practice: 2,502 verified repos show C# and TS dominance","Large-scale empirical study validates 2,502 DDD repositories","DDD adoption matures: 2,502 verified repos outlive typical GitHub projects","Open-source DDD projects show sustained engineering maturity","Semantic validation of 2,502 DDD repos challenges Java-centric assumptions"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":1462,"prompt_tokens":684,"completion_tokens":778,"prompt_tokens_details":null},"tokens_in":684,"tokens_out":778,"duration_ms":29177,"temperature":1.0,"reasoning_tokens":591,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T04:34:21.035136+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If the GPT-4o pipeline has systematic language-specific or size-specific biases not captured in the 50-repository benchmark, the composition of the 2,502-repository dataset could be skewed, undermining downstream claims about language distribution, architectural styles, and longevity.","supporting_citations":[],"review_version":1}