{"id":"62516c2c-a7e4-4a30-9eb9-4eab1a7f616e","arxiv_id":"2607.06505","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":8,"one_line_summary":"A retrieve-and-verify pipeline (BGE embeddings + GPT-4.1 rubric scoring) produces 6,588 NAICS-labeled GitHub repositories with 96.98% human-validated precision, plus a RoBERTa-large benchmark classifier at 86.45% F1.","lead":"This paper releases NAICS-GH, a dataset of 6,588 GitHub repositories labeled with 2-digit NAICS industry codes using a BGE+FAISS retrieval and GPT-4.1 verification pipeline. It enables empirical study of open-source production across economic sectors and provides a baseline classifier for downstream use.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Headline 96.98% precision is validated on only the USA subset (2,421 of 6,588 rows); the remaining ~62% of the released corpus (EU + AU) has zero human validation, and structural factors (NAICS taxonomy mismatch, English-only embeddings on potentially non-English EU repos) give reason to expect prec","rationale":"The reader correctly identified the most load-bearing concern: the headline precision is validated on a USA-only gold sample while the released corpus includes EU and AU labels with no human validation. This is the right concern for the right reason — it goes to the heart of whether the central claim (96.98% precision on the released corpus) actually holds for the full corpus.\n\nThe paper is notably transparent about this limitation (§5.6, §7, Appendix H.2), which is why CONDITIONAL rather than REJECT is appropriate. The dataset release, pipeline reproducibility (±0.03% candidate set replication), public code, and fine-tuned checkpoint are all genuine contributions. The precision validation on the USA subset is thorough, with per-sector breakdowns, score-conditional analysis, and Wilson confidence intervals.\n\nThe secondary concern about single-annotator protocol without inter-annotator agreement is real but less load-bearing: the discriminating per-sector precision pattern (73% for Manufacturing/Wholesale vs. 100% for six other sectors) suggests annotators are applying the rubric meaningfully rather than rubber-stamping. A future double-labeling pass on ~200 rows (as the paper proposes in §9) would address this.\n\nThe benchmark results (RoBERTa-large at 86.45% F1) are somewhat circular — the test set ground truth is GPT-4.1 labels, not human labels — but this is standard practice in weakly supervised learning and the paper does not claim the F1 is against human ground truth. The 96.98% precision validation provides an external check that the GPT labels are largely correct, which partially mitigates the circularity.\n\nThe empty Appendix G (qualitative error analysis) and empty §4.5 (score distribution) are minor structural gaps that don't affect the central claim.\n\nOverall, the reader's verdict of CONDITIONAL with HIGH confidence is well-calibrated. The concern is real, the paper acknowledges it, and the dataset is still a valuable contribution with appropriate caveats.","tokens_in":21772,"tokens_out":2664,"duration_ms":116516,"concrete_test":"Draw a stratified random sample of ~300 repositories from the EU and AU portions of the released corpus (proportional to their sector distributions), and have human annotators validate them using the same protocol and rubric as the USA gold set. Compute precision overall and per-sector. If EU/AU precision falls within ~3 percentage points of 96.98%, the headline claim generalizes and the concern does not land. If precision drops below ~93% overall, or if specific sectors (especially Utilities, Manufacturing, Wholesale Trade) show substantially lower precision than in the USA sample, the headline claim should be qualified as USA-only and the corpus-level precision should be reported as a weighted estimate with appropriate uncertainty.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that NAICS-GH achieves 96.98% precision (Wilson 95% CI [96.23, 97.59]) on a 2,421-repository human-validated random sample. However, Appendix H.2 states this sample is drawn entirely from the USA portion of the pipeline output. The released corpus contains 6,588 rows: 2,529 USA, 2,039 EU, and 2,021 AU (Table 1, Step 4). This means approximately 62% of the released labels — all EU and AU rows — have no human validation whatsoever. The paper acknowledges this in §5.6 and §7, but the abstract and headline results present 96.98% as the precision of the corpus without qualifying the geographic scope of validation.\n\nThere are concrete reasons to suspect EU/AU precision may differ from USA precision: (1) NAICS is a North American taxonomy, and the paper itself notes in §7 that sector definitions can diverge across jurisdictions (e.g., Sector 22 Utilities is structured around the US regulated-utility model). (2) BGE-large-en is an English embedding model; EU repositories may have non-English READMEs that are under-represented in retrieval, creating a different candidate distribution. (3) The GPT-4.1 rubric prompt (Appendix C) uses US-centric examples (e.g., 'Farm management software for Agriculture'), which could bias scoring differently for EU/AU repositories whose industry context doesn't map cleanly to NAICS.\n\nThe single-annotator protocol (no inter-annotator agreement, §5.1, §5.6) is a secondary concern: the 96.98% figure with per-sector variation (73% for Manufacturing and Wholesale Trade vs. 100% for six sectors) suggests annotators are genuinely discriminating rather than rubber-stamping, so systematic annotator bias seems less likely than geographic non-transferability. The paper is commendably transparent about both limitations, but the gap between the validated subset and the full released corpus is the single most load-bearing weakness in the headline precision claim.","agreement_with_reader":"agree"},"referee_report":{"model":"glm-5.2","summary":"This paper presents NAICS-GH, a corpus of 6,588 GitHub repositories from the USA, EU, and Australia labeled with 2-digit NAICS sector codes. Labels are produced by a two-stage retrieve-and-verify pipeline: BGE-large-en embeddings + FAISS retrieval narrows ~1.37M source repositories to 31,178 candidates, and GPT-4.1 rubric scoring (score >= 8) retains 6,588 high-confidence labels. The authors report 96.98% precision (Wilson 95% CI [96.23, 97.59]) on a 2,421-repository human-validated sample, and benchmark six pretrained encoders, with RoBERTa-large achieving 86.45% F1 on a held-out test set. The dataset, pipeline code, prompts, and fine-tuned checkpoint are publicly released.","tokens_in":22089,"tokens_out":1496,"duration_ms":361736,"significance":"The paper addresses a genuine gap: no publicly available corpus maps GitHub repositories to a standardized economic industry taxonomy. The pipeline is well-documented and reproducible, with an end-to-end re-run confirming candidate-set replication to within 0.03%. The authors release the dataset (CC-BY-4.0), pipeline code (MIT), prompts, and a fine-tuned RoBERTa-large checkpoint. The validation methodology uses Wilson confidence intervals with per-sector breakdowns, and the score-conditional precision analysis (Table 7) provides a usable confidence signal for downstream practitioners. The benchmark comparison across six encoders with identical hyperparameters is a fair and useful contribution.","major_comments":[{"comment":"§5.1 and Appendix H.2: The 2,421-row gold validation sample is drawn entirely from the USA portion of the pipeline output (Appendix H.2: 'Human re-validation of 2,421 USA repositories'). The released corpus contains 6,588 rows: 2,529 USA, 2,039 EU, and 2,021 AU (Table 1). This means approximately 62% of the released labels have no human validation. The abstract and headline results present 96.98% precision as the corpus-level figure without qualifying the geographic scope of validation. This is load-bearing because the central claim is about the precision of the released corpus, not just the USA subset. The paper acknowledges in §7 that NAICS is a North American taxonomy and that sector definitions can diverge across jurisdictions (e.g., Sector 22 Utilities), and that BGE-large-en is English-only while EU repositories may have non-English READMEs. These structural factors give concrete理由","section":null},{"comment":"to expect EU/AU precision may differ from USA precision. The authors should either (a) validate a sample of EU/AU rows and report stratified precision, or (b) explicitly reframe the headline precision as 'USA-validated precision' in the abstract and §5.2, with a clear statement that EU/AU precision is unvalidated. Option (b) is achievable within revision.","section":null},{"comment":"§5.1: The gold set was reviewed by single annotators with no inter-annotator agreement metric. The authors acknowledge this in §5.6 and propose a future double-labeling pass on ~200 rows. Given that the 96.98% precision figure is the paper's central validation claim, the absence of any agreement metric is a load-bearing gap. Even a small double-labeled subsample (the proposed 200 rows) with a Cohen's kappa would substantially strengthen the claim. If this cannot be completed before revision, the authors should at minimum add a sensitivity analysis: how low would annotator agreement need to be for the headline precision to shift meaningfully, given the observed error rate of 3.02%?","section":null}],"minor_comments":[{"comment":"Table 3: The sector name for code 72 is listed as 'Accomodation and Food Services' — the correct spelling is 'Accommodation.' The same misspelling appears in Table 6 and Table 9.","section":null},{"comment":"§3.3, effective_k formula: The piecewise definition reads 'effective_k = {20 if n > 20; max(20, ceil(400/n)) if n <= 20}'. For n > 20, the formula gives k = 20, but the text says 'For the 17 sectors with n > 20 this gives a uniform k = 20.' This is correct but the formula notation is slightly confusing since the second branch also applies max(20, ...) which is always >= 20. Consider clarifying that the second branch only activates for n <= 20.","section":null},{"comment":"Appendix C, prompt: The rubric examples are US-centric (e.g., 'Farm management software for Agriculture'). While this is understandable given NAICS is a North American taxonomy, the authors might note whether EU/AU-specific examples were considered or whether the US-centric framing could interact with the geographic validation gap noted above.","section":null},{"comment":"§4.4: The 113 duplicate (name_repo, code) pairs are described as distinct repositories sharing short names. The paper notes that upstream intermediate files retain nwo as a globally unique identifier. Consider noting whether these intermediate files are available to downstream users, or whether the name ambiguity is an accepted limitation of the public release.","section":null},{"comment":"Table 8: The finding that 'encoder family matters more than parameter count' is interesting but the differences between RoBERTa-large (86.45%), DeBERTa-v3-base (85.68%), and DeBERTa-v3-large (85.07%) are within ~1.4pp. Without confidence intervals or multiple seeds, it is unclear whether these differences are statistically significant. A note acknowledging this would be appropriate.","section":null},{"comment":"§3.1: The USA source dump was extracted April 15, 2025, while the EU README extraction is dated August 15, 2025. This four-month gap in source data freshness is not discussed. If repository content changed between these dates, the EU and USA pools are not temporally comparable. A brief note acknowledging this would be useful.","section":null},{"comment":"Appendix G is referenced in §5.5 as containing qualitative error analysis of the 73 incorrect labels, but the appendix appears to be empty (only the header 'G Qualitative error analysis' is present with no content). This should be populated or the reference removed.","section":null}],"recommendation":"major_revision","confidential_remarks":"The geographic validation gap is the primary concern. The paper is otherwise solid and well-documented, with genuine reproducibility artifacts. If the authors reframe the headline precision as USA-validated and add even a small inter-annotator agreement check, this should be publishable. The empty Appendix G suggests the manuscript may have been submitted before final completion; the authors should be asked to verify all appendices are complete."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive review. Both major comments identify genuine gaps in our validation scope that we will address in the revised manuscript. Below we respond point by point.","responses":[{"response":"The referee is correct on the facts. The gold sample is drawn exclusively from the USA portion of the pipeline output (Appendix H.2), and the abstract presents 96.98% precision without qualifying this geographic scope. We agree this is a load-bearing gap. We will adopt option (b) in the revision: the abstract, §5.1, §5.2, and the Datasheet (Appendix H.2) will be revised to explicitly state that the 96.98% precision figure reflects USA-validated precision only, and that EU and AU labels are produced by the same pipeline but have not yet been included in a manual gold sample. We will also add a brief discussion in §7 of the structural reasons EU/AU precision may differ — including the English-only BGE-large-en embeddings and jurisdiction-specific NAICS sector definitions (e.g., Sector 22 Utilities) — which the manuscript already partially acknowledges but does not connect to the validation gap. We note that the pipeline configuration is identical across all three jurisdictions (same embedding model, retrieval parameters, GPT-4.1 prompt, score threshold, and class-size filter), which provides some indirect reason to expect comparable precision, but we agree this does not substitute for direct validation and will not claim otherwise.","revision_made":"yes","referee_comment":"The 2,421-row gold validation sample is drawn entirely from the USA portion of the pipeline output. Approximately 62% of the released labels (EU + AU) have no human validation. The abstract and headline results present 96.98% precision as the corpus-level figure without qualifying the geographic scope of validation. The referee requests either (a) validation of EU/AU rows with stratified precision, or (b) explicit reframing of the headline precision as 'USA-validated precision' with a clear statement that EU/AU precision is unvalidated."},{"response":"The referee is correct that the absence of any inter-annotator agreement metric is a gap in the validation design. We will address this through both requested channels to the extent feasible in revision. First, we will conduct a double-labeling pass on approximately 200 rows drawn from the existing gold sample, with a second annotator independently reviewing each row, and report Cohen's kappa alongside the agreement-adjusted precision estimate. If logistical constraints prevent completion of the full double-labeling pass before the revision deadline, we will at minimum include the sensitivity analysis the referee requests. Concretely: given the observed error rate of 3.02% (73 of 2,421 rows judged incorrect), we will compute the range of annotator agreement (kappa) values under which the true precision would remain above a meaningful threshold (e.g., 95%), treating the observed disagreement between the annotator and GPT-4.1 as a mixture of genuine label errors and annotator disagreement. This analysis will be added to §5.6. We acknowledge that neither a 200-row kappa nor a sensitivity analysis fully closes this gap, and the revision will state this limitation transparently rather than overclaiming the strength of the validation.","revision_made":"partial","referee_comment":"The gold set was reviewed by single annotators with no inter-annotator agreement metric. The authors acknowledge this in §5.6 and propose a future double-labeling pass on ~200 rows. The referee requests either a small double-labeled subsample with Cohen's kappa, or at minimum a sensitivity analysis showing how low annotator agreement would need to be for the headline precision to shift meaningfully given the observed 3.02% error rate."}],"tokens_in":21578,"tokens_out":785,"duration_ms":184179,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"Bottom line: this paper releases a genuinely useful dataset — the first publicly available mapping of GitHub repositories to NAICS 2-digit sector codes, covering 6,588 repos across the USA, EU, and Australia. The pipeline is transparent, the code and data are public, and the USA validation is thorough. The one load-bearing weakness is that the headline 96.98% precision is validated only on the USA subset (2,421 of 6,588 rows); the EU and AU labels have zero human validation. The paper acknowledges this, but the abstract doesn't qualify it. That gap is the main thing to fix before publication. What's new and done well: the retrieve-and-verify pipeline (BGE-large-en + FAISS retrieval, GPT-4.1 rubric scoring at score ≥ 8) is well-engineered and documented. The precision validation on the USA sample is solid — Wilson intervals, per-sector breakdowns, and a clean monotonic relationship between GPT score and human-judged correctness (90.8% at score 8, 99.3% at score 10). The end-to-end re-run reproducing the candidate set to within 0.03% is good evidence of pipeline determinism. Releasing the dataset, pipeline code, prompts, and a fine-tuned RoBERTa-large checkpoint under permissive licenses is exactly what a dataset paper should do. The six-model benchmark comparison is a nice bonus, and the finding that encoder family matters more than parameter count at this corpus size is plausible and useful. The soft spots, in proportion: The geographic validation gap is the big one. The released corpus is 38% USA, 31% EU, 31% AU, but the 2,421-row gold sample is drawn entirely from USA output (Appendix H.2). There are concrete reasons to suspect EU/AU precision may differ: NAICS is a North American taxonomy with US-specific sector structure (the paper itself flags Sector 22 Utilities), BGE-large-en is English-only so non-English EU READMEs are under-represented in retrieval, and the GPT-4.1 rubric prompt uses US-centric examples. The paper is commendably transparent about all of this in §5.6 and §7, but the abstract presents 96.98% as the precision of the corpus without geographic qualification. A reader who only sees the abstract would over-trust the EU/AU labels. The single-annotator protocol (no inter-annotator agreement) is a secondary concern — the per-sector precision variation (73% for Manufacturing/Wholesale vs. 100% for six sectors) suggests annotators are genuinely discriminating, not rubber-stamping. The empty Appendix G (qualitative error analysis) and empty Section 4.5 (score distribution) are minor structural gaps. The free-parameter list (score threshold, min class size, k formula, truncation limits) is standard for a pipeline paper and not a concern. Who this is for: empirical software engineering researchers, innovation economists, and policy analysts who need industry-level mappings of open-source activity. Anyone building classifiers on GitHub repository metadata. This deserves a serious referee. The referee should push the authors to either validate a sample of EU/AU labels (even 200-300 rows would help) or explicitly scope the headline precision claim to the USA subset in the abstract. The Manufacturing and Wholesale Trade precision problem at score 8 (73%) should also be more prominently flagged for downstream users. Recommend accept for peer review. The contribution is real, the release is valuable, and the weaknesses are addressable through qualification and additional sampling rather than fundamental redesign.","headline":"First public NAICS-labeled GitHub corpus — useful and well-documented, but headline precision is validated on USA only while 62% of released labels (EU+AU) have no human validation","tokens_in":22677,"tokens_out":1274,"would_cite":true,"duration_ms":56140,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"GitHub repositories mapped to industry sectors at 97% precision","keywords":[],"falsifier":"If a future double-labeling pass on a subsample of EU and Australian repositories (which have no human validation) yields precision substantially below 96.98%, or if an inter-annotator agreement coefficient on a doubly labeled subset reveals systematic annotator bias, the core precision claim would be weakened.","tokens_in":21783,"feed_emoji":"🏭","tokens_out":1412,"duration_ms":138144,"temperature":0.7,"pith_summary":"The authors build NAICS-GH, a publicly released dataset of 6,588 GitHub repositories each labeled with a 2-digit NAICS industry sector code, covering 19 of the 20 top-level NAICS sectors and drawn from source pools in the United States, the European Union, and Australia. The central mechanism is a two-stage retrieve-and-verify pipeline: first, BGE-large-en embeddings indexed in FAISS retrieve candidate repositories for each of the 1,029 NAICS subindustry phrases; second, GPT-4.1 scores each candidate against a four-criterion rubric on a 1–10 scale, and only repositories scoring 8 or above are retained. This pipeline narrows roughly 1.37 million source repositories to 31,178 candidate pairs and then to 6,588 high-confidence labels. On a 2,421-repository human-validated random sample, the released labels achieve 96.98% precision (Wilson 95% CI [96.23, 97.59]), and precision rises monotonically with the GPT score—from 90.76% at score 8 to 99.30% at score 10—indicating that the rubric score carries genuine confidence information. The authors also benchmark six pretrained encoders on the corpus; RoBERTa-large reaches 86.45% F1 and 86.35% accuracy on a held-out 20% test set. The dataset, pipeline code, prompts, and fine-tuned checkpoint are all released under open licenses.","feed_headline":"GitHub repositories mapped to industry sectors at 97% precision","feed_subtitle":"A retrieve-and-verify pipeline using BGE embeddings and GPT-4.1 scoring labels 6,588 public repos with NAICS codes, validated against human","key_machinery":"The load-bearing mechanism is the retrieve-and-verify pipeline. Retrieval is deliberately over-inclusive: BGE cosine similarities of returned candidates concentrate in a narrow [0.78, 0.89] band, so the embedding score alone cannot separate true matches from near-misses. GPT-4.1's structured rubric—evaluating industry-specific software, sector-relevant functionality, industry-domain applications, and sector-specific data/research—performs the discrimination that the embedding similarity cannot. The 1–10 score serves as both a quality filter (threshold at 8) and a confidence signal (monotonically related to human-judged correctness).","core_discovery":"The paper's central claim is that a retrieve-and-verify pipeline combining dense semantic retrieval (BGE embeddings + FAISS) with structured LLM rubric scoring (GPT-4.1 at temperature 0) can assign standardized industry-sector labels to software repositories at high precision, validated against human judgment at 96.98% on a 2,421-row gold sample. The GPT rubric score functions as a calibrated confidence signal: precision increases monotonically from score 8 to score 10, allowing downstream users to trade coverage for precision by raising the threshold. Two sectors—Manufacturing (31–33) and Wholesale Trade (42)—show notably lower precision (~73% at score 8), and the authors recommend raising至","pith_inferences":[],"forward_implications":["Researchers can now study the industrial composition of open-source production across regions and over time using a standardized economic taxonomy, enabling comparisons with Census Bureau statistics on sector-level economic activity.","The fine-tuned RoBERTa-large classifier can propagate NAICS labels to arbitrary GitHub repositories beyond the 6,588 in the released corpus, enabling large-scale industry-mix analysis without re-running the GPT-4.1 verification step.","The retrieve-and-verify pipeline is taxonomy-agnostic in principle: swapping the NAICS JSON for another industry classification (e.g., ISIC, NACE) would adapt the pipeline to non-North-American economic taxonomies without architectural change.","The finding that encoder family matters more than parameter count at this corpus size (DeBERTa-v3-base outperforms ModernBERT-large) provides practical guidance for practitioners working with small labeled corpora in the 5,000-example range.","The score-conditional precision analysis gives downstream users a concrete knob: raising the threshold from 8 to 9 improves overall precision from ~97% to ~98% and fixes the Manufacturing and Wholesale Trade sectors, at the cost of reduced coverage."],"fun_headline_variants":["6,588 GitHub repos labeled with NAICS industry codes at 97% precision","Retrieve-and-verify pipeline assigns NAICS sectors to GitHub repos","Semantic retrieval plus GPT-4.1 scoring maps GitHub repos to industries","NAICS-GH: industry-sector labels for 6,588 public GitHub repositories","Dense retrieval and LLM rubric scoring label GitHub repos by industry"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The 2,421-repository human-validated gold sample was drawn entirely from the USA portion of the pipeline output, and each row was reviewed by a single annotator with no inter-annotator agreement metric. If the single annotators applied the rubric differently from the intended criteria, or if GPT-4.1's scoring behavior on USA repositories does not transfer to the EU and Australian portions, the headline 96.98% precision figure does not necessarily generalize to the full 6,588-","fun_headline_variants_meta":{"raw":{"variants":["6,588 GitHub repos labeled with NAICS industry codes at 97% precision","Retrieve-and-verify pipeline assigns NAICS sectors to GitHub repos","Semantic retrieval plus GPT-4.1 scoring maps GitHub repos to industries","NAICS-GH: industry-sector labels for 6,588 public GitHub repositories","Dense retrieval and LLM rubric scoring label GitHub repos by industry","GitHub repositories classified into NAICS sectors with 96.98% precision","6,588-repo NAICS-labeled corpus released with pipeline and checkpoints","BGE embeddings and GPT-4.1 score assign industry codes to GitHub repos"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":831,"prompt_tokens":641,"completion_tokens":190,"prompt_tokens_details":null},"tokens_in":641,"tokens_out":190,"duration_ms":20498,"temperature":1.0,"reasoning_tokens":59,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T03:25:36.782888+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If a future double-labeling pass on a subsample of EU and Australian repositories (which have no human validation) yields precision substantially below 96.98%, or if an inter-annotator agreement coefficient on a doubly labeled subset reveals systematic annotator bias, the core precision claim would be weakened.","supporting_citations":[],"review_version":1}