The reviewed record of science sign in
Pith

arxiv: 2607.06505 · v1 · pith:XSKUAPUG · submitted 2026-07-07 · cs.SE · cs.AI· cs.DB

Industry Classification of GitHub Repositories Using the North American Industry Classification System (NAICS)

Kevin Xu , Alexander Quispe This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-08 03:25 UTCglm-5.2pith:XSKUAPUGrecord.jsonopen to challenge →

classification cs.SE cs.AIcs.DB
keywords percentrepositoriesindustrypipelinereleasedclassificationgithublabels
0
0 comments X

The pith

GitHub repositories mapped to industry sectors at 97% precision

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The authors build NAICS-GH, a publicly released dataset of 6,588 GitHub repositories each labeled with a 2-digit NAICS industry sector code, covering 19 of the 20 top-level NAICS sectors and drawn from source pools in the United States, the European Union, and Australia. The central mechanism is a two-stage retrieve-and-verify pipeline: first, BGE-large-en embeddings indexed in FAISS retrieve candidate repositories for each of the 1,029 NAICS subindustry phrases; second, GPT-4.1 scores each candidate against a four-criterion rubric on a 1–10 scale, and only repositories scoring 8 or above are retained. This pipeline narrows roughly 1.37 million source repositories to 31,178 candidate pairs and then to 6,588 high-confidence labels. On a 2,421-repository human-validated random sample, the released labels achieve 96.98% precision (Wilson 95% CI [96.23, 97.59]), and precision rises monotonically with the GPT score—from 90.76% at score 8 to 99.30% at score 10—indicating that the rubric score carries genuine confidence information. The authors also benchmark six pretrained encoders on the corpus; RoBERTa-large reaches 86.45% F1 and 86.35% accuracy on a held-out 20% test set. The dataset, pipeline code, prompts, and fine-tuned checkpoint are all released under open licenses.

Core claim

The paper's central claim is that a retrieve-and-verify pipeline combining dense semantic retrieval (BGE embeddings + FAISS) with structured LLM rubric scoring (GPT-4.1 at temperature 0) can assign standardized industry-sector labels to software repositories at high precision, validated against human judgment at 96.98% on a 2,421-row gold sample. The GPT rubric score functions as a calibrated confidence signal: precision increases monotonically from score 8 to score 10, allowing downstream users to trade coverage for precision by raising the threshold. Two sectors—Manufacturing (31–33) and Wholesale Trade (42)—show notably lower precision (~73% at score 8), and the authors recommend raising至

What carries the argument

The load-bearing mechanism is the retrieve-and-verify pipeline. Retrieval is deliberately over-inclusive: BGE cosine similarities of returned candidates concentrate in a narrow [0.78, 0.89] band, so the embedding score alone cannot separate true matches from near-misses. GPT-4.1's structured rubric—evaluating industry-specific software, sector-relevant functionality, industry-domain applications, and sector-specific data/research—performs the discrimination that the embedding similarity cannot. The 1–10 score serves as both a quality filter (threshold at 8) and a confidence signal (monotonically related to human-judged correctness).

If this is right

  • Researchers can now study the industrial composition of open-source production across regions and over time using a standardized economic taxonomy, enabling comparisons with Census Bureau statistics on sector-level economic activity.
  • The fine-tuned RoBERTa-large classifier can propagate NAICS labels to arbitrary GitHub repositories beyond the 6,588 in the released corpus, enabling large-scale industry-mix analysis without re-running the GPT-4.1 verification step.
  • The retrieve-and-verify pipeline is taxonomy-agnostic in principle: swapping the NAICS JSON for another industry classification (e.g., ISIC, NACE) would adapt the pipeline to non-North-American economic taxonomies without architectural change.
  • The finding that encoder family matters more than parameter count at this corpus size (DeBERTa-v3-base outperforms ModernBERT-large) provides practical guidance for practitioners working with small labeled corpora in the 5,000-example range.
  • The score-conditional precision analysis gives downstream users a concrete knob: raising the threshold from 8 to 9 improves overall precision from ~97% to ~98% and fixes the Manufacturing and Wholesale Trade sectors, at the cost of reduced coverage.

Load-bearing premise

The 2,421-repository human-validated gold sample was drawn entirely from the USA portion of the pipeline output, and each row was reviewed by a single annotator with no inter-annotator agreement metric. If the single annotators applied the rubric differently from the intended criteria, or if GPT-4.1's scoring behavior on USA repositories does not transfer to the EU and Australian portions, the headline 96.98% precision figure does not necessarily generalize to the full 6,588-

What would settle it

If a future double-labeling pass on a subsample of EU and Australian repositories (which have no human validation) yields precision substantially below 96.98%, or if an inter-annotator agreement coefficient on a doubly labeled subset reveals systematic annotator bias, the core precision claim would be weakened.

Figures

Figures reproduced from arXiv: 2607.06505 by Alexander Quispe, Kevin Xu.

Figure 1
Figure 1. Figure 1: NAICS-GH end-to-end pipeline. A Presto/Trino SQL extraction from GitHub’s [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
read the original abstract

GitHub hosts hundreds of millions of public repositories, but the platform exposes no native mapping from repositories to standardized industry sectors. This gap limits empirical work on the geography of innovation, the industrial composition of open-source production, and the diffusion of new technologies across economic sectors. We present NAICS-GH, a publicly released corpus of 6,588 GitHub repositories drawn from source pools covering the United States, the European Union, and Australia, each labeled with a 2-digit sector from the North American Industry Classification System (NAICS 2022). Labels are produced by a retrieve-and-verify pipeline that combines BAAI/bge-large-en embeddings, FAISS retrieval, and GPT-4.1 rubric scoring. The pipeline narrows about 1.37 million source repositories to 31,178 candidate repository-sector pairs and retains 6,588 high-confidence labels with score at least 8. Re-running the retrieval pipeline end to end reproduces the candidate set to within 0.03 percent. On a 2,421-repository human-validated random sample, the released labels attain 96.98 percent precision, with Wilson 95 percent confidence interval [96.23, 97.59]. We benchmark six pretrained encoders on the released corpus; RoBERTa-large reaches 86.45 percent F1 and 86.35 percent accuracy on a held-out 20 percent test set. The dataset, Croissant metadata, pipeline code, prompts, and fine-tuned checkpoint are released under CC-BY-4.0 and MIT licenses.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 7 minor

Summary. This paper presents NAICS-GH, a corpus of 6,588 GitHub repositories from the USA, EU, and Australia labeled with 2-digit NAICS sector codes. Labels are produced by a two-stage retrieve-and-verify pipeline: BGE-large-en embeddings + FAISS retrieval narrows ~1.37M source repositories to 31,178 candidates, and GPT-4.1 rubric scoring (score >= 8) retains 6,588 high-confidence labels. The authors report 96.98% precision (Wilson 95% CI [96.23, 97.59]) on a 2,421-repository human-validated sample, and benchmark six pretrained encoders, with RoBERTa-large achieving 86.45% F1 on a held-out test set. The dataset, pipeline code, prompts, and fine-tuned checkpoint are publicly released.

Significance. The paper addresses a genuine gap: no publicly available corpus maps GitHub repositories to a standardized economic industry taxonomy. The pipeline is well-documented and reproducible, with an end-to-end re-run confirming candidate-set replication to within 0.03%. The authors release the dataset (CC-BY-4.0), pipeline code (MIT), prompts, and a fine-tuned RoBERTa-large checkpoint. The validation methodology uses Wilson confidence intervals with per-sector breakdowns, and the score-conditional precision analysis (Table 7) provides a usable confidence signal for downstream practitioners. The benchmark comparison across six encoders with identical hyperparameters is a fair and useful contribution.

major comments (3)
  1. §5.1 and Appendix H.2: The 2,421-row gold validation sample is drawn entirely from the USA portion of the pipeline output (Appendix H.2: 'Human re-validation of 2,421 USA repositories'). The released corpus contains 6,588 rows: 2,529 USA, 2,039 EU, and 2,021 AU (Table 1). This means approximately 62% of the released labels have no human validation. The abstract and headline results present 96.98% precision as the corpus-level figure without qualifying the geographic scope of validation. This is load-bearing because the central claim is about the precision of the released corpus, not just the USA subset. The paper acknowledges in §7 that NAICS is a North American taxonomy and that sector definitions can diverge across jurisdictions (e.g., Sector 22 Utilities), and that BGE-large-en is English-only while EU repositories may have non-English READMEs. These structural factors give concrete理由
  2. to expect EU/AU precision may differ from USA precision. The authors should either (a) validate a sample of EU/AU rows and report stratified precision, or (b) explicitly reframe the headline precision as 'USA-validated precision' in the abstract and §5.2, with a clear statement that EU/AU precision is unvalidated. Option (b) is achievable within revision.
  3. §5.1: The gold set was reviewed by single annotators with no inter-annotator agreement metric. The authors acknowledge this in §5.6 and propose a future double-labeling pass on ~200 rows. Given that the 96.98% precision figure is the paper's central validation claim, the absence of any agreement metric is a load-bearing gap. Even a small double-labeled subsample (the proposed 200 rows) with a Cohen's kappa would substantially strengthen the claim. If this cannot be completed before revision, the authors should at minimum add a sensitivity analysis: how low would annotator agreement need to be for the headline precision to shift meaningfully, given the observed error rate of 3.02%?
minor comments (7)
  1. Table 3: The sector name for code 72 is listed as 'Accomodation and Food Services' — the correct spelling is 'Accommodation.' The same misspelling appears in Table 6 and Table 9.
  2. §3.3, effective_k formula: The piecewise definition reads 'effective_k = {20 if n > 20; max(20, ceil(400/n)) if n <= 20}'. For n > 20, the formula gives k = 20, but the text says 'For the 17 sectors with n > 20 this gives a uniform k = 20.' This is correct but the formula notation is slightly confusing since the second branch also applies max(20, ...) which is always >= 20. Consider clarifying that the second branch only activates for n <= 20.
  3. Appendix C, prompt: The rubric examples are US-centric (e.g., 'Farm management software for Agriculture'). While this is understandable given NAICS is a North American taxonomy, the authors might note whether EU/AU-specific examples were considered or whether the US-centric framing could interact with the geographic validation gap noted above.
  4. §4.4: The 113 duplicate (name_repo, code) pairs are described as distinct repositories sharing short names. The paper notes that upstream intermediate files retain nwo as a globally unique identifier. Consider noting whether these intermediate files are available to downstream users, or whether the name ambiguity is an accepted limitation of the public release.
  5. Table 8: The finding that 'encoder family matters more than parameter count' is interesting but the differences between RoBERTa-large (86.45%), DeBERTa-v3-base (85.68%), and DeBERTa-v3-large (85.07%) are within ~1.4pp. Without confidence intervals or multiple seeds, it is unclear whether these differences are statistically significant. A note acknowledging this would be appropriate.
  6. §3.1: The USA source dump was extracted April 15, 2025, while the EU README extraction is dated August 15, 2025. This four-month gap in source data freshness is not discussed. If repository content changed between these dates, the EU and USA pools are not temporally comparable. A brief note acknowledging this would be useful.
  7. Appendix G is referenced in §5.5 as containing qualitative error analysis of the 73 incorrect labels, but the appendix appears to be empty (only the header 'G Qualitative error analysis' is present with no content). This should be populated or the reference removed.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for a careful and constructive review. Both major comments identify genuine gaps in our validation scope that we will address in the revised manuscript. Below we respond point by point.

read point-by-point responses

  1. Referee: The 2,421-row gold validation sample is drawn entirely from the USA portion of the pipeline output. Approximately 62% of the released labels (EU + AU) have no human validation. The abstract and headline results present 96.98% precision as the corpus-level figure without qualifying the geographic scope of validation. The referee requests either (a) validation of EU/AU rows with stratified precision, or (b) explicit reframing of the headline precision as 'USA-validated precision' with a clear statement that EU/AU precision is unvalidated.

    Authors: The referee is correct on the facts. The gold sample is drawn exclusively from the USA portion of the pipeline output (Appendix H.2), and the abstract presents 96.98% precision without qualifying this geographic scope. We agree this is a load-bearing gap. We will adopt option (b) in the revision: the abstract, §5.1, §5.2, and the Datasheet (Appendix H.2) will be revised to explicitly state that the 96.98% precision figure reflects USA-validated precision only, and that EU and AU labels are produced by the same pipeline but have not yet been included in a manual gold sample. We will also add a brief discussion in §7 of the structural reasons EU/AU precision may differ — including the English-only BGE-large-en embeddings and jurisdiction-specific NAICS sector definitions (e.g., Sector 22 Utilities) — which the manuscript already partially acknowledges but does not connect to the validation gap. We note that the pipeline configuration is identical across all three jurisdictions (same embedding model, retrieval parameters, GPT-4.1 prompt, score threshold, and class-size filter), which provides some indirect reason to expect comparable precision, but we agree this does not substitute for direct validation and will not claim otherwise. revision: yes

  2. Referee: The gold set was reviewed by single annotators with no inter-annotator agreement metric. The authors acknowledge this in §5.6 and propose a future double-labeling pass on ~200 rows. The referee requests either a small double-labeled subsample with Cohen's kappa, or at minimum a sensitivity analysis showing how low annotator agreement would need to be for the headline precision to shift meaningfully given the observed 3.02% error rate.

    Authors: The referee is correct that the absence of any inter-annotator agreement metric is a gap in the validation design. We will address this through both requested channels to the extent feasible in revision. First, we will conduct a double-labeling pass on approximately 200 rows drawn from the existing gold sample, with a second annotator independently reviewing each row, and report Cohen's kappa alongside the agreement-adjusted precision estimate. If logistical constraints prevent completion of the full double-labeling pass before the revision deadline, we will at minimum include the sensitivity analysis the referee requests. Concretely: given the observed error rate of 3.02% (73 of 2,421 rows judged incorrect), we will compute the range of annotator agreement (kappa) values under which the true precision would remain above a meaningful threshold (e.g., 95%), treating the observed disagreement between the annotator and GPT-4.1 as a mixture of genuine label errors and annotator disagreement. This analysis will be added to §5.6. We acknowledge that neither a 200-row kappa nor a sensitivity analysis fully closes this gap, and the revision will state this limitation transparently rather than overclaiming the strength of the validation. revision: partial

Circularity Check

0 steps flagged

No circularity found — pipeline uses external tools, external taxonomy, and independent human validation

full rationale

The paper's derivation chain is self-contained against external inputs at every load-bearing step. (1) The NAICS taxonomy is an external standard from the U.S. Census Bureau (§3.2). (2) BGE-large-en embeddings and FAISS retrieval are external, independently developed tools (§3.3). (3) GPT-4.1 rubric scoring uses an external model accessed via an API endpoint (§3.4). (4) The 96.98% precision claim is validated against a human-annotated gold set (§5.1–5.2) that is independent of the GPT-4.1 scoring — research assistants judged correctness by inspecting GitHub pages, not by consulting GPT-4.1 outputs. (5) The monotonic precision-vs-score result (Table 7) is measured against the human gold set, not against the GPT scores themselves, so it is not self-definitional. (6) The benchmark classifiers (RoBERTa, DeBERTa, ModernBERT) are pretrained models from independent sources (Hugging Face), fine-tuned on the released corpus and evaluated on a held-out split — standard ML practice, not circularity. (7) No self-citation chain is load-bearing: the authors cite only external prior work (BGE, FAISS, RoBERTa, DeBERTa, ModernBERT, BEACON, Snorkel, etc.), and no 'uniqueness theorem' or prior ansatz from the same authors is invoked to force a conclusion. The paper does have acknowledged limitations (single-annotator protocol, USA-only validation, no recall analysis), but these are correctness/generalization risks, not circularity.

Axiom & Free-Parameter Ledger

8 free parameters · 5 axioms · 0 invented entities

The paper introduces no new particles, forces, dimensions, or postulated entities. It uses existing models (BGE-large-en, GPT-4.1, RoBERTa, DeBERTa, ModernBERT), existing tools (FAISS), existing taxonomies (NAICS 2022), and existing data (GitHub public repositories). All free parameters are pipeline design choices, not physical constants or fitted model parameters in the theoretical sense.

free parameters (8)
  • score_threshold = 8
    The GPT-4.1 score cutoff for retaining labels (Section 3.4). Chosen to maximize coverage while maintaining high precision; validated post hoc by the gold sample.
  • min_class_size = 80
    Minimum repositories per NAICS sector to retain the sector (Section 3.5). Eliminates Sector 55 which had only 13 candidates.
  • effective_k_formula = max(20, ceil(400/n))
    Retrieval depth per subindustry query (Section 3.3). The formula and its constants (20, 400) are chosen by the authors to ensure adequate candidate coverage per sector.
  • readme_truncation_embed = 1000 chars
    README truncation for embedding input (Section 3.3). Chosen by the authors.
  • readme_truncation_llm = 3000 whitespace tokens
    README truncation for LLM verification input (Section 3.4). Chosen by the authors.
  • max_length_tokenizer = 512 WordPiece tokens
    Uniform tokenization length for all six benchmark models (Section 6.1). Chosen for direct comparability.
  • learning_rate = 1.5e-5
    Fine-tuning learning rate for all six benchmark runs (Section 6.2). Standard value but not tuned per model.
  • random_state = 42
    Seed for stratified train/validation/test split (Section 6.2).
axioms (5)
  • domain assumption BGE-large-en embeddings produce semantically meaningful representations of GitHub README content for industry-classification retrieval.
    The entire retrieval stage (Section 3.3) depends on this. The paper does not independently verify embedding quality but relies on the model's published capabilities.
  • domain assumption GPT-4.1 at temperature=0 produces near-deterministic and reliable rubric-based classification judgments for repository-sector pairs.
    The labeling pipeline (Section 3.4) depends on this. The paper validates output precision but assumes the model's judgments are consistent and trustworthy.
  • domain assumption NAICS 2-digit sector codes are a meaningful and applicable taxonomy for GitHub repositories across USA, EU, and AU jurisdictions.
    The paper acknowledges this is an assumption in Section 7 ('NAICS is a North American taxonomy. Applying NAICS to European or Australian repositories assumes that economic activities map cleanly across jurisdictions').
  • domain assumption The README, description, and topic tags of a GitHub repository are sufficient signal to determine the industry sector the repository serves.
    The pipeline uses only these fields (Section 3.4). If repository metadata does not reflect industry purpose, the labels cannot be correct.
  • ad hoc to paper Single-annotator judgments on the gold set are reliable proxies for ground truth.
    Section 5.1 states each row was reviewed by a single annotator with no inter-annotator agreement computed. The headline precision depends on this being acceptable.

pith-pipeline@v1.1.0-glm · 24700 in / 3280 out tokens · 287652 ms · 2026-07-08T03:25:36.782888+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 24 canonical work pages · 1 internal anchor

  1. [1]

    Topic Recommendation for Software Repositories using Multi-label Classification Algorithms

    URLhttps://arxiv.org/abs/2010.09116. Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547, 2021. Nazia Shehnaz Joynab and Soneya Binta Hossain. From threads to trajectories: A multi-LLM pipeline for community knowledge extraction from GitHub issue discussions.arXiv preprint a...

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  2. [2]

    **Industry-Specific Software**: Applications, tools, or systems designed specifically for use within this industry sector↪→ - Example: Farm management software for Agriculture (Sector 11) - Example: Church management systems for Religious Organizations (Sector 81)

    work page

  3. [3]

    **Sector-Relevant Functionality**: Code that implements processes, calculations, or workflows specific to this industry↪→ - Example: Prayer time calculators for Religious Organizations - Example: Crop yield prediction models for Agriculture

    work page

  4. [4]

    **Industry Domain Applications**: Software that directly serves businesses, organizations, or activities within this sector↪→ - Example: Restaurant POS systems for Food Services (Sector 72) - Example: Educational platforms for Educational Services (Sector 61)

    work page

  5. [5]

    **Sector-Specific Data/Research**: Datasets, analysis tools, or research implementations focused on this industry↪→ - Example: Agricultural sensor data analysis - Example: Healthcare outcome prediction models CLASSIFICATION STANDARDS: **INCLUDE ("Yes") when:** - The repository's primary purpose aligns with the sector - The software would be used by busine...

    work page

  6. [6]

    Identify the repository's primary purpose and functionality

    work page

  7. [7]

    Assess alignment with the target NAICS sector

    work page

  8. [8]

    Determine the most applicable classification criterion

    work page

  9. [9]

    NAICS {naics_code}

    Consider practical usage within the sector Provide your response in this exact JSON format: { "NAICS {naics_code}": { 17 "rationale": "Concise explanation of classification decision, including primary repository purpose, specific sector alignment criteria met, and justification for inclusion/exclusion", ↪→ ↪→ "score": "1-10", "match": "Yes" or "No" } } IM...

    work page 2025

  10. [10]

    Strip Markdown badges and shields (![...]([url])and[![...](...)](...))

    work page

  11. [11]

    strip license/copyright headers ( MIT License, Apache License, GPL, BSD, Copyright ...)

    work page

  12. [12]

    collapse URLs to the bare domain (https?://(domain)/path?...→domain)

    work page

  13. [13]

    strip Markdown headers (#...######) and bold/italic/code markers (*,_,~,`)

    work page

  14. [14]

    replace fenced code blocks with acode-{lang} placeholder; strip inline backticks but keep the content

    work page

  15. [15]

    normalize tech-stack mentions (js→javascript, py→python, reactjs→react, nodejs→nodejs)

    work page

  16. [16]

    normalize excessive punctuation (!!→!,??→?,....→...)

    work page

  17. [17]

    normalize whitespace (collapse multiple newlines and spaces)

    work page

  18. [18]

    Datasheet for Datasets

    strip installation-command noise (npm install, pip install, git clone, and the rest of those lines). The cleaned string is thetextcolumn passed to the tokenizer. F.2 Base models and parameter counts All six runs share data, splits, and hyperparameters; only the base model differs. Table 13: Pretrained encoders fine-tuned in §6. Hugging Face identifier Lab...

    work page 2021

  19. [19]

    README preprocessing.For embedding input, therawREADME is truncated to its first 1,000 characters (no markup stripping at this stage). For LLM scoring, code blocks, HTML tags, badge URLs, and image markdown are stripped and the text is truncated to 3,000 whitespace-separated tokens (word-count proxy, not BPE tokens)

    work page

  20. [20]

    description: {desc}, topics: {topics}, readme: {readme}

    Composite text construction.LLM scoring uses the combined string "description: {desc}, topics: {topics}, readme: {readme}", constructed at row time and inserted into the<readme>...</readme> block of the prompt. 24 Embedding uses the truncated README only; description and topics enter the pipeline at the scoring stage

    work page

  21. [21]

    Represent this document for retrieval:

    Embedding.BGE-large-en with the asymmetric BGE prefixes: "Represent this document for retrieval: " for documents and "Represent this query for retrieval: "for queries

    work page

  22. [22]

    Repositories about {subindustry}

    Retrieval.For each NAICS subindustry, the query "Repositories about {subindustry}" is embedded and the top-knearest repositories are retrieved (de- faultk= 20; the three sectors with fewer than 20 subindustries use a boostedk= max(20, ceil(400/n))= 24, 27, or 29)

    work page

  23. [23]

    NAICS {code}

    LLM scoring.Each (repository, sector) candidate is presented to GPT-4.1 with the rubric prompt (see the paper appendix). The model returns a JSON object nested under the outer key"NAICS {code}", with string-valuedmatch("Yes"/"No"), string-valuedscore("1"-"10"), and free-formrationale. Downstream code coerces matchto a boolean andscoreto an integer. 6.Filt...

    work page

  24. [24]

    Utilities,

    Minimum-class filter (training derivative only).Sectors with fewer than 80 repositories are dropped, removing sector 55. Was the raw data saved in addition to the preprocessed data?Yes. The original parquets of unfiltered GitHub repositories were preserved (USA, EU, AU dumps), as well as the three intermediate per-region GPT outputs ({usa,eu,au}_2k_gpt_ab...

    work page 2026