{"id":"77e4fb22-42a9-4cea-ba9f-6f3c4ca7ff4f","arxiv_id":"2607.06920","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":6,"one_line_summary":"A composed pipeline of betweenness cuts, per-edge logistic classifiers, and GitHub-asserted edges resolves ~107M author identities while reducing the largest cluster from 170,431 to under 7,000 and improving gold recall from 0.44 to 0.70.","lead":"This paper describes a methodology for resolving ~107 million free-text author identities in the World of Code (WoC) dataset into canonical persons, focusing on preventing 'mega-clusters' caused by bridge identities. A smart generalist might read it to understand how large-scale data disambiguation can be achieved without sacrificing precision or recall, a common problem in software engineering and network analysis.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The 30.8M+ expansion edges (Exps. 14–17) are precision-validated only on the 2,345-id ALFAA gold set; the 21M-alias GitHub benchmark is recall-only by construction and cannot detect moderate over-merge in those edges.","rationale":"The reader correctly identifies the ghid-transfer axis as a soft spot, but I think the more load-bearing concern is the evaluation gap that surrounds it: the recall-recovery stage adds ~70M edges whose precision is checked only on a 2,345-id gold set (159 FPs total) while the large-scale benchmark is recall-only. The ghid transfer is one contributor to this gap, but even a perfectly transferring classifier could produce moderate over-merge that neither benchmark detects. That said, the concern does not rise to the level of moving the verdict off ACCEPT, for three reasons. First, the structural mega-check (no cluster >10k; largest = 6,910) provides a hard ceiling on the worst-case over-merge from the added edges. Second, the ALFAA gold precision does rise (0.8733→0.8790), even if the sample is small — the direction is correct, not just neutral. Third, the paper is unusually self-aware about the benchmark inversion problem and explicitly frames it as a methodological lesson rather than hiding it. The concern is real but bounded: it questions the precision of the recall-recovery edges on the broader population, not the dissolution of the mega-cluster (which is structurally verified) or the recall gains (which are validated on a 21M-alias benchmark). The paper's evidence is sufficient for ACCEPT at MODERATE confidence, which is where the reader placed it. The concrete test I propose would either confirm the precision claim or reveal hidden over-merge, but its absence does not invalidate the current evidence.","tokens_in":34138,"tokens_out":4131,"duration_ms":145182,"concrete_test":"Draw a stratified random sample of ~800 pairs from the production map's added edges (the union of Exp. 14 τ=0.9 shingle edges, Exp. 17 ghid edges, and v2.1@0.8 edges), stratified by classifier score bin (0.5–0.7, 0.7–0.9, 0.9–1.0) and resulting cluster size (2, 3–10, 11–100, 101–1000). Human-adjudicate each pair for same-person status. If the false-positive rate exceeds 15% overall or 25% in any stratum, the 'at increasing precision' claim for the recall-recovery stage weakens, because neither the tiny gold set nor the recall-only GitHub benchmark would have caught it.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has three pillars: (1) largest cluster 170,431→<7,000, (2) gold recall 0.44→0.70 at increasing precision, (3) outperforms SOTA on the 21M-alias GitHub benchmark. Pillar (1) is a direct structural computation and is solid. The concern is with pillars (2) and (3) as they jointly validate the recall-recovery stage (Exps. 14–17), which adds 30.8M shingle-expansion edges, 5.7M ghid edges, and 34.1M v2.1-scored edges to the production map (SUG). The ALFAA gold set that validates precision is tiny (2,345 ids, 1,644 true matches, OpenStack-centric) and the paper's own audit (Exp. 13) identifies ~20 label errors among just 104 FPs, suggesting the gold set is noisy and narrow. The GitHub benchmark (21M aliases) validates recall but is recall-only by construction — false positives are invisible. So the precision of the ~70M added edges on the broader 107M-id population is measured only through 159 gold FPs. The paper itself states that 'recall-only and precision-only benchmarks invert verdicts unless read together,' yet the two benchmarks it reads together are each blind to one axis, and their intersection (the gold set's coverage of expansion-edge precision) is narrow. The ghid transfer concern the reader identifies is part of this — the classifier trained on noreply-id pairs scores 256.7M expansion pairs that are mostly non-noreply — but the deeper issue is that no benchmark directly measures moderate over-merge (wrong clusters of size 100–1000) in the added edge mass. The fragility evidence is internal: v2.1 transfers better (AUC 0.9910 vs 0.9861) yet produces a worse map (GitHub recall 0.5118 vs 0.5886), showing that small model changes produce large map-quality swings in a region no benchmark covers well.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"This paper documents the methodology used to construct the author identity map for World of Code (WoC) V2604, covering ~107M distinct author strings over ~6B commits—an order of magnitude beyond prior published resolutions. The central engineering problem identified is over-merge: a small number of bridge identities (bots, role accounts, placeholder emails, template strings) weld unrelated clusters into million-member mega-clusters through transitive union. The paper reports a complete experimental record of 20+ experiments, including unsuccessful approaches, culminating in a deployed composition: a betweenness-centrality cut over the exact union graph, a per-edge logistic classifier trained on 2.6M labels mined from GitHub no-reply identifiers, classifier-filtered expansion of dormant cross-project shingle groups, and GitHub-asserted same-account edges. The final production map reduces the largest cluster from 170,431 (ungated) to 6,910, raises gold recall from 0.44 to 0.70 at increasing precision, and outperforms predecessors and published state of the art on an independent 21M-alias GitHub ground truth.","tokens_in":34486,"tokens_out":3449,"duration_ms":149108,"significance":"The paper tackles a foundational infrastructure problem for software repository mining at unprecedented scale. Several aspects deserve explicit credit. First, the full experimental record including failures is a genuine contribution: the systematic demonstration that node-level gates (info-score, project spread, degree) and per-value blocklists each fail for articulable reasons, and that the principled fix requires moving from nodes to topology (betweenness cut) and from values to edges (learned classifier), is a transferable methodological lesson. Second, the 'labels for free' technique—mining 2.6M labeled identity pairs from GitHub no-reply identifiers at zero annotation cost—is an efficient and reproducible labeling strategy. Third, the transfer validation (ghid-trained classifier achieving AUC 0.987 on human-adjudicated ALFAA gold pairs, matching ALFAA's random forest without its expensive doc2vec fingerprint) is a falsifiable and confirmed prediction. Fourth, the paper's transparency about the shortcut-learning failure of gradient boosting (LGBM achieving 0.999 in-distribution AUC but collapsing to 0.56 on transfer) is a valuable cautionary result. Fifth, the deployed map and ","major_comments":[{"comment":"§2.9–2.11, Exps. 14–17 (Tables 18, 20): The recall-recovery stage adds approximately 70M edges to the production map (30.8M shingle-expansion edges at τ=0.9, 34.1M v2.1-scored edges at τ=0.8, and 5.7M ghid edges). The precision of these added edges on the broader 107M-id population is validated through only 159 false positives in the ALFAA gold set (Table 20: SUG gold precision 0.8790, FP=159). The GitHub benchmark (21M aliases) is recall-only by construction and cannot detect false positives at all. The paper itself states that 'recall-only and precision-only benchmarks invert verdicts unless read together,' yet the two benchmarks it reads together are each blind to one axis, and their intersection—the gold set's coverage of expansion-edge precision—is narrow (2,345 ids, OpenStack-centric, with the paper's own audit in Exp. 13 identifying ~20 label errors among just 104 FPs). No direct,","section":null},{"comment":"targeted audit of moderate-sized clusters (100–1000 members) formed by the expansion edges is reported. The paper notes that the 1k–10k band grows from 4 to 14 clusters at τ=0.9 (Table 18) and that SUG's largest cluster is 6,910, but whether these moderate clusters represent correct merges or over-merges is not directly measured. A sampled human audit of, say, 10–20 clusters in the 100–1000 range formed by expansion edges would substantially strengthen the claim that precision is maintained on the broader population, not just on the 2,345-id gold set. Without this, pillars (2) and (3) of the central claim rest on a precision measurement whose coverage of the added edge mass is thin.","section":null},{"comment":"§2.6, Exp. 6 (Table 25); §2.9, Exp. 15 (Table 19): The per-edge classifier is trained on 2.6M ghid-labeled links (pairs where both endpoints carry GitHub no-reply identifiers) but is then applied to score 256.7M expansion pairs (Exp. 15) that are predominantly non-noreply identities. The transfer to the ALFAA gold set (AUC 0.987) is encouraging, but the gold set comprises only 2,345 ids and is OpenStack-centric. The paper should explicitly discuss what fraction of the 107M-id corpus the gold set's population represents, and identify populations (e.g., non-English-name developers, corporate-email-only contributors, contributors from platforms other than GitHub) for which the ghid-trained classifier's transfer is least supported. The v2.1 vs. v1 production decision (Exp. 15–17) already demonstrates that in-distribution AUC is a poor selector for out-of-distribution performance; a parallel","section":null},{"comment":"analysis of where the classifier's calibration is least trustworthy would make the deployment claim more defensible.","section":null}],"minor_comments":[{"comment":"Several table references appear as 'Table ??' (e.g., in Exps. 2, 3, 7, 13, and the V3 comparison in §2.8), indicating placeholder cross-references that need to be resolved before publication.","section":null},{"comment":"The abstract states 'more than twenty experiments' while the body and conclusions say 'twenty experiments' and 'eighteen experiments' respectively (§3). The actual count in Tables 1–7 and 7 is 25 (Exps. 1–25). This inconsistency should be reconciled.","section":null},{"comment":"§2.14 (Engineering Lessons): The acknowledgment that the experiments were 'designed, executed, and written up by Claude (Anthropic)' is unusual for an academic paper. While this does not affect scientific merit, the journal may wish to clarify its policy on AI-assisted experimentation and authorship attribution for the analysis.","section":null},{"comment":"Table 20: The 'multi' recall column for V3 max9 (0.3675) is higher than the overall recall (0.4869), which is counterintuitive for a macro-averaged metric restricted to multi-alias handles. A footnote explaining the directionality would help readers interpret the benchmark correctly.","section":null},{"comment":"§2.7: The term 'bad high-quality attributes' is introduced without a formal definition at first use; the two-paragraph explanation that follows could benefit from a one-sentence definition upfront (e.g., 'values rare enough to escape inverse-frequency down-weighting yet semantically void').","section":null},{"comment":"Table 11: The 'ambiguous' intent category (19,211 emails) is introduced but its downstream treatment is not clearly specified—are ambiguous values treated as privacy or homonym? The text in §2.7 only discusses privacy and homonym handling.","section":null},{"comment":"The paper would benefit from a concise summary table listing all 25 experiments, their status (✓/✗), and the one-line lesson from each, as the current experiment log spans multiple tables and is difficult to navigate.","section":null}],"recommendation":"minor_revision","confidential_remarks":"The paper is dense and unusually long for a methodology paper, but the density is justified by the experimental record format. The main substantive concern (precision validation gap for the recall-recovery stage) is real but addressable through a targeted cluster audit and a discussion of transfer limitations, neither of which requires re-running the pipeline. The paper's transparency about failures and limitations is a genuine strength and should not be penalized. The AI-authorship disclosure is worth the editor's attention but is not a scientific concern."},"author_rebuttal":null,"desk_editor":{"model":"glm-5.2","letter":"The headline: this is a methodology paper for production identity resolution at 107M author strings, an order of magnitude beyond prior work. The core diagnosis — that over-merge at this scale is driven by 'bad high-quality attributes' (values rare enough to look identifying but semantically void) — and the betweenness-cut solution are the real contributions. The paper ships deployed infrastructure, released data, and a replication package. That earns it a serious read.","headline":"Real infrastructure paper at unprecedented scale; validation gap in the recall-recovery stage is genuine but the paper is honest about it","tokens_in":35038,"tokens_out":3394,"would_cite":true,"duration_ms":134638,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Mega-clusters dissolve when you cut edges, not nodes","keywords":["author identity disambiguation","over-merge","betweenness centrality","per-edge classification","mega-cluster dissolution","World of Code","GitHub no-reply identifiers","record linkage"],"falsifier":"If the per-edge classifier's transfer to human judgments did not hold at AUC 0.99 on the ALFAA gold set, or if the betweenness cut failed to reproduce its local prediction in the production union, the composed design would lack its two independent validations and the recall-recovery expansion would risk re-welding the mega-cluster.","tokens_in":34447,"feed_emoji":"🔬","tokens_out":1214,"duration_ms":222688,"temperature":0.7,"pith_summary":"The paper tackles a deceptively simple question at massive scale: when Git records authorship as free-text strings, how do you figure out which strings belong to the same person? At 107 million distinct author strings across 6 billion commits, the problem inverts from the usual one. The dominant failure is not missing real matches but over-merging: a handful of bridge identities (placeholder emails, bot accounts, shared template strings) weld millions of unrelated people into mega-clusters through transitive union. The paper's central claim is that no mechanism operating at the level of individual identities or attribute values can dissolve these mega-clusters without also destroying legitimate matches, because the welds form a redundant mesh that re-closes around each removed link. What works is changing the object of analysis twice: first from nodes to topology, using betweenness centrality to find the roughly 2,000 load-bearing nodes whose removal shatters the mega-cluster, and then from values to individual edges, training a classifier on 2.6 million free labels mined from GitHub no-reply identifiers to distinguish true from false links. The same classifier then safely re-expands dormant cross-project matches, recovering more recall than the precision work had cost, while the largest cluster falls from 170,431 to under 7,000.","feed_headline":"Mega-clusters dissolve when you cut edges, not nodes","feed_subtitle":"At 107M identities, the hard problem is over-merge. The fix is topology plus a free-label classifier.","key_machinery":"The working composition has four stages. First, a betweenness-centrality cut over the exact union graph identifies and gates roughly 2,000 to 5,000 load-bearing nodes that weld the mega-cluster. Second, a per-edge logistic classifier trained on 2.6 million labels mined from GitHub no-reply numeric identifiers prunes false links within the residual homonym fragments the cut leaves behind. Third, the same classifier filters the expansion of dormant cross-project shingle groups, admitting only high-confidence pairs as new union edges. Fourth, GitHub's own account assertions from no-reply identifiers add edges the classifier cannot see. The labels are free because GitHub embeds a numeric user id","core_discovery":"The decisive finding is that over-merge at this scale is driven by what the paper calls bad high-quality attributes: values rare enough to look identifying yet semantically void, such as Cloudflare relay hashes and unconfigured git template emails. These values score high on every standard information metric, so classifiers and thresholds cannot separate them from genuine aliases. The mega-cluster persists not through a few extreme hubs but through a long tail of moderate bridges that no node-level threshold can catch without collateral damage. The resolution is compositional: a betweenness cut on the exact union graph finds load-bearing nodes invisible to any attribute rule (98.4 percent of","pith_inferences":["The finding that betweenness centrality outperforms attribute-based detection for bridge identification suggests that network topology is a more reliable signal of over-merge than any property of the identities themselves, which may generalize to other record-linkage domains where shared placeholder values create transitive welds.","The observation that a better-calibrated classifier produced a worse map (because it pruned exactly the real within-handle aliases an external benchmark counted) implies that the relationship between model quality and system quality is non-monotonic when the training distribution differs from the deployment distribution, a gap that may require active learning on the deployment distribution to clos","If cryptographic signatures concentrate almost entirely in the already-resolvable good-class population, then signature-based identity anchoring is primarily a trust-tier instrument rather than a recall tool, and the hard tail of unresolvable local and generic identities will require fundamentally different evidence sources, such as behavioral fingerprints or institutional context."],"forward_implications":["Any global identity resolution system operating above roughly 100 million entities must treat over-merge, not missed merges, as the binding constraint, and must evaluate on both precision and recall axes simultaneously or the benchmark itself will drive the wrong design.","Structural cuts computed on one graph do not transfer to an augmented graph: every new edge source must re-earn safety end-to-end, which constrains how identity maps can be incrementally updated.","Gradient-boosted models can shortcut-learn label-construction artifacts that linear models survive, which means in-distribution accuracy is unreliable as a model selector when labels are mined from a shared platform.","The distinction between privacy masks (deliberate anonymization that must never be merged) and homonym defaults (accidental collisions that may be re-linked by later evidence) is a design principle for any system handling user-asserted identity strings."],"fun_headline_variants":["Over-merge, not missed merges, is the hard problem at 107M identities","Edge cuts break mega-clusters that node thresholds cannot","Rare-but-meaningless attributes weld 107M aliases into mega-clusters","Betweenness plus a per-edge classifier tames global author disambiguation","Node-level gates preserve recall but cannot dissolve mega-clusters"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The classifier is trained on 2.6 million labels derived from GitHub no-reply identifiers and assumes this signal transfers faithfully to the broader population of identities that do not carry such identifiers, a premise validated against a small human-labeled set of 2,345 identities but load-bearing for the recall-recovery stages that operate across the full 107-million-identity corpus.","fun_headline_variants_meta":{"raw":{"variants":["Over-merge, not missed merges, is the hard problem at 107M identities","Edge cuts break mega-clusters that node thresholds cannot","Rare-but-meaningless attributes weld 107M aliases into mega-clusters","Betweenness plus a per-edge classifier tames global author disambiguation","Node-level gates preserve recall but cannot dissolve mega-clusters","Cutting load-bearing edges, not nodes, fixes million-member clusters","Bad high-quality attributes beat every information metric at scale","Precision-only and recall-only benchmarks invert each other's verdicts","The fix is topology-aware edge filtering, not stricter attribute rules","A 170K-member cluster shrinks to under 7K via edge cuts and a classifier","Structural cuts do not transfer to edge sets they never saw","GitHub no-reply identifiers yield 2.6M free labels for edge classification","Recall rises from 0.44 to 0.70 while precision holds at 107M aliases","Compositional fixes beat any single gate for global identity resolution","Bridge identities, not hubs, hold mega-clusters together at scale"]},"model":"glm-5.2","effort":"low","cost_usd":0.0,"raw_usage":{"total_tokens":1447,"prompt_tokens":683,"completion_tokens":764,"prompt_tokens_details":null},"tokens_in":683,"tokens_out":764,"duration_ms":32619,"temperature":1.0,"reasoning_tokens":553,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T22:53:43.954638+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If the per-edge classifier's transfer to human judgments did not hold at AUC 0.99 on the ALFAA gold set, or if the betweenness cut failed to reproduce its local prediction in the production union, the composed design would lack its two independent validations and the recall-recovery expansion would risk re-welding the mega-cluster.","supporting_citations":[],"review_version":1}