{"id":"13841c7e-814d-4e54-98f0-9816c7b17ad3","arxiv_id":"1908.05354","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Public Git commit metadata can be merged across repositories to build a database of hundreds of thousands of developer identities, bypassing most existing GitHub privacy safeguards.","lead":"This paper builds a tool that harvests email addresses from public GitHub repositories and links them to named developer profiles at scale. It measures how often GitHub's privacy countermeasures fail and proposes new defenses.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Person-matching rule (Section 2.2.2) is unvalidated: merging on 'same name or same email' can join distinct developers, inflating Table 4's compromised rates and the extrapolated ~50% private-email claim. No precision/recall evidence is given.","rationale":"Good faith reading: the paper is a proof-of-concept that public Git metadata can be harvested cheaply. Its own Tables 2–4 provide real measurements, and the linear timing relation is plausible. The qualitative threat—commit emails are visible, noreply adoption is low, and some histories contain both noreply and private addresses—is supported even before the estimates. The problem is that the two headline numbers in §3.3/3.4 are derived through two fragile steps: identity merging and out-of-sample extrapolation. Of these, identity merging is more load-bearing because the 'compromised' label is defined by a merge outcome, and the matching rule has no validation whatsoever. The reader's CONDITIONAL verdict already captures this; I do not see grounds to reject the paper outright, since the underlying measurements and the central observation are not fraudulent or internally inconsistent. The condition should be: validate the matching rule (or bound its false-merge rate), and only then present 34.5%/50% as measurements rather than upper bounds. The extrapolation concern remains secondary: even with perfect matching, the power-law fit in §3.3.2 needs confidence intervals or a held-out run. The manuscript's own caveat that 'we do not know how far this method can be extended meaningfully' supports treating the 650,000 figure as an estimate, not a result.","tokens_in":16529,"tokens_out":9795,"duration_ms":94328,"concrete_test":"Audit a random sample of ~500 person objects from a 55,000-repo run that contain both a noreply address and a non-noreply email. Using the numeric GitHub user ID embedded in the noreply address, GitHub profile names, and repository .mailmap or GPG-signed identities, determine whether the second email belongs to the same actual account. Report the false-merge rate; if it exceeds a few percent, Table 4's compromised column and the ~50% extrapolation are materially inflated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is the person-matching equivalence in Section 2.2.2: 'Two lines that are matching (same name or same email address) come from the same person.' This rule converts raw shortlog lines into the 'persons' database, and Section 3.2.3 defines a noreply user as 'compromised' when their person object also contains a non-noreply address. If two distinct developers share a display name, one using a noreply address and the other using a private address, the merge rule creates a false compromise. Common names produce exactly this pattern, and the false-merge rate will grow as more repositories are added, which is the same shape as Table 4's rise from 8.3% to 34.5%. The paper provides no ground-truth validation: no .mailmap comparison, no manual audit, no precision/recall estimate, only the illustrative Figures 4 and 5. Because the extrapolated '~50%' in Section 3.3.3 inherits this definition, the strongest claim is an upper bound with unknown bias. The person count also suffers from overmerging (distinct people merged) and splitting (one person with multiple names), so the 650,000-person figure is likewise not anchored. Without validation, these are indicators of a possible leak, not measurements of it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper describes an attack pipeline that (1) uses the GitHub Search API to enumerate popular repositories, (2) clones a size-filtered subset and runs git shortlog -sne, (3) merges shortlog lines into person objects using the rule that lines with the same name or the same email belong to the same person, and (4) identifies GitHub noreply addresses that can be linked to other email addresses. The authors present measurements from an implementation named The Monster for repository counts from 100 to 55,000, reporting runtime, database size, and the fraction of noreply users for whom an additional email was found. They then fit regression curves to these seven data points and extrapolate to 7 million repositories, claiming an attacker can build a database of 650,000 persons in one day and that about 50% of noreply users can be linked to a private email address. The paper concludes with countermeasure proposals for Git, GitHub, and users.","tokens_in":16818,"tokens_out":7106,"duration_ms":64059,"significance":"If the measurement claims were fully supported, the paper would document a practical and scalable privacy weakness in the current Git/GitHub identity model, with concrete consequences for phishing risk. The paper's strengths are the direct measurement campaign (Tables 2–4 give concrete data points across seven repository counts) and the concrete, actionable countermeasures (default noreply addresses, push blocking, clone rate limits). However, the central quantitative claims rest on an unvalidated person-matching rule and on extrapolations from a small number of single-run measurements; the headline figures are therefore upper bounds with unknown bias rather than measured rates. The work is a useful measurement study but, as it stands, the strength of the conclusions exceeds the strength of the evidence.","major_comments":[{"comment":"The person-matching rule in §2.2.2 ('Two lines that are matching (same name or same email address) come from the same person') is load-bearing for all person-level statistics, but it is never validated against ground truth. The paper provides no precision/recall estimate, no comparison with .mailmap data, and no manual audit of a sample of merged identities. This matters because §3.2.3 defines a noreply user as 'compromised' when the merged person object also contains any non-noreply address. If two distinct developers share a display name, one using a noreply address and the other using a private address, the merge rule creates a false compromise. The observed rise in the compromised rate from 8.3% at numRepos=100 to 34.5% at numRepos=55,000 is consistent with the accumulation of such false merges as more repositories are added, so the extrapolated ~50% claim in §3.3.3 inherits an unknown positive bias. Please validate the matching rule (e.g., via a sample of known GitHub users' commit histories) and report the resulting precision/recall, or explicitly reframe the compromised percentages as 'potential associations' rather than confirmed leaks.","section":"§2.2.2, §3.2.3, Table 4"},{"comment":"The headline extrapolations (650,000 persons in one day; ~50% of noreply users compromised) are based on regression curves fitted to the seven measured points in Tables 2–4, with no error bars, no confidence intervals, and no cross-validation. The timing model in §3.3.1 (totalTime ≈ 0.1 sec × numRepos) is explicitly network-dependent, and the 40% smallest-repository threshold in §3.1 is an unoptimized free parameter. Because the extrapolation extends to numRepos values 100–10,000 times larger than the largest measured run, small changes in the fitted exponents (e.g., the 0.382 power for totalPersons) produce very large changes in the predicted totals. Please provide uncertainty bounds on every extrapolated figure, report repeat runs (especially at 55,000 repositories) to assess run-to-run variability, and present a sensitivity analysis over the repository-selection parameters.","section":"§3.3.2, §3.3.3, §3.1"},{"comment":"The comparison of the shortlogLines regression to the 'all-github-commit-emails' dataset (5.3 million addresses) is not an independent validation: that dataset was assembled from the full public GitHub corpus, whereas The Monster deliberately selects top-starred repositories and discards the largest 60% by size. The regression curve is fit to data generated under The Monster's selection bias, so the coincidence of the extrapolated value with the 5.3 million figure is a within-model consistency check at best. Please either remove this comparison or clearly label it as a coarse consistency check, not a validation of the extrapolation.","section":"§3.3.2"}],"minor_comments":[{"comment":"The text 'GitHub API The Github API itself...' contains an inconsistent capitalization ('Github' vs 'GitHub'), and the sentence 'This event information will only contain an actual email address if the user has not or just recently activated GitHub’s noreply-email-address service' is unclear; it should state explicitly whether the events endpoint returns the commit email or the user's primary email.","section":"§1.5.1"},{"comment":"The table note 'clone, shortlog and merge added up take longer than step 2 – this is because cloning, shortlog and merging are performed simultaneously' states the sum of component times exceeds the wall-clock time of step 2. Since the components overlap, the sum being larger than the wall-clock time is expected; the note should explain why the overlap does not imply that the sum is a meaningful measure of resource consumption.","section":"§3.2.1, Table 2"},{"comment":"The proposal to change noreply-email-addresses to contain only a numeric ID or a random string would break the existing address format and could leak the ID anyway; a discussion of backward compatibility and of the cost of the extra API lookup per user would strengthen the proposal.","section":"§4.2.2"},{"comment":"The term 'private email address' is used to describe any non-noreply address found for a person; since many developers use public, dedicated commit addresses (e.g., commits@example.com), the measured quantity is not necessarily private. Please adjust the terminology to 'non-noreply email address' or justify the assumption that all such addresses are private.","section":"§3.2.3"},{"comment":"The update in the conclusion (August 14, 2019) mentions that GitHub declined to address the proposals, but gives no details on the disclosure timeline or GitHub's response; a brief responsible-disclosure note would be useful to readers.","section":"Conclusion, Update"}],"recommendation":"major_revision","confidential_remarks":"The paper is a 2019 arXiv preprint; the journal should verify that the measurements and recommendations are still current given changes to GitHub's platform. The core issue (email leakage via commit metadata) remains relevant, but the unvalidated matching rule and extrapolation fragility are significant. I recommend major revision rather than rejection because the direct measurements are a useful contribution and the issues are addressable with additional validation and uncertainty analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real piece of measurement work, not a toy. The authors built a tool that fetches top GitHub repos by stars, clones the smallest 40%, runs git shortlog -sne, and merges lines into person records when name or email matches. The measurements are concrete: 55k repos -> 210k persons in about 1.5 hours, 12% of persons use noreply, and the fraction of noreply users for whom a second address appears rises from 8.3% to 34.5% as more repos are added. That last trend is the most interesting thing in the paper, because it quantifies how a privacy feature degrades as an attacker sees more of a person's commit history.\n\nWhat's new: earlier work like all-github-commit-emails collected raw addresses; this paper adds cross-repo identity merging and tries to measure how often noreply addresses are linked to real ones. That's a modest but real increment. The paper is also honest about the limits of the existing countermeasures and the proposals (e.g. clone rate limits) are sensible, if unsurprising.\n\nWhere it gets shaky: the person-matching rule in Section 2.2.2 is the load-bearing assumption, and it is never validated. 'Same name or same email' merges will create false identities for common names, and those false merges directly inflate Table 4's 'compromised' numbers: if two different developers share a name and one uses a noreply address while the other used a private address, the tool reports a compromise that didn't happen. The paper gives no precision/recall, no .mailmap check, no manual audit. That means the 34.5% and the extrapolated ~50% are upper bounds with unknown bias, not clean measurements. The extrapolations to 7M repos are also soft: they come from power-law fits to seven points and are used to produce the 650k person and 'single day' claims. Those figures should be read as illustrative, not predictive. Single-run measurements without error bars are a minor issue here; the trend is clear enough, but the absence of released code/data makes the quantitative claims harder to check.\n\nOverall, the central qualitative claim holds up. You can harvest commit metadata at scale, and noreply is not as private as users think. The paper deserves a serious referee, but the authors should validate the matching rule, release the tool and data, and soften the extrapolated numbers.","headline":"A credible proof-of-concept that mass identity harvesting from GitHub commit metadata works and that noreply addresses leak under a simple merge rule, but the headline numbers are extrapolations built on an unvalidated matching assumption.","tokens_in":17359,"tokens_out":2277,"would_cite":false,"duration_ms":22364,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that GitHub's public commit metadata lets an attacker build a phishing-ready database of hundreds of thousands of developers in about a day, and that noreply protection fails for many users.","keywords":["git metadata","email harvesting","GitHub API","person matching","phishing","noreply email addresses","privacy","repository metadata exploitation"],"falsifier":"Take a controlled group of, say, 1,000 known developers, record their true name-to-email mapping, let their public repositories be processed by the paper's matching algorithm, and count how often two distinct people are merged or one person is split; if the merge rate is substantial, the 34.5% compromised-noreply figure overstates the privacy breach.","tokens_in":16295,"feed_emoji":"🎣","tokens_out":4927,"duration_ms":47711,"temperature":0.7,"pith_summary":"This paper claims that the email addresses stored in every Git commit are a serious, easy-to-exploit privacy hole when commits are hosted publicly on GitHub. The authors build a scraping tool that fetches popular repositories, clones them, and runs git shortlog, then merges shortlog lines into person profiles using the rule that two lines with the same name or same email belong to one person. With 55,000 repositories they assemble a database of 210,164 people in about 1.5 hours, and they extrapolate that one day of computing yields roughly 650,000 person profiles. They further claim that about 12% of people use GitHub's noreply email service, and that for around half of those the private email can still be recovered from an older commit that used the real address. The paper's main conclusion is that existing countermeasures are insufficient and concrete changes—default noreply addresses, blocked pushes, clone rate limits, and better user education—should be adopted.","feed_headline":"Scraping GitHub commits can unmask 650k developers in a day","feed_subtitle":"Historical commits undo GitHub's noreply privacy for about half of users, enabling targeted phishing.","key_machinery":"The load-bearing mechanism is identity merging by git shortlog output: a shortlog line is (name, email, commit count), and the paper's person matching rule declares two lines to be the same person if either the name or the email matches. Iterated over thousands of repositories, this merges scattered commits into a person object containing all names, all emails, and all contributed repositories. The rule is what turns isolated email harvesting into a dense, phishing-ready profile database, because a single matching line in any repository is enough to attach a private email to a noreply-protected identity.","core_discovery":"The central discovery is that GitHub's own privacy feature is undone by Git's historical design. Every commit permanently records an author name and email; even if a user switches to username@users.noreply.github.com, a single earlier commit containing the real address, when merged into a person profile by the same-name rule, links the noreply identity to the private address. The paper's measurements show 210,164 persons from 55,000 repositories, with 31.6% contributing to at least two repositories, and 34.5% of noreply users compromised in the largest run. Extrapolating with regression curves, the authors estimate 1.36 million persons from all 7 million public repositories with at least one star, and roughly 650,000 persons in one day, with the private address known for about half of noreply users.","pith_inferences":["The person-matching rule has never been validated against known identities; if common names cause false merges, the reported 34.5% compromise rate is an overestimate, and real-world rates could be lower.","The same merging technique would work on any public Git host, not only GitHub, so the attack generalizes to GitLab, Bitbucket, or self-hosted mirrors if their repositories are crawlable.","The repository-coherence graph the paper describes for phishing could also be used defensively, e.g., to flag suspicious accounts that suddenly contribute to many unrelated projects, or to measure developer mobility across projects.","A direct test of the paper's central claim would be to run the pipeline on a cohort with known ground truth (e.g., a set of developers who confirm their own addresses) and compare merged identities against that ground truth."],"forward_implications":["Existing GitHub rate limits and abuse detection do not stop the attack, because after an initial API fetch the heavy work is ordinary git clone operations, which are not throttled.","A user who ever committed with a real email address cannot hide behind noreply afterwards; one leaked commit links all of their public commit activity to that address.","The person-repository graph makes targeted phishing feasible: the attacker can reference real repositories and programming languages, and the paper estimates click-through between 8% and 35%.","Simple defaults—enabling noreply for new accounts, blocking pushes that expose associated addresses, and adding clone rate limits—would reduce the attack's scale even if they cannot eliminate it.","The database grows sublinearly with repositories (roughly 3,310 × numRepos^0.382 persons), so the attack's marginal yield drops as it covers more repositories."],"supporting_citations":[{"why":"Prior exploit that collected over five million GitHub commit email addresses; used as a scale comparison for the raw email data the paper's approach achieves.","marker":"[4]"},{"why":"Documents GitHub API rate limits, which the paper argues do not stop the attack because cloned repositories are not subject to those limits.","marker":"[7]"},{"why":"Describes GitHub's noreply email address feature, the countermeasure whose effectiveness the paper tests and finds incomplete.","marker":"[9]"},{"why":"Introduces the block command line pushes that expose my email option, which the paper evaluates as a partial but insufficient protection.","marker":"[10]"},{"why":"Provides targeted phishing click-rate data from an academic study, used to estimate the success of the paper's proposed phishing attack.","marker":"[14]"},{"why":"Supplies GitHub's official script for rewriting commit author information, which the paper's automatic cleanup proposals rely on.","marker":"[16]"}],"fun_headline_variants":["GitHub noreply privacy half-defeated by commit history","Scraping GitHub metadata exposes 650k private emails","Historical commits leak half of GitHub noreply users","Git commit history breaks GitHub's email privacy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire person database and every compromised statistic rest on the unpublished rule that two shortlog lines with the same name or the same email address always describe the same real person, so distinct people who share a name are silently merged into one identity and one person using different names stays split.","fun_headline_variants_meta":{"raw":{"variants":["GitHub noreply privacy half-defeated by commit history","Scraping GitHub metadata exposes 650k private emails","Historical commits leak half of GitHub noreply users","Git commit history breaks GitHub's email privacy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000652,"raw_usage":{"total_tokens":2957,"prompt_tokens":882,"completion_tokens":2075,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":2012}},"tokens_in":498,"tokens_out":2075,"duration_ms":15621,"temperature":1.0,"reasoning_tokens":2012,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:16:13.932448+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a controlled group of, say, 1,000 known developers, record their true name-to-email mapping, let their public repositories be processed by the paper's matching algorithm, and count how often two distinct people are merged or one person is split; if the merge rate is substantial, the 34.5% compromised-noreply figure overstates the privacy breach.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior exploit that collected over five million GitHub commit email addresses; used as a scale comparison for the raw email data the paper's approach achieves."},{"cited_title":"url: https : / / developer","cited_arxiv_id":null,"evidence_quote":"Documents GitHub API rate limits, which the paper argues do not stop the attack because cloned repositories are not subject to those limits."},{"cited_title":"url: https://github.blog/2013-08-09-keep-your- email-private","cited_arxiv_id":null,"evidence_quote":"Describes GitHub's noreply email address feature, the countermeasure whose effectiveness the paper tests and finds incomplete."},{"cited_title":"url: https : / / github","cited_arxiv_id":null,"evidence_quote":"Introduces the block command line pushes that expose my email option, which the paper evaluates as a partial but insufficient protection."},{"cited_title":"Phishing in an Academic Community: A Study of User Susceptibility and Behavior","cited_arxiv_id":"1811.06078","evidence_quote":"Provides targeted phishing click-rate data from an academic study, used to estimate the success of the paper's proposed phishing attack."},{"cited_title":"url: https://help.github.com/en/articles/changing- author-info","cited_arxiv_id":null,"evidence_quote":"Supplies GitHub's official script for rewriting commit author information, which the paper's automatic cleanup proposals rely on."}],"review_version":1}