Identity Theft in AI Conference Peer Review

Andrew McCallum; Melisa Bok; Nihar B. Shah; Xukun Liu

REVIEW 3 major objections 5 minor 10 references

Identity Theft in AI Conference Peer Review

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper reports that platform staff uncovered 94 reviewer and meta-reviewer profiles created under fake identities to steer favorable reviews to attackers' own papers.

desk verdict A credible alarm about fake reviewer identities at AI conferences, but the paper's core count is an assertion from an interested party with no audit trail. read the letter →

arxiv 2508.04024 v1 pith:DCLD3OUX submitted 2025-08-06 cs.DL cs.AIcs.CR

Nihar B. Shah , Melisa Bok , Xukun Liu , Andrew McCallum This is my paper · ORCID

classification cs.DLcs.AIcs.CR

keywords peerreviewintegrityidentitytheftfakereviewerprofilesAIconferencesfraudemailaliasabuseacademicmisconduct

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that between February and April 2024 and again in April 2025, staff of the nonprofit platform that runs reviews for many top AI conferences uncovered 94 reviewer and meta-reviewer profiles built on fake identities. According to the paper, the people behind these profiles impersonated real researchers—using their affiliations and publication histories plus email addresses at trusted universities that passed verification—to get assigned to review their own submitted papers and write favorable reviews. If the finding is correct, a verified identity-theft attack has been operating inside AI peer review, exploiting the open recruitment forms, self-created reviewer profiles, round-trip email verification, and bidding mechanisms meant to build a qualified review pool. The paper argues the vulnerability is not confined to one venue, because many venues reuse reviewer lists from previous years, so undetected impersonators can be re-invited automatically. Its purpose is to alert organizers and platforms so they can adopt the proposed safeguards.

What carries the argument

The mechanism that makes the fraud work is the unverified reviewer profile combined with a round-trip-verified email alias. The paper's account depends on open-call reviewer recruitment, where program chairs check eligibility from the applicant's self-reported seniority and publication history, and on semi-automated assignment that lets reviewers bid on papers and be matched by similarity. The fake profile supplies the right surface credentials—another researcher's papers and affiliation—while the email alias at a trusted university domain passes the platform's verification check, so the attacker enters the pool as a legitimate reviewer and can then exploit bidding or text-based matching to

What would settle it

An independent audit of the 94 flagged profiles: for each, contact the person whose identity was used and the administrators of the university email domain, and check whether the profile can be traced to an actual dishonest actor. If even a few flagged profiles turn out to be legitimate reviewers misidentified by the internal investigation, or if none of the impersonations can be verified by the affected individuals, the central claim fails.

Watch

Extended reading notes

Core claim

The central claim is that dishonest researchers have created fraudulent reviewer profiles in multiple AI conferences by taking over another researcher's identity—submitting reviewer signup forms or platform profiles with the victim's affiliation and publication record while using an email address the attacker controls. All 94 fake profiles used an email address that passed round-trip verification; 92 drew on email domains of reputable universities and two used `.edu` domains of defunct institutions, in many cases because universities allow members and visitors to generate aliases resembling other people. Once recruited, the attacker bid for or tuned their profile to be assigned to papers aut

Load-bearing premise

The load-bearing premise is that the platform's staff correctly classified all 94 profiles as fraudulent—each one truly created by a dishonest researcher rather than a false positive—because the paper gives no detection methodology, case-level evidence, or independent audit; if that classification errs, the count and the inferred modus operandi collapse.

Editorial extensions

If this is right

If the 94-case finding is taken at face value, conference organizers should treat institutional email domains as weak identity evidence: in the paper's data, 92 of 94 fraudulent profiles used round-trip-verified email addresses from reputable universities.
Automated re-use of previous reviewer lists means an undetected impersonator is not a one-time event; the same fake profile can be re-invited to future editions unless profiles are deduplicated and re-verified.
The paper's proposed fixes—linking reviewer credentials to verified prior publications, requiring login through a persistent identifier system, vouching for new reviewers, and scrutinizing profiles created just before deadlines—follow directly from the observed modus operandi.
Because open-call recruitment is common outside AI, the same vulnerability plausibly affects other fields and platforms, though the paper only documents AI conferences.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's count says nothing about how many papers received fraudulent favorable reviews; knowing that number would determine whether the reported cases are an isolated nuisance or a systematic distortion of accept/reject decisions.
A testable consequence of the email-alias claim is that university alias policies are the strongest single enabler; platforms that require a verified non-alias institutional address or real-name matching should show lower fake-profile rates, and a comparison across platforms could quantify this.
The same loophole applies to recommendation-letter workflows in admissions and hiring, where the applicant controls the recommender's contact information; this is the paper's own analogy, and it suggests the fraud pattern can be detected by checking whether the recommender's email alias and letter metadata match.
The paper proposes transparency in publicizing investigations and outcomes; if adopted, the resulting dataset of confirmed cases could be used to build automated flagging models, but this is an extension the paper does not develop.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Read this paper if you care about peer-review integrity in AI. The headline finding — 94 reviewer/meta-reviewer profiles with fake identities, many using institutional email aliases — is new and important. It is the first platform-level documentation of this specific attack in AI conferences, and the email-alias variant is a genuinely new detail. The recommendations are sensible, and several are already implemented. The authors are candid that these are investigations by OpenReview staff, not the result of a public experiment.

The central number is as soft as it is load-bearing. The findings section states the 94 count without an operational definition of fake identity, no detection criteria, no case-level evidence, and no decision rule. The round-trip-verified email detail shows the controller clicked a link, not that the controller was another named researcher with an author account. That does not undermine the modus operandi — it is plausible and consistent with known collusion tactics — but it means the reader cannot check the classification. For an interested-party report, that is a real limitation. The paper itself does not flag this gap; it presents the count as established.

Two things keep this from being a reason to toss the paper. First, the authors are the platform, with unique access to logs, IP overlaps, and admissions that no outsider has. It is not unreasonable to give them some benefit of the doubt. Second, the practical threat model holds even if the precise 94 is off: the attack is possible, verified-emails do not protect against it, and university alias policies make it worse. The paper's value is in demonstrating a vulnerability, not in the exact numerator.

What is missing is a methodology appendix or an independent audit that could establish the count. Even a redacted list of criteria — "profile created via alias matching the impersonated person's name; account linked to author's IP; self-citation pattern" — would let the community assess the evidentiary standard. Without that, the headline claim remains an assertion, though a credible one.

The citation pattern is fine. Prior collusion-ring work is directly relevant, and the authors' own survey is the right background. Self-citation is not a problem here because the cited work is real and related.

Bottom line: this is a solid, important report that should go through peer review with the methodology made explicit. I would send it to reviewers, not desk reject it, and I would require the authors to add a transparency appendix or explain why that is impossible. It is also worth bringing to a reading group precisely because the evidence-standard question is a good test of how we treat platform self-reports.

Referee Report

3 major / 5 minor

Summary. The paper reports that OpenReview's internal investigation identified 94 reviewer and meta-reviewer profiles with fake identities across several AI conferences in 2024–2025. The alleged modus operandi is that dishonest researchers impersonated real researchers (including via university email aliases) to be recruited as reviewers, then bid for their own papers and wrote favorable reviews. The paper proposes identity-verification and anti-fraud measures for venues, platforms, and universities, and warns that similar vulnerabilities affect recommendation letters and other academic processes.

Significance. If the central factual claim were substantiated, it would document a serious, scalable attack on peer review at major AI venues and would justify the proposed safeguards. The paper's strength is that it publicizes a concrete vulnerability and offers sensible mitigations. However, the empirical core is an assertion by OpenReview-affiliated authors based on an undisclosed investigation; no methodology, detection criteria, case-level evidence, or independent verification is provided. The paper therefore functions at present as a high-level incident alert rather than a verifiable scientific report.

major comments (3)

[Findings of fraud (94-profile count)] The central claim rests entirely on the statement that 'This investigation unearthed 94 reviewer (and meta-reviewer) profiles involving fake identities.' No operational definition of 'fake identity' is given, no detection methodology or decision rule is described, and no case-level evidence or audit trail is presented. A round-trip-verified email address only proves that someone controlled the mailbox; it does not establish that the controller was a specific dishonest researcher or that the profile impersonated a specific real person. Without these details, the 94 count and the modus operandi are unfalsifiable.
[Findings of fraud (assignment/review outcomes)] The abstract claims that the fraudulent profiles were used 'to manipulate paper evaluations,' but the bullets only describe how a dishonest researcher 'attempts to get assigned' and, once assigned, 'provides favorable reviews.' No aggregate or case-level information is reported on how many of the 94 profiles actually received assignments to the target papers, wrote reviews, or influenced decisions. If none or few did, the severity and the 'identity theft' framing would need to be revised. This is load-bearing for the paper's central claim.
[Findings of fraud (email-alias inference)] The statement that 'In all 94 cases, the fake reviewer profiles included a round-trip-verified email' and that 92 addresses pertained to 'reputed universities' is not enough to support the conclusion that the dishonest researcher 'gained access to an email of a trusted institution' and 'created email alias(es) resembling someone else.' The investigation's method for linking an alias to a specific individual and for ruling out legitimate duplicate accounts or misconfigurations is not reported. A clear, reproducible classification protocol and redacted examples are needed.

minor comments (5)

[Overall structure] The paper lacks a Methods or Data Availability section; even the section headings are not formatted as conventional sections. Please add an explicit description of the investigation period, data sources, and anonymization procedures.
[Terminology] Define 'meta-reviewer' and 'reputed universities' for a general scientific audience.
[Recommendations] The recommendations, while sensible, are not prioritized or evaluated; some are said to be already implemented on OpenReview, but no before/after data are given.
[References] References [7] and [10] are informal sources; consider citing published versions or adding access dates.
[Counting clarity] The phrase 'in all 94 cases' could be read as involving 94 emails; please clarify whether the count refers to profiles, unique individuals, or email addresses.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's central claim is an empirical report of an investigation, not a derivation from its own assumptions.

full rationale

The manuscript makes no mathematical derivation and fits no parameters. Its central claim is that OpenReview staff identified 94 reviewer profiles with fake identities across AI conferences (section 'Findings of fraud'). This is presented as an empirical observation, not as a consequence of an assumption or a model. The named mechanisms (bidding, profile tuning) are supported by citations to prior work by Shah et al. ([5], [6], [9]), but those citations are background context and do not generate the 94-count or the modus operandi; the investigation itself is the source. The fact that the investigation was conducted by the authors' own organization and that its decision rules are not disclosed is a serious evidentiary limitation — the count is currently hard to audit — but that is a question of evidence and verifiability, not circularity. No equation, definition, or fitted input is recycled as a prediction. Self-citations are present but not load-bearing, so the paper does not exhibit self-definitional, fitted-input, or imported-uniqueness circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper contributes no mathematical derivations or models; its findings rest on the completeness and accuracy of an internal, undisclosed investigation. The count 94 is an empirical observation, not a fitted parameter. No new physical or formal entities are introduced. The main unstated premises are the reliability of the detection process and the representativeness of the described attack patterns.

assumptions (3)

domain assumption OpenReview staff investigations correctly identified all 94 profiles as fake and attributed them to dishonest researchers.
The entire findings section rests on the accuracy of the internal detection and attribution process, which is not described or independently verified.
domain assumption The reported modus operandi, including form signup with stolen identity and email alias at a trusted institution, is representative of how the fraudulent reviewers operated.
The paper states these patterns as findings but does not provide case-level evidence tying each pattern to the 94 profiles.
domain assumption Round-trip email verification is a meaningful signal of affiliation, and its presence in all 94 cases is accurately recorded.
The 'in all 94 cases' claim requires reliable record-keeping; no audit trail is given.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Identity Theft in AI Conference Peer Review." pith.science (2026). https://pith.science/paper/DCLD3OUX

@misc{pith2026250804024,
  author       = {Pith},
  title        = {Pith review of: Identity Theft in AI Conference Peer Review},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DCLD3OUX}},
  note         = {Machine review of arXiv:2508.04024}
}

read the original abstract

We discuss newly uncovered cases of identity theft in the scientific peer-review process within artificial intelligence (AI) research, with broader implications for other academic procedures. We detail how dishonest researchers exploit the peer-review system by creating fraudulent reviewer profiles to manipulate paper evaluations, leveraging weaknesses in reviewer recruitment workflows and identity verification processes. The findings highlight the critical need for stronger safeguards against identity theft in peer review and academia at large, and to this end, we also propose mitigating strategies.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

10 extracted references · 10 canonical work pages

[1]

August 2023

Aiken, A., Amato, N.M., Bowling, K., De Floriani, L., de Sturler, E., Gini, M., Hanson, V., Krishnamurthy, A., Larson, K., Li, W., Littman, M., Ozcan, F., Russell, M., Sarkar, V., Schwartz, A., Spafford, E.H., and Srivastava, D., Report of the CRA Working Group on Research Integrity. August 2023

work page 2023
[2]

and Webb, A.J., 2016

Cohen, A., Pattanaik, S., Kumar, P., Bies, R.R., De Boer, A., Ferro, A., Gilchrist, A., Isbister, G.K., Ross, S. and Webb, A.J., 2016. Organised crime against the academic peer review system. British Journal of Clinical Pharmacology, 81(6), p.1012

work page 2016
[3]

and Borchardt, G., 2018

Dadkhah, M., Lagzian, M. and Borchardt, G., 2018. Identity theft in the academic world leads to junk science. Science and Engineering Ethics, 24, pp.287-290

work page 2018
[4]

and Oransky, I., 2014

Ferguson, C., Marcus, A. and Oransky, I., 2014. The peer-review scam. Nature, 515(7528), p.480

work page 2014
[5]

Vulnerability of Text-Matching in ML/AI Conference Reviewer Assignments to Collusions

Hsieh J., Raghunathan A., Shah, N., 2024. Vulnerability of Text-Matching in ML/AI Conference Reviewer Assignments to Collusions. arXiv:2412.06606

work page Pith review arXiv 2024
[6]

and Fang, F., 2020

Jecmen, S., Zhang, H., Liu, R., Shah, N., Conitzer, V. and Fang, F., 2020. Mitigating manipulation in peer review via randomized reviewer assignments. Advances in Neural Information Processing Systems, 33, pp.12533-12545

work page 2020
[7]

Collusion rings threaten the integrity of computer science research

Littman M. Collusion rings threaten the integrity of computer science research. Communications of the ACM. 2021 May 24;64(6):43-4

work page 2021
[8]

A Randomized Controlled Trial on Anonymizing Reviewers to Each Other in Peer Review Discussions

Rastogi, C., Song, X., Jin, Z., Stelmakh, I., Daumé III, H., Zhang, K., and Shah, N, 2024. A Randomized Controlled Trial on Anonymizing Reviewers to Each Other in Peer Review Discussions. PLOS ONE. Dec 2024

work page 2024

Show all 10 references

[9]

Challenges, Experiments, and Computational Solutions in Peer Review (Extended Version)

Shah N. Challenges, Experiments, and Computational Solutions in Peer Review (Extended Version). Available online: https://www.cs.cmu.edu/~nihars/preprints/SurveyPeerReview.pdf. Shorter version published in the Communications of the ACM, 65(6), pp.76-87
[10]

T. N. Vijaykumar. Potential Organized Fraud in On-Going ASPLOS Reviews. Blog: https://medium.com/@tnvijayk/potential-organized-fraud-in-on-going-asplos-reviews-874ce14a3e be. Nov 2020

2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

August 2023

Aiken, A., Amato, N.M., Bowling, K., De Floriani, L., de Sturler, E., Gini, M., Hanson, V., Krishnamurthy, A., Larson, K., Li, W., Littman, M., Ozcan, F., Russell, M., Sarkar, V., Schwartz, A., Spafford, E.H., and Srivastava, D., Report of the CRA Working Group on Research Integrity. August 2023

work page 2023

[2] [2]

and Webb, A.J., 2016

Cohen, A., Pattanaik, S., Kumar, P., Bies, R.R., De Boer, A., Ferro, A., Gilchrist, A., Isbister, G.K., Ross, S. and Webb, A.J., 2016. Organised crime against the academic peer review system. British Journal of Clinical Pharmacology, 81(6), p.1012

work page 2016

[3] [3]

and Borchardt, G., 2018

Dadkhah, M., Lagzian, M. and Borchardt, G., 2018. Identity theft in the academic world leads to junk science. Science and Engineering Ethics, 24, pp.287-290

work page 2018

[4] [4]

and Oransky, I., 2014

Ferguson, C., Marcus, A. and Oransky, I., 2014. The peer-review scam. Nature, 515(7528), p.480

work page 2014

[5] [5]

Vulnerability of Text-Matching in ML/AI Conference Reviewer Assignments to Collusions

Hsieh J., Raghunathan A., Shah, N., 2024. Vulnerability of Text-Matching in ML/AI Conference Reviewer Assignments to Collusions. arXiv:2412.06606

work page Pith review arXiv 2024

[6] [6]

and Fang, F., 2020

Jecmen, S., Zhang, H., Liu, R., Shah, N., Conitzer, V. and Fang, F., 2020. Mitigating manipulation in peer review via randomized reviewer assignments. Advances in Neural Information Processing Systems, 33, pp.12533-12545

work page 2020

[7] [7]

Collusion rings threaten the integrity of computer science research

Littman M. Collusion rings threaten the integrity of computer science research. Communications of the ACM. 2021 May 24;64(6):43-4

work page 2021

[8] [8]

A Randomized Controlled Trial on Anonymizing Reviewers to Each Other in Peer Review Discussions

Rastogi, C., Song, X., Jin, Z., Stelmakh, I., Daumé III, H., Zhang, K., and Shah, N, 2024. A Randomized Controlled Trial on Anonymizing Reviewers to Each Other in Peer Review Discussions. PLOS ONE. Dec 2024

work page 2024