Source Code Authorship Attribution Does Not Generalize from Competitions to Classrooms

Marek Horv\'ath; Serhii Yemets

REVIEW 4 major objections 6 minor 18 references

Source Code Authorship Attribution Does Not Generalize from Competitions to Classrooms

T0 review · 4 major / 6 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A fine-tuned code transformer that achieves 70.7% Top-1 accuracy on 1,000 competitive programmers performs at or below chance on real student coursework, indicating that contest benchmarks overestimate practical applicability.

desk verdict Useful, honest negative result—GCJ-trained CodeBERT at chance on real coursework—but the volume confound blunts the title's generalization claim. read the letter →

arxiv 2608.00227 v1 pith:WYSLEUMB submitted 2026-07-31 cs.SE

Serhii Yemets , Marek Horváth This is my paper · ORCID

classification cs.SE

keywords sourcecodeauthorshipattributionstylometryBERTGoogleJameducationaldatabenchmarkgeneralizationtask-holdoutevaluationacademicintegrity

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether high-accuracy source code authorship attribution results, which are almost always measured on competitive programming data, transfer to real educational settings. It fine-tunes the same transformer pipeline on contest submissions and on institutional coursework, and finds a stark contrast: the model reaches 70.7% Top-1 accuracy for 1,000 contest authors, but only 0.2% and 0.06% Top-1 on two student datasets (690 and 812 authors), near or below random baselines. The authors argue that contest benchmarks overestimate practical applicability unless validated on the target coursework context, and they marshal cross-model evidence showing the gap is not specific to one architecture. The paper's contribution is a controlled transfer test that recasts authorship attribution as a dataset-context problem, not just a model-selection problem.

What carries the argument

The central object is the comparison between two data contexts under an identical fine-tuned CodeBERT pipeline with a task-holdout protocol (test tasks are never seen during training for any author). The key mechanism is the chance-baseline contrast: on coursework data the model's Top-1 accuracy lands near or below 1/N, indicating it memorizes training files rather than learning transferable author style. The paper also uses a multi-model benchmark with verification and abstention metrics to show the gap persists across architectures.

What would settle it

Run the same CodeBERT pipeline on a coursework subset where each student has at least 10 submissions (matching the contest setup) with labels independently verified; if Top-1 accuracy rises well above the 0.2% full-pool result, the generalization claim would be weakened. Alternatively, show that randomizing student labels on the same coursework data produces the same near-chance performance, confirming the signal is absent rather than data-limited.

Watch

Extended reading notes

Core claim

On competitive programming data, fine-tuning a pre-trained code transformer (CodeBERT) for authorship attribution scales to 1,000 authors with 70.7% Top-1 accuracy, but applying the identical pipeline to two full-scale institutional coursework datasets yields at-or-below-chance performance: 0.2% Top-1 on closed assignments (690 authors) and 0.06% Top-1 on open-ended projects (812 authors). A character-level CNN shows the same pattern on size-matched subsets, and a broader benchmark across model families confirms the contest-vs-coursework gap. The paper interprets this as evidence that Google Code Jam-based benchmarks overestimate the practical applicability of authorship attribution in educa

Load-bearing premise

The weakest load-bearing premise is that the near-chance coursework results are caused by domain shift rather than by the much smaller per-author sample sizes and possible label errors in coursework data; the paper does not test a matched-volume configuration that would rule those out.

Editorial extensions

If this is right

Published accuracy figures from competitive programming data should not be used to justify deploying attribution tools on coursework without validation on the target data.
At full candidate-pool scale, the examined coursework datasets do not support reliable closed-set attribution; final source files alone appear to be weak evidence.
Per-author code volume matters: reducing contest data to about ten solutions per author drops Top-1 accuracy from roughly 70% to 11-28%, so volume differences explain part of the gap.
Open-ended assignment design alone does not recover signal: the open-project dataset also fell below the random baseline on the full pool.
In educational settings, attribution should only be used as a weak screening aid with mandatory human review, never as standalone evidence of misconduct.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A matched-volume experiment (students with 10+ submissions, labels independently verified) would separate the 'domain shift' claim from a 'data scarcity' or 'label noise' explanation, since the paper's coursework authors have only about three submissions each.
The near-chance Top-1 on coursework might be partly due to label corruption from collaboration or copying, which the paper acknowledges as an upper bound on measurable accuracy; if so, the finding limits supervised attribution in classrooms regardless of model.
The paper's abstention analysis suggests confidence calibration, not raw accuracy, may be the more deployable metric for educational review workflows - a direction the paper only briefly explores.
The negative result motivates shifting from closed-set identification to verification or retrieval over a student's history, which the paper points to as future work.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a useful, honestly reported negative result that deserves referees, but the headline claim is stronger than the design supports. The new thing here is a direct transfer test: the same fine-tuned CodeBERT pipeline, evaluated with task holdout, reaches 70.7% Top-1 on 1,000 most-active Google Code Jam authors and then falls to 0.2% and 0.06% Top-1 on two institutional coursework pools (690 and 812 authors). That measurement is new, and it is a legitimate warning for anyone thinking of deploying GCJ-trained attribution in a classroom.

What the paper does well: the primary comparison is clean in the sense that the same preprocessing, model, and training recipe are used across all configurations. They report Top-10 and random baselines, not just Top-1. They are transparent about the most-active-author selection, the provenance difference between the Kaggle and curated GCJ sets, and the fact that the multi-round vs single-round comparison is descriptive rather than causal. The limitations section explicitly names collaboration/copying label noise, single-institution data, and the unmatched populations. That is honest scholarship.

The soft spots are real, though. The biggest is the per-author volume confound. The coursework datasets have roughly three submissions per author; the curated GCJ sets fix ten, and the multi-round sets select the most active authors. Table 3 shows that within GCJ, going from many solutions per author to ten drops CodeBERT from 70.7% to 11–28% Top-1. That means the near-chance coursework result is compatible with “three short files are not enough signal at pool scale” instead of “classroom code lacks transferable stylometric signal.” The paper acknowledges this but never tests a matched-volume GCJ condition, so the title’s generalization claim is not fully supported. Second, there are no confidence intervals anywhere; the below-chance open-assignment figure (0.06% vs 0.12% random) is almost certainly within sampling noise. Third, the companion cross-model evidence comes from the authors’ own companion preprint and, as they say, is secondary. None of these are fatal; they are fixable with a matched-volume arm, error bars, and a title that says “at the tested data conditions” rather than “does not generalize.”

Who should read it: anyone working on code authorship attribution or academic-integrity tooling. It is a worthwhile empirical contribution even if the mechanism is not isolated. I would send it to referees—preferably one from the stylometry side and one from the educational data mining side—and ask them to focus on the confound. It is the kind of paper where the measurement itself is useful, and the interpretation needs to be pulled back to exactly what the data support.

Referee Report

4 major / 6 minor

Summary. The paper fine-tunes CodeBERT on three provenance classes—public multi-round Google Code Jam (GCJ) data, curated single-round GCJ archives, and two institutional coursework corpora (690 authors / 2,345 closed assignments; 812 authors / 2,487 open assignments)—under a task-holdout protocol. It reports strong scaling on GCJ (92.6% Top-1 for 10 authors; 70.7% Top-1 / 88.2% Top-10 for 1000), but near-chance Top-1 on the full coursework pools (0.2% closed; 0.06% open). A char-CNN baseline and a companion multi-model benchmark are used to argue the gap is not architecture-specific. The paper concludes that GCJ-based benchmarks overestimate practical attribution accuracy in educational settings and recommends validation on target coursework contexts. The novelty is the direct transfer test using one pipeline; the main weakness is the absence of a volume-matched control.

Significance. If the negative result survives a matched-volume control, the paper would be a useful caution to academic-integrity tooling: it would show that competition benchmarks are ecologically misleading for classroom data and that per-author volume, shared scaffolds, and label quality must be reported. Strengths include the explicit same-pipeline comparison, task-holdout evaluation, transparent statement of limitations, pseudonymized institutional data, and the companion benchmark's breadth. However, the conclusion is currently conditional because the coursework/competition comparison is confounded with per-author data volume (≈3 vs 10 tasks) and acknowledged label noise; the paper itself provides evidence that volume alone explains a large part of the GCJ-side drop. The significance is therefore high in potential, but the evidence as presented is not yet conclusive.

major comments (4)

[§3.2/Table 1 and §5.1/Table 3] The central transfer claim is confounded with per-author volume. Coursework has ≈3.4 and ≈3.1 submissions/author (2,345/690, 2,487/812), while GCJ baselines use 10 tasks/author or deliberately select the most active authors. Table 3 already shows volume matters on the competitive side: 10 tasks/author drops Top-1 to 27.9% and 11.3% from 70.7%. Without a GCJ configuration at ~3 tasks/author and comparable pool size, the near-chance coursework numbers are equally explained by scarcity alone. The companion 5-vs-15-file comparison does not include a 3-file arm and is not run with CodeBERT. Add a matched-volume GCJ arm or restrict the conclusion to 'at the observed per-author volume.'
[§5.2/Tables 3–4] The negative result lacks uncertainty quantification. Closed-set Top-1 0.2% versus a 0.14% random baseline (Top-10 1.6% vs 1.45%) is a difference of only a few test samples; open-set 0.06% is below its 0.12% baseline. A single fixed split cannot separate near-chance from at-chance, and fine-tuning stochasticity is unquantified. Report per-author train/test task counts, total test samples, and bootstrapped CIs or repeated-seed ranges for all primary Table 3–4 numbers.
[§5.5 and §7] The acknowledged label noise (collaboration/copying/external help) is not a minor caveat: corrupted labels set an upper bound on measurable accuracy and can produce chance-level performance even when stylometric signal exists. The manuscript should either provide evidence about label quality (manual audit, consistency checks, or plagiarism analysis) or explicitly restrict the claim to 'accuracy under unverified submission labels,' rather than imply that student code lacks signal.
[§5.3–5.4/Table 5] RQ3 asks whether the behavior is architecture-specific, but the main cross-model evidence comes from the companion preprint [17], whose model configurations, hyperparameters, and split definitions are not self-contained in this paper. The primary char-CNN results cover only size-matched subsets. Either include the companion benchmark's essential methodological details and configuration inventory, or demote RQ3 and the 'persists across model families' conclusion to clearly secondary evidence.

minor comments (6)

[Abstract and §5.2] 'At or below corresponding chance baselines' is inaccurate for the closed dataset: Table 4 reports Top-1 0.2% > 0.14% and Top-10 1.6% > 1.45%. Use 'near chance' for the closed dataset.
[Figure 3] The figure uses '6·10^{-2}' for 0.06%; use decimal notation for readability.
[§5.2] The text mentions an earlier 100-author run reaching 11.3% Top-1; this result appears nowhere in the tables or figures. Add it to an appendix or remove.
[§4.1] The text reports 3–5 epochs without per-configuration detail; add epochs to the tables or a footnote.
[§5.5] Some items labeled 'Directly documented dataset properties'—task size, shared templates, style maturity—are asserted rather than directly measured in this study. Consider renaming to 'Observed/assumed dataset properties.'
[Table 5] Add a column indicating whether each configuration comes from the primary experiments or the companion benchmark, since the text relies on both.

Circularity Check

1 steps flagged · score 2.0 of 10

No derivation-level circularity: the central CodeBERT coursework result is independently run under task holdout. The only self-referential element is the same-author companion benchmark [17], which backs the secondary cross-model claim and is transparently labeled non-independent.

self citation load bearing [Section 5.4 (Companion Extended Benchmark), echoed in Abstract and Section 6]
"The following analysis is secondary evidence drawn from the companion benchmark [17]. Its role is to assess whether the pattern observed in the primary CodeBERT experiments persists across additional dataset configurations and model families, not to serve as an independent replication."

The paper's cross-model generalization claim ('not specific to CodeBERT and persists across the evaluated model families') is supported by importing Tables 5-7 from [17], a same-author arXiv preprint, rather than by experiments fully reported in this paper. This is a self-citation chain rather than a derivation-level circularity; moreover, the paper explicitly labels it secondary and the central CodeBERT comparison is internally generated, so the main negative result does not reduce to the citation.

full rationale

The paper has no formal derivation chain: the load-bearing claim is an empirical comparison. CodeBERT is fine-tuned separately on GCJ and on the coursework sets under a task-holdout protocol, and the near-chance coursework accuracy (0.2% and 0.06% Top-1) is an internally obtained result, not a fitted parameter renamed as a prediction. The comparison is confounded by per-author volume and label noise, and the paper acknowledges this in Section 7 ('a perfectly matched comparison is impossible because the contest and coursework populations differ in more than one dimension'; labels may be affected by collaboration/copying). That is a validity threat to the generalization title, but not circularity. The only notable self-referential element is the companion benchmark [17], a same-author preprint used for RQ3 and the cross-model robustness claim; it is disclosed as secondary evidence and does not carry the central CodeBERT finding. Accordingly the paper is largely self-contained on its primary claim, with one minor same-author citation on a secondary axis.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim is an empirical measurement rather than a derivation, so there are no fitted equations. However, several dataset-construction choices and domain assumptions carry the interpretation: the most-active-author selection inflates GCJ accuracy, the coursework labels are treated as ground truth despite acknowledged noise, and the cross-model generalization relies on a self-cited companion preprint.

free parameters (3)

Most-active-author selection threshold (10/500/1000) = 10, 500, 1000
Multi-round GCJ scaling configurations are built by selecting the N most active authors, deliberately creating a high-volume, favorable benchmark (Section 3.1); this choice directly sets the high GCJ accuracy numbers.
Solutions per author in curated GCJ (10) = 10
Curated single-round configs fix ten solutions per author as a compromise between training cost and accuracy (Section 3.1); affects the 11-28% single-round Top-1 results.
Training epochs (3-5) = 3-5
Hyperparameters tuned modestly per configuration (Section 4.1); the paper admits more aggressive optimization could improve coursework performance (Section 7).

assumptions (4)

domain assumption Task-holdout with per-author disjoint tasks measures attribution generalization
Used throughout Section 4.2; if task overlap leaks task semantics, the comparison would be invalid.
domain assumption Coursework submission labels correspond to true authorship
Needed to interpret low Top-1 as lack of signal; Section 5.5 and Section 7 acknowledge collaboration/copying/outside help can break the label-author correspondence.
ad hoc to paper Companion benchmark [17] correctly implements the same protocol across model families
RQ3 cross-model evidence and most of Section 5.4 rest on this self-cited preprint by the same corresponding author; not independently verified in this manuscript.
domain assumption CodeBERT fine-tuning is a representative deep attribution pipeline
The paper's primary evidence is one model; cross-model evidence is secondary (Section 4.1).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Source Code Authorship Attribution Does Not Generalize from Competitions to Classrooms." pith.science (2026). https://pith.science/paper/WYSLEUMB

@misc{pith2026260800227,
  author       = {Pith},
  title        = {Pith review of: Source Code Authorship Attribution Does Not Generalize from Competitions to Classrooms},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WYSLEUMB}},
  note         = {Machine review of arXiv:2608.00227}
}

read the original abstract

Source code authorship attribution aims to identify the author of a program fragment from its writing style. We fine-tune the pre-trained transformer CodeBERT on three sources of data: publicly available Google Code Jam (GCJ) submissions from an open Kaggle repository, a curated GCJ archive, and institutional coursework datasets collected at a technical university. On multi-round GCJ data, CodeBERT reaches 92.6% Top-1 accuracy for 10 authors and retains 70.7% Top-1 (88.2% Top-10) for 1000 authors. On the examined coursework datasets, the same pipeline performs at or below the corresponding chance baselines: 0.2% Top-1 on a closed-assignment dataset of 690 authors and 0.06% Top-1 on open-ended assignments evaluated over 812 authors. A companion multi-model benchmark provides consistent cross-model evidence across additional dataset configurations, indicating that the observed performance gap is not specific to CodeBERT and persists across the evaluated model families. We analyze dataset and task properties that plausibly explain this gap and argue that GCJ-based benchmarks overestimate the practical applicability of authorship attribution in educational settings unless they are validated on the target coursework context.

Figures

Figures reproduced from arXiv: 2608.00227 by the authors.

**Figure 1.** CodeBERT accuracy on multi-round GCJ data as the number of candidate authors grows. 70%, while Top-10 accuracy remains near 88%. Given random baselines of 0.2% and 0.1% respectively, the model performs far above chance and reliably narrows 1000 candidates down to a shortlist of ten. The lower half of [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗

**Figure 3.** CodeBERT on the full-scale coursework datasets compared with random Top-1 baselines. sizes it achieves only 17.6%, 11.9%, and 7.7%. In the companion benchmark ( [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

**Figure 4.** Char-CNN Top-1 accuracy on size-matched competitive (GCJ 2021) and student subsets. performing model for each configuration [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

Figures from the paper (1 more)

**Figure 5.** Figure 5: Representative companion-benchmark Top-1 accuracy [17]. The companion evidence is consistent with the primary CodeBERT comparison, but it also shows that the coursework side is not uniform. The two full assignment datasets approach the chance baseline, whereas some sm…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 2 linked inside Pith

[17]

Horváth M., Evaluating Static and Process Evidence for Code Authorship in Programming Education, arXiv preprint arXiv:2607.07400, 2026

arXiv 2026
[1]

Feng F., Guo X., Tang D., Duan N., Feng X., Gong M., et al., CodeBERT: A Pre-Trained Model for Pro- gramming and Natural Languages, In: Findings of the Association for Computational Linguistics: EMNLP 2020, 2020

2020
[2]

Caliskan-Islam A., Harang R., Liu A., Narayanan A., Voss C., Yamaguchi F., et al., De-anonymizing Pro- grammers via Code Stylometry, In: Proceedings of the 24th USENIX Security Symposium, 2015

2015
[3]

Abuhamad M., AbuHmed T., Mohaisen A., Nyang D., Large-Scale and Language-Oblivious Code Au- thorship Identification, In: Proceedings of the 2018 ACM SIGSAC Conference on Computer and Commu- nications Security (CCS), 2018

2018
[4]

10492, 2017

Alsulami B., Dauber E., Harang R., Mancoridis S., Greenstadt R., Source Code Authorship Attribution Using Long Short-Term Memory Based Networks, In: Computer Security - ESORICS 2017, LNCS vol. 10492, 2017

2017
[5]

Kalgutkar V., Kaur R., Gonzalez H., Stakhanova N., Matyukhina A., Code Authorship Attribution: Meth- ods and Challenges, ACM Computing Surveys, 2019, 52(1)

2019
[6]

Bogomolov E., Kovalenko V., Rebryk Y., Bacchelli A., Bryksin T., Authorship Attribution of Source Code: A Language-Agnostic Approach and Applicability in Software Engineering, In: Proceedings of ESEC/FSE, 2021 12

2021
[7]

Dauber E., Caliskan A., Harang R., Shearer G., Weis- man M., Nelson F., et al., Git Blame Who? Stylistic Authorship Attribution of Small, Incomplete Source Code Fragments, Proceedings on Privacy Enhancing Technologies, 2019, 2019(3)

2019

Show all 18 references

[8]

Frantzeskou G., Stamatatos E., Gritzalis S., Katsikas S., Effective Identification of Source Code Authors Using Byte-Level Information, In: Proceedings of the 28th International Conference on Software Engineer- ing (ICSE), 2006

2006
[9]

Burrows S., Uitdenbogerd A.L., Turpin A., Compar- ing Techniques for Authorship Attribution of Source Code, Software: Practice and Experience, 2014, 44(1)

2014
[10]

Krsul I., Spafford E.H., Authorship Analysis: Identify- ing the Author of a Program, Computers and Security, 1997, 16(3)

1997
[11]

Devlin J., Chang M.-W., Lee K., Toutanova K., BERT: Pre-training of Deep Bidirectional Transform- ers for Language Understanding, In: Proceedings of NAACL-HLT 2019, 2019

2019
[12]

Zhang X., Zhao J., LeCun Y., Character-level Con- volutional Networks for Text Classification, In: Ad- vances in Neural Information Processing Systems 28 (NeurIPS), 2015

2015
[13]

Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A.N., et al., Attention Is All You Need, In: Advances in Neural Information Processing Systems 30 (NeurIPS), 2017

2017
[14]

jur1cek, Google Code Jam dataset, Kaggle dataset, https://www.kaggle.com/datasets/jur1cek/gcj- dataset (accessed 2026); Google, Coding Competi- tions Archive: Code Jam and Kick Start, GitHub repository, https://github.com/google/coding- competitions-archive (accessed 2026)

2026
[15]

Horváth M., Pietriková E., Spinellis D., Bridging Be- havioral Biometrics and Source Code Stylometry: A Survey of Programmer Attribution, arXiv preprint arXiv:2603.11150, 2026

2026
[16]

18267/j.aip.283

Horváth M., Pietriková E., Gurbáľ F., et al., Person- alized Learning Analytics Through Static Code Anal- ysis in Computer Science Education, Acta Informat- ica Pragensia, 2026, 15(1), 54–71, https://doi.org/10. 18267/j.aip.283

2026
[18]

Horváth M., Pietriková E., Evaluating Semantic and Quality-AwareRetrievalforSourceCodeRepositories, arXiv preprint arXiv:2607.09161, 2026

2026 arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.

[1] [17]

Horváth M., Evaluating Static and Process Evidence for Code Authorship in Programming Education, arXiv preprint arXiv:2607.07400, 2026

arXiv 2026

[2] [1]

Feng F., Guo X., Tang D., Duan N., Feng X., Gong M., et al., CodeBERT: A Pre-Trained Model for Pro- gramming and Natural Languages, In: Findings of the Association for Computational Linguistics: EMNLP 2020, 2020

2020

[3] [2]

Caliskan-Islam A., Harang R., Liu A., Narayanan A., Voss C., Yamaguchi F., et al., De-anonymizing Pro- grammers via Code Stylometry, In: Proceedings of the 24th USENIX Security Symposium, 2015

2015

[4] [3]

Abuhamad M., AbuHmed T., Mohaisen A., Nyang D., Large-Scale and Language-Oblivious Code Au- thorship Identification, In: Proceedings of the 2018 ACM SIGSAC Conference on Computer and Commu- nications Security (CCS), 2018

2018

[5] [4]

10492, 2017

Alsulami B., Dauber E., Harang R., Mancoridis S., Greenstadt R., Source Code Authorship Attribution Using Long Short-Term Memory Based Networks, In: Computer Security - ESORICS 2017, LNCS vol. 10492, 2017

2017

[6] [5]

Kalgutkar V., Kaur R., Gonzalez H., Stakhanova N., Matyukhina A., Code Authorship Attribution: Meth- ods and Challenges, ACM Computing Surveys, 2019, 52(1)

2019

[7] [6]

Bogomolov E., Kovalenko V., Rebryk Y., Bacchelli A., Bryksin T., Authorship Attribution of Source Code: A Language-Agnostic Approach and Applicability in Software Engineering, In: Proceedings of ESEC/FSE, 2021 12

2021

[8] [7]

Dauber E., Caliskan A., Harang R., Shearer G., Weis- man M., Nelson F., et al., Git Blame Who? Stylistic Authorship Attribution of Small, Incomplete Source Code Fragments, Proceedings on Privacy Enhancing Technologies, 2019, 2019(3)

2019