REVIEW 3 major objections 8 minor 1 cited by
Evaluating Static and Process Evidence for Code Authorship in Programming Education
T0 review · 3 major / 8 minor · reviewed 2026-07-09 · glm-5.2
Pith's one-line read Commit timing beats code style for identifying student programmers
desk verdict Process features (commit timing, deadline distance, activity intensity) consistently improve educational code authorship attribution across six matched comparisons; contest accuracy does not transfer to course settings. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Task-aware cross-assignment evaluation protocol and matched static vs. static-process educational comparisons
What would settle it
If process features were tested in a course with mandatory auto-formatting and prescribed commit schedules, and they no longer improved attribution beyond static-only code, the central claim that process evidence complements weak final-code signal would be weakened. Alternatively, if the process-feature improvement disappeared when labels were independently verified for sole authorship (removing noise from collaboration or copied work), it would suggest the signal was capturing label noise rather than auth
Extended reading notes
Core claim
The paper's central finding is that repository-derived process features—commit timing, deadline distance, change volume, and activity intensity—provide the largest single contribution to author-related signal in educational programming repositories, and they do so consistently across all six matched comparisons. In the detailed feature analysis of one selected dataset, removing process metrics caused the largest performance drop (top-1 from 0.286 to 0.163), and nine of the ten highest-ranked individual features described repository activity rather than static code properties. This means the timing and rhythm of how a student works in a repository, not the style of the code they submit, is a
Load-bearing premise
The paper assumes that pseudonymous student labels derived from repository ownership or submission records are a valid proxy for authorship identity, and that repository-visible commit history reflects actual development behavior. If commit patterns are heavily shaped by course infrastructure (e.g., mandatory commit workflows) or if labels are noisy due to untracked collaboration, the measured process-feature signal could reflect repository-usage conventions rather than auth
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper investigates source code authorship attribution and verification in educational programming repositories, contrasting performance with programming-contest datasets. The central empirical contribution is a set of six matched comparisons showing that repository-visible process features (commit timing, deadline distance, change volume, activity intensity) improve mean top-1 attribution from 0.094 to 0.233 and mean pairwise verification ROC-AUC from 0.556 to 0.752. The study employs a task-aware cross-assignment evaluation protocol to prevent assignment leakage, fits preprocessing within folds, and includes feature-removal analyses to identify which feature groups drive the observed gains. The paper is methodologically careful and appropriately conservative in its claims, framing results as instructor-mediated decision support rather than autonomous authorship identification.
Significance. The paper makes a valuable contribution to the code authorship literature by systematically examining the educational setting, which is underrepresented compared to contest and open-source benchmarks. The task-aware evaluation protocol is a genuine strength: it directly addresses the assignment-leakage confound that plagues naive cross-validation in educational contexts. The six matched comparisons provide replication across three academic years and two candidate-set sizes, strengthening the central claim. The feature-removal analysis (Table 12) and the task-adjusted information score (Eq. 4) provide interpretable diagnostics. The public release of the cleaned contest dataset (Zenodo) and synthetic dataset is commendable and supports reproducibility. The paper is also appropriately bounded in its claims, consistently framing results as context-dependent rather than universal.
major comments (3)
- §5.3, Eq. (4) and §6.6: The task-adjusted information score I(f;L) − I(f;T) is used to assess whether features carry more information about labels than tasks. However, this score cannot distinguish individual-level from section- or cohort-level patterns. As acknowledged in §7.2, features like 'evening activity' or 'time-of-day activity' could reflect section scheduling rather than individual work habits. If students in different sections face different deadline structures or lab times, process features could encode section membership, which would correlate with student labels without reflecting individual behavioral regularities. This is load-bearing for the central claim that process features capture 'author-related signal' (§2.3). The paper should add an explicit analysis or at minimum a quantitative bound on how much of the process-feature signal could be explained by section or lab-组
- Table 8 and §6.2: The six matched comparisons show consistent improvement from process features, but no statistical significance testing is reported. With only six comparisons and absolute gains ranging from 0.098 to 0.181, it would strengthen the claim to report a simple paired test (e.g., Wilcoxon signed-rank) or at least a bootstrap confidence interval on the mean gain. The consistency across years is noted qualitatively, but the paper's central quantitative claim ('mean top-1 from 0.094 to 0.233') would benefit from a formal uncertainty estimate.
- §6.6, Table 12: The feature-removal analysis is conducted on a single educational dataset (651 records, 100 labels, 7 assignment groups). While the paper acknowledges this limitation in §7.4 ('The relative importance of these groups is specific to the analyzed educational static-process dataset'), the group-removal results are used to support the general claim that process features provide the largest contribution (§7.4, RQ4). Since the six matched comparisons in Table 8 span three years and two candidate-set sizes, but the feature analysis covers only one configuration, the generalizability of the feature-ranking claim is not established. At minimum, the paper should clarify that RQ4 is answered for one configuration only and should not be read as a general finding about educational datasets.
minor comments (8)
- §4.1: The course subsets in Table 4 are anonymized, but no information is given about whether different sections or tutorial groups exist within courses. If section information is available, it should be mentioned even if not used in the analysis, as it bears on the confound discussed above.
- Table 3: The educational row reports '–' for participant records, which is understandable, but the table caption or a footnote should clarify that candidate-label filters are applied later in derived datasets (this is mentioned in the text but not in the table itself).
- §5.4: The random forest uses 400 trees and 'balanced class weights' with a 'fixed random state.' The specific random state value should be reported for full reproducibility, or the paper should state that results are averaged over multiple seeds.
- Figure 1: The hatching/fill distinction is a good accessibility choice. However, the y-axis starts at 0, which compresses the visual differences. Consider whether a broken axis or annotated differences would improve readability.
- §6.4: The sequential model comparison reports that 'longer training improved average sequential results' but does not specify the final number of epochs used for the reported results. This should be clarified.
- Table 11: The educational rows show both static-process and static-only results for 2022 and 2024, but not for 2021. If 2021 verification results are available, they should be included for completeness; if not, a brief note explaining the omission would help.
- §8.1: The statement 'No formal ethics-board review was obtained for this retrospective analysis' is noted. While the paper explains the rationale (retrospective, pseudonymized, aggregate), some readers may find the absence of IRB review notable. Consider acknowledging this as a limitation more prominently or confirming that institutional policy does not require it for this type of analysis.
- References: The paper cites work from 2025 and 2026, suggesting it is very recent. Ensure that all cited works are publicly available or provide accessible preprint links where possible.
Simulated Author's Rebuttal
We thank the referee for the careful and constructive review. The referee correctly identifies the task-aware evaluation protocol, the six matched comparisons, and the bounded framing of claims as strengths of the paper. We address each major comment below. We agree with two of the three comments in full and will revise accordingly; on the third (section-level confounding of process features), we agree the concern is legitimate and will add a quantitative analysis where the data permit it, while being transparent about what cannot be resolved with the available data.
read point-by-point responses
-
Referee: §5.3, Eq. (4) and §6.6: The task-adjusted information score I(f;L) − I(f;T) cannot distinguish individual-level from section- or cohort-level patterns. Features like 'evening activity' could reflect section scheduling rather than individual work habits. The paper should add an explicit analysis or at minimum a quantitative bound on how much of the process-feature signal could be explained by section or lab-group membership.
Authors: This is a legitimate and important concern. We agree that the task-adjusted information score I(f;L) − I(f;T) does not by itself separate individual-level from section-level signal. If students in different sections face different deadline structures or lab times, process features could encode section membership, which would correlate with student labels without reflecting individual behavioral regularities. This is directly relevant to the central claim that process features capture author-related signal. We will add a quantitative analysis in the revised manuscript. Specifically, we will compute the mutual information between each process feature and the section/lab-group variable, and compare it to I(f;L). This will provide a bound on how much of the label-associated signal in process features could be explained by section membership. We will also add a section-level stratified analysis where the data permit it: if multiple sections are present within a single academic year and candidate set, we can test whether process features retain predictive value when section is controlled for. However, we must be transparent about a limitation: the educational dataset was collected from institutional repositories where section metadata is not uniformly recorded across all academic years and course subsets. For configurations where section information is available, we will report the analysis; where it is not, we will state this explicitly as a standing limitation rather than claiming the confound has been fully ruled out. The manuscript text in §7.2 and §8.1 will be revised to incorporate this analysis and to sharpen the distinction between 'author-related signal' and 'section-correlated signal.' revision: partial
-
Referee: Table 8 and §6.2: No statistical significance testing is reported for the six matched comparisons. With only six comparisons and absolute gains ranging from 0.098 to 0.181, a paired test or bootstrap confidence interval on the mean gain would strengthen the central quantitative claim.
Authors: The referee is correct. With six matched comparisons showing consistent improvement, a formal uncertainty estimate is appropriate and straightforward to add. We will report a Wilcoxon signed-rank test on the six paired top-1 differences (static vs. static-process), and a bootstrap 95% confidence interval on the mean gain. We will also report the same for the verification AUC comparisons where matched pairs are available (2022 and 2024). We agree that the central quantitative claim ('mean top-1 from 0.094 to 0.233') benefits from a formal uncertainty estimate, and we will add this to Table 8 or the accompanying text in §6.2. We note that with n=6, the Wilcoxon test has limited power, but the consistency of direction (6/6 improvements) is itself informative, and the confidence interval on the mean gain will convey the appropriate uncertainty. revision: yes
-
Referee: §6.6, Table 12: The feature-removal analysis is conducted on a single educational dataset (651 records, 100 labels, 7 assignment groups). The group-removal results are used to support the general claim that process features provide the largest contribution (§7.4, RQ4), but the feature analysis covers only one configuration. The paper should clarify that RQ4 is answered for one configuration only and should not be read as a general finding about educational datasets.
Authors: The referee is correct that the feature-removal analysis covers only one configuration and that the RQ4 finding should not be read as a general claim about all educational datasets. The manuscript already acknowledges this in §7.4 ('The relative importance of these groups is specific to the analyzed educational static-process dataset') and in the Table 12 caption ('It does not imply that the same ranking will hold across courses, assignments, or repository policies'). However, we agree that the framing of RQ4 in §7.4 could still be read as making a broader claim than the data support. We will revise the RQ4 discussion to state explicitly that the feature-ranking result is established for one configuration only and that generalizability across courses, years, and institutions is not claimed. We will also add a sentence noting that running the feature-removal analysis on additional matched configurations would strengthen the RQ4 claim and is a direction for future work. The abstract and conclusion will be checked to ensure they do not overstate the scope of the feature-analysis finding. revision: yes
- The section-level confounding analysis (Comment 1) can only be conducted where section metadata is available in the educational dataset. Section information is not uniformly recorded across all academic years and course subsets in the institutional repository. For configurations where section data exist, we will report the analysis; for those where it does not, we cannot provide a quantitative bound and will state this as a limitation. We cannot fully resolve this confound for all six matched comparisons.
Circularity Check
No circularity found: process features are derived from repository metadata, not from author labels, and the central claim is supported by matched comparisons with independent evaluation.
full rationale
The paper's central claim is that repository-visible process features (commit timing, deadline distance, change volume, activity intensity) add complementary author-related signal beyond final source code in educational programming repositories. This claim is tested through six matched comparisons (Table 8) where the only variable changed is the inclusion of process features, with the same academic years, candidate-set sizes, and evaluation protocol. The process features are derived from repository metadata (commit timestamps, line counts, deadline distances) rather than from author labels, so the improvement from 0.094 to 0.233 mean top-1 is not tautological. The feature-removal analysis (Table 12) provides independent corroborating evidence by showing that removing process features causes the largest performance drop. The task-adjusted information score I(f;L) - I(f;T) (Eq. 4) is a diagnostic that checks whether features are more associated with labels than tasks; it is not used to define or fit the process features themselves. No parameter is fitted to a subset of data and then 'predicted' on a closely related quantity. No self-citation chain is load-bearing for the central claim: the paper cites external prior work (Caliskan-Islam et al. 2015, Abuhamad et al. 2019/2021, White and Sprague 2021, etc.) for context and motivation, but the empirical results are self-contained, derived from the described datasets, feature extraction pipeline, and evaluation protocol. The single author (Marek Horváth) does not appear in the cited references, so there is no self-citation at all. The derivation chain—from repository metadata to process features to matched static-vs-static-process comparisons—is not circular by construction. The paper's acknowledged limitations (label noise, confounding from course infrastructure, section-level patterns) are correctness risks, not circularity: they concern whether the measured signal reflects individual authorship or cohort-level patterns, but they do not make the result equivalent to its inputs by definition.
Assumptions & free parameters
free parameters (5)
- Random forest: 400 trees =
400
- Logistic regression: max 4000 iterations =
4000
- TF-IDF: count repetition cap =
unspecified
- Score-level combination weight =
0.5 (equal weight)
- Correlation threshold for feature removal =
|r| >= 0.85
assumptions (4)
- domain assumption Student labels derived from repository ownership or submission records are a valid experimental proxy for authorship identity, even though they are not independently verified sole authorship.
- domain assumption Repository-visible commit history reflects meaningful aspects of development behavior, not solely artifacts of course infrastructure or commit policy.
- domain assumption Task-aware cross-assignment splitting sufficiently controls for assignment-specific structure so that measured performance reflects transferable author signal rather than task recognition.
- domain assumption Random forest with balanced class weights is an appropriate and stable classifier for heterogeneous tabular code features across all evaluated datasets.
Cite this review
Pith. "Pith review of Evaluating Static and Process Evidence for Code Authorship in Programming Education." pith.science (2026). https://pith.science/paper/JJCE7BTU
@misc{pith2026260707400,
author = {Pith},
title = {Pith review of: Evaluating Static and Process Evidence for Code Authorship in Programming Education},
year = {2026},
howpublished = {\url{https://pith.science/paper/JJCE7BTU}},
note = {Machine review of arXiv:2607.07400}
}
read the original abstract
In programming courses, instructors may need to interpret whether a submission is consistent with a student's prior programming profile, especially when code similarity alone is inconclusive. Existing source-code authorship methods are often evaluated on programming-contest or open-source datasets, where reusable templates and local code patterns can produce strong author-related signal. Educational repositories present a different setting. Students solve shared assignments while their programming practices are still developing. This study uses task-aware evaluation to contrast these production contexts and tests whether repository-visible process features add information beyond final code in six matched educational comparisons. Contest data provide a high-signal contrast, with a Kick Start mean top-1 of 0.938. Educational datasets produce substantially lower attribution performance. Adding process features raises the educational mean from 0.094 to 0.233 and mean pairwise verification ROC-AUC from 0.556 to 0.752. The comparisons show that measured signal depends on production context and that process patterns can complement weak final-code signal in educational repositories. Such models are therefore appropriate only as instructor-mediated decision support, not as independent proof of authorship.
Figures
Forward citations
Cited by 1 Pith paper
-
Source Code Authorship Attribution Does Not Generalize from Competitions to Classrooms
A fine-tuned CodeBERT classifier reaches 70.7% Top-1 on 1,000 Code Jam authors but 0.2% or less on real coursework submissions, showing that competition benchmarks do not transfer to classrooms.
Reference graph
Works this paper leans on
-
[1]
doi:10.1109/TDSC.2021.3138700 Mohammed Abuhamad, Tamer Abuhmed, David Mohaisen, and Daehun Nyang
Dataset Characteristics for Reliable Code Authorship Attribution.IEEE Transactions on Dependable and Secure Computing20, 1 (2023), 506–521. doi:10.1109/TDSC.2021.3138700 Mohammed Abuhamad, Tamer Abuhmed, David Mohaisen, and Daehun Nyang
-
[2]
Large-scale and Robust Code Authorship Identification with Deep Feature Learning.ACM Transactions on Privacy and Security24, 4 (2021), 1–35. doi:10.1145/3461666 Mohammed Abuhamad, Ji-su Rhim, Tamer AbuHmed, Sana Ullah, Sanggil Kang, and DaeHun Nyang
-
[3]
doi:10.1016/ j.future.2018.12.038 Stefano Balla, Maurizio Gabbrielli, and Stefano Zacchiroli
Code authorship identification using convolutional neural networks.Future Generation Computer Systems95 (2019), 104–115. doi:10.1016/ j.future.2018.12.038 Stefano Balla, Maurizio Gabbrielli, and Stefano Zacchiroli
work page 2019
-
[4]
Code stylometry vs formatting and minification.PeerJ Computer Science10 (2024), e2142. doi:10.7717/PEERJ-CS.2142 Aylin Caliskan-Islam, Richard Harang, Andrew Liu, Arvind Narayanan, Clare Voss, Fabian Yamaguchi, and Rachel Greenstadt
-
[5]
doi:10.7717/PEERJ-CS.2429 Sepideh HajiHosseinKhani, Bhaskar Joshi, and Arash Habibi Lashkari
Stylometry for real-world expert coders: a zero-shot approach.PeerJ Computer Science10 (2024), e2429. doi:10.7717/PEERJ-CS.2429 Sepideh HajiHosseinKhani, Bhaskar Joshi, and Arash Habibi Lashkari
-
[6]
AuthAttLyzer-V2: Unveiling Code Authorship Attribution Using Enhanced Ensemble Learning Models and Generating Benchmark Dataset. InData Science (Communi- cations in Computer and Information Science, Vol. 2253). Springer, Cham, Switzerland, 148–165. doi:10.1007/978-3-031- 85856-7_13 Xie He, Arash Habibi Lashkari, Nikhill Vombatkere, and Dilli Prasad Sharma
-
[7]
Authorship Attribution Methods, Challenges, and Future Research Directions: A Comprehensive Survey.Information15, 3 (2024),
work page 2024
-
[8]
Code Authorship Attribution: Methods and Challenges.Comput. Surveys52, 1 (2020), 1–36. doi:10.1145/3292577 Ivan Murenin, Evgenia Novikova, Roman Ushakov, and Ivan Kholod
Show all 16 references
-
[9]
InInternet of Things, Smart Spaces, and Next Generation Networks and Systems (Lecture Notes in Computer Science, Vol
Explaining android application authorship attribution based on source code analysis. InInternet of Things, Smart Spaces, and Next Generation Networks and Systems (Lecture Notes in Computer Science, Vol. 12525). Springer, Cham, Switzerland, 43–56. doi:10.1007/978-3-030-65726-0_...
-
[10]
Surveys50, 6 (2018), 1–36
Surveying Stylometry Techniques and Applications.Comput. Surveys50, 6 (2018), 1–36. doi:10.1145/3132039 Weihan Ou, Steven Ding, Mohammad Zulkernine, Li Tao Li, and Sarah Labrosse
2018 doi
-
[11]
doi:10.1145/3669901 ACM Trans
VeriBin: A Malware Authorship Verification Approach for APT Tracking through Explainable and Functionality-Debiasing Adversarial Representation Learning.ACM Transactions on Privacy and Security27, 3 (2024), 1–37. doi:10.1145/3669901 ACM Trans. Comput. Educ., Vol. 1, No. 1, Art...
2024 doi
-
[12]
InInternational Conference on Computer Systems and Technologies ’21
The effect of time drift in source code authorship attribution: Time drifting in source code – stylochronometry. InInternational Conference on Computer Systems and Technologies ’21. ACM, Ruse, Bulgaria, 87–92. doi:10.1145/3472410.3472445 Aleksandr Romanov, Anna Kurtukova, Anas...
-
[13]
doi:10.1109/ACCESS.2025.3584616 Matthew F
Integrated Technique of Natural Language Texts and Source Codes Authorship Verification in the Academic Environment.IEEE Access13 (2025), 113274–113290. doi:10.1109/ACCESS.2025.3584616 Matthew F. Tennyson
2025 doi
-
[14]
In2013 3rd International Workshop on Replication in Empirical Software Engineering Research
A Replicated Comparative Study of Source Code Authorship Attribution. In2013 3rd International Workshop on Replication in Empirical Software Engineering Research. IEEE, Piscataway, NJ, USA, 76–83. doi:10.1109/ RESER.2013.12 Ningfei Wang, Shouling Ji, and Ting Wang
2013
-
[15]
InProceedings of the 11th ACM Workshop on Artificial Intelligence and Security
Integration of Static and Dynamic Code Stylometry Analysis for Programmer De-anonymization. InProceedings of the 11th ACM Workshop on Artificial Intelligence and Security. ACM, Toronto, Canada, 74–84. doi:10.1145/3270101.3270110 Riley White and Nathan Sprague
-
[16]
In 2021 20th IEEE International Conference on Machine Learning and Applications
Deep Metric Learning for Code Authorship Attribution and Verification. In 2021 20th IEEE International Conference on Machine Learning and Applications. IEEE, Piscataway, NJ, USA, 1089–1093. doi:10.1109/ICMLA52953.2021.00178 ACM Trans. Comput. Educ., Vol. 1, No. 1, Article . Pu...
2021 doi
Reviewed July 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.