{"id":"c4a8165b-22fe-4f5e-91bf-a3d9f5ba2699","arxiv_id":"2607.07400","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":5,"one_line_summary":"Repository-visible process features (commit timing, change volume, deadline distance) improve educational code authorship attribution from 0.094 to 0.233 top-1 and verification ROC-AUC from 0.556 to 0.752 across six matched comparisons, while contest data achieves 0.938 top-1.","lead":"This paper tests whether code authorship analysis works in educational settings where students solve shared assignments, finding that final-code signal is weak but repository process features (commit timing, change volume) meaningfully improve attribution. A smart generalist might read it to understand why high authorship accuracy on contest benchmarks doesn't transfer to classrooms, and what evidence instructors can actually use.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Process features may encode cohort- or infrastructure-level patterns rather than individual behavioral regularities; the task-aware split controls for assignment overlap but not for section or cohort confounds.","rationale":"The reader's weakest_assumption identifies a related concern about label validity and whether commit history reflects actual development behavior. My concern is more specific: the process features most responsible for the improvement may capture cohort- or infrastructure-level patterns rather than individual behavioral regularities, and the experimental design does not include a control that separates these levels. The reader's concern and mine both point to the validity of process features as a signal source, but from different angles — the reader focuses on label noise and commit-behavior fidelity, while I focus on the confounding structure of the process features themselves. The paper is already conservatively framed and explicitly acknowledges these limitations in §7.2 and §8.1. The CONDITIONAL verdict is appropriate: the empirical evidence (six consistent matched comparisons, feature-removal analysis, verification results) is sufficient to support the bounded claim as stated, but the absence of a section/cohort control means the practical interpretation of what the process signal captures remains uncertain. This does not change the verdict because the paper's claims are already carefully bounded to 'patterns statistically associated with the available experimental label' rather than claims about individual behavioral traits. However, the concrete test proposed above would substantially strengthen the paper's practical contribution by clarifying whether the signal is individual-level or cohort-level. The paper's other strengths — task-aware evaluation, fold-independent preprocessing, multiple complementary metrics, and the feature-removal analysis — provide solid methodological grounding. The absence of significance tests (noted by the reader) is a secondary concern; the sign-test p-value for 6/6 improvements is approximately 0.016, and the verification results provide independent corroborating evidence from a different decision formulation.","tokens_in":24665,"tokens_out":5363,"duration_ms":349824,"concrete_test":"Within a single course section where all students share identical deadline structures, lab schedules, and commit infrastructure, re-run the matched static vs. static-process comparison. If process features still improve top-1 attribution and pairwise AUC by a margin comparable to Table 8, the signal is likely individual-level. If the improvement disappears or shrinks substantially, the signal is confounded with section/cohort effects. This test could be performed on any existing course subset where section metadata is available, using the same task-aware protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that repository-visible process features add complementary author-related signal beyond final code. The six matched comparisons (Table 8) and feature-removal analysis (Table 12) support this under the paper's own definition of 'author-related signal' as 'patterns statistically associated with the available experimental label' (§2.3). However, the process features most responsible for the improvement — activity intensity, deadline distance, time-of-day activity, and change volume (§6.6) — are also the features most susceptible to confounding with course infrastructure, section scheduling, or cohort-level commit policies. The task-aware split (§5.3) holds out assignment groups but does not control for section or cohort membership within a course. If students in different sections or tutorial groups face different deadline structures, lab schedules, or infrastructure configurations, process features could encode section membership, which would correlate with student labels without reflecting individual work habits. The task-adjusted information score I(f;L) − I(f;T) (Eq. 4) checks whether features are more associated with labels than tasks, but it cannot distinguish individual-level from section-level patterns: a feature like 'time-of-day activity' could have high I(f;L) if sections meet at different times, even if within-section variation is uninformative. The paper acknowledges this in §7.2 ('Evening activity may be stable for one student but shaped by timetable constraints for another') and §8.1, but the experimental design includes no control that separates these confounds. If the process-feature signal is primarily cohort-level, the practical utility for instructor-mediated review — the paper's stated application (§2.1, §7.6) — would be weaker than implied, because the features would not distinguish between students within the same cohort.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"This paper investigates source code authorship attribution and verification in educational programming repositories, contrasting performance with programming-contest datasets. The central empirical contribution is a set of six matched comparisons showing that repository-visible process features (commit timing, deadline distance, change volume, activity intensity) improve mean top-1 attribution from 0.094 to 0.233 and mean pairwise verification ROC-AUC from 0.556 to 0.752. The study employs a task-aware cross-assignment evaluation protocol to prevent assignment leakage, fits preprocessing within folds, and includes feature-removal analyses to identify which feature groups drive the observed gains. The paper is methodologically careful and appropriately conservative in its claims, framing results as instructor-mediated decision support rather than autonomous authorship identification.","tokens_in":24821,"tokens_out":1496,"duration_ms":354876,"significance":"The paper makes a valuable contribution to the code authorship literature by systematically examining the educational setting, which is underrepresented compared to contest and open-source benchmarks. The task-aware evaluation protocol is a genuine strength: it directly addresses the assignment-leakage confound that plagues naive cross-validation in educational contexts. The six matched comparisons provide replication across three academic years and two candidate-set sizes, strengthening the central claim. The feature-removal analysis (Table 12) and the task-adjusted information score (Eq. 4) provide interpretable diagnostics. The public release of the cleaned contest dataset (Zenodo) and synthetic dataset is commendable and supports reproducibility. The paper is also appropriately bounded in its claims, consistently framing results as context-dependent rather than universal.","major_comments":[{"comment":"§5.3, Eq. (4) and §6.6: The task-adjusted information score I(f;L) − I(f;T) is used to assess whether features carry more information about labels than tasks. However, this score cannot distinguish individual-level from section- or cohort-level patterns. As acknowledged in §7.2, features like 'evening activity' or 'time-of-day activity' could reflect section scheduling rather than individual work habits. If students in different sections face different deadline structures or lab times, process features could encode section membership, which would correlate with student labels without reflecting individual behavioral regularities. This is load-bearing for the central claim that process features capture 'author-related signal' (§2.3). The paper should add an explicit analysis or at minimum a quantitative bound on how much of the process-feature signal could be explained by section or lab-组","section":null},{"comment":"Table 8 and §6.2: The six matched comparisons show consistent improvement from process features, but no statistical significance testing is reported. With only six comparisons and absolute gains ranging from 0.098 to 0.181, it would strengthen the claim to report a simple paired test (e.g., Wilcoxon signed-rank) or at least a bootstrap confidence interval on the mean gain. The consistency across years is noted qualitatively, but the paper's central quantitative claim ('mean top-1 from 0.094 to 0.233') would benefit from a formal uncertainty estimate.","section":null},{"comment":"§6.6, Table 12: The feature-removal analysis is conducted on a single educational dataset (651 records, 100 labels, 7 assignment groups). While the paper acknowledges this limitation in §7.4 ('The relative importance of these groups is specific to the analyzed educational static-process dataset'), the group-removal results are used to support the general claim that process features provide the largest contribution (§7.4, RQ4). Since the six matched comparisons in Table 8 span three years and two candidate-set sizes, but the feature analysis covers only one configuration, the generalizability of the feature-ranking claim is not established. At minimum, the paper should clarify that RQ4 is answered for one configuration only and should not be read as a general finding about educational datasets.","section":null}],"minor_comments":[{"comment":"§4.1: The course subsets in Table 4 are anonymized, but no information is given about whether different sections or tutorial groups exist within courses. If section information is available, it should be mentioned even if not used in the analysis, as it bears on the confound discussed above.","section":null},{"comment":"Table 3: The educational row reports '–' for participant records, which is understandable, but the table caption or a footnote should clarify that candidate-label filters are applied later in derived datasets (this is mentioned in the text but not in the table itself).","section":null},{"comment":"§5.4: The random forest uses 400 trees and 'balanced class weights' with a 'fixed random state.' The specific random state value should be reported for full reproducibility, or the paper should state that results are averaged over multiple seeds.","section":null},{"comment":"Figure 1: The hatching/fill distinction is a good accessibility choice. However, the y-axis starts at 0, which compresses the visual differences. Consider whether a broken axis or annotated differences would improve readability.","section":null},{"comment":"§6.4: The sequential model comparison reports that 'longer training improved average sequential results' but does not specify the final number of epochs used for the reported results. This should be clarified.","section":null},{"comment":"Table 11: The educational rows show both static-process and static-only results for 2022 and 2024, but not for 2021. If 2021 verification results are available, they should be included for completeness; if not, a brief note explaining the omission would help.","section":null},{"comment":"§8.1: The statement 'No formal ethics-board review was obtained for this retrospective analysis' is noted. While the paper explains the rationale (retrospective, pseudonymized, aggregate), some readers may find the absence of IRB review notable. Consider acknowledging this as a limitation more prominently or confirming that institutional policy does not require it for this type of analysis.","section":null},{"comment":"References: The paper cites work from 2025 and 2026, suggesting it is very recent. Ensure that all cited works are publicly available or provide accessible preprint links where possible.","section":null}],"recommendation":"minor_revision","confidential_remarks":"The paper is well-suited for ACM TOCE. The methodological care is above average for this venue, and the task-aware evaluation protocol is a genuine contribution. The main concern (section/cohort confounding) is acknowledged by the authors but not fully addressed; however, it does not invalidate the central claim under the paper's own bounded definition of 'author-related signal' as 'patterns statistically associated with the available experimental label' (§2.3). A revision that adds a section-level analysis or at least explicitly quantifies the risk would strengthen the paper, but the current evidence is sufficient for publication with minor revisions. The lack of significance testing on six matched comparisons is a presentation gap that should be easy to address."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for the careful and constructive review. The referee correctly identifies the task-aware evaluation protocol, the six matched comparisons, and the bounded framing of claims as strengths of the paper. We address each major comment below. We agree with two of the three comments in full and will revise accordingly; on the third (section-level confounding of process features), we agree the concern is legitimate and will add a quantitative analysis where the data permit it, while being transparent about what cannot be resolved with the available data.","responses":[{"response":"This is a legitimate and important concern. We agree that the task-adjusted information score I(f;L) − I(f;T) does not by itself separate individual-level from section-level signal. If students in different sections face different deadline structures or lab times, process features could encode section membership, which would correlate with student labels without reflecting individual behavioral regularities. This is directly relevant to the central claim that process features capture author-related signal. We will add a quantitative analysis in the revised manuscript. Specifically, we will compute the mutual information between each process feature and the section/lab-group variable, and compare it to I(f;L). This will provide a bound on how much of the label-associated signal in process features could be explained by section membership. We will also add a section-level stratified analysis where the data permit it: if multiple sections are present within a single academic year and candidate set, we can test whether process features retain predictive value when section is controlled for. However, we must be transparent about a limitation: the educational dataset was collected from institutional repositories where section metadata is not uniformly recorded across all academic years and course subsets. For configurations where section information is available, we will report the analysis; where it is not, we will state this explicitly as a standing limitation rather than claiming the confound has been fully ruled out. The manuscript text in §7.2 and §8.1 will be revised to incorporate this analysis and to sharpen the distinction between 'author-related signal' and 'section-correlated signal.'","revision_made":"partial","referee_comment":"§5.3, Eq. (4) and §6.6: The task-adjusted information score I(f;L) − I(f;T) cannot distinguish individual-level from section- or cohort-level patterns. Features like 'evening activity' could reflect section scheduling rather than individual work habits. The paper should add an explicit analysis or at minimum a quantitative bound on how much of the process-feature signal could be explained by section or lab-group membership."},{"response":"The referee is correct. With six matched comparisons showing consistent improvement, a formal uncertainty estimate is appropriate and straightforward to add. We will report a Wilcoxon signed-rank test on the six paired top-1 differences (static vs. static-process), and a bootstrap 95% confidence interval on the mean gain. We will also report the same for the verification AUC comparisons where matched pairs are available (2022 and 2024). We agree that the central quantitative claim ('mean top-1 from 0.094 to 0.233') benefits from a formal uncertainty estimate, and we will add this to Table 8 or the accompanying text in §6.2. We note that with n=6, the Wilcoxon test has limited power, but the consistency of direction (6/6 improvements) is itself informative, and the confidence interval on the mean gain will convey the appropriate uncertainty.","revision_made":"yes","referee_comment":"Table 8 and §6.2: No statistical significance testing is reported for the six matched comparisons. With only six comparisons and absolute gains ranging from 0.098 to 0.181, a paired test or bootstrap confidence interval on the mean gain would strengthen the central quantitative claim."},{"response":"The referee is correct that the feature-removal analysis covers only one configuration and that the RQ4 finding should not be read as a general claim about all educational datasets. The manuscript already acknowledges this in §7.4 ('The relative importance of these groups is specific to the analyzed educational static-process dataset') and in the Table 12 caption ('It does not imply that the same ranking will hold across courses, assignments, or repository policies'). However, we agree that the framing of RQ4 in §7.4 could still be read as making a broader claim than the data support. We will revise the RQ4 discussion to state explicitly that the feature-ranking result is established for one configuration only and that generalizability across courses, years, and institutions is not claimed. We will also add a sentence noting that running the feature-removal analysis on additional matched configurations would strengthen the RQ4 claim and is a direction for future work. The abstract and conclusion will be checked to ensure they do not overstate the scope of the feature-analysis finding.","revision_made":"yes","referee_comment":"§6.6, Table 12: The feature-removal analysis is conducted on a single educational dataset (651 records, 100 labels, 7 assignment groups). The group-removal results are used to support the general claim that process features provide the largest contribution (§7.4, RQ4), but the feature analysis covers only one configuration. The paper should clarify that RQ4 is answered for one configuration only and should not be read as a general finding about educational datasets."}],"tokens_in":24518,"tokens_out":1203,"duration_ms":273729,"standing_objections":["The section-level confounding analysis (Comment 1) can only be conducted where section metadata is available in the educational dataset. Section information is not uniformly recorded across all academic years and course subsets in the institutional repository. For configurations where section data exist, we will report the analysis; for those where it does not, we cannot provide a quantitative bound and will state this as a limitation. We cannot fully resolve this confound for all six matched comparisons."]},"desk_editor":{"model":"glm-5.2","letter":"Two things matter here: (1) contest authorship accuracy does not transfer to educational repositories, and (2) repository-derived process features (commit timing, deadline distance, change volume, activity intensity) consistently improve attribution and verification in educational settings across six matched comparisons. The improvement is real and replicated — top-1 goes from 0.094 to 0.233 mean, pairwise AUC from 0.556 to 0.752, with gains in every year and candidate-set size tested. The task-aware cross-assignment split is the right design choice and prevents the most obvious form of leakage. The feature-removal analysis (Table 12) and the task-adjusted information score are good diagnostic moves that most papers in this area skip. The paper is also appropriately conservative about what its labels mean — it never claims sole authorship ground truth, and it frames the practical application as instructor-mediated review prioritization, not automated accusation. That framing is earned by the data. The contest dataset is publicly released (Zenodo), which is a plus for reproducibility on that side; the educational data cannot be released for privacy reasons, which is understandable but limits external validation. The stress-test concern about section or cohort confounding is legitimate but probably not load-bearing. The paper acknowledges it explicitly in Sections 7.2 and 8.1, and the task-adjusted information score I(f;L) − I(f;T) partially addresses it by checking whether features associate more with labels than tasks. What it cannot do is separate individual-level from section-level signal — if sections meet at different times, time-of-day features could encode section membership rather than personal habits. This matters for the practical claim about within-cohort discrimination, but it does not undermine the core finding that process features add complementary signal beyond final code. The consistency across three academic years and two candidate-set sizes is hard to explain purely by cohort confound. Two minor issues: no significance tests or confidence intervals on the matched-pair differences (the consistency across all six pairs partially compensates), and the TF-IDF feature-document branch adds essentially nothing when combined with the random forest, which is honestly reported but raises the question of why it occupies so much space. This paper is for researchers in code authorship attribution and for computing-education people thinking about academic-integrity tooling. It deserves a serious referee. The main thing I would push a reviewer to ask for is a within-section or within-cohort control on at least one dataset, to test whether the process-feature signal survives when section-level variation is removed.","headline":"Process features (commit timing, deadline distance, activity intensity) consistently improve educational code authorship attribution across six matched comparisons; contest accuracy does not transfer to course settings.","tokens_in":25636,"tokens_out":597,"would_cite":true,"duration_ms":159534,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Commit timing beats code style for identifying student programmers","keywords":[],"falsifier":"If process features were tested in a course with mandatory auto-formatting and prescribed commit schedules, and they no longer improved attribution beyond static-only code, the central claim that process evidence complements weak final-code signal would be weakened. Alternatively, if the process-feature improvement disappeared when labels were independently verified for sole authorship (removing noise from collaboration or copied work), it would suggest the signal was capturing label noise rather than auth","tokens_in":24719,"feed_emoji":"⏱️","tokens_out":644,"duration_ms":104235,"temperature":0.7,"pith_summary":"This paper argues that in educational programming courses, the final source code a student submits carries very little signal about who wrote it, because shared assignments, starter code, and developing skills compress the differences between students. The author tests whether repository-visible process features—commit timing, distance from deadline, change volume, and activity intensity—add complementary author-related information beyond the static code artifact. Across six matched comparisons spanning three academic years and two candidate-set sizes, adding process features consistently improved attribution ranking (mean top-1 rising from 0.094 to 0.233) and pairwise verification (mean ROC-AUC rising from 0.556 to 0.752). The paper also contrasts educational repositories with programming-contest data, where reusable personal templates and stable routines produce much stronger authorship signal (Kick Start mean top-1 of 0.938). The central mechanism is that when an assignment constrains the final artifact, the development history preserves variation that the submitted code does not. The author frames the result as bounded: these methods are appropriate as instructor-mediated decision support for prioritizing review, not as autonomous proof of authorship.","feed_headline":"Commit timing beats code style for identifying student programmers","feed_subtitle":"When shared assignments erase individual code fingerprints, repository work history recovers author signal that final code alone cannot.","key_machinery":"Task-aware cross-assignment evaluation protocol and matched static vs. static-process educational comparisons","core_discovery":"The paper's central finding is that repository-derived process features—commit timing, deadline distance, change volume, and activity intensity—provide the largest single contribution to author-related signal in educational programming repositories, and they do so consistently across all six matched comparisons. In the detailed feature analysis of one selected dataset, removing process metrics caused the largest performance drop (top-1 from 0.286 to 0.163), and nine of the ten highest-ranked individual features described repository activity rather than static code properties. This means the timing and rhythm of how a student works in a repository, not the style of the code they submit, is a ","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Commit patterns outperform code style for student authorship attribution","Process features beat static code for identifying student programmers","How students work matters more than what they code for authorship","Commit timing and work rhythm reveal student authors better than code style","Work patterns outpace code style for student attribution in coursework"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The paper assumes that pseudonymous student labels derived from repository ownership or submission records are a valid proxy for authorship identity, and that repository-visible commit history reflects actual development behavior. If commit patterns are heavily shaped by course infrastructure (e.g., mandatory commit workflows) or if labels are noisy due to untracked collaboration, the measured process-feature signal could reflect repository-usage conventions rather than auth","fun_headline_variants_meta":{"raw":{"variants":["Commit patterns outperform code style for student authorship attribution","Process features beat static code for identifying student programmers","How students work matters more than what they code for authorship","Commit timing and work rhythm reveal student authors better than code style","Work patterns outpace code style for student attribution in coursework","When assignments are shared, commit history identifies student authors","Process evidence recovers author signal that final code cannot","Static code features underperform process metrics in educational attribution","Repository activity patterns surpass code style for student identification","Code style fails where commit history succeeds for student authorship","Educational code authorship relies on process signal over static features","Nine of ten top authorship features describe repository activity, not code"]},"model":"glm-5.2","effort":"low","cost_usd":0.0,"raw_usage":{"total_tokens":1301,"prompt_tokens":507,"completion_tokens":794,"prompt_tokens_details":null},"tokens_in":507,"tokens_out":794,"duration_ms":40540,"temperature":1.0,"reasoning_tokens":723,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T11:55:30.461878+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If process features were tested in a course with mandatory auto-formatting and prescribed commit schedules, and they no longer improved attribution beyond static-only code, the central claim that process evidence complements weak final-code signal would be weakened. Alternatively, if the process-feature improvement disappeared when labels were independently verified for sole authorship (removing noise from collaboration or copied work), it would suggest the signal was capturing label noise rather than auth","supporting_citations":[],"review_version":1}