{"id":"977d4512-e1db-47b5-bbb9-54602ce446ed","arxiv_id":"2506.17355","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A classroom study claims the IDE-based tool PasteTrace detects single-source code plagiarism better than MOSS, but the evaluation labels come from PasteTrace's own logs.","lead":"PasteTrace is a modified programming editor that hides invisible watermarks in copied code so teachers can see where each line came from, and it logs every edit a student makes. The paper reports it caught more cheating than the standard MOSS tool, but the comparison is circular because the 'true' cheating labels were read from PasteTrace's own logs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 15-of-16 detection claim is unsubstantiated because the 'Method' ground-truth labels are read from PasteTrace's own metaComment logs, the same data the automated detector consumes.","rationale":"The reader's weakest assumption is exactly the load-bearing spot. PasteTrace is an interesting engineering artifact with real affordances: embedding UUID watermarks, infection stacks, edit logs, and a released open-source implementation. But the paper's headline contribution is an empirical comparison claiming 15/16 detection and one false positive against MOSS. That comparison is only meaningful if the labels are known independently of the tool being evaluated. Section IV shows they are not: the narrative consistently appeals to the metaComment to decide legitimacy or plagiarism, and the same metaComment drives the automated check. The paper itself flags the fragility (Student I's corrupted log, Student C's unclear status), which strengthens rather than weakens the concern. The proposed check is straightforward and uses the already-released data: external blinded adjudication without any PasteTrace logs. If such an adjudication reproduces the same labels, the central claim would be substantially strengthened; until then, the reported detection rate and the MOSS comparison rest on circular evidence. I therefore agree with the REJECT verdict and see no need to weaken or strengthen it based on this stress-test pass.","tokens_in":8348,"tokens_out":4837,"duration_ms":52386,"concrete_test":"Use the released figshare case-study data to build an independent gold standard: strip every metaComment and PasteTrace log from all 24 submissions, then have two external annotators who have never used PasteTrace classify each submission as plagiarized or legitimate using only the final source code, the assignment prompt, the known seeded solutions, and ordinary code-similarity inspection (e.g., MOSS output without the metaComment). Pre-register that any submission matching a seeded solution, or near-identical to another student's submission in a way visible in the plain code, counts as plagiarized. Compare this independent label set against the paper's 'Method' column and recompute the 15/16 and MOSS comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparative claim of Section IV-C ('our tool detected 15 of 16 and had one false positive') requires the 'Method' labels in Tables I and II to be independent ground truth. They are not independent. Section IV-A describes Student B's legitimacy as 'shown in the metaComments' and Student F's plagiarism as 'logged as a large foreign paste'; Section IV-B says Student I's metaComment was 'corrupted' yet I is still counted among the plagiarized, and Student C in course I is called 'unclear.' The automated detector in Section III-C consumes the same metaComment/InfectionStack and paste logs to produce its verdicts. The paper also collects self-reports but never uses them as an independent label source, and the MOSS pairs are manually interpreted by the same authors. The only externally verifiable cases are the seeded files; the rest of the 16-case count is anchored to the tool's own logs. This makes the headline comparison, and the asserted 'immunity' to traditional false positives in Section III-A, unsupported. The open-source artifact and released data are real, but they support a weaker claim: PasteTrace flags the events recorded in its own logs.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PasteTrace, an open-source Integrated Development Environment (IDE) plugin for detecting single-source plagiarism in introductory programming courses. PasteTrace embeds machine and project identifiers into source files via zero-width characters in a hidden metadata comment, logs all edit and paste events, and provides an automated script that flags plagiarism based on paste provenance and line-count thresholds. The authors evaluate PasteTrace in two introductory programming courses (24 total submissions), compare its performance to MOSS, and report that PasteTrace detected 15 of 16 plagiarized submissions with one false positive, while MOSS detected at most 9 of 16. The artifact, data, and case study files are made publicly available.","tokens_in":8554,"tokens_out":4652,"duration_ms":47863,"significance":"If the evaluation were sound, PasteTrace would address a genuine gap: pairwise similarity tools like MOSS cannot detect single-source plagiarism (e.g., from ChatGPT, the web, or a previous project) when the final code is heavily edited. The provenance-tracking approach is promising, and the open-source artifact and released data are strong assets. However, the central empirical claim — that PasteTrace outperforms MOSS — is undermined by a circular evaluation: the ground-truth labels are derived from the same metaComment logs that the automated detector consumes. The current evidence supports only a weaker claim about PasteTrace's ability to flag events recorded in its own logs, not its external detection accuracy. The paper's contribution at present is primarily a tool design and case study, not a validated comparative evaluation.","major_comments":[{"comment":"The headline result, 'our tool detected 15 of 16 and had one false positive,' is not supported by independent ground truth. The 'Method' column in Tables I and II is read from PasteTrace's own metaComment logs: Section IV-A states that Student B's legitimacy is 'shown in the metaComments' and that Student F's plagiarism 'is logged as a large foreign paste'; Section IV-B states that Student I's metaComment was 'corrupted' yet I is still counted among the plagiarized, and Student C in course I is called 'unclear.' The automated detector in Section III-C consumes the same metaComment/InfectionStack and paste events to produce its verdicts. Thus the reported detection rate is a measure of self-consistency between the tool's rules and its own logs, not a validated detection rate. The comparison with MOSS is consequently unsubstantiated.","section":"Section IV-C"},{"comment":"The claim that the method 'is immune to traditional false positives as organically created code is never considered plagiarized regardless of similarity to another student' is contradicted by the tool's own behavior in the evaluation. Section III-C states that pastes from a different project on the same machine are flagged as plagiarism if they exceed a 50-line threshold, and pastes from an unknown machine or project are flagged at a 20-line threshold. These are structural sources of false positives for legitimate code reuse. Indeed, Table I shows that Student E in course I, who the authors classify as 'Legitimate,' is flagged by the automated check as 'Plagiarism Detected, 767 Edits.' The immunity claim is therefore not supported by the presented evidence.","section":"Section III-A"},{"comment":"The detection thresholds (50 lines, 20 lines, and 3 lines) are hand-set with only informal rationale, and the paper provides no sensitivity analysis. Given the small sample size (16 plagiarized, 2 legitimate with clear labels, 2 unclear), the reported 15/16 detection rate could be highly sensitive to these thresholds, but the manuscript does not explore how varying them would change the outcome. This is load-bearing because the central claim is that PasteTrace achieves superior detection, and the thresholds are central to that claim.","section":"Section III-C"}],"minor_comments":[{"comment":"The text claims that physically typing code from a reference is 'theoretically detected by analyzing the activity,' but no method for detecting such behavior is described or implemented in Section III-C, which relies exclusively on paste events and line counts. This claim should be softened or clarified.","section":"Section III-A"},{"comment":"Table II contains a typo: 'Blank Submision' should be 'Blank Submission.'","section":"Section IV-B"},{"comment":"The comparison with MOSS would benefit from explicit counts for each category. The paper reports '15 of 16' for PasteTrace but does not clearly specify which of the 16 cases were missed or which case constituted the false positive, despite the information being present in the tables. A per-case reconciliation would help the reader verify the claim.","section":"Section IV-C"},{"comment":"The self-report data collected from students is mentioned but never used in the analysis, and the anonymity design prevents linking self-reports to submissions. The paper should either explain why self-reports were not usable for validation or describe how they might be incorporated in future work.","section":"Section IV"},{"comment":"The distinction between peer-to-peer sharing, collaboration, and theft is described conceptually, but the automated detector in Section III-C does not appear to implement these distinctions, instead using only paste origin and line thresholds. Clarify which scenarios the automated script actually distinguishes.","section":"Section III-B"}],"recommendation":"reject","confidential_remarks":"The circularity of the evaluation is the decisive issue. The claimed detection rates and the MOSS comparison cannot be salvaged with the current data because no independent ground truth was collected. A future study could address this by using seeded files (which were already included) and pre-registered manual labels from independent coders who are blinded to the metaComment content. For this submission, however, the central empirical claim is unsupported, and the paper would need substantial reframing — dropping the 'outperforming MOSS' claim — to be acceptable as a tool demonstration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nPasteTrace is a genuinely useful piece of engineering: a modified Processing IDE that embeds per-install and per-project UUIDs in hidden metadata comments, tags copied code with zero-width spaces, and logs every edit so you can reconstruct how a submission came to be. The open-source release and case-study data are real, and the problem it targets is real — similarity scanners like MOSS miss single-source plagiarism, including ChatGPT-derived or previously written code. There is nothing else quite like this combination in the cited literature.\n\nThe evaluation, though, does not support the headline claim that PasteTrace outperforms MOSS. The 15-of-16 detection count rests on labels that are read from the same logs the tool produces and analyzes. Section IV-A calls Student B legitimate because it is 'shown in the metaComments,' and Student F plagiarized because it is 'logged as a large foreign paste.' The automated script consumes those same metaComment and InfectionStack entries. So the comparison is more an internal consistency check than an external validation. The seeded internet solutions are the only externally verifiable cases. Thresholds (50, 20, and 3 lines) are hand-set with no sensitivity analysis, the sample is small and voluntary, and two cases are honestly labeled unclear (Student C in course I, Student M in course II) while one metaComment was corrupted (Student I) — yet the counts treat these as resolvable.\n\nThat is a load-bearing flaw, not a cosmetic one. The tool may well work, but the paper does not demonstrate it. What would fix it: independent adjudication where the human reviewer does not see PasteTrace's logs (even better, known-source submissions), threshold sensitivity analysis, and a clean separation between the tool's raw logs and the human judgment that labels a case.\n\nThe paper deserves a serious referee — the artifact is real and the evaluation is fixable — but not acceptance in its current form. This is a reject-and-resubmit or major-revision candidate. I'd want the reworked evaluation before putting it in the literature. It's aimed at computing-education researchers and instructors who want a local, process-based plagiarism detector, and could be a solid tool paper once the claims catch up with the evidence.","headline":"PasteTrace is a clever engineering artifact, but its headline detection claim is circular and needs independent validation.","tokens_in":9077,"tokens_out":4216,"would_cite":false,"duration_ms":39733,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PasteTrace detects plagiarism by tracing where each paste came from, not by comparing final code.","keywords":["plagiarism detection","single-source plagiarism","IDE instrumentation","code provenance","zero-width space watermark","introductory programming","programming education","edit trace"],"falsifier":"Take a set of submissions whose true sources are known independently—for example, by planting watermarked copies from controlled accounts and by having students produce original code under observation—then run PasteTrace's automated checker and compare its verdicts with that independent source knowledge; if the checker's own metaComment is the only evidence used to label the cases, the claimed detection rate cannot be distinguished from circular reasoning.","tokens_in":8105,"feed_emoji":"📋","tokens_out":7106,"duration_ms":70884,"temperature":0.7,"pith_summary":"PasteTrace is an attempt to shift code plagiarism detection in introductory programming courses from comparing finished submissions to tracing the act of copying itself. The authors build a modified Processing IDE that hides machine and project identifiers inside copied code using zero-width characters, then records every paste and its origin in a hidden metadata comment. Because the tool watches how code was made rather than how it looks at the end, it is designed to catch plagiarism that similarity detectors overlook: a student pasting from an AI chatbot, from a website, or from a previous project, even when the pasted code is heavily edited afterward. The paper reports two course deployments in which PasteTrace flagged 15 of 16 submissions the authors labeled plagiarized, with one false positive, compared with 9 of 16 (or 5 of 16 under stricter scoring) for an established similarity-based tool.","feed_headline":"Watermarking pastes catches cheating that similarity tools miss","feed_subtitle":"Hidden IDs in pasted code let instructors trace it to a chatbot, the web, or an old project.","key_machinery":"The load-bearing object is the InfectionStack: an ordered list of mismatched machine and project identifiers embedded in a hidden metaComment in each source file. It is populated by zero-width-space watermarks that survive ordinary message-passing and partial pastes, and it records each file's chain of sources. Working with the edit log, it lets the checker distinguish an organic file with no foreign IDs and linear typing, a paste from another student's machine or project, a paste from the same student's older project, an outside-the-IDE paste, and a copied file loaded directly. The line-count thresholds (50 for same-machine pastes, 20 for unknown-machine pastes, 3 for outside pastes) and the count of edits after the last flagged paste turn the raw provenance stack into a plagiarism verdict.","core_discovery":"The core discovery is that plagiarism in introductory programming can be treated as a provenance problem rather than a similarity problem. PasteTrace's modified IDE stamps every copied region with hidden zero-width space characters encoding the source machine's persistent InstallID, the source project's ProjectID, and an InfectionStack of previous owners. Any paste whose hidden IDs do not match the receiving project appends the mismatch to that stack, and the stack is stored in a metadata comment inside the file alongside a full log of edit events. The tool's automated checker then classifies pastes by origin and size, flags pastes from other students' machines or projects, applies line-count thresholds for same-machine and unknown-machine pastes, and counts edits after the last suspicious paste. The paper claims this catches single-source copies that leave no second submission to compare against, and reports 15 of 16 detections with one false positive across two introductory courses.","pith_inferences":["If the mechanism generalizes, the same hidden-watermark-and-stack idea could be embedded in other IDEs or in web-based coding environments, allowing single-source plagiarism detection outside the Processing ecosystem.","The reported detection statistics are only as strong as the independent ground truth behind them; an external audit with planted sources and blinded human review would settle whether the 15-of-16 rate holds.","The line-count thresholds are arbitrary and likely course-dependent; a testable extension would calibrate them per assignment using known legitimate reuse patterns.","A striking but unexplored corollary is that the method also produces a detailed reconstruction of how each student worked, which could support formative feedback on study habits rather than only policing cheating."],"forward_implications":["Single-source plagiarism—copying from an AI chatbot, a website, an expert service, or a previous assignment—becomes detectable even though no second submission shares the final code.","Heavy editing after a copy does not hide the plagiarism, because the original paste event and its source IDs remain in the file's metadata.","Class-wide analysis of infection stacks can separate peer file-sharing, collaboration, theft, and search or expert copying, and can tie a copied file back to the assignment and author that produced it.","Tools that only compare finished code cannot detect plagiarism when only one student submitted the copied solution; PasteTrace's method is designed to work in that case.","If the tool says a file is clean, that claim is based on a visible chain of organic edits rather than on the absence of a similar file."],"supporting_citations":[{"why":"Supplies the winnowing-based similarity measure used as the MOSS baseline that PasteTrace claims to outperform.","marker":"[12]"},{"why":"Supplies the open-source Processing IDE that PasteTrace modifies to track paste events.","marker":"[22]"},{"why":"Frames collaboration, collusion, and plagiarism in computer science coursework as the problem PasteTrace addresses.","marker":"[6]"},{"why":"Defines the general plagiarism-detection problem as finding a detection function D, which the paper contrasts with its provenance-based approach.","marker":"[11]"}],"fun_headline_variants":["Watermark pastes with hidden IDs to trace stolen code","Single-source plagiarism? PasteTrace finds it via provenance","Zero-width space watermarks unmask copied code origins","Plagiarism detection goes underground with invisible tags","Code provenance: the new front in cheating detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes the authors' reading of PasteTrace's own metadata comments is an independent ground truth for which students plagiarized, yet those comments are written by the tool itself; one student's status was called unclear and another's comment was corrupted, so the 15-of-16 figure rests on self-referential logs.","fun_headline_variants_meta":{"raw":{"variants":["Watermark pastes with hidden IDs to trace stolen code","Single-source plagiarism? PasteTrace finds it via provenance","Zero-width space watermarks unmask copied code origins","Plagiarism detection goes underground with invisible tags","Code provenance: the new front in cheating detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000273,"raw_usage":{"total_tokens":1608,"prompt_tokens":888,"completion_tokens":720,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":645}},"tokens_in":504,"tokens_out":720,"duration_ms":7311,"temperature":1.0,"reasoning_tokens":645,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:19:28.478794+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of submissions whose true sources are known independently—for example, by planting watermarked copies from controlled accounts and by having students produce original code under observation—then run PasteTrace's automated checker and compare its verdicts with that independent source knowledge; if the checker's own metaComment is the only evidence used to label the cases, the claimed detection rate cannot be distinguished from circular reasoning.","supporting_citations":[{"cited_title":"Winnowing: local algorithms for document fingerprinting,","cited_arxiv_id":null,"evidence_quote":"Supplies the winnowing-based similarity measure used as the MOSS baseline that PasteTrace claims to outperform."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the open-source Processing IDE that PasteTrace modifies to track paste events."},{"cited_title":"Collaboration, collusion and plagiarism in computer science coursework,","cited_arxiv_id":null,"evidence_quote":"Frames collaboration, collusion, and plagiarism in computer science coursework as the problem PasteTrace addresses."},{"cited_title":"Attitudes of second year computer science undergraduates toward plagiarism,","cited_arxiv_id":null,"evidence_quote":"Defines the general plagiarism-detection problem as finding a detection function D, which the paper contrasts with its provenance-based approach."}],"review_version":2}