Pith. sign in

REVIEW 3 major objections 5 minor 22 references

PasteTrace: A Single Source Plagiarism Detection Tool For Introductory Programming Courses

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PasteTrace detects plagiarism by tracing where each paste came from, not by comparing final code.

desk verdict PasteTrace is a clever engineering artifact, but its headline detection claim is circular and needs independent validation. read the letter →

arxiv 2506.17355 v1 pith:F5YVUZAP submitted 2025-06-20 cs.CY

Jesse McDonald , Scott Robertson , Anthony Peruma This is my paper · ORCID
classification cs.CY
keywords plagiarismdetectionsingle-sourceIDEinstrumentationcodeprovenancezero-widthspacewatermarkintroductoryprogrammingeducationedittrace
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PasteTrace is an attempt to shift code plagiarism detection in introductory programming courses from comparing finished submissions to tracing the act of copying itself. The authors build a modified Processing IDE that hides machine and project identifiers inside copied code using zero-width characters, then records every paste and its origin in a hidden metadata comment. Because the tool watches how code was made rather than how it looks at the end, it is designed to catch plagiarism that similarity detectors overlook: a student pasting from an AI chatbot, from a website, or from a previous project, even when the pasted code is heavily edited afterward. The paper reports two course deployments in which PasteTrace flagged 15 of 16 submissions the authors labeled plagiarized, with one false positive, compared with 9 of 16 (or 5 of 16 under stricter scoring) for an established similarity-based tool.

What carries the argument

The load-bearing object is the InfectionStack: an ordered list of mismatched machine and project identifiers embedded in a hidden metaComment in each source file. It is populated by zero-width-space watermarks that survive ordinary message-passing and partial pastes, and it records each file's chain of sources. Working with the edit log, it lets the checker distinguish an organic file with no foreign IDs and linear typing, a paste from another student's machine or project, a paste from the same student's older project, an outside-the-IDE paste, and a copied file loaded directly. The line-count thresholds (50 for same-machine pastes, 20 for unknown-machine pastes, 3 for outside pastes) and the count of edits after the last flagged paste turn the raw provenance stack into a plagiarism verdict.

What would settle it

Take a set of submissions whose true sources are known independently—for example, by planting watermarked copies from controlled accounts and by having students produce original code under observation—then run PasteTrace's automated checker and compare its verdicts with that independent source knowledge; if the checker's own metaComment is the only evidence used to label the cases, the claimed detection rate cannot be distinguished from circular reasoning.

Watch

Extended reading notes

Core claim

The core discovery is that plagiarism in introductory programming can be treated as a provenance problem rather than a similarity problem. PasteTrace's modified IDE stamps every copied region with hidden zero-width space characters encoding the source machine's persistent InstallID, the source project's ProjectID, and an InfectionStack of previous owners. Any paste whose hidden IDs do not match the receiving project appends the mismatch to that stack, and the stack is stored in a metadata comment inside the file alongside a full log of edit events. The tool's automated checker then classifies pastes by origin and size, flags pastes from other students' machines or projects, applies line-count thresholds for same-machine and unknown-machine pastes, and counts edits after the last suspicious paste. The paper claims this catches single-source copies that leave no second submission to compare against, and reports 15 of 16 detections with one false positive across two introductory courses.

Load-bearing premise

The evaluation assumes the authors' reading of PasteTrace's own metadata comments is an independent ground truth for which students plagiarized, yet those comments are written by the tool itself; one student's status was called unclear and another's comment was corrupted, so the 15-of-16 figure rests on self-referential logs.

Editorial extensions

If this is right

  • Single-source plagiarism—copying from an AI chatbot, a website, an expert service, or a previous assignment—becomes detectable even though no second submission shares the final code.
  • Heavy editing after a copy does not hide the plagiarism, because the original paste event and its source IDs remain in the file's metadata.
  • Class-wide analysis of infection stacks can separate peer file-sharing, collaboration, theft, and search or expert copying, and can tie a copied file back to the assignment and author that produced it.
  • Tools that only compare finished code cannot detect plagiarism when only one student submitted the copied solution; PasteTrace's method is designed to work in that case.
  • If the tool says a file is clean, that claim is based on a visible chain of organic edits rather than on the absence of a similar file.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism generalizes, the same hidden-watermark-and-stack idea could be embedded in other IDEs or in web-based coding environments, allowing single-source plagiarism detection outside the Processing ecosystem.
  • The reported detection statistics are only as strong as the independent ground truth behind them; an external audit with planted sources and blinded human review would settle whether the 15-of-16 rate holds.
  • The line-count thresholds are arbitrary and likely course-dependent; a testable extension would calibrate them per assignment using known legitimate reuse patterns.
  • A striking but unexplored corollary is that the method also produces a detailed reconstruction of how each student worked, which could support formative feedback on study habits rather than only policing cheating.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces PasteTrace, an open-source Integrated Development Environment (IDE) plugin for detecting single-source plagiarism in introductory programming courses. PasteTrace embeds machine and project identifiers into source files via zero-width characters in a hidden metadata comment, logs all edit and paste events, and provides an automated script that flags plagiarism based on paste provenance and line-count thresholds. The authors evaluate PasteTrace in two introductory programming courses (24 total submissions), compare its performance to MOSS, and report that PasteTrace detected 15 of 16 plagiarized submissions with one false positive, while MOSS detected at most 9 of 16. The artifact, data, and case study files are made publicly available.

Significance. If the evaluation were sound, PasteTrace would address a genuine gap: pairwise similarity tools like MOSS cannot detect single-source plagiarism (e.g., from ChatGPT, the web, or a previous project) when the final code is heavily edited. The provenance-tracking approach is promising, and the open-source artifact and released data are strong assets. However, the central empirical claim — that PasteTrace outperforms MOSS — is undermined by a circular evaluation: the ground-truth labels are derived from the same metaComment logs that the automated detector consumes. The current evidence supports only a weaker claim about PasteTrace's ability to flag events recorded in its own logs, not its external detection accuracy. The paper's contribution at present is primarily a tool design and case study, not a validated comparative evaluation.

major comments (3)
  1. [Section IV-C] The headline result, 'our tool detected 15 of 16 and had one false positive,' is not supported by independent ground truth. The 'Method' column in Tables I and II is read from PasteTrace's own metaComment logs: Section IV-A states that Student B's legitimacy is 'shown in the metaComments' and that Student F's plagiarism 'is logged as a large foreign paste'; Section IV-B states that Student I's metaComment was 'corrupted' yet I is still counted among the plagiarized, and Student C in course I is called 'unclear.' The automated detector in Section III-C consumes the same metaComment/InfectionStack and paste events to produce its verdicts. Thus the reported detection rate is a measure of self-consistency between the tool's rules and its own logs, not a validated detection rate. The comparison with MOSS is consequently unsubstantiated.
  2. [Section III-A] The claim that the method 'is immune to traditional false positives as organically created code is never considered plagiarized regardless of similarity to another student' is contradicted by the tool's own behavior in the evaluation. Section III-C states that pastes from a different project on the same machine are flagged as plagiarism if they exceed a 50-line threshold, and pastes from an unknown machine or project are flagged at a 20-line threshold. These are structural sources of false positives for legitimate code reuse. Indeed, Table I shows that Student E in course I, who the authors classify as 'Legitimate,' is flagged by the automated check as 'Plagiarism Detected, 767 Edits.' The immunity claim is therefore not supported by the presented evidence.
  3. [Section III-C] The detection thresholds (50 lines, 20 lines, and 3 lines) are hand-set with only informal rationale, and the paper provides no sensitivity analysis. Given the small sample size (16 plagiarized, 2 legitimate with clear labels, 2 unclear), the reported 15/16 detection rate could be highly sensitive to these thresholds, but the manuscript does not explore how varying them would change the outcome. This is load-bearing because the central claim is that PasteTrace achieves superior detection, and the thresholds are central to that claim.
minor comments (5)
  1. [Section III-A] The text claims that physically typing code from a reference is 'theoretically detected by analyzing the activity,' but no method for detecting such behavior is described or implemented in Section III-C, which relies exclusively on paste events and line counts. This claim should be softened or clarified.
  2. [Section IV-B] Table II contains a typo: 'Blank Submision' should be 'Blank Submission.'
  3. [Section IV-C] The comparison with MOSS would benefit from explicit counts for each category. The paper reports '15 of 16' for PasteTrace but does not clearly specify which of the 16 cases were missed or which case constituted the false positive, despite the information being present in the tables. A per-case reconciliation would help the reader verify the claim.
  4. [Section IV] The self-report data collected from students is mentioned but never used in the analysis, and the anonymity design prevents linking self-reports to submissions. The paper should either explain why self-reports were not usable for validation or describe how they might be incorporated in future work.
  5. [Section III-B] The distinction between peer-to-peer sharing, collaboration, and theft is described conceptually, but the automated detector in Section III-C does not appear to implement these distinctions, instead using only paste origin and line thresholds. Clarify which scenarios the automated script actually distinguishes.

Circularity Check

2 steps flagged · score 7.0 of 10

The 15-of-16 detection claim is undermined because the ground-truth 'Method' labels are read from PasteTrace's own metaComment logs, the same data consumed by the automated detector.

  1. self definitional [Section IV-A and Tables I/II (Method column)]
    "Student B completed the assignment legitimately, and it is shown in the metaComments. There are multiple well-organized files, each organically coded with no large external pastes or irregularly linear typing sections."

    The Method column that supplies the 16-case ground truth is read from PasteTrace's own metaComment/log evidence (B is 'Legitimate' because the metaComments look organic; F's plagiarism is 'logged as a large foreign paste'; Student I is labeled despite a 'corrupted' metaComment on the basis of the InfectionStack). The Section III-C automated detector consumes the same metaComment/InfectionStack and paste-event data to classify submissions. Consequently the automated verdicts and the human Method labels are two interpretations of one data source, so the detector is not being tested against an independent ground truth.

  2. fitted input called prediction [Section IV-C (PasteTrace Vs. MOSS)]
    "In comparison, our tool detected 15 of 16 and had one false positive."

    The '16 plagiarised assignments' count and the 'one false positive' are taken from the Method labels that were themselves derived from PasteTrace's logs, while the automated check output in Tables I/II is PasteTrace's script reading those same logs. The 15-of-16 figure is therefore an agreement rate between two readings of PasteTrace's own data, not a detection rate against externally verified plagiarism. The claimed advantage over MOSS, whose pairs were manually inspected from similarity output, is not established by this evaluation design.

full rationale

The central evaluation of PasteTrace is not anchored to an independent ground truth. Section IV-A states Student B's legitimacy is 'shown in the metaComments,' Student F's plagiarism is 'logged as a large foreign paste,' and Section IV-B reports Student I's metaComment was corrupted yet still labels I as plagiarized on the basis of the InfectionStack. The automated detector in Section III-C consumes the same metaComment/InfectionStack and paste-event logs to classify submissions. Section IV-C then reports 'our tool detected 15 of 16 and had one false positive' against those labels. Thus the headline comparison to MOSS measures agreement between a human reading and a rule-based reading of the same data source, not detection of independently verified plagiarism. The only externally anchored cases are the seeded solution files; self-reports were collected but are not used as an independent check. This makes the claimed 'outperforming' result unsupported as an evaluation, although the logging mechanism itself and the released open-source artifact are not circular. Score 7 reflects that the central comparative claim is substantially forced by the non-independent evaluation design, while still acknowledging the tool's real artifact and the fact that the automated detector did disagree with human labels in some cases.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical or theoretical entities are postulated. The UUIDs and InfectionStack are implemented software components with source code linked, so they are engineering artifacts rather than invented entities. The main load-bearing inventions are the hand-set detection thresholds and the circular evaluation labels that make the reported detection rate a self-consistency check.

free parameters (4)
  • Internal paste threshold = 50 lines
    Pastes from the same machine into a different project are assumed legitimate below 50 lines and flagged as plagiarism above it. Hand-set with no sensitivity analysis, Section III-C.
  • Foreign-machine paste threshold = 20 lines
    Pastes from a machine not associated with any enrolled student are assumed to be the student's own work below 20 lines and flagged above it. Hand-set with no sensitivity analysis, Section III-C.
  • External-origin paste threshold = 3 lines
    Pastes not originating in the IDE are treated as online copying; more than three lines is flagged. The paper reasons about docs-page copying but does not test the cutoff, Section III-C.
  • MOSS Major Link threshold = 40%
    MOSS pairs are classified as major or minor links using a 40% shared-line cutoff. The authors call it arbitrary, Section IV.
assumptions (4)
  • domain assumption Students used the supplied PasteTrace IDE and USB drives for all assignment work, so the logs capture all relevant coding activity.
    Section IV instructs students to use PasteTrace and save project files on the flash drive. If students worked elsewhere, provenance is incomplete.
  • domain assumption The metaComment and keystroke log cannot be stripped or corrupted without notice.
    The detection relies on these logs, yet the paper itself reports Student I's metaComment corrupted and Student M's typing history destroyed, Section IV-B.
  • ad hoc to paper The manual labels in the 'Method' columns of Tables I and II are independent ground truth.
    The labels are assigned from the metaComment logs, so using them to score PasteTrace is circular, Section IV-A and IV-C.
  • domain assumption Zero-width-space watermarks survive all relevant sharing channels except email.
    Section III-A says testing showed resilience on messaging software and stripping in email, but the full range of student sharing channels is not demonstrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PasteTrace: A Single Source Plagiarism Detection Tool For Introductory Programming Courses." pith.science (2026). https://pith.science/paper/F5YVUZAP

@misc{pith2026250617355,
  author       = {Pith},
  title        = {Pith review of: PasteTrace: A Single Source Plagiarism Detection Tool For Introductory Programming Courses},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F5YVUZAP}},
  note         = {Machine review of arXiv:2506.17355}
}
read the original abstract

Introductory Computer Science classes are important for laying the foundation for advanced programming courses. However, students without prior programming experience may find these courses challenging, leading to difficulties in understanding concepts and engaging in academic dishonesty such as plagiarism. While there exists plagiarism detection techniques and tools, not all of them are suitable for academic settings, especially in introductory programming courses. This paper introduces PasteTrace, a novel open-source plagiarism detection tool designed specifically for introductory programming courses. Unlike traditional methods, PasteTrace operates within an Integrated Development Environment that tracks the student's coding activities in real-time for evidence of plagiarism. Our evaluation of PasteTrace in two introductory programming courses demonstrates the tool's ability to provide insights into student behavior and detect various forms of plagiarism, outperforming an existing well-established tool. A video demonstration of PasteTrace and its source code, and case study data are made available at https://doi.org/10.6084/m9.figshare.27115852

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 22 canonical work pages

  1. [1]

    Introductory programming course: review and future implications,

    U. Omer, M. S. Farooq, and A. Abid, “Introductory programming course: review and future implications,” PeerJ Computer Science, vol. 7, p. e647, 2021. 1

    work page 2021

  2. [2]

    Challenges with learning to program and problem solve: An analysis of student online discussions,

    P. Piwek and S. Savage, “Challenges with learning to program and problem solve: An analysis of student online discussions,” in Proceedings of the 51st ACM Technical Symposium on Computer Science Education , SIGCSE ’20, (New York, NY , USA), p. 494–499, Association for Computing Machinery, 2020. 1 5

    work page 2020

  3. [3]

    Python and roles of variables in introductory programming: experiences from three educational institutions,

    U. Nikula, J. Sajaniemi, M. Tedre, and S. Wray, “Python and roles of variables in introductory programming: experiences from three educational institutions,” Journal of Information Technology Education: Research, vol. 6, no. 1, pp. 199–214, 2007. 1

    work page 2007

  4. [4]

    A systematic literature review on teaching and learning introductory programming in higher education,

    R. P. Medeiros, G. L. Ramalho, and T. P. Falc ˜ao, “A systematic literature review on teaching and learning introductory programming in higher education,” IEEE Transactions on Education , vol. 62, no. 2, pp. 77–90,

    work page

  5. [5]

    Students’ misconceptions and other difficulties in introductory programming: A literature review,

    Y . Qian and J. Lehman, “Students’ misconceptions and other difficulties in introductory programming: A literature review,” ACM Trans. Comput. Educ., vol. 18, oct 2017. 1

    work page 2017

  6. [6]

    Collaboration, collusion and plagiarism in computer science coursework,

    R. Fraser, “Collaboration, collusion and plagiarism in computer science coursework,” Informatics in Education-An International Journal, vol. 13, no. 2, pp. 179–195, 2014. 1

    work page 2014

  7. [7]

    Plagiarism detection: A tool survey and comparison,

    V . T. Martins, D. Fonte, P. R. Henriques, and D. da Cruz, “Plagiarism detection: A tool survey and comparison,” in 3rd Symposium on Languages, Applications and Technologies (2014) , Schloss-Dagstuhl-Leibniz Zentrum f ¨ur Informatik, 2014. 1

    work page 2014

  8. [8]

    Plagiarism detection tools and techniques: A comprehensive survey,

    M. Jiffriya, M. Jahan, and R. Ragel, “Plagiarism detection tools and techniques: A comprehensive survey,” Journal of Science-FAS-SEUSL , vol. 2, no. 02, pp. 47–64, 2021. 1

    work page 2021

Show all 22 references
  1. [9]

    O. U. press, ed., Oxford English Dictionary . 1

  2. [10]

    Software reuse and plagiarism: a code of practice,

    J. P. Gibson, “Software reuse and plagiarism: a code of practice,” ACM SIGCSE Bulletin, vol. 41, p. 55–59, July 2009. 1

    2009

  3. [11]

    Attitudes of second year computer science undergraduates toward plagiarism,

    P. Walcott, “Attitudes of second year computer science undergraduates toward plagiarism,” Caribbean Teaching Scholar, vol. 41, p. 63–80, Dec

  4. [12]

    Winnowing: local algorithms for document fingerprinting,

    S. Schleimer, D. S. Wilkerson, and A. Aiken, “Winnowing: local algorithms for document fingerprinting,” in Proceedings of the 2003 ACM SIGMOD international conference on Management of data, ACM, June 2003. 2

    2003

  5. [13]

    Detecting computer code plagiarism in higher education,

    M. Konecki, T. Orehovacki, and A. Lovrencic, “Detecting computer code plagiarism in higher education,” in Proceedings of the ITI 2009 31st International Conference on Information Technology Interfaces , IEEE, June 2009. 2

    2009

  6. [14]

    Heres, Source Code Plagiarism Detection using Machine Learning

    D. Heres, Source Code Plagiarism Detection using Machine Learning . PhD thesis, Utrecht University, Aug. 2017. 2

    2017

  7. [15]

    Machine learning for source-code plagiarism detection,

    J. Y . B. Katta, “Machine learning for source-code plagiarism detection,”

  8. [16]

    Software plagiarism detection in multiprogramming languages using machine learning approach,

    F. Ullah, J. Wang, M. Farhan, M. Habib, and S. Khalid, “Software plagiarism detection in multiprogramming languages using machine learning approach,” Concurrency and Computation: Practice and Experience, vol. 33, Oct 2018. 2

    2018

  9. [17]

    Plagiarism detection in programming assignments using machine learning,

    N. Awale, M. Pandey, A. Dulal, and B. Timsina, “Plagiarism detection in programming assignments using machine learning,” September 2020, vol. 2, p. 177–184, Jul 2020. 2

    2020

  10. [18]

    Outfox: Llm-generated essay detection through in-context learning with adversarially generated examples,

    R. Koike, M. Kaneko, and N. Okazaki, “Outfox: Llm-generated essay detection through in-context learning with adversarially generated examples,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, Mar. 2024. 2

    2024

  11. [19]

    New ai classifier for indicating ai-written text,

    J. H. Kirchner, L. Ahmad, S. Aaronson, and J. Leike, “New ai classifier for indicating ai-written text,” Jan 2023. 2

    2023

  12. [20]

    Evaluating the efficacy of ai content detection tools in differentiating between human and ai-generated text,

    A. M. Elkhatat, K. Elsaid, and S. Almeer, “Evaluating the efficacy of ai content detection tools in differentiating between human and ai-generated text,” International Journal for Educational Integrity , vol. 19, Sep 2023. 2

    2023

  13. [21]

    An integrated approach for intrinsic plagiarism detection,

    M. AlSallal, R. Iqbal, V . Palade, S. Amin, and V . Chang, “An integrated approach for intrinsic plagiarism detection,” Future Generation Computer Systems, vol. 96, p. 700–712, Jul 2019. 2

    2019

  14. [22]

    The Processing Environment, pp. 143–170. Berkeley, CA: Apress, 2007. 2 6

    2007

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.