{"id":"b3792a82-784e-4489-a82e-4c54ce6d8d39","arxiv_id":"2505.08244","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Code similarity detectors miss obfuscated or AI-generated submissions in competitive programming, so the author recommends combining automated screening, manual review, and oral interviews.","lead":"This paper argues that standard plagiarism checkers such as Moss and Kattis often fail in competitive programming courses when students transform code or use generative AI, and that oral interviews are more reliable but costly. A practitioner report from seven semesters of a Purdue course, it offers a concrete workflow and recommendations for educators facing AI-assisted cheating.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Interview-as-gold-standard makes the 'near 100% success rate' circular; Section 4.4 admits the one case that breaks it, so the paper's flagship empirical claim is not established.","rationale":"I agree with the reader's weakest assumption; this is the point where the central claim is least secure. The conceptual thesis (automated similarity checkers are evadable, and interviews are strong but labor-intensive) is well supported by prior literature and by the author's detailed experience, so this is not a rejection of the paper. But the paper makes a stronger quantitative-sounding claim: 'our success rate approaches 100% in detecting plagiarism this way,' and 'in nearly 100% of the cases, the suspects faltered or confessed.' Two problems follow. First, circularity: interviews are used to confirm exactly what they are used to detect; no independent authorship ground truth exists for the cases that pass. Second, the manuscript itself explicitly concedes the adversarial case in Section 4.4: 'what if a student uses AI to generate code and then thoroughly learns that code? In an interview they might pass as if they wrote it.' That concession is not a minor caveat; it is the exact counterexample to the gold-standard assumption. The empirical incident numbers in Table 1 are also not incidence estimates because the detection pipeline changed over time (MOSS introduced in S24, manual BSTA search in S25, interviews introduced in S25, and genAI policy varied across semesters). The proposed test directly measures false negatives in the interview protocol. If the concern holds, the paper should reframe the success-rate sentence as anecdotal and add this limitation explicitly; the practical recommendations can remain conditional. I therefore keep the reader's CONDITIONAL verdict.","tokens_in":31945,"tokens_out":3540,"duration_ms":35165,"concrete_test":"Run a blinded validation on the S25 pipeline: take about 20 unflagged accepted submissions from the same weekly problem sets, 10 known to be authored by the submitting student and 10 generated by ChatGPT or GitHub Copilot and then studied by a different student for about one hour, with the interviewer blind to provenance. Apply Protocol 1 verbatim and record pass or fail per submission. If the studied-AI group passes at rates comparable to the genuine authors, the 'near 100% success' claim and the recommendation to scale interviews are unsupported; if the studied-AI group fails nearly all cases, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's main empirical claim is that a multi-faceted strategy caught or deterred most plagiarism, and specifically that suspect interviews have a 'near 100% success rate' in confirming plagiarism (Section 3; Section 4.3.2). The load-bearing assumption is that failure in an interview is a valid gold standard for non-authorship. This is circular: the same Protocol 1 interview is used both to generate suspicion and to 'confirm' it, and there is no external validation of false negatives. The paper itself identifies the failure mode in Section 4.4: a student who uses genAI and then thoroughly learns the generated code can pass an interview, so detection fails while academic integrity was still breached. Because a plagiarist who prepares can be indistinguishable from an author under Protocol 1, the 'nearly 100%' figure measures only agreement among detection steps, not actual detection performance. The recommendation to scale interviews inherits this uncertainty. Without independent ground truth, Table 1's incident counts are also best read as detection-effort counts (for example, the 15 BSTA cases were found by manual search, not by the tools), not as plagiarism incidence or as a validated success rate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper is an experience-based analysis of plagiarism detection in competitive programming courses, drawing on the author's seven semesters of teaching CP1 at Purdue and on a redesign of the CP1/2/3 sequence. It reviews the literature on code plagiarism, automated similarity tools (Moss, Kattis, JPlag), manual review, oral interviews, and generative AI, and combines this with a set of incident counts (Table 1) and case studies. The central claims are that automated similarity checkers can be evaded by simple transformations and by novel AI-generated code, that human-centric interviews are the most effective verification method but are labor-intensive, and that a multi-faceted strategy combining tools, manual review, and interviews catches or deters most plagiarism. The paper also introduces the term 'whiteboard similarity,' provides an interview protocol (Protocol 1), a strengths/weaknesses comparison (Table 2), and appendices on tracked solution repositories, a plagiarism ring, and example MOSS outputs.","tokens_in":32186,"tokens_out":3740,"duration_ms":41675,"significance":"If read as a practitioner's experience report, the paper is useful: it gives a concrete interview protocol, documents real failure modes of Moss/Kattis, and offers a sensible set of recommendations for multi-pronged detection and authentic assessment. The detailed appendices and the explicit acknowledgment of the genAI edge case in Section 4.4 are strengths. However, the paper's quantitative empirical claims—especially the 'near 100% success rate' of interviews and the interpretation of Table 1—are not supported by the evidence as presented, and they are load-bearing for the recommendation to scale interviews. The central conceptual argument is defensible, but the empirical framing needs to be corrected before the claims can be accepted.","major_comments":[{"comment":"The claim that suspect interviews have a 'near 100% success rate' in confirming plagiarism is circular as stated: Protocol 1 is used both to decide which students are suspected and to 'confirm' that the suspicion is correct, with no independent ground truth for false negatives. Section 4.4 explicitly acknowledges the edge case of a student who uses AI, thoroughly learns the generated code, and passes an interview, which means the interview is not a validated gold standard. The reported rate therefore measures agreement among detection steps rather than detection accuracy. Please reframe the claim as 'near 100% agreement among screening signals' or provide external validation (for example, confessions, follow-up assessment outcomes, or blinded re-interviews by a second interviewer).","section":"Sections 3 and 4.3.2"},{"comment":"The incident counts in Table 1 are not comparable across semesters and do not by themselves support the conclusion that the multi-faceted strategy 'caught or deterred the majority' of plagiarism cases. The table spans periods with different detection effort and policy: MOSS runs began in S24, F24 included a genAI-allowed intervention for part of the course, S25 introduced weekly interviews and a required final, and the 15 BSTA cases in S25 were found by manual search rather than by the automated tools. The counts are therefore better described as detection-effort counts, not as plagiarism incidence. Please provide the relevant denominators, time at risk, and any changes in detection effort alongside the table, and avoid statements of success rates that go beyond what the counts can show.","section":"Table 1 and Section 3"},{"comment":"The BSTA case is presented as a 'failure of automated detection,' but the same paragraph indicates that a few students were initially flagged and that the remaining cases were found only after a manual search through all accepted submissions. The interpretation of this example depends on which of the 15 students were flagged by which tool and on the untested assumption that the interview outcome is ground truth. Please specify the detection pathway for each of the 15 students (Moss, Kattis, TA report, or manual search) and state how the interview protocol was validated in this particular episode, so readers can assess what exactly the case demonstrates.","section":"Section 3, BSTA example"}],"minor_comments":[{"comment":"Reference [54] is cited as 'arXiv preprint arXiv:TBA'; please provide a complete citation or mark it as forthcoming.","section":"References"},{"comment":"The phrase 'our success rate approaches 100%' is stated without the underlying denominator; please report the number of suspect interviews conducted and the number of confirmed outcomes so the statement is meaningful.","section":"Section 4.3.2"},{"comment":"The definition of 'serious incidents' appears only in the table note and is easy to miss; please define it in the main text near the table.","section":"Table 1"},{"comment":"The new term 'whiteboard similarity' is defined only by contrast with 'code collaboration'; a short sentence relating it to existing notions of legitimate collaboration would help readers who are unfamiliar with the term.","section":"Section 2.1"}],"recommendation":"major_revision","confidential_remarks":"This is a practitioner-oriented experience report rather than a controlled empirical study, and its value lies in the detailed protocol and concrete observations. The self-citations to the author's related work are used for context and are not load-bearing for the main argument, but the placeholder citation [54] should be corrected. Given that the central claims are plausible and the main empirical overreach can be fixed by reframing the interview success rate and the Table 1 counts, I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a candid practitioner report from someone who taught a competitive programming course for seven semesters. The main claim—Moss and Kattis are easily evaded by AI-generated or obfuscated code, so human interviews are needed—is correct, but it is not new. The paper itself cites the work that already established it (Biderman and Raff 2022, Taylor et al. 2023, Devore-McDonald and Berger 2020). What the paper adds is concrete field experience: a codified interview protocol, a detailed case study of a plagiarism ring, MOSS examples, and practical workflow notes that will be genuinely useful to instructors.\n\nIt does several things well. The literature review is competent and well-sourced. The paper is unusually honest about its limitations: it explicitly flags the F24 genAI experiment, admits in Section 4.4 that a student who learns AI-generated code can pass an interview, and notes that Table 1's incident counts are confounded by changes in detection effort. That kind of candor is welcome.\n\nThe soft spots are where the paper tries to convert experience into evidence. The 'near 100% success rate' for interviews is circular in a narrow but real sense: the interview protocol is used to confirm suspicions raised by the same detection pipeline, and there is no independent ground truth. The paper's own Section 4.4 describes the case that breaks the gold standard. So the number measures agreement among detection steps, not detection performance, and Table 1 counts detection effort rather than validated plagiarism incidence. The central claim survives—tools do fail and interviews are a reasonable complement—but the empirical support is thinner than the narrative suggests.\n\nOne more concrete issue: reference [54] is a placeholder ('arXiv:TBA'), and it is cited as the parallel curriculum-design paper. That needs fixing before publication.\n\nMinor: the term 'whiteboard similarity' is fine but lightly defined. The self-citations are supporting context, not load-bearing, so they do not bother me.\n\nWho this is for: CS educators, especially instructors of programming courses that use online judges, who want a field-tested playbook. Researchers will find little new here. It deserves a serious referee because it is a useful experience report with actionable detail, but it should be reviewed as such, not as rigorous empirical research. My recommendation: send it out, and ask the author to reframe the interview claim as a practice observation, explicitly acknowledge the circularity and lack of ground truth, and present Table 1 as detection-effort counts.","headline":"A useful, honest practitioner report whose core claim is already established; the interview 'success rate' is circular and the empirical evidence is thinner than the narrative suggests.","tokens_in":32685,"tokens_out":4034,"would_cite":false,"duration_ms":38890,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Similarity checkers miss AI-generated code and lightly rewritten copies, so competitive programming courses must add human interviews to the detection mix.","keywords":["code plagiarism","competitive programming","similarity detection","Moss","Kattis","generative AI","oral interviews","academic integrity"],"falsifier":"A controlled study using the paper's Protocol 1: recruit students who genuinely wrote their own solutions and another group who copied or used AI but then studied the code until they could explain it; run blinded interviews and measure how many of the second group pass. If a substantial fraction passes, the near-100% success claim fails, while high failure rates in the first group would indicate a false-accusation problem.","tokens_in":31748,"feed_emoji":"🤖","tokens_out":5488,"duration_ms":53741,"temperature":0.7,"pith_summary":"This paper argues that in competitive programming courses, routine automated plagiarism detection fails against two common tactics: lightly transformed copies of known solutions and freshly generated AI code, both of which evade fingerprint-matching. Drawing on seven semesters of running a competitive programming course, the author claims that a multi-layered strategy—automated screening, manual review, and one-on-one code interviews—catches or deters the majority of plagiarism cases, with interviews doing the decisive work. The paper's central recommendation is that institutions combine improved detectors with authentic, human verification rather than relying on any single tool.","feed_headline":"Similarity checkers miss AI-generated code in programming courses","feed_subtitle":"A seven-semester teaching record shows oral interviews catch what automated tools don't.","key_machinery":"The analysis is carried by comparing four detection mechanisms: token-fingerprinting similarity tools (Moss), n-gram and hash comparisons (Kattis), manual code review, and oral code-authorship interviews, with code stylometry suggested as a future addition. The load-bearing object is the interview protocol: a short oral exam in which a student explains their accepted solution, reproduces key reasoning, and describes a bug they fixed, without access to the code. The paper uses this interview as the ground truth against which the other methods are measured, and it introduces the notion of 'whiteboard similarity'—similarity that arises from two students solving a problem together without seeing each other's code—to explain why independent students rarely produce matches that similarity tools should flag.","core_discovery":"The paper's central discovery is that automated code-similarity systems such as Moss and Kattis's built-in checker are reliable only against direct copy-paste and lightly disguised copying; they are defeated by systematic rewrite tools and by generative-AI output that is original by construction. In the author's competitive programming course, a weekly routine of running similarity reports, manually inspecting flagged pairs, and interviewing a random subset of students uncovered plagiarism that no automated tool flagged—including fifteen students who submitted an advanced technique they had not been taught and could not explain. The paper claims that the interview protocol, in which a student must explain their solution, its runtime, and a struggle they overcame, approaches 100% success in confirming which students do not understand their own code, and that the mere prospect of interviews deters most repeat cheating.","pith_inferences":["If interview-based verification is genuinely near-100% accurate, the most scalable institutional response is to shift more assessment weight to live, proctored demonstrations rather than trying to perfect text-similarity algorithms.","The paper's 'whiteboard similarity' observation implies that pairwise code similarity is a noisy proxy for collusion; a cleaner target for automated tools may be authorship consistency over time within one student, that is, stylometric anomaly detection.","Because the paper reports that most confirmed cheaters were caught early in the semester and few repeated after a warning, a testable extension is whether early-term interviews alone can reduce end-of-term plagiarism rates.","The reported rise in denials and the edge case of a student who learns AI-generated code well enough to pass an interview suggest that as interviews become expected, cheaters will invest in understanding copied code, eroding the ground truth the method relies on."],"forward_implications":["Courses that rely only on similarity scores will miss AI-generated submissions, since novel AI output has no matching source to flag.","A weekly rotation of short interviews, covering every student at least once, can serve as both a deterrent and a confirmation step at modest staff cost.","Detectors improve when instructors add known online solution repositories and a few generated AI solutions as base files in each similarity run.","Process anomalies, such as one-shot perfect solutions from students who usually struggle, are a useful complement to text similarity.","Cross-language plagiarism—translating a solution from one programming language to another—is currently invisible to all commercial tools and requires manual or stylistic review."],"supporting_citations":[{"why":"Defines the winnowing fingerprinting algorithm behind Moss and states that similarity alone does not prove plagiarism.","marker":"[12]"},{"why":"Controlled experiment showing ChatGPT-generated code has lower pairwise similarity to student code and is not flagged by Moss.","marker":"[27]"},{"why":"Describes Kattis's plagiarism detection on every submission, its n-gram and hash approach, and its limitations.","marker":"[14]"},{"why":"Presents the Mossad tool that inserts dummy statements and rearranges code to defeat similarity detectors.","marker":"[47]"},{"why":"Shows generative AI availability increased plagiarism incidence and shifted cheating from solution repositories to AI output.","marker":"[23]"},{"why":"Documents false positives on short problems where 93% of submissions appear similar despite no evidence of cheating.","marker":"[46]"},{"why":"Provides the scale of manual-review burden in large classes, with thousands of flagged pairs winnowed to dozens of confirmed cases.","marker":"[7]"},{"why":"Argues oral or viva voce assessment can counter AI-generated answers because impromptu questioning is hard to fake.","marker":"[16]"},{"why":"Supplies code stylometry research that could flag authorship divergence, the paper's proposed future detector.","marker":"[51]"}],"fun_headline_variants":["AI code evades plagiarism checkers in competitive programming","Why plagiarism detectors fail on AI-written code","Oral interviews catch what Moss misses in competitive programming","Automated checkers beaten by AI and clever rewrites","Plagiarism detection fails against AI-generated submissions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported success rate rests on treating the interview as a reliable lie detector: a student who cannot explain their submitted code is assumed to have plagiarized, and the paper assumes this failure cannot be convincingly faked.","fun_headline_variants_meta":{"raw":{"variants":["AI code evades plagiarism checkers in competitive programming","Why plagiarism detectors fail on AI-written code","Oral interviews catch what Moss misses in competitive programming","Automated checkers beaten by AI and clever rewrites","Plagiarism detection fails against AI-generated submissions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000607,"raw_usage":{"total_tokens":2815,"prompt_tokens":916,"completion_tokens":1899,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":1826}},"tokens_in":532,"tokens_out":1899,"duration_ms":12029,"temperature":1.0,"reasoning_tokens":1826,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:59:13.258734+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled study using the paper's Protocol 1: recruit students who genuinely wrote their own solutions and another group who copied or used AI but then studied the code until they could explain it; run blinded interviews and measure how many of the second group pass. If a substantial fraction passes, the near-100% success claim fails, while high failure rates in the first group would indicate a false-accusation problem.","supporting_citations":[{"cited_title":"Winnowing: lo- cal algorithms for document fingerprinting,","cited_arxiv_id":null,"evidence_quote":"Defines the winnowing fingerprinting algorithm behind Moss and states that similarity alone does not prove plagiarism."},{"cited_title":"Plagiarism in entry- level computer science courses using chatgpt,","cited_arxiv_id":null,"evidence_quote":"Controlled experiment showing ChatGPT-generated code has lower pairwise similarity to student code and is not flagged by Moss."},{"cited_title":"Discovering plagiarism in introductory programming courses through the application of multiple methods,","cited_arxiv_id":null,"evidence_quote":"Describes Kattis's plagiarism detection on every submission, its n-gram and hash approach, and its limitations."},{"cited_title":"Plagiarism in the age of generative ai: cheating method change and learning loss in an intro to cs course,","cited_arxiv_id":null,"evidence_quote":"Shows generative AI availability increased plagiarism incidence and shifted cheating from solution repositories to AI output."},{"cited_title":"An Approach to Detect Abnormal Submissions for CodeWorkout Dataset","cited_arxiv_id":"2407.17475","evidence_quote":"Documents false positives on short problems where 93% of submissions appear similar despite no evidence of cheating."},{"cited_title":"Why we still can’t stop plagiarism in undergraduate computer science,","cited_arxiv_id":null,"evidence_quote":"Provides the scale of manual-review burden in large classes, with thousands of flagged pairs winnowed to dozens of confirmed cases."},{"cited_title":"How to avoid plagiarism: Is viva voce the answer?","cited_arxiv_id":null,"evidence_quote":"Argues oral or viva voce assessment can counter AI-generated answers because impromptu questioning is hard to fake."},{"cited_title":"De-anonymizing programmers via code stylometry,","cited_arxiv_id":null,"evidence_quote":"Supplies code stylometry research that could flag authorship divergence, the paper's proposed future detector."}],"review_version":1}