{"id":"7300ec6a-a050-4097-a6d3-00de9275d1b6","arxiv_id":"2505.05679","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Prompt bias lessons derived from an LLM's own error explanations improve its clone detection F1 score by up to 10.81% on two benchmarks.","lead":"This paper tests whether large language models can detect code clones and whether adding short lessons based on the model's own mistakes improves their scores. It reports up to 10.81% higher F1 when such lessons are added to the prompt.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No control prompt means the reported F1 gains are not shown to be caused by the error-derived lessons; the poolC result is also in-sample, so the central causal claim is underdetermined.","rationale":"I read the paper as making two separable claims: (1) appending the eight lessons to the default prompt improves F1, and (2) this improvement is caused by the error-derived taxonomy. Claim (1) is supported by Tables 7-8 and the prediction-shift analysis in Table 6; I do not doubt the numbers. Claim (2) is the load-bearing part because it is what makes the paper a bias-mitigation framework rather than a single instance of prompt editing. The weakest point in the evidence for claim (2) is the absence of any control prompt. A matched control is standard for prompt-engineering studies because prompt length, style and instruction content all affect LLM outputs. The default versus all-lessons contrast changes all three at once. The ablation does provide some internal control—lessons 4, 5, 6 and 8 fail to improve or hurt—which shows that not any appended sentence works, but it still leaves open the possibility that a generic instruction to focus on semantics reproduces most of the gain. The poolC in-sample reuse further weakens claim (2) for that dataset, though avatar partially rescues generalizability. I also flag the confidence-table inconsistency because it bears on reproducibility: if poolC errors all have confidence 0, the described filtering cannot yield the 100 examples used to define categories. Because the avatar out-of-sample result is genuine evidence, I would not reject the paper; I would keep the reader's CONDITIONAL verdict and require the control condition and data provenance.","tokens_in":19401,"tokens_out":9399,"duration_ms":111224,"concrete_test":"Add a control condition to the evaluation in Tables 7-8: append a placebo instruction of comparable length and structure to the default prompt on both datasets—e.g., 'Carefully analyze the semantics of both code snippets before deciding whether they are clones'—and, separately, append the eight lessons in randomized order. If either control raises F1 by an amount statistically comparable to the all_lessons prompt, the specific error-derived content is not necessary for the reported gain. Also report, per dataset, how many incorrect predictions pass the 80% confidence filter; if poolC yields no such examples, state from which predictions the 100 rationales in Section 3.2.3 were sampled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The raw F1 improvements in Finding 3 are plausible, and the avatar result is a genuine out-of-sample test because the taxonomy and lessons were derived from poolC. What is not established is the causal attribution: that the improved prompts work because they encode the error categories. The evaluation (Section 3.5, Tables 7-8) compares the default prompt against prompts with lessons, but never includes a control prompt of matched length and style that is not derived from model errors. At the default operating point, recall is the bottleneck (e.g., poolC-with-comment default recall 76.89 vs all-lessons recall 91.58; avatar-with-comment default recall 76.75 vs 97.64), so any instruction that pushes the model toward labeling more pairs as clones will raise F1; the eight lessons mostly say 'textual differences should not determine clones,' which is exactly such a push. The ablation shows that some individual lessons hurt, which is evidence of content specificity, but it does not rule out a generic semantic-equivalence instruction. In addition, the poolC lessons are derived from the same test set on which the poolC gain is measured, so that half of Finding 3 is in-sample. Table 3 adds a reproducibility problem: for poolC-with-comment, 100% of incorrect predictions have confidence 0, so the stated 80%-confidence filter retains no poolC-with-comment errors; the provenance of the '100 examples' in Section 3.2.3 is therefore unclear. None of this disproves the improvement, but it means the central claim that the method works by leveraging the error taxonomy is unverified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies prompt bias of large language models (LLMs) for code clone detection. Using PaLM as the selected model, the authors run experiments on two datasets: poolC (within-language, Python clones) and an Avatar-derived cross-language dataset (Java/Python). From PaLM's incorrect predictions on poolC, they elicit high-level rationales, manually distill eight 'prompt bias mistake categories,' derive eight corresponding 'prompt bias lessons,' and append combinations of these lessons to the default prompt. They report F1 improvements up to 10.81% on Avatar and 9.77% on poolC and claim that the method is effective and statistically significant. The paper also reports prevalence statistics of the error categories and ablations of individual lessons.","tokens_in":19712,"tokens_out":2433,"duration_ms":27718,"significance":"If the central claim holds, the paper offers a practical and potentially generalizable recipe for improving LLM performance on clone detection by converting observed model errors into prompt instructions. The deterministic temperature setting, the explicit replication-package link, and the use of an out-of-sample Avatar evaluation for lessons derived from poolC are genuine strengths. The ablation study showing that some individual lessons hurt performance is also useful evidence of content specificity. However, the causal attribution of the F1 gains to the error-derived lessons is not established because of the missing control prompt and the in-sample nature of the poolC evaluation. These issues are fixable within the manuscript's scope, so the contribution is defensible but needs substantial revision.","major_comments":[{"comment":"The evaluation compares only the default prompt against prompts containing the error-derived lessons; there is no control prompt of matched length and style that conveys generic semantic-equivalence instructions without being derived from the model's errors. This matters because at the default operating point recall is the bottleneck: for example, avatar-with-comment recall rises from 76.75 to 97.64 and poolC-with-comment recall from 76.89 to 91.58 with all_lessons. Any instruction that pushes the model toward labeling more pairs as clones, such as 'textual differences should not determine clones,' would raise F1 even if the specific error taxonomy were irrelevant. The causal claim that the improved prompts work because they encode the eight identified error categories therefore requires a matched control prompt.","section":"§3.5, Tables 7-8"},{"comment":"The poolC result is in-sample: the eight error categories and the lessons were derived from PaLM's errors on poolC, and the improved prompts were then evaluated on the same poolC test set. The reported 9.77% improvement for poolC is therefore not an out-of-sample estimate. The Avatar result is a genuine external check, but it only supports generalization to the cross-language setting. To validate the poolC claim, the authors should derive the categories and lessons on a training or validation subset and evaluate on a held-out test set, or at least report a cross-validation-style procedure.","section":"§3.2.3-§3.5, Tables 7-8"},{"comment":"The stated 80% confidence filter is inconsistent with the reported confidence distribution for poolC-with-comment. Table 3 shows that 100.00% of incorrect predictions in that dataset have confidence 0, so the 80%-confidence filter would retain zero poolC-with-comment errors. Yet Section 3.2.3 says the authors 'randomly selected 100 examples' from the reliable incorrect predictions to build the taxonomy. The provenance of those 100 examples is unclear: were they drawn only from poolC-without-comment, from a mix of datasets, or from unfiltered errors? This needs clarification because it affects the reproducibility of the taxonomy and the claimed relationship between the filter and the lessons.","section":"§3.2.2, Table 3"},{"comment":"The entire taxonomy rests on the assumption that PaLM's self-reported high-level rationales for its wrong answers accurately reflect the true cause of the errors. LLMs often produce plausible post-hoc explanations that do not correspond to the actual computational cause of a misclassification. The paper does not validate the rationales against an independent, code-level analysis (for example, by checking whether the cited textual difference actually drove the prediction). If the rationales are not faithful, the eight categories and the lessons derived from them are not causally linked to the subsequent F1 improvements. The authors should either provide such validation, for instance by testing the lessons against rationales obtained from a different model or from controlled input perturbations, or explicitly weaken the causal claim.","section":"§3.2.3, Figure 4"},{"comment":"The statistical significance analysis uses a t-test for each lesson and each dataset individually, with many comparisons, but no multiple-comparison correction is reported. Given that eight lessons are tested on four datasets, some significant results would be expected by chance even if the lessons had no effect. Reporting adjusted p-values or a Bonferroni-style correction would strengthen the claim that the improvements are not artifacts of multiple testing.","section":"§4.3, Tables 7-8"}],"minor_comments":[{"comment":"The CodeT5 accuracy for poolC with comment is reported as '0.0.45', which appears to be a typo and should read 0.45.","section":"Table 2"},{"comment":"The category 'Overemphasis On Textual Similarity' appears in the prevalence table but is not defined in the list of eight categories in Section 4.1.2; either add its definition or explain the discrepancy with the eight named categories.","section":"Table 4"},{"comment":"The threats-to-validity section acknowledges that the categories were derived from only 100 poolC instances and may not generalize, but it does not acknowledge the missing control prompt as a threat to the causal claim; this should be added and discussed explicitly.","section":"§6"},{"comment":"The rationale for choosing 80% as the confidence threshold is based on the observed distribution in Table 3, but that distribution is presented only later in Section 4.1.1; consider moving the table earlier or describing the distribution when the threshold is introduced.","section":"§3.2.2"},{"comment":"The claim that this is 'the first exploration of LLMs within the domain of clone detection' is too strong given the existing literature; the authors should temper this claim or provide a more careful review of prior LLM-based clone detection work.","section":"§1"}],"recommendation":"major_revision","confidential_remarks":"The paper's central empirical finding—that adding error-derived lessons to the prompt improves F1—is plausible and the Avatar out-of-sample result is encouraging. However, the missing control prompt and the in-sample poolC evaluation are substantial gaps that prevent the causal claim from being accepted as stated. These are fixable with additional experiments, so I do not recommend rejection. The paper would also benefit from a more careful treatment of the confidence-filter inconsistency and the reliance on self-reported rationales."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know up front: this is a genuine empirical case study with one real out-of-sample check, but the central causal claim outruns the design. The paper reports F1 gains of roughly 7–11 points from appending eight manually written “lessons” to PaLM’s prompt on two clone benchmarks (avatar, cross-language; poolC, within-language), plus an eight-category taxonomy of the model’s errors. The avatar result is the most convincing piece: the taxonomy and lessons were derived entirely from poolC, and they still transfer to a different-language dataset. That is real evidence the lessons carry signal rather than merely memorizing the test set.\n\nWhat is actually new: among the cited work, no one has evaluated instruction-tuned LLMs for clone detection, and mining model rationales to build prompt-bias categories is a first. The per-lesson ablation is a nice touch and shows content specificity—lesson 7 (“code logic matters more than minor code differences”) is robustly the best, while lessons 4 and 8 hurt on most datasets. There is a replication package.\n\nThe soft spots are real. Most important: no control prompt. The all-lessons prompt is longer, more permissive, and full of “textual differences should not determine clones” instructions, so any instruction pushing the model toward “yes” would lift recall—and recall is exactly the bottleneck at the default operating point. A length-and-style-matched generic semantic-equivalence control would have shown whether the error-derived lessons cause the gain. Second, poolC is in-sample: lessons were built from errors on the same test set that later measures the poolC improvement, and the authors’ own threats-to-validity section concedes the categories came from poolC. Third, Table 3 contradicts the method: for poolC-with-comment, 100% of incorrect predictions have confidence 0, so the 80% confidence filter retains zero errors from that dataset, yet the text says 100 reliable incorrect examples were reviewed. The provenance of those examples is unexplained. Fourth, the rationales are self-reported and may be post-hoc; the avatar transfer softens this, but it remains load-bearing for the mechanism. Minor rough edges: PaLM is called a “GPT model,” the category list in Section 4.1.2 does not quite match Table 4’s “Overemphasis on Textual Similarity,” and the ACM placeholder boilerplate needs cleanup.\n\nNone of this disproves the improvement; the causal attribution is what is unproven, and the fix is one control experiment plus a clarified sampling pipeline. I would send this to serious peer review—the right reviewer will ask for exactly that.","headline":"Real, partially out-of-sample F1 gains and a first error taxonomy for LLM clone detection, but no control prompt and an in-sample poolC half leave the causal story unproven.","tokens_in":20239,"tokens_out":5216,"would_cite":true,"duration_ms":52178,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that prompting a large language model with lessons distilled from its own confident wrong answers lifts clone-detection F1 by up to 10.81 percentage points on a cross-language benchmark and 9.77 on a within-language…","keywords":["code clone detection","prompt bias","large language models","prompt engineering","error analysis","semantic clones","cross-language clone detection","PaLM"],"falsifier":"Run the eight lessons against a control prompt of matched length and tone whose instructions are not derived from the model's errors, such as generic advice that code can be functionally equivalent despite textual differences: if the control prompt produces the same F1 improvement, the error-driven derivation is not what matters. Alternatively, build lessons from deliberately shuffled or fabricated rationales and check whether the improvement still appears.","tokens_in":19212,"feed_emoji":"🤖","tokens_out":9635,"duration_ms":93292,"temperature":0.7,"pith_summary":"The paper attempts to show that a large language model's systematic mistakes on code-clone detection can be mined, classified, and converted into prompt instructions that fix a substantial share of those mistakes. On two clone-detection benchmarks, one within Python and one cross-language (Java/Python), the PaLM model scored F1 near 90 before any modification. The authors extracted the model's self-reported rationales for its confident wrong answers, grouped them into eight recurring error categories, and rewrote each category as a one-sentence 'lesson' appended to the prompt. Adding all eight lessons raised F1 by up to 10.81 points on the cross-language benchmark and 9.77 points on the within-language benchmark. If the result holds, it means a model's biases can be identified and partially corrected using nothing but its own explanations, with no retraining and no access to weights.","feed_headline":"Lessons from a model's errors boost clone-detection F1 by 11 points","feed_subtitle":"Injecting eight distilled lessons from wrong answers lifts PaLM's F1 on avatar and poolC clone benchmarks.","key_machinery":"The load-bearing object is the 'prompt bias lesson': a one-sentence instruction, derived from a mistake category, appended to the clone-detection prompt. The pipeline is: run the model with a default prompt, keep only confident wrong answers, ask the model to explain those errors in its own words, cluster the explanations into eight categories, translate each category into a lesson, and measure F1 before and after appending the lessons. Two human raters did the categorization and lesson-writing, with high reported inter-rater agreement. The clone-type taxonomy (identical, renamed, near-miss, semantic) supplies the interpretive frame: most categories describe the model treating semantic equivalence as non-clones because it overweights surface text.","core_discovery":"The paper's central claim is that prompt bias in LLM-based clone detection is not random fluctuation but a small set of recurring conceptual confusions, and that naming those confusions in the prompt recovers most of the lost performance. The eight categories range from a narrow reading of 'clone' (only identical code counts) to confusions about operators, variable naming, data structures, function and library names, thematic content, comprehension failures, and divergent implementation styles; most map to the model failing on Type 2-4 clones, especially semantic clones. The authors report that the most frequent category, misinterpretation of function and library API nomenclature, appears in 65.39% of wrong answers averaged across datasets. When all eight lessons were appended to the default prompt, PaLM's F1 rose from 89.30 to 96.48 on the avatar without-comment set and from 86.41 to 94.55 on poolC with-comment, with all gains reported as statistically significant; the single strongest lesson was that code logic outweighs minor code differences.","pith_inferences":["Inference: A natural extension is to automate the loop, having the model propose lessons from its own rationales and selecting them by validation-set performance, which the paper flags as future work.","Inference: If the causal story is right, the same rationale-to-lesson procedure should transfer to other code tasks such as bug detection, code summarization, or security analysis, where models also show recurring, prompt-sensitive failure modes.","Inference: A placebo-prompt control (generic advice matched in length, not derived from observed errors) would separate the effect of domain knowledge injection from the effect of error-specific diagnosis; the paper does not report such a control.","Inference: The improvement may in part reflect that the lessons are effective generic reminders about clone semantics rather than corrections tied to the model's specific biases; comparing lessons derived from a different model's errors would test that."],"forward_implications":["LLMs without fine-tuning can serve as strong clone detectors, reaching F1 scores near 90 on both within-language and cross-language benchmarks.","Prompt bias is classifiable: eight error categories account for the bulk of confident mistakes, and the same lessons generalize from the Python dataset to the Java/Python dataset.","The all-lessons prompt converts many more errors to correct answers than it introduces (for example 1,166 vs. 101 on poolC with comment), so the improvement is not a simple precision-recall trade.","Lesson 7 alone, which tells the model that code logic matters more than minor differences, gives the largest per-lesson gain on all four dataset variants."],"supporting_citations":[{"why":"Supplies the poolC dataset used for within-language clone detection.","marker":"[5]"},{"why":"Supplies the avatar translation pairs that the authors converted into a cross-language clone-detection benchmark.","marker":"[7]"},{"why":"Introduces the PaLM model whose predictions and rationales drive the analysis.","marker":"[17]"},{"why":"Provides the topic-modeling approach used to measure how often each error category occurs.","marker":"[24]"},{"why":"Defines the Type 1-4 clone taxonomy used to interpret the model's failure modes.","marker":"[46]"},{"why":"Supplies CodeT5 as the BERT-family baseline in the model comparison.","marker":"[54]"},{"why":"Provides the method for generating the default prompt and is the prior prompt-bias work this paper extends.","marker":"[62]"},{"why":"Documents the GPT baseline models (Turbo-3.5 and Text-002) compared during model selection.","marker":"[4]"}],"fun_headline_variants":["Turn model mistakes into prompts to fix clone detection bias","Eight error categories unlock 11-point F1 boost in clone detection","Prompt bias tamed: model errors teach better clone detection","From bias to boost: error-driven prompts sharpen clone detection","Leveraging LLM errors lifts clone detection by 11 F1 points"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that PaLM's self-reported rationales for its wrong answers accurately reflect the true causes of those errors; if the rationales are post-hoc stories rather than genuine causes, the eight categories and lessons are not the reason the prompt helped.","fun_headline_variants_meta":{"raw":{"variants":["Turn model mistakes into prompts to fix clone detection bias","Eight error categories unlock 11-point F1 boost in clone detection","Prompt bias tamed: model errors teach better clone detection","From bias to boost: error-driven prompts sharpen clone detection","Leveraging LLM errors lifts clone detection by 11 F1 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000261,"raw_usage":{"total_tokens":1629,"prompt_tokens":1018,"completion_tokens":611,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":525}},"tokens_in":634,"tokens_out":611,"duration_ms":6306,"temperature":1.0,"reasoning_tokens":525,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:59:31.223312+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the eight lessons against a control prompt of matched length and tone whose instructions are not derived from the model's errors, such as generic advice that code can be functionally equivalent despite textual differences: if the control prompt produces the same F1 improvement, the error-driven derivation is not what matters. Alternatively, build lessons from deliberately shuffled or fabricated rationales and check whether the improvement still appears.","supporting_citations":[{"cited_title":"PoolC-5-fold-clone-detection-600k-5fold, howpublished = https://huggingface.co/datasets/poolc/5-fold-clone- detection-600k-5fold, note = Accessed: 2023-09-23","cited_arxiv_id":null,"evidence_quote":"Supplies the poolC dataset used for within-language clone detection."},{"cited_title":"wasiahmad-AVATAR, howpublished = https://github.com/wasiahmad/avatar, note = Accessed: 2023-09-23","cited_arxiv_id":null,"evidence_quote":"Supplies the avatar translation pairs that the authors converted into a cross-language clone-detection benchmark."},{"cited_title":"Bertopic: Neural topic modeling with a class-based tf-idf procedure,","cited_arxiv_id":null,"evidence_quote":"Provides the topic-modeling approach used to measure how often each error category occurs."},{"cited_title":"A survey on software clone detection research,","cited_arxiv_id":null,"evidence_quote":"Defines the Type 1-4 clone taxonomy used to interpret the model's failure modes."},{"cited_title":"OpenAI-models, howpublished = https://platform.openai.com/docs/models/gpt-3-5, note = Accessed: 2023-09-23","cited_arxiv_id":null,"evidence_quote":"Documents the GPT baseline models (Turbo-3.5 and Text-002) compared during model selection."}],"review_version":1}