{"id":"45b3713d-2b15-4b6e-b5ef-5c26894918e9","arxiv_id":"2501.09888","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Large new Python/Java benchmarks, diff-based metrics (BLEU-diff, CrystalBLEU-diff, LEMOD), and an LLM evaluation showing ~10% exact-match SATD repayment and larger models winning on fine-grained metrics.","lead":"This paper builds large new datasets of real-world code fixes where developers repaid 'self-admitted technical debt' (known suboptimal code marked in comments), and tests how well large language models can generate such fixes. It also proposes new ways to score the generated fixes, and finds that large models with prompts often outperform small fine-tuned models on fine-grained quality, though exact matching remains below 11%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reference diffs are not line-level pure: the step-10 LLM filter validates whole-method relevance only, so EM, BLEU-diff, CrystalBLEU-diff, and LEMOD all inherit unrelated edits and may misstate both absolute and relative SATD-repayment ability.","rationale":"The reader's weakest assumption identifies essentially the same risk: the diff between the method before and after SATD removal is treated as exactly the SATD repayment, with no line-level verification. I agree that this is the most load-bearing concern because it underlies every reported number. If the reference diffs contain unrelated edits, then EM is not a valid measure of SATD repayment ability, and the proposed diff-based metrics are not valid either, since they reward or penalize reproduction of those unrelated lines. This directly threatens the central claims that (a) current LLMs address 10.1%/8.1% of SATDs and (b) fine-tuned small models underperform prompt-based large models on diff-based metrics. The concern is not merely theoretical: the paper's own pre-filter manual evaluation found a large fraction of unclear or unrelated updates, and the post-filter validation is limited to 100 samples per language at whole-method granularity. The proposed concrete test would settle whether contamination is material by measuring line-level purity and re-running the headline comparisons on clean diffs. I do not recommend changing the reader's verdict because the paper's results are plausible and the issue is an addressable validity threat rather than a demonstrated error; CONDITIONAL already captures the need for this evidence. A separate concern about possible RQ2/RQ4 test-set overlap in prompt selection is secondary; it could be checked by computing the overlap between the RQ2 random 1000-sample sets and the RQ4 test splits, but the diff-purity issue is more fundamental to the benchmark's construct validity.","tokens_in":28759,"tokens_out":9419,"duration_ms":106216,"concrete_test":"Manually annotate at line level a random sample of 200 accepted Python and 200 accepted Java items from the final filtered datasets, labeling each reference-diff line as SATD-repayment-related, unrelated, or mixed, and compute the fraction of items containing at least one unrelated line. Then recompute EM, BLEU-diff, CrystalBLEU-diff, and LEMOD for Tables 12 and 13 on the subset of items with fully pure diffs, or with unrelated lines masked out of both reference_diff and candidate_diff. If the headline EM values or the relative ordering of fine-tuned versus prompt-based models changes materially, the central claim is not robust to reference-diff contamination.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's ground truth is the full method diff, not the lines that actually repay the SATD. Section 3.1 (step 10) uses Llama-3-70B to judge whether the whole method update is related to the SATD, and Table 5 reports precision of 0.830 (Python) and 0.926 (Java) on predicted \"Yes\" items. This validates whole-method relevance, not line-level purity. The authors' own pre-filter manual check found 55/100 Python and 38/100 Java samples were \"unclear or unrelated\" (Section 3.1 and Figure 1 context), and after filtering about 8-17% of accepted items can still be false positives. Moreover, even true repayments often occur alongside unrelated edits, as the paper itself notes from Zampetti et al. [39]. Because EM, BLEU-diff, CrystalBLEU-diff, and LEMOD are all computed against reference_diff = diff(input_code, ground_truth) (Section 3.2), any unrelated changed line must be reproduced exactly for EM=1 and contributes to every diff-based score. An LLM that correctly repays the SATD but does not reproduce the developer's unrelated contemporaneous edits is penalized. Thus the headline 10.1%/8.1% EM figures and the RQ4 ranking (fine-tuned models underperform on diff metrics) are not clean estimates of SATD-repayment ability; they are contaminated by the noise in the reference diffs. The threat is acknowledged only indirectly in Section 7 as \"some items with incorrect ground truth remain,\" but the paper does not quantify line-level contamination or show that the main results survive it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces two large-scale SATD-repayment datasets (58,722 Python and 97,347 Java items), constructed with an SATD tracking tool and a 10-step filtering pipeline whose final step uses an LLM judge to remove updates unrelated to SATD repayment. It proposes three diff-based evaluation metrics—BLEU-diff, CrystalBLEU-diff, and LEMOD—and uses them, together with EM, to compare fine-tuned CodeT5p models against five prompt-based LLMs across four prompt templates. The main empirical findings are that fine-tuned small models achieve EM scores comparable to prompt-based approaches while lagging on the fine-grained diff metrics, and that Gemma-2-9B with a simple prompt achieves the best EM (10.1% Python, 8.1% Java), well above the 2.3% reported by prior work.","tokens_in":29169,"tokens_out":5661,"duration_ms":56630,"significance":"If the central claims hold, this is a substantial empirical contribution to automated SATD repayment. The paper contributes large public datasets, a reproducible evaluation pipeline, and new metrics that correlate strongly with EM, and it provides the first multi-language comparison of fine-tuned and prompt-based LLMs on this task. The finding that fine-tuned small models match prompt-based models on EM but underperform on finer-grained diff metrics is useful and non-obvious, and the large improvement over the previous 2.3% EM baseline would change the practical outlook for the task. The availability of datasets and source code is a clear strength, as is the attention to metric interpretability.","major_comments":[{"comment":"The benchmark ground truth is the full method diff, not the lines that actually repay the SATD. The step-10 filter asks Llama-3-70B whether the whole method update is related to the SATD comment; Table 5 reports precision of 0.830 (Python) and 0.926 (Java) on the predicted 'Yes' items, so 8-17% of accepted items can still be false positives, and even true repayments often co-occur with unrelated edits, as the paper notes from Zampetti et al. [39]. Because EM, BLEU-diff, CrystalBLEU-diff, and LEMOD are all computed against reference_diff = diff(input_code, ground_truth), any unrelated changed line must be reproduced exactly for EM=1 and contributes to every diff-based score. Section 7 acknowledges that 'some items with incorrect ground truth remain' but does not quantify line-level contamination or show that the headline results in Tables 12/13 survive it. I ask for a line-level purity analysis on a random sample, and a recomputation of the RQ4 results using reference diffs restricted to SATD-related lines.","section":"Section 3.1 (filter step 10) and Section 3.2 (reference_diff)"},{"comment":"The 1,000 RQ2 samples are selected 'from each dataset' without any explicit statement about their relation to the repository-wise split used in RQ4. The per-model best prompt identified in RQ2 (Tables 7/8) is then used to produce the RQ4 rankings in Tables 12/13. If the RQ2 sample overlaps the RQ4 test set, the prompt selection is informed by test data, which can inflate RQ4 scores and potentially change the comparison between fine-tuned and prompt-based models. The manuscript must state explicitly whether the RQ2 samples are a subset of the RQ4 test set or are drawn disjointly from it, and should preferably select prompts on training or validation data only.","section":"Section 3.2 (RQ2) and Section 3.4 (RQ4)"},{"comment":"The operational definitions of BLEU-diff and CrystalBLEU-diff are underspecified. The text says that diffs are computed with difflib and that BLEU/CrystalBLEU are then calculated 'on these diffs,' but it does not specify whether deleted lines, inserted lines, or a concatenation of both are used, how the brevity penalty applies to a diff string, or how line-level matches are tokenized. Since these metrics are central to the RQ4 comparison and are claimed as a contribution, the definitions need to be precise enough to reproduce Tables 12/13 from the paper alone.","section":"Section 3.2 (BLEU-diff and CrystalBLEU-diff)"}],"minor_comments":[{"comment":"The caption refers to the 'Maldonado dataset,' but the comparison is with the Mastropaolo dataset; please correct the attribution.","section":"Table 3 caption"},{"comment":"The statement that the precision and recall errors of the filter 'counteract each other' is not justified; precision and recall errors can bias scores in either direction and should be quantified separately rather than assumed to cancel.","section":"Section 7 (Internal Validity)"},{"comment":"The baseline row uses the input code after ICD removal, which is not a model output; please state explicitly that this baseline is illustrative and is not part of a statistical comparison among models.","section":"Section 5.2, Table 6"},{"comment":"The paper alternates between 'adopt' and 'design' for the 10-step filtering pipeline; since steps 1-8 are explicitly taken from Mastropaolo et al., the contribution should be phrased as the two new filtering steps (9 and 10).","section":"Section 3.1"},{"comment":"The layout of the correlation matrix is confusing because the Python and Java results are combined in one table with unclear upper/lower triangle labels; please present two separate matrices or label the triangles unambiguously.","section":"Table 18"},{"comment":"The oracle-template results are an upper bound and should be accompanied by an explicit caveat that they are not attainable without a working selector.","section":"Section 6.1"}],"recommendation":"major_revision","confidential_remarks":"The two main technical concerns—line-level contamination of the reference diffs and the unclear relation between the RQ2 sample and the RQ4 test split—are fixable within the scope of the paper and do not require rejecting the work. I would ask the authors to address them head-on, including a quantitative sensitivity analysis, before the paper can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful thing here is the benchmark and metrics. They built the first large Python SATD repayment dataset (58,722 items) and a Java dataset roughly 20x larger than Mastropaolo et al.'s, using a language-independent tracker and a ten-step filter. The three diff-based metrics — BLEU-diff, CrystalBLEU-diff, and LEMOD — are simple extensions of existing ideas but they are well-motivated and the correlation analysis (diff metrics 0.65–0.84 with EM; whole-code BLEU and CrystalBLEU 0.01–0.08) makes a convincing case that whole-code similarity is the wrong tool for this task. All code and data are public, fine.\n\nThe headline result – Gemma-2-9B reaching 10.1% EM on Python and 8.1% on Java versus the previous 2.3% – should not be taken at face value. The stress-test note is correct: step 10 of the filter only judges whether the whole method update is related to the SATD, not whether the diff is clean at line level. The paper's own pre-filter manual check found 55/100 Python and 38/100 Java samples were unclear or unrelated, and even true repayments often come with unrelated contemporaneous changes (Zampetti et al.). Since every metric is computed against the full method diff, any unrelated inserted or deleted line must be reproduced exactly for EM=1 and contributes to BLEU-diff, CrystalBLEU-diff, and LEMOD. So the absolute scores are probably underestimates if the model correctly addresses the SATD but misses unrelated edits, and the relative ranking between fine-tuned and prompt-based models could shift if contamination is systematic. The authors acknowledge in Section 7 that 'some items with incorrect ground truth remain' but do not quantify line-level contamination or show the results survive it.\n\nTwo smaller issues. First, RQ2 selects each model's best prompt on 1,000 random samples per dataset; it isn't stated whether those samples are excluded from the RQ4 test set. If there is overlap, prompt selection can overfit and inflate RQ4 numbers. Second, there are no statistical tests; differences like 10.1% versus 9.7% EM may be within noise. Both are addressable.\n\nWho is this for? Anyone working on SATD or code-repair benchmarks. The datasets and metrics are the contribution; the specific model rankings are plausible but not definitive. With a robustness analysis on reference purity and overlap, this could be a solid TOSEM paper.\n\nI'd send it to peer review, with the request for those checks.","headline":"New large SATD repayment benchmarks and diff-based metrics are worth refereeing, but the headline EM numbers may be contaminated by unrelated edits in reference diffs.","tokens_in":29707,"tokens_out":3030,"would_cite":true,"duration_ms":30618,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that automated SATD repayment is more feasible than prior work indicated, with prompt-based large language models correctly addressing 10.1% of Python and 8.1% of Java self-admitted technical debts exactly, roughly four…","keywords":["self-admitted technical debt","SATD repayment","large language models","code generation","evaluation metrics","BLEU-diff","LEMOD","benchmark dataset"],"falsifier":"Manually inspect a random sample of the reference diffs (e.g., 300 items) and count the fraction containing edits unrelated to the SATD comment; if that fraction is high, BLEU-diff, CrystalBLEU-diff, and LEMOD would be rewarding or punishing models for reproducing irrelevant changes, and the reported rankings would need recomputation on purity-filtered diffs.","tokens_in":28592,"feed_emoji":"🛠️","tokens_out":5148,"duration_ms":43267,"temperature":0.7,"pith_summary":"This paper sets out to determine whether large language models can repay self-admitted technical debt (SATD) — the suboptimal code developers flag in comments like \"TODO: optimize\" — and how to measure that capability fairly. The authors build two large, filtered datasets of real SATD repayments (58,722 Python and 97,347 Java samples), argue that existing whole-code similarity metrics mislead, and propose evaluating only the diff between the method before and after repayment. Using BLEU-diff, CrystalBLEU-diff, and a new line-level metric (LEMOD), they find that prompt-based models like Gemma-2-9B address 10.1% of Python and 8.1% of Java SATDs exactly, far above the 2.3% previously reported. The work suggests automated SATD repayment is more achievable than earlier benchmarks implied, and that fine-tuned small models match large models on exact matches but produce worse code changes.","feed_headline":"LLMs repay 'TODO' code debt at 10% — up from 2.3%","feed_subtitle":"New diff-based metrics and larger datasets show prompt-based models beat fine-tuned ones on code quality.","key_machinery":"The load-bearing mechanism is diff-based evaluation: instead of scoring similarity between the whole generated method and the whole ground-truth method, the paper computes the diff between the input method (containing the SATD) and each output, then scores those diffs. This yields BLEU-diff and CrystalBLEU-diff, and the new LEMOD metric, which computes precision, recall, and F1 over the set of changed lines in the reference diff versus the candidate diff. The other key object is the 10-step filtering pipeline, whose final step uses a zero-shot chain-of-thought prompt to Llama-3-70B to judge whether a method update actually repays the SATD, which removes roughly 60% of candidate samples as noise.","core_discovery":"On its own terms, the paper claims that automated SATD repayment is a tractable task for current LLMs once the data is clean and the evaluation looks at code changes rather than whole methods. The central discovery is twofold: first, a 10-step filtering pipeline built on the language-independent SATD Tracker and an LLM-as-judge relevance check removes noise that made prior datasets pessimistic, yielding datasets roughly twenty times larger than the only previous benchmark; second, evaluating on diffs changes the picture of model quality, since whole-code BLEU and CrystalBLEU correlate near zero with exact match while the diff-based metrics correlate 0.65–0.84. On these new benchmarks a simple prompt to Gemma-2-9B repays 10.1% of Python and 8.1% of Java SATDs exactly, versus 2.3% for the best earlier fine-tuned model, and larger prompt-based models lead on diff-based quality metrics while fine-tuned small models only match large ones on exact match and lose on the diff-based metrics.","pith_inferences":["The diff-purity assumption could be tested directly: manually check a random sample of reference diffs for edits unrelated to the SATD, and re-evaluate models on a subset restricted to pure diffs to see how much the reported scores change.","The LEMOD metric, being interpretable, could transfer to other code-editing tasks, such as program repair or refactoring, where the edit diff matters more than the whole file.","The paper's easy-versus-hard split suggests a practical triage: models reliably handle SATDs requiring one or two inserted lines, so tools could route only those to automation and leave larger refactors to humans.","The oracle-template result implies that learning a prompt router — a small classifier that picks the prompt per SATD — could push exact match well beyond any single prompt, since 15–20% of items are addressed by at least one model-prompt combination in the 1,000-sample study."],"forward_implications":["If the central claim holds, automated SATD repayment at the method level is within reach of current prompt-based LLMs, with exact-match rates around 10% and much higher line-level overlap on easy cases.","Whole-code BLEU and CrystalBLEU should not be used to rank SATD repayment models, since they do not correlate with exact correctness; diff-based metrics or LEMOD should be the standard.","Fine-tuned small models (220M–770M parameters) can match large models on exact match but lag on change quality, so the choice of approach depends on whether exact replication or line-level fidelity matters more.","A cleaner dataset directly improves both prompt-based and fine-tuned performance, as shown by the RQ3 comparison on the filtered version of the prior Java-only dataset.","An oracle that picks the best prompt per SATD item raises exact match to 11–14%, suggesting prompt selection is a major lever for further gains."],"supporting_citations":[{"why":"The prior Java-only benchmark and its 2.3% exact-match result; the paper's datasets and comparisons extend and challenge this baseline.","marker":"[23]"},{"why":"SATDBailiff, the Java-parser-based tool whose 12% repository success rate motivates switching to SATD Tracker.","marker":"[1]"},{"why":"SATD Tracker, the language-independent tool that enables large-scale SATD extraction and the larger datasets.","marker":"[33]"},{"why":"CrystalBLEU, the code similarity metric whose whole-code application is shown to be misleading and which is adapted to diffs.","marker":"[12]"},{"why":"Zampetti et al.'s finding that 33–63% of method-level SATD removals involve code changes, used to justify the method-update filter.","marker":"[39]"},{"why":"SEART, the repository metadata tool used to select GitHub projects with the same criteria as the prior study.","marker":"[11]"}],"fun_headline_variants":["LLMs repay 10% of TODO debt exactly; diff metrics favor prompts","Diff-based metrics expose prompt-based LLM edge in code debt","Fine-tuned LLMs lose to prompts on diff-based SATD quality","New benchmarks and diff metrics show LLMs hit 10% exact repayment","Prompt engineering beats fine-tuning for LLM repayment of code debt"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole evaluation assumes that the diff between the method before and after the commit contains exactly the changes that repay the SATD, with no unrelated edits; the LLM filter only checks whether the update is related to the SATD comment, not that every changed line is.","fun_headline_variants_meta":{"raw":{"variants":["LLMs repay 10% of TODO debt exactly; diff metrics favor prompts","Diff-based metrics expose prompt-based LLM edge in code debt","Fine-tuned LLMs lose to prompts on diff-based SATD quality","New benchmarks and diff metrics show LLMs hit 10% exact repayment","Prompt engineering beats fine-tuning for LLM repayment of code debt"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000239,"raw_usage":{"total_tokens":1613,"prompt_tokens":1142,"completion_tokens":471,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":758,"completion_tokens_details":{"reasoning_tokens":378}},"tokens_in":758,"tokens_out":471,"duration_ms":5208,"temperature":1.0,"reasoning_tokens":378,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:33:41.742598+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually inspect a random sample of the reference diffs (e.g., 300 items) and count the fraction containing edits unrelated to the SATD comment; if that fraction is high, BLEU-diff, CrystalBLEU-diff, and LEMOD would be rewarding or punishing models for reproducing irrelevant changes, and the reported rankings would need recomputation on purity-filtered diffs.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SATD Tracker, the language-independent tool that enables large-scale SATD extraction and the larger datasets."}],"review_version":1}