{"id":"627634a6-1699-463b-ba73-f29f9fc1082e","arxiv_id":"2507.19390","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ReCatcher systematically measures regressions in LLM code generation across correctness, static quality, and performance, and its evaluation shows fine-tuning, merging, and new releases each introduce specific regressions.","lead":"This paper introduces ReCatcher, a tool that compares an old and a new AI coding model on correctness, code quality, and speed, flagging where the new model is worse. The authors test it on three common upgrade paths and find that upgrades often cause syntax errors, missing imports, and slower code even when overall accuracy is similar.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Execution-time regression is measured only on unit-test-sized inputs (Section 4.2.2) and reported as a task fraction, so GPT-4o-mini's 80.49% 'degradation' may not hold on realistic workloads.","rationale":"ReCatcher's central contribution is a regression testing framework, and its headline empirical value includes the execution-time regression claim for GPT-4o-mini. For that claim to support the abstract's '80% performance degradation' wording, the measurement must reflect something more than microbenchmark behavior on unit-test inputs. The reader's weakest assumption identifies exactly this gap, and Section 4.2.2 confirms large inputs were unavailable. I also note that the paper reports only a task fraction with a significant Mann-Whitney U result, not an effect size, which makes the 80.49% number easy to over-read. The paper is otherwise transparent: it discloses the large-input limitation, the correct-code-only filtering, and the interdependency between syntax errors and logical correctness. These are addressable experimental limitations, not fatal flaws, so the conditional verdict remains appropriate. If the large-input rerun reproduces the regression, the headline is strengthened; if not, the abstract and Section 5.5.3 should be revised to scope the finding to unit-test-scale inputs. I therefore leave the reader's verdict unchanged.","tokens_in":23250,"tokens_out":6404,"duration_ms":61754,"concrete_test":"Re-run the GPT-4o vs GPT-4o-mini comparison on a large-input variant of HumanEval+: for each task, add test inputs that scale parameter sizes by 10x to 1000x (e.g., for sorting tasks use random lists of length 10^4 to 10^6 instead of the few elements in the unit tests), keep ReCatcher's correct-code filtering and n=5 repetitions, and recompute the task-level regression ratio and report effect sizes such as median slowdown for the regressed tasks. If the 80.49% ratio drops substantially or reverses on large inputs, the headline claim is input-size dependent and should be reframed as 'regression on unit-test-sized inputs.' As a secondary check, report distribution overlap or effect sizes for the current unit-test-input runs to distinguish statistical significance from practical magnitude.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is in the performance dimension, one of the paper's three headline claims. Section 4.2.2 states that 'large inputs were not available' and performance testing therefore used the inputs from the benchmark unit tests; for HumanEval+ those inputs are tiny and constructed for correctness checking, not for exposing algorithmic complexity. Table 1's 80.49% is the fraction of tasks where the Mann-Whitney U test found a significant execution-time difference, not a measured 80% slowdown: effect sizes and confidence intervals are not reported. On such small inputs, timing can be dominated by interpreter overhead, allocation, or noise, and an algorithmically worse solution can appear faster. The comparison is also restricted to tasks where both models produced unit-test-passing code, so the regressed subset is selected rather than representative. If execution-time ordering changes with input scale, the abstract's claim that GPT-4o-mini 'suffers up to 80% performance degradation' overstates the finding, and the practical recommendation to prefer GPT-4o for algorithmic tasks is not established. The paper acknowledges large-input support but did not use it, so this is a testable gap rather than an internal inconsistency.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ReCatcher, a regression testing framework that systematically compares two LLMs for Python code generation across three dimensions: logical correctness, static code quality, and execution performance. The framework uses unit tests, Pylint, PMD-CPD, and profiling/tracemalloc, and generates a regression report based on repeated generation and statistical testing. It is applied to three model-update scenarios—fine-tuning, merging, and model release—using CodeLlama, DeepSeek-Coder, and GPT-family models on HumanEval+ and BigCodeBench. Headline findings include syntax-error regressions up to 12.93% after cross-language fine-tuning, logical-correctness regression up to 18.72% after merging CodeLlama with Llama2, missing-import regressions up to 51.64% for GPT-4o relative to GPT-3.5-turbo, and an 80.49% execution-time regression for GPT-4o-mini on HumanEval+. The paper also compares ReCatcher with LLM-as-judge baselines and standard metrics for a small subset of models.","tokens_in":23509,"tokens_out":6297,"duration_ms":55781,"significance":"If the headline results are correct, ReCatcher is a practical quality gate for deciding whether to adopt a new LLM for code generation, and the findings usefully demonstrate that more capable or cost-efficient models are not universally better in correctness, quality, or performance. The paper has concrete strengths: the framework is open-sourced, a dataset of generated snippets is released, the correctness oracles are external (unit tests, Pylint, CPD, profiler) rather than fitted to the conclusions, and the performance comparison uses a non-parametric test. The claim of being the first LLM regression testing framework for code generation is plausible relative to RETAIN and RACE, although RACE is closely related. However, the execution-time headline is currently not adequately supported, and the statistical reporting of the main regression rates is too thin to distinguish real effects from noise.","major_comments":[{"comment":"The 80.49% execution-time regression for GPT-4o-mini on HumanEval+ is the fraction of tasks where a Mann-Whitney U test found a significant difference, not a measured 80% slowdown; no effect sizes or confidence intervals are reported for any performance entry. Performance testing used only unit-test inputs because 'large inputs were not available' (Section 4.2.2), and the comparison is restricted to tasks where both models produced unit-test-passing code. On the tiny, correctness-oriented inputs of HumanEval+, timing can be dominated by interpreter overhead or noise, so an algorithmically worse solution can appear faster. The paper's own large-input mechanism was not exercised, leaving the practical recommendation to prefer GPT-4o for algorithmic tasks unsubstantiated. Please report effect sizes (e.g., common-language effect size or median ratios), state how many tasks were excluded per scenario, and either use large inputs or limit the performance claims to unit-test-scale inputs.","section":"Section 4.2.2 and Section 5.5.3, Table 1"},{"comment":"All correctness and static-code regression rates are reported as point estimates without confidence intervals, significance tests, or effect sizes. Many entries are at the level of 0.61% to 3%, which corresponds to 1 to 5 tasks out of 164 or 1,140; with only 10 generations per task, sampling variability is substantial. As a result, the paper cannot distinguish genuine regressions from noise, and conclusions such as 'readability and maintainability were relatively stable' rest on very small differences. Please provide task-level rates with binomial confidence intervals or otherwise quantify the uncertainty of the reported percentages.","section":"Table 1 and Sections 5.5.1-5.5.3"},{"comment":"The abstract's claim that ReCatcher achieves 'better and consistent accuracy' against baseline solutions is not supported by the design of the comparison. For logical correctness and performance, ReCatcher's accuracy is 1.0 by construction because unit tests and the Mann-Whitney U test define the ground truth; the experiment measures only the LLM-as-judge baseline's accuracy against that ground truth. No comparison is made to RACE, RETAIN, or any other regression-testing framework, and the 'static code issues' baseline uses only cyclomatic complexity and maintainability index, which measure different properties. Please restate the comparison as an evaluation of LLM-as-judge and two standard metrics, or conduct a head-to-head comparison with an alternative framework before claiming superior accuracy.","section":"Section 5.6 and Table 2"}],"minor_comments":[{"comment":"The text states that the CodeLlama merged variant 'exhibited a regression of 30.77% in Syntax Error on BigCodeBench and 18.72% on HumanEval+', but Table 1 reports Syntax Error as -9.02% on HumanEval+; the -18.72% value in the table is the General Logic regression. Please align the narrative with the table.","section":"Section 5.5.2"},{"comment":"The repetition variables are used inconsistently: Section 4.2.1 defines generation repetitions as m and Section 5.3.2 labels them n, while performance repetitions are labeled m in Section 5.3.2. Please use consistent notation.","section":"Sections 4.2.1 and 5.3.2"},{"comment":"The table header contains typos: 'Fintuned' should be 'Fine-tuned' and 'Errros' should be 'Errors'.","section":"Table 1"},{"comment":"The sentence 'We observed overall improvements across all code aspects except for a minor minor exception exception regression regression of 1.28%' contains duplicated words; please clean up the prose.","section":"Section 5.5.2"},{"comment":"The phrase 'suffers up to 80% performance degradation' is imprecise; the reported 80.49% is the percentage of tasks with a statistically significant execution-time difference, not the magnitude of the slowdown. Please phrase this as 'shows a significant execution-time regression on up to 80% of tasks' or report the actual magnitude.","section":"Abstract and Section 5.5.3"},{"comment":"The library name is misspelled as 'tarcemalloc' in the sentence about memory measurement; it should be 'tracemalloc'.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper's framework is a reasonable contribution and the artifacts are valuable, but the central execution-time claim needs substantial additional evidence and the statistical reporting needs to be brought up to the standard expected for a software engineering venue. The baseline comparison is currently presented in a way that overstates what was measured. I believe these issues are fixable within the manuscript's scope, hence major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: ReCatcher is worth a serious look. It integrates unit tests, Pylint, PMD-CPD, and profiling into a taxonomy-driven regression workflow for comparing two code-gen LLMs, and it actually ships code and a dataset. The empirical findings are mostly plausible and some are genuinely non-obvious, e.g., GPT-4o regressing by over 50% on missing imports/declarations versus GPT-3.5-turbo on BigCodeBench, and merging CodeLlama with Llama2 hurting correctness and syntax. That kind of concrete evidence is useful for anyone deciding whether to adopt a new model.\n\nThe framework itself is clean: generation repeated 10 times, performance runs repeated 5 times, and the use of Mann-Whitney U for performance distributions is a sensible non-parametric choice. The baseline comparison against LLM-as-judge is also a fair stress test, even though the results there are mixed.\n\nSoft spots, in rough order of importance. First, the performance claim is the weakest link. Section 4.2.2 admits large inputs were not available, so execution-time comparisons ran on unit-test inputs only. For HumanEval+, those are tiny and built for correctness, not for exposing algorithmic complexity. The abstract's \"up to 80% performance degradation\" is really the fraction of tasks where a significant difference was detected, not an 80% slowdown; effect sizes and confidence intervals are missing. That combination means the practical recommendation to prefer GPT-4o over GPT-4o-mini for algorithmic tasks is not yet established at realistic input scales. Second, the percentage differences in Table 1 are reported without confidence intervals or significance tests, and with 10 repetitions some of the smaller values are likely noise. Third, the baseline-validation section uses ReCatcher's own outputs as ground truth for comparing ReCatcher against LLM-judges, which is somewhat circular; the authors acknowledge this, and it is an honest limitation rather than a fatal one.\n\nWho gets value from this: practitioners evaluating model updates, and researchers working on LLM evaluation beyond pass@k. The paper deserves a serious referee. I would send it out, but with a request to put the performance findings on a more honest statistical footing and to state explicitly that the execution-time results are benchmark-input-scale results, not workload-scale results.\n\nRecommendation: accept for peer review with revisions expected. I'd bring it to our reading group to discuss both the framework and the performance measurement pitfall.","headline":"ReCatcher is a coherent and useful framework for LLM code-generation regression testing, with plausible headline findings, but the execution-time claims outrun the input scale and the significance statistics are thin in places.","tokens_in":24014,"tokens_out":1347,"would_cite":true,"duration_ms":15477,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ReCatcher claims to be the first regression-testing framework for LLM code generation, comparing models on logic, static quality, and execution speed and finding regressions of up to 80%.","keywords":["regression testing","LLM code generation","code quality","execution performance","fine-tuning","model merging","GPT-4o","static analysis"],"falsifier":"Run the same GPT-4o versus GPT-4o-mini comparison on a benchmark with large, real-world-shaped inputs or on a production workload and profile execution time; if the candidate is not slower on a substantial fraction of tasks, the headline execution-time regression claim would be confined to the benchmark setting. Similarly, repeat the fine-tuning scenario with a same-language dataset; if syntax errors do not rise, the cross-language mechanism fails to reproduce.","tokens_in":23080,"feed_emoji":"🐍","tokens_out":4581,"duration_ms":39684,"temperature":0.7,"pith_summary":"ReCatcher is proposed as the first regression-testing framework aimed specifically at LLM-generated Python code. Instead of asking whether a new model is more accurate, it asks whether swapping in a candidate model for the current one introduces regressions in logical correctness, static code quality, or execution performance. Applied to fine-tuned, merged, and newly released CodeLlama, DeepSeek-Coder, and GPT-family models, the framework reports that cross-language fine-tuning raises syntax errors by up to 12.93%, merging with a general-purpose model can reduce logical correctness by up to 18.72%, GPT-4o misses imports or declarations up to 51.64% more often than GPT-3.5-turbo on BigCodeBench, and GPT-4o-mini is slower than GPT-4o on 80.49% of HumanEval+ tasks. The point of the paper is that more capable or newer models are not automatically better on every axis, and that systematic regression testing should precede model adoption.","feed_headline":"New LLM versions can regress code speed by up to 80%","feed_subtitle":"A regression-testing framework finds syntax, correctness, and import errors that leaderboards miss.","key_machinery":"The load-bearing object is the three-part Test Suite: unit-test execution for logical correctness, Pylint (plus PMD-CPD for duplication and a custom rule for unnecessary conditional blocks) for static code issues, and repeated profiling of execution time and memory with the Mann-Whitney U test for performance. Each inefficiency category in the taxonomy of LLM-generated code inefficiencies is mapped to a detectable test or Pylint message, giving the framework a structured, reproducible way to turn code-quality concerns into pass or fail signals. The direct comparison of two models on the same tasks is what turns ordinary quality measurement into regression testing.","core_discovery":"ReCatcher's central claim is that regressions in LLM code generation can be detected systematically by comparing, per benchmark task, code generated by the deployed model and the candidate model across three dimensions. Logical correctness is judged by executing each snippet against the benchmark's unit tests; static code issues are detected by Pylint and PMD-CPD using a taxonomy of LLM-code inefficiencies; performance is profiled by repeated execution and compared with the Mann-Whitney U test to determine whether the candidate's code is statistically slower or faster. The framework emits a regression report with an inefficiency-difference percentage for logic and static issues and a task ratio of improvements or regressions for time and memory. Using it across fine-tuning, merging, and model-release scenarios, the paper reports the specific regressions listed above and concludes that logical correctness, syntax and import errors, and execution time are the most regression-prone aspects.","pith_inferences":["If the regression patterns generalize, error-handling signals such as syntax errors and missing imports act as an early canary: they shift with fine-tuning and merging, so cheap static checks alone could serve as a low-cost gate before full unit-test and profiling runs.","The framework's reliance on unit-test inputs for performance means the 80.49% slowdown could be a small-input artifact; extending the same comparison to large, workload-shaped inputs is the natural next test, and ReCatcher's architecture already allows such inputs.","A direct extension would be to rank candidate models for a specific codebase by running ReCatcher on the team's own tasks, turning the framework from a post-hoc evaluator into a release gate in a CI pipeline.","The finding that an LLM judge agrees with test-based logic judgment only about 80% of the time and performs poorly on performance suggests a hybrid approach: test execution for logic, profiling for speed, and LLM judges only for aspects that resist automation."],"forward_implications":["Teams planning to fine-tune a code model on data from another language should expect syntax regression and budget for syntax checks before deployment.","Merging a code model with a general-purpose model can cost up to roughly 18% in logical correctness and roughly 20-30% in syntax reliability, so merged models should be regression-tested rather than assumed beneficial.","Model releases within one family can trade correctness or import handling for speed or vice versa; GPT-4o-mini's 80.49% execution-time regression on HumanEval+ means cheaper models should be profiled on the intended workload.","Memory usage stayed relatively stable across most updates, while execution time, syntax errors, and missing imports were the volatile axes, so regression suites should prioritize these three.","More capable models are not universally better, so adopting a new model release or merged variant on the strength of leaderboard accuracy alone risks silent quality or performance losses."],"supporting_citations":[{"why":"Supplies the taxonomy of inefficiencies in LLM-generated code that ReCatcher maps to tests.","marker":"[2]"},{"why":"Provides the HumanEval+ tasks and unit tests used to judge logical correctness and to drive performance profiling.","marker":"[37]"},{"why":"Provides the BigCodeBench real-world function-call tasks and tests used as the second benchmark.","marker":"[79]"},{"why":"Pylint is the static-analysis engine that detects most static code issues through its messages.","marker":"[53]"},{"why":"PMD-CPD detects code and comment duplication using token-based structural comparison.","marker":"[52]"},{"why":"Defines RETAIN, the general LLM regression-testing tool whose text-only focus ReCatcher positions itself against.","marker":"[14]"},{"why":"Describes TIES merging, the technique used to build the merged CodeLlama and DeepSeek-Coder variants.","marker":"[71]"},{"why":"Defines Llama2, the general-purpose model whose merging with CodeLlama produced the correctness and syntax regressions.","marker":"[66]"},{"why":"Describes OpenCodeInterpreter, whose LeetCode-style training explains the merged DeepSeek-Coder's improvements.","marker":"[77]"}],"fun_headline_variants":["ReCatcher: catching LLM code regressions before they ship","LLM update? Check for 80% slower code first","ReCatcher finds LLM code regressions in syntax, correctness, speed","Up to 80% slower code from LLM updates? ReCatcher catches it","Before switching LLMs: test code for regressions - ReCatcher does it"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The regressions, especially execution-time regressions, are measured only on tasks and small inputs drawn from two benchmarks because large inputs were not available for performance testing; if those inputs do not resemble real workloads, the performance findings may not transfer to practice.","fun_headline_variants_meta":{"raw":{"variants":["ReCatcher: catching LLM code regressions before they ship","LLM update? Check for 80% slower code first","ReCatcher finds LLM code regressions in syntax, correctness, speed","Up to 80% slower code from LLM updates? ReCatcher catches it","Before switching LLMs: test code for regressions - ReCatcher does it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00063,"raw_usage":{"total_tokens":2935,"prompt_tokens":995,"completion_tokens":1940,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":1841}},"tokens_in":611,"tokens_out":1940,"duration_ms":13678,"temperature":1.0,"reasoning_tokens":1841,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:53:20.686681+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same GPT-4o versus GPT-4o-mini comparison on a benchmark with large, real-world-shaped inputs or on a production workload and profile execution time; if the candidate is not slower on a substantial fraction of tasks, the headline execution-time regression claim would be confined to the benchmark setting. Similarly, repeat the fine-tuning scenario with a same-language dataset; if syntax errors do not rise, the cross-language mechanism fails to reproduce.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Pylint is the static-analysis engine that detects most static code issues through its messages."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PMD-CPD detects code and comment duplication using token-based structural comparison."},{"cited_title":"RETAIN: Interactive Tool for Regression Testing Guided LLM Migration","cited_arxiv_id":"2409.03928","evidence_quote":"Defines RETAIN, the general LLM regression-testing tool whose text-only focus ReCatcher positions itself against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes TIES merging, the technique used to build the merged CodeLlama and DeepSeek-Coder variants."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes OpenCodeInterpreter, whose LeetCode-style training explains the merged DeepSeek-Coder's improvements."}],"review_version":2}