{"id":"a5095e2e-c919-4669-ba43-804fc5278ace","arxiv_id":"2509.06429","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"GPT-4's bug fixes on QuixBugs vary across repeated runs, and increasing temperature from 0 to 1 lowers both syntactic similarity and the rate of passing tests.","lead":"This paper asks whether ChatGPT produces the same fix every time it is asked to repair the same bug, and whether the temperature setting changes that. It reports that fixes vary across runs and that higher temperatures reduce both syntax similarity and the chance of passing tests, though the study uses only three runs per setting.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"n=3 per condition makes the reported temperature trend statistically unsupported; the central instability claim rests on noise-level differences.","rationale":"The reader identified the same weakest assumption: n=3 per condition is insufficient to estimate variance and success rates. This is the single most load-bearing concern because the paper's headline and proposed method both depend on a real temperature-driven increase in instability and functional failure. If the observed differences are within sampling noise, the empirical foundation collapses, regardless of the other internal inconsistencies (e.g., OER definition vs computation, inconsistent temperature counts). The concern can be settled directly from the reported tables via resampling tests, or by replicating with more generations. Since the reader already issued a CONDITIONAL verdict partly for this reason, my stress test does not change the verdict; it reinforces it. No stronger objection is needed.","tokens_in":17128,"tokens_out":4390,"duration_ms":50894,"concrete_test":"Using the paper's own Tables 5 and 7, compute the 20 per-problem success rates at T=0 and T=1, then run a paired bootstrap (10,000 resamples) for the mean difference. If the 95% confidence interval includes 0, the functional-failure trend is not statistically supported. Independently, run a permutation test on the per-problem Levenshtein variances from Tables 2–4 for the null that variance does not increase with temperature. If both are non-significant, the claim fails on current evidence. Ideally, also rerun the experiment with ≥30 generations per (problem, temperature) to see whether the trend emerges with adequate power.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that outputs become 'much more unstable and variable' as temperature rises, with functional failure increasing (OER 0.70→0.62)—is quantified from only 3 repeated generations per problem per temperature (Tables 5–7). For binary success/failure, the per-problem standard deviation is at its ceiling (0.58) for any problem with a 1/3 or 2/3 success rate, and aggregate success is 42/60 vs 37/60 at T=0 vs T=1. Treating the 60 binary trials as independent, the standard error of this difference is about 0.086, making the observed drop of 0.083 not significant (z≈0.97). A paired analysis over 20 problems is unlikely to rescue significance given the 0, ±1/3 granularity. No confidence intervals or statistical tests are reported anywhere. The Levenshtein variance tables (2–4) are also built from 3 similarity values per problem; one divergent output can create variance ~0.1, while many problems show exactly zero variance at all temperatures (e.g., flatten, kth, to_base, depth_first_search), contradicting any across-the-board instability claim. Thus the size of the effect, which motivates the paper's variance-aware validation protocol, is not distinguishable from sampling noise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical study of GPT-4's consistency in repairing 20 Python bugs selected from QuixBugs. For each problem, three fixes are generated at each of three temperatures (0, 0.5, 1.0). The authors compute normalized Levenshtein similarity between generated fixes and an Output Equivalence Rate (OER) intended to capture functional consistency. They report that mean OER declines from 0.70 to 0.62 as temperature increases, that syntactic similarity becomes more dispersed, and that temperature 0 does not guarantee determinism. They conclude by recommending variance-aware, multi-sample validation protocols before LLM-generated patches are deployed.","tokens_in":17467,"tokens_out":6561,"duration_ms":70229,"significance":"The topic is timely and the study design has good intentions: it uses a public program-repair benchmark, varies temperature systematically, and combines structural and functional measures rather than reporting best-of-N accuracy. If the empirical results were statistically solid, the paper would provide a useful cautionary result for LLM-based repair tooling. However, the current evidence is not solid: the OER metric is not computed as defined, the sample size per condition (n=3) cannot support the variance and trend claims, and several tables contain internal numerical contradictions. The paper also fails to provide replication data, exact model version, or inferential statistics. With revision, the central idea could become a contribution; in its current form it is not established.","major_comments":[{"comment":"The OER is defined as the proportion of test inputs for which two programs produce identical outputs, but the experimental tables report only binary pass/fail per generated fix against the reference test suite (successes/failures out of 3 trials). These are different quantities. As reported, 'OER=0.70' is actually the mean per-problem success rate, not an output-equivalence rate between two programs. This mismatch is load-bearing because the paper's central functional-instability measure is built on it. Please either redefine the metric as a pass@k-style success rate or compute true pairwise OER among the three generations per (problem, temperature).","section":"§3.3, Eq. (3); §4.2 Tables 5-7"},{"comment":"Several descriptive statistics are internally inconsistent. For example, Table 2 'breadth_first_search' reports Average=0.40, Maximum=0.40, Minimum=0.39, yet Variance=0 and StdDev=0; 'bucketsort' reports Variance=0 but StdDev=0.02; 'rpn_eval' reports Variance=0 but StdDev=0.02. Similar issues occur in Tables 3 and 4. Because these tables support the syntactic-instability claim, they must be recomputed and verified. Provide raw pairwise similarity values or code/scripts so readers can check the computations.","section":"§4.2 Tables 2-4"},{"comment":"With n=3 per (problem, temperature), the aggregate OER difference between T=0 (42/60) and T=1 (37/60) has a standard error of approximately 0.086; the observed drop is 0.083 (z≈0.97), so the headline 'OER decreases with temperature' is not distinguishable from sampling noise. For individual problems, any 1/3 or 2/3 success count yields a sample standard deviation of 0.58—the maximum possible—so the reported variance bars in Figure 5 are largely artifacts of the tiny sample. The manuscript reports no confidence intervals, hypothesis tests, or effect-size measures. The claims 'much more unstable and variable' and 'high temperatures showing especially high rates of functional failure' require either substantially more generations per cell or appropriate statistical modeling.","section":"§4.2 Tables 5-7, Figure 5"},{"comment":"The experimental design is described inconsistently. §3.1 states two temperatures (1.0 and 0.0), while the abstract and §4.2 use three (0, 0.5, 1.0). The abstract says nine outputs per problem (20×9=180 total), while the conclusion says 'the model was run nine times for each problem at three different temperature values' and reports 540 outputs (20×9×3=540). This discrepancy affects the interpretation of every table. Specify the exact number of generations per (problem, temperature, run) and correct the totals.","section":"§3.1; §4.2; Abstract; Conclusion"}],"minor_comments":[{"comment":"The sentence 'The major contributions of this study are summarized as follows:' is duplicated immediately after 'The main contributions of this study are summarized as follows:'.","section":"Introduction"},{"comment":"QuixBugs is cited twice as [7] and [40]; please merge or distinguish them if they refer to different versions.","section":"References"},{"comment":"There are typos in figure/table labels: Figure 2 caption 'Tempreature', Table 7 header 'genel', and the text near Figure 4 refers to 'breadth basis search'.","section":"Figures/Tables"},{"comment":"It is not stated whether Levenshtein similarity is computed pairwise among the three generated fixes or between each fix and a golden reference fix. This must be clarified because it changes the meaning of the variance and low-similarity ratios.","section":"§3.1"},{"comment":"The threshold for the 'Low Similarity Ratio (<0.7)' column is arbitrary and not justified. Consider reporting the full distribution or a sensitivity analysis around this threshold.","section":"§4.2 Tables 2-4"},{"comment":"The paper does not state the exact GPT-4 model version, API access date, or decoding parameters beyond temperature. These details are necessary for replication given known model-version drift.","section":"Reproducibility"},{"comment":"The limitations paragraph mentions AST-based metrics and model-version drift, but it does not acknowledge the small number of repetitions (n=3) as a limitation. This should be stated explicitly.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The paper's core question is relevant and the authors chose a public benchmark with executable tests, which is a strength. The main problems—OER definition mismatch, small-sample inference, and internally inconsistent tables—are fixable through additional experiments and careful reporting. I would not reject on the basis of possible training-data leakage alone, but the authors should address how benchmark familiarity might inflate per-problem success rates. Given the load-bearing statistical and definitional issues, major revision is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know up front: this paper asks a genuinely useful question — how unstable is GPT-4 when asked to fix the same bug repeatedly — and it applies the existing non-determinism analysis to the less-studied bug-repair setting. That is a legitimate extension, and the practical takeaway (don't trust a single patch; sample, test, and select) is correct and worth saying.\n\nWhat the paper does well: it explicitly varies temperature, which most bug-fixing studies ignore, and it reports both structural (Levenshtein) and functional (pass/fail) consistency. The per-problem breakdown in Tables 5–7 honestly shows that some problems are temperature-insensitive while others are brittle; that heterogeneity is a real observation. Using a public benchmark with executable test oracles is the right way to ground this work, and the authors deserve credit for trying multi-trial estimation instead of best-of-N.\n\nNow the soft spots, in order of severity.\n\nFirst, the statistics are not there. Three generations per (problem, temperature) is simply too few to estimate variance. The stress-test note gets it exactly right: for binary pass/fail, the per-problem standard deviation is at its ceiling (0.58) for any 1/3 or 2/3 split, and the aggregate OER drop from 0.70 to 0.62 is not statistically distinguishable from noise (z ≈ 0.97 on 60 trials). No confidence intervals, no paired tests, no effect sizes. The trend is plausible and consistent with prior work, but this paper does not demonstrate it with the data it reports.\n\nSecond, there are internal contradictions that are hard to wave away. Section 3.1 says two temperatures (0 and 1.0); the abstract and Section 4.2 say three (0, 0.5, 1). The abstract and conclusion say 20 problems × 9 outputs = 180 total outputs; the conclusion also says 540. More seriously, Eq. 3 defines OER as pairwise output equivalence between two programs, but the tables compute it as the success rate of a generated fix against the test oracle. These are not the same thing. A charitable reader can reconstruct the intent, but the paper as written contradicts its own central metric.\n\nThird, minor framing issue: the Levenshtein tables show zero variance for several problems even at T=1 (flatten, kth, to_base), which undercuts the blanket claim of across-the-board instability. The authors do acknowledge per-problem heterogeneity, so this is a nuance problem, not a fatal one.\n\nWho is this for? Practitioners building CI gates around LLM-generated patches and researchers working on APR evaluation methodology. It deserves a serious referee, but it needs a revision that fixes the numbers, clarifies what OER actually means, and either increases N or at least reports proper statistical tests. As-is, the claims outrun the data.\n\nRecommendation: send to peer review, but expect major revision.","headline":"Right question, small sample, and a few numbers that don't add up — fixable, but not ready as-is.","tokens_in":17889,"tokens_out":3844,"would_cite":false,"duration_ms":43122,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that ChatGPT's bug-fix outputs become both structurally and functionally less stable as temperature rises, and that setting temperature to zero does not remove nondeterminism.","keywords":["LLM instability","bug fixing","automated program repair","temperature parameter","Levenshtein similarity","output equivalence rate","nondeterminism","QuixBugs"],"falsifier":"Run the same 20 QuixBugs problems with many more generations per temperature, for example 30 or 100, logging API version and sampling flags; if the per-problem categories and the 0.70-to-0.62 OER trend remain stable across repeated replications, the central claim holds, and if they swing, the instability estimates themselves are unstable. Separately, issue repeated identical requests at temperature 0 and check whether any two outputs differ; if they never differ in a controlled setting, the paper's residual-nondeterminism claim would need revision.","tokens_in":17081,"feed_emoji":"🐛","tokens_out":8228,"duration_ms":86301,"temperature":0.7,"pith_summary":"This paper sets out to measure whether a large language model gives stable, functionally correct fixes when asked repeatedly to repair the same buggy code. Using 20 buggy Python algorithm problems from the QuixBugs benchmark and the ChatGPT (GPT-4) model, the authors compare nine outputs per problem across temperatures 0, 0.5, and 1, measuring structural similarity with normalized Levenshtein distance and functional equivalence with a test-based Output Equivalence Rate. They find that output variability grows as temperature rises: average syntactic similarity between repeated fixes falls, and functional success drops from 70% at temperature 0 to 62% at temperature 1. They also report that temperature 0 does not make the model deterministic, because repeated runs still differ and some fixes still fail tests. If this holds, automated bug-fixing tools need variance-aware validation—multiple samples, test-based selection, and output-equivalence thresholds—rather than single-run acceptance.","feed_headline":"Raising temperature makes LLM bug fixes diverge and fail more","feed_subtitle":"Functional success falls from 70% to 62%, and even temperature 0 leaves fixes nondeterministic","key_machinery":"The measuring pair is normalized Levenshtein similarity, which scores character-level syntactic distance between two generated fixes, and the Output Equivalence Rate (OER), which scores how often two programs produce identical outputs on a shared test set. These are applied over a repeated-generation design: 20 QuixBugs problems, three temperatures (0, 0.5, 1), and three generations per temperature, giving nine fixes per problem and 540 outputs total. The combination is what lets the paper separate 'looks different' from 'behaves differently' for the same bug.","core_discovery":"The paper's central claim is that the same buggy code, given to the same model with the same prompt, produces a distribution of fixes rather than one fix, and that this distribution widens as the sampling temperature rises. Across 20 buggy Python algorithms from QuixBugs, three generations at each of temperatures 0, 0.5, and 1 show average Levenshtein similarity between repeated fixes falling and functional agreement with test oracles (OER) dropping from 0.70 at temperature 0 to 0.62 at temperature 1. A second claim is that temperature 0 does not restore determinism: repeated runs at temperature 0 still differ syntactically and some still fail tests, so residual randomness is attributed to s","pith_inferences":["The paper does not test whether sampling more fixes at higher temperature and then selecting by test results would recover the lost reliability; that is a natural next experiment implied by its protocol.","With only three runs per condition, the exact stability rankings of individual problems are fragile; a larger replication would be needed to know which problems truly sit on the stable or unstable side.","The protocol could be applied to other models and decoding settings beyond temperature; if the same pattern holds, variance reporting would become a standard part of LLM-repair evaluation.","One could also test whether API version changes or non-temperature sampling flags account for the residual nondeterminism seen at temperature 0."],"forward_implications":["Single-shot fixes from an LLM should not be treated as dependable; a CI/CD admission gate should require multiple generated fixes and test-based verification before accepting a patch.","Lowering temperature improves stability but does not guarantee determinism, so production pipelines that assume deterministic output at temperature 0 are unsound.","A combined view of structural similarity and functional equivalence is a workable way to quantify how variable an LLM repair pipeline is.","Automation of bug fixing is safer for problems with standard, template-like solutions, while graph and state-heavy tasks need stronger guardrails or human review.","Multi-sample ensembling with test-aware selection and clustering of similar patches can reduce variance-induced failures in LLM-based repair."],"supporting_citations":[{"why":"Surveys LLM-based code generation and frames bug fixing as a growing task where stability has not been thoroughly studied.","marker":"[1]"},{"why":"Establishes LLM-based automated program repair as the practical context the paper is testing.","marker":"[3]"},{"why":"Supplies the earlier finding that nondeterminism in ChatGPT code generation persists even at low temperature, which this paper extends to bug fixing.","marker":"[4]"},{"why":"Introduces the QuixBugs benchmark of buggy programs with test oracles used for functional evaluation.","marker":"[7]"},{"why":"Provides a comprehensive program-repair study on QuixBugs that supports treating this benchmark as a standard testbed.","marker":"[8]"},{"why":"Reports bit-for-bit identical output rates for ChatGPT across code benchmarks, grounding the expectation of run-to-run divergence.","marker":"[9]"},{"why":"Defines Levenshtein distance, the basis of the syntactic similarity metric used throughout the experiments.","marker":"[34]"},{"why":"Is the QuixBugs dataset reference that supplies the 20 Python problems used in the experiments.","marker":"[40]"}],"fun_headline_variants":["Same bug, same prompt: LLM fixes diverge with rising temperature","Higher temperature makes LLM bug fixes diverge and fail more often","Even at temperature 0, ChatGPT bug fixes remain nondeterministic","LLM bug-fixing instability worsens as temperature rises","Temperature 0 doesn't make LLM bug fixes deterministic"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The study assumes that three generations per problem per temperature are enough to estimate a problem's success rate and output variance; with only three binary trials, one different run changes an entire problem's category, so the reported stability differences rest on that small-sample premise.","fun_headline_variants_meta":{"raw":{"variants":["Same bug, same prompt: LLM fixes diverge with rising temperature","Higher temperature makes LLM bug fixes diverge and fail more often","Even at temperature 0, ChatGPT bug fixes remain nondeterministic","LLM bug-fixing instability worsens as temperature rises","Temperature 0 doesn't make LLM bug fixes deterministic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001256,"raw_usage":{"total_tokens":5031,"prompt_tokens":840,"completion_tokens":4191,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":4102}},"tokens_in":584,"tokens_out":4191,"duration_ms":32544,"temperature":1.0,"reasoning_tokens":4102,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T23:36:26.087806+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 20 QuixBugs problems with many more generations per temperature, for example 30 or 100, logging API version and sampling flags; if the per-problem categories and the 0.70-to-0.62 OER trend remain stable across repeated replications, the central claim holds, and if they swing, the instability estimates themselves are unstable. Separately, issue repeated identical requests at temperature 0 and check whether any two outputs differ; if they never differ in a controlled setting, the paper's residual-nondeterminism claim would need revision.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes LLM-based automated program repair as the practical context the paper is testing."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides a comprehensive program-repair study on QuixBugs that supports treating this benchmark as a standard testbed."},{"cited_title":"Ouyang, J.M","cited_arxiv_id":null,"evidence_quote":"Reports bit-for-bit identical output rates for ChatGPT across code benchmarks, grounding the expectation of run-to-run divergence."},{"cited_title":"Levenshtein, Binary codes capable of correcting deletions, insertions, and reversals, Soviet Physics","cited_arxiv_id":null,"evidence_quote":"Defines Levenshtein distance, the basis of the syntactic similarity metric used throughout the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the QuixBugs dataset reference that supplies the 20 Python problems used in the experiments."}],"review_version":1}