{"id":"4c07794f-6299-4f13-b5a9-5bb72fa6714c","arxiv_id":"2505.04441","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Execution traces in prompts give inconsistent gains for LLM-based program repair, while LLM-optimized trace summaries improve consistency.","lead":"This paper tests whether adding runtime execution traces to prompts helps large language models fix buggy code, and finds that plain traces help in only a minority of cases. It shows that summarizing traces with the LLM itself gives more consistent repair gains, which is useful guidance for building program repair tools.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OPT trace gains are confounded by the 200-line prompt truncation; raw traces lose the tail containing the failing return value, so the OPT advantage may reflect prompt length rather than trace optimization.","rationale":"I read the paper in good faith and acknowledge that the RQ1 negative finding is well supported by the table: raw traces do not consistently beat error-only prompts, and the paper is appropriately cautious there. The central positive contribution is the RQ3 claim that OPT traces are the most consistent prompting technique. That claim is load-bearing for the paper's value, and it is where the argument is least secure. The reader's weakest assumption, that OPT summaries may omit repair-relevant state changes, is a valid concern. However, an even more direct and concrete confound exists: Section 3.1's uniform 200-line truncation policy disproportionately shortens raw trace prompts, cutting off the tail of the trace where the failing return value and final state changes appear. Since OPT summaries are generated to be shorter, they are less likely to be truncated and can retain information from the full trace. The paper does not control for this difference in prompt length or truncation behavior. This confound could fully explain the OPT advantage over raw traces, independent of summary fidelity. It also explains why the paper's RQ2 finding (longer traces hurt) might be an artifact: longer traces are more likely to be truncated, and the truncation removes the diagnostic tail. The proposed tests are straightforward and would settle the issue. Because this concern can be addressed with additional experiments and does not invalidate the paper's broader empirical contribution, the CONDITIONAL verdict from the reader remains appropriate; I would not change it. I agree with the reader only partially because the identified mechanism differs, though both point to the OPT result being potentially artifactual.","tokens_in":11403,"tokens_out":7377,"duration_ms":68543,"concrete_test":"Re-run the RQ3 evaluation on the subset of programs whose raw traces exceed 200 lines, using three prompt variants: (a) the paper's raw trace truncated as currently implemented, (b) a length-matched raw trace truncated to the same line count as the OPT summary but preserving the final lines of the trace (tail-preserving), and (c) the OPT trace. Compare CFA and CPA for both GPT-3.5 and GPT-4. If variant (b) meets or exceeds (c), the OPT advantage is due to truncation. As a second check, rerun the full experiment with the 200-line limit removed or raised to 2048 lines for both raw and OPT traces; if raw traces then match OPT, the RQ3 conclusion is an artifact of the truncation policy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's most positive result, that LLM-optimized traces (OPT) give the most consistent APR gains (RQ3 Summary, Table 2), is confounded by the prompt truncation policy in Section 3.1: 'we truncate the content of the prompt if the number of lines exceeds 200.' Footnote 1 reports that a significant number of trace prompts are truncated (5% for Refactory and almost 10% for RunBugRun) and that traces can exceed 10,000 entries. Because the trace is placed after the buggy program and failing test (Figure 1), truncating the tail of the prompt removes the final trace entries, including the return value and last variable state that reveal the bug, such as 'Return value:.. None'. OPT traces are deliberately shortened by GPT-4-32k, so they fit within 200 lines and can preserve a summary of the full execution tail. Thus the comparison between OPT and raw Trace prompts is not length-controlled: raw traces may lose the very information needed for repair, while OPT retains it. The observed OPT advantage could be an artifact of truncation rather than evidence that LLM-generated summaries are a better trace representation. The paper does not report a baseline with deterministically truncated, length-matched raw traces, so this confound is unresolved.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether augmenting LLM prompts with program execution traces improves automatic program repair, using GPT-3.5 and GPT-4 on three APR datasets: Refactory, HumanEval-Java, and RunBugRun. It reports three main results: (RQ1) raw trace prompts do not consistently outperform prompts containing only the failing test case; (RQ2) longer traces and more variable modifications are associated with fewer successful repairs; and (RQ3) among several modified trace representations, LLM-optimized (OPT) traces provide the most consistent gains, especially on Correct Fix Accuracy. The paper also compares trace-based prompting against fine-tuning a 1.3B DeepSeek-Coder model and probes GPT-4's ability to align execution traces with code and to predict traces from code.","tokens_in":11621,"tokens_out":8479,"duration_ms":74491,"significance":"If the claims are supported, the paper would be a useful empirical qualification of execution-trace augmentation for LLM-based APR: it would show that simply appending raw traces is not reliably helpful, while LLM-generated trace summaries can be a more practical representation. Strengths include the explicit framing of a negative RQ1 result, the comparison against trace-free Self-Debug and error-only baselines, and the probing experiments that suggest GPT-4 cannot perfectly reproduce or collate traces. The main limitation is that the paper's positive OPT result is not yet cleanly supported because of a prompt-truncation confound and missing methodological detail for Java trace generation; the quantitative support also lacks variance estimates and significance tests.","major_comments":[{"comment":"The paper's main positive result—that LLM-optimized (OPT) traces outperform deterministic Trace prompts—is confounded by the 200-line prompt truncation policy. Section 3.1 states that 'we truncate the content of the prompt if the number of lines exceeds 200,' and because the trace appears after the program and failing test (Figure 1), truncation removes the tail of long raw traces, including the final return value and last variable states (e.g., 'Return value:.. None' in Figure 1). Footnote 1 reports that roughly 5% of Refactory prompts and nearly 10% of RunBugRun prompts are truncated, with traces exceeding 10,000 entries. OPT traces are deliberately shortened by GPT-4-32k and therefore fit within the limit while retaining end-of-trace information, so the comparison is not length-controlled. The absence of a deterministic, length-matched truncated-trace baseline means the observed OPT advantage could be explained by prompt truncation rather than by trace optimization. This is load-bearing for the RQ3 summary and should be resolved with a controlled experiment or a re-analysis restricted to untruncated instances.","section":"§3.1, §3.3 (footnote 1), §4.1"},{"comment":"The trace generation methodology is described only for Python (PySnooper), but the evaluation includes HumanEval-Java, a Java dataset. The paper does not state which tool or procedure produced execution traces for Java, how exceptions or standard output were captured, or how the Java traces were postprocessed and truncated. Section 4.1 refers to 'execution traces for both languages,' confirming that Java traces exist, but the missing description is a reproducibility gap for one of the three datasets. Please add the exact trace-generation setup for Java and, if possible, release the trace data or a representative sample of traces.","section":"§3.1 (Execution Traces Generation), §4.1"},{"comment":"All quantitative comparisons are single-run point estimates without confidence intervals or significance tests, despite the stochasticity of GPT-3.5/GPT-4 and the fact that one fix is generated per test-case prompt. Several differences supporting the RQ1/RQ3 narrative are small: for example, GPT-4 HumanEval-Java CPA is 0.713 for Trace vs 0.662 for Error (Table 1), and the OPT-vs-RQ1-best CFA gaps in Table 2 are often 0.02–0.07. The phrase 'significantly higher' in §3.3 is used without a statistical test, and 'most consistent' in §4.2 is based on point estimates. Please provide paired tests across the same programs or repeated sampling with variance estimates so that the load-bearing consistency claim can be evaluated.","section":"Tables 1 and 2, §3.2, §3.3, §4.2"},{"comment":"The RQ3 claim that OPT is 'among the top three performing prompting techniques' for CPA is not supported by Table 2 in every configuration. For GPT-3.5 Refactory CPA, RQ1 Best is 0.659, TRL OPT (EP) and TRL OPT (OPT) are 0.623, OPT is 0.601, Collated is 0.587, and Conf OPT is 0.384, so OPT is fourth if RQ1 Best is counted. For GPT-4 RunBugRun CPA, OPT's 0.589 is below RQ1 Best (0.629) and TRL OPT (OPT) (0.603). On CFA, OPT does not improve over RQ1 Best in three of six configurations (Refactory with both models and RunBugRun with GPT-4). The RQ3 summary should be qualified to a per-configuration competitive claim rather than a uniform gain.","section":"§4.2, Table 2"}],"minor_comments":[{"comment":"The reported TRL OPT numbers in Table 2 appear to be the best observed over the threshold grid N ∈ {25,30,35,40,45,50} without a validation split; please label these as oracle-selected or provide a cross-validated threshold choice.","section":"§4.1, Figures 6–7"},{"comment":"The sentence 'Section 4 covers RQ2' should read 'Section 4 covers RQ3'.","section":"Introduction, §3"},{"comment":"Since the text relies on median trace length and median number of variable modifications, please mark the median explicitly on the box plots and define the box and whisker ranges in the caption.","section":"Figure 2 and §3.3"},{"comment":"The instruction given to GPT-4-32k for trace summarization is not included; adding the exact prompt, the requested output length, and one or two original-versus-summarized trace pairs would substantially improve reproducibility.","section":"§4.1"},{"comment":"The fine-tuning comparison uses a 1.3B DeepSeek-Coder model, whereas the prompting baselines use GPT-3.5/GPT-4; the abstract's wording 'trace-based prompting to be superior to finetuning a smaller LLM' is accurate, but the caption of Figure 4 should state model sizes and training data sizes to avoid overgeneralization.","section":"§5.1, Figure 4"},{"comment":"The paper does not include a data/code availability statement; releasing the trace-generation pipeline, the summarization prompts, or at least a sample of traces would help the community reproduce the study.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The RQ1 negative result is honest and likely valuable, but the central positive OPT claim needs a length-controlled control before publication. The missing Java trace-generation methodology is also a substantive reproducibility gap. I do not see a circularity problem beyond the shared GPT model family for summarization and evaluation, which the authors acknowledge."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First: the paper is worth reading for its honest negative result. It systematically tests what happens when you put real execution traces into prompts for GPT-3.5/GPT-4 on three APR datasets, and finds that raw traces only beat a simple error-prompt baseline in two of six configurations. That is useful evidence for people designing APR systems, and the RQ2 complexity analysis (longer traces, fewer correct fixes) is plausible. The two probing studies in Section 5 are a nice addition: they show GPT-4 cannot perfectly reproduce execution traces, which justifies passing real traces in at all.\n\nThe novelty is real relative to prior work. TraceFixer, TRACED, and Self-Debug use traces for finetuning, pretraining, or self-generated explanations. This paper is the first systematic study of real traces injected into prompts of an off-the-shelf LLM, and it includes a good literature comparison.\n\nThe soft spot is load-bearing. The paper's positive RQ3 claim is that LLM-optimized traces (OPT) give the most consistent gains. But the prompt construction truncates anything over 200 lines (Section 3.1), and the footnote says 5–10% of trace prompts are truncated, with some traces over 10,000 entries. Since the trace appears after the program and failing test, truncation cuts off the end of the trace—exactly where the return value and final variable states live. OPT traces are summarized to be short, so they survive truncation and keep that tail. The comparison is not length-controlled. The OPT advantage could come from simply preserving the tail, not from intelligent summarization. The paper does not include a baseline of mechanically truncated raw traces matched to OPT length, so this confound is unresolved. That weakens the central constructive claim.\n\nOther issues are milder. No error bars, significance tests, or multiple runs. The TRL OPT routing thresholds are tuned on the evaluation set (they report the best N over 25–50), which inflates those numbers. No code or exact prompts are released, which makes reproduction harder. And the summary-fidelity assumption—that GPT-4-32k's reductions keep repair-relevant changes—is never directly validated.\n\nNone of this looks like deliberate distortion. The paper is honestly written; the authors flag unexplained results and state the RQ1 summary as a negative. The missing controls look like oversights rather than spin, and they are fixable. With a length-matched baseline and variance estimates, the paper would be solid.\n\nRecommendation: yes, send this to peer review. It deserves referee time. But the referee should require the truncation control and error bars before publication. This is useful for APR researchers, and the probing studies have standalone value.","headline":"Honest negative result on raw execution traces, but the optimistic OPT result is confounded by prompt truncation; needs a length-matched baseline before the main positive claim is credible.","tokens_in":12193,"tokens_out":2905,"would_cite":true,"duration_ms":26179,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding raw execution traces to LLM prompts rarely improves automated program repair, but LLM-optimized trace summaries do.","keywords":["automatic program repair","execution traces","large language models","prompt engineering","trace optimization","GPT program repair","code repair benchmarks","trace complexity"],"falsifier":"Compare OPT against a matched control that removes the same number of lines from the raw trace by simple truncation or random deletion, holding total prompt length constant; if truncated traces match OPT's fix accuracy, then the content of the LLM's summary is irrelevant and the gain is just shorter prompts. Additionally, instrument the summaries against the ground-truth trace and count omitted variable modifications; a correlation between omitted state changes and failed fixes would show the summaries lose repair-critical information.","tokens_in":11167,"feed_emoji":"🐞","tokens_out":7151,"duration_ms":64085,"temperature":0.7,"pith_summary":"This paper asks whether execution traces—logs of variable changes, calls, and returns from actually running the buggy program—improve LLM-based automatic program repair when added to the prompt. The answer is nuanced: raw traces rarely help and sometimes hurt; on only two of six dataset/model combinations do they beat error-only prompts. Longer and more complex traces reduce the chance of a correct fix. The most reliable way to use traces is to have a long-context LLM summarize them into an optimized trace, which produces more consistent repair gains. The paper also finds trace-based prompting beats finetuning a small model, and that real traces add information LLMs cannot yet fully reproduce themselves.","feed_headline":"Raw execution traces rarely help LLM program repair","feed_subtitle":"Summarized traces from a long-context LLM produce the most consistent fix gains across datasets.","key_machinery":"The central object is the execution-trace prompt: a Python function decorated with the PySnooper tracer logs every variable initialization and change, call, return, and exception with its line number, and this log is appended to the buggy program and failing test case. The paper manipulates this object along two axes. Trace complexity, measured by trace length and number of variable modifications, is shown to moderate success. The improved variant, the OPT trace, is produced by asking GPT-4-32k to rewrite the deterministic trace into a shorter version optimized for downstream APR; this compression step is what carries the main positive result. The alternative mechanisms—collating trace lines as inline comments, confidence-based routing, and trace-length threshold routing—all underperform OPT.","core_discovery":"The paper examines whether putting a program's execution trace into an APR prompt helps a GPT model repair the bug. On three datasets (Refactory, RunBugRun, HumanEval-Java) and two models (GPT-3.5, GPT-4), raw trace prompts beat the error-only prompt baseline in only two of six configurations and never consistently beat it. Trace complexity explains part of the pattern: fixes are less likely to be correct when traces are longer and contain more variable assignments. The paper then shows that replacing the raw trace with a shortened, LLM-generated summary (OPT) yields the most consistent gains, with OPT among the top three on Correct Program Accuracy for all six dataset/model pairs and the best or second-best Correct Fix Accuracy in all six. A finetuned 1.3B model performs worse than all prompting variants, and probing experiments show GPT-4 cannot perfectly produce or align execution traces, suggesting real traces still carry information the model cannot infer.","pith_inferences":["The OPT result suggests a division of labor: let a long-context model compress raw logs and let the repair model use the compressed form; this pattern likely transfers to other log-rich code tasks beyond APR, such as test failure diagnosis.","Because GPT-4 benefits while GPT-3.5 does not, trace augmentation may only pay off above a capability threshold; a testable extension is to measure whether open-weights models of comparable size show the same threshold.","The summarization prompt is evaluated only through downstream APR metrics, not trace fidelity; a direct evaluation could check whether omitted variable modifications explain fix failures, and whether simple truncation would match OPT, isolating the contribution of summarization semantics.","Trace-based prompting beating finetuning on small data may not hold with larger finetuning budgets, since the paper's own finetuning uses only about 500 examples per dataset."],"forward_implications":["Raw execution traces should not be assumed to help APR: adding them to prompts can reduce fix accuracy, so systems need a complexity-aware trace strategy.","Trace length and number of variable assignments are usable predictors of when trace augmentation will fail.","LLM-generated trace summaries are a more consistent augmentation than raw traces, producing top-three Correct Program Accuracy in all six settings and best or second-best Correct Fix Accuracy in all six.","Trace-based prompting outperforms finetuning a small 1.3B code model on the same small training sets, in both Correct Fix Accuracy and Correct Program Accuracy.","Probing results imply self-generated traces are unreliable because LLMs cannot reproduce exact traces, so external execution information remains potentially valuable."],"supporting_citations":[{"why":"Provides the Self-Debug chain-of-thought baseline that trace prompts are compared against.","marker":"(Chen et al., 2023)"},{"why":"Supplies the TraceFixer finetuning setup that the prompting comparison is modeled on.","marker":"(Bouzenia et al., 2023)"},{"why":"Provides the Refactory dataset of student-written Python bugs.","marker":"(Hu et al., 2019)"},{"why":"Provides the RunBugRun dataset of executable Python submissions.","marker":"(Prenner and Robbes, 2023)"},{"why":"Supplies HumanEval-Java, one of the three APR evaluation datasets.","marker":"(Jiang et al., 2023)"},{"why":"Provides PySnooper, the library used to generate execution traces.","marker":"(Rachum et al., 2019)"},{"why":"Provides the prompt instruction template on which all prompt variants are based.","marker":"(Xia et al., 2023)"},{"why":"Defines the GPT-4 model used in the main experiments.","marker":"(OpenAI, 2023)"},{"why":"Contributes the test-execution-diagnostics baseline underlying the error-only prompt.","marker":"(Ye et al., 2022)"}],"fun_headline_variants":["Summarized traces outperform raw traces for LLM repair","Raw execution traces seldom aid LLM program repair","For code LLMs, raw traces underdeliver; summaries shine","Trace complexity erodes LLM repair; summaries fix that","LLM repair: raw traces flop, but summarized traces win"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that GPT-4-32k's shortened optimized traces preserve the variable-state changes the repair model needs while only cutting redundant log content; the paper never checks whether the summaries drop, distort, or add state information, so the observed OPT advantage could in principle come from shorter prompts or from the summarizer sharing the repair model's own failure modes rather than from better trace content.","fun_headline_variants_meta":{"raw":{"variants":["Summarized traces outperform raw traces for LLM repair","Raw execution traces seldom aid LLM program repair","For code LLMs, raw traces underdeliver; summaries shine","Trace complexity erodes LLM repair; summaries fix that","LLM repair: raw traces flop, but summarized traces win"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000713,"raw_usage":{"total_tokens":3197,"prompt_tokens":923,"completion_tokens":2274,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":2206}},"tokens_in":539,"tokens_out":2274,"duration_ms":14659,"temperature":1.0,"reasoning_tokens":2206,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:28:11.095880+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare OPT against a matched control that removes the same number of lines from the raw trace by simple truncation or random deletion, holding total prompt length constant; if truncated traces match OPT's fix accuracy, then the content of the LLM's summary is irrelevant and the gain is just shorter prompts. Additionally, instrument the summaries against the ground-truth trace and count omitted variable modifications; a correlation between omitted state changes and failed fixes would show the summaries lose repair-critical information.","supporting_citations":[],"review_version":1}