Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMs

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read Adding raw execution traces to LLM prompts rarely improves automated program repair, but LLM-optimized trace summaries do.

desk verdict Honest negative result on raw execution traces, but the optimistic OPT result is confounded by prompt truncation; needs a length-matched baseline before the main positive claim is credible. read the letter →

arxiv 2505.04441 v1 pith:FBE23ELN submitted 2025-05-07 cs.LG cs.SE

classification cs.LGcs.SE
keywords automaticprogramrepairexecutiontraceslargelanguagemodelspromptengineeringtraceoptimizationGPTcodebenchmarkscomplexity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether execution traces—logs of variable changes, calls, and returns from actually running the buggy program—improve LLM-based automatic program repair when added to the prompt. The answer is nuanced: raw traces rarely help and sometimes hurt; on only two of six dataset/model combinations do they beat error-only prompts. Longer and more complex traces reduce the chance of a correct fix. The most reliable way to use traces is to have a long-context LLM summarize them into an optimized trace, which produces more consistent repair gains. The paper also finds trace-based prompting beats finetuning a small model, and that real traces add information LLMs cannot yet fully reproduce themselves.

What carries the argument

The central object is the execution-trace prompt: a Python function decorated with the PySnooper tracer logs every variable initialization and change, call, return, and exception with its line number, and this log is appended to the buggy program and failing test case. The paper manipulates this object along two axes. Trace complexity, measured by trace length and number of variable modifications, is shown to moderate success. The improved variant, the OPT trace, is produced by asking GPT-4-32k to rewrite the deterministic trace into a shorter version optimized for downstream APR; this compression step is what carries the main positive result. The alternative mechanisms—collating trace lines as inline comments, confidence-based routing, and trace-length threshold routing—all underperform OPT.

What would settle it

Compare OPT against a matched control that removes the same number of lines from the raw trace by simple truncation or random deletion, holding total prompt length constant; if truncated traces match OPT's fix accuracy, then the content of the LLM's summary is irrelevant and the gain is just shorter prompts. Additionally, instrument the summaries against the ground-truth trace and count omitted variable modifications; a correlation between omitted state changes and failed fixes would show the summaries lose repair-critical information.

Watch

Extended reading notes

Core claim

The paper examines whether putting a program's execution trace into an APR prompt helps a GPT model repair the bug. On three datasets (Refactory, RunBugRun, HumanEval-Java) and two models (GPT-3.5, GPT-4), raw trace prompts beat the error-only prompt baseline in only two of six configurations and never consistently beat it. Trace complexity explains part of the pattern: fixes are less likely to be correct when traces are longer and contain more variable assignments. The paper then shows that replacing the raw trace with a shortened, LLM-generated summary (OPT) yields the most consistent gains, with OPT among the top three on Correct Program Accuracy for all six dataset/model pairs and the best or second-best Correct Fix Accuracy in all six. A finetuned 1.3B model performs worse than all prompting variants, and probing experiments show GPT-4 cannot perfectly produce or align execution traces, suggesting real traces still carry information the model cannot infer.

Load-bearing premise

The load-bearing premise is that GPT-4-32k's shortened optimized traces preserve the variable-state changes the repair model needs while only cutting redundant log content; the paper never checks whether the summaries drop, distort, or add state information, so the observed OPT advantage could in principle come from shorter prompts or from the summarizer sharing the repair model's own failure modes rather than from better trace content.

Editorial extensions

If this is right

  • Raw execution traces should not be assumed to help APR: adding them to prompts can reduce fix accuracy, so systems need a complexity-aware trace strategy.
  • Trace length and number of variable assignments are usable predictors of when trace augmentation will fail.
  • LLM-generated trace summaries are a more consistent augmentation than raw traces, producing top-three Correct Program Accuracy in all six settings and best or second-best Correct Fix Accuracy in all six.
  • Trace-based prompting outperforms finetuning a small 1.3B code model on the same small training sets, in both Correct Fix Accuracy and Correct Program Accuracy.
  • Probing results imply self-generated traces are unreliable because LLMs cannot reproduce exact traces, so external execution information remains potentially valuable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The OPT result suggests a division of labor: let a long-context model compress raw logs and let the repair model use the compressed form; this pattern likely transfers to other log-rich code tasks beyond APR, such as test failure diagnosis.
  • Because GPT-4 benefits while GPT-3.5 does not, trace augmentation may only pay off above a capability threshold; a testable extension is to measure whether open-weights models of comparable size show the same threshold.
  • The summarization prompt is evaluated only through downstream APR metrics, not trace fidelity; a direct evaluation could check whether omitted variable modifications explain fix failures, and whether simple truncation would match OPT, isolating the contribution of summarization semantics.
  • Trace-based prompting beating finetuning on small data may not hold with larger finetuning budgets, since the paper's own finetuning uses only about 500 examples per dataset.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper investigates whether augmenting LLM prompts with program execution traces improves automatic program repair, using GPT-3.5 and GPT-4 on three APR datasets: Refactory, HumanEval-Java, and RunBugRun. It reports three main results: (RQ1) raw trace prompts do not consistently outperform prompts containing only the failing test case; (RQ2) longer traces and more variable modifications are associated with fewer successful repairs; and (RQ3) among several modified trace representations, LLM-optimized (OPT) traces provide the most consistent gains, especially on Correct Fix Accuracy. The paper also compares trace-based prompting against fine-tuning a 1.3B DeepSeek-Coder model and probes GPT-4's ability to align execution traces with code and to predict traces from code.

Significance. If the claims are supported, the paper would be a useful empirical qualification of execution-trace augmentation for LLM-based APR: it would show that simply appending raw traces is not reliably helpful, while LLM-generated trace summaries can be a more practical representation. Strengths include the explicit framing of a negative RQ1 result, the comparison against trace-free Self-Debug and error-only baselines, and the probing experiments that suggest GPT-4 cannot perfectly reproduce or collate traces. The main limitation is that the paper's positive OPT result is not yet cleanly supported because of a prompt-truncation confound and missing methodological detail for Java trace generation; the quantitative support also lacks variance estimates and significance tests.

major comments (4)
  1. [§3.1, §3.3 (footnote 1), §4.1] The paper's main positive result—that LLM-optimized (OPT) traces outperform deterministic Trace prompts—is confounded by the 200-line prompt truncation policy. Section 3.1 states that 'we truncate the content of the prompt if the number of lines exceeds 200,' and because the trace appears after the program and failing test (Figure 1), truncation removes the tail of long raw traces, including the final return value and last variable states (e.g., 'Return value:.. None' in Figure 1). Footnote 1 reports that roughly 5% of Refactory prompts and nearly 10% of RunBugRun prompts are truncated, with traces exceeding 10,000 entries. OPT traces are deliberately shortened by GPT-4-32k and therefore fit within the limit while retaining end-of-trace information, so the comparison is not length-controlled. The absence of a deterministic, length-matched truncated-trace baseline means the observed OPT advantage could be explained by prompt truncation rather than by trace optimization. This is load-bearing for the RQ3 summary and should be resolved with a controlled experiment or a re-analysis restricted to untruncated instances.
  2. [§3.1 (Execution Traces Generation), §4.1] The trace generation methodology is described only for Python (PySnooper), but the evaluation includes HumanEval-Java, a Java dataset. The paper does not state which tool or procedure produced execution traces for Java, how exceptions or standard output were captured, or how the Java traces were postprocessed and truncated. Section 4.1 refers to 'execution traces for both languages,' confirming that Java traces exist, but the missing description is a reproducibility gap for one of the three datasets. Please add the exact trace-generation setup for Java and, if possible, release the trace data or a representative sample of traces.
  3. [Tables 1 and 2, §3.2, §3.3, §4.2] All quantitative comparisons are single-run point estimates without confidence intervals or significance tests, despite the stochasticity of GPT-3.5/GPT-4 and the fact that one fix is generated per test-case prompt. Several differences supporting the RQ1/RQ3 narrative are small: for example, GPT-4 HumanEval-Java CPA is 0.713 for Trace vs 0.662 for Error (Table 1), and the OPT-vs-RQ1-best CFA gaps in Table 2 are often 0.02–0.07. The phrase 'significantly higher' in §3.3 is used without a statistical test, and 'most consistent' in §4.2 is based on point estimates. Please provide paired tests across the same programs or repeated sampling with variance estimates so that the load-bearing consistency claim can be evaluated.
  4. [§4.2, Table 2] The RQ3 claim that OPT is 'among the top three performing prompting techniques' for CPA is not supported by Table 2 in every configuration. For GPT-3.5 Refactory CPA, RQ1 Best is 0.659, TRL OPT (EP) and TRL OPT (OPT) are 0.623, OPT is 0.601, Collated is 0.587, and Conf OPT is 0.384, so OPT is fourth if RQ1 Best is counted. For GPT-4 RunBugRun CPA, OPT's 0.589 is below RQ1 Best (0.629) and TRL OPT (OPT) (0.603). On CFA, OPT does not improve over RQ1 Best in three of six configurations (Refactory with both models and RunBugRun with GPT-4). The RQ3 summary should be qualified to a per-configuration competitive claim rather than a uniform gain.
minor comments (6)
  1. [§4.1, Figures 6–7] The reported TRL OPT numbers in Table 2 appear to be the best observed over the threshold grid N ∈ {25,30,35,40,45,50} without a validation split; please label these as oracle-selected or provide a cross-validated threshold choice.
  2. [Introduction, §3] The sentence 'Section 4 covers RQ2' should read 'Section 4 covers RQ3'.
  3. [Figure 2 and §3.3] Since the text relies on median trace length and median number of variable modifications, please mark the median explicitly on the box plots and define the box and whisker ranges in the caption.
  4. [§4.1] The instruction given to GPT-4-32k for trace summarization is not included; adding the exact prompt, the requested output length, and one or two original-versus-summarized trace pairs would substantially improve reproducibility.
  5. [§5.1, Figure 4] The fine-tuning comparison uses a 1.3B DeepSeek-Coder model, whereas the prompting baselines use GPT-3.5/GPT-4; the abstract's wording 'trace-based prompting to be superior to finetuning a smaller LLM' is accurate, but the caption of Figure 4 should state model sizes and training data sizes to avoid overgeneralization.
  6. [General] The paper does not include a data/code availability statement; releasing the trace-generation pipeline, the summarization prompts, or at least a sample of traces would help the community reproduce the study.

Circularity Check

1 steps flagged · score 2.0 of 10

No load-bearing circularity; central OPT result is an empirical benchmark finding, with one minor secondary result (TRL OPT) that selects the best threshold on the evaluation data.

  1. fitted input called prediction [Section 4.2 (RQ3 Results, Table 2) and Section 4.1 (TRL OPT description)]
    "We use the following N values for the experiment: 25,30,35,40,45,50. ... For trace length-based prompt selection, we only report the best results in the table."

    The Table 2 TRL OPT entries are, by construction, the best CFA/CPA over the six routing thresholds N evaluated on the same test data used to report performance. No held-out validation selection is described, so the reported numbers are selection maxima rather than the performance of a fixed, pre-specified routing policy. This fits the fitted-input-called-prediction pattern, but it is confined to a secondary ablation: the paper itself says changing N had limited impact and does not use TRL OPT to support its main conclusion that OPT traces are the most consistent technique.

full rationale

The paper's central claims are empirical comparisons on external APR datasets (Refactory, RunBugRun, HumanEval-Java) against trace-free baselines, with no fitted equations or derivation that reduce a predicted quantity to its own inputs. The main OPT result is not circular: GPT-4-32k trace summarization is an external generation step, not a parameter fitted to the repair outcomes, and the comparison to Error/Trace prompts is an independent benchmark evaluation. The same-model-family confound (GPT-4-32k summarizes, GPT-4 repairs) is a validity threat, not a definitional circularity. The 200-line truncation policy is likewise a prompt-length confound, not a circular reduction. One secondary issue is the TRL OPT result, where the best threshold N is reported after searching N in {25,...,50} on the evaluation data; this is a selection-on-test-set artifact, but the paper explicitly downplays routing's benefit and does not rest the main conclusion on it. No self-citation chain or imported uniqueness theorem is load-bearing, so the overall circularity is minor.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

This is an empirical study, so there are no mathematical axioms beyond the domain assumptions listed. The load-bearing assumptions are about trace fidelity, dataset representativeness, evaluation via test cases, and the unverified fidelity of LLM-generated trace summaries. Free parameters include the truncation limit, the confidence threshold, and routing thresholds, with the best routing results selected post hoc.

free parameters (3)
  • Trace-length routing threshold N = Selected from {25, 30, 35, 40, 45, 50}; best value per dataset/model not reported
    In Section 4.1, TRL OPT routes to optimized or error prompts when trace length exceeds N. Section 4.2 reports the best results over this grid, so the threshold is tuned on evaluation data rather than fixed a priori.
  • Confidence threshold for Conf OPT = 3 (Likert scale 1-5)
    In Section 4.1, confidence scores below 3 are treated as low confidence. The threshold is arbitrary and is not ablated or justified by data.
  • Prompt truncation limit = 200 lines
    Section 3.1 caps prompt content at 200 lines. Because trace prompts are longer, truncation may disproportionately affect them, and the choice of limit is a design decision with no supporting experiment.
assumptions (6)
  • domain assumption PySnooper traces are a faithful and complete record of runtime variable changes.
    Section 3.1; the entire trace-augmentation claim depends on traces accurately reflecting execution, yet the generated traces are not validated against a ground-truth execution record.
  • domain assumption Passing all provided test cases defines a correct fix.
    Section 3.1 metrics; this is standard in APR but does not measure behavior on hidden or held-out tests.
  • domain assumption One generated fix per prompt is sufficient to compare prompt strategies.
    Section 3.1 states 'we generate a single prediction per test case-specific prompt'; no temperature, seed, or multiple-sample variance is reported, so reported differences may be sampling noise.
  • ad hoc to paper LLM-optimized trace summaries preserve the information needed for repair.
    Section 4.1; the summarization step is never evaluated for omission or alteration of state changes, yet OPT gains are the main positive result of RQ3.
  • domain assumption The selected datasets are representative enough for conclusions about APR.
    Section 3.1; the datasets are self-contained algorithmic problems rather than full open-source projects, and the paper acknowledges this limitation.
  • domain assumption Commercial API model behavior is stable across the evaluation period.
    The models are accessed through the OpenAI API without pinned versions or access dates, so results may not reproduce after model updates.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMs." pith.science (2026). https://pith.science/paper/FBE23ELN

@misc{pith2026250504441,
  author       = {Pith},
  title        = {Pith review of: Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FBE23ELN}},
  note         = {Machine review of arXiv:2505.04441}
}
read the original abstract

Large Language Models (LLMs) show promising performance on various programming tasks, including Automatic Program Repair (APR). However, most approaches to LLM-based APR are limited to the static analysis of the programs, while disregarding their runtime behavior. Inspired by knowledge-augmented NLP, in this work, we aim to remedy this potential blind spot by augmenting standard APR prompts with program execution traces. We evaluate our approach using the GPT family of models on three popular APR datasets. Our findings suggest that simply incorporating execution traces into the prompt provides a limited performance improvement over trace-free baselines, in only 2 out of 6 tested dataset / model configurations. We further find that the effectiveness of execution traces for APR diminishes as their complexity increases. We explore several strategies for leveraging traces in prompts and demonstrate that LLM-optimized prompts help outperform trace-free prompts more consistently. Additionally, we show trace-based prompting to be superior to finetuning a smaller LLM on a small-scale dataset; and conduct probing studies reinforcing the notion that execution traces can complement the reasoning abilities of the LLMs.

Figures

Figures reproduced from arXiv: 2505.04441 by the authors.

Figure 1
Figure 1. Example buggy program, a failing test case [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Distributions of trace lengths and variable [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The Flow of Conditional Selection of Traces [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: CPA and CFA of Prompting-Based Tech￾niques vs. Model Fine-Tuned for APR Tasks. number of training examples for TraceFixer was significantly higher, too. In our future work, we plan to use a larger training dataset and larger mod￾els for finetuning. 5.2 Probing Studies …
Figure 5
Figure 5. Figure 5: Distributions of trace lengths and variable changes across correct vs incorrect program fixes generated by [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Ablations of trace length threshold values used with the routing strategy for GPT3.5 [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Ablations of trace length threshold values used with the routing strategy for GPT-4. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ReLog: Execution-Aware Logging with Runtime Feedback for LLM-Oriented Debugging

    cs.SE 2026-03 conditional novelty 6.0 of 10

    ReLog iteratively writes and rewrites logging statements guided by runtime feedback, and its logs beat static logging baselines on Defects4J debugging tasks.

Reference graph

Works this paper leans on

21 extracted references · 5 canonical work pages · cited by 1 Pith paper

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Elena N Akimova, Alexander Yu Bersenev, Artem A Deikov, Konstantin S Kobylkin, Anton V Konygin, Ilya P Mezentsev, and Vladimir E Misilov. 2021. Pytracebugs: A large python code dataset for supervised machine learning in software defect prediction. In 2021 28th Asia-Pacific Software Engineering Conference (APSEC), pages 141--151. IEEE

  4. [4]

    Islem Bouzenia, Yangruibo Ding, Kexin Pei, Baishakhi Ray, and Michael Pradel. 2023. http://arxiv.org/abs/2304.12743 Tracefixer: Execution trace-driven program repair

  5. [5]

    Xinyun Chen, Maxwell Lin, Nathanael Sch \"a rli, and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128

  6. [6]

    Yangruibo Ding, Ben Steenhoek, Kexin Pei, Gail Kaiser, Wei Le, and Baishakhi Ray. 2023. http://arxiv.org/abs/2306.07487 Traced: Execution-aware pre-training for source code

  7. [7]

    Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. http://arxiv.org/abs/2203.03850 Unixcoder: Unified cross-modal pre-training for code representation

  8. [8]

    Yang Hu, Umair Z Ahmed, Sergey Mechtaev, Ben Leong, and Abhik Roychoudhury. 2019. Re-factoring based program repair applied to programming assignments. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 388--398. IEEE

Show all 21 references
  1. [9]

    Kung-Hsiang Huang, Philippe Laban, Alexander R Fabbri, Prafulla Kumar Choubey, Shafiq Joty, Caiming Xiong, and Chien-Sheng Wu. 2023. Embrace divergence for richer insights: A multi-document summarization benchmark and a case study on summarizing diverse information from news a...

  2. [10]

    Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. 2023. http://arxiv.org/abs/2302.05020 Impact of code language models on automated program repair

  3. [11]

    Chenxiao Liu, Shuai Lu, Weizhu Chen, Daxin Jiang, Alexey Svyatkovskiy, Shengyu Fu, Neel Sundaresan, and Nan Duan. 2023. http://arxiv.org/abs/2305.05383 Code execution with pre-trained language models

  4. [12]

    OpenAI. 2023. https://api.semanticscholar.org/CorpusID:257532815 Gpt-4 technical report . ArXiv, abs/2303.08774

  5. [13]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 3...

  6. [14]

    Julian Aron Prenner and Romain Robbes. 2023. http://arxiv.org/abs/2304.01102 Runbugrun -- an executable dataset for automated program repair

  7. [15]

    Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, et al. 2021. Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks. arXiv preprint arXiv:2105.12655

  8. [16]

    Ram Rachum, Alex Hall, Iori Yanokura, et al. 2019. https://doi.org/10.5281/zenodo.10462459 Pysnooper: Never use print for debugging again

  9. [17]

    Charles Sutton, David Bieber, Kensen Shi, Kexin Pei, and Pengcheng Yin. 2023. Can large language models reason about program invariants?

  10. [18]

    Bissyandé

    Haoye Tian, Weiqi Lu, Tsz On Li, Xunzhu Tang, Shing-Chi Cheung, Jacques Klein, and Tegawendé F. Bissyandé. 2023. http://arxiv.org/abs/2304.11938 Is chatgpt the ultimate programming assistant -- how far is it?

  11. [19]

    Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. https://doi.org/10.1109/ICSE48619.2023.00129 Automated program repair in the era of large pre-trained language models . In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1482--1494

  12. [20]

    Chunqiu Steven Xia and Lingming Zhang. 2022. Less training, more repairing please: revisiting automated program repair via zero-shot learning. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering...

  13. [21]

    He Ye, Matias Martinez, Xiapu Luo, Tao Zhang, and Martin Monperrus. 2022. Selfapr: Self-supervised program repair with test execution diagnostics. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, pages 1--13

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.