REVIEW 4 major objections 6 minor 1 cited by
Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMs
T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Adding raw execution traces to LLM prompts rarely improves automated program repair, but LLM-optimized trace summaries do.
desk verdict Honest negative result on raw execution traces, but the optimistic OPT result is confounded by prompt truncation; needs a length-matched baseline before the main positive claim is credible. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the execution-trace prompt: a Python function decorated with the PySnooper tracer logs every variable initialization and change, call, return, and exception with its line number, and this log is appended to the buggy program and failing test case. The paper manipulates this object along two axes. Trace complexity, measured by trace length and number of variable modifications, is shown to moderate success. The improved variant, the OPT trace, is produced by asking GPT-4-32k to rewrite the deterministic trace into a shorter version optimized for downstream APR; this compression step is what carries the main positive result. The alternative mechanisms—collating trace lines as inline comments, confidence-based routing, and trace-length threshold routing—all underperform OPT.
What would settle it
Compare OPT against a matched control that removes the same number of lines from the raw trace by simple truncation or random deletion, holding total prompt length constant; if truncated traces match OPT's fix accuracy, then the content of the LLM's summary is irrelevant and the gain is just shorter prompts. Additionally, instrument the summaries against the ground-truth trace and count omitted variable modifications; a correlation between omitted state changes and failed fixes would show the summaries lose repair-critical information.
Extended reading notes
Core claim
The paper examines whether putting a program's execution trace into an APR prompt helps a GPT model repair the bug. On three datasets (Refactory, RunBugRun, HumanEval-Java) and two models (GPT-3.5, GPT-4), raw trace prompts beat the error-only prompt baseline in only two of six configurations and never consistently beat it. Trace complexity explains part of the pattern: fixes are less likely to be correct when traces are longer and contain more variable assignments. The paper then shows that replacing the raw trace with a shortened, LLM-generated summary (OPT) yields the most consistent gains, with OPT among the top three on Correct Program Accuracy for all six dataset/model pairs and the best or second-best Correct Fix Accuracy in all six. A finetuned 1.3B model performs worse than all prompting variants, and probing experiments show GPT-4 cannot perfectly produce or align execution traces, suggesting real traces still carry information the model cannot infer.
Load-bearing premise
The load-bearing premise is that GPT-4-32k's shortened optimized traces preserve the variable-state changes the repair model needs while only cutting redundant log content; the paper never checks whether the summaries drop, distort, or add state information, so the observed OPT advantage could in principle come from shorter prompts or from the summarizer sharing the repair model's own failure modes rather than from better trace content.
Editorial extensions
If this is right
- Raw execution traces should not be assumed to help APR: adding them to prompts can reduce fix accuracy, so systems need a complexity-aware trace strategy.
- Trace length and number of variable assignments are usable predictors of when trace augmentation will fail.
- LLM-generated trace summaries are a more consistent augmentation than raw traces, producing top-three Correct Program Accuracy in all six settings and best or second-best Correct Fix Accuracy in all six.
- Trace-based prompting outperforms finetuning a small 1.3B code model on the same small training sets, in both Correct Fix Accuracy and Correct Program Accuracy.
- Probing results imply self-generated traces are unreliable because LLMs cannot reproduce exact traces, so external execution information remains potentially valuable.
Reading between the lines
- The OPT result suggests a division of labor: let a long-context model compress raw logs and let the repair model use the compressed form; this pattern likely transfers to other log-rich code tasks beyond APR, such as test failure diagnosis.
- Because GPT-4 benefits while GPT-3.5 does not, trace augmentation may only pay off above a capability threshold; a testable extension is to measure whether open-weights models of comparable size show the same threshold.
- The summarization prompt is evaluated only through downstream APR metrics, not trace fidelity; a direct evaluation could check whether omitted variable modifications explain fix failures, and whether simple truncation would match OPT, isolating the contribution of summarization semantics.
- Trace-based prompting beating finetuning on small data may not hold with larger finetuning budgets, since the paper's own finetuning uses only about 500 examples per dataset.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates whether augmenting LLM prompts with program execution traces improves automatic program repair, using GPT-3.5 and GPT-4 on three APR datasets: Refactory, HumanEval-Java, and RunBugRun. It reports three main results: (RQ1) raw trace prompts do not consistently outperform prompts containing only the failing test case; (RQ2) longer traces and more variable modifications are associated with fewer successful repairs; and (RQ3) among several modified trace representations, LLM-optimized (OPT) traces provide the most consistent gains, especially on Correct Fix Accuracy. The paper also compares trace-based prompting against fine-tuning a 1.3B DeepSeek-Coder model and probes GPT-4's ability to align execution traces with code and to predict traces from code.
Significance. If the claims are supported, the paper would be a useful empirical qualification of execution-trace augmentation for LLM-based APR: it would show that simply appending raw traces is not reliably helpful, while LLM-generated trace summaries can be a more practical representation. Strengths include the explicit framing of a negative RQ1 result, the comparison against trace-free Self-Debug and error-only baselines, and the probing experiments that suggest GPT-4 cannot perfectly reproduce or collate traces. The main limitation is that the paper's positive OPT result is not yet cleanly supported because of a prompt-truncation confound and missing methodological detail for Java trace generation; the quantitative support also lacks variance estimates and significance tests.
major comments (4)
- [§3.1, §3.3 (footnote 1), §4.1] The paper's main positive result—that LLM-optimized (OPT) traces outperform deterministic Trace prompts—is confounded by the 200-line prompt truncation policy. Section 3.1 states that 'we truncate the content of the prompt if the number of lines exceeds 200,' and because the trace appears after the program and failing test (Figure 1), truncation removes the tail of long raw traces, including the final return value and last variable states (e.g., 'Return value:.. None' in Figure 1). Footnote 1 reports that roughly 5% of Refactory prompts and nearly 10% of RunBugRun prompts are truncated, with traces exceeding 10,000 entries. OPT traces are deliberately shortened by GPT-4-32k and therefore fit within the limit while retaining end-of-trace information, so the comparison is not length-controlled. The absence of a deterministic, length-matched truncated-trace baseline means the observed OPT advantage could be explained by prompt truncation rather than by trace optimization. This is load-bearing for the RQ3 summary and should be resolved with a controlled experiment or a re-analysis restricted to untruncated instances.
- [§3.1 (Execution Traces Generation), §4.1] The trace generation methodology is described only for Python (PySnooper), but the evaluation includes HumanEval-Java, a Java dataset. The paper does not state which tool or procedure produced execution traces for Java, how exceptions or standard output were captured, or how the Java traces were postprocessed and truncated. Section 4.1 refers to 'execution traces for both languages,' confirming that Java traces exist, but the missing description is a reproducibility gap for one of the three datasets. Please add the exact trace-generation setup for Java and, if possible, release the trace data or a representative sample of traces.
- [Tables 1 and 2, §3.2, §3.3, §4.2] All quantitative comparisons are single-run point estimates without confidence intervals or significance tests, despite the stochasticity of GPT-3.5/GPT-4 and the fact that one fix is generated per test-case prompt. Several differences supporting the RQ1/RQ3 narrative are small: for example, GPT-4 HumanEval-Java CPA is 0.713 for Trace vs 0.662 for Error (Table 1), and the OPT-vs-RQ1-best CFA gaps in Table 2 are often 0.02–0.07. The phrase 'significantly higher' in §3.3 is used without a statistical test, and 'most consistent' in §4.2 is based on point estimates. Please provide paired tests across the same programs or repeated sampling with variance estimates so that the load-bearing consistency claim can be evaluated.
- [§4.2, Table 2] The RQ3 claim that OPT is 'among the top three performing prompting techniques' for CPA is not supported by Table 2 in every configuration. For GPT-3.5 Refactory CPA, RQ1 Best is 0.659, TRL OPT (EP) and TRL OPT (OPT) are 0.623, OPT is 0.601, Collated is 0.587, and Conf OPT is 0.384, so OPT is fourth if RQ1 Best is counted. For GPT-4 RunBugRun CPA, OPT's 0.589 is below RQ1 Best (0.629) and TRL OPT (OPT) (0.603). On CFA, OPT does not improve over RQ1 Best in three of six configurations (Refactory with both models and RunBugRun with GPT-4). The RQ3 summary should be qualified to a per-configuration competitive claim rather than a uniform gain.
minor comments (6)
- [§4.1, Figures 6–7] The reported TRL OPT numbers in Table 2 appear to be the best observed over the threshold grid N ∈ {25,30,35,40,45,50} without a validation split; please label these as oracle-selected or provide a cross-validated threshold choice.
- [Introduction, §3] The sentence 'Section 4 covers RQ2' should read 'Section 4 covers RQ3'.
- [Figure 2 and §3.3] Since the text relies on median trace length and median number of variable modifications, please mark the median explicitly on the box plots and define the box and whisker ranges in the caption.
- [§4.1] The instruction given to GPT-4-32k for trace summarization is not included; adding the exact prompt, the requested output length, and one or two original-versus-summarized trace pairs would substantially improve reproducibility.
- [§5.1, Figure 4] The fine-tuning comparison uses a 1.3B DeepSeek-Coder model, whereas the prompting baselines use GPT-3.5/GPT-4; the abstract's wording 'trace-based prompting to be superior to finetuning a smaller LLM' is accurate, but the caption of Figure 4 should state model sizes and training data sizes to avoid overgeneralization.
- [General] The paper does not include a data/code availability statement; releasing the trace-generation pipeline, the summarization prompts, or at least a sample of traces would help the community reproduce the study.
Circularity Check
No load-bearing circularity; central OPT result is an empirical benchmark finding, with one minor secondary result (TRL OPT) that selects the best threshold on the evaluation data.
-
fitted input called prediction
[Section 4.2 (RQ3 Results, Table 2) and Section 4.1 (TRL OPT description)]
"We use the following N values for the experiment: 25,30,35,40,45,50. ... For trace length-based prompt selection, we only report the best results in the table."
The Table 2 TRL OPT entries are, by construction, the best CFA/CPA over the six routing thresholds N evaluated on the same test data used to report performance. No held-out validation selection is described, so the reported numbers are selection maxima rather than the performance of a fixed, pre-specified routing policy. This fits the fitted-input-called-prediction pattern, but it is confined to a secondary ablation: the paper itself says changing N had limited impact and does not use TRL OPT to support its main conclusion that OPT traces are the most consistent technique.
full rationale
The paper's central claims are empirical comparisons on external APR datasets (Refactory, RunBugRun, HumanEval-Java) against trace-free baselines, with no fitted equations or derivation that reduce a predicted quantity to its own inputs. The main OPT result is not circular: GPT-4-32k trace summarization is an external generation step, not a parameter fitted to the repair outcomes, and the comparison to Error/Trace prompts is an independent benchmark evaluation. The same-model-family confound (GPT-4-32k summarizes, GPT-4 repairs) is a validity threat, not a definitional circularity. The 200-line truncation policy is likewise a prompt-length confound, not a circular reduction. One secondary issue is the TRL OPT result, where the best threshold N is reported after searching N in {25,...,50} on the evaluation data; this is a selection-on-test-set artifact, but the paper explicitly downplays routing's benefit and does not rest the main conclusion on it. No self-citation chain or imported uniqueness theorem is load-bearing, so the overall circularity is minor.
Assumptions & free parameters
free parameters (3)
- Trace-length routing threshold N =
Selected from {25, 30, 35, 40, 45, 50}; best value per dataset/model not reported
- Confidence threshold for Conf OPT =
3 (Likert scale 1-5)
- Prompt truncation limit =
200 lines
assumptions (6)
- domain assumption PySnooper traces are a faithful and complete record of runtime variable changes.
- domain assumption Passing all provided test cases defines a correct fix.
- domain assumption One generated fix per prompt is sufficient to compare prompt strategies.
- ad hoc to paper LLM-optimized trace summaries preserve the information needed for repair.
- domain assumption The selected datasets are representative enough for conclusions about APR.
- domain assumption Commercial API model behavior is stable across the evaluation period.
Cite this review
Pith. "Pith review of Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMs." pith.science (2026). https://pith.science/paper/FBE23ELN
@misc{pith2026250504441,
author = {Pith},
title = {Pith review of: Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/FBE23ELN}},
note = {Machine review of arXiv:2505.04441}
}
read the original abstract
Large Language Models (LLMs) show promising performance on various programming tasks, including Automatic Program Repair (APR). However, most approaches to LLM-based APR are limited to the static analysis of the programs, while disregarding their runtime behavior. Inspired by knowledge-augmented NLP, in this work, we aim to remedy this potential blind spot by augmenting standard APR prompts with program execution traces. We evaluate our approach using the GPT family of models on three popular APR datasets. Our findings suggest that simply incorporating execution traces into the prompt provides a limited performance improvement over trace-free baselines, in only 2 out of 6 tested dataset / model configurations. We further find that the effectiveness of execution traces for APR diminishes as their complexity increases. We explore several strategies for leveraging traces in prompts and demonstrate that LLM-optimized prompts help outperform trace-free prompts more consistently. Additionally, we show trace-based prompting to be superior to finetuning a smaller LLM on a small-scale dataset; and conduct probing studies reinforcing the notion that execution traces can complement the reasoning abilities of the LLMs.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 1 Pith paper
-
ReLog: Execution-Aware Logging with Runtime Feedback for LLM-Oriented Debugging
ReLog iteratively writes and rewrites logging statements guided by runtime feedback, and its logs beat static logging baselines on Defects4J debugging tasks.
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Elena N Akimova, Alexander Yu Bersenev, Artem A Deikov, Konstantin S Kobylkin, Anton V Konygin, Ilya P Mezentsev, and Vladimir E Misilov. 2021. Pytracebugs: A large python code dataset for supervised machine learning in software defect prediction. In 2021 28th Asia-Pacific Software Engineering Conference (APSEC), pages 141--151. IEEE
work page 2021
-
[4]
Islem Bouzenia, Yangruibo Ding, Kexin Pei, Baishakhi Ray, and Michael Pradel. 2023. http://arxiv.org/abs/2304.12743 Tracefixer: Execution trace-driven program repair
arXiv 2023
-
[5]
Xinyun Chen, Maxwell Lin, Nathanael Sch \"a rli, and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128
arXiv 2023
-
[6]
Yangruibo Ding, Ben Steenhoek, Kexin Pei, Gail Kaiser, Wei Le, and Baishakhi Ray. 2023. http://arxiv.org/abs/2306.07487 Traced: Execution-aware pre-training for source code
work page Pith review arXiv 2023
-
[7]
Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. http://arxiv.org/abs/2203.03850 Unixcoder: Unified cross-modal pre-training for code representation
arXiv 2022
-
[8]
Yang Hu, Umair Z Ahmed, Sergey Mechtaev, Ben Leong, and Abhik Roychoudhury. 2019. Re-factoring based program repair applied to programming assignments. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 388--398. IEEE
work page 2019
Show all 21 references
-
[9]
Kung-Hsiang Huang, Philippe Laban, Alexander R Fabbri, Prafulla Kumar Choubey, Shafiq Joty, Caiming Xiong, and Chien-Sheng Wu. 2023. Embrace divergence for richer insights: A multi-document summarization benchmark and a case study on summarizing diverse information from news a...
2023 arXiv
-
[10]
Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. 2023. http://arxiv.org/abs/2302.05020 Impact of code language models on automated program repair
2023 arXiv
-
[11]
Chenxiao Liu, Shuai Lu, Weizhu Chen, Daxin Jiang, Alexey Svyatkovskiy, Shengyu Fu, Neel Sundaresan, and Nan Duan. 2023. http://arxiv.org/abs/2305.05383 Code execution with pre-trained language models
2023 arXiv
-
[12]
OpenAI. 2023. https://api.semanticscholar.org/CorpusID:257532815 Gpt-4 technical report . ArXiv, abs/2303.08774
2023 arXiv
-
[13]
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 3...
2022
-
[14]
Julian Aron Prenner and Romain Robbes. 2023. http://arxiv.org/abs/2304.01102 Runbugrun -- an executable dataset for automated program repair
2023 arXiv
-
[15]
Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, et al. 2021. Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks. arXiv preprint arXiv:2105.12655
2021 arXiv
-
[16]
Ram Rachum, Alex Hall, Iori Yanokura, et al. 2019. https://doi.org/10.5281/zenodo.10462459 Pysnooper: Never use print for debugging again
2019 doi
-
[17]
Charles Sutton, David Bieber, Kensen Shi, Kexin Pei, and Pengcheng Yin. 2023. Can large language models reason about program invariants?
2023
-
[18]
Bissyandé
Haoye Tian, Weiqi Lu, Tsz On Li, Xunzhu Tang, Shing-Chi Cheung, Jacques Klein, and Tegawendé F. Bissyandé. 2023. http://arxiv.org/abs/2304.11938 Is chatgpt the ultimate programming assistant -- how far is it?
2023 arXiv
-
[19]
Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. https://doi.org/10.1109/ICSE48619.2023.00129 Automated program repair in the era of large pre-trained language models . In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1482--1494
2023
-
[20]
Chunqiu Steven Xia and Lingming Zhang. 2022. Less training, more repairing please: revisiting automated program repair via zero-shot learning. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering...
2022
-
[21]
He Ye, Matias Martinez, Xiapu Luo, Tao Zhang, and Martin Monperrus. 2022. Selfapr: Self-supervised program repair with test execution diagnostics. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, pages 1--13
2022
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.