Pith. sign in

REVIEW 5 major objections 5 minor 40 references

Optimizing Code Runtime Performance through Context-Aware Retrieval-Augmented Generation

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AUTOPATCH claims that adding control-flow context to LLM prompts speeds up generated code by 7.3 percent.

desk verdict A clear, reproducible RAG/ICL pipeline with a plausible speedup, but the headline number is uninterpretable without a correctness check. read the letter →

arxiv 2501.16692 v2 pith:4TKCHQHU submitted 2025-01-28 cs.SE cs.AI

Manish Acharya , Yifan Zhang , Kevin Leach , Yu Huang This is my paper · ORCID
classification cs.SEcs.AI
keywords largelanguagemodelscodeoptimizationretrieval-augmentedgenerationcontrolflowgraphin-contextlearningruntimeperformanceCFGdiffprogramanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AUTOPATCH is an in-context-learning pipeline that tries to make large language models better at optimizing code by giving them structural context: the control-flow-graph difference between an unoptimized program and its optimized counterpart, plus one retrieved example of a structurally similar code pair. The paper reports that on 116 executable C++ programs drawn from the CodeNet dataset, code generated with this context ran in 0.3815 seconds on average, compared with 0.4115 seconds for zero-shot GPT-4o generation, a 7.3 percent improvement. A naive retrieval baseline using source-code embeddings was slower than zero-shot, suggesting that how the retrieved example is chosen matters. The authors conclude that LLMs can be steered toward meaningful runtime optimizations without fine-tuning, using cheap static-analysis signals and a small corpus of before/after code pairs.

What carries the argument

The load-bearing object is the control-flow-graph diff, written $\Delta G = (\Delta S, \Delta F, \Delta C)$, which records structural block changes, flow-connectivity changes, and statement-level content changes between an unoptimized code pair and its optimized counterpart. The pipeline stores historical code pairs with CFG-derived embeddings, retrieves the single example whose CFG embedding has the highest cosine similarity to the target code, and assembles a prompt $P = (\Delta G, R_{\text{opt}}, E_r)$ containing the diff, optimization rationales, and that one retrieved pair. The empirical finding that one retrieved example is enough, while two or three add no gain, is what makes the mechanism cheap and practical.

What would settle it

Run the 116 test programs and compare the output of each optimized program with the output of the original program on the same GPT-4o-generated test cases; if the 7.3 percent speedup shrinks or reverses when only output-preserving programs are counted, the central claim fails. A second check is to vary the number of retrieved examples and the embedding source to see whether the gain is specifically tied to CFG-based retrieval.

Watch

Extended reading notes

Core claim

The central claim is that combining control-flow-graph differences with retrieval-augmented prompting improves the runtime of LLM-generated optimized code. The authors argue that human programmers optimize by comparing control flow before and after an edit, and that encoding this comparison as a structured diff lets an LLM focus on high-impact structural changes instead of token-level rewrites. On the paper's 116-program executable test set, the context-aware pipeline outperforms zero-shot generation by 7.3 percent in mean execution time, while a baseline that retrieves by raw source-code embedding is 27.3 percent slower than zero-shot. The paper also reports higher lexical similarity to ground-truth optimized code, which it treats as supporting evidence that the generated patches track genuine optimization patterns.

Load-bearing premise

The paper's speedup figure assumes that every generated program still does what the original program did; the evaluation measures only how long the new code runs, never whether its outputs match on the test cases, so faster-but-incorrect code would count as an optimization.

Editorial extensions

If this is right

  • A single structurally similar example plus a CFG diff is sufficient prompt context to shift generated code toward the ground-truth optimized version.
  • Retrieval signal quality determines whether added context helps or hurts: naive source-code retrieval performed worse than no retrieval.
  • The approach transfers, in principle, to other tasks requiring structural program understanding, such as debugging and program repair, as the paper's discussion suggests.
  • Without fine-tuning, the context-aware pipeline improves average execution time by 7.3 percent over GPT-4o zero-shot on the common executable test set.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported speedup is not yet evidence of correct optimization because the paper measures only execution time; checking that generated programs produce identical outputs on the test cases is the natural next experiment.
  • The 27.3 percent slowdown from naive embedding retrieval suggests that retrieval is a double-edged sword and that the geometric structure captured by CFG embeddings, not raw token similarity, carries the benefit.
  • Whether the CFG-diff mechanism transfers to other languages, to larger real-world codebases, or to non-runtime goals such as security patching remains a conjecture rather than a demonstrated result.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. Automated code optimization remains difficult for LLMs. The paper proposes AUTOPATCH, a retrieval-augmented in-context learning method for C++ that augments a GPT-4o prompt with (a) a representation of the control-flow-graph difference between an original and an optimized code pair, and (b) one retrieved historical example selected by CFG-based embeddings. The method is evaluated on a 1,200-pair sample of the IBM Project CodeNet dataset, with 1,000 pairs in a vector database and 200 held out for testing. The main result is a 7.3% average execution-time improvement over zero-shot GPT-4o on a filtered set of 116 executable programs, together with modest improvements in line overlap, edit distance similarity, and token overlap relative to two baselines. The paper also includes a public repository with preprocessing, CFG extraction, and retrieval code.

Significance. If the 7.3% result held under rigorous evaluation, AUTOPATCH would provide a useful and fairly simple recipe for LLM-based runtime optimization: compute structural differences, retrieve one analogous example, and prompt with both. The CFG-diff representation is more semantically motivated than raw token retrieval, and the authors deserve credit for making preprocessing and implementation code available. The main barriers to accepting the contribution are empirical: the evaluation lacks functional-equivalence checking, statistical inference, and safeguards against retrieval leakage, so the headline number is not yet established.

major comments (5)
  1. [Section IV.B, Table IV] The central claim of a 7.3% runtime improvement is uninterpretable as an optimization unless the generated programs preserve the observable behavior of the original program. Section IV.B reports only execution times; there is no comparison of program outputs, exit codes, or even successful termination on the generated test cases. The low lexical overlap reported in Table III, together with the authors' own admission in Section IV.A that lexical similarity 'does not guarantee logical correctness,' makes it plausible that some of the speedup comes from programs that behave differently or fail to run. The authors should verify, for every generated program and every test case, that compilation succeeds, the program runs to completion, and the output and exit status match the reference, and they should recompute the timing comparison on the subset of behavior-preserving programs.
  2. [Section III.D] The reduction of the test set from 200 to 116 programs is described only as excluding 'non-executable or anomalous code,' with no explicit criteria, no counts of exclusions per reason, and no statement about whether the same filtering was applied to all three generation methods. If the filter is applied after seeing which programs compile or run, it can bias the comparison toward a method whose output happens to be executable. The paper must state the exact filter, apply it identically and before evaluation, and report the number and reason for every excluded pair.
  3. [Section III.A, Table I] The split into 1,000 vector-database pairs and 200 test pairs is at the level of code pairs, but the paper does not say whether any test pair and retrieved pair share the same CodeNet problem. Because CodeNet contains multiple submissions per problem, a retrieved example from the same problem could be very close to the target or even reveal the intended optimized version, making the retrieval gain spurious. The authors should either construct a problem-disjoint split or quantify the overlap between retrieved and target problems and report results restricted to non-overlapping cases.
  4. [Table IV] The headline improvement is presented as a single mean difference (0.3815 s vs. 0.4115 s) with no variance, confidence interval, paired significance test, or effect size. With 116 programs and typically noisy runtime measurements, this difference may not be statistically reliable. The authors should report the per-program runtime distributions, standard deviations, a paired test such as the Wilcoxon signed-rank test, and the number of programs improved, unchanged, or worsened, and they should state how many repeated runs were averaged.
  5. [Section II.C] The choice of exactly one retrieved example is justified only by 'preliminary experiments' that are not reported. Since the retrieval count is a central design parameter of the method, the paper should include the preliminary data or, preferably, an ablation over k (for example, 0, 1, 2, and 3) in the main results to show the sensitivity of the 7.3% figure to this choice.
minor comments (5)
  1. [Throughout] The method name is inconsistent: 'AUTOPATCH' and 'AUTO PATCH' are used interchangeably; please standardize throughout.
  2. [Section II.B.1] The notation ∆G = (∆S, ∆F, ∆C) does not state how these components are extracted from the CFG pair or how they are serialized into the prompt; some concrete examples or pseudocode would help reproducibility.
  3. [Table II, Fig. 2] Table II reports 942 optimization-type labels for what becomes a set of 116 test programs; the paper should clarify whether the labels are per submission, how multiple labels per program are handled in the per-category averages of Fig. 2, and why the counts sum to 942.
  4. [Section III.D] The statement that 'execution testcases are generated by GPT-4o' should specify the generation prompt, the number of test cases per program, and how the same test cases are guaranteed for all methods and for repeated runs.
  5. [Data Availability Statement] No stable DOI or version is given for the repository; the authors should add a versioned identifier or commit hash so that the claimed artifacts are actually reproducible.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the 7.3% speedup is a direct measurement, not a fitted or self-referential result; unverified behavior preservation is a validity threat, not circularity.

full rationale

The paper's central claim is an execution-time comparison between AUTOPATCH (CFG-diff context plus one retrieved example in the prompt) and zero-shot GPT-4o on 116 executable C++ programs (Table IV: 0.3815 s vs 0.4115 s). No parameter is fitted to the test outputs: the retrieved example comes from the disjoint 1,000-pair vector database, and the prompt components are defined in Section II.C as P = (ΔG, R_opt, E_r), where ΔG is described as 'structural edits found in prior code pairs' and E_r as 'a training code pair.' The target's ground-truth optimized code is therefore not fed into the test-time prompt, so the reported speedup is not equivalent by construction to an input fit. The choice of exactly one retrieved example was set by preliminary experiments, which is a hyperparameter-selection concern rather than definitional circularity. The self-citations ([13]-[18], [22], [38]) support the human-attention motivation in Related Work and are not load-bearing for the runtime claim; no uniqueness theorem or pre-adopted ansatz is imported from those works. The skeptical concern about behavioral equivalence is a real evaluation-validity issue: Section IV.A concedes 'higher lexical similarity does not guarantee logical correctness,' and Section IV.B reports execution time only, so a faster-but-wrong program would be counted as an optimization. That is a correctness threat, not a circularity of the derivation chain.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The method has no fitted numerical parameters in the sense of an optimization objective, but it relies on a handful of hand-chosen design decisions and domain assumptions. The most consequential choices are the retrieved-example count, the test set filter, and the untested semantic equivalence of generated code.

free parameters (2)
  • retrieved example count k = 1
    Chosen from preliminary experiments (Section II.C); retrieving two or three examples 'did not yield additional gains'. This is a data-driven design choice, not a derived quantity.
  • test set inclusion threshold = 116 of 200 programs
    Section III.D refines the test set by excluding non-executable or anomalous code without stating criteria; the effective sample is selected rather than fixed in advance.
assumptions (6)
  • domain assumption Historical code pairs in the vector database are correct original-optimized pairs.
    Table I and Section II.B treat CodeNet pairs as groundtruth optimizations; errors in these pairs would propagate into retrieved guidance.
  • domain assumption CFG differences capture transferable optimization knowledge.
    Section II.B.1 defines delta-G as the difference between original and optimized graphs and assumes structural, flow, and content changes are the right signal for guiding code optimization.
  • domain assumption CodeBERT embeddings of CFG-enriched code provide useful similarity for retrieval.
    Section II.B.3 and Section III.D use cosine similarity on CodeBERT embeddings; the paper provides no validation that nearest neighbors are relevant.
  • domain assumption The LLM's optimized output preserves the original program's outputs.
    The evaluation uses execution time only (Section IV.B); if outputs change, the speedup is not a valid optimization.
  • domain assumption GPT-4o-generated execution test cases are representative and unbiased.
    Section III.D says execution testcases are generated by GPT-4o, but the paper does not assess their coverage or quality.
  • domain assumption Single-run execution times are stable enough for comparison.
    No repeated runs or variance statistics are reported in Table IV.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimizing Code Runtime Performance through Context-Aware Retrieval-Augmented Generation." pith.science (2026). https://pith.science/paper/4TKCHQHU

@misc{pith2026250116692,
  author       = {Pith},
  title        = {Pith review of: Optimizing Code Runtime Performance through Context-Aware Retrieval-Augmented Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4TKCHQHU}},
  note         = {Machine review of arXiv:2501.16692}
}
read the original abstract

Optimizing software performance through automated code refinement offers a promising avenue for enhancing execution speed and efficiency. Despite recent advancements in LLMs, a significant gap remains in their ability to perform in-depth program analysis. This study introduces AUTOPATCH, an in-context learning approach designed to bridge this gap by enabling LLMs to automatically generate optimized code. Inspired by how programmers learn and apply knowledge to optimize software, AUTOPATCH incorporates three key components: (1) an analogy-driven framework to align LLM optimization with human cognitive processes, (2) a unified approach that integrates historical code examples and CFG analysis for context-aware learning, and (3) an automated pipeline for generating optimized code through in-context prompting. Experimental results demonstrate that AUTOPATCH achieves a 7.3% improvement in execution efficiency over GPT-4o across common generated executable code, highlighting its potential to advance automated program runtime optimization.

Figures

Figures reproduced from arXiv: 2501.16692 by the authors.

Figure 1
Figure 1. Overview of the AUTOPATCH Workflow: AUTOPATCH mimics human cognitive processes to optimize code using historical examples and CFG analysis. ⃝1 represents the programmer learning phase, extracting insights from historical code and CFG differences. ⃝2 integrates CFG analysis and optimization patterns into a context-aware LLM. ⃝3 applies this knowledge to generate optimized code for new inputs. and transitions to ident… view at source ↗
Figure 2
Figure 2. Average execution times for different optimization types across [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 11 canonical work pages

  1. [1]

    Utilizing Deep Learning to Optimize Software Development Processes

    K. Li, A. Zhu, P. Zhao, J. Song, and J. Liu, “Utilizing deep learning to optimize software development processes,” arXiv preprint arXiv:2404.13630, 2024

    work page Pith review arXiv 2024

  2. [2]

    Learning performance-improving code edits,

    A. Shypula, A. Madaan, Y . Zeng, U. Alon, J. Gardner, M. Hashemi, G. Neubig, P. Ranganathan, O. Bastani, and A. Yazdanbakhsh, “Learning performance-improving code edits,” arXiv preprint arXiv:2302.07867 , 2023

    arXiv 2023

  3. [3]

    Search-based llms for code optimization,

    S. Gao, C. Gao, W. Gu, and M. Lyu, “Search-based llms for code optimization,” in 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, 2024, pp. 254– 266

    work page 2025

  4. [4]

    Jarvix: A llm no code platform for tab- ular data analysis and optimization,

    S.-C. Liu, S. Wang, W. Lin, C.-W. Hsiung, Y .-C. Hsieh, Y .-P. Cheng, S.- H. Luo, T. Chang, and J. Zhang, “Jarvix: A llm no code platform for tab- ular data analysis and optimization,” arXiv preprint arXiv:2312.02213 , 2023

    arXiv 2023

  5. [5]

    How efficient is llm-generated code? a rigorous & high-standard benchmark,

    R. Qiu, W. W. Zeng, H. Tong, J. Ezick, and C. Lott, “How efficient is llm-generated code? a rigorous & high-standard benchmark,” arXiv preprint arXiv:2406.06647, 2024

    arXiv 2024

  6. [6]

    Llm- assisted content analysis: Using large language models to support deductive coding,

    R. Chew, J. Bollenbacher, M. Wenger, J. Speer, and A. Kim, “Llm- assisted content analysis: Using large language models to support deductive coding,” arXiv preprint arXiv:2306.14924 , 2023

    arXiv 2023

  7. [7]

    Static code analysis in the ai era: An in-depth exploration of the concept, function, and potential of intelligent code analysis agents,

    G. Fan, X. Xie, X. Zheng, Y . Liang, and P. Di, “Static code analysis in the ai era: An in-depth exploration of the concept, function, and potential of intelligent code analysis agents,” arXiv preprint arXiv:2310.08837 , 2023

    arXiv

  8. [8]

    Frustrated with code quality issues? llms can help!

    N. Wadhwa, J. Pradhan, A. Sonwane, S. P. Sahu, N. Natarajan, A. Kanade, S. Parthasarathy, and S. Rajamani, “Frustrated with code quality issues? llms can help!” arXiv preprint arXiv:2309.12938 , 2023

    arXiv 2023

Show all 40 references
  1. [9]

    Using an llm to help with code understanding,

    D. Nam, A. Macvean, V . Hellendoorn, B. Vasilescu, and B. Myers, “Using an llm to help with code understanding,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , 2024, pp. 1–13

    2024

  2. [10]

    Codenet: A large- scale ai for code dataset for learning a diversity of coding tasks,

    R. Puri, D. S. Kung, G. Janssen, W. Zhang, G. Domeniconi, V . Zolotov, J. Dolby, J. Chen, M. Choudhury, L. Decker et al., “Codenet: A large- scale ai for code dataset for learning a diversity of coding tasks,” arXiv preprint arXiv:2105.12655, 2021

    2021 arXiv

  3. [11]

    Gpt-4o system card,

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford et al., “Gpt-4o system card,” arXiv preprint arXiv:2410.21276 , 2024

    2024 arXiv

  4. [12]

    Codebert: A pre-trained model for programming and natural languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang et al., “Codebert: A pre-trained model for programming and natural languages,” arXiv preprint arXiv:2002.08155 , 2020

    2002 arXiv

  5. [13]

    Do machines and humans focus on similar code? exploring explainability of large language models in code summarization,

    J. Li, Y . Zhang, Z. Karas, C. McMillan, K. Leach, and Y . Huang, “Do machines and humans focus on similar code? exploring explainability of large language models in code summarization,” in Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension, 202...

    2024

  6. [14]

    Modeling programmer attention as scanpath prediction,

    A. Bansal, C.-Y . Su, Z. Karas, Y . Zhang, Y . Huang, T. J.-J. Li, and C. McMillan, “Modeling programmer attention as scanpath prediction,” in 2023 38th IEEE/ACM International Conference on Automated Soft- ware Engineering (ASE) . IEEE, 2023, pp. 1732–1736

    2023

  7. [15]

    Eyetrans: Merging human and machine attention for neural code summarization,

    Y . Zhang, J. Li, Z. Karas, A. Bansal, T. J.-J. Li, C. McMillan, K. Leach, and Y . Huang, “Eyetrans: Merging human and machine attention for neural code summarization,” Proceedings of the ACM on Software Engineering, vol. 1, no. FSE, pp. 115–136, 2024

    2024

  8. [16]

    A tale of two comprehensions? analyzing student programmer attention during code summarization,

    Z. Karas, A. Bansal, Y . Zhang, T. Li, C. McMillan, and Y . Huang, “A tale of two comprehensions? analyzing student programmer attention during code summarization,” ACM Transactions on Software Engineering and Methodology, 2024

    2024

  9. [17]

    Pre-training representations of binary code using contrastive learning,

    Y . Zhang, C. Huang, K. Cao, Y . Zhang, S. T. Andersen, H. Shao, K. Leach, and Y . Huang, “Pre-training representations of binary code using contrastive learning,” arXiv preprint arXiv:2210.05102 , 2022

    2022

  10. [18]

    Leveraging artificial intelligence on binary code comprehen- sion,

    Y . Zhang, “Leveraging artificial intelligence on binary code comprehen- sion,” in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering , 2022, pp. 1–3

    2022

  11. [19]

    Rag-enhanced commit message generation,

    L. Zhang, H. Zhang, C. Wang, and P. Liang, “Rag-enhanced commit message generation,” arXiv preprint arXiv:2406.05514 , 2024

    2024 arXiv

  12. [20]

    Prompt-based code completion via multi-retrieval augmented genera- tion,

    H. Tan, Q. Luo, L. Jiang, Z. Zhan, J. Li, H. Zhang, and Y . Zhang, “Prompt-based code completion via multi-retrieval augmented genera- tion,” arXiv preprint arXiv:2405.07530 , 2024

    2024 arXiv

  13. [21]

    Evaluating retrieval-augmented generation (rag) tech- niques in enhancing lms for coding tasks,

    E. Mozharovskii, “Evaluating retrieval-augmented generation (rag) tech- niques in enhancing lms for coding tasks,” Universum: tekhnicheskie nauki: elektron. nauchn. zhurn , no. 6, p. 123, 2024

    2024

  14. [22]

    Malmixer: Few-shot malware classification with retrieval-augmented semi-supervised learning,

    E. Li, Y . Zhang, Y . Huang, and K. Leach, “Malmixer: Few-shot malware classification with retrieval-augmented semi-supervised learning,” arXiv preprint arXiv:2409.13213, 2024

    2024 arXiv

  15. [23]

    Enhancing code translation in language models with few-shot learning via retrieval-augmented generation,

    M. Bhattarai, J. E. Santos, S. Jones, A. Biswas, B. Alexandrov, and D. O’Malley, “Enhancing code translation in language models with few-shot learning via retrieval-augmented generation,” arXiv preprint arXiv:2407.19619, 2024

    2024 arXiv

  16. [24]

    Arks: Active retrieval in knowledge soup for code generation,

    H. Su, S. Jiang, Y . Lai, H. Wu, B. Shi, C. Liu, Q. Liu, and T. Yu, “Arks: Active retrieval in knowledge soup for code generation,” arXiv preprint arXiv:2402.12317, 2024

    2024 arXiv

  17. [25]

    Coderag-bench: Can retrieval augment code generation?

    Z. Z. Wang, A. Asai, X. V . Yu, F. F. Xu, Y . Xie, G. Neubig, and D. Fried, “Coderag-bench: Can retrieval augment code generation?” arXiv preprint arXiv:2406.14497 , 2024

    2024 arXiv

  18. [26]

    Llm-based and retrieval-augmented control code generation,

    H. Koziolek, S. Gr ¨uner, R. Hark, V . Ashiwal, S. Linsbauer, and N. Es- kandani, “Llm-based and retrieval-augmented control code generation,” in Proceedings of the 1st International Workshop on Large Language Models for Code , 2024, pp. 22–29

    2024

  19. [27]

    A survey on rag meeting llms: Towards retrieval-augmented large language models,

    W. Fan, Y . Ding, L. Ning, S. Wang, H. Li, D. Yin, T.-S. Chua, and Q. Li, “A survey on rag meeting llms: Towards retrieval-augmented large language models,” in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2024, pp. 6491–6501

    2024

  20. [28]

    Retrieval-augmented generation for ai-generated content: A survey,

    P. Zhao, H. Zhang, Q. Yu, Z. Wang, Y . Geng, F. Fu, L. Yang, W. Zhang, and B. Cui, “Retrieval-augmented generation for ai-generated content: A survey,” arXiv preprint arXiv:2402.19473 , 2024

    2024 arXiv

  21. [29]

    What makes good examples for visual in-context learning?

    Y . Zhang, K. Zhou, and Z. Liu, “What makes good examples for visual in-context learning?” Advances in Neural Information Processing Systems, vol. 36, pp. 17 773–17 794, 2023

    2023

  22. [30]

    The learnability of in-context learning,

    N. Wies, Y . Levine, and A. Shashua, “The learnability of in-context learning,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  23. [31]

    Compositional exemplars for in-context learning,

    J. Ye, Z. Wu, J. Feng, T. Yu, and L. Kong, “Compositional exemplars for in-context learning,” in International Conference on Machine Learning . PMLR, 2023, pp. 39 818–39 833

    2023

  24. [32]

    Finding support examples for in-context learning,

    X. Li and X. Qiu, “Finding support examples for in-context learning,” arXiv preprint arXiv:2302.13539 , 2023

    2023 arXiv

  25. [33]

    Democraft: Using in-context learning to improve code generation in large language models,

    N. J. Kapu and M. Sreejith, “Democraft: Using in-context learning to improve code generation in large language models,” arXiv preprint arXiv:2411.00865, 2024

    2024 arXiv

  26. [34]

    In-context unlearning: Lan- guage models as few shot unlearners,

    M. Pawelczyk, S. Neel, and H. Lakkaraju, “In-context unlearning: Lan- guage models as few shot unlearners,” arXiv preprint arXiv:2310.07579, 2023

    2023 arXiv

  27. [35]

    Evaluating the effectiveness of deep learning models for foundational program analysis tasks,

    Q. Chen, C. Yu, R. Liu, C. Zhang, Y . Wang, K. Wang, T. Su, and L. Wang, “Evaluating the effectiveness of deep learning models for foundational program analysis tasks,” Proceedings of the ACM on Programming Languages, vol. 8, no. OOPSLA1, pp. 500–528, 2024

    2024

  28. [36]

    Enchanting program specification synthesis by large language models using static analysis and program verification,

    C. Wen, J. Cao, J. Su, Z. Xu, S. Qin, M. He, H. Li, S.-C. Cheung, and C. Tian, “Enchanting program specification synthesis by large language models using static analysis and program verification,” in International Conference on Computer Aided Verification . Springer, 2024, pp....

    2024

  29. [37]

    Mea- suring the runtime performance of code produced with github copilot,

    D. Erhabor, S. Udayashankar, M. Nagappan, and S. Al-Kiswany, “Mea- suring the runtime performance of code produced with github copilot,” arXiv preprint arXiv:2305.06439 , 2023

    2023 arXiv

  30. [38]

    Detecting code comment inconsistencies using llm and program analysis,

    Y . Zhang, “Detecting code comment inconsistencies using llm and program analysis,” in Companion Proceedings of the 32nd ACM Inter- national Conference on the Foundations of Software Engineering , 2024, pp. 683–685

    2024

  31. [39]

    Codeplan: Repository-level coding using llms and planning,

    R. Bairi, A. Sonwane, A. Kanade, A. Iyer, S. Parthasarathy, S. Rajamani, B. Ashok, and S. Shet, “Codeplan: Repository-level coding using llms and planning,” Proceedings of the ACM on Software Engineering, vol. 1, no. FSE, pp. 675–698, 2024

    2024

  32. [40]

    Enhancing fault localization through ordered code analysis with llm agents and self- reflection,

    M. N. Rafi, D. J. Kim, T.-H. Chen, and S. Wang, “Enhancing fault localization through ordered code analysis with llm agents and self- reflection,” arXiv preprint arXiv:2409.13642 , 2024

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.