Pith. sign in

REVIEW 4 major objections 7 minor 74 references

Uncertainty in retrieved evidence should drive what code models keep, generate, and repair—not relevance alone.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 20:16 UTC pith:OXCZZWJM

load-bearing objection Careful multi-source repo RAG paper whose main GPT gain is real but fully explained by verify/repair; the factorial interactions and API refinement are the cleaner contributions. the 4 major comments →

arxiv 2607.24884 v1 pith:OXCZZWJM submitted 2026-07-27 cs.SE cs.AIcs.CLcs.LG

Beyond "What to Retrieve": Uncertainty in Retrieval-Augmented Code Generation

Chandan Kumar Sah , Xiaoli Lian , Li Zhang This is my paper
classification cs.SE cs.AIcs.CLcs.LG
keywords repository-level code generationretrieval-augmented generationuncertainty-aware retrievalheterogeneous evidenceAPI knowledgeexecutable verificationcode repairmulti-source fusion
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Repository-level code generation depends on mixed evidence—similar code, local context, and project APIs—that can help or hurt depending on what else is present and which model is used. Most retrieval-augmented systems still optimize mainly for relevance and paste the top hits into the prompt. This paper argues that the missing control is source-specific uncertainty: how confident we should be that each piece of evidence is relevant, complete, and compatible with the target. OpenCoder estimates that uncertainty, filters and ranks multi-source evidence with it, conditions generation on an uncertainty trace, then verifies candidates and repairs failures. A full factorial study finds no fixed ranking of evidence types—only interactions that change with the accompanying sources and the LLM. On 32 matched executable tasks, the approach lifts GPT selected-output correctness from 56.25% to 78.13% over plain multi-source RAG, though it ties a matched verify-and-repair control and the Gemini gain is not statistically supported. Target-aware API refinement also markedly improves recovered API sets. The practical claim is that uncertainty is useful when treated as an operational dial over retrieval, generation, and repair—not as a post-hoc confidence number.

Core claim

The paper establishes that retrieval utility for repository-level code generation is interaction-dependent rather than additive by source type, and that modeling source-specific uncertainty over API knowledge, repository context, and similar code—then using it to filter, fuse, generate, verify, and repair—can raise selected-output correctness over plain multi-source RAG (GPT: 56.25% to 78.13% on 32 RepoExec-inline tasks), while benefits remain backend- and evidence-completeness-dependent and match a verification-and-repair control under the same budget.

What carries the argument

OpenCoder’s uncertainty-aware pipeline: per-source uncertainty scores combined with relevance and predicted source utility into a fused ranking score γ, top-70% retention and shared-budget fusion, generation conditioned on a source-wise uncertainty trace, then validation-driven selection with at most two repair rounds on the mode candidate when all raw candidates fail.

Load-bearing premise

Estimated uncertainty is reliable enough to drop lower-confidence evidence without discarding necessary alternatives when the repository index is incomplete.

What would settle it

On a larger matched set of executable repository tasks with incomplete context, check whether uncertainty-aware filtering still beats or even matches plain multi-source RAG plus the same verify/repair budget; if filtering consistently hurts selected-output correctness whenever coverage is partial, the control-signal claim fails outside complete indexes.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Repository RAG systems should score and filter API, context, and similar-code evidence jointly under uncertainty rather than by a fixed source ranking.
  • Selected-output reliability gains can come from coupling uncertainty-aware fusion with executable validation and limited repair, not from retrieval relevance alone.
  • API grounding improves when retrieval is refined against the target intent, with uncertainty used to flag false-positive APIs.
  • Reported benefits must be conditioned on LLM backend and repository-evidence completeness; partial indexes can make aggressive filtering harmful.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If interaction-dependent utility is general, static “always retrieve X” policies in coding agents are mis-specified; policies should condition on co-retrieved sources and model identity.
  • The partial-context failure mode suggests a needed extension: uncertainty estimates that detect missing coverage and loosen filters or trigger broader retrieval instead of only down-weighting items.
  • Calibration differing sharply across backends implies production systems may need backend-specific uncertainty maps rather than one shared threshold.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces OpenCoder, an uncertainty-aware pipeline for retrieval-augmented repository-level code generation. It estimates source- and item-level uncertainty over three evidence types (project APIs, repository context, similar code), uses these scores to filter and fuse evidence (Eqs. 4–9), and couples generation with executable verification and up to two repair rounds (Eq. 10). A 2³ factorial study over evidence sources on ten ExecRepoBench tasks finds no significant additive source effects but two Holm-significant interactions, which the authors take as motivation for uncertainty-aware integration. On a pre-audited 32-task RepoExec-inline set with a frozen five-candidate protocol, OpenCoder improves GPT selected-output correctness from 56.25% to 78.13% over Baseline RAG (CI [6.25, 40.63], nominal McNemar p=.039) but ties a matched RAG+Verify/Repair control (78.13%, Δ=0); Gemini differences are not statistically supported; under partial-context stress testing the control dominates OpenCoder at every metric. Target-aware API refinement improves macro API F1 from 43.4% to 64.8% (GPT) and 59.1% (Gemini). The paper concludes that uncertainty is an actionable control signal for retrieval, verification, and repair.

Significance. If the claims are recalibrated to the evidence, this is a useful and unusually honest contribution. The factorial demonstration that retrieval utility is interaction-dependent rather than additively rankable is a genuine empirical finding for repository-level RAG, the frozen pre-output task audit and matched five-candidate budget set a good methodological standard, the explicit isolation of Baseline RAG vs. RAG+Verify/Repair is exactly the right control design, and the API-refinement results (macro F1 43.4→64.8 GPT, →59.1 Gemini) are solid and practically relevant. The public code release supports reproducibility. The significance is currently capped by the gap between the framing (uncertainty as the operative mechanism) and the evidence (a tie with the verify/repair control and an inverse GPT AUROC); narrowing that gap would make the paper a clean negative/boundary-condition result with a validated API-grounding component, which the field needs.

major comments (4)
  1. [RQ3 / Table 3; Abstract] The headline framing — uncertainty as 'an actionable control signal' for retrieval/verification/repair — is not supported by the paper's own primary comparison. On the 32-task RepoExec-inline set with GPT, OpenCoder ties RAG+Verify/Repair exactly at 78.13% selected-output correctness (Δ=0, CI [-9.38, 9.38]), with nearly identical Pass@1/3/5 (61.88/72.50/78.13 vs 61.88/74.38/78.13). On the partial-context ExecRepoBench stress test the control beats OpenCoder at every metric for both backends. RQ2 itself attributes 'most of the GPT improvement to verification-based candidate selection.' As written, no experiment isolates a positive marginal contribution of the uncertainty machinery — γ_m scoring/filtering (Eq. 6, α=0.5, top-70% retention), the fused ranking (Eq. 7), or the uncertainty-trace conditioning (Eq. 8–9) — over plain concatenation under the same verify/repair budget. The claim is
  2. [RQ2 (uncertainty quantification); Eq. (6)] The reported failure-detection AUROC for GPT is 0.167 — far below chance — for the same backend on which the headline improvement is reported. The authors note an 'inverse association under the evaluated score orientation,' but this needs more than a parenthetical: if the uncertainty score anti-predicts failure for GPT, then the max{0, 1 − α·u_m(e)} penalty in Eq. 6 is, for GPT, systematically down-weighting the better evidence (or the score orientation is mis-specified, in which case the calibration claims elsewhere are suspect). Either way, the mechanism said to drive filtering is not functioning as described on the GPT backend. The paper should (i) clarify the score orientation and verify the sign convention, and (ii) report a direct comparison of γ_m ranking vs. relevance-only ranking (r_m alone) on evidence quality or downstream correctness, to show the uncertainty term is doing mea
  3. [RQ1 / Table 2, Figure 2] Finding 1 and the abstract's claim of 'significant cross-source interactions' rest on n=10 ExecRepoBench tasks (160 task–condition runs, but the unit of inference is the task). The two Holm-surviving effects — a 33.3pp Pass@1 interaction for GPT context×similar-code and a 0.059 uncertainty reduction for Gemini API×similar-code, both p_Holm=.032 — are single-task-scale effects on a 10-task set and are fragile to the choice of adapted tasks. Moreover, these ten tasks were 'adapted' from a statement/expression-masking completion benchmark to function-level generation, which raises construct questions about what the factorial design measures. The interaction-dependence conclusion is plausible and the design is appropriate in principle, but the inferential weight placed on it (it 'empirically motivates' the whole integration scheme) exceeds what 10 tasks support. Replication on a larger task
  4. [§OpenCoder, Eqs. (6)–(7); RQ3 stress test] The framework introduces several free choices that are load-bearing for the filtering behavior — α=0.5 in Eq. (6), the top-70% per-source retention, the fused budget K=10 in Eq. (7), the eight-candidate per-source cap, and the 0.4/0.4/0.2 generation-uncertainty mixture weights — all described as fixed a priori with no sensitivity analysis. Given that the partial-context stress test shows filtering actively suppressing useful evidence (OpenCoder Pass@1 drops to 28.00 vs the control's 62.00 for GPT), the retention percentile in particular is doing real harm in at least one regime. A small sensitivity sweep over retention percentile and α on the 32-task set would substantially strengthen the paper's practical guidance and is cheap within the existing frozen protocol.
minor comments (7)
  1. [Table 3 / Metrics] The 'Sel.' (selected-output correctness) column is used as the primary metric but never given a formal definition. It appears to be correctness of the single output chosen by the Eq. (10) procedure; please state this explicitly and clarify how it differs from effective Pass@1 as defined in RQ2.
  2. [Experimental Setup] RQ1–RQ2 use temperature 0.2 while RQ3–RQ4 use 0.7 with five candidates. This is defensible (diagnostic vs. end-to-end), but it means the RQ2 component-attribution results (e.g., 'most of the GPT improvement' from verification-based selection) are not directly transferable to the RQ3 regime; a sentence acknowledging this would help.
  3. [Metrics; RQ3] Eq. (11): n and c are used before being defined ('we generate n candidate implementations, of which c pass'); reorder. Also W/L/T = 8/1/23 in RQ3 is used without expansion; define on first use.
  4. [Problem Formulation] Notation for uncertainty alternates between u_m(e|q) (Eq. 4) and u_m(e) (Eq. 6); please unify. Similarly e_{S,m} in Eq. (7) is undefined typography — presumably the per-source retained set for intent s and source m.
  5. [RQ2] ECE worsens for GPT from 0.162 to 0.222 across the pipeline while improving for Gemini; this is reported but not discussed alongside the AUROC anomaly. Both belong in one coherent paragraph on GPT-side calibration behavior.
  6. [Figure 3] Figure 3 shows effective vs. raw-sample Pass@1 'as uncertainty-aware components are introduced,' but the six configurations are never enumerated in the caption or text; the reader cannot tell which component is added at each step.
  7. [Table 1; References] Table 1 is a useful positioning artifact, but several cited systems (RepoMinCoder, SRACG, Incoherence, AllianceCoder/Gu et al. 2026) are 2024–2026 venues with arXiv-only or in-press entries; please verify venue/status at camera-ready and add the CoderEval API-set evaluation protocol details (how ground-truth API sets were derived) to the supplement pointer.

Circularity Check

0 steps flagged

No circularity: empirical systems paper evaluated against external executable benchmarks; uncertainty is a control input, not the success criterion.

full rationale

OpenCoder’s claims are empirical, not first-principles derivations. Source-specific uncertainty u_m, the fused score γ_m (Eq. 6), top-70% retention, and the uncertainty trace u(q) are methodological design choices that condition retrieval and generation; functional correctness is measured by external executable tests (Pass@k, selected-output correctness) on pre-audited RepoExec-inline, ExecRepoBench, and CoderEval tasks, with held-out splits for API AUROC/ECE. Success is never defined in terms of the paper’s own uncertainty scores, so there is no self-definitional loop, no fitted parameter renamed as a prediction of a closely related quantity, and no load-bearing uniqueness theorem or ansatz imported via overlapping-author citation. The matched RAG+Verify/Repair tie and the partial-context degradation are contribution and boundary-condition issues, not circular reductions. The derivation chain is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 3 invented entities

The work is empirical systems/SE research. Load-bearing background includes standard RAG and executable Pass@k evaluation; the paper adds hand-chosen fusion/uncertainty hyperparameters and the modeling choice that heterogeneous repo evidence is well captured by three sources whose item-level uncertainty can be estimated and used as a filter. No new physical entities; invented constructs are methodological (uncertainty scores, fusion rule, pipeline stages).

free parameters (6)
  • uncertainty penalty α in γ_m = 0.5
    Scales how much evidence uncertainty down-weights relevance×utility; set to 0.5 without a reported sensitivity sweep on the 32-task main evaluation.
  • per-source retention percentile = 70%
    Keeps top 70% of candidates by γ_m before fuse/dedup; directly implements the filter whose failure mode appears under partial context.
  • fused context budget K = 10
    Shared prompt budget after merge/dedup/rank; fixed at 10 items for all methods in the matched protocol.
  • generation uncertainty mixture weights = 0.4 / 0.4 / 0.2
    Token entropy, self-consistency, and semantic variance combined with fixed weights across all benchmarks/backends.
  • max repair rounds and candidate count = 2 repairs; n=5 candidates
    Protocol knobs that dominate effective Pass@1 in RQ2; two repair rounds and five candidates are fixed, not learned.
  • per-source retriever candidate cap = 8
    Each source retriever returns up to eight candidates before uncertainty filtering/fusion.
axioms (5)
  • domain assumption Functional correctness is adequately measured by the benchmarks’ executable tests (Pass@k / selected-output pass) under the frozen harness.
    Central dependent variable for RQ1–RQ3; standard in code generation but incomplete for security, licensing, and untested behaviors (noted in Limitations).
  • domain assumption Repository knowledge for generation is usefully partitioned as C = A ∪ X ∪ S (APIs, context, similar code).
    Eq. (1) and problem formulation; follows prior repo-level settings (Wu et al.; Gu et al.) and structures all retrieval and factorial analysis.
  • ad hoc to paper Source- and item-level uncertainty u_m(e|q) ∈ [0,1] can be estimated well enough from the paper’s signals to improve filtering and control.
    Core methodological premise (Eqs. 4–9); operational use continues despite GPT failure-detection AUROC 0.167 and backend-dependent ECE.
  • domain assumption Task inclusion for the expanded 32-task RepoExec-inline set can be fixed by a pre-output audit (dependency completeness, passing reference tests, harness compatibility) without selection bias.
    Experimental Setup / RQ3; 18 of 25 remaining candidates retained before inspecting method outputs—credible if audit truly blinded, still a small curated slice.
  • standard math Paired comparisons with bootstrap CIs and exact McNemar tests on binary task outcomes support the stated backend-specific conclusions at the reported sample sizes.
    Statistical Analysis section; appropriate methods, but low power and unadjusted headline p limit strength of inference.
invented entities (3)
  • OpenCoder uncertainty-aware pipeline (Phases I–V) no independent evidence
    purpose: Unify uncertainty profiling, query decomposition, multi-source filter/fuse, uncertainty-conditioned generation, and verify/repair.
    Primary proposed system; evaluated empirically against RAG baselines rather than derived from theory.
  • Source-wise uncertainty trace u(q) = [ū_A, ū_X, ū_S] no independent evidence
    purpose: Condition generation on aggregate per-source uncertainty alongside selected evidence E*(q).
    Eqs. (8)–(9); internal control signal without external physical referent.
  • Combined evidence score γ_m(e|s) = r_m · w_m(s) · max{0, 1 − α u_m(e)} no independent evidence
    purpose: Rank/filter items using relevance, predicted source utility, and uncertainty under a shared budget.
    Eq. (6); hand-parameterized fusion rule central to the method’s difference from Baseline RAG.

pith-pipeline@v1.2.0-grok45-kimik3 · 17577 in / 4418 out tokens · 85323 ms · 2026-07-31T20:16:26.574402+00:00 · methodology

0 comments
read the original abstract

Repository-level code generation relies on heterogeneous evidence whose relevance, compatibility, and completeness are inherently uncertain. Similar-code examples, repository context, and project-specific APIs may provide complementary information, but can also introduce noisy, redundant, or conflicting signals. Existing retrieval-augmented approaches primarily optimize retrieval relevance without explicitly modeling how uncertainty in retrieved evidence affects downstream generation. We introduce OpenCoder, an uncertainty-aware framework that estimates source-specific uncertainty, uses it to filter and rank heterogeneous evidence, and guides generation, verification, and repair. A factorial analysis over API knowledge, repository context, and similar-code evidence reveals no universal additive source ranking; instead, significant cross-source interactions depend on the accompanying evidence and LLM backend. On an expanded 32-task RepoExec-inline evaluation, OpenCoder improves GPT selected-output correctness over Baseline RAG from 56.25\% to 78.13\%. However, it matches a verification-and-repair control, and the corresponding Gemini improvement is not statistically supported, indicating backend-dependent benefits. Target-aware API refinement also substantially improves API-set retrieval. These findings support treating uncertainty as an actionable control signal for repository-level retrieval, verification, and repair.

Figures

Figures reproduced from arXiv: 2607.24884 by Chandan Kumar Sah, Li Zhang, Xiaoli Lian.

Figure 1
Figure 1. Figure 1: Overview of OpenCoder’s uncertainty-aware pipeline for repository indexing, evidence retrieval and fusion, code generation, verification, and repair. where u(q) = [¯uA(q), u¯X (q), u¯S (q)] (9) is the source-wise uncertainty trace. Given five candidates Y(q) = (y1, . . . , y5), OpenCoder validates them in gener￾ation order and returns the first passing candidate. If none passes, the earliest candidate ymod… view at source ↗
Figure 2
Figure 2. Figure 2: Factorial effects of API knowledge, repository context, and similar-code evidence on aggregate uncertainty and [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: API-set retrieval quality (left) and held-out false [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

74 extracted references · 15 linked inside Pith

  1. [1]

    Liao, Dianshu and Pan, Shidong and Sun, Xiaoyu and Ren, Xiaoxue and Huang, Qing and Xing, Zhenchang and Jin, Huan and Li, Qinying , journal =

  2. [2]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages =

    RepoCoder: Repository-Level Code Completion through Iterative Retrieval and Generation , author =. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages =. 2023 , doi =

    2023

  3. [3]

    Proceedings of the 41st International Conference on Machine Learning , series =

    Repoformer: Selective Retrieval for Repository-Level Code Completion , author =. Proceedings of the 41st International Conference on Machine Learning , series =

  4. [4]

    Li, Yifan and Shi, Ensheng and Zheng, Dewu and Duan, Kefeng and Chen, Jiachi and Wang, Yanlin , booktitle =

  5. [5]

    Wang, Yanlin and Wang, Yanli and Guo, Daya and Chen, Jiachi and Zhang, Ruikai and Ma, Yuchi and Zheng, Zibin , journal =

  6. [6]

    Deng, Ken and Liu, Jiaheng and Zhu, He and Liu, Congnan and Li, Jingxin and Wang, Jiakai and Zhao, Peng and Zhang, Chenchen and Wu, Yanan and Yin, Xueqiao and Zhang, Yuanxing and Su, Wenbo and Xiang, Bangyu and Ge, Tiezheng and Zheng, Bo , journal =

  7. [7]

    Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering , year =

    GraphCoder: Enhancing Repository-Level Code Completion via Coarse-to-Fine Retrieval Based on Code Context Graph , author =. Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering , year =

  8. [8]

    Liang, Ming and Xie, Xiaoheng and Zhang, Gehao and Zheng, Xunjin and Di, Peng and Jiang, Wei and Chen, Hongwei and Wang, Chengpeng and Fan, Gang , journal =

  9. [9]

    Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE) , year =

    What to Retrieve for Effective Retrieval-Augmented Code Generation? An Empirical Study and Beyond , author =. Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE) , year =

    2026

  10. [10]

    Asai, Akari and Wu, Zeqiu and Wang, Yizhong and Sil, Avirup and Hajishirzi, Hannaneh , booktitle =

  11. [13]

    Journal of Information Processing Systems , volume =

    A Survey of Automatic Code Generation from Natural Language , author =. Journal of Information Processing Systems , volume =

  12. [14]

    Science , volume =

    Competition-Level Code Generation with AlphaCode , author =. Science , volume =

  13. [15]

    Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , pages =

    IntelliCode Compose: Code Generation Using Transformer , author =. Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , pages =

  14. [20]

    arXiv preprint arXiv:2312.10997 , year =

    Retrieval-Augmented Generation for Large Language Models: A Survey , author =. arXiv preprint arXiv:2312.10997 , year =

    Pith/arXiv arXiv

  15. [21]

    Guo, Daya and Lu, Shuai and Duan, Nan and Wang, Yanlin and Zhou, Ming and Yin, Jian , journal =

  16. [22]

    Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , year =

    CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models , author =. Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , year =

  17. [24]

    Yang, Jian and Zhang, Jiajun and Yang, Jiaxi and Jin, Ke and Zhang, Lei and Peng, Qiyao and Deng, Ken and Miao, Yibo and Liu, Tianyu and Cui, Zeyu and Hui, Binyuan and Lin, Junyang , journal =

  18. [25]

    Proceedings of the 2018 conference on empirical methods in natural language processing , pages=

    Retrieval-based neural code generation , author=. Proceedings of the 2018 conference on empirical methods in natural language processing , pages=

    2018

  19. [26]

    Parvez, Md Rizwan and Ahmad, Wasi Uddin and Chakraborty, Saikat and Ray, Baishakhi and Chang, Kai-Wei , journal =

  20. [27]

    Lu, Shuai and Duan, Nan and Han, Hojae and Guo, Daya and Hwang, Seung-won and Svyatkovskiy, Alexey , journal =

  21. [28]

    and Wang, Zhiruo and Jiang, Zhengbao and Neubig, Graham , journal =

    Zhou, Shuyan and Alon, Uri and Xu, Frank F. and Wang, Zhiruo and Jiang, Zhengbao and Neubig, Graham , journal =

  22. [29]

    Knowledge-Based Systems , volume=

    Private-library-oriented code generation with large language models , author=. Knowledge-Based Systems , volume=. 2025 , publisher=

    2025

  23. [30]

    Eghbali, Aryaz and Pradel, Michael , journal =

  24. [31]

    Findings of the Association for Computational Linguistics: NAACL 2025 , pages=

    Probing-rag: Self-probing to guide language models in selective document retrieval , author=. Findings of the Association for Computational Linguistics: NAACL 2025 , pages=

    2025

  25. [32]

    Proceedings of the 34th International Conference on Machine Learning , pages =

    On Calibration of Modern Neural Networks , author =. Proceedings of the 34th International Conference on Machine Learning , pages =

  26. [33]

    Nature , volume =

    Detecting Hallucinations in Large Language Models Using Semantic Entropy , author =. Nature , volume =. 2024 , doi =

    2024

  27. [34]

    Proceedings of the 41st International Conference on Machine Learning , series =

    Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling , author =. Proceedings of the 41st International Conference on Machine Learning , series =

  28. [35]

    Proceedings of the 41st International Conference on Machine Learning , series =

    Thermometer: Towards Universal Calibration for Large Language Models , author =. Proceedings of the 41st International Conference on Machine Learning , series =

  29. [36]

    Proceedings of the 41st International Conference on Machine Learning , series =

    Multicalibration for Confidence Scoring in Large Language Models , author =. Proceedings of the 41st International Conference on Machine Learning , series =

  30. [37]

    Ni, Ansong and Iyer, Srini and Radev, Dragomir and Stoyanov, Veselin and Yih, Wen-Tau and Wang, Sida and Lin, Xi Victoria , booktitle =

  31. [38]

    Chen, Bei and Zhang, Fengji and Nguyen, Anh and Zan, Daoguang and Lin, Zeqi and Lou, Jian-Guang and Chen, Weizhu , booktitle =

  32. [39]

    The Twelfth International Conference on Learning Representations , year =

    Teaching Large Language Models to Self-Debug , author =. The Twelfth International Conference on Learning Representations , year =

  33. [40]

    Le, Hung and Wang, Yue and Gotmare, Akhilesh Deepak and Savarese, Silvio and Hoi, Steven Chu Hong , booktitle =

  34. [41]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume =

    Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation , author =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =. 2024 , doi =

    2024

  35. [42]

    Findings of the Association for Computational Linguistics: EMNLP 2022 , pages =

    When Language Model Meets Private Library , author =. Findings of the Association for Computational Linguistics: EMNLP 2022 , pages =. 2022 , doi =

    2022

  36. [43]

    Advances in Neural Information Processing Systems , volume=

    Execution guided line-by-line code generation , author=. Advances in Neural Information Processing Systems , volume=

  37. [44]

    2026 , doi =

    Wang, Mengzhen and Ma, Shukai and Gong, Songwen and Wang, Jiexin and Chen, Ruolin and Cao, Liuwen and Cai, Yi , booktitle =. 2026 , doi =

    2026

  38. [45]

    Incoherence as Oracle-less Measure of Error in

    Valentin, Thomas Jean-Michel and Madadi, Ardi and Sapia, Gaetano and B. Incoherence as Oracle-less Measure of Error in. Proceedings of the AAAI Conference on Artificial Intelligence , volume =. 2026 , doi =

    2026

  39. [46]

    Asai, A.; Wu, Z.; Wang, Y.; Sil, A.; and Hajishirzi, H. 2024. Self-RAG : Learning to Retrieve, Generate, and Critique through Self-Reflection. In The Twelfth International Conference on Learning Representations

    2024

  40. [47]

    Baek, I.; Chang, H.; Kim, B.; Lee, J.; and Lee, H. 2025. Probing-rag: Self-probing to guide language models in selective document retrieval. 3287--3304

    2025

  41. [48]

    Chen, B.; Zhang, F.; Nguyen, A.; Zan, D.; Lin, Z.; Lou, J.-G.; and Chen, W. 2023. CodeT : Code Generation with Generated Tests. In The Eleventh International Conference on Learning Representations

    2023

  42. [49]

    Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; Pinto, H. P. d. O.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.; et al. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374

    Pith/arXiv arXiv 2021

  43. [50]

    Chen, X.; Lin, M.; Sch \"a rli, N.; and Zhou, D. 2024. Teaching Large Language Models to Self-Debug. In The Twelfth International Conference on Learning Representations

    2024

  44. [51]

    Deng, K.; Liu, J.; Zhu, H.; Liu, C.; Li, J.; Wang, J.; Zhao, P.; Zhang, C.; Wu, Y.; Yin, X.; Zhang, Y.; Su, W.; Xiang, B.; Ge, T.; and Zheng, B. 2024. R2C2-Coder : Enhancing and Benchmarking Real-World Repository-Level Code Completion Abilities of Code Large Language Models. arXiv preprint arXiv:2406.01359

    Pith/arXiv arXiv 2024

  45. [52]

    Farquhar, S.; Kossen, J.; Kuhn, L.; and Gal, Y. 2024. Detecting Hallucinations in Large Language Models Using Semantic Entropy. Nature, 630: 625--630

    2024

  46. [53]

    Gu, W.; Chen, J.; Wang, Y.; Jiang, T.; Li, X.; Liu, M.; Liu, X.; Ma, Y.; and Zheng, Z. 2026. What to Retrieve for Effective Retrieval-Augmented Code Generation? An Empirical Study and Beyond. In Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE). ACM

    2026

  47. [54]

    Guo, C.; Pleiss, G.; Sun, Y.; and Weinberger, K. Q. 2017. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference on Machine Learning, 1321--1330

    2017

  48. [55]

    Guo, D.; Lu, S.; Duan, N.; Wang, Y.; Zhou, M.; and Yin, J. 2022. UniXcoder : Unified Cross-Modal Pre-training for Code Representation. arXiv preprint arXiv:2203.03850

    Pith/arXiv arXiv 2022

  49. [56]

    Guo, D.; Zhu, Q.; Yang, D.; Xie, Z.; Dong, K.; Zhang, W.; Chen, G.; Bi, X.; Wu, Y.; Li, Y.; et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming--The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196

    Pith/arXiv arXiv 2024

  50. [57]

    L.; Nguyen, D

    Hai, N. L.; Nguyen, D. M.; and Bui, N. D. Q. 2024. On the Impacts of Contexts on Repository-Level Code Generation. arXiv preprint arXiv:2406.11927

    Pith/arXiv arXiv 2024

  51. [58]

    A.; Olivier, R.; Avvaru, P.; Yin, P.; Tomasic, A.; and Neubig, G

    Hayati, S. A.; Olivier, R.; Avvaru, P.; Yin, P.; Tomasic, A.; and Neubig, G. 2018. Retrieval-based neural code generation. 925--930

    2018

  52. [59]

    Hou, B.; Liu, Y.; Qian, K.; Andreas, J.; Chang, S.; and Zhang, Y. 2024. Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, 19023--19042

    2024

  53. [60]

    Hwang, J.; Park, J.; Park, H.; Park, S.; and Ok, J. 2024. Retrieval-Augmented Generation with Estimation of Source Reliability. arXiv preprint arXiv:2410.22954

    arXiv 2024

  54. [61]

    u ttler, H.; Lewis, M.; Yih, W.-t.; Rockt \

    Lewis, P.; Perez, E.; Piktus, A.; Petroni, F.; Karpukhin, V.; Goyal, N.; K \"u ttler, H.; Lewis, M.; Yih, W.-t.; Rockt \"a schel, T.; Riedel, S.; and Kiela, D. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv preprint arXiv:2005.11401

    Pith/arXiv arXiv 2020

  55. [62]

    Li, Y.; Shi, E.; Zheng, D.; Duan, K.; Chen, J.; and Wang, Y. 2024. RepoMinCoder : Improving Repository-Level Code Generation Based on Information Loss Screening. In Proceedings of the 15th Asia-Pacific Symposium on Internetware, 229--238

    2024

  56. [63]

    Liang, M.; Xie, X.; Zhang, G.; Zheng, X.; Di, P.; Jiang, W.; Chen, H.; Wang, C.; and Fan, G. 2024. RepoFuse : Repository-Level Code Completion with Fused Dual Context. arXiv preprint arXiv:2402.14323

    Pith/arXiv arXiv 2024

  57. [64]

    Liao, D.; Pan, S.; Sun, X.; Ren, X.; Huang, Q.; Xing, Z.; Jin, H.; and Li, Q. 2024. A ^3 -CodGen : A Repository-Level Code Generation Framework for Code Reuse with Local-Aware, Global-Aware, and Third-Party-Library-Aware. IEEE Transactions on Software Engineering

    2024

  58. [65]

    Liu, W.; Yu, A.; Zan, D.; Shen, B.; Zhang, W.; Zhao, H.; Jin, Z.; and Wang, Q. 2024. GraphCoder: Enhancing Repository-Level Code Completion via Coarse-to-Fine Retrieval Based on Code Context Graph. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering

    2024

  59. [66]

    Lu, S.; Duan, N.; Han, H.; Guo, D.; Hwang, S.-w.; and Svyatkovskiy, A. 2022. ReACC : A Retrieval-Augmented Code Completion Framework. arXiv preprint arXiv:2203.07722

    Pith/arXiv arXiv 2022

  60. [67]

    Ni, A.; Iyer, S.; Radev, D.; Stoyanov, V.; Yih, W.-T.; Wang, S.; and Lin, X. V. 2023. LEVER : Learning to Verify Language-to-Code Generation with Execution. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, 26106--26128

    2023

  61. [68]

    R.; Ahmad, W

    Parvez, M. R.; Ahmad, W. U.; Chakraborty, S.; Ray, B.; and Chang, K.-W. 2021. REDCODER : Retrieval Augmented Code Generation and Summarization. arXiv preprint arXiv:2108.11601

    Pith/arXiv arXiv 2021

  62. [69]

    E.; Adi, Y.; Liu, J.; Sauvestre, R.; Remez, T.; et al

    Roziere, B.; Gehring, J.; Gloeckle, F.; Sootla, S.; Gat, I.; Tan, X. E.; Adi, Y.; Liu, J.; Sauvestre, R.; Remez, T.; et al. 2023. Code Llama: Open Foundation Models for Code. arXiv preprint arXiv:2308.12950

    Pith/arXiv arXiv 2023

  63. [70]

    W.; and Ghosh, S

    Shen, M.; Das, S.; Greenewald, K.; Sattigeri, P.; Wornell, G. W.; and Ghosh, S. 2024. Thermometer: Towards Universal Calibration for Large Language Models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, 44687--44711

    2024

  64. [71]

    J.-M.; Madadi, A.; Sapia, G.; and B \"o hme, M

    Valentin, T. J.-M.; Madadi, A.; Sapia, G.; and B \"o hme, M. 2026. Incoherence as Oracle-less Measure of Error in LLM -Based Code Generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, 33305--33313

    2026

  65. [72]

    Wang, M.; Ma, S.; Gong, S.; Wang, J.; Chen, R.; Cao, L.; and Cai, Y. 2026. SRACG : A Code Generation Framework with Selective Retrieval Augmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, 33584--33592

    2026

  66. [73]

    Wang, Y.; Wang, Y.; Guo, D.; Chen, J.; Zhang, R.; Ma, Y.; and Zheng, Z. 2024. RLCoder : Reinforcement Learning for Repository-Level Code Completion. arXiv preprint arXiv:2407.19487

    Pith/arXiv arXiv 2024

  67. [74]

    U.; Zhang, D.; Ramanathan, M

    Wu, D.; Ahmad, W. U.; Zhang, D.; Ramanathan, M. K.; and Ma, X. 2024. Repoformer: Selective Retrieval for Repository-Level Code Completion. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, 53270--53290

    2024

  68. [75]

    Yan, S.-Q.; Gu, J.-C.; Zhu, Y.; and Ling, Z.-H. 2024. Corrective Retrieval Augmented Generation. arXiv preprint arXiv:2401.15884

    Pith/arXiv arXiv 2024

  69. [76]

    Yang, J.; Zhang, J.; Yang, J.; Jin, K.; Zhang, L.; Peng, Q.; Deng, K.; Miao, Y.; Liu, T.; Cui, Z.; Hui, B.; and Lin, J. 2024. ExecRepoBench : Multi-level Executable Code Completion Evaluation. arXiv preprint arXiv:2412.11990

    Pith/arXiv arXiv 2024

  70. [77]

    Yu, H.; Shen, B.; Ran, D.; Zhang, J.; Zhang, Q.; Ma, Y.; Liang, G.; Li, Y.; Wang, Q.; and Xie, T. 2024. CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering

    2024

  71. [78]

    Zan, D.; Chen, B.; Gong, Y.; Cao, J.; Zhang, F.; Wu, B.; Guan, B.; Yin, Y.; and Wang, Y. 2025. Private-library-oriented code generation with large language models. Knowledge-Based Systems, 326: 113934

    2025

  72. [79]

    Zhang, F.; Chen, B.; Zhang, Y.; Keung, J.; Liu, J.; Zan, D.; Mao, Y.; Lou, J.-G.; and Chen, W. 2023. RepoCoder: Repository-Level Code Completion through Iterative Retrieval and Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2471--2484

    2023

  73. [80]

    Zhong, L.; and Wang, Z. 2024. Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 21841--21849

    2024

  74. [81]

    F.; Wang, Z.; Jiang, Z.; and Neubig, G

    Zhou, S.; Alon, U.; Xu, F. F.; Wang, Z.; Jiang, Z.; and Neubig, G. 2022. DocPrompting : Generating Code by Retrieving the Docs. arXiv preprint arXiv:2207.05987

    Pith/arXiv arXiv 2022