Pith. sign in

REVIEW 3 major objections 4 minor 34 references

In-Context Learning as an Effective Estimator of Functional Correctness of LLM-Generated Code

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Providing few-shot examples of functionally correct code, retrieved by CodeBERT similarity, improves an LLM's ranking of generated solutions by functional correctness over zero-shot and embedding baselines.

desk verdict Cross-lingual claim is invalidated by MBPP/MBJP leakage; the in-domain few-shot result is plausible but modest. read the letter →

arxiv 2507.05200 v1 pith:PFN27RLM submitted 2025-07-07 cs.SE cs.IR

Susmita Das , Madhusudan Ghosh , Priyanka Swami , Debasis Ganguly , Gul Calikli This is my paper
classification cs.SEcs.IR
keywords in-contextlearningcodequalityestimationfunctionalcorrectnessqueryperformancepredictiongenerationfew-shotpromptingBERTsimilaritynDCG
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper targets a practical gap: when a developer asks an LLM for code, the model returns several candidate solutions ranked by likelihood, and the team must pick one without necessarily having unit tests. Treating functional correctness as the analogue of relevance in information retrieval, the authors argue that what matters is a ranking in which correct solutions appear before incorrect ones. They propose using few-shot in-context learning, retrieving labeled problem-solution pairs from a training set and appending them to the prompt, to estimate whether a solution is functionally correct. The central claim is that these labeled examples consistently improve ranking quality over both zero-shot prompting and embedding-similarity baselines, in same-language, cross-language, same-model, and cross-model settings.

What carries the argument

The load-bearing mechanism is the field-based similarity function $\sigma((P_{\text{test}}, S_{\text{test}}), (P, S)) = \alpha (\mathbf{e}_{P_{\text{test}}} \cdot \mathbf{e}_P) + (1-\alpha)(\mathbf{e}_{S_{\text{test}}} \cdot \mathbf{e}_S)$, where $\mathbf{e}_Z$ is a CodeBERT embedding of text $Z$, and $\alpha \in \{0, 0.5, 1\}$ selects problem-only, solution-only, or combined retrieval. For each input pair, the system retrieves $k$ correct and $k$ incorrect examples from the MBPP training set, builds a balanced few-shot prompt, and computes the posterior likelihood $P(\text{'yes'}) / (P(\text{'yes'}) + P(\text{'no'}))$ from the LLM's token probabilities. That likelihood score orders the candidate solutions. The balancing of correct and incorrect examples and the retrieval similarity are the two controls that distinguish this method from zero-shot estimation.

What would settle it

Take the same few-shot pipeline but replace the retrieved examples with randomly selected labeled examples of the same correct/incorrect balance; if the nDCG gains over zero-shot vanish or remain unchanged, then the similarity-based retrieval in Equation 1 is not doing the work the paper attributes to it.

Watch

Extended reading notes

Core claim

The paper's central discovery is that feeding an LLM a small balanced set of retrieved examples, some labeled functionally correct and some incorrect, lets a 7-billion-parameter instruction-tuned CodeLlama model estimate the functional correctness of generated code better than the zero-shot version of the same prompt and better than CodeBERT-embedding baselines. The improvement holds whether the estimator ranks solutions for one problem (local nDCG) or across all problems (global nDCG), and it transfers across programming languages, where Python examples help score Java code, and across generator models. Among the three retrieval variants, combining problem-level and solution-level similarity (FS-PS) generally yields the best results, with problem-only retrieval winning in some cross-lingual cases.

Load-bearing premise

The whole method rests on the assumption that CodeBERT embedding similarity, as defined in Equation 1, selects few-shot examples that actually help the LLM judge functional correctness; if the retrieved examples are merely textually similar but uninformative, the few-shot prompt could mislead the estimator, and the paper never separates retrieval quality from prompt effects.

Editorial extensions

If this is right

  • Developers can rank generated candidate solutions by estimated functional correctness without running unit tests, saving time in feature-driven development.
  • A smaller 7B model can stand in as a quality judge for code produced by larger generators of a different family, reducing deployment cost.
  • Python-labeled examples transfer to Java code, suggesting the estimator is not locked to one language.
  • Combining problem and solution similarity in retrieval is usually better than either alone, giving practitioners a simple tuning choice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The success of retrieval-based ICL here suggests that the quality of the retrieved examples, not just their count $k$, is the true lever; a study that varies retrieval accuracy while holding $k$ fixed could confirm this.
  • Because the estimator produces a continuous score, it could be recycled as a reward signal for reranking generated solutions or for selecting which solution to refine, a use the paper does not explore.
  • The query-performance-prediction analogy implies that query-drift or clarity-style measures from information retrieval could be adapted to code-ranking lists, an extension beyond the paper's own similarity function.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper proposes an in-context-learning (ICL) approach for estimating the functional correctness of LLM-generated code in the absence of unit tests. For a test problem-solution pair, k correct and k incorrect example pairs are retrieved from an MBPP-derived training pool using CodeBERT embedding similarity over problem and/or solution fields (Eq. 1), and a CodeLlama-7B prompt is asked to emit a yes/no correctness judgment; the posterior probability of 'yes' is used to rank generated solutions. The method is evaluated against ELS, TLS, and zero-shot (ZS) baselines on HEval (in-domain) and MBJP (cross-lingual) with CodeStral-22B and CodeLlama-7B as generators, using global and local nDCG. The paper claims few-shot examples consistently improve code quality estimation and that Java quality prediction improves from Python examples.

Significance. The idea of adapting query performance prediction to ranked lists of generated code is timely, and the proposed prompt-based estimator is simple, reproducible (code is publicly released), and uses objective test-case-derived labels. If the claimed gains were robust, the method would be a practically useful ranking signal for code generation workflows. However, the empirical support is currently mixed: several Table 2 cells show few-shot variants below zero-shot, none of the differences are accompanied by significance tests, and the only cross-lingual benchmark (MBJP) is contaminated by test-problem overlap with the MBPP example pool. The contribution is therefore plausible but not yet established at the level claimed.

major comments (3)
  1. [§3 (Datasets) and §4 (RQ-4)] The cross-lingual experiment cannot support the RQ-4 conclusion. Section 3 says MBJP 'is the Java version of MBPP', and MBPP is the training pool from which the few-shot examples are retrieved (Table 1). For a given MBJP test problem, the corresponding MBPP Python problem appears in the example pool, so the problem-problem similarity term of Eq. (1) will retrieve examples of the same problem; this is true for alpha=1 (FS-P) and alpha=0.5 (FS-PS). The prompt then presents the LLM with correct and incorrect Python implementations of exactly that problem statement before asking it to judge a Java implementation of the same statement. This is answer leakage rather than cross-lingual generalization, and the RQ-4 claim in Section 4 ('prediction effectiveness of Java code improves with Python examples') is not evidence for the central claim in a deployment setting where the test problem does not occur in the training set. The authors should re-evaluate on a Java benchmark disjoint from MBPP, or remove the cross-lingual claim.
  2. [§4, Table 2 (RQ-1)] The claim that ICL 'outperforms' zero-shot consistently is contradicted by several cells of Table 2. For L-nDCG on MBJP with CS, ZS reaches 0.570 while FS-PS, FS-P, and FS-S reach only 0.549, 0.545, and 0.542, respectively; for G-nDCG on HEval with CS, FS-P (0.891) is below ZS (0.904). In the CL HEval local row, the best few-shot gain is 0.318 vs 0.317. No significance tests are reported, and with 974 (MBJP) or 164 (HEval) problems such small differences may be noise. The paper should restrict RQ-1 to configurations with non-negligible positive gains and should report paired significance tests (e.g., per-problem permutation or bootstrap) and effect sizes.
  3. [§2 and §4 (RQ-3)] RQ-3's conclusion that FS-PS 'mostly outperforms' the other variants is not supported in the cross-lingual condition and is partly contradicted by the local MBJP CS row, where FS-PS (0.549) is worse than ZS (0.570) and only marginally better than FS-P (0.545). The paper attributes this to unreliable cross-language solution similarities, but no experiment isolates retrieval quality from prompt effectiveness. At minimum, the authors should report retrieval precision of the selected examples (e.g., whether the same-problem leakage explains the MBJP results) or ablate by using random examples instead of similarity-based ones.
minor comments (4)
  1. [Figure 2] The word 'Functionally' is typeset as 'F unctionally' in the prompt figure; this typographical issue should be corrected.
  2. [§3 (Datasets and ICL examples)] The relationship between k and the 10 generated solutions per problem should be clarified: are the k correct and k incorrect examples sampled with replacement from the same pool of 10 solutions, and are duplicates removed before prompting?
  3. [§4 (Figure 4)] The sensitivity plots do not include confidence intervals or significance markers; for L-nDCG on MBJP (Figure 4f), the y-axis range 0.53-0.56 makes it difficult to assess whether differences across k are meaningful.
  4. [References] Several entries, e.g., [16] and [30], use incomplete author formats; the bibliography should be made consistent with the ACM style.

Circularity Check

1 steps flagged · score 4.0 of 10

Cross-lingual MBJP results are undermined by the MBPP training pool containing the same problems; the in-domain HEval results remain independent evidence.

  1. self definitional [Section 3 'Datasets' (MBJP overlap with MBPP training pool; used with Eq. 1 in Section 2)]
    "We use MBPP as our training set for ICL examples (see Figure 1). For in-domain testing, we use HumanEval (HEval), whereas for out-domain (cross-lingual) evaluation, we use the MBJP dataset, which is the Java version of MBPP."

    MBJP is defined as the Java translation of MBPP, and MBPP's training split is the exact pool from which Eq. 1 retrieves few-shot examples. Consequently, for each MBJP test problem, the identical Python problem statement exists in the training pool. With alpha=1 (FS-P) or alpha=0.5 (FS-PS), problem-problem similarity will retrieve that same problem's labeled correct and incorrect Python solutions, so the LLM sees the same problem's reference implementations before judging the Java solution. RQ-4's claim that 'prediction effectiveness of Java code improves with Python examples' is therefore a same-problem answer-leakage effect, not evidence of cross-lingual generalization.

full rationale

The paper's main idea, that adding retrieved labeled examples helps LLM-based code quality estimation, is tested on HEval and MBJP. HEval is separate from the MBPP training pool, and the ground-truth labels come from executing external test cases, so the in-domain and out-domain-model results are not circular. The hyperparameter k is tuned on a held-out dev split, which is standard practice. There is no load-bearing self-citation or imported uniqueness theorem. The significant problem is confined to the cross-lingual evaluation: MBJP is the Java version of MBPP, and MBPP is the training set used to build the few-shot examples. Equation 1's problem-problem similarity therefore retrieves the exact same problem from the training pool for the MBJP test query, and the resulting few-shot prompt contains the known correct and incorrect Python implementations of that same problem. The MBJP improvements reported for FS-P and FS-PS therefore do not measure cross-lingual ICL generalization; they measure the model's ability to exploit an answer-leakage shortcut. This invalidates RQ-4 as a cross-lingual test while leaving the HEval-based evidence intact. The appropriate score is therefore moderate, not extreme.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard benchmark assumptions (test-case labels, nDCG), the empirical hypothesis that ICL examples help, and the design choices of k and alpha. No invented entities are introduced.

free parameters (3)
  • k (number of correct and incorrect examples) = k=3 for FS-S, k=4 for FS-P and FS-PS
    Tuned on the MBPP development split by optimizing G-nDCG (Section 3, Evaluation Metrics).
  • alpha (similarity field weight) = 0 (FS-S), 0.5 (FS-PS), 1 (FS-P)
    Manually chosen design variants of Equation 1; the paper reports results for all three.
  • Number of generated solutions per problem = 10
    Fixed by experimental design for both training and test lists (Section 3).
assumptions (5)
  • domain assumption A solution that passes all available test cases is deemed functionally correct.
    Used to construct ground-truth labels for training examples (Section 2, Figure 1 step 1).
  • domain assumption Functional correctness of code can be analogized to relevance of documents in IR, and nDCG is an appropriate evaluation measure.
    Justifies the QPP framing and the metrics used in Section 3.
  • domain assumption The ratio of 'yes' to 'no' token probabilities from the estimator LLM is a reliable scalar for ranking solutions by correctness.
    This posterior is the score assigned to each solution (Section 2, LLM-Prompting Details).
  • domain assumption CodeBERT embedding similarity between problem-solution pairs is a useful signal for choosing ICL examples.
    The similarity function in Equation 1 assumes that embedding proximity indicates helpful examples.
  • domain assumption Few-shot examples of correct and incorrect code will shift the LLM's predictions in the right direction.
    The central hypothesis of the paper, tested empirically in Section 4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of In-Context Learning as an Effective Estimator of Functional Correctness of LLM-Generated Code." pith.science (2026). https://pith.science/paper/PFN27RLM

@misc{pith2026250705200,
  author       = {Pith},
  title        = {Pith review of: In-Context Learning as an Effective Estimator of Functional Correctness of LLM-Generated Code},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PFN27RLM}},
  note         = {Machine review of arXiv:2507.05200}
}
read the original abstract

When applying LLM-based code generation to software development projects that follow a feature-driven or rapid application development approach, it becomes necessary to estimate the functional correctness of the generated code in the absence of test cases. Just as a user selects a relevant document from a ranked list of retrieved ones, a software generation workflow requires a developer to choose (and potentially refine) a generated solution from a ranked list of alternative solutions, ordered by their posterior likelihoods. This implies that estimating the quality of a ranked list -- akin to estimating "relevance" for query performance prediction (QPP) in IR -- is also crucial for generative software development, where quality is defined in terms of "functional correctness". In this paper, we propose an in-context learning (ICL) based approach for code quality estimation. Our findings demonstrate that providing few-shot examples of functionally correct code from a training set enhances the performance of existing QPP approaches as well as a zero-shot-based approach for code quality estimation.

Figures

Figures reproduced from arXiv: 2507.05200 by the authors.

Figure 1
Figure 1. Schematic workflow of our proposed few-shot code quality estimation: [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Prompt instruction for code quality estimation with [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The final estimation likelihood is obtained by the posterior [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Effect of variations in 𝑘 on the global and local tasks - respectively, ranking all problem-solution pairs, vs. ranking a list of 10 solutions for each problem for the two different test sets. and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXi…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 14 canonical work pages

  1. [1]

    Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala, and Christopher Ré

    Simran Arora, Avanika Narayan, Mayee F. Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala, and Christopher Ré. 2022. Ask Me Anything: A simple strategy for prompting language models. arXiv:2210.02441 [cs.CL]

    arXiv 2022

  2. [2]

    Owura Asare, Meiyappan Nagappan, and N. Asokan. 2024. Is GitHub’s Copilot as Bad as Humans at Introducing Vulnerabilities in Code? arXiv:2204.04741 [cs.SE] https://arxiv.org/abs/2204.04741

    arXiv 2024

  3. [3]

    Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ra- manathan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Su...

    arXiv 2023

  4. [4]

    ranking a list of 10 solutions for each problem for the two different test sets

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, 1https://github.com/Susmitacse/ICL-Code-Quality-Estimation.git Estimating the Quality of LLM-Generated Program Solutions SIGIR ’25, July 13–18, 2025, Padua, Italy 1 2 3 4 5 Number of Examples (k) 0.88 0.90 0.92 0.94nD...

    arXiv 2025

  5. [5]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  6. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    arXiv 2021

  7. [7]

    Desmarais, and Hironori Washizaki

    Arghavan Moradi Dakhel, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, and Hironori Washizaki. 2024. Generative AI for Software Development: A Family of Studies on Code Generation . Springer Nature Switzerland, Cham, 151–172. https://doi.org/10.1007/978-3-031-55642-5_7

    work page doi:10.1007/978-3-031-55642-5_7 2024

  8. [8]

    Suchana Datta, Debasis Ganguly, Mandar Mitra, and Derek Greene. 2023. A Relative Information Gain-based Query Performance Prediction Framework with Generated Query Variants. ACM Trans. Inf. Syst. 41, 2 (2023), 38:1–38:31

    work page 2023

Show all 34 references
  1. [9]

    Tuan Dinh, Jinman Zhao, Samson Tan, Renato Negrinho, Leonard Lausen, Sheng Zha, and George Karypis. 2023. Large Language Models of Code Fail at Completing Code with Potential Bugs. arXiv:2306.03438 [cs.LG] https: //arxiv.org/abs/2306.03438

    2023 arXiv

  2. [10]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. arXiv:2002.08155 [cs.CL] https://arxiv.org/abs/2002.08155

    2020 arXiv

  3. [11]

    Debasis Ganguly, Suchana Datta, Mandar Mitra, and Derek Greene. 2022. An Analysis of Variations in the Effectiveness of Query Performance Prediction. In ECIR (1) (Lecture Notes in Computer Science, Vol. 13185) . Springer, 215–229

    2022

  4. [12]

    Daya Guo, Canwen Xu, Nan Duan, Jian Yin, and Julian McAuley. 2023. Long- Coder: A Long-Range Pre-trained Language Model for Code Completion. arXiv:2306.14893 [cs.SE] https://arxiv.org/abs/2306.14893

    2023 arXiv

  5. [13]

    E. I. Karac, B. Turhan, , and N. Juristo. 2019. A Controlled Experiment with Novice Developers on the Impact of Task Description Granularity on Software Quality in Test-Driven Development. IEEE Transactions on Software Engineering (2019)

    2019

  6. [14]

    Minghan Li, Xueguang Ma, and Jimmy Lin. 2022. An Encoder Attribution Analysis for Dense Passage Retriever in Open-Domain Question Answering. InProceedings of the 2nd Workshop on Trustworthy Natural Language Processing (TrustNLP 2022). Association for Computational Linguistics,...

    2022 doi

  7. [15]

    Tianle Li, Xueguang Ma, Alex Zhuang, Yu Gu, Yu Su, and Wenhu Chen. 2023. Few- shot In-context Learning on Knowledge Base Question Answering. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Association for Comp...

    2023 doi

  8. [16]

    Feng Lin, Dong Jae Kim, and Tse-Husn (Peter) Chen. 2023. When LLM-based Code Generation Meets the Software Development Process. arXiv:2403.15852

    2023 arXiv

  9. [17]

    Sheshera Mysore, Andrew Mccallum, and Hamed Zamani. 2023. Large Language Model Augmented Narrative Driven Recommendations (RecSys ’23). Association for Computing Machinery, New York, NY, USA, 777–783. https://doi.org/10. 1145/3604915.3608829

    2023

  10. [18]

    Chanathip Pornprasit and Chakkrit Tantithamthavorn. 2024. Fine-tuning and prompt engineering for large language models-based code review automation. Information and Software Technology 175 (2024), 107523

    2024

  11. [19]

    Lelkes, Honglei Zhuang, Jimmy Lin, Donald Metzler, and Vinh Q

    Ronak Pradeep, Kai Hui, Jai Gupta, Adam D. Lelkes, Honglei Zhuang, Jimmy Lin, Donald Metzler, and Vinh Q. Tran. 2023. How Does Generative Retrieval Scale to Millions of Passages? arXiv:2305.11841 [cs.IR]

    2023 arXiv

  12. [20]

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Im- proving language understanding with unsupervised learning. (2018). https: //openai.com/research/language-unsupervised

    2018

  13. [21]

    Robertson, Hugo Zaragoza, and Michael J

    Stephen E. Robertson, Hugo Zaragoza, and Michael J. Taylor. 2004. Simple BM25 extension to multiple weighted fields. In CIKM. ACM, 42–49

    2004

  14. [22]

    HJ Rognerud and JE Hannay. 2009. Challenges in enterprise software integration: An industrial study using repertory grids. In 2009 3rd International Symposium on Empirical Software Engineering and Measurement . IEEE, 11–22

    2009

  15. [23]

    Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation. arXiv:2302.06527 [cs.SE] https://arxiv.org/abs/2302.06527

    2023 arXiv

  16. [24]

    Anna Shtok, Oren Kurland, and David Carmel. 2010. Using Statistical Decision Theory and Relevance Models for Query-Performance Prediction. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’10) . Associati...

    2010

  17. [25]

    Anna Shtok, Oren Kurland, David Carmel, Fiana Raiber, and Gad Markovits. 2012. Predicting Query Performance by Query-Drift Estimation. ACM Trans. Inf. Syst. 30, 2, Article 11 (2012), 35 pages

    2012

  18. [26]

    Yuting Tang, Ratish Puduppully, Zhengyuan Liu, and Nancy Chen. 2023. In- context Learning of Large Language Models for Controlled Dialogue Summa- rization: A Holistic Benchmark and Empirical Analysis. In Proceedings of the 4th New Frontiers in Summarization Workshop, Yue Dong,...

    2023 doi

  19. [27]

    Yining Wang, Liwei Wang, Yuanzhi Li, Di He, and Tie-Yan Liu. 2013. A Theoreti- cal Analysis of NDCG Type Ranking Measures. In COLT (JMLR Workshop and SIGIR ’25, July 13–18, 2025, Padua, Italy Susmita Das, Madhusudan Ghosh, Priyanka Swami, Debasis Ganguly, and Gul Calikli Confe...

    2013

  20. [28]

    Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, et al

  21. [29]

    Di Wu, Wasi Uddin Ahmad, Dejiao Zhang, Murali Krishna Ramanathan, and Xiaofei Ma. 2024. Repoformer: Selective Retrieval for Repository-Level Code Completion. arXiv:2403.10059 [cs.SE] https://arxiv.org/abs/2403.10059

    2024 arXiv

  22. [30]

    Yixi Wu et al. 2024. A Comprehensive Framework for Evaluating API-oriented Code Generation in Large Language Models. arXiv:2409.15228

    2024 arXiv

  23. [31]

    Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Alex Polozov, and Charles Sutton. 2022. Natural Language to Code Generation in Interactive Data Science Notebooks. arXiv:2212.09248 ...

    2022 arXiv

  24. [32]

    Shuyan Zhou, Uri Alon, Sumit Agarwal, and Graham Neubig. 2023. Code- BERTScore: Evaluating Code Generation with Pretrained Models of Code. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds....

    2023

  25. [33]

    Terry Yue Zhuo. 2024. ICE-Score: Instructing Large Language Models to Evaluate Code. arXiv:2304.14317 [cs.AI]

    2024 arXiv

  26. [2022]

    In 2022 ACM Conference on Fairness, Accountability, and Transparency

    Taxonomy of risks posed by language models. In 2022 ACM Conference on Fairness, Accountability, and Transparency. 214–229

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.