{"id":"57b2f458-b6ee-477b-894c-92881bbfd8e5","arxiv_id":"2507.05200","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Few-shot in-context examples improve LLM-based functional correctness estimation for generated code relative to zero-shot judgment, but the gains are modest and uneven.","lead":"This paper tests whether adding a few labeled examples of correct and incorrect code to a large language model prompt helps the model judge whether newly generated code works. It reports small but mostly positive gains in ranking quality on Python benchmarks, with larger-looking but potentially inflated gains on a Java benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-lingual evidence is invalidated by MBPP/MBJP overlap: the XL test set contains the same problems as the MBPP training set, so retrieved few-shot examples leak the answer and do not measure cross-lingual ICL.","rationale":"The reader's verdict (CONDITIONAL) is reasonable, and the rationale already flags the MBPP/MBJP overlap. However, the formal 'weakest assumption' in the reader's analysis concerned retrieval quality (Equation 1 selecting uninformative examples). I see the dataset overlap as more load-bearing: it invalidates the cross-lingual experimental branch entirely, not just the retrieval mechanism. If the XL results are discarded, the paper's support for the central claim reduces to HEval alone, where effect sizes are small (e.g., +0.011 G-nDCG for CS, +0.001 L-nDCG for CL) and no significance testing is provided. A disjoint cross-lingual benchmark is the decisive check: without it, the paper cannot claim that ICL generalizes across languages, and the abstract's unqualified claim overreaches. The in-domain results remain plausible, so rejection is too harsh; conditional acceptance with a required re-evaluation is the appropriate outcome. I credit the authors for releasing code and using a standard metric (nDCG), but the overlap is a correctness risk in the experimental design, not a stylistic preference. My recommendation aligns with the reader's condition but sharpens the justification: the overlap does not merely weaken the cross-lingual evidence; it makes that evidence structurally incapable of supporting the claim.","tokens_in":9675,"tokens_out":7483,"duration_ms":94936,"concrete_test":"Replace the XL test set with a Java benchmark that has no overlap with MBPP, e.g., a Java translation of HumanEval generated independently, or MultiPL-E Java problems not derived from MBPP. Re-run FS-PS, FS-P, FS-S, and ZS; if FS no longer beats ZS on this disjoint set, the RQ-4 claim is an artifact of the overlap. Also run the in-domain comparison with bootstrap CIs or multiple seeds and report whether the HEval deltas (G-nDCG +0.011, L-nDCG +0.018 for CS; +0.083/0.001 for CL) are significant.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim ('ICL improves code quality estimation') is tested on two benchmarks: HEval (in-domain) and MBJP (cross-lingual). Section 3 states that MBJP 'is the Java version of MBPP' and MBPP is the training set for retrieved examples. Equation 1 computes problem-problem similarity via CodeBERT embeddings. For any MBJP test problem, the corresponding MBPP problem appears in the training set with nearly identical problem text, so alpha-weighting (including alpha=0.5 in FS-PS) will retrieve the exact same problem's labeled Python solutions. The few-shot prompt then shows the LLM the same problem statement with a correct and an incorrect Python implementation, and asks it to judge a Java implementation of that same problem. This is answer leakage, not cross-lingual generalization: the model is effectively doing translation-aware comparison against a reference solution, a much easier task than estimating correctness for a novel problem in a new language. The RQ-4 conclusion (Section 4) that 'prediction effectiveness of Java code improves with Python examples' is therefore not evidence for the central claim in deployed settings where the training set does not contain the test problem. The in-domain HEval results are not affected by this overlap, but the paper's claimed 'consistent' improvements are overstated: for local nDCG, CL HEval improves from .317 (ZS) to .318 (FS-PS, FS-P), and MBJP CL from .269 to .271; FS-S actually drops below ZS in the CL HEval local row. No variance or significance testing is reported, so even the in-domain deltas (e.g., G-nDCG CS HEval .904 to .915) may be within noise. The load-bearing weakness is that the cross-lingual experiments, one of the paper's two test branches, cannot support the stated claim because of the training/test problem overlap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes an in-context-learning (ICL) approach for estimating the functional correctness of LLM-generated code in the absence of unit tests. For a test problem-solution pair, k correct and k incorrect example pairs are retrieved from an MBPP-derived training pool using CodeBERT embedding similarity over problem and/or solution fields (Eq. 1), and a CodeLlama-7B prompt is asked to emit a yes/no correctness judgment; the posterior probability of 'yes' is used to rank generated solutions. The method is evaluated against ELS, TLS, and zero-shot (ZS) baselines on HEval (in-domain) and MBJP (cross-lingual) with CodeStral-22B and CodeLlama-7B as generators, using global and local nDCG. The paper claims few-shot examples consistently improve code quality estimation and that Java quality prediction improves from Python examples.","tokens_in":10025,"tokens_out":6378,"duration_ms":70091,"significance":"The idea of adapting query performance prediction to ranked lists of generated code is timely, and the proposed prompt-based estimator is simple, reproducible (code is publicly released), and uses objective test-case-derived labels. If the claimed gains were robust, the method would be a practically useful ranking signal for code generation workflows. However, the empirical support is currently mixed: several Table 2 cells show few-shot variants below zero-shot, none of the differences are accompanied by significance tests, and the only cross-lingual benchmark (MBJP) is contaminated by test-problem overlap with the MBPP example pool. The contribution is therefore plausible but not yet established at the level claimed.","major_comments":[{"comment":"The cross-lingual experiment cannot support the RQ-4 conclusion. Section 3 says MBJP 'is the Java version of MBPP', and MBPP is the training pool from which the few-shot examples are retrieved (Table 1). For a given MBJP test problem, the corresponding MBPP Python problem appears in the example pool, so the problem-problem similarity term of Eq. (1) will retrieve examples of the same problem; this is true for alpha=1 (FS-P) and alpha=0.5 (FS-PS). The prompt then presents the LLM with correct and incorrect Python implementations of exactly that problem statement before asking it to judge a Java implementation of the same statement. This is answer leakage rather than cross-lingual generalization, and the RQ-4 claim in Section 4 ('prediction effectiveness of Java code improves with Python examples') is not evidence for the central claim in a deployment setting where the test problem does not occur in the training set. The authors should re-evaluate on a Java benchmark disjoint from MBPP, or remove the cross-lingual claim.","section":"§3 (Datasets) and §4 (RQ-4)"},{"comment":"The claim that ICL 'outperforms' zero-shot consistently is contradicted by several cells of Table 2. For L-nDCG on MBJP with CS, ZS reaches 0.570 while FS-PS, FS-P, and FS-S reach only 0.549, 0.545, and 0.542, respectively; for G-nDCG on HEval with CS, FS-P (0.891) is below ZS (0.904). In the CL HEval local row, the best few-shot gain is 0.318 vs 0.317. No significance tests are reported, and with 974 (MBJP) or 164 (HEval) problems such small differences may be noise. The paper should restrict RQ-1 to configurations with non-negligible positive gains and should report paired significance tests (e.g., per-problem permutation or bootstrap) and effect sizes.","section":"§4, Table 2 (RQ-1)"},{"comment":"RQ-3's conclusion that FS-PS 'mostly outperforms' the other variants is not supported in the cross-lingual condition and is partly contradicted by the local MBJP CS row, where FS-PS (0.549) is worse than ZS (0.570) and only marginally better than FS-P (0.545). The paper attributes this to unreliable cross-language solution similarities, but no experiment isolates retrieval quality from prompt effectiveness. At minimum, the authors should report retrieval precision of the selected examples (e.g., whether the same-problem leakage explains the MBJP results) or ablate by using random examples instead of similarity-based ones.","section":"§2 and §4 (RQ-3)"}],"minor_comments":[{"comment":"The word 'Functionally' is typeset as 'F unctionally' in the prompt figure; this typographical issue should be corrected.","section":"Figure 2"},{"comment":"The relationship between k and the 10 generated solutions per problem should be clarified: are the k correct and k incorrect examples sampled with replacement from the same pool of 10 solutions, and are duplicates removed before prompting?","section":"§3 (Datasets and ICL examples)"},{"comment":"The sensitivity plots do not include confidence intervals or significance markers; for L-nDCG on MBJP (Figure 4f), the y-axis range 0.53-0.56 makes it difficult to assess whether differences across k are meaningful.","section":"§4 (Figure 4)"},{"comment":"Several entries, e.g., [16] and [30], use incomplete author formats; the bibliography should be made consistent with the ACM style.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"Novelty is moderate and the manuscript is clearly written, but the central empirical evidence needs repair: one benchmark is contaminated, and the headline improvement is not consistent across Table 2. These issues are fixable within the scope of a revision; I do not see a fundamental flaw in the idea."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: the paper's cross-lingual evidence doesn't hold up. Section 3 says MBJP 'is the Java version of MBPP,' and MBPP is the pool from which few-shot examples are retrieved. So for any MBJP test problem, retrieval will grab the same problem's labeled Python solutions, and the LLM is being shown the correct and incorrect implementations of the exact problem it is judging. That's answer leakage, not cross-lingual generalization. The paper's own text admits the overlap, so this isn't a subtle defect; it's the central experiment for RQ-4.\n\nThe in-domain half is more believable. The method is a reasonable extension of ICE-Score: retrieve balanced correct/incorrect examples using CodeBERT similarity, combine problem and solution field similarities, and few-shot a 7B model. On HEval, FS-PS nudges G-nDCG from .904 to .915 for Codestral-generated solutions, and most FS variants beat zero-shot. They also release code, which helps reproducibility.\n\nThe soft spots beyond the leak: no error bars or significance tests, and the gains are often tiny. L-nDCG for CL HEval goes from .317 (ZS) to .318 (FS-PS); MBJP CL from .269 to .271. One FS-S row is below ZS. Claims of 'consistent' improvement are overstated. The choice of FS-PS as best is also post-hoc, since FS-PS, FS-P, and FS-S differ by small margins.\n\nProportionate verdict: this is a legitimate but modest empirical contribution. The in-domain result is a new data point. The cross-lingual branch should be redone with a genuinely held-out language pair, or dropped. If that happens, the core claim about ICL helping for in-domain ranking would probably survive in weaker form.\n\nWorth a serious referee? Yes—the method is simple, reproducible, and the in-domain effect is plausible. But I'd want any reviewer to catch the MBPP/MBJP overlap and require revision before acceptance.","headline":"Cross-lingual claim is invalidated by MBPP/MBJP leakage; the in-domain few-shot result is plausible but modest.","tokens_in":10589,"tokens_out":2629,"would_cite":false,"duration_ms":30298,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Providing few-shot examples of functionally correct code, retrieved by CodeBERT similarity, improves an LLM's ranking of generated solutions by functional correctness over zero-shot and embedding baselines.","keywords":["in-context learning","code quality estimation","functional correctness","query performance prediction","code generation","few-shot prompting","CodeBERT similarity","nDCG"],"falsifier":"Take the same few-shot pipeline but replace the retrieved examples with randomly selected labeled examples of the same correct/incorrect balance; if the nDCG gains over zero-shot vanish or remain unchanged, then the similarity-based retrieval in Equation 1 is not doing the work the paper attributes to it.","tokens_in":9506,"feed_emoji":"🧑💻","tokens_out":4996,"duration_ms":51678,"temperature":0.7,"pith_summary":"The paper targets a practical gap: when a developer asks an LLM for code, the model returns several candidate solutions ranked by likelihood, and the team must pick one without necessarily having unit tests. Treating functional correctness as the analogue of relevance in information retrieval, the authors argue that what matters is a ranking in which correct solutions appear before incorrect ones. They propose using few-shot in-context learning, retrieving labeled problem-solution pairs from a training set and appending them to the prompt, to estimate whether a solution is functionally correct. The central claim is that these labeled examples consistently improve ranking quality over both zero-shot prompting and embedding-similarity baselines, in same-language, cross-language, same-model, and cross-model settings.","feed_headline":"Few-shot examples sharpen LLM code-quality rankings","feed_subtitle":"A 7B model ranks generated solutions better with a few labeled correct and incorrect examples than zero-shot baselines.","key_machinery":"The load-bearing mechanism is the field-based similarity function $\\sigma((P_{\\text{test}}, S_{\\text{test}}), (P, S)) = \\alpha (\\mathbf{e}_{P_{\\text{test}}} \\cdot \\mathbf{e}_P) + (1-\\alpha)(\\mathbf{e}_{S_{\\text{test}}} \\cdot \\mathbf{e}_S)$, where $\\mathbf{e}_Z$ is a CodeBERT embedding of text $Z$, and $\\alpha \\in \\{0, 0.5, 1\\}$ selects problem-only, solution-only, or combined retrieval. For each input pair, the system retrieves $k$ correct and $k$ incorrect examples from the MBPP training set, builds a balanced few-shot prompt, and computes the posterior likelihood $P(\\text{'yes'}) / (P(\\text{'yes'}) + P(\\text{'no'}))$ from the LLM's token probabilities. That likelihood score orders the candidate solutions. The balancing of correct and incorrect examples and the retrieval similarity are the two controls that distinguish this method from zero-shot estimation.","core_discovery":"The paper's central discovery is that feeding an LLM a small balanced set of retrieved examples, some labeled functionally correct and some incorrect, lets a 7-billion-parameter instruction-tuned CodeLlama model estimate the functional correctness of generated code better than the zero-shot version of the same prompt and better than CodeBERT-embedding baselines. The improvement holds whether the estimator ranks solutions for one problem (local nDCG) or across all problems (global nDCG), and it transfers across programming languages, where Python examples help score Java code, and across generator models. Among the three retrieval variants, combining problem-level and solution-level similarity (FS-PS) generally yields the best results, with problem-only retrieval winning in some cross-lingual cases.","pith_inferences":["The success of retrieval-based ICL here suggests that the quality of the retrieved examples, not just their count $k$, is the true lever; a study that varies retrieval accuracy while holding $k$ fixed could confirm this.","Because the estimator produces a continuous score, it could be recycled as a reward signal for reranking generated solutions or for selecting which solution to refine, a use the paper does not explore.","The query-performance-prediction analogy implies that query-drift or clarity-style measures from information retrieval could be adapted to code-ranking lists, an extension beyond the paper's own similarity function."],"forward_implications":["Developers can rank generated candidate solutions by estimated functional correctness without running unit tests, saving time in feature-driven development.","A smaller 7B model can stand in as a quality judge for code produced by larger generators of a different family, reducing deployment cost.","Python-labeled examples transfer to Java code, suggesting the estimator is not locked to one language.","Combining problem and solution similarity in retrieval is usually better than either alone, giving practitioners a simple tuning choice."],"supporting_citations":[{"why":"supplies the MBPP training set from which few-shot examples are retrieved.","marker":"[4]"},{"why":"provides the HumanEval benchmark used for in-domain testing.","marker":"[6]"},{"why":"CodeBERT supplies the dense embeddings used in the retrieval similarity function.","marker":"[10]"},{"why":"the field-based BM25F retrieval idea underlies the linear combination of problem and solution similarities.","marker":"[21]"},{"why":"defines nDCG, the ranking measure used to evaluate the estimators.","marker":"[27]"},{"why":"CodeBERTScore is the basis of the TLS baseline and of the broader embedding-similarity comparison.","marker":"[32]"},{"why":"ICE-Score is the zero-shot code quality estimator that the proposed few-shot method is compared against.","marker":"[33]"}],"fun_headline_variants":["Few-shot examples boost LLM code-quality ranking","ICL improves functional correctness estimation for code","7B model picks correct code with few-shot ICL examples","Few labeled examples sharpen code correctness ranking in LLMs","In-context learning sharpens ranking of generated code correctness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method rests on the assumption that CodeBERT embedding similarity, as defined in Equation 1, selects few-shot examples that actually help the LLM judge functional correctness; if the retrieved examples are merely textually similar but uninformative, the few-shot prompt could mislead the estimator, and the paper never separates retrieval quality from prompt effects.","fun_headline_variants_meta":{"raw":{"variants":["Few-shot examples boost LLM code-quality ranking","ICL improves functional correctness estimation for code","7B model picks correct code with few-shot ICL examples","Few labeled examples sharpen code correctness ranking in LLMs","In-context learning sharpens ranking of generated code correctness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000372,"raw_usage":{"total_tokens":1951,"prompt_tokens":865,"completion_tokens":1086,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":1010}},"tokens_in":481,"tokens_out":1086,"duration_ms":9923,"temperature":1.0,"reasoning_tokens":1010,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:29:25.999558+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same few-shot pipeline but replace the retrieved examples with randomly selected labeled examples of the same correct/incorrect balance; if the nDCG gains over zero-shot vanish or remain unchanged, then the similarity-based retrieval in Equation 1 is not doing the work the paper attributes to it.","supporting_citations":[{"cited_title":"Robertson, Hugo Zaragoza, and Michael J","cited_arxiv_id":null,"evidence_quote":"the field-based BM25F retrieval idea underlies the linear combination of problem and solution similarities."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines nDCG, the ranking measure used to evaluate the estimators."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CodeBERTScore is the basis of the TLS baseline and of the broader embedding-similarity comparison."}],"review_version":1}