{"id":"35e0aeae-64f4-4926-b393-165c21063731","arxiv_id":"2412.12852","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SSL_ner, a named-entity-based few-shot example selector, improves code explanation over token-based selection but not significantly over embedding-based selection across open-source Code-LLMs.","lead":"This paper benchmarks four open-source code LLMs on code explanation and proposes a new few-shot example selection method, SSL_ner, using named entity recognition to choose similar code samples. The method yields small gains over token-based selection but does not significantly beat embedding-based selection.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ4 already concedes SSL_ner is not significantly better than Selection_semantic, so the abstract's 'best SSL approach' claim is not supported; the numerical margins in Tables 2–7 are small and undercut the central novelty claim.","rationale":"I read the paper in good faith. The benchmarking of four open-source Code-LLMs on CoNaLa and TLC is a useful contribution, and the paper is transparent about limitations. The strongest claim, however, is that SSL_ner is the best selective-shot approach. That claim is load-bearing because it motivates the novelty. RQ4 explicitly reports no statistically significant difference over Selection_semantic; the tables' margins are tiny; and no p-values or test details are given. This is an internal inconsistency with the abstract/introduction, not a disagreement with consensus. The reader's weakest_assumption focuses on entity extraction validity; I agree that is an unresolved mechanism, but the more decisive issue is the significance gap because even perfect entities would not make the 'best' claim true if the comparison is not significant. Therefore I do not move the verdict; CONDITIONAL remains appropriate, with the condition that the superiority claim be either supported by proper corrected significance testing or toned down.","tokens_in":13830,"tokens_out":6407,"duration_ms":56607,"concrete_test":"Recompute the SSL_ner vs Selection_semantic comparison for every reported (model, dataset/intent, metric) cell using paired bootstrap or Wilcoxon signed-rank tests over individual test samples, with Benjamini-Hochberg correction across all cells, and report p-values. If no cell reaches adjusted p<0.05, revise the abstract and conclusion to 'competitive with embedding-based selection and better than token-based selection'; the current 'best/effectiveness over SOTA' phrasing should be removed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing claim is that SSL_ner is the best selective-shot method. The paper's own RQ4 (Sec. 6) states that differences between SSL_ner and Selection_semantic are 'trivial' and 'not statistically significant' on CoNaLa and across all TLC intents. Without significance, numerical margins of roughly 0.003–0.02 in BLEU/ROUGE-L/METEOR (e.g., Table 2 CodeLlama BLEU 0.399 vs 0.395; Table 7 'what' METEOR 0.341 vs 0.335) do not establish superiority over the embedding baseline. RQ3 reports a significant gain over Selection_token, so 'best token-based' is defensible; 'effectiveness over SOTA' and 'best performing SSL approach' are not. The unvalidated UniversalNER entities and the hand-set weights (Sec. 4.3, Ethical Considerations) are a second-order concern: they make the mechanism unverified, but the primary issue is that the central 'best' claim is contradicted by the paper's own significance testing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies selective-shot learning (SSL) for code explanation with open-source Code-LLMs. It benchmarks four Code-LLMs (Llama-2-Coder-7B, CodeUp-13B, StarCoder-15.5B, CodeLlama-34B) on two datasets (CoNaLa inline-level; TLC function-level) and compares three example-selection strategies: token-based Jaccard similarity, embedding-based CodeBERT cosine similarity, and a new method, SSL_ner, which uses UniversalNER to extract code entities and computes a weighted Jaccard similarity over entity types (Eq. 1). The central claim is that SSL_ner is the best-performing SSL approach, and the paper reports quantitative gains across BLEU, ROUGE-L, and METEOR. The paper also provides qualitative examples and discusses limitations and ethical considerations.","tokens_in":14047,"tokens_out":4164,"duration_ms":37719,"significance":"The work is the first broad benchmarking of open-source Code-LLMs for code explanation under selected few-shot examples, and it proposes a simple, interpretable selection method. If the findings are solid, the study is a useful reference for practitioners selecting open-source models and example-retrieval strategies. The evaluation is broad (4 models x 2 datasets x 5 TLC intents) and the method is simple to implement. However, the central novelty claim is weakened by the paper's own significance testing: RQ4 reports that gains over Selection_semantic are not statistically significant, and the entity-extraction component is explicitly unvalidated. These issues need to be addressed before the main claims can be accepted.","major_comments":[{"comment":"The abstract and Section 7 claim that SSL_ner is 'effective' over state-of-the-art methods and is 'the best performing SSL approach,' but Section 6 (RQ4) states: 'The improvements over the best token-based method SSL_ner and Selection semantic are trivial and is not statistically significant' for CoNaLa and all TLC intents. The numerical margins are small (e.g., Table 2, CodeLlama BLEU 0.399 vs 0.395 and METEOR 0.381 vs 0.375; Table 7, 'what' METEOR 0.341 vs 0.335). Without statistically significant differences over Selection_semantic, the 'best SSL approach' claim is not supported by the evidence. The defensible claim is the one in Section 7, that SSL_ner is the best token-based method. Please report pairwise significance tests for all comparisons and revise the abstract and conclusion accordingly.","section":"Section 6 (RQ4) and Abstract/§7"},{"comment":"The proposed method depends critically on (i) the quality of entities extracted by UniversalNER and (ii) the hand-set weights (wei = 0 for 'data type', 'variable', 'value'). Section 9 admits: 'We did not perform a validation of the code entities predicted by the UniversalNER model.' This unvalidated component is load-bearing for SSL_ner's similarity score. Add an evaluation of entity extraction accuracy on the two datasets, an ablation that varies the entity types and weights, or a sensitivity analysis, to show that the method's advantage is not an artifact of noisy or arbitrary entity selection.","section":"Section 4.3, Eq. (1) and Section 9"},{"comment":"The CodeLlama prompt template states: 'Below is a line of python code that describes a task,' but the TLC dataset contains Java code. If the same template is used for both datasets, this language mismatch could systematically disadvantage CodeLlama on TLC and confound the results. Please clarify whether the template was adapted for Java, and if not, address this as a potential limitation.","section":"Section 5 (Prompt templates)"}],"minor_comments":[{"comment":"The table header contains a typo, 'A verage length', and the layout of subdomain rows is hard to read; please reformat and correct.","section":"Section 3, Table 1"},{"comment":"The sentence 'the improvements reported are 3.8%, 7.85%, 5.84% and 2.57%' is ambiguous about whether these are relative percentages or percentage-point differences; please specify.","section":"Section 6, RQ3"},{"comment":"The sentence 'The improvements over the best token-based method SSL ner and Selection semnatic are trivial' is grammatically ambiguous and contains a typo ('semnatic'); clarify which method is compared with which and correct the spelling.","section":"Section 6, RQ4"},{"comment":"The claim of being the 'first systematic benchmarking' should be qualified relative to the existing evaluations of transformer models and LLMs on code summarization mentioned in Section 2 (e.g., Mondal et al., 2023; Dvivedi et al., 2024), to avoid overstating novelty.","section":"Abstract / Section 1"},{"comment":"The data and code release link points to an internal Bosch domain (github.boschdevcloud.com); a publicly accessible repository or hosted archive should be provided to support the reproducibility claim.","section":"Section 9 / Data availability"},{"comment":"The paper sets temperature=0.7 but also do_sample=False for the Code-LLMs; please state that the sampling is deterministic or clarify the role of the temperature parameter.","section":"Section 5 (LLM inference)"}],"recommendation":"major_revision","confidential_remarks":"The reader's conditional verdict is appropriate: the paper has useful breadth and a clear, simply-described method, but the headline claim ('best SSL approach') is contradicted by the manuscript's own significance statement in RQ4, and the entity-extraction module is admittedly unvalidated. The 'best token-based' claim is defensible and should be the central message. The TLC prompt-template issue is an additional confound that the authors can clarify. I recommend major revision rather than rejection because the load-bearing problems are fixable by revised claims and targeted experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a solid empirical study with a modest new selection method, but the headline claim that SSL_ner beats state-of-the-art selection is not supported by the paper's own significance testing. What holds up is the first systematic benchmark of open-source Code-LLMs for code explanation and a clean comparison of token, embedding, and entity-based selection; what does not hold up is calling SSL_ner the best overall approach.\n\nThe new piece is SSL_ner: weighted Jaccard similarity over named entities extracted by UniversalNER, with weights hand-set to ignore variable/value/data-type entities. That is a reasonable, interpretable idea, and the paper makes it concrete. The evaluation is broad: four open-source Code-LLMs, two datasets (CoNaLa inline, TLC function-level with five intents), three selection strategies, three metrics. The qualitative examples are useful and show cases where entity-based selection picks more coherent demonstrations. The paper also honestly lists limitations, including the lack of entity validation, the overhead, and examples where SSL_ner does worse.\n\nThe soft spots are in the central claim. RQ4 explicitly says the differences between SSL_ner and the embedding-based Selection_semantic are 'trivial' and not statistically significant on CoNaLa and across TLC intents. The abstract says 'effectiveness over state-of-the-art methods' and the introduction says SSL_ner is the best performing SSL approach; those are overstated. What the evidence supports is 'best token-based method' (RQ3 shows significant gains over Selection_token), not best overall. The margins over Selection_semantic are small (0.003-0.02 BLEU/ROUGE/METEOR) and could be noise. Second, the entity extraction module is unvalidated; the authors say this themselves. Since the whole similarity computation depends on those entities and hand-set weights, the mechanism is unverified. That is a moderate concern, not fatal, because the method still works on the surface. Third, the code/data link is a boschdevcloud.com URL, which looks internal; for a reproducibility claim, that needs to be a real public link.\n\nOverall: the empirical work is honest, the benchmark is genuinely useful, and SSL_ner is a plausible addition to the toolbox. But the paper needs a toned-down abstract and a clearer separation between 'best token-based' and 'competitive with embeddings.' I would send it to peer review; with access to the code and a corrected significance story, it is a solid contribution. If the link is not public, that alone might be enough for a reject in practice, but the science deserves referee time.","headline":"Useful benchmark and a plausible interpretable selection method, but the 'best overall' claim is contradicted by the paper's own significance test.","tokens_in":14586,"tokens_out":2247,"would_cite":false,"duration_ms":19938,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Entity-aware selection beats token matching in code-LLM few-shot prompts","keywords":["selective shot learning","code explanation","few-shot prompting","named entity recognition","code summarization","open-source code LLMs","in-context learning","entity-based similarity"],"falsifier":"Run SSL_ner against token and embedding selection on a new code-explanation benchmark after independently validating entity extraction; if BLEU, ROUGE-L, and METEOR are unchanged when entity predictions are corrupted or weights are randomized, the claimed entity-driven gain is not real.","tokens_in":13627,"feed_emoji":"🧩","tokens_out":7232,"duration_ms":64235,"temperature":0.7,"pith_summary":"The paper tries to establish that when an open-source code-focused large language model explains a short code snippet, the choice of few-shot examples matters, and that example selection based on named code entities—functions, libraries, classes—works better than token-overlap selection and at least as well as embedding-based selection. The authors benchmark four open-source Code-LLMs on two datasets (inline Python from CoNaLa and function-level Java from TLC) and propose SSL_ner, which scores candidate examples by weighted Jaccard similarity of extracted code entities. Across most settings SSL_ner posts the highest BLEU, ROUGE-L, and METEOR scores, with gains over token selection statistically significant; the advantage over embedding-based selection is small and not statistically significant. This matters because interpretable, syntax-aware example selection could lower the cost and increase the reliability of code documentation and learning tools.","feed_headline":"Entity-aware selection beats token matching for code-LLM shots","feed_subtitle":"A new entity-based example selector beats token matching on code-explanation metrics across four open-source LLMs.","key_machinery":"The load-bearing object is the entity-based similarity score in Eq. (1): $sim_{ne}(q,d) = \\sum_{i=1}^{|E|} w_{e_i} \\cdot s_{e_i}(q,d)$, where $E$ is the set of code-entity types recognized by the extraction model, $s_{e_i}$ is the Jaccard similarity between query and candidate for entity type $e_i$, and $w_{e_i}$ is a hand-set weight (zero for data types, variables, and values, one otherwise). It is what turns raw code into a small set of interpretable semantic signposts, and it is the only component that distinguishes SSL_ner from token-based selection. The ranking produced by this score determines which (code, explanation) pairs enter the prompt, so the whole empirical gain attributed to SSL_ner flows through this equation and the entity extraction feeding it.","core_discovery":"The paper's central claim is that code entities—what a snippet imports, calls, defines, or instantiates—are a better unit of few-shot example similarity for code explanation than raw tokens or dense embeddings alone. SSL_ner extracts entities from query and candidate code with UniversalNER, then computes $sim_{ne}(q,d)$ as the sum over entity types $e_i$ of weight $w_{e_i}$ times the Jaccard similarity of the entities of that type, with weights set to zero for 'data type', 'variable', and 'value'. Across the CoNaLa and TLC datasets and four open-source models, this selection strategy yields the best BLEU/ROUGE-L/METEOR in most configurations, with the token-matching baseline consistently and significantly below it, and the embedding-based baseline close but not significantly different. The authors also interpret this behavior: entity matches such as shared libraries, method calls, and classes make the selected demonstrations visibly relevant, whereas token matching can surface superficially similar but semantically unrelated examples.","pith_inferences":["Beyond the paper, if UniversalNER's code-entity predictions were validated and weights tuned per language or per intent, SSL_ner's margin over embedding selection could widen; the authors explicitly flag entity validation as undone.","A natural extension the paper leaves implicit is combining entity and embedding similarities, since the two appear to capture complementary signals, syntax-level matching versus distributional semantics; the paper lists combining them as future work, so this is an editorial projection.","If entity weights were optimized rather than hand-set, the method could become a lightweight, interpretable alternative to retrieval-augmented prompting in production code documentation pipelines.","A testable prediction: on languages with richer named entities, such as Java with classes and methods, SSL_ner should help more than on entity-poor languages, and on highly abstract code with few recognized entities it should degrade toward token selection."],"forward_implications":["On open-source Code-LLMs, replacing random few-shot examples with token-based or entity-based selection raises explanation quality on both inline and function-level code, with medium-sized models (StarCoder 15B, CodeUp 13B) gaining more from selective shots than the largest model (CodeLlama 34B).","SSL_ner's advantage over token-overlap selection is statistically significant on both datasets, so entity-aware demonstration selection is a reliable upgrade over that baseline.","The gap between SSL_ner and embedding-based selection is not statistically significant, so both are defensible choices; embedding selection remains competitive without entity extraction.","Entity-based selection is interpretable and tunable: a user can reweight entity types, such as emphasizing class over function, to steer which examples are retrieved.","The first systematic benchmark of these four open-source Code-LLMs on code explanation establishes that CodeLlama leads zero-shot while smaller models catch up more quickly in few-shot settings."],"supporting_citations":[{"why":"Defines the token-based and embedding-based selection baselines SSL_ner is compared against, and the in-context multi-intent comment generation setup.","marker":"Geng et al., 2024"},{"why":"Provides UniversalNER, the entity extractor SSL_ner relies on to obtain code entities.","marker":"Zhou et al., 2023"},{"why":"Supplies the CoNaLa inline-Python dataset used for one of the two evaluation benchmarks.","marker":"Yin et al., 2018"},{"why":"Supplies the TLC function-level Java dataset with five code intents used as the second benchmark.","marker":"Mu et al., 2023"},{"why":"Supplies CodeLlama, the largest open-source Code-LLM evaluated in the study.","marker":"Rozière et al., 2023"},{"why":"Supplies StarCoder, the medium-sized open-source Code-LLM whose few-shot gains are highlighted.","marker":"Li, 2023"},{"why":"Establishes the k=10 few-shot exemplar setting and prior evidence that few-shot prompting helps code summarization.","marker":"Ahmed and Devanbu, 2022"},{"why":"Formulates the code explanation task and motivates LLM-generated code explanations for developers.","marker":"MacNeil et al., 2023"}],"fun_headline_variants":["Entity-based shot selection wins for code explanation across LLMs","Code entities beat tokens for choosing few-shot examples in LLMs","Open-source LLMs get better code explanations with entity-aware shots","Entity-aware example selection improves code explanation in LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the named code entities produced by UniversalNER are accurate enough and the hand-set weights on entity types are appropriate; the authors state they did not validate the entity predictions, so if extraction is noisy or the weights are misfit, SSL_ner's reported advantage could disappear.","fun_headline_variants_meta":{"raw":{"variants":["Entity-based shot selection wins for code explanation across LLMs","Code entities beat tokens for choosing few-shot examples in LLMs","Open-source LLMs get better code explanations with entity-aware shots","Entity-aware example selection improves code explanation in LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000247,"raw_usage":{"total_tokens":1528,"prompt_tokens":916,"completion_tokens":612,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":544}},"tokens_in":532,"tokens_out":612,"duration_ms":5443,"temperature":1.0,"reasoning_tokens":544,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:39:24.822990+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SSL_ner against token and embedding selection on a new code-explanation benchmark after independently validating entity extraction; if BLEU, ROUGE-L, and METEOR are unchanged when entity predictions are corrupted or weights are randomized, the claimed entity-driven gain is not real.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the token-based and embedding-based selection baselines SSL_ner is compared against, and the in-context multi-intent comment generation setup."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CoNaLa inline-Python dataset used for one of the two evaluation benchmarks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the k=10 few-shot exemplar setting and prior evidence that few-shot prompting helps code summarization."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Formulates the code explanation task and motivates LLM-generated code explanations for developers."}],"review_version":1}