Pith. sign in

REVIEW 3 major objections 6 minor 33 references

Selective Shot Learning for Code Explanation

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Pith's one-line read Entity-aware selection beats token matching in code-LLM few-shot prompts

desk verdict Useful benchmark and a plausible interpretable selection method, but the 'best overall' claim is contradicted by the paper's own significance test. read the letter →

arxiv 2412.12852 v1 pith:WPPD56MC submitted 2024-12-17 cs.SE cs.CLcs.IR

classification cs.SEcs.CLcs.IR
keywords selectiveshotlearningcodeexplanationfew-shotpromptingnamedentityrecognitionsummarizationopen-sourceLLMsin-contextentity-basedsimilarity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that when an open-source code-focused large language model explains a short code snippet, the choice of few-shot examples matters, and that example selection based on named code entities—functions, libraries, classes—works better than token-overlap selection and at least as well as embedding-based selection. The authors benchmark four open-source Code-LLMs on two datasets (inline Python from CoNaLa and function-level Java from TLC) and propose SSL_ner, which scores candidate examples by weighted Jaccard similarity of extracted code entities. Across most settings SSL_ner posts the highest BLEU, ROUGE-L, and METEOR scores, with gains over token selection statistically significant; the advantage over embedding-based selection is small and not statistically significant. This matters because interpretable, syntax-aware example selection could lower the cost and increase the reliability of code documentation and learning tools.

What carries the argument

The load-bearing object is the entity-based similarity score in Eq. (1): $sim_{ne}(q,d) = \sum_{i=1}^{|E|} w_{e_i} \cdot s_{e_i}(q,d)$, where $E$ is the set of code-entity types recognized by the extraction model, $s_{e_i}$ is the Jaccard similarity between query and candidate for entity type $e_i$, and $w_{e_i}$ is a hand-set weight (zero for data types, variables, and values, one otherwise). It is what turns raw code into a small set of interpretable semantic signposts, and it is the only component that distinguishes SSL_ner from token-based selection. The ranking produced by this score determines which (code, explanation) pairs enter the prompt, so the whole empirical gain attributed to SSL_ner flows through this equation and the entity extraction feeding it.

What would settle it

Run SSL_ner against token and embedding selection on a new code-explanation benchmark after independently validating entity extraction; if BLEU, ROUGE-L, and METEOR are unchanged when entity predictions are corrupted or weights are randomized, the claimed entity-driven gain is not real.

Watch

Extended reading notes

Core claim

The paper's central claim is that code entities—what a snippet imports, calls, defines, or instantiates—are a better unit of few-shot example similarity for code explanation than raw tokens or dense embeddings alone. SSL_ner extracts entities from query and candidate code with UniversalNER, then computes $sim_{ne}(q,d)$ as the sum over entity types $e_i$ of weight $w_{e_i}$ times the Jaccard similarity of the entities of that type, with weights set to zero for 'data type', 'variable', and 'value'. Across the CoNaLa and TLC datasets and four open-source models, this selection strategy yields the best BLEU/ROUGE-L/METEOR in most configurations, with the token-matching baseline consistently and significantly below it, and the embedding-based baseline close but not significantly different. The authors also interpret this behavior: entity matches such as shared libraries, method calls, and classes make the selected demonstrations visibly relevant, whereas token matching can surface superficially similar but semantically unrelated examples.

Load-bearing premise

The load-bearing premise is that the named code entities produced by UniversalNER are accurate enough and the hand-set weights on entity types are appropriate; the authors state they did not validate the entity predictions, so if extraction is noisy or the weights are misfit, SSL_ner's reported advantage could disappear.

Editorial extensions

If this is right

  • On open-source Code-LLMs, replacing random few-shot examples with token-based or entity-based selection raises explanation quality on both inline and function-level code, with medium-sized models (StarCoder 15B, CodeUp 13B) gaining more from selective shots than the largest model (CodeLlama 34B).
  • SSL_ner's advantage over token-overlap selection is statistically significant on both datasets, so entity-aware demonstration selection is a reliable upgrade over that baseline.
  • The gap between SSL_ner and embedding-based selection is not statistically significant, so both are defensible choices; embedding selection remains competitive without entity extraction.
  • Entity-based selection is interpretable and tunable: a user can reweight entity types, such as emphasizing class over function, to steer which examples are retrieved.
  • The first systematic benchmark of these four open-source Code-LLMs on code explanation establishes that CodeLlama leads zero-shot while smaller models catch up more quickly in few-shot settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, if UniversalNER's code-entity predictions were validated and weights tuned per language or per intent, SSL_ner's margin over embedding selection could widen; the authors explicitly flag entity validation as undone.
  • A natural extension the paper leaves implicit is combining entity and embedding similarities, since the two appear to capture complementary signals, syntax-level matching versus distributional semantics; the paper lists combining them as future work, so this is an editorial projection.
  • If entity weights were optimized rather than hand-set, the method could become a lightweight, interpretable alternative to retrieval-augmented prompting in production code documentation pipelines.
  • A testable prediction: on languages with richer named entities, such as Java with classes and methods, SSL_ner should help more than on entity-poor languages, and on highly abstract code with few recognized entities it should degrade toward token selection.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies selective-shot learning (SSL) for code explanation with open-source Code-LLMs. It benchmarks four Code-LLMs (Llama-2-Coder-7B, CodeUp-13B, StarCoder-15.5B, CodeLlama-34B) on two datasets (CoNaLa inline-level; TLC function-level) and compares three example-selection strategies: token-based Jaccard similarity, embedding-based CodeBERT cosine similarity, and a new method, SSL_ner, which uses UniversalNER to extract code entities and computes a weighted Jaccard similarity over entity types (Eq. 1). The central claim is that SSL_ner is the best-performing SSL approach, and the paper reports quantitative gains across BLEU, ROUGE-L, and METEOR. The paper also provides qualitative examples and discusses limitations and ethical considerations.

Significance. The work is the first broad benchmarking of open-source Code-LLMs for code explanation under selected few-shot examples, and it proposes a simple, interpretable selection method. If the findings are solid, the study is a useful reference for practitioners selecting open-source models and example-retrieval strategies. The evaluation is broad (4 models x 2 datasets x 5 TLC intents) and the method is simple to implement. However, the central novelty claim is weakened by the paper's own significance testing: RQ4 reports that gains over Selection_semantic are not statistically significant, and the entity-extraction component is explicitly unvalidated. These issues need to be addressed before the main claims can be accepted.

major comments (3)
  1. [Section 6 (RQ4) and Abstract/§7] The abstract and Section 7 claim that SSL_ner is 'effective' over state-of-the-art methods and is 'the best performing SSL approach,' but Section 6 (RQ4) states: 'The improvements over the best token-based method SSL_ner and Selection semantic are trivial and is not statistically significant' for CoNaLa and all TLC intents. The numerical margins are small (e.g., Table 2, CodeLlama BLEU 0.399 vs 0.395 and METEOR 0.381 vs 0.375; Table 7, 'what' METEOR 0.341 vs 0.335). Without statistically significant differences over Selection_semantic, the 'best SSL approach' claim is not supported by the evidence. The defensible claim is the one in Section 7, that SSL_ner is the best token-based method. Please report pairwise significance tests for all comparisons and revise the abstract and conclusion accordingly.
  2. [Section 4.3, Eq. (1) and Section 9] The proposed method depends critically on (i) the quality of entities extracted by UniversalNER and (ii) the hand-set weights (wei = 0 for 'data type', 'variable', 'value'). Section 9 admits: 'We did not perform a validation of the code entities predicted by the UniversalNER model.' This unvalidated component is load-bearing for SSL_ner's similarity score. Add an evaluation of entity extraction accuracy on the two datasets, an ablation that varies the entity types and weights, or a sensitivity analysis, to show that the method's advantage is not an artifact of noisy or arbitrary entity selection.
  3. [Section 5 (Prompt templates)] The CodeLlama prompt template states: 'Below is a line of python code that describes a task,' but the TLC dataset contains Java code. If the same template is used for both datasets, this language mismatch could systematically disadvantage CodeLlama on TLC and confound the results. Please clarify whether the template was adapted for Java, and if not, address this as a potential limitation.
minor comments (6)
  1. [Section 3, Table 1] The table header contains a typo, 'A verage length', and the layout of subdomain rows is hard to read; please reformat and correct.
  2. [Section 6, RQ3] The sentence 'the improvements reported are 3.8%, 7.85%, 5.84% and 2.57%' is ambiguous about whether these are relative percentages or percentage-point differences; please specify.
  3. [Section 6, RQ4] The sentence 'The improvements over the best token-based method SSL ner and Selection semnatic are trivial' is grammatically ambiguous and contains a typo ('semnatic'); clarify which method is compared with which and correct the spelling.
  4. [Abstract / Section 1] The claim of being the 'first systematic benchmarking' should be qualified relative to the existing evaluations of transformer models and LLMs on code summarization mentioned in Section 2 (e.g., Mondal et al., 2023; Dvivedi et al., 2024), to avoid overstating novelty.
  5. [Section 9 / Data availability] The data and code release link points to an internal Bosch domain (github.boschdevcloud.com); a publicly accessible repository or hosted archive should be provided to support the reproducibility claim.
  6. [Section 5 (LLM inference)] The paper sets temperature=0.7 but also do_sample=False for the Code-LLMs; please state that the sampling is deterministic or clarify the role of the temperature parameter.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: SSL_ner is an independently evaluated selection heuristic; the paper's own significance tests and limitations weaken the 'best' claim but do not make it self-referential.

full rationale

The claimed derivation chain is empirical rather than definitional. SSL_ner's similarity score in Eq. 1 is defined directly on extracted code entities with hand-set weights (zero for 'data type', 'variable', 'value'; one otherwise), and those weights are fixed design choices, not fitted to the test outputs. The evaluation on held-out CoNaLa and TLC test sets compares SSL_ner against Selection_token and Selection_semantic using BLEU, ROUGE-L, and METEOR, with the paper reporting paired t-test results for the token-based comparison. The paper's own RQ4 states that differences between SSL_ner and Selection_semantic are 'trivial' and 'not statistically significant', and the Ethical Considerations section concedes that the UniversalNER entity predictions were not validated. These are limitations on the strength of the central 'best' claim and on the mechanism's verification, but they do not make the derivation circular: no predicted quantity is constructed from the fit, no fitted parameter is renamed as a prediction, and no load-bearing premise is justified only by a self-citation. The one self-citation (Bhattacharya et al., 2023) appears in related work on LLM code explanation and is not used to justify the proposed method's effectiveness. The method is a testable hypothesis, and the paper even reports cases where it underperforms Selection_token (Table 9). Therefore no circular step is present.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the quality of UniversalNER entity extraction, the suitability of a weighted Jaccard similarity over entity types, and the choice of automatic metrics. The entity weights are hand-set rather than fitted, which keeps the method simple but leaves the design choice unvalidated.

free parameters (2)
  • Entity type weights wei = 0 for data type, variable, value; 1 otherwise
    Hand-set in Section 4.3; not fitted to data. The choice favors structural entities like functions and libraries.
  • Number of few-shot examples k = 10
    Taken from prior work (Geng et al., 2024; Ahmed & Devanbu, 2022) in Section 5; not tuned on these datasets.
assumptions (4)
  • domain assumption UniversalNER correctly labels code entities (functions, libraries, classes, etc.) for the input and training snippets.
    Used in Section 4.3, Code Entity Extraction. The authors note in Ethical Considerations that they did not validate these predictions.
  • domain assumption Weighted Jaccard similarity over entity types is an effective proxy for relevance between code snippets for explanation generation.
    This is the core hypothesis of SSL_ner in Section 4.3. The weights are set manually and not justified with data.
  • domain assumption BLEU, ROUGE-L, and METEOR adequately measure the quality of generated code explanations.
    Adopted from prior work (Geng et al., 2024; Hu et al., 2018) in Section 5. The authors acknowledge in Ethical Considerations that these metrics may not correlate with end-user satisfaction.
  • domain assumption The open-source models generate explanations in the same style and format as the ground-truth comments.
    Used throughout Section 6 without fine-tuning; the paper relies on matching n-grams between generated and reference explanations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Selective Shot Learning for Code Explanation." pith.science (2026). https://pith.science/paper/WPPD56MC

@misc{pith2026241212852,
  author       = {Pith},
  title        = {Pith review of: Selective Shot Learning for Code Explanation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WPPD56MC}},
  note         = {Machine review of arXiv:2412.12852}
}
read the original abstract

Code explanation plays a crucial role in the software engineering domain, aiding developers in grasping code functionality efficiently. Recent work shows that the performance of LLMs for code explanation improves in a few-shot setting, especially when the few-shot examples are selected intelligently. State-of-the-art approaches for such Selective Shot Learning (SSL) include token-based and embedding-based methods. However, these SSL approaches have been evaluated on proprietary LLMs, without much exploration on open-source Code-LLMs. Additionally, these methods lack consideration for programming language syntax. To bridge these gaps, we present a comparative study and propose a novel SSL method (SSL_ner) that utilizes entity information for few-shot example selection. We present several insights and show the effectiveness of SSL_ner approach over state-of-the-art methods across two datasets. To the best of our knowledge, this is the first systematic benchmarking of open-source Code-LLMs while assessing the performances of the various few-shot examples selection approaches for the code explanation task.

Figures

Figures reproduced from arXiv: 2412.12852 by the authors.

Figure 3
Figure 3. An example demonstrating the Query Code method, the top 1 demonstration example selected by [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figure 4
Figure 4. Figure demonstrating a query code sample, the top 3 examples selected by [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure demonstrating a query code sample and the top 3 examples selected by [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 23 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2020. A transformer-based approach for source code summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4998--5007

  4. [4]

    Toufique Ahmed and Premkumar Devanbu. 2022. Few-shot training llms for project-specific code-summarization. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, pages 1--5

  5. [5]

    Toufique Ahmed, Kunal Suresh Pai, Premkumar Devanbu, and Earl T Barr. 2024. Automatic semantic augmentation of language model prompts (for code summarization). In 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE), pages 1004--1004. IEEE Computer Society

  6. [6]

    Paheli Bhattacharya, Manojit Chakraborty, Kartheek NSN Palepu, Vikas Pandey, Ishan Dindorkar, Rakesh Rajpurohit, and Rishabh Gupta. 2023. Exploring large language models for code explanation. arXiv preprint arXiv:2310.16673

  7. [7]

    Ruichu Cai, Zhihao Liang, Boyan Xu, Zijian Li, Yuexing Hao, and Yao Chen. 2020. TAG : Type auxiliary guiding for code comment generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 291--301

  8. [8]

    Andor Diera, Abdelhalim Dahou, Lukas Galke, Fabian Karl, Florian Sihler, and Ansgar Scherp. 2023. Gencodesearchnet: A benchmark test suite for evaluating generalization in programming language understanding. In GenBench: The first workshop on generalisation (benchmarking) in NLP, page 12

Show all 33 references
  1. [9]

    Shubhang Shekhar Dvivedi, Vyshnav Vijay, Sai Leela Rahul Pujari, Shoumik Lodh, and Dhruv Kumar. 2024. https://doi.org/10.1145/3664646.3664765 A comparative analysis of large language models for code documentation generation . In Proceedings of the 1st ACM International Confere...

  2. [10]

    Brian P Eddy, Jeffrey A Robinson, Nicholas A Kraft, and Jeffrey C Carver. 2013. Evaluating source code summarization techniques: Replication and expansion. In 2013 21st International Conference on Program Comprehension (ICPC), pages 13--22. IEEE

  3. [11]

    Mingyang Geng, Shangwen Wang, Dezun Dong, Haotian Wang, Ge Li, Zhi Jin, Xiaoguang Mao, and Xiangke Liao. 2024. Large language models are few-shot summarizers: Multi-intent comment generation via in-context learning. In 46th International Conference on Software Engineering

  4. [12]

    Sonia Haiduc, Jairo Aponte, Laura Moreno, and Andrian Marcus. 2010. On the use of automated text summarization techniques for summarizing source code. In 2010 17th Working conference on reverse engineering, pages 35--44. IEEE

  5. [13]

    Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2018. Deep code comment generation. In Proceedings of the 26th Conference on Program Comprehension, page 200–210. Association for Computing Machinery

  6. [14]

    Yufan Huang, Mengnan Qi, Yongqiang Yao, Maoquan Wang, Bin Gu, Colin Clement, and Neel Sundaresan. 2023. Program translation via code distillation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10903--10914

  7. [15]

    Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing source code using a neural attention model. In 54th Annual Meeting of the Association for Computational Linguistics 2016, pages 2073--2083. Association for Computational Linguistics

  8. [16]

    Juyong Jiang and Sunghun Kim. 2023. Codeup: A multilingual code generation llama2 model with parameter-efficient instruction-tuning. https://huggingface.co/deepse

  9. [17]

    Haau-Sing Xiaocheng Li, Mohsen Mesgar, Andr \'e FT Martins, and Iryna Gurevych. 2023. Python code generation by asking clarification questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14287--14306

  10. [18]

    Raymond Li. 2023. Starcoder: may the source be with you! https://huggingface.co/bigcode/starcoder

  11. [19]

    Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, and Juho Leinonen. 2023. Experiences from using code explanations generated by large language models in a web software development e-book. In Proceedings of the 54th ACM Technical Sy...

  12. [20]

    Manuel Romero . 2023. https://doi.org/10.57967/hf/0931 Llama-2-coder-7b

  13. [21]

    Debanjan Mondal, Abhilasha Lodha, Ankita Sahoo, and Beena Kumari. 2023. Understanding code semantics: An evaluation of transformer models in summarization. In GenBench: The first workshop on generalisation (benchmarking) in NLP, page 65

  14. [22]

    Laura Moreno, Jairo Aponte, Giriprasad Sridhara, Andrian Marcus, Lori Pollock, and K Vijay-Shanker. 2013. Automatic generation of natural language summaries for java classes. In 2013 21st International conference on program comprehension (ICPC), pages 23--32. IEEE

  15. [23]

    Fangwen Mu, Xiao Chen, Lin Shi, Song Wang, and Qing Wang. 2023. Developer-intent driven code comment generation. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 768--780. IEEE

  16. [24]

    Baptiste Rozi \`e re, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J \'e r \'e my Rapin, et al. 2023. Code llama: Open foundation models for code. https://huggingface.co/codellama

  17. [25]

    Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023. http://arxiv.org/abs/2302.06527 An empirical evaluation of using large language models for automated unit test generation

  18. [26]

    Rishab Sharma, Fuxiang Chen, and Fatemeh Fard. 2022. Lamner: code comment generation using character language model and named entity recognition. In Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension, pages 48--59

  19. [27]

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03...

  20. [28]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. https://huggingface.co/meta-llama/Llama-2-70b-chat-hf Llama 2: Open foundation and fine-tuned chat models . ...

  21. [29]

    Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, and Xia Hu. 2023. Harnessing the power of llms in practice: A survey on chatgpt and beyond. arXiv preprint arXiv:2304.13712

  22. [30]

    Wei Ye, Rui Xie, Jinglei Zhang, Tianxiang Hu, Xiaoyin Wang, and Shikun Zhang. 2020. Leveraging code generation to improve code retrieval and summarization via dual learning. In Proceedings of The Web Conference 2020, pages 2309--2319

  23. [31]

    Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018. https://conala-corpus.github.io/ Learning to mine aligned code and natural language pairs from stack overflow . In International Conference on Mining Software Repositories, pages 476--486. ACM

  24. [32]

    Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Oleksandr Polozov, and Charles Sutton. 2023. Natural language to code generation in interactive data science notebooks. In Proceedin...

  25. [33]

    Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, and Hoifung Poon. 2023. http://arxiv.org/abs/2308.03279 Universalner: Targeted distillation from large language models for open named entity recognition

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.