REVIEW 3 major objections 6 minor 33 references
Selective Shot Learning for Code Explanation
T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Entity-aware selection beats token matching in code-LLM few-shot prompts
desk verdict Useful benchmark and a plausible interpretable selection method, but the 'best overall' claim is contradicted by the paper's own significance test. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the entity-based similarity score in Eq. (1): $sim_{ne}(q,d) = \sum_{i=1}^{|E|} w_{e_i} \cdot s_{e_i}(q,d)$, where $E$ is the set of code-entity types recognized by the extraction model, $s_{e_i}$ is the Jaccard similarity between query and candidate for entity type $e_i$, and $w_{e_i}$ is a hand-set weight (zero for data types, variables, and values, one otherwise). It is what turns raw code into a small set of interpretable semantic signposts, and it is the only component that distinguishes SSL_ner from token-based selection. The ranking produced by this score determines which (code, explanation) pairs enter the prompt, so the whole empirical gain attributed to SSL_ner flows through this equation and the entity extraction feeding it.
What would settle it
Run SSL_ner against token and embedding selection on a new code-explanation benchmark after independently validating entity extraction; if BLEU, ROUGE-L, and METEOR are unchanged when entity predictions are corrupted or weights are randomized, the claimed entity-driven gain is not real.
Extended reading notes
Core claim
The paper's central claim is that code entities—what a snippet imports, calls, defines, or instantiates—are a better unit of few-shot example similarity for code explanation than raw tokens or dense embeddings alone. SSL_ner extracts entities from query and candidate code with UniversalNER, then computes $sim_{ne}(q,d)$ as the sum over entity types $e_i$ of weight $w_{e_i}$ times the Jaccard similarity of the entities of that type, with weights set to zero for 'data type', 'variable', and 'value'. Across the CoNaLa and TLC datasets and four open-source models, this selection strategy yields the best BLEU/ROUGE-L/METEOR in most configurations, with the token-matching baseline consistently and significantly below it, and the embedding-based baseline close but not significantly different. The authors also interpret this behavior: entity matches such as shared libraries, method calls, and classes make the selected demonstrations visibly relevant, whereas token matching can surface superficially similar but semantically unrelated examples.
Load-bearing premise
The load-bearing premise is that the named code entities produced by UniversalNER are accurate enough and the hand-set weights on entity types are appropriate; the authors state they did not validate the entity predictions, so if extraction is noisy or the weights are misfit, SSL_ner's reported advantage could disappear.
Editorial extensions
If this is right
- On open-source Code-LLMs, replacing random few-shot examples with token-based or entity-based selection raises explanation quality on both inline and function-level code, with medium-sized models (StarCoder 15B, CodeUp 13B) gaining more from selective shots than the largest model (CodeLlama 34B).
- SSL_ner's advantage over token-overlap selection is statistically significant on both datasets, so entity-aware demonstration selection is a reliable upgrade over that baseline.
- The gap between SSL_ner and embedding-based selection is not statistically significant, so both are defensible choices; embedding selection remains competitive without entity extraction.
- Entity-based selection is interpretable and tunable: a user can reweight entity types, such as emphasizing class over function, to steer which examples are retrieved.
- The first systematic benchmark of these four open-source Code-LLMs on code explanation establishes that CodeLlama leads zero-shot while smaller models catch up more quickly in few-shot settings.
Reading between the lines
- Beyond the paper, if UniversalNER's code-entity predictions were validated and weights tuned per language or per intent, SSL_ner's margin over embedding selection could widen; the authors explicitly flag entity validation as undone.
- A natural extension the paper leaves implicit is combining entity and embedding similarities, since the two appear to capture complementary signals, syntax-level matching versus distributional semantics; the paper lists combining them as future work, so this is an editorial projection.
- If entity weights were optimized rather than hand-set, the method could become a lightweight, interpretable alternative to retrieval-augmented prompting in production code documentation pipelines.
- A testable prediction: on languages with richer named entities, such as Java with classes and methods, SSL_ner should help more than on entity-poor languages, and on highly abstract code with few recognized entities it should degrade toward token selection.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies selective-shot learning (SSL) for code explanation with open-source Code-LLMs. It benchmarks four Code-LLMs (Llama-2-Coder-7B, CodeUp-13B, StarCoder-15.5B, CodeLlama-34B) on two datasets (CoNaLa inline-level; TLC function-level) and compares three example-selection strategies: token-based Jaccard similarity, embedding-based CodeBERT cosine similarity, and a new method, SSL_ner, which uses UniversalNER to extract code entities and computes a weighted Jaccard similarity over entity types (Eq. 1). The central claim is that SSL_ner is the best-performing SSL approach, and the paper reports quantitative gains across BLEU, ROUGE-L, and METEOR. The paper also provides qualitative examples and discusses limitations and ethical considerations.
Significance. The work is the first broad benchmarking of open-source Code-LLMs for code explanation under selected few-shot examples, and it proposes a simple, interpretable selection method. If the findings are solid, the study is a useful reference for practitioners selecting open-source models and example-retrieval strategies. The evaluation is broad (4 models x 2 datasets x 5 TLC intents) and the method is simple to implement. However, the central novelty claim is weakened by the paper's own significance testing: RQ4 reports that gains over Selection_semantic are not statistically significant, and the entity-extraction component is explicitly unvalidated. These issues need to be addressed before the main claims can be accepted.
major comments (3)
- [Section 6 (RQ4) and Abstract/§7] The abstract and Section 7 claim that SSL_ner is 'effective' over state-of-the-art methods and is 'the best performing SSL approach,' but Section 6 (RQ4) states: 'The improvements over the best token-based method SSL_ner and Selection semantic are trivial and is not statistically significant' for CoNaLa and all TLC intents. The numerical margins are small (e.g., Table 2, CodeLlama BLEU 0.399 vs 0.395 and METEOR 0.381 vs 0.375; Table 7, 'what' METEOR 0.341 vs 0.335). Without statistically significant differences over Selection_semantic, the 'best SSL approach' claim is not supported by the evidence. The defensible claim is the one in Section 7, that SSL_ner is the best token-based method. Please report pairwise significance tests for all comparisons and revise the abstract and conclusion accordingly.
- [Section 4.3, Eq. (1) and Section 9] The proposed method depends critically on (i) the quality of entities extracted by UniversalNER and (ii) the hand-set weights (wei = 0 for 'data type', 'variable', 'value'). Section 9 admits: 'We did not perform a validation of the code entities predicted by the UniversalNER model.' This unvalidated component is load-bearing for SSL_ner's similarity score. Add an evaluation of entity extraction accuracy on the two datasets, an ablation that varies the entity types and weights, or a sensitivity analysis, to show that the method's advantage is not an artifact of noisy or arbitrary entity selection.
- [Section 5 (Prompt templates)] The CodeLlama prompt template states: 'Below is a line of python code that describes a task,' but the TLC dataset contains Java code. If the same template is used for both datasets, this language mismatch could systematically disadvantage CodeLlama on TLC and confound the results. Please clarify whether the template was adapted for Java, and if not, address this as a potential limitation.
minor comments (6)
- [Section 3, Table 1] The table header contains a typo, 'A verage length', and the layout of subdomain rows is hard to read; please reformat and correct.
- [Section 6, RQ3] The sentence 'the improvements reported are 3.8%, 7.85%, 5.84% and 2.57%' is ambiguous about whether these are relative percentages or percentage-point differences; please specify.
- [Section 6, RQ4] The sentence 'The improvements over the best token-based method SSL ner and Selection semnatic are trivial' is grammatically ambiguous and contains a typo ('semnatic'); clarify which method is compared with which and correct the spelling.
- [Abstract / Section 1] The claim of being the 'first systematic benchmarking' should be qualified relative to the existing evaluations of transformer models and LLMs on code summarization mentioned in Section 2 (e.g., Mondal et al., 2023; Dvivedi et al., 2024), to avoid overstating novelty.
- [Section 9 / Data availability] The data and code release link points to an internal Bosch domain (github.boschdevcloud.com); a publicly accessible repository or hosted archive should be provided to support the reproducibility claim.
- [Section 5 (LLM inference)] The paper sets temperature=0.7 but also do_sample=False for the Code-LLMs; please state that the sampling is deterministic or clarify the role of the temperature parameter.
Circularity Check
No circularity: SSL_ner is an independently evaluated selection heuristic; the paper's own significance tests and limitations weaken the 'best' claim but do not make it self-referential.
full rationale
The claimed derivation chain is empirical rather than definitional. SSL_ner's similarity score in Eq. 1 is defined directly on extracted code entities with hand-set weights (zero for 'data type', 'variable', 'value'; one otherwise), and those weights are fixed design choices, not fitted to the test outputs. The evaluation on held-out CoNaLa and TLC test sets compares SSL_ner against Selection_token and Selection_semantic using BLEU, ROUGE-L, and METEOR, with the paper reporting paired t-test results for the token-based comparison. The paper's own RQ4 states that differences between SSL_ner and Selection_semantic are 'trivial' and 'not statistically significant', and the Ethical Considerations section concedes that the UniversalNER entity predictions were not validated. These are limitations on the strength of the central 'best' claim and on the mechanism's verification, but they do not make the derivation circular: no predicted quantity is constructed from the fit, no fitted parameter is renamed as a prediction, and no load-bearing premise is justified only by a self-citation. The one self-citation (Bhattacharya et al., 2023) appears in related work on LLM code explanation and is not used to justify the proposed method's effectiveness. The method is a testable hypothesis, and the paper even reports cases where it underperforms Selection_token (Table 9). Therefore no circular step is present.
Assumptions & free parameters
free parameters (2)
- Entity type weights wei =
0 for data type, variable, value; 1 otherwise
- Number of few-shot examples k =
10
assumptions (4)
- domain assumption UniversalNER correctly labels code entities (functions, libraries, classes, etc.) for the input and training snippets.
- domain assumption Weighted Jaccard similarity over entity types is an effective proxy for relevance between code snippets for explanation generation.
- domain assumption BLEU, ROUGE-L, and METEOR adequately measure the quality of generated code explanations.
- domain assumption The open-source models generate explanations in the same style and format as the ground-truth comments.
Cite this review
Pith. "Pith review of Selective Shot Learning for Code Explanation." pith.science (2026). https://pith.science/paper/WPPD56MC
@misc{pith2026241212852,
author = {Pith},
title = {Pith review of: Selective Shot Learning for Code Explanation},
year = {2026},
howpublished = {\url{https://pith.science/paper/WPPD56MC}},
note = {Machine review of arXiv:2412.12852}
}
read the original abstract
Code explanation plays a crucial role in the software engineering domain, aiding developers in grasping code functionality efficiently. Recent work shows that the performance of LLMs for code explanation improves in a few-shot setting, especially when the few-shot examples are selected intelligently. State-of-the-art approaches for such Selective Shot Learning (SSL) include token-based and embedding-based methods. However, these SSL approaches have been evaluated on proprietary LLMs, without much exploration on open-source Code-LLMs. Additionally, these methods lack consideration for programming language syntax. To bridge these gaps, we present a comparative study and propose a novel SSL method (SSL_ner) that utilizes entity information for few-shot example selection. We present several insights and show the effectiveness of SSL_ner approach over state-of-the-art methods across two datasets. To the best of our knowledge, this is the first systematic benchmarking of open-source Code-LLMs while assessing the performances of the various few-shot examples selection approaches for the code explanation task.
Figures
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2020. A transformer-based approach for source code summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4998--5007
work page 2020
-
[4]
Toufique Ahmed and Premkumar Devanbu. 2022. Few-shot training llms for project-specific code-summarization. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, pages 1--5
work page 2022
-
[5]
Toufique Ahmed, Kunal Suresh Pai, Premkumar Devanbu, and Earl T Barr. 2024. Automatic semantic augmentation of language model prompts (for code summarization). In 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE), pages 1004--1004. IEEE Computer Society
work page 2024
-
[6]
Paheli Bhattacharya, Manojit Chakraborty, Kartheek NSN Palepu, Vikas Pandey, Ishan Dindorkar, Rakesh Rajpurohit, and Rishabh Gupta. 2023. Exploring large language models for code explanation. arXiv preprint arXiv:2310.16673
arXiv 2023
-
[7]
Ruichu Cai, Zhihao Liang, Boyan Xu, Zijian Li, Yuexing Hao, and Yao Chen. 2020. TAG : Type auxiliary guiding for code comment generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 291--301
work page 2020
-
[8]
Andor Diera, Abdelhalim Dahou, Lukas Galke, Fabian Karl, Florian Sihler, and Ansgar Scherp. 2023. Gencodesearchnet: A benchmark test suite for evaluating generalization in programming language understanding. In GenBench: The first workshop on generalisation (benchmarking) in NLP, page 12
work page 2023
Show all 33 references
-
[9]
Shubhang Shekhar Dvivedi, Vyshnav Vijay, Sai Leela Rahul Pujari, Shoumik Lodh, and Dhruv Kumar. 2024. https://doi.org/10.1145/3664646.3664765 A comparative analysis of large language models for code documentation generation . In Proceedings of the 1st ACM International Confere...
2024
-
[10]
Brian P Eddy, Jeffrey A Robinson, Nicholas A Kraft, and Jeffrey C Carver. 2013. Evaluating source code summarization techniques: Replication and expansion. In 2013 21st International Conference on Program Comprehension (ICPC), pages 13--22. IEEE
2013
-
[11]
Mingyang Geng, Shangwen Wang, Dezun Dong, Haotian Wang, Ge Li, Zhi Jin, Xiaoguang Mao, and Xiangke Liao. 2024. Large language models are few-shot summarizers: Multi-intent comment generation via in-context learning. In 46th International Conference on Software Engineering
2024
-
[12]
Sonia Haiduc, Jairo Aponte, Laura Moreno, and Andrian Marcus. 2010. On the use of automated text summarization techniques for summarizing source code. In 2010 17th Working conference on reverse engineering, pages 35--44. IEEE
2010
-
[13]
Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2018. Deep code comment generation. In Proceedings of the 26th Conference on Program Comprehension, page 200–210. Association for Computing Machinery
2018
-
[14]
Yufan Huang, Mengnan Qi, Yongqiang Yao, Maoquan Wang, Bin Gu, Colin Clement, and Neel Sundaresan. 2023. Program translation via code distillation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10903--10914
2023
-
[15]
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing source code using a neural attention model. In 54th Annual Meeting of the Association for Computational Linguistics 2016, pages 2073--2083. Association for Computational Linguistics
2016
-
[16]
Juyong Jiang and Sunghun Kim. 2023. Codeup: A multilingual code generation llama2 model with parameter-efficient instruction-tuning. https://huggingface.co/deepse
2023
-
[17]
Haau-Sing Xiaocheng Li, Mohsen Mesgar, Andr \'e FT Martins, and Iryna Gurevych. 2023. Python code generation by asking clarification questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14287--14306
2023
-
[18]
Raymond Li. 2023. Starcoder: may the source be with you! https://huggingface.co/bigcode/starcoder
2023
-
[19]
Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, and Juho Leinonen. 2023. Experiences from using code explanations generated by large language models in a web software development e-book. In Proceedings of the 54th ACM Technical Sy...
2023
-
[20]
Manuel Romero . 2023. https://doi.org/10.57967/hf/0931 Llama-2-coder-7b
2023 doi
-
[21]
Debanjan Mondal, Abhilasha Lodha, Ankita Sahoo, and Beena Kumari. 2023. Understanding code semantics: An evaluation of transformer models in summarization. In GenBench: The first workshop on generalisation (benchmarking) in NLP, page 65
2023
-
[22]
Laura Moreno, Jairo Aponte, Giriprasad Sridhara, Andrian Marcus, Lori Pollock, and K Vijay-Shanker. 2013. Automatic generation of natural language summaries for java classes. In 2013 21st International conference on program comprehension (ICPC), pages 23--32. IEEE
2013
-
[23]
Fangwen Mu, Xiao Chen, Lin Shi, Song Wang, and Qing Wang. 2023. Developer-intent driven code comment generation. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 768--780. IEEE
2023
-
[24]
Baptiste Rozi \`e re, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J \'e r \'e my Rapin, et al. 2023. Code llama: Open foundation models for code. https://huggingface.co/codellama
2023
-
[25]
Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023. http://arxiv.org/abs/2302.06527 An empirical evaluation of using large language models for automated unit test generation
2023 arXiv
-
[26]
Rishab Sharma, Fuxiang Chen, and Fatemeh Fard. 2022. Lamner: code comment generation using character language model and named entity recognition. In Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension, pages 48--59
2022
-
[27]
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03...
2023
-
[28]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. https://huggingface.co/meta-llama/Llama-2-70b-chat-hf Llama 2: Open foundation and fine-tuned chat models . ...
2023 arXiv
-
[29]
Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, and Xia Hu. 2023. Harnessing the power of llms in practice: A survey on chatgpt and beyond. arXiv preprint arXiv:2304.13712
2023 arXiv
-
[30]
Wei Ye, Rui Xie, Jinglei Zhang, Tianxiang Hu, Xiaoyin Wang, and Shikun Zhang. 2020. Leveraging code generation to improve code retrieval and summarization via dual learning. In Proceedings of The Web Conference 2020, pages 2309--2319
2020
-
[31]
Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018. https://conala-corpus.github.io/ Learning to mine aligned code and natural language pairs from stack overflow . In International Conference on Mining Software Repositories, pages 476--486. ACM
2018
-
[32]
Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Oleksandr Polozov, and Charles Sutton. 2023. Natural language to code generation in interactive data science notebooks. In Proceedin...
2023
-
[33]
Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, and Hoifung Poon. 2023. http://arxiv.org/abs/2308.03279 Universalner: Targeted distillation from large language models for open named entity recognition
2023 arXiv
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.