Pith. sign in

REVIEW 4 major objections 5 minor 22 references

Code2API: A Tool for Generating Reusable APIs from Stack Overflow Code Snippets

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Code2API claims that a prompt-guided LLM can turn Stack Overflow snippets into reusable APIs at human level, beating the rule-based APIzator by a wide margin.

desk verdict A functional Chrome-extension demo wrapped around the authors' already-published FSE 2024 results; the extension and Python pilot are real, but the evaluation needs a validation split for prompt tuning before the 'large margin' claim can be trusted. read the letter →

arxiv 2504.14331 v1 pith:TTYCJVRE submitted 2025-04-19 cs.SE

Yubo Mai , Zhipeng Gao , Xing Hu , Lingfeng Bao , Jingyuan Chen , Jianling Sun This is my paper · ORCID
classification cs.SE
keywords APIzationStackOverflowcodesnippetslargelanguagemodelschain-of-thoughtpromptingfew-shotin-contextlearningreusableAPIsChromeextensionreuse
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Code2API is a browser extension that claims a large language model, guided by a carefully constructed prompt, can convert Stack Overflow code snippets into reusable, callable APIs. The paper argues this works because the prompt encodes the task the way a skilled developer thinks about it: an eight-step chain of thought covering imports, method naming, parameters, return values, and exceptions, plus one worked example. On a 200-API Java benchmark, Code2API is reported to beat the rule-based APIzator on parameter inference, return statements, and equivalent implementations, and judges in a user study ranked its APIs best slightly more often than human-written ones. If these results hold, developers could reuse Stack Overflow solutions in seconds instead of minutes, and the same prompt-driven recipe could be carried to other programming languages.

What carries the argument

The load-bearing object is the prompt itself: a role designation that tells the model to act as a skilled Java developer, an eight-step chain of thought that mirrors the developer's workflow, one few-shot input/output example, the test snippet with its question title and body, and format constraints on the output. Each reasoning step is a named sub-task—recover import statements, wrap the code in a public class, add public static modifiers, create a method name from context, infer parameter list, return statements, and throws statements, then output the complete code. The prompt does the work that APIzator's hand-crafted rules did; because it is expressed in natural language, the paper argues the same machinery transfers to Python by swapping the worked example and the language-specific steps.

What would settle it

A decisive check is to freeze the prompt after selecting it on a validation set, run it on a fresh, never-seen sample of Stack Overflow snippets, and compare parameter, return, and implementation accuracy against APIzator on the same sample; if the reported 11%, 9.5%, and 7.5% margins do not reappear, the central claim fails. The Python claim could be settled independently by having different annotators build the ground-truth corpus and re-measuring the 69/80/57 accuracies.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that APIzation—the transformation of a partial Stack Overflow code snippet into a named, compilable, callable method—can be automated by prompting a general-purpose LLM rather than by writing transformation rules. The paper reports that Code2API infers the correct parameter list for 66.0% of 200 Java APIs, the correct return statement for 65.0%, and an implementation equivalent to the human-written one for 43.5%, beating APIzator by 11, 9.5, and 7.5 percentage points respectively. In a blind user study, 74.3% of Code2API's method names earned the top descriptiveness score, above the human rate of 60.0%, and 50.5% of evaluators picked Code2API's API as best versus 48.0% for human-written APIs. With only the few-shot example and reasoning steps changed, the same approach reaches 69.0%, 80.0%, and 57.0% on a 100-API Python dataset, which the paper takes as evidence that the method generalizes across languages.

Load-bearing premise

The load-bearing premise is that the evaluation is unbiased—the one-shot prompt appears to have been chosen by comparing n-shot variants without a separate validation split, and the Python claim rests on 100 APIs manually constructed by the authors—so if either source of bias inflates the numbers, the claimed margin over APIzator and the generalization to Python would weaken.

Editorial extensions

If this is right

  • A developer using the extension can obtain a reusable Java API from a Stack Overflow answer in about 10 seconds, versus roughly four and a half minutes for manual APIzation.
  • Because no hand-written transformation rules are involved, extending Code2API to another language reduces to editing the few-shot example and the chain-of-thought, as demonstrated on Python.
  • The tool can be run over large corpora to produce reusable API datasets, and the paper reports creating 6,023 Java and 5,000 Python APIs this way.
  • If the user-study results are taken at face value, LLM-generated APIs are a practical substitute for human-written method names and overall API structure, not merely a faster fallback.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's claims, if APIzation is effectively solved, the bottleneck shifts to the trustworthiness of the original snippet: the generated API would inherit any bug in the Stack Overflow code, and the paper does not test generated APIs by executing them.
  • Beyond the paper's Java and Python evidence, the same eight-step reasoning trace could likely be distilled into a smaller specialized model for faster, offline, cheaper APIzation; this is an editorial extrapolation, not something the paper asserts.
  • The prompt-driven recipe is plausibly transferable to other programming languages, but the Python result alone is weak evidence because its ground truth was manually constructed by the authors; an independent multilingual benchmark would be the natural next test.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper presents Code2API, a Chrome extension that converts Stack Overflow code snippets into reusable APIs using GPT-3.5-turbo with role designation, an eight-step chain-of-thought prompt, and one-shot in-context learning. The tool extracts the question title, question body, and answer body from a Stack Overflow post, builds a prompt, and post-processes the LLM output with regular expressions. The evaluation reports that on the 200-API Java benchmark of Terragni et al., Code2API outperforms APIzator by 11, 9.5, and 7.5 percentage points on parameter-list, return-statement, and equivalent-method accuracy, respectively; a user study rates Code2API's method names and overall API quality as comparable or superior to human developers; and a generalization study on 100 manually constructed Python APIs reports accuracies of 69%, 80%, and 57%. The paper also provides a demo video, a replication package, and two large-scale generated datasets.

Significance. If the reported results are unbiased, the tool is practically valuable: it would let developers reuse Stack Overflow snippets directly by generating testable APIs with a lightweight browser extension, and the Python generalization suggests the prompt-based recipe transfers across languages. The replication package, demo video, Chrome extension, and generated datasets are concrete reproducibility artifacts and are a strength of the submission. The central significance rests on the fairness of the evaluation; as presented, the evaluation protocol is not yet convincing because the prompt's main free parameters appear to be selected on the evaluation benchmark and the Python ground truth is author-constructed.

major comments (4)
  1. [Section 2.2.3 and Section 4.1] The final 1-shot prompt was chosen by comparing the performance of different n-shot variants, but the manuscript does not state what data were used for that comparison or describe any held-out validation split. Since Section 3 sets the temperature to zero, the hand-selected few-shot examples and the n-shot count are effectively the system's only free parameters. If the n-shot comparison was performed on the same 200-API benchmark used in Section 4.1, the reported 66.0%/65.0%/43.5% accuracies and the claimed 11/9.5/7.5 percentage-point margins over APIzator, as well as the user-study results in Section 4.2, would be affected by test-set tuning. This directly underlies the abstract's 'large margin' claim and the Section 4.2 claim of being comparable or superior to human developers. Please rerun the evaluation with the final prompt on a disjoint validation set, or explicitly describe how the FSE'24 full paper [11] kept a separate validation split from prompt selection.
  2. [Section 4.1] The comparison with APIzator is not fully specified. The paper does not define how 'equivalent method implementation' was determined, and it does not state whether APIzator was rerun by the authors under identical conditions on the same 200 snippets or whether the APIzator numbers were taken from the original paper [15]. Without a precise equivalence criterion and an apples-to-apples execution protocol, the headline margins are difficult to verify. Please define the matching procedure and state explicitly whether the APIzator baseline was run by the authors on the same benchmark.
  3. [Section 4.3] The Python generalization results (69.0%, 80.0%, and 57.0%) are computed against 100 APIs that the authors 'manually constructed' and 'manually wrote,' with no inter-rater agreement, no annotation protocol, and no external baseline. Because the Python prompt was also adjusted by the authors, the claim that the framework 'can be easily extended to other programming languages without losing performance' is not yet supported by the evidence provided. Please report the annotation procedure, include a second annotator or use a public dataset, and compare against at least one baseline.
  4. [Section 4.2 and Table 1] The user study reports that 101 (50.5%) of the best APIs were generated by Code2API versus 96 (48.0%) by humans, and Table 1 shows 74.3% versus 60.0% of method names receiving the top score. The paper then concludes that Code2API is 'comparable or even superior to human developers,' but no significance test is reported. A difference of 101 versus 96 out of 200 is well within sampling variability, and the method-name scores also lack any inter-rater agreement or statistical test. Please report an appropriate significance test (e.g., an exact binomial test for the best-API counts and a suitable test for the ordinal method-name scores) and temper the conclusion unless the test supports it.
minor comments (5)
  1. [Abstract] The abstract contains two typos: 'AIPzation task' should be 'APIzation task,' and 'reusalbe' should be 'reusable.'
  2. [Section 1] The sentence 'technical their queries' should be 'their technical queries,' and the sentence 'For 66.0% and 65.0% generated APIs, APIzator and human expert extracted identical parameters and return statements respectively' is grammatically ambiguous and should be rewritten to state that Code2API's outputs were compared against the human ground truth.
  3. [Section 2] The claim that Code2API 'doesn't require any manually designed rules' conflicts with the manually crafted chain-of-thought steps, the hand-selected few-shot examples, and the format constraints described in Section 2.2. Please rephrase to 'no hand-coded transformation rules' or a similar formulation.
  4. [Table 1] The header 'MNE SCORE' is undefined; please spell out 'Method Name Evaluation Score' or define the abbreviation in the caption.
  5. [Section 2.2.2] The chain-of-thought steps were derived from only 15 snippets, but the sampling procedure for those snippets is not described; please specify how they were chosen and whether the resulting 8-step process was validated beyond the first author's discussion with the two developers.

Circularity Check

1 steps flagged · score 6.0 of 10

The few-shot prompt is selected on the same 200-API benchmark used for the reported comparison, so the claimed margin over APIzator is a test-set-tuned result rather than an independent prediction.

  1. fitted input called prediction [Section 2.2.3 (Few-Shot Learning) and Section 4.1 (Parameter Lists, Return Statements, and Equivalent Methods)]
    ""we selected five examples that best represent our needs, and through further comparison of the performance of different n-shot learning, we ultimately chose 1-shot learning to construct our final prompt." "We compared the performance of Code2API with the state-of-the-art model APIzator using the same evaluation set by Terragni et al. [15] (containing 200 APIs written by developers as ground truth).""

    The number of few-shot examples is the main free parameter of the system (temperature is fixed at 0), and the paper states this parameter was chosen by comparing performance across n-shot variants. No validation split or held-out set is described. The evaluation in Section 4.1 then reports accuracies on the same 200-API benchmark used for that comparison, so the headline numbers (66.0% parameters, 65.0% returns, 43.5% equivalent methods, and the 11/9.5/7.5 point margins over APIzator) are the result of selecting the prompt on the test set. Reporting those selected numbers as Code2API's predictive accuracy makes the central claim of outperforming APIzator at least partly self-confirming rather than an independent prediction.

full rationale

The paper contains no equations and the method itself is a straightforward LLM-prompting pipeline, so the usual self-definitional circularity is absent. The self-citation to the authors' FSE'24 paper [11] is not load-bearing: the current paper describes the prompt components and evaluation in enough detail to stand alone. The main circularity is in the evaluation protocol: Section 2.2.3 says the final 1-shot prompt was chosen 'through further comparison of the performance of different n-shot learning,' while Section 4.1 evaluates on the same 200-API set from Terragni et al. with no validation split, so the reported margins are fitted to the benchmark. The Python generalization study (Section 4.3) additionally uses an author-constructed 100-API ground truth with no reported inter-rater agreement, which is a validity concern but not a derivation-chain circularity. Overall, the central claim that Code2API 'significantly outperforms' APIzator rests on test-set-tuned prompt selection, warranting a 6 rather than a lower score; a rerun with a held-out validation split would resolve the concern.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities are postulated. The tool, prompts, and datasets are artifacts, not invented scientific entities.

free parameters (4)
  • few-shot example count (n-shot) = 1
    The paper states 'through further comparison of the performance of different n-shot learning, we ultimately chose 1-shot learning' (Sec. 2.2.3). This is a hyperparameter selected by performance on the evaluation benchmark, i.e., fitted to the test data.
  • temperature of GPT-3.5-turbo = 0
    Set to zero to ensure consistent output; this is a hand-chosen configuration that affects determinism but not accuracy per se.
  • hand-picked few-shot examples = 5 selected, 1 used
    The first author examined 100 Stack Overflow posts and selected examples covering most CoT steps (Sec. 2.2.3); the choice is subjective and not cross-validated.
  • chain-of-thought step definitions = 8 steps
    CoT was derived from two developers' reports and distilled by the first author (Sec. 2.2.2); this is a manual modeling choice that shapes all outputs.
assumptions (4)
  • domain assumption The 200 Java snippets and human-written APIs from Terragni et al. [15] are a valid and unbiased ground truth for APIzation.
    All Java accuracy numbers are measured against this set (Sec. 4.1); if the ground truth is flawed or unrepresentative, the reported improvements over APIzator are not meaningful.
  • domain assumption GPT-3.5-turbo reliably follows the hand-crafted CoT prompt and produces compilable, reusable APIs.
    The whole approach relies on the model's instruction-following and code-generation capability (Sec. 2.2, 3).
  • domain assumption Volunteer ratings of method names and API quality are a valid proxy for developer-perceived reusability.
    The user study (Sec. 4.2) uses 18 volunteers and majority voting to judge quality; if these ratings are noisy or biased, the claim of superiority over humans is unsupported.
  • ad hoc to paper The 100 Python APIs manually constructed by the authors are an unbiased ground truth for the generalization study.
    Sec. 4.3 states the dataset was 'manually constructed' by the authors, introducing possible bias in favor of the tool's expected output.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Code2API: A Tool for Generating Reusable APIs from Stack Overflow Code Snippets." pith.science (2026). https://pith.science/paper/TTYCJVRE

@misc{pith2026250414331,
  author       = {Pith},
  title        = {Pith review of: Code2API: A Tool for Generating Reusable APIs from Stack Overflow Code Snippets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TTYCJVRE}},
  note         = {Machine review of arXiv:2504.14331}
}
read the original abstract

Nowadays, developers often turn to Stack Overflow for solutions to daily problems, however, these code snippets are partial code that cannot be tested and verified properly. One way to test these code snippets is to transform them into APIs (Application Program Interface) that developers can be directly invoked and executed. However, it is often costly and error-prone for developers to manually perform this transformation (referred to as AIPzation task) due to different actions to be taken (e.g., summarizing proper method names, inferring input parameters list and return statements). To help developers quickly reuse code snippets in Stack Overflow, in this paper, we propose Code2API, a Google Chrome extension that uses Large Language Models (LLMs) to automatically perform APIzation of code snippets on Stack Overflow. \toolname guides LLMs through well-designed prompts to generate reusable APIs, using Chain-of-Thought reasoning and few-shot in-context learning to help LLMs understand and solve the APIzation task in a developer-like manner. The evaluation results show that Code2API significantly outperforms the rule-based approach by a large margin.

Figures

Figures reproduced from arXiv: 2504.14331 by the authors.

Figure 1
Figure 1. The Workflow of Our Approach. In this section, we introduce the method details of Code2API. The framework of this method is depicted in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The Overview of Code2API Chrome Extension We employ regular expressions to post-process the LLMs’ output and save the generated APIs in local files for further assessment. 3 TOOL AVAILABILITY We have implemented Code2API as a Google Chrome extension, which can be downloaded at https://doi.org/10.6084/m9.figshare. 24426961.v1. We demonstrated the process of installing Code2API Google Chrome extension and how to use o… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 13 canonical work pages

  1. [11]

    Yubo Mai, Zhipeng Gao, Xing Hu, Lingfeng Bao, Yu Liu, and JianLing Sun. 2024. Are Human Rules Necessary? Generating Reusable APIs with CoT Reasoning and In-Context Learning. Proc. ACM Softw. Eng. 1, FSE, Article 104 (July 2024), 23 pages. https://doi.org/10.1145/3660811

    work page doi:10.1145/3660811 2024

  2. [15]

    Valerio Terragni and Pasquale Salza. 2021. APIzation: Generating reusable APIs from StackOverflow code snippets. In 2021 36th IEEE/ACM International Confer- ence on Automated Software Engineering (ASE) . IEEE, 542–554

    work page 2021

  3. [1]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

    arXiv 2021

  4. [2]

    Zhenlong Dai, Bingrui Chen, Zhuoluo Zhao, Xiu Tang, Sai Wu, Chang Yao, Zhipeng Gao, and Jingyuan Chen. 2025. Less is More: Adaptive Program Re- pair with Bug Localization and Preference Learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 128–136

    work page 2025

  5. [3]

    Zhenlong Dai, Chang Yao, WenKang Han, Yuanying Yuanying, Zhipeng Gao, and Jingyuan Chen. 2024. MPCoder: Multi-user Personalized Code Generator with Explicit and Implicit Style Representation Learning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 3765–3780

    work page 2024

  6. [4]

    Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. 2023. Self-collaboration Code Generation via ChatGPT. arXiv preprint arXiv:2304.07590 (2023)

    arXiv 2023

  7. [5]

    Sidong Feng and Chunyang Chen. 2023. Prompting Is All Your Need: Automated Android Bug Replay with Large Language Models.arXiv preprint arXiv:2306.01987 (2023)

    arXiv 2023

  8. [6]

    Zhipeng Gao, Xin Xia, John Grundy, David Lo, and Yuan-Fang Li. 2020. Generat- ing question titles for stack overflow from mined code snippets.ACM Transactions on Software Engineering and Methodology (TOSEM) 29, 4 (2020), 1–37

    work page 2020

Show all 22 references
  1. [7]

    Zhipeng Gao, Xin Xia, David Lo, and John Grundy. 2020. Technical Q8A Site Answer Recommendation via Question Boosting. ACM Transactions on Software Engineering and Methodology (TOSEM) 30, 1 (2020), 1–34

    2020

  2. [8]

    Zhipeng Gao, Xin Xia, David Lo, John Grundy, Xindong Zhang, and Zhenchang Xing. 2023. I know what you are searching for: Code snippet recommenda- tion from stack overflow posts. ACM Transactions on Software Engineering and Methodology 32, 3 (2023), 1–42

    2023

  3. [9]

    Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla

  4. [10]

    Qing Huang, Yanbang Sun, Zhenchang Xing, Yuanlong Cao, Jieshan Chen, Xiwei Xu, Huan Jin, and Jiaxing Lu. 2024. Let’s Discover More API Relations: A Large Language Model-Based AI Chain for Unsupervised API Relation Inference. ACM Transactions on Software Engineering and Methodo...

    2024

  5. [12]

    Yubo Mai, Zhipeng Gao, Haoye Wang, Tingting Bi, Xing Hu, Xin Xia, and Jianling Sun. 2024. Towards Better Answers: Automated Stack Overflow Post Updating. arXiv preprint arXiv:2408.09095 (2024)

    2024 arXiv

  6. [13]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35...

    2022

  7. [14]

    Rishov Paul, Md Mohib Hossain, Mohammed Latif Siddiq, Masum Hasan, Anindya Iqbal, and Joanna CS Santos. [n.d.]. Enhancing Automated Program Repair through Fine-tuning and Prompt Engineering. ([n. d.])

  8. [16]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35 (2022), 24824–24837

    2022

  9. [17]

    Zhenchang Xing, Yang Liu, Zhuo Cheng, Qing Huang, Dehai Zhao, Daniel SUN, and Chenhua Liu. [n.d.]. When Prompt Engineering Meets Software Engineering: CNL-P as Natural and Robust" APIs”for Human-AI Interaction. In The Thirteenth International Conference on Learning Representations

  10. [18]

    Zhipeng Xue, Zhipeng Gao, Shaohua Wang, Xing Hu, Xin Xia, and Shanping Li

  11. [19]

    Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen, and Wei Cheng. 2023. Explor- ing the limits of chatgpt for query or aspect-based text summarization. arXiv preprint arXiv:2302.08081 (2023)

    2023 arXiv

  12. [20]

    Zhiqiang Yuan, Junwei Liu, Qiancheng Zi, Mingwei Liu, Xin Peng, and Yiling Lou. 2023. Evaluating instruction-tuned large language models on code compre- hension and generation. arXiv preprint arXiv:2308.01240 (2023)

    2023 arXiv

  13. [2023]

    arXiv preprint arXiv:2302.09210 (2023)

    How good are gpt models at machine translation? a comprehensive evalu- ation. arXiv preprint arXiv:2302.09210 (2023)

    2023 arXiv

  14. [2024]

    In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis

    Selfpico: Self-guided partial code execution with llms. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis . 1389–1401

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.