{"id":"8f1e3d66-629e-4be1-9e66-a29e6d20bc57","arxiv_id":"2504.14331","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Code2API prompts GPT-3.5-turbo with chain-of-thought and few-shot examples to turn Stack Overflow snippets into reusable APIs, and reports higher quality than the rule-based APIzator and near-human performance.","lead":"This paper presents Code2API, a Chrome extension that uses a large language model to convert code snippets from Stack Overflow answers into ready-to-use APIs. The authors report that it outperforms an existing rule-based tool and sometimes matches or beats human-written APIs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The n-shot prompt was selected on the same 200-API benchmark (Section 2.2.3 vs Section 4.1), so the reported margins over APIzator may be test-set-tuned; a validation-split rerun would settle this.","rationale":"The paper's central claim is empirical: Code2API outperforms the rule-based APIzator by a large margin and approaches or exceeds human performance. The evidence for this claim is the benchmark accuracy in Section 4.1 and the user study in Section 4.2. The benchmark numbers depend on the constructed prompt, and the prompt's n-shot setting was selected by comparing performance on the same evaluation set (Section 2.2.3), with no validation split described. Because the LLM is deterministic at temperature 0, the prompt is the system's main tunable component, and selecting it on the test set is a textbook way to inflate measured accuracy. The same test-tuned prompt is used to generate the APIs shown to user-study evaluators, so the potential bias propagates to the headline 'comparable or even superior to human developers' claim. The Python generalization study is also weakened by author-constructed ground truth without inter-rater reliability, but the Java evaluation is the more load-bearing issue. The paper does have real strengths: a working Chrome extension, a replication package, and a plausible prompt-engineering approach. The concern is not fatal to the tool's usefulness, but it does mean the reported quantitative superiority may not be trustworthy. A validation-split rerun, as proposed in concrete_test, would determine whether the claimed margins hold when prompt selection is done honestly. Therefore the reader's CONDITIONAL verdict is appropriate, and no change to that verdict is needed.","tokens_in":7863,"tokens_out":4402,"duration_ms":41322,"concrete_test":"Using the provided replication package, rerun the Java experiment with a validation split for prompt selection: split the 200 snippets into 5 folds; for each fold, select the n-shot setting (0, 1, 2, or 5) that maximizes parameter and return accuracy on the other four folds' snippets, then evaluate the selected prompt on the held-out fold. Aggregate the held-out accuracy across folds and compare with APIzator's published numbers. If the aggregated Code2API margins over APIzator fall below the reported 11, 9.5, and 7.5 percentage points, or if 1-shot is not consistently selected by the validation folds, the reported superiority is attributable to test-set prompt tuning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.2.3 states that the final 1-shot prompt was chosen \"through further comparison of the performance of different n-shot learning,\" while Section 4.1 evaluates on the same 200-API benchmark used by Terragni et al. No validation split or held-out set is described anywhere in the paper. Since GPT-3.5-turbo is run with temperature 0, the prompt (including the n-shot choice and selected examples) is effectively the only free parameter of the system. Choosing that parameter on the evaluation benchmark can inflate all downstream numbers: the 66.0%/65.0%/43.5% parameter/return/equivalent-method accuracies, the 11/9.5/7.5 percentage-point margins over APIzator, and the user-study results in Section 4.2, which present APIs generated with the test-tuned prompt to evaluators. The abstract's \"large margin\" claim and the Section 4.2 claim of being \"comparable or even superior to human developers\" rest directly on these numbers. The Python generalization results in Section 4.3 additionally rely on an author-constructed ground truth with no reported inter-rater agreement, but the Java benchmark is the more load-bearing link because it supports the paper's central claim. This is not an internal inconsistency, but it is a correctness risk in the evaluation protocol that a simple validation split would address.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents Code2API, a Chrome extension that converts Stack Overflow code snippets into reusable APIs using GPT-3.5-turbo with role designation, an eight-step chain-of-thought prompt, and one-shot in-context learning. The tool extracts the question title, question body, and answer body from a Stack Overflow post, builds a prompt, and post-processes the LLM output with regular expressions. The evaluation reports that on the 200-API Java benchmark of Terragni et al., Code2API outperforms APIzator by 11, 9.5, and 7.5 percentage points on parameter-list, return-statement, and equivalent-method accuracy, respectively; a user study rates Code2API's method names and overall API quality as comparable or superior to human developers; and a generalization study on 100 manually constructed Python APIs reports accuracies of 69%, 80%, and 57%. The paper also provides a demo video, a replication package, and two large-scale generated datasets.","tokens_in":8189,"tokens_out":9483,"duration_ms":79703,"significance":"If the reported results are unbiased, the tool is practically valuable: it would let developers reuse Stack Overflow snippets directly by generating testable APIs with a lightweight browser extension, and the Python generalization suggests the prompt-based recipe transfers across languages. The replication package, demo video, Chrome extension, and generated datasets are concrete reproducibility artifacts and are a strength of the submission. The central significance rests on the fairness of the evaluation; as presented, the evaluation protocol is not yet convincing because the prompt's main free parameters appear to be selected on the evaluation benchmark and the Python ground truth is author-constructed.","major_comments":[{"comment":"The final 1-shot prompt was chosen by comparing the performance of different n-shot variants, but the manuscript does not state what data were used for that comparison or describe any held-out validation split. Since Section 3 sets the temperature to zero, the hand-selected few-shot examples and the n-shot count are effectively the system's only free parameters. If the n-shot comparison was performed on the same 200-API benchmark used in Section 4.1, the reported 66.0%/65.0%/43.5% accuracies and the claimed 11/9.5/7.5 percentage-point margins over APIzator, as well as the user-study results in Section 4.2, would be affected by test-set tuning. This directly underlies the abstract's 'large margin' claim and the Section 4.2 claim of being comparable or superior to human developers. Please rerun the evaluation with the final prompt on a disjoint validation set, or explicitly describe how the FSE'24 full paper [11] kept a separate validation split from prompt selection.","section":"Section 2.2.3 and Section 4.1"},{"comment":"The comparison with APIzator is not fully specified. The paper does not define how 'equivalent method implementation' was determined, and it does not state whether APIzator was rerun by the authors under identical conditions on the same 200 snippets or whether the APIzator numbers were taken from the original paper [15]. Without a precise equivalence criterion and an apples-to-apples execution protocol, the headline margins are difficult to verify. Please define the matching procedure and state explicitly whether the APIzator baseline was run by the authors on the same benchmark.","section":"Section 4.1"},{"comment":"The Python generalization results (69.0%, 80.0%, and 57.0%) are computed against 100 APIs that the authors 'manually constructed' and 'manually wrote,' with no inter-rater agreement, no annotation protocol, and no external baseline. Because the Python prompt was also adjusted by the authors, the claim that the framework 'can be easily extended to other programming languages without losing performance' is not yet supported by the evidence provided. Please report the annotation procedure, include a second annotator or use a public dataset, and compare against at least one baseline.","section":"Section 4.3"},{"comment":"The user study reports that 101 (50.5%) of the best APIs were generated by Code2API versus 96 (48.0%) by humans, and Table 1 shows 74.3% versus 60.0% of method names receiving the top score. The paper then concludes that Code2API is 'comparable or even superior to human developers,' but no significance test is reported. A difference of 101 versus 96 out of 200 is well within sampling variability, and the method-name scores also lack any inter-rater agreement or statistical test. Please report an appropriate significance test (e.g., an exact binomial test for the best-API counts and a suitable test for the ordinal method-name scores) and temper the conclusion unless the test supports it.","section":"Section 4.2 and Table 1"}],"minor_comments":[{"comment":"The abstract contains two typos: 'AIPzation task' should be 'APIzation task,' and 'reusalbe' should be 'reusable.'","section":"Abstract"},{"comment":"The sentence 'technical their queries' should be 'their technical queries,' and the sentence 'For 66.0% and 65.0% generated APIs, APIzator and human expert extracted identical parameters and return statements respectively' is grammatically ambiguous and should be rewritten to state that Code2API's outputs were compared against the human ground truth.","section":"Section 1"},{"comment":"The claim that Code2API 'doesn't require any manually designed rules' conflicts with the manually crafted chain-of-thought steps, the hand-selected few-shot examples, and the format constraints described in Section 2.2. Please rephrase to 'no hand-coded transformation rules' or a similar formulation.","section":"Section 2"},{"comment":"The header 'MNE SCORE' is undefined; please spell out 'Method Name Evaluation Score' or define the abbreviation in the caption.","section":"Table 1"},{"comment":"The chain-of-thought steps were derived from only 15 snippets, but the sampling procedure for those snippets is not described; please specify how they were chosen and whether the resulting 8-step process was validated beyond the first author's discussion with the two developers.","section":"Section 2.2.2"}],"recommendation":"major_revision","confidential_remarks":"This is a short tool paper that largely summarizes the authors' FSE'24 research paper [11]. The main risk is not the idea but the self-contained evaluation protocol: the n-shot selection procedure and the author-constructed Python ground truth make the headline numbers difficult to trust as presented. If the authors can add a held-out validation split, report inter-rater agreement, and provide significance tests, the tool paper would be acceptable. The editor may also wish to consider whether the tool-demo contribution is sufficiently distinct from the already-published FSE'24 paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a tool-demo paper whose scientific payload was mostly published as the authors' FSE 2024 paper (their ref [11]). What's new here is the working Chrome extension, the tool-availability artifacts, and a small Python pilot. That is a legitimate extension of previous work, not a new research result, and the paper is honest about the FSE origin in the abstract.\n\nWhat it does well: the tool is real — there is a demo video, a Figshare release, and a replication package; the design is simple and plausible; the backend uses GPT-3.5-turbo with temperature 0, which is reproducible; the Java evaluation reuses Terragni's 200-API benchmark; and the user study is blind, with majority voting. The chain-of-thought steps are drawn from asking developers to document their process, which is reasonable prompt design rather than an arbitrary hack.\n\nSoft spots, in proportion: the stress-test concern about n-shot selection lands. Section 2.2.3 says the final 1-shot prompt was chosen by comparing different n-shot configurations, and Section 4.1 evaluates on the same 200-API benchmark. No held-out validation split is described. Since temperature is 0, the prompt is effectively the only free parameter, so this is test-set tuning, and it inflates the reported margins over APIzator. The 'large margin' in the abstract also overstates the actual numbers: the exact-match gains are 11/9.5/7.5 percentage points, which are meaningful but not large in absolute terms. The Python pilot in 4.3 uses an author-constructed ground truth with no inter-rater agreement; that is a minor concern for a demo but should be disclosed. No significance tests are reported anywhere; for a five-page tool paper that is not disqualifying, but it is another reason to phrase claims carefully, especially the 'comparable or even superior to human developers' line, which rests on a user study with 18 volunteers and 3 voters per item.\n\nWho this is for: people working on LLM-based code reuse or Stack Overflow mining will want to know the tool exists. The FSE paper is the right citation for the method; this paper is the right citation for the Chrome extension and the Python evidence. It deserves serious peer review as a tool demo, assuming the reviewers ask the authors to add a validation split or at least explicitly acknowledge the tuning, and to soften the overclaims.","headline":"A functional Chrome-extension demo wrapped around the authors' already-published FSE 2024 results; the extension and Python pilot are real, but the evaluation needs a validation split for prompt tuning before the 'large margin' claim can be trusted.","tokens_in":8710,"tokens_out":2476,"would_cite":false,"duration_ms":21290,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Code2API claims that a prompt-guided LLM can turn Stack Overflow snippets into reusable APIs at human level, beating the rule-based APIzator by a wide margin.","keywords":["APIzation","Stack Overflow code snippets","large language models","chain-of-thought prompting","few-shot in-context learning","reusable APIs","Chrome extension","code reuse"],"falsifier":"A decisive check is to freeze the prompt after selecting it on a validation set, run it on a fresh, never-seen sample of Stack Overflow snippets, and compare parameter, return, and implementation accuracy against APIzator on the same sample; if the reported 11%, 9.5%, and 7.5% margins do not reappear, the central claim fails. The Python claim could be settled independently by having different annotators build the ground-truth corpus and re-measuring the 69/80/57 accuracies.","tokens_in":7660,"feed_emoji":"🧩","tokens_out":7545,"duration_ms":64480,"temperature":0.7,"pith_summary":"Code2API is a browser extension that claims a large language model, guided by a carefully constructed prompt, can convert Stack Overflow code snippets into reusable, callable APIs. The paper argues this works because the prompt encodes the task the way a skilled developer thinks about it: an eight-step chain of thought covering imports, method naming, parameters, return values, and exceptions, plus one worked example. On a 200-API Java benchmark, Code2API is reported to beat the rule-based APIzator on parameter inference, return statements, and equivalent implementations, and judges in a user study ranked its APIs best slightly more often than human-written ones. If these results hold, developers could reuse Stack Overflow solutions in seconds instead of minutes, and the same prompt-driven recipe could be carried to other programming languages.","feed_headline":"LLM turns Stack Overflow snippets into ready-to-use APIs","feed_subtitle":"Prompt-guided Code2API beats the rule-based APIzator on 200 Java APIs and rivals human-written code.","key_machinery":"The load-bearing object is the prompt itself: a role designation that tells the model to act as a skilled Java developer, an eight-step chain of thought that mirrors the developer's workflow, one few-shot input/output example, the test snippet with its question title and body, and format constraints on the output. Each reasoning step is a named sub-task—recover import statements, wrap the code in a public class, add public static modifiers, create a method name from context, infer parameter list, return statements, and throws statements, then output the complete code. The prompt does the work that APIzator's hand-crafted rules did; because it is expressed in natural language, the paper argues the same machinery transfers to Python by swapping the worked example and the language-specific steps.","core_discovery":"On the paper's own terms, the central claim is that APIzation—the transformation of a partial Stack Overflow code snippet into a named, compilable, callable method—can be automated by prompting a general-purpose LLM rather than by writing transformation rules. The paper reports that Code2API infers the correct parameter list for 66.0% of 200 Java APIs, the correct return statement for 65.0%, and an implementation equivalent to the human-written one for 43.5%, beating APIzator by 11, 9.5, and 7.5 percentage points respectively. In a blind user study, 74.3% of Code2API's method names earned the top descriptiveness score, above the human rate of 60.0%, and 50.5% of evaluators picked Code2API's API as best versus 48.0% for human-written APIs. With only the few-shot example and reasoning steps changed, the same approach reaches 69.0%, 80.0%, and 57.0% on a 100-API Python dataset, which the paper takes as evidence that the method generalizes across languages.","pith_inferences":["Beyond the paper's claims, if APIzation is effectively solved, the bottleneck shifts to the trustworthiness of the original snippet: the generated API would inherit any bug in the Stack Overflow code, and the paper does not test generated APIs by executing them.","Beyond the paper's Java and Python evidence, the same eight-step reasoning trace could likely be distilled into a smaller specialized model for faster, offline, cheaper APIzation; this is an editorial extrapolation, not something the paper asserts.","The prompt-driven recipe is plausibly transferable to other programming languages, but the Python result alone is weak evidence because its ground truth was manually constructed by the authors; an independent multilingual benchmark would be the natural next test."],"forward_implications":["A developer using the extension can obtain a reusable Java API from a Stack Overflow answer in about 10 seconds, versus roughly four and a half minutes for manual APIzation.","Because no hand-written transformation rules are involved, extending Code2API to another language reduces to editing the few-shot example and the chain-of-thought, as demonstrated on Python.","The tool can be run over large corpora to produce reusable API datasets, and the paper reports creating 6,023 Java and 5,000 Python APIs this way.","If the user-study results are taken at face value, LLM-generated APIs are a practical substitute for human-written method names and overall API structure, not merely a faster fallback."],"supporting_citations":[{"why":"Provides the rule-based APIzator baseline and the 200-API Java dataset with human-written ground truth that Code2API is scored against.","marker":"[15]"},{"why":"The full research paper that defines the APIzation method and evaluation on which this tool paper builds.","marker":"[11]"},{"why":"Supplies the chain-of-thought prompting technique that the paper adapts into the eight-step developer reasoning trace.","marker":"[16]"},{"why":"Shows the process of eliciting developer reasoning traces that the authors follow when constructing their chain-of-thought examples.","marker":"[5]"},{"why":"Establishes GPT-3.5-turbo as a state-of-the-art instruction-tuned model and justifies its use as the backend LLM.","marker":"[13]"}],"fun_headline_variants":["LLM-driven Code2API converts Stack Overflow snippets to callable APIs","Chrome extension uses LLM to auto-generate APIs from Stack Overflow code","Code2API's LLM outperforms rule-based APIzation of code snippets","From Stack Overflow snippet to API: Code2API's LLM approach wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the evaluation is unbiased—the one-shot prompt appears to have been chosen by comparing n-shot variants without a separate validation split, and the Python claim rests on 100 APIs manually constructed by the authors—so if either source of bias inflates the numbers, the claimed margin over APIzator and the generalization to Python would weaken.","fun_headline_variants_meta":{"raw":{"variants":["LLM-driven Code2API converts Stack Overflow snippets to callable APIs","Chrome extension uses LLM to auto-generate APIs from Stack Overflow code","Code2API's LLM outperforms rule-based APIzation of code snippets","From Stack Overflow snippet to API: Code2API's LLM approach wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001243,"raw_usage":{"total_tokens":5120,"prompt_tokens":982,"completion_tokens":4138,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":4055}},"tokens_in":598,"tokens_out":4138,"duration_ms":25709,"temperature":1.0,"reasoning_tokens":4055,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:50:50.064615+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check is to freeze the prompt after selecting it on a validation set, run it on a fresh, never-seen sample of Stack Overflow snippets, and compare parameter, return, and implementation accuracy against APIzator on the same sample; if the reported 11%, 9.5%, and 7.5% margins do not reappear, the central claim fails. The Python claim could be settled independently by having different annotators build the ground-truth corpus and re-measuring the 69/80/57 accuracies.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the rule-based APIzator baseline and the 200-API Java dataset with human-written ground truth that Code2API is scored against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The full research paper that defines the APIzation method and evaluation on which this tool paper builds."}],"review_version":1}