{"id":"a61b154d-c445-4cd7-bb7a-78ab0dc22975","arxiv_id":"2504.16292","paper_version":1,"verdict":"UNVERDICTED","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":4,"one_line_summary":"GENCNIPPET is a planned browser plugin that would use a fine-tuned Llama-3 model to generate code snippets for Stack Overflow questions missing them, with evaluation plans but no implemented results.","lead":"This paper describes GENCNIPPET, a proposed browser plugin that would generate example code snippets for Stack Overflow questions that lack them. The plan proposes fine-tuning Llama-3 on Stack Overflow question-code pairs and evaluating the tool against ground truth, experts, and real users.","discovery_kind":"unclear","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training/evaluation inputs contain the code snippets the model is supposed to generate, so the planned experiments do not test the actual missing-snippet use case.","rationale":"The reader's weakest assumption concerned the validity of using the existing code snippet as ground truth; that is a real concern about evaluation validity. My concern is more directly about the task formulation: the data used for training and evaluation comes from questions that have code snippets, while the central claim concerns questions that lack them. If the model input includes the target code, the proposed experiments test copying, not generation. If the input excludes code, the distribution shift between complete and incomplete questions remains unaddressed. Either way, the planned evaluation would not support the headline claim. This does not change the reader's verdict that the paper is unverifiable without implementation, so UNCHANGED is appropriate, but it identifies a specific methodological soft spot that should be fixed before any experiments are run.","tokens_in":9704,"tokens_out":4443,"duration_ms":46936,"concrete_test":"Take a held-out set of 200 questions from the paper's planned dataset. For each question, create two input variants: (A) the original question body as used in the proposed fine-tuning, and (B) the same body with all code blocks removed. Run the fine-tuned model on both variants and compute the planned automatic metrics against the original code snippet. Then have two human raters judge relevance of the generated snippets for variant B only. If variant A scores much higher than B, or if B's snippets are rated irrelevant, the proposed evaluation does not reflect the missing-snippet use case. Also inspect the actual training instances to verify whether code blocks are present in the 'problem description' inputs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that GENCNIPPET will generate relevant code snippets for SO questions that lack them. However, the proposed dataset consists entirely of questions that already contain code snippets (Section III-A, Table I: 'Questions with Code Snippets'), and the fine-tuning input/output pairs are 'problem descriptions and their corresponding code snippets' (Section III-B). The paper never states that code blocks are removed from the problem descriptions before being fed to the model. If the code snippet remains in the input, the model can learn to copy it, making the automatic evaluation (ROUGE, BLEU, BERTScore against that same snippet, Section III-C) a measure of copying rather than generation. Even if code blocks are stripped, the training descriptions are written for questions that include code and often reference it (e.g., 'in the following code'), so they are not representative of the descriptions of questions that omit code. Thus the proposed training and evaluation protocol, as described, cannot support the claim that the tool will generate relevant snippets for the intended deployment scenario.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes GENCNIPPET, a browser plug-in for Stack Overflow (SO) that is intended to generate example code snippets for user questions that lack them. The authors plan to filter SO questions that require code snippets and have positive scores, fine-tune a Llama-3-8B model on problem-description/ground-truth-snippet pairs, and evaluate the generated snippets using automatic metrics (ROUGE, BLEU, BERTScore), manual expert review, and a live wild test. The paper presents dataset construction statistics (558,552 questions across Java and Python), a detailed fine-tuning and evaluation plan, the tool architecture, and a survey design for user studies. No experiments, model training, or evaluation results are reported; the entire contribution is a proposal in future tense.","tokens_in":10059,"tokens_out":3138,"duration_ms":32180,"significance":"If the proposed approach worked, GENCNIPPET could address a real problem documented in the software engineering literature: SO questions that miss required code snippets receive acceptable answers far less often (23.8% vs. 61.4%) and with greater delay. The paper is clearly motivated and the planned evaluation is unusually comprehensive, including user studies and wild testing. The authors also exhibit good awareness of threats to validity and limitations of automatic metrics. However, the manuscript contains no empirical evidence whatsoever, and, more importantly, the proposed training and evaluation protocol has a fundamental mismatch with the deployment scenario: the model is trained and evaluated on questions that already contain code snippets, but the tool is intended to serve questions that omit code. As submitted, the paper is a research plan rather than a completed study, so its significance cannot yet be assessed.","major_comments":[{"comment":"The training and evaluation protocol does not match the deployment use case. The dataset consists exclusively of questions that already contain code snippets (Table I: 'Questions with Code Snippets'), and the fine-tuning input/output pairs are 'problem descriptions and their corresponding code snippets' (Section III-B). The manuscript never states that code blocks are stripped from the problem description before being fed to the model. If code remains in the input, the model can learn to copy it, and the automatic evaluation against that same snippet (ROUGE/BLEU/BERTScore, Section III-C) would measure copying fidelity, not generation relevance. If code is stripped, the remaining descriptions are still written for questions that include code and often reference it (e.g., 'in the following code'), so they are not representative of descriptions of questions that omit code. The proposed protocol, as described, cannot support the central claim that GENCNIPPET will generate relevant snippets for questions that miss them.","section":"III-A, III-B, III-C"},{"comment":"The paper contains no experiments, no results, and no validation. The abstract states 'We plan to introduce GENCNIPPET', and the methodology is phrased entirely in future tense ('We will fine-tune', 'We will evaluate', 'will be assessed'). The central claim that the fine-tuned model will generate relevant code snippets is asserted, not demonstrated. For a full research paper, this is a proposal rather than a completed study, and the reader cannot verify any of the claims about the tool's effectiveness.","section":"Entire manuscript"},{"comment":"The automatic evaluation against a single ground-truth snippet is not a valid measure of snippet relevance for the intended task. For a given problem description, many different code snippets may be equally relevant and correct; ROUGE, BLEU, and BERTScore reward surface similarity to one reference and will penalize equally valid alternative snippets. The manual review of 400 samples is a reasonable complement, but it is still applied to questions that already have code, so it does not assess the tool's behavior on questions that lack code. The planned wild test on recent questions without code is the only evaluation component that matches the deployment scenario, but it is only a plan and is not reported.","section":"III-C"},{"comment":"The comparison with foundation models (GPT-4, CodeLlama) uses 'the same set of questions from Section III-C', i.e., questions already containing code snippets. This evaluates the models' ability to reproduce existing snippets, not their ability to generate snippets for questions that miss them. Since the research questions RQ1 and RQ2 explicitly target 'questions that miss required code snippets', the proposed comparison does not directly answer those research questions.","section":"III-D"}],"minor_comments":[{"comment":"Reference [14] is cited as 'StackExchage. StackExchage API' with a URL to data.stackexchange.com; the spelling and the URL-to-citation match should be corrected.","section":"References"},{"comment":"Figure 3 shows a mock interface with a small matplotlib example. It would help readers if the figure caption explained the source of this example and noted it is illustrative only, not an output of the proposed system.","section":"IV-A, Fig. 3"},{"comment":"The weighted Net Promoter Score formula uses response scores 2, 1, 0, -1, -2, which differs from the standard 0-10 NPS scale. The text should justify this choice or clarify the mapping to the standard NPS calculation.","section":"IV-C, Eq. (1)"},{"comment":"The term 'Open Circular' is capitalized without explanation; this appears to be a label for open recruitment and should be described more precisely.","section":"IV-C"},{"comment":"The paper states 'we will incorporate more recent data up to the point when we begin fine-tuning', but the dataset statistics in Table I are fixed as of April 2024. It would be clearer to state whether the reported counts are final or preliminary.","section":"III-A"}],"recommendation":"reject","confidential_remarks":"The manuscript is a well-organized research proposal but contains no empirical content, which alone would normally warrant rejection for a full research paper. More fundamentally, the planned training and evaluation design has a train/deploy mismatch: it trains and tests on questions that already contain code snippets while the tool is meant to serve questions that lack them. This is not a minor fixable flaw; it affects the validity of the core evaluation. The paper might be suitable for a 'new ideas' or 'vision' track, or as a basis for a revised submission after the authors actually run the experiments and redesign the evaluation to target questions without code snippets."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this if you want to see a concrete LLM-for-SO proposal with a clear blind spot. The paper is a plan, not a study: no model, no experiments, no data analysis. The motivation is solid — questions missing code get far fewer answers, prior work identified them, and generating examples could help. Credit where due: the dataset construction is reasonable (filtering via a learned classifier, positive-score questions, single snippets for low noise), fine-tuning Llama-3 with LoRA is sensible, and the three-way evaluation (automatic metrics, manual review, wild testing with suggested edits) shows they understand the limits of similarity scores alone. The citation pattern is fine; the self-citations point to their own prior classifier, which is the natural basis.\n\nThe soft spot is load-bearing. The dataset consists of questions that already contain code snippets (Section III-A), and the fine-tuning pairs are \"problem descriptions and their corresponding code snippets\" (Section III-B). The paper never says code is stripped from the problem description before training. If it isn't, the model learns to copy the snippet, and automatic evaluation against that same snippet measures copying. If it is stripped, the descriptions are still written for questions that include code and often reference it, so they don't look like the real missing-code cases the tool targets. Either way, the proposed evaluation does not test the actual use case. Manual review and wild testing are good ideas but remain promises. This is not a minor flaw; it's the central claim.\n\nOther soft spots are minor. Restricting to single-snippet positive-score questions may create a clean but unrepresentative training set. The comparison with GPT-4 is standard. The survey plan is detailed but 20 participants is small, and snowball sampling is biased; they acknowledge this.\n\nWho is this for? Someone working on question-quality tools for Q&A forums. The paper gives a clear design to critique, but there is no result to verify. I would not cite it in the next year. If it were submitted as a full paper, I would desk reject it — not enough substance. The authors would be better served by a workshop or a registered report where reviewers can shape the experimental design before the work is done.","headline":"A well-motivated proposal for generating missing code snippets on Stack Overflow, but the planned training and evaluation never actually test the missing-snippet scenario, and the paper contains no results.","tokens_in":10374,"tokens_out":2782,"would_cite":false,"duration_ms":26574,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes GENCNIPPET, a browser plug-in that uses a fine-tuned Llama-3 model to generate relevant example code snippets for Stack Overflow questions that lack them, aiming to help those questions get timely answers.","keywords":["Stack Overflow","question quality","code snippets","code generation","fine-tuning","Llama-3","browser plugin","user study"],"falsifier":"Execute 100 generated snippets against the problem descriptions they are meant to support and record how many reproduce the reported fault; a majority failing to reproduce it would falsify the central claim that GENCNIPPET generates relevant supporting examples.","tokens_in":9525,"feed_emoji":"💻","tokens_out":8309,"duration_ms":74456,"temperature":0.7,"pith_summary":"The paper tackles a documented problem: Stack Overflow questions that omit code snippets receive acceptable answers far less often than questions that include them, and many askers omit code because of time pressure, employer policy, confidentiality, or uncertainty about what to share. It proposes GENCNIPPET, a browser plug-in that reads the question's problem description and programming language, sends them to a fine-tuned Llama-3 model, and returns a short example code snippet the asker can paste into the question. The authors plan to fine-tune on hundreds of thousands of high-scoring Java and Python questions, each paired with the single snippet it contains, and to evaluate output through similarity metrics, expert review of 400 samples, and live submissions of 50 generated snippets to Stack Overflow as suggested edits. The paper is a plan: it specifies the architecture and evaluation rather than reporting trained-model results. If the proposal works, it would lower a common barrier to getting help without replacing the human Q&A platform.","feed_headline":"Planned tool writes missing code snippets for Stack Overflow","feed_subtitle":"A fine-tuned Llama-3 browser plug-in would give askers a relevant example snippet when they cannot share their own.","key_machinery":"The central object is the fine-tuned snippet generator: Llama-3-8B adapted with LoRA on structured pairs of (question text, language, date) → (code snippet), where the snippets come from positive-scoring questions that genuinely need code, as judged by an existing machine-learning filter. Around that core sits GENCNIPPET's two-part architecture: a browser-client script captures the problem description and language tag, a server builds a prompt asking for a concise example that demonstrates the problem 'including intentional faults,' and the model's output is returned to the user for pasting into the question. The evaluation machinery is three-layered: ROUGE, BLEU, and BERTScore for an initial similarity screen; a 400-sample expert review on a 5-point Likert scale; and a live test that submits generated snippets as Stack Overflow suggested edits and tracks acceptance and response times.","core_discovery":"The paper's central claim is that an open-weights language model tailored to Stack Overflow's question style can generate a relevant, concise example snippet for a question that lacks one, and that supplying that snippet during submission improves the question's chances of a timely and appropriate answer. The authors ground this in earlier findings that only 23.8% of snippet-missing questions receive acceptable answers, versus 61.4% of questions that include code, and they design GENCNIPPET around the missing piece: a fine-tuned Llama-3-8B that takes a problem description, language tag, and date as input and produces 'Code: [code]' as output. The paper does not yet present evidence that the model can generate such snippets; it lays out how that evidence will be obtained, through automatic similarity scores, manual expert ratings, and a wild test in which 50 generated snippets are submitted as suggested edits on Stack Overflow.","pith_inferences":["A stronger test than similarity metrics would be execution: run each generated snippet against the described problem and check whether it reproduces the reported error; the paper's planned evaluation does not include this check, and the training objective of matching one attached snippet is only a proxy for usefulness.","Because the prompt explicitly asks for code that contains the intentional fault, a natural extension is paired generation: produce the failing snippet together with a corrected version, which would serve diagnosis as well as the learning goals Stack Overflow is meant to support.","The same pipeline could generalize beyond Stack Overflow to issue trackers and discussion forums, where missing minimal reproductions are a common barrier; the paper limits itself to Java and Python questions on one platform."],"forward_implications":["If the fine-tuned model performs as planned, askers who cannot share real code can still post questions with a relevant example snippet, and the known gap in acceptable-answer rates between snippet-missing and snippet-including questions should narrow.","A working GENCNIPPET would strengthen Stack Overflow as a human-curated knowledge base instead of substituting automated answers for expert discussion, preserving the community-review mechanism.","The planned comparison against zero-shot and few-shot GPT-4 and CodeLlama would quantify how much domain-specific fine-tuning adds over general-purpose code generation for this task.","If community moderators accept a substantial share of the 50 wild-test suggested edits, that acceptance would be direct evidence that generated snippets meet real Stack Overflow quality expectations."],"supporting_citations":[{"why":"Establishes that programmers ask and answer questions on the web, setting up the premise that question completeness matters for getting help.","marker":"[8]"},{"why":"Load-bearing: supplies the filter for questions that genuinely require code snippets and the statistic that snippet-missing questions get acceptable answers only 23.8% of the time versus 61.4%.","marker":"[10]"},{"why":"Provides evidence-based guidelines showing that well-specified questions receive better answers, motivating the addition of generated snippets.","marker":"[11]"},{"why":"Shows that quality questions need quality code fragments, supporting the claim that snippets are required for resolution.","marker":"[12]"},{"why":"Documents the scale of unanswered Stack Overflow questions, framing the problem GENCNIPPET addresses.","marker":"[13]"},{"why":"The Stack Exchange data dump is the source of the 3.37 million Java and Python questions used to build the dataset.","marker":"[14]"},{"why":"Gives insight into unresolved Stack Overflow questions, supporting the motivation for improving question completeness.","marker":"[15]"}],"fun_headline_variants":["AI writes missing code for Stack Overflow","Llama-3 plug-in auto-generates code snippets","Tool crafts code examples for code-less SO questions","Planned AI fills code gaps on Stack Overflow","Auto-snippet generator for SO posts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The plan rests on treating the single code snippet attached to each high-scoring question as the correct, representative example: if those snippets are arbitrary or unrepresentative, the model learns to mimic them rather than to produce something genuinely useful.","fun_headline_variants_meta":{"raw":{"variants":["AI writes missing code for Stack Overflow","Llama-3 plug-in auto-generates code snippets","Tool crafts code examples for code-less SO questions","Planned AI fills code gaps on Stack Overflow","Auto-snippet generator for SO posts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000511,"raw_usage":{"total_tokens":2541,"prompt_tokens":1053,"completion_tokens":1488,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":669,"completion_tokens_details":{"reasoning_tokens":1419}},"tokens_in":669,"tokens_out":1488,"duration_ms":12701,"temperature":1.0,"reasoning_tokens":1419,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:05:53.257898+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Execute 100 generated snippets against the problem descriptions they are meant to support and record how many reproduce the reported fault; a majority failing to reproduce it would falsify the central claim that GENCNIPPET generates relevant supporting examples.","supporting_citations":[{"cited_title":"Treude, O","cited_arxiv_id":null,"evidence_quote":"Establishes that programmers ask and answer questions on the web, setting up the premise that question completeness matters for getting help."},{"cited_title":"Can we identify stack overflow ques- tions requiring code snippets? investigating the cause & effect of missing code snippets","cited_arxiv_id":null,"evidence_quote":"Load-bearing: supplies the filter for questions that genuinely require code snippets and the statistic that snippet-missing questions get acceptable answers only 23.8% of the time versus 61.4%."},{"cited_title":"Calefato, F","cited_arxiv_id":null,"evidence_quote":"Provides evidence-based guidelines showing that well-specified questions receive better answers, motivating the addition of generated snippets."},{"cited_title":"Quality questions need quality code: Classifying code fragments on stack overflow","cited_arxiv_id":null,"evidence_quote":"Shows that quality questions need quality code fragments, supporting the claim that snippets are required for resolution."},{"cited_title":"Asaduzzaman, A","cited_arxiv_id":null,"evidence_quote":"Documents the scale of unanswered Stack Overflow questions, framing the problem GENCNIPPET addresses."},{"cited_title":"StackExchage API, 2024","cited_arxiv_id":null,"evidence_quote":"The Stack Exchange data dump is the source of the 3.37 million Java and Python questions used to build the dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives insight into unresolved Stack Overflow questions, supporting the motivation for improving question completeness."}],"review_version":1}