{"id":"2c936ae3-1a41-4757-8b00-95a277525582","arxiv_id":"2507.21285","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A fine-tuned classifier and question generator let a small coding assistant detect under-specified prompts and ask for clarification, which users rated better than a baseline in a small study.","lead":"This paper builds a coding assistant that first decides whether a developer's request is vague, and if so asks a clarifying question before writing code. The authors fine-tune a small language model on synthetic examples and report that users prefer its questions and final answers over a baseline assistant.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"External validity of both user studies is the load-bearing risk: evaluation prompts are synthesized with the same GPT-4o-mini template used for training, and RQ2 injects simulated user responses absent from the baseline, so the central preference claim may not generalize to real developer queries.","rationale":"The paper's central claim is that a trained clarification module and end-to-end pipeline improve coding assistance. This is supported by genuine user-preference studies and the approach is internally coherent, so I do not see an internal logical flaw. The decisive risk is external validity. Section 4.2 says RQ1 prompts were synthesized with the same GPT-4o-mini template as training data; Section 5.1 concedes that real GitHub issue data was too noisy and led to lower model performance. Thus the model's apparent advantage may be an artifact of distribution overlap rather than robust ability. RQ2 compounds this by comparing a pipeline that receives a GPT-4o-mini-simulated user response (effectively injecting missing constraints) against a baseline with only the original prompt; no clarification-generating baseline (e.g., ClarifyGPT) is included, so the 66-82% preference margins conflate the effect of clarification with the value of additional information. The reported Cohen's d inconsistency (medium/small in §4.3 vs >0.8 in Finding 1) is a reporting error and should be fixed, but it is not the load-bearing issue. The proposed real-prompt/real-user test would settle whether the central claim generalizes.","tokens_in":10834,"tokens_out":4816,"duration_ms":54425,"concrete_test":"Re-run RQ1 and RQ2 on a held-out set of real under-specified developer queries (e.g., Stack Overflow questions that elicited clarifying comments, or GitHub issues), with clarification responses supplied by human participants rather than GPT-4o-mini, and include ClarifyGPT or another clarification-generating baseline in RQ2. If the preference margins shrink below significance or reverse, the central claim does not generalize to real developer prompts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Both user studies are conducted on synthetic prompts generated by GPT-4o-mini using the same template as the training data (§4.2), while the Limitations section reports that real GitHub issue data was too noisy and produced lower model performance (§5.1). This makes the measured user preference potentially an artifact of train/eval distribution overlap rather than evidence that the clarification module helps on realistic developer queries. RQ2 has an additional confound: the pipeline's chat history includes a simulated user response (also GPT-4o-mini) that supplies the missing specifications, while the baseline receives only the original prompt; there is no clarification-generating baseline such as ClarifyGPT, so the reported 66-82% preference for the final answer conflates the value of the injected information with the quality of the clarification. These two linked external-validity gaps are the load-bearing weakness of the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an end-to-end LLM-based coding assistant that first classifies whether a coding prompt is under-specified and, if so, generates clarification questions before producing a final code answer. The system consists of a DistilBERT intent-clarity classifier and a LoRA-fine-tuned Gemma-3-1B-IT clarification module, both trained on synthetic prompt–clarification data generated by GPT-4o-mini. Two user studies are reported: RQ1 compares the clarification questions from the fine-tuned module against a one-shot baseline, and RQ2 compares the full pipeline's final answer (including a simulated user response) against a baseline that receives only the original prompt. The authors report strong user preferences for both the clarification questions and the final answers, and also report classifier accuracy, perplexity, and pipeline overhead results.","tokens_in":10938,"tokens_out":4617,"duration_ms":56638,"significance":"If the findings hold, the paper makes a practical contribution: it shows that a small, fine-tuned model can generate clarification questions that users prefer over a one-shot baseline, and that inserting a clarification turn can improve the perceived quality of final code answers. The authors explicitly release their code, datasets, user study templates, and models, which strengthens reproducibility. The use of a small DeciLM/Gemma-3-1B-IT model with LoRA is a reasonable efficiency-oriented design. However, the central external-validity claims rest on user studies conducted entirely on synthetic prompts generated with the same template used for training, and the RQ2 comparison is confounded by the inclusion of a simulated user response that provides extra information only to the treatment condition. These issues limit what can currently be concluded about real-world developer queries.","major_comments":[{"comment":"The external validity of both user studies is the load-bearing weakness. Section 4.2 states that the RQ1 evaluation prompts were synthesized with GPT-4o-mini using the same prompt template as the training dataset, and Section 5.1 (Internal Validity) reports that real GitHub issue data was too noisy and led to lower model performance. Consequently, the user preferences reported in Findings 1 and 2 may reflect in-distribution performance on the training distribution rather than an ability to handle realistic, out-of-distribution developer queries. To support the generalization claim, the authors should evaluate on genuinely held-out real-world queries (e.g., StackOverflow or GitHub issues) or on systematically out-of-distribution prompts, reporting classifier accuracy and clarification quality separately for that data.","section":"§4.2 and §5.1"},{"comment":"The RQ2 comparison is confounded. The treatment condition receives the original under-specified prompt, the generated clarification question, and a simulated user response that supplies the missing specifications, while the baseline receives only the original prompt before generating its final answer. The reported 66–82% preference for the pipeline's final answer could therefore be driven by the additional information in the simulated response rather than by the quality of the generated clarification question. A proper control would give the baseline the same simulated user response without the clarification question (or an oracle-prompt baseline with gold specifications), and a clarification-generating baseline such as ClarifyGPT should be included to isolate the contribution of the trained module. The paper's own Section 5.1 conclusion-validity caveat acknowledges that simulation may misrepresent real interactions, but the confound remains in the RQ2 design.","section":"§4.4 and §5.1"},{"comment":"The effect-size reporting is internally inconsistent. The RQ1 findings state a medium effect for precision and focus (d = 0.61) and small effects for the other metrics (0.3 < d < 0.5), but the same paragraph and Finding 1 claim 'large effect sizes (Cohen's d > 0.8)'. Additionally, the statistical test used to test the null hypothesis that the rating equals 3 is not named, and no correction for multiple comparisons is reported. Please reconcile the effect-size statements and specify the exact test procedure.","section":"§4.3 and Finding 1"}],"minor_comments":[{"comment":"The text says participants reviewed 10 interactions for a total of 100 unique interactions, which implies 10 participants, but the number of participants should be stated explicitly.","section":"§4.2"},{"comment":"The sentence '100% of cases were deemed at least as correct as the baseline' should be clarified by stating the breakdown, e.g., 66% better and 34% equal, so that the distribution is unambiguous.","section":"§4.5"},{"comment":"The Intent Classifier's routing threshold on the 4-point clarity scale is not reported; please describe how the 4-point scores were mapped to the binary route decision and report a small sensitivity analysis around that cutoff.","section":"§4.6"},{"comment":"The reported 4% perplexity reduction is presented without variance or a significance test; confidence intervals or a paired test would make the quantitative result more informative.","section":"§4.6"},{"comment":"ClarifyGPT is discussed in related work but is not used as a baseline in either user study; adding it as a clarification-generating baseline would strengthen the RQ1 comparison.","section":"§2.3 and §4.2"}],"recommendation":"major_revision","confidential_remarks":"The central idea is promising and the release of artifacts is commendable, but the external-validity gap and the RQ2 confound are real and load-bearing. Both are addressable with additional experiments: an out-of-distribution/real-data evaluation for the classifier and clarification module, and a properly controlled RQ2 design. The internal inconsistency in RQ1 effect-size reporting must also be corrected. I would be supportive of a revised version that addresses these points."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is an honest, clearly written systems paper with one defensible result (RQ1) and one confounded result (RQ2). The central preference claim for the final answer is not as clean as the authors present.\n\nWhat's new: they fine-tune a DistilBERT clarity classifier and a Gemma-3-1B clarification-question generator, integrate them into a loop that asks follow-ups until the prompt is clear, and run two within-subjects user studies. That is a real extension of ClarifyGPT, which used zero-shot prompting. They also claim to release code, data, and models, and they are transparent about trying GitHub issue data and failing.\n\nWhat works: RQ1 is a fair comparison — both the fine-tuned model and the baseline see the same under-specified prompt, and users rate the generated questions. The 68% preference on precision/focus is plausible. The user study is small but reasonable, and the randomization is good. The limitation section does not hide the synthetic-data dependency.\n\nWhere it gets soft: RQ2 is confounded. The pipeline's chat history includes a simulated user response that supplies the missing specification, while the baseline gets the original vague prompt and nothing else. Any preference for the pipeline's answer could come from the extra information, not from the quality of the clarification. That is the load-bearing weakness, and it is not a minor issue. The paper should compare against a baseline that receives the same augmented context, or at least include ClarifyGPT as a clarification-generating baseline.\n\nThe external-validity worry is real but acknowledged. The evaluation prompts are generated with the same GPT-4o-mini template used for training, and the authors admit real GitHub data was too noisy and lowered performance. So the measured preferences may not transfer to real developer queries. That is a serious limitation, but they say it in the Limitations section, which earns them credit.\n\nThere is also an internal inconsistency: Section 4.3 reports Cohen's d = 0.61 for precision/focus and small effects elsewhere, but the summary text and Finding 1 claim large effects (d > 0.8). That is a factual error that needs fixing, likely a leftover from an earlier draft.\n\nBottom line: this paper deserves a serious referee, but needs major revision before it can be accepted. The RQ1 result can stand; RQ2 needs an unconfounded redesign and the effect-size numbers need to be consistent. I'd send it to review with a request for those changes. I wouldn't cite the RQ2 numbers in my own work until they hold up.","headline":"An honest and clearly written systems paper with a defensible RQ1 result but a confounded RQ2 comparison; worth reviewing with major revision.","tokens_in":11532,"tokens_out":2926,"would_cite":false,"duration_ms":29763,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that an LLM coding assistant that detects vague prompts and asks a clarification question before generating code produces answers users prefer over a one-shot baseline.","keywords":["clarification questions","coding assistants","under-specified prompts","intent classification","code generation","fine-tuning","user study","LLM"],"falsifier":"Run an A/B user study on a held-out set of naturally occurring under-specified prompts collected from public issue trackers or Q&A sites, using the same rating scales; if the clarification pipeline no longer beats the one-shot baseline at a statistically significant margin on precision/focus and correctness, the central claim fails outside the synthetic distribution.","tokens_in":10550,"feed_emoji":"🤖","tokens_out":5814,"duration_ms":62240,"temperature":0.7,"pith_summary":"The paper claims that an LLM coding assistant can do better than guessing when a developer's prompt is vague: it should detect the under-specification and ask a targeted clarification question before generating code. To test this, the authors built a two-stage pipeline out of a DistilBERT clarity classifier and a fine-tuned Gemma-3-1B-IT question generator, and compared it against a one-shot baseline in two user studies. Users preferred the clarification questions in 68% of cases on precision and focus, and rated the final answers from the full pipeline higher in 82% of cases for precision and focus, 78% for contextual fit, 80% for answer faithfulness, and 66% for correctness, all statistically significant. If the result holds, interactive clarification becomes a cheap, trainable improvement for coding assistants rather than a prompt-engineering trick.","feed_headline":"Clarifying before coding wins user preference in 82% of comparisons","feed_subtitle":"A fine-tuned assistant that spots vague prompts and asks first beats a one-shot baseline on precision, fit, and faithfulness.","key_machinery":"The engine of the system is a two-stage clarification loop: a DistilBERT classifier fine-tuned on 4,161 synthetic examples rates each prompt on a four-point clarity scale and, if under-specified, routes it to a Gemma-3-1B-IT model fine-tuned with LoRA on 9,969 prompt–clarification pairs; the generated question is answered by the user, the augmented prompt is re-checked, and only then does the answering module produce code. The loop is what converts an ambiguous request into a concrete specification before generation begins.","core_discovery":"On its own terms, the paper claims that under-specified coding prompts are better handled by an explicit clarification turn than by one-shot generation. It reports that a fine-tuned Gemma-3-1B-IT clarification module, trained on 9,969 synthetic prompt–question pairs generated by GPT-4o-mini, produces questions users significantly prefer to a one-shot baseline across precision and focus (mean 3.9 out of 5, 68% favoring), immediate editability (mean 3.55, 53%), and contextual fit (mean 3.56, 45%, with 87% equivalent-or-better), at p < 0.001. When those questions are inserted into a full pipeline with a DistilBERT intent classifier and the user's answer is fed back before final code generation, users prefer the pipeline's final answer in 82% of cases on precision and focus, 78% on contextual fit, 80% on answer faithfulness, and 66% on correctness (with 100% at least as correct as the baseline), again at p < 0.001. The paper presents these findings as evidence that clarification-driven interaction is a promising direction for LLM-based coding assistants.","pith_inferences":["Because both user studies use prompts synthesized with the same template as the training data, a natural next test is to rerun the comparison on naturally occurring under-specified prompts from version-control issues or Q&A sites; the paper's own note that real GitHub data was too noisy for training suggests this distribution gap is the main open risk.","The clarification model's tendency to ask failure-handling questions (for example, what should happen if the API call fails) hints that clarification may push generated code toward defensive or security-aware patterns; a controlled vulnerability comparison with and without clarification would test this.","The same two-stage ask-then-answer loop could transfer to other under-specified generation tasks, such as data-analysis or documentation requests, where the intent gap is similarly wide."],"forward_implications":["Users prefer clarification-generated answers over one-shot answers on four quality axes, so interactive clarification can be a default behavior rather than a special mode.","A 1B-parameter model fine-tuned on synthetic pairs can generate questions users rate useful, suggesting the clarification skill is learnable without frontier-scale models.","A lightweight classifier can gate whether to ask, keeping detection overhead below 0.1 seconds; the main cost is question generation (about 133 seconds per prompt on the tested setup).","Because answer faithfulness and correctness also improve, clarification plausibly reduces the need for users to engineer prompts or supply tests upfront."],"supporting_citations":[{"why":"Supplies the base model for both the clarification module and the one-shot answering baseline.","marker":"[41]"},{"why":"Supplies the DistilBERT architecture used for the intent clarity classifier.","marker":"[36]"},{"why":"Establishes the prior zero-shot clarification approach for code generation that this work extends by fine-tuning.","marker":"[30]"},{"why":"Provides the four-point clarity rating scale adopted by the intent classifier.","marker":"[1]"},{"why":"Supports the choice to train clarification behavior on synthetic data.","marker":"[2]"},{"why":"Supports the use of synthetic data for fine-tuning when real data is noisy or hard to obtain.","marker":"[26]"},{"why":"Represents the test-driven alternative for resolving intent, which is contrasted with asking clarification questions.","marker":"[12]"},{"why":"Motivates synthetic training data by pointing out that human annotations are prone to error.","marker":"[49]"}],"fun_headline_variants":["Ask first, code better: 82% prefer clarification assistant","Coding assistant that asks questions gets 82% preference","Clarify first, then code: 82% prefer the answer","Asking before coding boosts answer preference to 82%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that synthetic under-specified prompts generated by GPT-4o-mini with the same template as the training data represent how real developers phrase vague coding requests, since both user studies are conducted on that distribution.","fun_headline_variants_meta":{"raw":{"variants":["Ask first, code better: 82% prefer clarification assistant","Coding assistant that asks questions gets 82% preference","Clarify first, then code: 82% prefer the answer","Asking before coding boosts answer preference to 82%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001072,"raw_usage":{"total_tokens":4490,"prompt_tokens":943,"completion_tokens":3547,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":3476}},"tokens_in":559,"tokens_out":3547,"duration_ms":26956,"temperature":1.0,"reasoning_tokens":3476,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T12:55:48.717875+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an A/B user study on a held-out set of naturally occurring under-specified prompts collected from public issue trackers or Q&A sites, using the same rating scales; if the clarification pipeline no longer beats the one-shot baseline at a statistically significant margin on precision/focus and correctness, the central claim fails outside the synthetic distribution.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the four-point clarity rating scale adopted by the intent classifier."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the choice to train clarification behavior on synthetic data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the use of synthetic data for fine-tuning when real data is noisy or hard to obtain."},{"cited_title":"Challenges in Trustworthy Human Evaluation of Chatbots","cited_arxiv_id":"2412.04363","evidence_quote":"Motivates synthetic training data by pointing out that human annotations are prone to error."}],"review_version":1}