{"id":"486b94e9-c4fd-49d1-883b-75c48b4c1101","arxiv_id":"2607.22898","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"AssumptionMiner extracts implicit assumptions from LLM-generated code with a cross-model ensemble F1 of 0.816 on a new 180-task benchmark, and enables assumption-guided targeted regeneration.","lead":"This paper introduces AssumptionMiner, a framework that makes the implicit assumptions hidden in LLM-generated code explicit, inspectable, and editable through targeted regeneration. It includes a new 180-task benchmark showing far better assumption extraction than offline baselines, while honestly reporting that exact decision matching and cascading edits remain unsolved.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.816 F1 headline is computed on 180 tasks, but only 30 are human-verified; an unverified HumanEval/MBPP tier may be inflating the result.","rationale":"The reader's weakest assumption identifies the same underlying concern: the reference annotations are GPT-4o-generated, and human verification covers only a small subset. My attack sharpens this into a concrete, testable failure mode: the aggregate F1 mixes human-verified and unverified annotations, and the paper does not report per-tier results. The paper is transparent about the circularity and contamination risks, and the human verification on the curated subset is a genuine strength. However, the headline claim is presented without qualification as a single number for all 180 tasks, and the unverified tier could dominate it. The proposed test is directly feasible from the released artifact and would settle whether the 3.6x improvement holds on the human-verified subset. Since the reader's conditions for acceptance already include expanding human verification and reporting stricter metrics, my concern does not change the CONDITIONAL verdict; it reinforces the necessity of those conditions. I therefore recommend UNCHANGED, noting that the paper should be accepted only if the per-tier analysis supports the headline or the claim is revised to be tier-specific.","tokens_in":26217,"tokens_out":6062,"duration_ms":62730,"concrete_test":"Using the released benchmark (data/benchmark.json), recompute the CW ensemble F1 (category-level and strict) restricted to the 30 curated tasks and their 109 human-verified references. Separately, have two independent developers (neither an author) apply the released annotation protocol to a random sample of 30 extended-tier tasks. If the curated-only F1 is more than ~0.05 below 0.816, or if the sample validity rate is below ~95%, the headline 3.6x claim should be reported with per-tier breakdowns and qualified as applying to the unverified extended tier.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To support the central claim that the confidence-weighted ensemble 'improves on the strongest offline baseline by 3.6x', the reference annotations A* must be a valid ground truth for implicit assumptions. The paper constructs A* with GPT-4o using the same extraction prompt as AssumptionMiner, and human-verifies only the 30-task curated subset (109 assumptions). The other 150 tasks are drawn from HumanEval/MBPP — likely in the training data of all evaluated models — and their 567 reference assumptions are not human-verified. The paper reports no per-tier F1, but the per-category table (Table IV) shows the rare categories T4/T6 (which occur only in the curated subset) have F1 around 0.5–0.67, well below the aggregate 0.816. If the ensemble's F1 on the curated 30 tasks is materially lower than 0.816, the headline improvement is an artifact of easy, stereotyped tasks plus unverified reference labels. The paper's own threat analysis (§VIII) acknowledges the circularity and contamination, but the absence of a per-tier breakdown means the reader cannot tell how much of the headline is carried by the unverified tier. This is the most load-bearing gap because the central claim is stated as a single aggregate number, and the unverified tier dominates the aggregate (150/180 tasks).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AssumptionMiner, an LLM-based framework that takes a natural-language prompt and the code an LLM generated for it, and produces an explicit 'assumption layer' of implicit design decisions (categorized into a six-part taxonomy), links each assumption to the code regions that implement it via a keyword-overlap plus AST-narrowing dependency mapper, and supports targeted regeneration when a developer revises an assumption. It also introduces a benchmark of 180 ambiguous programming tasks with 676 GPT-4o-generated reference assumptions, of which 109 assumptions across 30 tasks are human-verified. The evaluation reports three results: (i) assumption extraction, where a confidence-weighted ensemble of Qwen2.5-Coder-32B and DeepSeek-Coder-V2-Lite reaches F1 = 0.816 under category-level matching (F1 = 0.662 under a stricter decision-level metric); (ii) dependency localization, where AST-guided mapping yields tighter regions than keyword matching, though threshold accuracy is similar; and (iii) assumption-guided regeneration, which produces smaller edits than full regeneration or in-place editing but with lower syntactic validity (83.3% vs 100%).","tokens_in":26459,"tokens_out":9284,"duration_ms":100062,"significance":"If the extraction results are robust, AssumptionMiner is a useful contribution: it turns an invisible side effect of LLM code generation into a structured, inspectable, and revisable artifact, and it provides a public benchmark, source code, and reproducibility package. The paper is unusually careful in several respects: it reports bootstrap confidence intervals, distinguishes lenient category-level from strict decision-level matching, acknowledges the self-consistency risk of using GPT-4o as annotator and evaluator, and performs human verification on a curated subset. The honest disclosure of the strict F1 drop to 0.66 and the RQ3 syntactic-validity gap is a strength. However, because the reference labels for 150 of the 180 tasks are generated by GPT-4o with the same extraction prompt used by AssumptionMiner, and because no per-tier breakdown is reported, the magnitude of the headline 0.816 F1 and the 3.6x improvement over baselines is not yet established. The paper is best read as a framework-plus-benchmark contribution whose central quantitative claim requires additional validation.","major_comments":[{"comment":"The headline RQ1 result is reported on the full 180-task benchmark, but 150 of those tasks (567 of 676 reference assumptions) have reference annotations produced by GPT-4o using the same structured extraction prompt as the system under evaluation (Appendix A), and these annotations are not human-verified. The paper acknowledges this in §VIII, but it does not report AR/AP/F1 separately for the 30-task human-verified curated subset and the 150-task extended tier. This is load-bearing because the aggregate F1 = 0.816 is dominated by the unverified, potentially easier and contaminated HumanEval/MBPP tier, and Table IV suggests the curated-only categories T4/T6 have F1 values of only 0.50–0.67 for the open-source backbones. The authors should report per-tier extraction results for every backbone and the ensemble, including confidence intervals, and state explicitly whether the 3.6x improvemen","section":"Table III, §V, §VIII"},{"comment":"The reference assumption set A* is generated with the same two-phase extraction prompt (Appendix A) that AssumptionMiner itself uses. The human-verified portion covers only 109 assumptions on 30 tasks, and one of the two annotators is the author. Consequently, the reported F1 scores measure agreement between an extractor and the annotating model's own framing of 'implicit assumptions'; they do not independently establish that the extracted items correspond to what developers would regard as important hidden decisions. The strict decision-level metric mitigates this in part, but the primary abstract/conclusion claims are based on the lenient category-level metric. The authors should either (a) human-verify a representative sample of the extended tier, or (b) explicitly reframe the headline claim as 'agreement with a GPT-4o-derived reference' rather than as fidelity to developer intent, an","section":"§III-B, §V, Definition 1"},{"comment":"The 'category-level (lenient)' matching rule counts a predicted assumption as correct whenever its taxonomy category matches an unused reference category for the same task. This is essentially a category-coverage metric, not a measure of whether the same design decision was identified. The paper acknowledges this in §VIII and reports the stricter decision-level metric, but the abstract's headline 'F1 = 0.816' is not qualified as category-level at the point where it is introduced. Since the strict decision-level F1 is 0.662, the gap between the two metrics is large and should be foregrounded whenever the 0.816 number is cited. Please add the word 'category-level' to the abstract's numeric claim, or move the strict metric into the headline statement.","section":"§VI-A, category-level matching"}],"minor_comments":[{"comment":"Reference [16] contains the literal placeholder 'tODO: complete author list from arXiv:2504.16331' and must be completed. Reference [27] (AVATAR) is cited for 'automated design-rationale extraction' in §VII-D, but AVATAR is a program-repair paper; the citation does not match the claim.","section":"References [16] and [27]"},{"comment":"The contribution statement C2 says AssumptionMiner produces 'code and a formalized assumption layer in a single augmented generation step,' but the architecture in §III-A separates code generation (C2) from assumption extraction (C3), which is a second LLM call. Please rephrase to avoid implying joint decoding.","section":"§I and §III-A"},{"comment":"The semantic-similarity threshold τ = 0.56 is calibrated on only 30 author-labeled pairs with 80% agreement. This small, single-annotator calibration is not listed in the threats-to-validity section; please add it, or provide a second annotator for the calibration pairs.","section":"§VI-A, §VIII"},{"comment":"The 'Calls' column lists '1' for both AM-LLM rows in the full-set columns, which is ambiguous: it presumably means one additional LLM call per assumption, not one call total. Please clarify in the caption or table notes.","section":"Table V caption"},{"comment":"The notation alternates between AF1 and F1 (e.g., Tables III–IV vs. the text). Please use a single notation or define both at first use.","section":"Throughout"},{"comment":"The abstract says 'improving on the strongest offline baseline by 3.6x.' This is a ratio of F1 scores (0.816/0.225), not a relative improvement in error rate; since the baseline is near zero and the matching is category-level, the ratio can be misleading. Consider reporting the absolute gain and the strict F1 as well.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The central empirical claim is not fully supported by the current evidence because the reference annotations are LLM-generated with the same prompt as the evaluated system and the human-verified subset covers only 30 of 180 tasks. However, the framework, benchmark, and open-source artifact are valuable, and the per-tier results are readily computable from the released code. This is therefore a major revision rather than a rejection: the authors can address the main concern by adding per-tier breakdowns and by qualifying the headline claim. The citation errors (references [16] and [27]) should be fixed before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is good and worth engaging with: making the assumptions an LLM bakes into generated code a first-class, inspectable, revisable artifact. That is a real gap, and the paper's framing against clarification questions and self-repair is fair. The AST-based dependency graph and targeted regeneration are sensible technical pieces, and the release of a benchmark, code, and replication package is a seriousplus.\n\nWhat the paper actually delivers: a taxonomy that is plausible if not proven complete, a 180-task benchmark, an extraction method that clearly beats non-LLM baselines, and a dependency mapper that gives tighter regions than keyword matching on the human-verified subset. It also does the right thing by reporting a strict decision-level F1 (0.66), per-category breakdowns, and a detailed error analysis. The transparency is unusual and welcome.\n\nThe soft spots are real and not fully neutralized by that transparency. The 0.816 F1 headline is computed over all 180 tasks, but 150 of them are HumanEval/MBPP tasks whose reference assumptions were produced by GPT-4o using the very same extraction prompt as the system being evaluated. Those tasks are also almost certainly in the training data of every evaluated model, so they are the easy, stereotyped tier. Only 30 tasks are human-verified, and even there one of the two annotators is the author. The paper never reports per-tier F1, so the reader cannot tell how much of the 3.6x improvement over baselines is carried by the unverified tier. The per-category table hints that rare categories like T4/T6, which exist only in the curated set, score far below the aggregate. This is a load-bearing gap, and the stress-test note is correct to flag it.\n\nI also note that the dependency mapper's category-to-node-type map was tuned on the evaluation set. The ablation suggests the tuning was worth only a couple of points, so this is a minor concern, not a fatal one. The bigger issue for RQ3 is that syntactic validity at 83.3% versus 100% for the non-targeted baselines, with no functional testing, means the regeneration claim is really \"smaller edits, but less safe\" -- which the paper does say plainly.\n\nWho should read this: anyone working on LLM code generation, requirements engineering, or human-AI collaboration for code. The framework and benchmark are worth having even if the empirical results need to be re-interpreted. It deserves a serious referee, but the referee should insist on per-tier results, a larger and more independent human-verification sample, and functional testing of regenerated code before the headline numbers are taken at face value.","headline":"A genuinely new framework and benchmark for surfacing implicit assumptions in LLM-generated code, but the headline F1 is carried by an unverified, likely contaminated tier and should be read as upper-bound evidence.","tokens_in":26994,"tokens_out":1610,"would_cite":true,"duration_ms":19034,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"When an LLM writes code, it silently commits to assumptions that never appear in the prompt; this paper's AssumptionMiner surfaces those decisions, links each to the code it governs, and regenerates only the affected region when a developer","keywords":["assumption mining","LLM code generation","implicit assumptions","code traceability","AST dependency graph","incremental regeneration","code adaptability","benchmark"],"falsifier":"Ask a group of professional developers to independently list what they consider the most important hidden decisions in a set of LLM-generated code samples, then run AssumptionMiner on the same samples and compare. If the majority of developer-flagged decisions are absent from the extracted assumption layer, or if the surfaced assumptions are mostly judged trivial or irrelevant, the central claim—that making assumptions explicit improves transparency and controllability—would fail.","tokens_in":26009,"feed_emoji":"🔍","tokens_out":2952,"duration_ms":31675,"temperature":0.7,"pith_summary":"The paper argues that LLM-generated code is full of implicit assumptions—design decisions the model makes to fill gaps in underspecified prompts—and that these hidden choices are a major source of 'working-but-wrong' code that passes tests yet violates developer intent. It presents AssumptionMiner, a framework that makes these assumptions an explicit, structured output of code generation, linked to the exact code regions they govern through an AST-based dependency graph. The paper backs this with a benchmark of 180 ambiguous tasks and 676 annotated assumptions, showing that a confidence-weighted ensemble of open-source LLMs can extract assumptions at an F1 of 0.816, far above offline baselines. It also demonstrates that when a developer revises an assumption, targeted regeneration changes far less code than full regeneration, though with a cost in syntactic validity. The sympathetic reader sees a practical path toward more inspectable and controllable AI-generated code: rather than hoping the model guesses right, treat its guesses as first-class artifacts that can be reviewed and corrected.","feed_headline":"LLM code hides assumptions; tool surfaces them at 82% F1","feed_subtitle":"AssumptionMiner extracts the unstated design choices in AI-generated code, links them to code lines, and rewrites only what an edited assump","key_machinery":"The load-bearing machinery is the AssumptionRecord schema combined with the two-phase extraction prompt and the AST-based dependency graph. The extraction prompt first asks the LLM to reason about which choices in the code go beyond what the prompt requires, then to format its discoveries into JSON records with category, description, alternatives, and confidence—the two-phase structure is what outperforms direct generation, clarifying questions, and plain chain-of-thought in preliminary experiments. The dependency mapper then uses a category-to-node-type map to select AST nodes whose identifiers overlap with keywords from the assumption's description, narrows to the minimal enclosing subtree","core_discovery":"The central discovery is that implicit assumptions in LLM-generated code are not incidental noise but a structural consequence of collapsing requirements-to-design-to-implementation into a single opaque step, and that they can be systematically surfaced as structured AssumptionRecords. Each record captures the assumption's category (from a six-part taxonomy covering input validation, data format, error policy, persistence, performance, and security), a natural-language description, a rationale, at least one realistic alternative, and a confidence score. The records are linked to the code regions they govern via a two-pass dependency mapper (keyword-overlap candidate selection then minimal-su","pith_inferences":["If assumption extraction generalized beyond Python and short single-file tasks, it could turn IDE code assistants from black-box generators into tools that surface 'why this code looks this way'—a form of automated design rationale that may reduce debugging time for developers unfamiliar with the generated artifact.","The conflict between targeted regeneration and cascading edits suggests a natural next mechanism: a post-revision pass that re-validates all dependency links and flags regions invalidated by a change, rather than the current single-pass splice; such a pass would likely close most of the 17-point syntactic-validity gap.","The extraction F1 of 0.816 is measured against an LLM-produced reference; a user study where practicing developers judge the usefulness of the surfaced assumptions would test whether the benchmark's 'implicit assumption' construct maps to what developers actually care about in real code review.","Because the confidence-weighted ensemble outperforms either backbone alone by combining a recall-biased and a precision-biased model, the same ensembling trick may transfer to other LLM-augmented engineering tasks where a trade-off between coverage and exactness is the bottleneck."],"forward_implications":["Making assumptions explicit gives developers a way to audit and correct LLM-generated code without re-prompting from scratch, which should reduce the rate of working-but-wrong code shipped to production.","The dependency graph turns a revised assumption into a surgical edit: regenerating only the governed region changes 1.3–2.4x less code than full regeneration or manual reprompting, so reviewers can reason about the blast radius of a decision change.","The benchmark's strict decision-level metric (F1 = 0.66 for the best cross-model configuration) shows that extracting the exact decision, not just its category, remains an open problem—future systems must improve description fidelity, not just category coverage.","Because every revision is traceable to a specific AssumptionRecord, the framework naturally produces a live decision log for generated code, which could serve as automated design documentation and support parallel, independent revisions."],"fun_headline_variants":["AssumptionMiner surfaces LLM code's hidden assumptions at 82% F1","Make AI code's implicit assumptions explicit and editable — AssumptionMiner","LLM code assumptions exposed: 82% F1 and targeted rewrites","Uncover the unstated in LLM code — AssumptionMiner hits 82% F1","AssumptionMiner: 82% F1 on finding and fixing AI code assumptions"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark's reference annotations are produced by the same kind of model that is being evaluated (GPT-4o) using the same extraction prompt, and only 30 of the 180 tasks have human-verified references—so the reported F1 scores may measure how well the extractor agrees with an LLM's framing of 'implicit assumption' rather than how well it surfaces decisions that human developers would actually consider important.","fun_headline_variants_meta":{"raw":{"variants":["AssumptionMiner surfaces LLM code's hidden assumptions at 82% F1","Make AI code's implicit assumptions explicit and editable — AssumptionMiner","LLM code assumptions exposed: 82% F1 and targeted rewrites","Uncover the unstated in LLM code — AssumptionMiner hits 82% F1","AssumptionMiner: 82% F1 on finding and fixing AI code assumptions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000312,"raw_usage":{"total_tokens":1636,"prompt_tokens":793,"completion_tokens":843,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":734}},"tokens_in":537,"tokens_out":843,"duration_ms":7493,"temperature":1.0,"reasoning_tokens":734,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T04:10:47.052534+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask a group of professional developers to independently list what they consider the most important hidden decisions in a set of LLM-generated code samples, then run AssumptionMiner on the same samples and compare. If the majority of developer-flagged decisions are absent from the extracted assumption layer, or if the surfaced assumptions are mostly judged trivial or irrelevant, the central claim—that making assumptions explicit improves transparency and controllability—would fail.","supporting_citations":[],"review_version":1}