{"id":"e6ba9bab-d6c0-413f-b6fd-66056a11b151","arxiv_id":"2606.20585","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"SpecBench benchmarks agents on intent-to-spec translation via interaction, and Buddy improves it by using morphological analysis, simulated users, and targeted questioning.","lead":"This paper introduces SpecBench, a benchmark for testing how well AI agents turn vague user intent into structured executable specifications through limited interaction, and presents Buddy, an agent that decomposes intent into design dimensions and uses simulated users to decide what to ask. A smart generalist might read it to see how current AI coding agents often fail at collaboration by either assuming too much or asking too many questions.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Morphological decomposition plus simulated-user evaluation may miss user-relevant dimensions or inject LLM biases into ambiguity selection.","rationale":"The reader's weakest_assumption directly identifies the unverified inference step in the proposed workflow. The abstract supplies no empirical check on decomposition coverage or simulated-user fidelity, so the UNVERDICTED verdict is appropriate; full-text results would be needed to move it.","tokens_in":1750,"tokens_out":262,"duration_ms":13901,"concrete_test":"Run a within-subjects user study (n>=20) where the same intents are processed by Buddy and by a baseline that asks the real user directly; compare final spec alignment (via user satisfaction ratings and preference match) and number of real-user questions needed. If Buddy's specs score >15% lower on alignment, the simulated-user filter is not reliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the decomposition into design dimensions reliably surfaces the ambiguities that matter to real users, and that simulated users (created from the same decomposition) can rank choices without their own systematic biases. The abstract gives no mechanism for validating either step against actual user data, so the workflow could silently drop important dimensions or reinforce LLM priors rather than user preferences.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript introduces SpecBench, a benchmark for assessing agents' ability to translate vague user intent into structured, executable specifications via a fixed number of clarification rounds with access to past conversations. It claims existing agents exhibit two extremes—prematurely entering implementation while overestimating preference alignment, or exhausting the question budget on all ambiguities—and proposes Buddy, an agent that decomposes intent via morphological analysis into design dimensions and choices, generates simulated users to rank options, and then queries the real user only on remaining ambiguities.","tokens_in":1826,"tokens_out":377,"duration_ms":19862,"significance":"If the workflow were shown to reliably surface user-relevant ambiguities without introducing systematic biases from the decomposition or simulated users, the work could meaningfully advance agent design toward collaborative specification rather than pure execution. The morphological-analysis inspiration and simulated-user step are conceptually distinctive, but the absence of any reported metrics, user studies, or validation leaves the practical significance unestablished.","major_comments":[{"comment":"Abstract: the characterization of existing agents' two extreme behaviors is presented without any metrics, experimental methodology, dataset details, or quantitative results, so it is impossible to assess whether the claimed failure modes are supported or whether Buddy's improvements can be measured.","section":"Abstract"},{"comment":"Abstract (Buddy workflow description): the central claim that morphological decomposition plus simulated-user evaluation will identify the ambiguities worth surfacing to the real user rests on the untested assumptions that (a) the decomposition reliably captures dimensions that matter to actual users and (b) simulated users do not inject their own systematic biases; no validation against real-user data or bias audit is described, which is load-bearing for the proposed solution.","section":"Abstract"}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the detailed and constructive comments. We address each major point below and indicate planned revisions to strengthen the manuscript.","responses":[{"response":"The abstract summarizes findings from the full evaluation on SpecBench. The experimental methodology, dataset construction, and quantitative results (including metrics on premature implementation and question-budget exhaustion) are reported in Sections 4 and 5. To address the concern that the abstract itself is not self-contained, we will revise the abstract to include one or two key quantitative indicators of the observed failure modes while remaining within length limits.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the characterization of existing agents' two extreme behaviors is presented without any metrics, experimental methodology, dataset details, or quantitative results, so it is impossible to assess whether the claimed failure modes are supported or whether Buddy's improvements can be measured."},{"response":"We agree that the assumptions underlying the morphological decomposition and simulated-user ranking step require explicit scrutiny. The current manuscript introduces the workflow and benchmark but does not contain a dedicated real-user validation study or bias audit. We will revise the manuscript to add a dedicated limitations subsection that (i) states these assumptions explicitly, (ii) discusses potential sources of bias in the simulated-user component, and (iii) outlines how SpecBench can be used for future validation. If space permits, we will also include a small-scale pilot comparison of decomposition dimensions against a modest set of real-user preferences.","revision_made":"yes","referee_comment":"[Abstract] Abstract (Buddy workflow description): the central claim that morphological decomposition plus simulated-user evaluation will identify the ambiguities worth surfacing to the real user rests on the untested assumptions that (a) the decomposition reliably captures dimensions that matter to actual users and (b) simulated users do not inject their own systematic biases; no validation against real-user data or bias audit is described, which is load-bearing for the proposed solution."}],"tokens_in":1352,"tokens_out":391,"duration_ms":24981,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's core contribution is SpecBench, a benchmark for turning vague user intent into structured specs through limited rounds of clarification, plus the Buddy agent that decomposes intent into design dimensions via morphological analysis, generates simulated users to score options, and only queries the real user on leftover ambiguities.\n\nIt does a clean job naming the two failure modes in existing agents: they either jump straight to implementation while overestimating what the user wants, or they waste the entire question budget on every possible detail. That framing is practical and matches what people building these systems actually see.\n\nThe workflow itself is a reasonable attempt to make the interaction more targeted. Shifting emphasis from code generation to specification is a useful pivot.\n\nThe soft spot is obvious and load-bearing: the abstract contains no evaluation results, no metrics, no comparison numbers, and no description of how they measured the existing agents or Buddy's gains. Without that data it is impossible to tell whether the simulated users avoid their own biases or whether the decomposition actually surfaces the dimensions real users care about. The stress-test concern about missing dimensions or LLM priors landing in the ambiguity filter therefore cannot be checked from what is here.\n\nThis is for researchers working on interactive agents in HCI and software design. A reader already thinking about clarification strategies might pick up the benchmark idea. I would bring the full paper to a reading group only after seeing the experiments. It deserves peer review because the problem is concrete and the proposed workflow is a distinct direction, even though the current version leaves the central claims unsupported.","headline":"SpecBench and Buddy identify a real collaboration problem in agents but the abstract supplies zero results or metrics, so the claims stay untested.","tokens_in":2295,"tokens_out":382,"would_cite":false,"duration_ms":13973,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Buddy agent decomposes vague user intent into design dimensions and choices, then uses simulated users to decide what to ask the real user.","keywords":["user intent","specification generation","collaborative agents","morphological analysis","simulated users","clarifying questions","SpecBench"],"falsifier":"A study in which real users rate the final specifications produced by Buddy as less aligned with their original preferences than specifications produced by a baseline agent that asks more questions.","tokens_in":2654,"feed_emoji":"🤖","tokens_out":420,"duration_ms":25956,"temperature":0.7,"pith_summary":"The paper introduces SpecBench, a benchmark where agents must turn vague user intent into structured executable specifications through a fixed number of clarifying questions. Existing agents either start implementing too soon while overestimating their grasp of preferences or exhaust their question budget on every ambiguity. Buddy follows a morphological analysis workflow that breaks intent into a space of design dimensions and candidate choices, creates simulated users to evaluate those choices, and engages the real user only on the remaining ambiguities. This approach shifts the emphasis from code execution to collaborative specification building.","feed_headline":"Buddy agent uses simulated users to clarify vague intent into specs","feed_subtitle":"Decomposes design dimensions, evaluates options with fakes, then asks the real user only on unresolved gaps.","key_machinery":"Morphological analysis that decomposes user intent into design dimensions and candidate choices, followed by simulated-user evaluation to prune which ambiguities require real-user input.","core_discovery":"An agent can produce specifications that better align with user preferences by first decomposing intent into a structured space of design dimensions and candidate choices, creating simulated users to evaluate those choices, and then asking the real user only to resolve the remaining ambiguities.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["SpecBench benchmarks agents converting vague intent to structured specs","Buddy decomposes design choices for evaluation with simulated users","Workflow tests options via fakes before final real user queries","Benchmark reveals extremes in agent question and implementation behavior"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Morphological analysis decomposition plus simulated-user evaluation will reliably identify which ambiguities are worth asking the real user about without the simulations introducing bias or the decomposition missing dimensions that matter to actual users.","fun_headline_variants_meta":{"raw":{"variants":["SpecBench benchmarks agents converting vague intent to structured specs","Buddy decomposes design choices for evaluation with simulated users","Workflow tests options via fakes before final real user queries","Benchmark reveals extremes in agent question and implementation behavior"]},"model":"grok-4.3","cost_usd":0.008159,"raw_usage":{"total_tokens":3605,"prompt_tokens":629,"num_sources_used":0,"completion_tokens":61,"cost_in_usd_ticks":81590500,"prompt_tokens_details":{"text_tokens":629,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2915,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":629,"tokens_out":61,"duration_ms":23441,"temperature":1.0,"reasoning_tokens":2915,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-30T21:23:08.194640+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A study in which real users rate the final specifications produced by Buddy as less aligned with their original preferences than specifications produced by a baseline agent that asks more questions.","supporting_citations":[],"review_version":1}