{"id":"e0154dca-eeb9-430e-b3c1-d575fae663d2","arxiv_id":"2508.16077","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An LLM-guided Bayesian optimization framework enables designers to intervene in design optimization via natural language, improving agency and reducing cognitive load without sacrificing performance.","lead":"This paper presents a design-optimization system that lets designers steer Bayesian optimization through natural language requests and read AI explanations of suggestions. A user study found higher perceived agency than a system-led approach and lower cognitive load than an existing constraint-based cooperative method.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Agency claim is confounded: Cooperative-NL also had manual sliders; Study2 shows the no-NL Explicit-Constraint condition scored significantly higher agency, so NL interaction is not shown to drive the agency benefit.","rationale":"The paper's central claim is experiential. The strongest evidence offered is the Study1 agency contrast. That contrast is internally valid as a system-level comparison, but it cannot support the paper's conceptual claim that natural language interaction is what gives users agency, because the Cooperative condition also provided unrestricted sliders, and the BO-led condition did not. The pattern predicted by the 'sliders drive agency' explanation—Designer-led ≈ Cooperative > BO-led—is exactly what was observed (p.adj = 1.000 between Designer-led and Cooperative; p.adj = 0.002 for both vs BO-led). Study2 provides a sharper test and goes against the NL mechanism: Cooperative-EC, which has sliders but no NL, yielded significantly higher agency than Cooperative-NL (p=0.032). The authors acknowledge this in 6.4.2 but do not adjust the abstract's framing. This is not a statistical error or a question of LLM fidelity; it is an attribution problem in the experimental design. The reader's concern about LLM selection fidelity (Appendix B) is real but secondary: even perfect fidelity would not establish the agency claim if the effect is due to sliders. The cognitive-load benefit (NASA-TLX p=0.034, mental demand p=0.000) and the absence of a hypervolume difference vs EC (p=0.204) remain useful results. I give credit for using the public D3MOBO implementation, BoTorch, and reporting effect sizes, but the central agency claim as phrased in the abstract overstates what the design supports. The verdict should remain CONDITIONAL: the paper needs a properly controlled comparison before the NL-specific agency benefit is claimed.","tokens_in":26610,"tokens_out":5392,"duration_ms":60756,"concrete_test":"Run a 2x2 factorial Study 1: {sliders: yes/no} × {NL chat: yes/no}, keeping BO suggestions available in all conditions except pure manual. Compare Agency Score across the four cells. If Agency(sliders+NL) ≈ Agency(sliders, no NL) > Agency(no sliders, NL) ≈ Agency(no sliders, no NL), the agency benefit is from sliders, not natural language.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's headline UX claim—'higher user agency than a system-led method'—is supported by Study 1 (Agency Score, BO-led vs Cooperative-NL p.adj=0.002), but the Cooperative-NL condition combined two interventions: natural-language interaction AND unrestricted manual slider control (Section 5.1.1). BO-led had no sliders. Agency could therefore come entirely from direct manipulation. The paper's own Study 2 is evidence for this: when both conditions had sliders, Cooperative-EC (no NL) scored significantly higher on agency than Cooperative-NL (p=0.032, Section 6.3.2, E.1.2). Thus the natural-language component, the paper's claimed novelty, is not shown to increase agency; the comparison conflates interface modality with amount of manual control. This is more load-bearing than LLM selection fidelity: even a perfect LLM would not rescue the agency claim if the effect is caused by sliders. The optimization-performance and cognitive-load findings are less affected, but the abstract overstates the role of NL in agency.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a cooperative design-optimization framework that combines multi-objective Bayesian optimization (BO) with a large language model (LLM). At each iteration, batch BO proposes q candidate parameter sets, and an LLM selects one based on the designer's natural-language request, also providing a textual rationale. Two within-participant studies are reported, both using synthetic quadratic test functions with simulated user testing on three web-app design tasks. Study 1 (N=18) compares Designer-led, BO-led, and Cooperative-NL; Study 2 (N=12) compares Cooperative-EC (Mo et al.) and Cooperative-NL. The abstract claims higher user agency than a system-led method, promising optimization performance versus manual design, and matched performance of an existing cooperative method with lower cognitive load. The core results are: a significant agency difference between BO-led and Cooperative-NL, a significantly lower NASA-TLX for Cooperative-NL in Study 2, and no significant hypervolume difference in Study 2. However, the agency claim is confounded by the simultaneous availability of manual sliders in the Cooperative-NL condition, and Study 2's no-NL condition scored significantly higher on agency.","tokens_in":26878,"tokens_out":7016,"duration_ms":80203,"significance":"If substantiated, the idea is timely and relevant to human-in-the-loop optimization: using an LLM to let designers steer BO sampling in natural language, with textual explanations, could make system-led optimization more designer-centered. The technical integration is simple, clearly described, and reproducible from the provided prompts and the publicly available implementation of Mo et al. The authors also provide a useful comparison against an existing cooperative baseline and report qualitatively interesting findings on how designers express intent. The main weakness is that the headline agency benefit is not attributable to natural-language interaction based on the current experimental design, and the optimization-performance claim rests on a non-significant p-value. These issues are load-bearing because they directly support the paper's principal contributions.","major_comments":[{"comment":"The agency comparison is confounded. In Study 1, BO-led participants could not use sliders, while Cooperative-NL participants could both adjust sliders and issue natural-language requests. The headline result (BO-led vs Cooperative-NL, p.adj=0.002) therefore cannot be attributed to natural-language interaction. Study 2's Cooperative-EC condition, which had sliders but no NL, scored significantly higher on the same Agency Score than Cooperative-NL (p=0.032; Appendix E.1.2). The abstract's 'higher user agency' claim must be restricted to the full combined interface (sliders + NL) or better, the NL-specific contribution should be tested with a condition that controls for manual slider availability.","section":"Section 5.1.1 and Section 6.3.2"},{"comment":"The claim of 'promising optimization performance compared to manual design' is based on a non-significant pairwise comparison (p.adj=0.071), and the authors emphasize r=0.533 as supportive. With N=18, a large effect size from a non-significant test is not reliable evidence; the large r may reflect small-sample variability. Since H1.2 is explicitly not supported, the abstract and conclusion should not assert promising performance without qualification. Report a confidence interval or pre-specified equivalence bound if this claim is retained.","section":"Section 5.5.1 and Section 5.6.2"},{"comment":"The technical validation does not establish that the LLM faithfully interprets free-form designer requests. It alternates only two canned prompts ('increase Objective 1/2'), uses the same synthetic functions as the user studies, and reports centroid distances without a chance-level baseline or significance test. The real requests from the user studies (Section 7.4) include compound and complex instructions that are not tested. Since the entire mechanism depends on this mapping, add a random-selection baseline and evaluate on a sample of actual user requests.","section":"Appendix B"}],"minor_comments":[{"comment":"Typo: 'higher values of Objective 1 (top side)' in the description of blue dots should presumably read 'higher values of Objective 2'; the context indicates the two objectives are being contrasted.","section":"Appendix B"},{"comment":"The NASA-TLX subscale analysis reports p=0.000 for mental demand after testing multiple subscales, but no multiple-comparison correction is described. The weighted total p=0.034 is the primary result; please clarify whether subscale p-values are adjusted and report effect sizes for the significant findings.","section":"Section 6.3.2 / Figure 10"},{"comment":"The content analysis of 187 requests is described as manually tagged with multiple tags, but no inter-rater reliability or detailed coding rubric is reported. If multiple coders were used, report agreement (e.g., Cohen's kappa); otherwise, acknowledge the single-coder nature of this analysis.","section":"Section 7.4"},{"comment":"The batch size q=8 is selected based on 'empirical observations.' Since the LLM can only choose among q candidates, batch size is central to request fidelity. Consider reporting a small sensitivity analysis or at least candidate-diversity metrics to support the choice.","section":"Section 8.1"}],"recommendation":"major_revision","confidential_remarks":"The paper fits UIST's scope and has a useful contribution in the cognitive-load comparison and in showing a full cooperative interface can combine BO and LLM. However, the agency claim needs major repair: the current design conflates natural-language interaction with manual slider control, and Study 2's no-NL condition scoring higher on agency is direct evidence against the NL-specific attribution. If the authors cannot run an additional condition that separates sliders from NL, they should substantially soften the abstract and conclusion. The 'promising optimization performance' phrasing should also be hedged in light of p.adj=0.071."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is a real system, not a slideware proposal. The technical core—using an LLM to pick one point from batch-BO candidates based on a designer's natural-language request, with an explanation—is a legitimate new mechanism, distinct from Mo et al.'s explicit constraints and from prior LLM-BO hybrids that optimize algorithmically. The paper also does several things right: within-participant design with counterbalancing, reuse of Mo et al.'s synthetic tasks, full prompt in the appendix, and honest reporting of null results (trust, and the p=0.071 performance comparison).\n\nThe stress-test concern lands. In Study 1, Cooperative-NL had sliders while BO-led didn't; the agency difference could come entirely from direct manipulation. Study 2 is the tell: when both cooperative conditions had sliders, Cooperative-EC (no natural language) scored significantly higher on agency (p=0.032). So the abstract's 'higher user agency than a system-led method' is really 'a cooperative interface with sliders and BO suggestions beats bare BO.' That's still a useful finding, but it cannot be credited to natural language. The authors themselves hint at this in Section 6.4.2 and the appendix, so it's an overclaim, not a hidden flaw.\n\nThe optimization-performance claim is also softer than advertised: the difference vs. manual design is not significant (p.adj=0.071). The cognitive-load result is the cleanest positive finding—lower NASA-TLX and mental demand for NL vs. explicit constraints—though the authors acknowledge an automation confound. The LLM fidelity check in Appendix B is thin: alternating formulaic 'increase objective 1/2' requests, no random-selection baseline, no open-ended language. That matters because the whole interaction premise depends on the LLM tracking open-ended designer intent.\n\nSample sizes (18 and 12) and synthetic tasks are typical for this subfield, so I wouldn't hang the paper on those. On balance, this is a solid contribution to a specific HCI line—human-in-the-loop multi-objective optimization with LLM interaction. A serious referee would be worth the time: the mechanism is new, the studies are carefully run, and the limitations are mostly stated, just not always heeded in the abstract. I'd send it to review, but with a clear note to the authors that a revision should either add a control condition or reframe the agency claim. The paper is honest enough that I'd be willing to see it improve rather than be rejected on the confound alone.","headline":"A useful LLM-BO cooperative design system, but the paper's own Study 2 undercuts the agency headline by showing the no-NL condition scored higher on agency.","tokens_in":27326,"tokens_out":2009,"would_cite":true,"duration_ms":24842,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes that letting designers talk to a Bayesian optimizer in plain language improves their sense of agency and reduces mental load while keeping optimization quality on par with manual and constraint-based cooperation.","keywords":["cooperative design optimization","Bayesian optimization","large language models","natural language interfaces","human-in-the-loop optimization","user agency","cognitive load","multi-objective optimization"],"falsifier":"Run the same cooperative interface with the LLM's selection step replaced by random selection among the eight candidates while keeping the explanations and sliders unchanged; if agency and cognitive-load benefits persist, they come from the interaction shell rather than intent-faithful selection. Separately, collect open-ended designer requests, have independent raters rank which candidate best matches each request, and measure agreement between the LLM's choice and the raters' choice.","tokens_in":26550,"feed_emoji":"💬","tokens_out":4104,"duration_ms":45554,"temperature":0.7,"pith_summary":"The paper proposes a cooperative design-optimization workflow in which a designer and a Bayesian-optimization (BO) system decide together which parameter setting to test next, communicating through natural language. The system generates eight candidate parameter sets with a batch BO algorithm, then a large language model (LLM) picks the one that best matches the designer's request and explains its choice in plain words. Two user studies with designers optimizing web-app interfaces support the claim that this interaction gives users a stronger sense of agency than a system-led BO loop, performs about as well as manual design, and matches an existing constraint-based cooperative method in optimization quality while reducing cognitive load. The paper treats natural language as the key to letting designers steer optimization without needing to know the parameter space.","feed_headline":"Talk to the optimizer: less cognitive load, same design quality","feed_subtitle":"A natural-language bridge between designers and Bayesian optimization raises user agency and matches a constraint-based cooperative method.","key_machinery":"The core mechanism is a batch BO pipeline (qLogNEHVI acquisition on a Gaussian-process surrogate) that produces eight candidate parameter sets per iteration, followed by an LLM 'selector' that receives a structured prompt containing the task description, the designer's natural-language request, the full evaluation history, and each candidate's predicted mean, predictive variance, and acquisition value. The LLM chooses one candidate and generates a natural-language explanation of its reasoning. Designers can also override the suggestion with sliders, so the system supports three control modes—full manual, system-led, and cooperative—within one interface.","core_discovery":"The central claim is that cooperative design optimization can be realized by inserting a large language model as an arbiter inside a Bayesian optimization loop: the underlying BO engine proposes a small batch of diverse candidate parameter sets, and the LLM selects the single candidate that best fits the designer's natural-language request, then writes a plain-language rationale. In Study 1, this natural-language cooperative condition produced agency scores statistically indistinguishable from fully manual design and significantly higher than a system-led BO condition (adjusted p = 0.002), while optimization quality (relative hypervolume) was not significantly worse than manual design and th","pith_inferences":["The fidelity of the LLM's candidate selection is the linchpin: if it fails to track open-ended requests, the agency and cognitive-load benefits may come from the interactive wrapper (sliders, explanations, a sense of response) rather than from true intent-following.","The cognitive-load reduction may be partly an automation effect—designers delegate parameter manipulation to the system—and might persist even if the chosen candidate were effectively random; a control condition with random selection would separate these explanations.","Explanations did not raise trust scores, suggesting that a textual rationale is insufficient; designers seem to need evidence that their request was actually incorporated, such as a visible mapping between the request and the selected candidate.","The 'candidate menu plus LLM arbiter' pattern is a general recipe that could extend beyond UI design to any human-in-the-loop optimization setting where users can articulate goals but not parameter values.","If the selection step is the bottleneck, prompting the LLM to reason about predictive variance more explicitly, or increasing batch size to give the LLM more choices, could improve both fidelity and perceived responsiveness."],"forward_implications":["Designers can steer multi-objective search by stating high-level goals ('make it faster', 'improve readability') instead of translating intentions into numeric parameter ranges.","The same BO backend can be reused across tasks; only the candidate-selection step changes, so the approach is a modular addition to existing multi-objective BO tooling.","Optimization quality with natural-language cooperation is statistically indistinguishable from manual design and from a constraint-based cooperative method, while imposing lower mental demand.","Because the LLM's prompt includes only BO statistics and evaluation history, the method may transfer to design spaces where the LLM has no prior domain knowledge."],"supporting_citations":[{"why":"Provides the baseline cooperative method (Explicit Constraint), the three web-app design tasks, the synthetic test functions, and the study structure that both user studies adapt.","marker":"[28]"},{"why":"Source of the design-fixation and agency problems that motivate natural-language cooperation, and of the human-in-the-loop optimization framing.","marker":"[4]"},{"why":"Supplies the Gaussian-process surrogate model (BoTorch defaults) used as the BO backend.","marker":"[2]"},{"why":"qLogNEHVI is the batch acquisition function that generates the eight candidate points from which the LLM selects.","marker":"[7]"},{"why":"Supplies the three agency questionnaire items (Q1-Q3) used to compute the agency composite score in both studies.","marker":"[33]"},{"why":"NASA-TLX is the workload instrument that showed significantly lower cognitive load for the natural-language condition in Study 2.","marker":"[11]"},{"why":"The Multidimensional Trust Questionnaire is used to measure purpose, transparency, and utility in Study 2.","marker":"[31]"}],"fun_headline_variants":["LLM as BO arbiter boosts designer agency","Chat with the optimizer: same quality, less load","Natural language design optimization with higher agency","Cooperative BO via LLM: agency up, load down","Talk to the optimizer: less load, same design quality"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The LLM reliably picks the candidate that actually matches the designer's natural-language intent from the eight options, so that the designer's feeling of being heard reflects real fidelity.","fun_headline_variants_meta":{"raw":{"variants":["LLM as BO arbiter boosts designer agency","Chat with the optimizer: same quality, less load","Natural language design optimization with higher agency","Cooperative BO via LLM: agency up, load down","Talk to the optimizer: less load, same design quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000144,"raw_usage":{"total_tokens":968,"prompt_tokens":659,"completion_tokens":309,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":403,"completion_tokens_details":{"reasoning_tokens":248}},"tokens_in":403,"tokens_out":309,"duration_ms":4344,"temperature":1.0,"reasoning_tokens":248,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T17:31:23.637187+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same cooperative interface with the LLM's selection step replaced by random selection among the eight candidates while keeping the explanations and sliders unchanged; if agency and cognitive-load benefits persist, they come from the interaction shell rather than intent-faithful selection. Separately, collect open-ended designer requests, have independent raters rank which candidate best matches each request, and measure agreement between the LLM's choice and the raters' choice.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the baseline cooperative method (Explicit Constraint), the three web-app design tasks, the synthetic test functions, and the study structure that both user studies adapt."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"qLogNEHVI is the batch acquisition function that generates the eight candidate points from which the LLM selects."}],"review_version":1}