{"id":"428fc426-bd70-4702-aeb5-c96d835957bf","arxiv_id":"2608.11631","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CLAIM trains an open-domain clarification policy using only synthetic labels from multi-model answer-disagreement entropy, and it matches or beats several baselines without human annotations.","lead":"This paper presents CLAIM, a pipeline that decides when a chatbot should ask a clarifying question by measuring disagreement among different language models, then trains a clarification policy without human-labeled data. The system automatically builds synthetic training examples from this uncertainty signal and uses SFT plus GRPO, matching or beating several baselines on three open-domain ambiguity benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training-query provenance is never specified; if the ~10k synthetic instances were built from the three test benchmarks, the SOTA and data-efficiency claims are circular.","rationale":"The reader's weakest_assumption targets the validity of the entropy+LLM labeling proxy; that is a genuine concern, and Appendix B's clarification-ratio diagnostics (77.80 vs 61.90, 73.15 vs 87.96, 61.09 vs 50.00) show the raw multi-model signal is miscalibrated. However, the paper partially addresses it with LLM-based judgement and arbitration, and the final benchmark numbers could still be valid with a noisy proxy. The provenance of the training queries is more load-bearing: if the 10k uncertainty-constructed instances were generated from the same queries used for evaluation, the central data-efficiency and generalization claims are circular regardless of label quality. The paper explicitly says the benchmarks are 'used exclusively for evaluation' but never discloses the query pool for synthetic data construction, so the current evidence cannot distinguish leakage from a legitimate distribution shift. Hence I recommend UNVERDICTED pending the repository/training-source check rather than CONDITIONAL, which presumes the framework is sound modulo minor fixes.","tokens_in":21697,"tokens_out":7264,"duration_ms":77320,"concrete_test":"Inspect the released repository's data-construction script to identify the exact query source for the ~10k synthetic instances, then compute exact-match (or embedding-threshold) overlap with ClariLM-test, IN3 test, and CLAMBER. If any overlap exists, the headline claim fails. If the repo is ambiguous, rebuild the pipeline using only ClariLM train or another explicitly disjoint open-domain query pool, retrain CLAIM, and re-run Table 1; if the claimed margins over SFT-IN3/ClariLM shrink or reverse, the §4.5 generalization argument is unsupported.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"Section 4.4 claims SOTA/near-SOTA on 'approximately 10k uncertainty-constructed training instances' versus ClariLM's ~120k, and Section 4.5 argues generalization from the SFT-IN3 vs. SFT-Full gap. The paper never states where the user queries feeding the §3.2–3.5 pipeline come from. Section 4.1 says the three benchmarks are used 'exclusively for evaluation,' but that phrase only excludes their training splits; it does not establish that ClariLM-test, IN3 test, or CLAMBER queries were not used as the input pool for CLAIM-Agent. If the 10k training queries overlap with the test queries, the model has seen the evaluation inputs during SFT/GRPO and the headline data-efficiency/generalization comparison is circular. This is the most load-bearing gap because no amount of label-quality tuning fixes a train/test overlap; the paper's central claim presupposes a distribution shift that is never documented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CLAIM, an uncertainty-driven framework for open-domain clarification in LLMs. The method estimates query uncertainty via entropy over semantic clusters of answers sampled from k1=5 heterogeneous LLMs, decides whether clarification is needed using a hand-set entropy threshold tau=0.45 combined with LLM-based judgment and conflict arbitration, and selects among k2=3 candidate clarifying questions by information gain computed with a simulated user answer. Training uses SFT followed by GRPO on roughly 10k synthetic instances generated without human annotations. Evaluation on ClariLM-test, IN3, and CLAMBER compares against LLMs, reasoning models, SFT variants, and the previous ClariLM system. The paper claims SOTA or near-SOTA on the majority of metrics and superior data efficiency versus ClariLM's ~120k training examples.","tokens_in":21906,"tokens_out":7899,"duration_ms":69412,"significance":"If the results hold, the paper provides a low-cost, scalable alternative to human-annotated clarification training: a single 8B model trained on ~10k automatically generated examples can match or beat a system trained on ~120k examples on two of three benchmarks. The multi-model semantic-entropy signal is principled and well-motivated by prior uncertainty-estimation work. The paper includes useful ablations (SFT-Entropy only, SFT-LLM only, SFT-without IG, SFT-IN3 vs. SFT-Full) that isolate the contributions of the judgment signals, the information-gain selection, and the domain-generalization behavior. The release of code and prompts (claimed in Section 3) would aid reproducibility. However, the strength of the central claims is weakened by the lack of statistical support, the unvalidated synthetic labels, and the absence of provenance for the training queries.","major_comments":[{"comment":"The paper never states the source of the user queries used to construct the approximately 10k synthetic training instances. Section 4.1 only states that the three benchmarks are used 'exclusively for evaluation,' but this phrasing does not rule out that queries from the ClariLM-test, IN3, or CLAMBER test sets were fed into the CLAIM-Agent pipeline before evaluation. If the training queries overlap with the test queries, the data-efficiency claim in Section 4.4 and the generalization analysis in Section 4.5 become circular. I request an explicit statement of where the training queries came from and a documented decontamination procedure (e.g., n-gram overlap removal) between the training queries and all test sets.","section":"Sections 3.2-3.6 and 4.1"},{"comment":"The training labels are generated entirely by the authors' own pipeline: the hand-set entropy threshold tau=0.45, the LLM-based judgment, and the conflict-arbitration step. There is no independent validation that these synthetic labels agree with human judgments of clarification necessity or question quality, and the GRPO reward is defined as alignment with these same synthetic labels. Without such validation, the trained model may be learning the pipeline's biases rather than user needs, and the only support for the pipeline's correctness is the final benchmark accuracy, which risks a self-referential loop. Please provide a label-quality study, for example human agreement on a sample of the training labels or a comparison of the pipeline's decisions against the human-annotated labels in IN3 and CLAMBER.","section":"Sections 3.2-3.3, Eq. (5), and 3.6.2"},{"comment":"The conclusion states that CLAIM 'consistently outperforms' strong baseline models, but Table 1 shows that ClariLM outperforms CLAIM on all four metrics on IN3: Accuracy 89.72 vs. 87.04, F1 94.36 vs. 92.55, CDA 66.32 vs. 63.16, and CQSS 72.68 vs. 72.23. The claim should be qualified to reflect that CLAIM's advantage is benchmark-specific (ClariLM-test and CLAMBER) rather than consistent across all benchmarks.","section":"Section 5 vs. Table 1"},{"comment":"All results are reported as single runs without variance or significance tests. Several key comparisons are within 1-2 points, for example ClariLM-test F1 (CLAIM 84.97 vs. ClariLM 85.48) and IN3 CQSS (CLAIM 72.23 vs. ClariLM 72.68). Without standard deviations across seeds or pairwise significance tests, the 'SOTA or near-SOTA' claim is not statistically supported for those metrics. Please report mean and standard deviation over at least three seeds and, where appropriate, significance tests for the main comparisons.","section":"Section 4.4, Table 1"},{"comment":"The information gain in Eq. (7) depends on a simulated user answer A generated by DeepSeek-V3. The selection of the optimal clarifying question is therefore conditioned on the assumption that real users answer the way DeepSeek-V3 does. No validation is provided that these simulated answers are representative of real user responses. This premise is load-bearing for the IG-based selection claim (the SFT-without IG vs. SFT-Full comparison in Section 4.5) and should be tested, for example by comparing simulated and real user answers on a sample of queries.","section":"Section 3.5 and Appendix A"},{"comment":"The entropy threshold tau=0.45 is motivated by an argument specific to the choice of k1=5 sampling models, but the paper reports no sensitivity analysis for tau, k1, the semantic clustering similarity threshold, or the sampling temperature. These are free parameters of the synthetic data generation pipeline, and the robustness of the method to their variation is unestablished. I request an ablation or sensitivity study for at least the threshold and the number of answer-sampling models.","section":"Section 3.3.1 and Appendix A"}],"minor_comments":[{"comment":"Figure 3 appears garbled in the manuscript, with sequences such as '/uni00000013' replacing the plotted content. A readable figure is needed.","section":"Figure 3"},{"comment":"Table 1 has formatting errors where numbers appear merged, for example '54.4470.2084.26' in the Qwen3-32B row. The table should be cleaned up so that each metric is clearly separated.","section":"Table 1"},{"comment":"The phrase 'approximately 10k' is used for the training instance count, but the exact number is never stated. Please provide the exact count and clarify whether it includes both clarification and non-clarification instances.","section":"Section 4.4"},{"comment":"The term 'near-SOTA' is not defined. Please specify the margin that qualifies as near state-of-the-art.","section":"Section 4.4"},{"comment":"The human evaluation uses three expert annotators and general users, but no inter-annotator agreement (e.g., Cohen's kappa or Fleiss' kappa) is reported. Please include this information.","section":"Section 4.6"},{"comment":"The notation for the cluster probability p_i is defined assuming k1 answers, but the clustering threshold used for semantic clustering is never specified in the main text or appendix. Please document it.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the missing provenance of the training queries. If the authors cannot demonstrate that the approximately 10k training queries are disjoint from the ClariLM-test, IN3, and CLAMBER test sets, the paper's central data-efficiency and generalization claims collapse. I would ask the editor to insist on this clarification and on a label-quality validation before publication. The paper is also a direct follow-up to the same group's ClariLM system; the authors should more explicitly position the novelty relative to that prior work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThis paper is worth a serious look. It shows that a clarification policy for open-domain queries can be trained on roughly 10k automatically labeled instances, where the labels come from entropy over answer disagreements among five heterogeneous LLMs, plus an LLM completeness judgment and an information-gain question selector. That combination is new relative to ClariLM, EVPI, and semantic-entropy papers, and the ablations are informative: removing the entropy signal, the LLM judgment, or the information-gain selection each degrades performance. The trained 8B model is competitive with ClariLM (which uses ~120k human/preference labels) on two of three benchmarks, and code and prompts are promised.\n\nThe main soft spot is the missing training-query provenance. The paper says nothing about where the user queries that feed the Section 3 pipeline come from. Section 4.1 says the three benchmarks are 'used exclusively for evaluation,' but that sentence does not rule out using their test queries as the input pool for the 10k synthetic training instances. If those queries overlap with the test queries, then the SOTA and data-efficiency claims would be circular. This is not a label-quality issue; it is a fundamental validity question. The paper must state the source of the training queries, and if they are drawn from the test benchmarks, the claims need substantial qualification. This is the first thing I would ask an author to fix.\n\nSecondary issues: Table 1 reports no variance or significance tests, and the conclusion's 'consistently outperforms' is contradicted by IN3, where ClariLM wins on all four metrics. The clustering step is also underspecified (algorithm and threshold are not given), and the GRPO reward is defined as agreement with synthetic labels, which is a partial self-referential loop—though the external human benchmarks mitigate that. These are addressable.\n\nIf the training queries come from an independent source, this is a genuinely useful low-annotation recipe for clarification training. As written, the paper deserves peer review, but the provenance question is load-bearing and must be resolved before publication. I would send it to referees with a clear request to check that, and I would not cite it in my own work until it is.","headline":"Low-annotation clarification training shows promise, but the paper must document where its 10k training queries come from before the SOTA claims can be trusted.","tokens_in":22402,"tokens_out":3038,"would_cite":false,"duration_ms":30049,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"When several language models disagree on an answer, that disagreement can be turned into training data that teaches a model when to ask clarifying questions.","keywords":["clarifying questions","open-domain clarification","uncertainty estimation","semantic entropy","synthetic data","information gain","group relative policy optimization","large language models"],"falsifier":"Build a held-out set of queries judged ambiguous by human raters but for which the five models produce nearly identical answers; if CLAIM answers directly on most of them, the entropy signal misses an entire class of missing-information queries and the synthetic labels inherit that blind spot.","tokens_in":21517,"feed_emoji":"❓","tokens_out":6701,"duration_ms":68609,"temperature":0.7,"pith_summary":"This paper argues that the disagreement among several language models answering the same user query is a usable signal for teaching a model when to ask clarifying questions and what to ask. It builds CLAIM, an offline pipeline that turns that signal into roughly ten thousand synthetic training examples without human preference labels, then trains a single 8B model with supervised fine-tuning and group-relative policy optimization. On three open-domain and task-oriented benchmarks, the trained model matches or approaches the best prior clarification system, which was trained on about 120k manually supervised and preference-annotated examples. A sympathetic reader would care because the result points to a low-cost, scalable route to proactive clarification: the supervision comes from model disagreement itself rather than expensive annotation.","feed_headline":"Model disagreement can teach when to ask clarifying questions","feed_subtitle":"Ten thousand synthetic examples, no human preference labels, and one 8B model match far larger supervised baselines.","key_machinery":"The load-bearing object is the entropy-driven synthetic data pipeline: a multi-model answer sampler using five models, semantic clustering over answers, and entropy as an uncertainty score, followed by a threshold, an LLM-based completeness judgment, and conflict arbitration. Clarifying-question candidates are generated with history-based diversity constraints, and the chosen question is the one that maximizes information gain, $IG(q, c_q) = E_1(q) - E_2(q, c_q, A)$, where $A$ is a simulated user answer and $E_2$ is the post-clarification entropy. That synthetic data defines a structured decision-generation target: either answer directly, or name a clarification dimension and ask a targeted question. Training then runs in two stages: supervised fine-tuning to internalize the behaviors and group-relative policy optimization to stabilize decisions for high-uncertainty queries. The multi-model cost is paid only offline; at inference CLAIM is a single model.","core_discovery":"The paper's central claim is that query ambiguity can be operationalized as entropy over semantically clustered answers from a set of five heterogeneous models. For a user query $q$, the pipeline collects candidate direct answers, clusters them by semantic similarity, and computes $E_1(q) = -\\sum_i p_i \\log p_i$. If entropy exceeds a fixed threshold, or an LLM completeness judgment also flags missing information, with a third model arbitrating disagreements, the query is labeled as needing clarification. The pipeline then generates several candidate clarifying questions and keeps the one with the largest information gain, defined as the drop in entropy after a simulated user answer is added. Those labels form about 10k training instances; SFT teaches the decision and generation format, and GRPO sharpens decisions near the boundary. The paper reports that the resulting single model achieves top or near-top results on the clarification-necessity and question-quality metrics of ClariLM-test, IN3, and CLAMBER, using far less supervised data than the ClariLM baseline.","pith_inferences":["The same disagreement-as-uncertainty label generator could plausibly be reused for other underspecification decisions, such as when a model should ask for confirmation, request a missing tool argument, or abstain from answering.","Because the threshold is fixed at 0.45 based on the five-model sampling scheme, the method's calibration is tied to that specific model pool; an inexpensive extension would be to re-estimate the threshold for each pool rather than using a global constant.","The information-gain ranking rests on a simulated user answer, so questions that the simulating model finds easy to disambiguate may be favored even when real users would not answer that way; testing with real user answers would show whether the ranking holds.","Extending the same uncertainty measurement across turns, which the paper leaves to future work, would turn CLAIM into an active information-acquisition loop that re-estimates entropy after each user reply."],"forward_implications":["Clarification policies can be trained without human preference labels, since the supervision is synthesized from model outputs.","Roughly ten thousand uncertainty-constructed examples are enough to approach a system trained on roughly one hundred twenty thousand supervised and preference-annotated examples.","A multi-model agent pipeline can be distilled into a single 8B model that needs one inference call per query at deployment.","The information-gain selection step is what improves clarification-question quality: removing it drops dimension accuracy and semantic similarity while necessity accuracy stays similar.","Adding GRPO after SFT further improves decision accuracy on the harder boundary cases, as shown by the gains from SFT-Full to CLAIM."],"supporting_citations":[{"why":"Supplies the ClariLM-test benchmark and the supervised, preference-trained baseline whose roughly 120k-annotated-example cost CLAIM is compared against.","marker":"[38]"},{"why":"Provides the CLAMBER open-domain benchmark used to evaluate clarification necessity and clarifying-question generation.","marker":"[36]"},{"why":"Provides the IN3 task-oriented interaction dataset used as a held-out evaluation benchmark.","marker":"[21]"},{"why":"Introduces semantic entropy, the underlying signal connecting answer-cluster disagreement to model uncertainty.","marker":"[6]"},{"why":"Defines semantic uncertainty via clustering and entropy, which the pipeline adapts to multi-model answer sets.","marker":"[12]"},{"why":"Formulates expected value of perfect information, the information-gain rationale behind choosing the highest-IG clarifying question.","marker":"[23]"},{"why":"Introduces group-relative policy optimization, the objective used in the second training stage to sharpen clarification decisions.","marker":"[28]"}],"fun_headline_variants":["Uncertainty from model disagreement fuels clarification training","Entropy over answer clusters replaces human labels for clarification","Ask better questions: train with entropy-sampled synthetic data","No labels needed: LLM learns when to clarify from model entropy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the assumption that the spread of answers among a fixed set of five models, combined with the fixed threshold and LLM completeness judgments, faithfully identifies which queries need clarification and which clarifying question is most useful to real users.","fun_headline_variants_meta":{"raw":{"variants":["Uncertainty from model disagreement fuels clarification training","Entropy over answer clusters replaces human labels for clarification","Ask better questions: train with entropy-sampled synthetic data","No labels needed: LLM learns when to clarify from model entropy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00071,"raw_usage":{"total_tokens":3237,"prompt_tokens":1024,"completion_tokens":2213,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":640,"completion_tokens_details":{"reasoning_tokens":2146}},"tokens_in":640,"tokens_out":2213,"duration_ms":16269,"temperature":1.0,"reasoning_tokens":2146,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:33:07.626719+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a held-out set of queries judged ambiguous by human raters but for which the five models produce nearly identical answers; if CLAIM answers directly on most of them, the entropy signal misses an entire class of missing-information queries and the synthetic labels inherit that blind spot.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CLAMBER open-domain benchmark used to evaluate clarification necessity and clarifying-question generation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the IN3 task-oriented interaction dataset used as a held-out evaluation benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Formulates expected value of perfect information, the information-gain rationale behind choosing the highest-IG clarifying question."}],"review_version":1}