{"id":"240cf35f-76f1-419d-bf9d-b918c85373a3","arxiv_id":"2502.01523","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Introduces a 2,000-question benchmark where ambiguous QA queries are annotated with explicit conditions, and shows that condition-aware answering improves accuracy across seven LLMs.","lead":"CondAmbigQA is a new benchmark of 2,000 ambiguous questions, each paired with explicit 'conditions' that clarify different valid interpretations. The authors report that models which identify and use these conditions answer more accurately, suggesting some apparent AI hallucinations are really unresolved question ambiguity.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported improvement is not independently measurable: GPT-4o authors the ground-truth conditions, is the top-scoring model, and the G-Eval judge references those same conditions; the headline percentages do not match the tables.","rationale":"The paper is a useful resource: 2,000 instances with condition-answer-citation triples and a documented annotation pipeline (Appendix C). I read the central claim as stated in the Abstract and Section 5.5: condition-aware generation improves answer accuracy by 11.75%, with an additional 7.15% gain when conditions are explicitly provided. The weakest link is not that GPT-4o assisted in creating the data; human validation and the reported Cohen's kappa are real evidence of some data quality. The weakest link is that the evaluation instrument is built from the same model family that created the references and is then ranked first. G-Eval's Condition and Answer rubrics (Appendix D) instruct the judge to compare the actual condition or answer to the expected condition or answer. When the expected answer is GPT-4o's own output, GPT-4o has a structural advantage over smaller models that answer differently but still correctly. The standard-RAG prompt is not asked to output conditions, so the judge will penalize it according to a rubric that rewards conditions. The 20-example human correlation is too small to exclude systematic judge bias. The headline percentages are not recoverable from the tables, and the Limitations section gives a combined score for GPT-4o that conflicts with Table 2. These are not signs of misconduct; they are signs that the empirical claim has not been pinned down. The proposed concrete test of independent human re-annotation and a different judge would settle whether the gain is real. Because the resource remains valuable, I keep the CONDITIONAL verdict rather than moving to accept or reject.","tokens_in":22691,"tokens_out":5211,"duration_ms":45577,"concrete_test":"Sample 100 CondAmbigQA instances and have independent human annotators, blind to GPT-4o's annotations, write condition-answer-citation triples from the same retrieved fragments. Then score all seven models' outputs against these independent references using a non-GPT-4o judge (with human adjudication on disagreements) and exact citation overlap. If GPT-4o's advantage and the condition-vs-no-condition gap shrink by more than 0.10 absolute, the reported gain is substantially an artifact of evaluator circularity. Separately, recompute the abstract's 11.75% and 7.15% from Tables 2 and 3 and Figure 4; if no consistent mapping exists, report corrected numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Abstract; Section 5.5) is that condition-aware RAG improves answer accuracy because explicit conditions resolve genuine query ambiguity. For that claim to hold, the evaluation must measure answer quality independently of the annotation process. It does not. Ground-truth condition-answer-citation triples are drafted by GPT-4o and validated by human annotators (Section 3.3); the primary metric is G-Eval, an LLM-as-judge (DeepEval) whose reference is those same GPT-4o-derived triples; and the top-scoring model is GPT-4o evaluated on its own annotation distribution. Thus GPT-4o's high Condition, Answer, and Citation scores, and the gap over standard RAG, may reflect format/content alignment with the judge's reference rather than better ambiguity resolution. The comparison is also not controlled: the standard RAG prompt (Appendix D) is not instructed to state conditions, so the judge naturally penalizes it for missing condition structure even when the answer is semantically equivalent. The reliability evidence is thin: G-Eval-human correlation is computed on 20 examples (Appendix E, Table 9), with wide confidence intervals, and the abstract's headline numbers (11.75%, 7.15%) do not appear in the result tables; Table 3 reports +128% improvement for GPT-4o and the Limitations section quotes a GPT-4o combined score of 0.701 versus Table 2's 0.662. These inconsistencies make the exact quantitative claim unverifiable from the paper. The dataset resource itself remains useful, but the empirical inference that condition reasoning causes the improvement is not yet supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CondAmbigQA, a benchmark and dataset of 2,000 ambiguous question-answering instances derived from ALCE-ASQA/AmbigNQ, each annotated with a set of condition-answer-citation triples where the conditions represent explicit contextual constraints that resolve query ambiguity. The annotation process uses GPT-4o to draft initial triples and a six-person human team to validate and calibrate them. The paper proposes G-Eval-based metrics (Condition, Answer, Citation, and Combined scores), evaluates seven (or eight) models under three settings — closed-book, standard RAG, and condition-based RAG with either self-generated or ground-truth conditions — and reports consistent improvements from condition-aware answering, plus a closed-book ablation and an external generalization experiment on ALCE-ASQA. The central claim is that incorporating explicit conditions before answering improves response quality over standard RAG and that apparent hallucinations often stem from query ambiguity rather than model failure.","tokens_in":23001,"tokens_out":5543,"duration_ms":47848,"significance":"If the claims hold, CondAmbigQA would be a useful resource for studying ambiguity-driven hallucination, and the condition-based framing could inform future QA system design. The dataset itself is released under CC BY-SA 4.0, which is a concrete contribution, and the paper reports a detailed annotation workflow with cost and time estimates. The experimental direction — that conditioning on explicit interpretations improves answer quality — is plausible and broadly consistent across the models tested. However, the magnitude and even the interpretation of the reported improvements are currently not verifiable because several load-bearing evaluation components are underspecified or circular: the headline improvement percentages are untraceable to the tables, the Combined Score formula is not defined, the LLM-as-judge metric is validated on only 20 examples, and the ground-truth conditions are generated by the same model that is the top performer and that also defines the reference used by the judge. These issues affect the core quantitative claims rather than just the presentation.","major_comments":[{"comment":"The abstract's headline numbers — an 11.75% improvement from condition reasoning and an additional 7.15% gain when conditions are explicitly provided — do not appear in any table or equation. Section 5.5 reports an average absolute gain of 0.20 on Answer Score, and Section 5.6 (Table 3) reports relative improvements ranging from +93% to +400%, none of which matches 11.75% or 7.15%. In addition, the Limitations section states GPT-4o's combined score as 0.701 and Qwen2.5 as 0.469, while Table 2 lists 0.662 and 0.360 respectively. The central quantitative claims are therefore not reproducible from the manuscript as written. Please provide exact formulas, a dedicated results table, and reconcile these inconsistencies.","section":"Abstract and Sections 5.5–5.6"},{"comment":"The Combined Score is used throughout the results (Table 2, Figure 2, Limitations) and drives the model rankings, but it is never defined mathematically. The text states only that it is a weighted average of Condition, Answer, and Citation scores adjusted by penalties for overgeneration, undergeneration, and single-answer output; no weights, penalty values, or normalization steps are given. Without an exact specification, the combined-score comparisons cannot be reproduced or interpreted.","section":"Section 4.1, Combined Score"},{"comment":"The entire evaluation relies on G-Eval as an LLM judge, yet its agreement with human judgment is validated on only 20 randomly sampled examples (Table 9). With n=20, the reported Pearson correlations (0.88 and 0.83) have wide confidence intervals, and the Answer Quality Spearman correlation of 0.68 is considerably weaker. Furthermore, the human rubric uses holistic 10-point scores for condition quality and answer quality, which may not align with the specific step-by-step criteria in Appendix D; this mapping is not described. This is insufficient evidence for the reliability of the metric on 2,000 instances across eight models.","section":"Section 4.1 and Appendix E"},{"comment":"The evaluation loop is circular: GPT-4o drafted the ground-truth condition-answer-citation triples (Section 3.3), the primary metric is G-Eval as implemented in DeepEval with those same triples as the reference (Section 4.1, Appendix D), and GPT-4o is the top-scoring model (Section 5.1). The measured improvements from condition guidance may therefore reflect how well models imitate GPT-4o's annotation style and output format rather than how well they resolve genuine ambiguity. Human calibration reduces but does not remove this concern. Please re-evaluate with (a) human-written conditions for a subset of queries, (b) a judge model that is not GPT-based and is blind to model identity, and (c) a format-controlled analysis.","section":"Sections 3.3, 4.1, and 5.1"},{"comment":"The comparison between condition-based RAG and standard RAG is not controlled. The standard RAG prompt (Appendix D) explicitly instructs the model only to answer based on the fragments and cites up to five sources, with no request to state conditions or separate answers by condition. The evaluation criteria, however, require comparing extracted conditions against expected conditions and 'heavily penalise omission of critical details in the condition.' Thus the standard-RAG baseline is penalized for missing a required output structure, which likely inflates the gap. A controlled baseline should use the same output format (e.g., an empty or fixed condition block) or a format-invariant metric.","section":"Section 5.5 and Appendix D"}],"minor_comments":[{"comment":"The paper uses inconsistent percentage conventions. Section 5.8 describes a change from 0.374 to 0.471 as a 'substantial gain of 10%', but that is an absolute percentage-point increase; the relative gain is approximately 26%. Similarly, the abstract's 11.75% and 7.15% should state whether they are relative or absolute improvements.","section":"Section 5.8 and Abstract"},{"comment":"Citation Score as defined is precision-only: it divides by the number of model citations. A model that outputs a single correct citation receives a perfect score of 1.0, and the formula is undefined when the model outputs zero citations. Please add a recall term or F1-style combination and specify the zero-citation convention.","section":"Equation (1)"},{"comment":"The reported Cohen's κ of approximately 0.72 is not accompanied by a description of what is being measured for agreement (which annotators, how many items, and what codings). Please specify the inter-annotator agreement protocol.","section":"Section 3.3"},{"comment":"The retrieval setup is underspecified for reproducibility. The text mentions a 'Wikipedia FAISS index', but does not state which retriever (e.g., DPR, BM25, sentence-level embeddings), chunk size, or number of retrieved fragments per query is used. Since the benchmark is retrieval-included, this is needed to reconstruct the dataset and to compare against ALCE-ASQA's DPR passages.","section":"Section 3.2 and Figure 1"},{"comment":"The alternative setting, where models are given ground-truth conditions, is described in a single sentence. Since this setting underlies the central improvement claims, the exact prompt (e.g., the 'Modified Condition-based Prompt' in Appendix D) and output format expectations should be spelled out in the main text or at least explicitly cross-referenced.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The dataset resource appears genuinely useful, and the manuscript is likely within scope for the journal. The main obstacle to publication is not the existence of the benchmark but the rigor of the evaluation section: the headline percentages are unreproducible, the combined metric is undefined, and the GPT-4o-to-GPT-4o annotation-judge loop undermines the central causal claim. I would advise the editor that these issues are fixable within the manuscript's scope (adding exact formulas, a larger human-validation sample, and at least one de-circularized control experiment), but they must be addressed substantively rather than cosmetically."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me give you the short version: the CondAmbigQA dataset is a real resource, and the condition-answer-citation triple structure is a reasonable way to think about ambiguous QA. But the paper's headline improvement figures are not supported by its own tables, and the evaluation loop is too circular to back the causal claim that conditions help.\n\nWhat's actually new: a 2,000-instance benchmark built on ALCE-ASQA/AmbigNQ with retrieved Wikipedia fragments, where each query gets a set of condition-answer-citation triples. The human-LLM annotation pipeline is described concretely (GPT-4o drafts, six NLP people validate, ~$800 total, Cohen's κ≈0.72). That is a useful contribution, and the authors are upfront about the dataset's limitations.\n\nThe soft spots are load-bearing. Ground-truth conditions were drafted by GPT-4o (Section 3.3), and GPT-4o is the best-scoring model. The G-Eval metric is an LLM-as-judge (DeepEval) that compares outputs to those same GPT-4o-derived triples. So the gap between GPT-4o and the others may reflect alignment with the annotation distribution rather than better ambiguity resolution. The standard RAG baseline is not instructed to state conditions, so the judge will naturally penalize it for missing structure even when the answer is semantically equivalent. The G-Eval-human validation is only 20 examples, with wide confidence intervals. And the numbers are internally inconsistent: the abstract's 11.75%/7.15% don't appear in any table; the Limitations section quotes a 0.701 combined score for GPT-4o while Table 2 says 0.662; Table 3 reports +128% for GPT-4o. That makes the precise quantitative claims unverifiable from the paper.\n\nThe condition framework itself is not deeply novel—AmbigQA and ASQA already handle multiple interpretations, and the idea that ambiguous questions carry implicit assumptions is not new. But the specific triple structure and the emphasis on conditions as a first-class annotation target is a modest, real step.\n\nWho this is for: researchers building ambiguous-QA benchmarks or studying LLM hallucination from query ambiguity will want the dataset. The empirical inference needs independent validation before anyone should cite the improvement numbers.\n\nRecommendation: send it to peer review—the resource and framing deserve referee time—but the reviewers should ask for a controlled baseline (e.g., RAG instructed to state conditions without ground truth), a non-LLM or independently validated metric, and a reconciliation of the reported numbers. I would not desk-reject it.","headline":"A genuinely useful dataset and annotation framework undermined by a circular evaluation loop and internally inconsistent headline numbers.","tokens_in":23536,"tokens_out":2292,"would_cite":false,"duration_ms":18939,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Apparent LLM hallucinations often stem from ambiguous queries, and making the hidden interpretation explicit improves QA accuracy by 11.75%.","keywords":["ambiguous question answering","conditions","retrieval-augmented generation","hallucination","LLM evaluation","benchmark dataset","disambiguation","citation accuracy"],"falsifier":"Take a random sample of CondAmbigQA queries, give independent annotators only the query and the retrieved fragments (no model-drafted conditions), and ask them to list every plausible interpretation and answer. If a substantial share of gold condition-answer pairs omit interpretations those annotators identify, or if model rankings change when scored against the independent annotations, the claim that explicit conditions resolve inherent ambiguity is falsified.","tokens_in":22496,"feed_emoji":"❓","tokens_out":9545,"duration_ms":74021,"temperature":0.7,"pith_summary":"The paper argues that when a user asks an ambiguous question, an LLM's answer is often not wrong—the model simply picked one of several equally valid interpretations without saying which one. It introduces \"conditions,\" explicit contextual constraints that pin down one valid interpretation, and builds CondAmbigQA, a 2,000-query benchmark in which each query comes with retrieved encyclopedia fragments and condition–answer–citation triples. Across several open and proprietary models, having the model identify conditions before answering improves answer accuracy by 11.75% over standard retrieval-augmented generation, and supplying the gold conditions adds another 7.15%. The claim is that apparent hallucinations in QA are often expectation mismatches caused by query ambiguity, and that explicit condition reasoning gives models and users a shared frame for resolving them.","feed_headline":"Stating hidden assumptions lifts QA accuracy by 11.75%","feed_subtitle":"A 2,000-query benchmark shows that many hallucinations are unstated interpretations—explicit conditions fix them.","key_machinery":"The machinery is the condition, defined as a set of contextual constraints that must be satisfied for an answer to be correct within a particular scope. Conditions are generated by retrieving encyclopedia fragments for a query, having an LLM draft condition-answer pairs from those fragments, and having human annotators refine and validate them. Each condition is tied to a ground-truth answer and to the specific fragments that support it, so the triple (condition, answer, citations) does the work of separating \"wrong answer\" from \"right answer under a different interpretation.\" The evaluation then scores condition identification, answer correctness, and citation precision, with a combined score that penalizes models for generating too many or too few condition-answer pairs.","core_discovery":"The central discovery is that much of what is called hallucination in question answering is really the model answering a different, but co-valid, interpretation of the same query. The paper formalizes the missing context as conditions and shows that when a model states the condition it is answering under—either by discovering it from retrieved passages or by being given it—both answer accuracy and citation accuracy improve consistently across model sizes. The evidence includes a condition-quality/answer-quality correlation (Pearson 0.598, Spearman 0.637, p < 0.001) and a closed-book ablation in which condition reasoning raises answer scores by an average of 135% over direct answering. The benchmark's design makes this visible by pairing every query with multiple condition-answer-citation triples instead of forcing one answer.","pith_inferences":["If the central claim holds, user-facing assistants could usefully ask a clarifying question or display the condition they are assuming before answering, turning silent expectation mismatches into explicit choices; the paper does not test interactive clarification.","A natural extension is to probe interpretations that do not appear in the retrieved fragments, since the benchmark's conditions are bounded by what the encyclopedia passages contain; an independent-annotation study could reveal how much ambiguity the retrieval step misses.","The benchmark's condition-answer triples could be used as training supervision for smaller models, and if that training narrows the gap between API and local models, it would strengthen the paper's claim that condition reasoning, not raw scale, is the active ingredient.","The citation score could double as a groundedness check: a fluent answer with low citation precision signals that the model is not actually using the retrieved evidence, which may explain some failures that look like hallucination."],"forward_implications":["QA systems that first identify and state a condition before answering should show higher factual accuracy and better source attribution than standard RAG systems, across both large and small models.","Supplying the correct condition helps even models that fail to discover it themselves, which means improving condition identification is a concrete target for retrieval and reasoning pipelines.","The condition–answer–citation structure gives an evaluation that distinguishes ambiguity resolution from factual correctness, so benchmark scores can separate a model's world knowledge from its ability to handle ambiguous queries.","Because the condition-based approach transferred to an existing long-form QA benchmark with about a 10% gain, the method is not limited to the new dataset.","Closed-book ablation results suggest that condition reasoning contributes beyond simple retrieval, since models with self-generated conditions substantially outperform direct answering even without retrieved passages."],"supporting_citations":[{"why":"Supplies the original ambiguous open-domain questions that CondAmbigQA selects and re-annotates with conditions.","marker":"(Min et al., 2020)"},{"why":"Provides the long-form QA benchmark with retrieved Wikipedia passages and citation setup that CondAmbigQA builds on and later uses for the generalisation test.","marker":"(Gao et al., 2023a)"},{"why":"Contributes the long-form answers used in the initial screening to identify queries whose ambiguity leads to genuinely different answers.","marker":"(Stelmakh et al., 2022)"},{"why":"Defines standard retrieval-augmented generation, the baseline that condition-based answering is compared against.","marker":"(Lewis et al., 2020)"},{"why":"Defines G-Eval, the LLM-based evaluation procedure used to compute the condition and answer scores.","marker":"(Liu et al., 2023)"},{"why":"Frames the premise that multiple valid answers to one query require clarification, motivating the condition construct.","marker":"(Qian et al., 2024)"}],"fun_headline_variants":["Hallucinations often just unstated interpretations in QA","Condition-aware QA lifts accuracy by 11.75% on 2,000 queries","Benchmark reveals hidden assumptions behind AI hallucinations","Condition reasoning boosts QA accuracy by 11.75% and more"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the condition-answer triples produced by the retrieval-based annotation process and validated by a small human team faithfully and completely capture the reasonable interpretations of each query; if those gold conditions are incomplete or biased toward the drafting model's reading, the measured gains show agreement with that view rather than genuine ambiguity resolution.","fun_headline_variants_meta":{"raw":{"variants":["Hallucinations often just unstated interpretations in QA","Condition-aware QA lifts accuracy by 11.75% on 2,000 queries","Benchmark reveals hidden assumptions behind AI hallucinations","Condition reasoning boosts QA accuracy by 11.75% and more"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000663,"raw_usage":{"total_tokens":2998,"prompt_tokens":884,"completion_tokens":2114,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":2043}},"tokens_in":500,"tokens_out":2114,"duration_ms":13213,"temperature":1.0,"reasoning_tokens":2043,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T15:01:16.081526+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of CondAmbigQA queries, give independent annotators only the query and the retrieved fragments (no model-drafted conditions), and ask them to list every plausible interpretation and answer. If a substantial share of gold condition-answer pairs omit interpretations those annotators identify, or if model rankings change when scored against the independent annotations, the claim that explicit conditions resolve inherent ambiguity is falsified.","supporting_citations":[],"review_version":1}