{"id":"9f8738b4-ef31-4b38-a4c7-a5e13fcc77f7","arxiv_id":"2608.06539","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"FPQA methods that score higher on false-presupposition questions tend to score lower on normal questions, because their fact-checking step over-rejects true presuppositions.","lead":"This paper measures how methods that detect false presuppositions in questions affect normal question answering. It finds a consistent tradeoff: methods that are better at catching false assumptions also reject many valid questions, and this hurts real-world QA performance.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TPQ scores reward not correcting rather than answering correctly; the 'general QA performance' conclusion is not yet supported by the reported metric.","rationale":"The reader's weakest assumption, the 13% WildChat estimate, is real but explicitly acknowledged in the Limitations section and can be addressed by sensitivity analysis or domain-specific estimates. The TPQ evaluation metric is more load-bearing because it is used for every TPQ datapoint in every table and figure. A high TPQ score currently means 'did not cast doubt on the question,' not 'answered correctly.' The paper's own full results tables show Direct QA at 100% TPQ score-5 in numerous settings, which is implausible as factual QA accuracy and strongly suggests the judge is not checking content. The fact-checking bottleneck analysis in Section 3.2 and Tables 1–2 is independent evidence that over-rejection exists, so the tradeoff may be real as a behavioral tendency; the problem is specifically the quantitative 'general QA' claim built on TPQ scores. Thus the verdict should remain conditional, but with an added condition: TPQ scores must be validated or replaced with correctness-aware evaluation before the headline claim is accepted.","tokens_in":31816,"tokens_out":8784,"duration_ms":87409,"concrete_test":"Select a stratified sample of TPQ outputs (e.g., 100 each from Direct QA, FP Identification, and Decompose-then-Fact-Check across Cancer-Myth, Syn-QA2, CREPE, and QA2) and score them under a rubric that requires factual correctness against gold answers (available for QA2, Syn-QA2, and CREPE) or expert or lay human annotation for Cancer-Myth. Recompute the Figure 1/3 tradeoff and the PF-weighted scores at PF = 0.13, 0.25, and 0.5. If Direct QA's TPQ advantage shrinks or reverses under the correctness-aware rubric, the 'general QA' conclusion is a rubric artifact. Also report agreement between the original TPQ score and the correctness-aware score.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central quantitative move (Section 3.1) is to weight FPQ and TPQ scores by a realistic FPQ rate and conclude that Direct QA is best, and the abstract generalizes this to 'general QA performance.' The TPQ rubric in Appendix E (Table 5) does not require the answer to be factually correct: score 5 is 'fully treats the question as a normal question... answers directly and naturally without questioning...', score 4 allows 'minor unnecessary qualifications'; nothing checks whether the answer content is true. The judge prompt (Appendix B, TPQ Evaluation Template) explicitly says to focus only on false-claim semantics and provides no gold answer. Consequently, Direct QA, which never raises presupposition concerns, can receive top TPQ scores even when it gives an incorrect answer, and indeed Direct QA gets roughly 100% score-5 TPQ rates in many settings (e.g., Table 7). Under the weighted score, the advantage of Direct QA may therefore be an artifact of a rubric that measures skepticism and formatting rather than QA correctness. The Alternative Annotator Test (Appendix F) only validates agreement with human annotators under the same rubric; it does not validate that the rubric captures answer correctness. This undermines the headline claim that FPQA methods 'degrade general QA performance' and the conclusion that Direct QA is best under realistic settings.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies false-presupposition QA (FPQA) methods, which extract and fact-check presuppositions before answering a question. The authors evaluate several FPQA methods across four benchmarks, five model families, multiple RAG conditions, and both false-presupposition questions (FPQs) and true-presupposition questions (TPQs). They report a systematic tradeoff: methods that improve FPQ performance tend to reduce TPQ performance, and they attribute this to fact-checking components that over-reject true presuppositions. Using a 13% FPQ rate estimated from 100 manually annotated WildChat questions, they compute a weighted score and conclude that a plain Direct QA baseline outperforms all FPQA-targeted methods under realistic FPQ/TPQ distributions. The paper also releases manual presupposition annotations for Cancer-Myth TPQs and the WildChat FPQ/TPQ labels.","tokens_in":32018,"tokens_out":3038,"duration_ms":30942,"significance":"If the central claim is established, the paper has clear practical significance: it cautions against evaluating FPQA methods on FPQ-only benchmarks and argues that current fact-checking-based pipelines may hurt general QA reliability. The study's breadth is a major strength: it covers four benchmarks, five model families, multiple methods including GEPA and fine-tuning, and several retrieval settings, and it includes a direct ablation of the fact-checking component. The release of code, data, and manual annotations supports reproducibility. The main risk is that the headline conclusion about 'general QA performance' rests on a TPQ evaluation rubric that appears to reward not correcting rather than answering correctly, and on a single small-sample estimate of the real-world FPQ rate. These issues are load-bearing for the paper's central quantitative claim with the weighted score.","major_comments":[{"comment":"The TPQ rubric does not measure whether the model's answer is factually correct. Table 5's score 5 criterion says only that the answer 'fully treats the question as a normal question... answers directly and naturally without questioning,' and score 4 allows 'minor unnecessary qualifications.' The judge prompt in Appendix B instructs the evaluator to focus only on 'the semantics of the answer in relation to the false claim' and provides no gold answer. Consequently, Direct QA, which never raises presupposition concerns, receives near-100% score-5 rates on many TPQ settings (for example, Table 7 shows Direct QA at 100% score 5 on Cancer-Myth TPQs across RAG conditions, and similar patterns appear in Tables 8, 10, 12, and 17). Under the weighted score in Section 3.1, the advantage of Direct QA can therefore be an artifact of a rubric that rewards fluent non-correction rather than answering correctly. The claim that FPQA methods 'degrade general QA performance' needs support from a correctness-based TPQ evaluation, for instance by scoring against gold answers where available (e.g., CREPE and Syn-QA2) or with human correctness judgments.","section":"§2.5, Appendix E (Table 5), Appendix B (TPQ Evaluation Template), §3.1"},{"comment":"The 13% FPQ rate used for the weighted score is estimated from only 100 manually annotated WildChat questions. Appendix H reports 85% agreement between two annotators, with a second annotator covering only half of the data, and the annotation was performed by the authors. The Limitations section explicitly acknowledges that domain-specific distributions (medicine, law) may have much higher FPQ rates. The conclusion that Direct QA is best under 'realistic' settings is a point estimate with no sensitivity analysis; it is load-bearing for the paper's quantitative conclusion. The authors should report expected scores as a function of PF (or at least the PF range over which Direct QA remains optimal), and ideally use a larger or stratified sample to support the realism of the 13% estimate.","section":"§3.1, Appendix H, Limitations"},{"comment":"The Alternative Annotator Test validates that the LLM judge agrees with human annotators under the same TPQ and FPQ rubrics, but it does not validate that those rubrics capture answer correctness. Because the TPQ rubric lacks any check of factual content, the test cannot support the paper's inference that high TPQ scores mean good general QA performance. To address this, the authors should either augment the TPQ evaluation with correctness labels or explicitly reframe the paper's claim as being about 'non-correction behavior' rather than general QA accuracy.","section":"Appendix F (Alternative Annotator Test)"}],"minor_comments":[{"comment":"The word 'percents' should be 'percentages' in the sentence describing the FPQ ratios in synthetic benchmarks.","section":"§3.1"},{"comment":"The method is called 'FP Identification' in most of the paper but 'FP Interpretation' in the final answer template header; please make the naming consistent.","section":"§2.2, §B (Final Answer Template (FP Interpretation))"},{"comment":"The dataset name is written inconsistently as 'CancerMyth' and 'Cancer-Myth'; please standardize.","section":"Tables 1–2 and throughout"},{"comment":"The paper reports 'TPQ accuracy' as the percentage of score-5 responses after excluding zeros, but this is not stated explicitly in the main text. Please clarify the definition of accuracy in the main text or in the caption.","section":"§2.5, Tables 7–26"},{"comment":"The note that 'Direct QA + all RAG' and 'Self-Dual-Critique + all RAG' produced 100% gibberish for OLMO-3-7B is important for interpreting the averaged plots; consider excluding or separately marking such degenerate settings in the figures.","section":"Table 11, note"}],"recommendation":"major_revision","confidential_remarks":"The paper is a broad, well-organized empirical study with a plausible qualitative tradeoff. The main blocker is that the TPQ metric rewards not-correcting rather than answering correctly, so the 'general QA performance' conclusion is not yet supported. A correctness-based TPQ evaluation and a sensitivity analysis for the 13% FPQ rate would substantially strengthen the paper. I also recommend that the authors avoid overstating the generality of the conclusion until those two pieces are in place."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on arXiv:2608.06539. The core empirical finding is real: across a large matrix of models, methods, benchmarks, and RAG settings, methods that do better on false-presupposition questions do worse on true-presupposition questions, and the fact-checking component is the bottleneck. The paper is worth a serious referee, but the headline quantitative claim—that FPQA methods degrade 'general QA performance'—does not survive a close look at the TPQ rubric. The TPQ criteria in Appendix E reward answering directly without questioning the premise, with no check on whether the answer is factually correct. Score 5 can be awarded for a confident wrong answer. Direct QA gets near-universal score 5 on TPQs, so the weighted score that makes it the best method is measuring non-skepticism, not QA correctness. The Alternative Annotator Test only shows agreement on that rubric; it doesn't validate the rubric. The paper should either score TPQ answers for factual correctness against gold answers (where available) or drop the 'general QA performance' language.\n\nWhat the paper does well: the fact-checking diagnosis is solid and directly measured. Tables 1 and 2 show models rejecting true presuppositions at high rates even with evidence, which supports the over-rejection story. The experiments are extensive, prompts and code are released, and the manual annotations for Cancer-Myth TPQs are a useful resource. This is a reproducible empirical contribution.\n\nThe soft spots beyond the rubric issue: the 13% FPQ rate rests on 100 WildChat questions with 85% agreement, and the paper itself acknowledges domain-specific rates could differ. Fine as a sensitivity check, not as a definitive real-world distribution. No confidence intervals or significance tests anywhere, though the pattern across dozens of configurations is convincing enough for the qualitative claim. The Cancer-Myth TPQ presupposition annotations appear single-annotator; agreement stats would strengthen the fact-checking numbers, but those results match SynQA2, so it's minor.\n\nWho should read it: anyone working on FPQA, fact-checking, or LLM-as-a-judge evaluation. It deserves peer review, but only after the authors fix the TPQ metric or temper the 'accuracy' claims.","headline":"A solid FPQ-TPQ tradeoff study whose 'general QA performance' conclusion is undercut by a TPQ rubric that never checks whether answers are correct.","tokens_in":32585,"tokens_out":4073,"would_cite":true,"duration_ms":37760,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rejecting false presuppositions makes LLMs worse at normal questions.","keywords":["false presupposition QA","presupposition verification","fact-checking","question answering","LLM evaluation","FPQ-TPQ tradeoff","real-world QA distribution","true presupposition rejection"],"falsifier":"Annotate a sample of real patient questions from a medical advice portal, compute the weighted score with the resulting domain-specific FPQ rate, and check whether any FPQA method overtakes direct QA; a domain rate high enough to flip the ranking would falsify the paper's main conclusion.","tokens_in":31550,"feed_emoji":"❓","tokens_out":14461,"duration_ms":112127,"temperature":0.7,"pith_summary":"False-presupposition QA (FPQA) asks a model to notice when a question takes a false claim for granted and to correct or refuse instead of answering. This paper tests the standard prescription—extract presuppositions, fact-check each one, then respond—across four benchmarks, five model families, and several evidence settings. The central finding is a consistent tradeoff: methods that improve scores on false-presupposition questions (FPQs) lower scores on ordinary true-presupposition questions (TPQs), because the fact-checking step rejects true presuppositions it cannot verify. When responses are weighted by a realistic false-presupposition rate of about 13%, estimated from real user questions, the plain direct-QA baseline is the best overall method and the most aggressive FPQA method is the worst.","feed_headline":"Catching false premises makes LLMs worse at normal questions","feed_subtitle":"Fact-check-first methods reject true assumptions too; plain QA wins at a realistic 13% FPQ rate.","key_machinery":"The central object is the weighted expected QA score $\\mathbb{E}_{q\\sim QA}\\{V_{QA}\\}=P_F V_F + (1-P_F)V_T$, with $P_F=0.13$ taken from a manual annotation of 100 real user questions, and with $V_F$ and $V_T$ the symmetric 0–5 scores for false- and true-presupposition questions. The mechanism under scrutiny is the decompose-then-fact-check pipeline: extract atomic presuppositions, verify each against retrieved evidence or parametric knowledge, then answer. Fact-checking accuracy on gold presuppositions is the component measurement that isolates the failure, showing high rejection of false presuppositions coexisting with substantial rejection of true ones.","core_discovery":"The paper establishes that FPQA mitigation does not come for free: across almost all tested settings, improvement on FPQ is bought with worse TPQ performance, and the mechanism is a fact-checking component that over-rejects. On gold-labeled presuppositions, fact-checking accuracy on true presuppositions sits below 50% in most configurations, with only the strongest model plus web evidence reaching 76% on one benchmark, while false-presupposition rejection is near-perfect. Since synthetic and even natural FPQA benchmarks contain 21–100% FPQs while real user questions contain roughly 13%, benchmark scores mislead about real-world QA: replacing direct QA with any tested FPQA method reduces expected answer quality under the weighted score, and FP Identification, the strongest FPQ method, is the worst overall performer on every dataset.","pith_inferences":["My inference: a gating strategy that runs fact-checking only when the question is suspicious, rather than on every question, would likely preserve TPQ performance while still catching many FPQs, since the paper shows the verification step itself is what over-rejects.","My inference: the 13% prior is the natural quantity to re-estimate per domain; medical or legal user logs may have far higher FPQ rates, and under such priors the ranking of methods could reverse, as the paper's own limitation note implies.","My inference: the over-rejection pattern suggests fact-checkers conflate 'unverifiable' with 'false', so adding an explicit 'unverifiable' outcome and abstaining only on confirmed false presuppositions is a direct, testable intervention.","My inference: the symmetric TPQ rubric, which scores unnecessary corrections of true presuppositions as wrong, could be adopted more widely in QA evaluation since it exposes a failure mode that FPQ-only benchmarks hide."],"forward_implications":["Under the estimated 13% FPQ rate, a general-purpose chatbot that adopts any tested FPQA method is expected to produce lower-quality answers overall than one that simply answers the question.","FPQA benchmarks that report only false-presupposition accuracy overstate the practical value of mitigation methods; meaningful evaluation must report TPQ accuracy and a domain-appropriate FPQ rate.","The tradeoff points to fact-checking, not presupposition extraction or prompt wording, as the component that must improve for FPQA methods to generalize.","Improving external evidence retrieval helps open-weight models verify true presuppositions, but supplying too many passages can hurt; web search for the largest model narrows but does not close the TPQ accuracy gap.","Fine-tuning on FPQs together with additional complex TPQs, the recipe suggested by prior work, does not escape the tradeoff in the settings tested."],"supporting_citations":[{"why":"Introduces false-presupposition QA and the decompose-then-fact-check pipeline that all compared FPQA methods build on.","marker":"Kim et al. (2021)"},{"why":"Provides PreWoMe, one of the tested methods that performs fact-checking through LLM feedback and action.","marker":"Han et al. (2023)"},{"why":"Supplies the FP Identification and Question-to-Statement methods, the strongest FPQ performer and a pipeline variant.","marker":"Wang and Blanco (2025)"},{"why":"Supplies the fine-tuning recipe on FalseQA with ARC-DA complex TPQs that the paper replicates.","marker":"Hu et al. (2023)"},{"why":"Supplies (QA)^2, a benchmark of real search-engine questions with both false- and true-presupposition items.","marker":"Kim et al. (2023)"},{"why":"Supplies CREPE, a naturally occurring FPQA benchmark from ELI5 used to test the tradeoff on real information-seeking questions.","marker":"Yu et al. (2023)"},{"why":"Supplies Syn-QA2, the synthetic FPQ-TPQ pair benchmark used for controlled tradeoff measurements.","marker":"Daswani et al. (2024)"},{"why":"Supplies Cancer-Myth, a physician-verified medical FPQA benchmark, and the GEPA prompt-optimization recipe.","marker":"Zhu et al. (2026)"},{"why":"Supplies WildChat, the real user interaction corpus whose manual annotation yields the paper's 13% FPQ rate.","marker":"Zhao et al. (2024)"},{"why":"Provides prior evidence that LLM fact-checkers have a rejection prior, supporting the paper's over-rejection explanation.","marker":"Wagner (2026)"}],"fun_headline_variants":["Fact-checking false premises hurts LLMs on normal questions","Fixing fake presuppositions tanks everyday QA performance","FPQA rescue plans over-reject true facts: plain QA wins","False premise filters degrade LLM accuracy on real queries","Catching fake premises backfires on ordinary questions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 13% false-presupposition rate measured in 100 annotated real-world chat questions is representative of real question distributions; if medicine, law, or other domains turn out to have much higher rates, the paper's conclusion that direct QA is best would not transfer to them.","fun_headline_variants_meta":{"raw":{"variants":["Fact-checking false premises hurts LLMs on normal questions","Fixing fake presuppositions tanks everyday QA performance","FPQA rescue plans over-reject true facts: plain QA wins","False premise filters degrade LLM accuracy on real queries","Catching fake premises backfires on ordinary questions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000657,"raw_usage":{"total_tokens":2976,"prompt_tokens":885,"completion_tokens":2091,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":2012}},"tokens_in":501,"tokens_out":2091,"duration_ms":15575,"temperature":1.0,"reasoning_tokens":2012,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T04:16:08.642703+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Annotate a sample of real patient questions from a medical advice portal, compute the weighted score with the resulting domain-specific FPQ rate, and check whether any FPQA method overtakes direct QA; a domain rate high enough to flip the ranking would falsify the paper's main conclusion.","supporting_citations":[],"review_version":1}