Pith. sign in

REVIEW 3 major objections 5 minor 16 references

Don't `Well, Actually' Me Unless You Know What You're Talking About: Weak Presupposition Verification Degrades General QA Performance

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Rejecting false presuppositions makes LLMs worse at normal questions.

desk verdict A solid FPQ-TPQ tradeoff study whose 'general QA performance' conclusion is undercut by a TPQ rubric that never checks whether answers are correct. read the letter →

arxiv 2608.06539 v1 pith:HPP3DKCA submitted 2026-08-06 cs.CL

Shenran Wang , Vered Shwartz , Hila Gonen This is my paper · ORCID
classification cs.CL
keywords falsepresuppositionQAverificationfact-checkingquestionansweringLLMevaluationFPQ-TPQtradeoffreal-worlddistributiontruerejection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

False-presupposition QA (FPQA) asks a model to notice when a question takes a false claim for granted and to correct or refuse instead of answering. This paper tests the standard prescription—extract presuppositions, fact-check each one, then respond—across four benchmarks, five model families, and several evidence settings. The central finding is a consistent tradeoff: methods that improve scores on false-presupposition questions (FPQs) lower scores on ordinary true-presupposition questions (TPQs), because the fact-checking step rejects true presuppositions it cannot verify. When responses are weighted by a realistic false-presupposition rate of about 13%, estimated from real user questions, the plain direct-QA baseline is the best overall method and the most aggressive FPQA method is the worst.

What carries the argument

The central object is the weighted expected QA score $\mathbb{E}_{q\sim QA}\{V_{QA}\}=P_F V_F + (1-P_F)V_T$, with $P_F=0.13$ taken from a manual annotation of 100 real user questions, and with $V_F$ and $V_T$ the symmetric 0–5 scores for false- and true-presupposition questions. The mechanism under scrutiny is the decompose-then-fact-check pipeline: extract atomic presuppositions, verify each against retrieved evidence or parametric knowledge, then answer. Fact-checking accuracy on gold presuppositions is the component measurement that isolates the failure, showing high rejection of false presuppositions coexisting with substantial rejection of true ones.

What would settle it

Annotate a sample of real patient questions from a medical advice portal, compute the weighted score with the resulting domain-specific FPQ rate, and check whether any FPQA method overtakes direct QA; a domain rate high enough to flip the ranking would falsify the paper's main conclusion.

Watch

Extended reading notes

Core claim

The paper establishes that FPQA mitigation does not come for free: across almost all tested settings, improvement on FPQ is bought with worse TPQ performance, and the mechanism is a fact-checking component that over-rejects. On gold-labeled presuppositions, fact-checking accuracy on true presuppositions sits below 50% in most configurations, with only the strongest model plus web evidence reaching 76% on one benchmark, while false-presupposition rejection is near-perfect. Since synthetic and even natural FPQA benchmarks contain 21–100% FPQs while real user questions contain roughly 13%, benchmark scores mislead about real-world QA: replacing direct QA with any tested FPQA method reduces expected answer quality under the weighted score, and FP Identification, the strongest FPQ method, is the worst overall performer on every dataset.

Load-bearing premise

The load-bearing premise is that the 13% false-presupposition rate measured in 100 annotated real-world chat questions is representative of real question distributions; if medicine, law, or other domains turn out to have much higher rates, the paper's conclusion that direct QA is best would not transfer to them.

Editorial extensions

If this is right

  • Under the estimated 13% FPQ rate, a general-purpose chatbot that adopts any tested FPQA method is expected to produce lower-quality answers overall than one that simply answers the question.
  • FPQA benchmarks that report only false-presupposition accuracy overstate the practical value of mitigation methods; meaningful evaluation must report TPQ accuracy and a domain-appropriate FPQ rate.
  • The tradeoff points to fact-checking, not presupposition extraction or prompt wording, as the component that must improve for FPQA methods to generalize.
  • Improving external evidence retrieval helps open-weight models verify true presuppositions, but supplying too many passages can hurt; web search for the largest model narrows but does not close the TPQ accuracy gap.
  • Fine-tuning on FPQs together with additional complex TPQs, the recipe suggested by prior work, does not escape the tradeoff in the settings tested.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: a gating strategy that runs fact-checking only when the question is suspicious, rather than on every question, would likely preserve TPQ performance while still catching many FPQs, since the paper shows the verification step itself is what over-rejects.
  • My inference: the 13% prior is the natural quantity to re-estimate per domain; medical or legal user logs may have far higher FPQ rates, and under such priors the ranking of methods could reverse, as the paper's own limitation note implies.
  • My inference: the over-rejection pattern suggests fact-checkers conflate 'unverifiable' with 'false', so adding an explicit 'unverifiable' outcome and abstaining only on confirmed false presuppositions is a direct, testable intervention.
  • My inference: the symmetric TPQ rubric, which scores unnecessary corrections of true presuppositions as wrong, could be adopted more widely in QA evaluation since it exposes a failure mode that FPQ-only benchmarks hide.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies false-presupposition QA (FPQA) methods, which extract and fact-check presuppositions before answering a question. The authors evaluate several FPQA methods across four benchmarks, five model families, multiple RAG conditions, and both false-presupposition questions (FPQs) and true-presupposition questions (TPQs). They report a systematic tradeoff: methods that improve FPQ performance tend to reduce TPQ performance, and they attribute this to fact-checking components that over-reject true presuppositions. Using a 13% FPQ rate estimated from 100 manually annotated WildChat questions, they compute a weighted score and conclude that a plain Direct QA baseline outperforms all FPQA-targeted methods under realistic FPQ/TPQ distributions. The paper also releases manual presupposition annotations for Cancer-Myth TPQs and the WildChat FPQ/TPQ labels.

Significance. If the central claim is established, the paper has clear practical significance: it cautions against evaluating FPQA methods on FPQ-only benchmarks and argues that current fact-checking-based pipelines may hurt general QA reliability. The study's breadth is a major strength: it covers four benchmarks, five model families, multiple methods including GEPA and fine-tuning, and several retrieval settings, and it includes a direct ablation of the fact-checking component. The release of code, data, and manual annotations supports reproducibility. The main risk is that the headline conclusion about 'general QA performance' rests on a TPQ evaluation rubric that appears to reward not correcting rather than answering correctly, and on a single small-sample estimate of the real-world FPQ rate. These issues are load-bearing for the paper's central quantitative claim with the weighted score.

major comments (3)
  1. [§2.5, Appendix E (Table 5), Appendix B (TPQ Evaluation Template), §3.1] The TPQ rubric does not measure whether the model's answer is factually correct. Table 5's score 5 criterion says only that the answer 'fully treats the question as a normal question... answers directly and naturally without questioning,' and score 4 allows 'minor unnecessary qualifications.' The judge prompt in Appendix B instructs the evaluator to focus only on 'the semantics of the answer in relation to the false claim' and provides no gold answer. Consequently, Direct QA, which never raises presupposition concerns, receives near-100% score-5 rates on many TPQ settings (for example, Table 7 shows Direct QA at 100% score 5 on Cancer-Myth TPQs across RAG conditions, and similar patterns appear in Tables 8, 10, 12, and 17). Under the weighted score in Section 3.1, the advantage of Direct QA can therefore be an artifact of a rubric that rewards fluent non-correction rather than answering correctly. The claim that FPQA methods 'degrade general QA performance' needs support from a correctness-based TPQ evaluation, for instance by scoring against gold answers where available (e.g., CREPE and Syn-QA2) or with human correctness judgments.
  2. [§3.1, Appendix H, Limitations] The 13% FPQ rate used for the weighted score is estimated from only 100 manually annotated WildChat questions. Appendix H reports 85% agreement between two annotators, with a second annotator covering only half of the data, and the annotation was performed by the authors. The Limitations section explicitly acknowledges that domain-specific distributions (medicine, law) may have much higher FPQ rates. The conclusion that Direct QA is best under 'realistic' settings is a point estimate with no sensitivity analysis; it is load-bearing for the paper's quantitative conclusion. The authors should report expected scores as a function of PF (or at least the PF range over which Direct QA remains optimal), and ideally use a larger or stratified sample to support the realism of the 13% estimate.
  3. [Appendix F (Alternative Annotator Test)] The Alternative Annotator Test validates that the LLM judge agrees with human annotators under the same TPQ and FPQ rubrics, but it does not validate that those rubrics capture answer correctness. Because the TPQ rubric lacks any check of factual content, the test cannot support the paper's inference that high TPQ scores mean good general QA performance. To address this, the authors should either augment the TPQ evaluation with correctness labels or explicitly reframe the paper's claim as being about 'non-correction behavior' rather than general QA accuracy.
minor comments (5)
  1. [§3.1] The word 'percents' should be 'percentages' in the sentence describing the FPQ ratios in synthetic benchmarks.
  2. [§2.2, §B (Final Answer Template (FP Interpretation))] The method is called 'FP Identification' in most of the paper but 'FP Interpretation' in the final answer template header; please make the naming consistent.
  3. [Tables 1–2 and throughout] The dataset name is written inconsistently as 'CancerMyth' and 'Cancer-Myth'; please standardize.
  4. [§2.5, Tables 7–26] The paper reports 'TPQ accuracy' as the percentage of score-5 responses after excluding zeros, but this is not stated explicitly in the main text. Please clarify the definition of accuracy in the main text or in the caption.
  5. [Table 11, note] The note that 'Direct QA + all RAG' and 'Self-Dual-Critique + all RAG' produced 100% gibberish for OLMO-3-7B is important for interpreting the averaged plots; consider excluding or separately marking such degenerate settings in the figures.

Circularity Check

1 steps flagged · score 6.0 of 10

The TPQ rubric defines the top score as Direct QA's defining behavior, so the headline tradeoff and the 'Direct QA is best under realistic settings' conclusion are partly artifacts of the evaluation definition; the fact-checking diagnosis is independent.

  1. self definitional [Section 2.5, Section 3.1, Appendix B (Direct QA Template), Appendix E Table 5]
    "We design a symmetric set of criteria for TPQ, where 1 indicates wrongly trying to correct a TP, and 5 means no correction attempts. ... 5: The answer fully treats the question as a normal question. It answers directly and naturally without questioning the validity of the user's question, introducing unnecessary skepticism, or discussing nonexistent false presuppositions. ... Under such a weighted score, direct QA is the best method."

    Direct QA's prompt instructs the model to answer without any presupposition analysis, which is exactly what TPQ score 5 rewards. FPQ score 5 rewards the opposite behavior: identifying and correcting the false claim. Therefore the 'FPQ improvement hurts TPQ' pattern is in part encoded in the symmetric rubric rather than discovered from answer correctness; the TPQ judge prompt provides no gold answer and tells the judge to focus only on false-claim semantics, so a direct but incorrect answer still earns 5. The conclusion that Direct QA wins under the 13% FPQ weighting is thus substantially a restatement of the evaluation's definitions.

full rationale

The paper is an empirical study, not a derivation, so there is no equation-level circularity in the weighted score computation: the 13% FPQ ratio is estimated from 100 annotated WildChat questions rather than fitted to produce the conclusion, and the fact-checking bottleneck is measured directly on gold-standard presuppositions. I do not treat the WildChat limitation as circular, and the self-citations to the authors' own benchmarks (e.g., Jung and Gonen 2026, Taranukhin and Shwartz 2026) are not load-bearing for the central claim. The circularity concern is narrower but real: the TPQ evaluation rubric defines its maximum score as answering without questioning presuppositions, and Direct QA is defined exactly as answering without presupposition handling. Consequently, Direct QA's near-perfect TPQ scores and its winning position under the realistic weighting are largely guaranteed by the rubric's definition, not by answer correctness. The paper's Limitations section acknowledges the domain-specificity of the 13% FPQ estimate but does not acknowledge that the TPQ score omits factual correctness; this missing support is the basis for the partial-circularity finding. Because the paper also contains independent evidence (fact-checking accuracy below 50% on true presuppositions, tradeoffs that are not universal across all settings such as GEPA(FPQ+TPQ) sometimes scoring well on both), the circularity is partial rather than total, yielding a score of 6 rather than 8-10.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper's conclusions rest on three domain assumptions: the validity of an LLM judge, the representativeness of the WildChat-derived FPQ ratio, and the quality of the newly annotated TPQ presuppositions. No new theoretical entities are introduced.

free parameters (1)
  • PF (realistic FPQ proportion) = 0.13
    Estimated from manual annotation of 100 WildChat questions in Appendix H, then used as a fixed weight in the expected QA score in Section 3.1. The paper's conclusion that direct QA is best depends on this value.
assumptions (3)
  • domain assumption The GEMINI-3-FLASH LLM judge's 1-5 response scores are a valid stand-in for human evaluation of both FPQ and TPQ responses.
    Validated with the Alternative Annotator Test on 150 samples in Appendix F, but every aggregate result in Section 3 inherits this assumption.
  • ad hoc to paper The 13% FPQ rate estimated from 100 WildChat questions reflects the realistic FPQ-TPQ distribution for general QA.
    Used to compute the weighted QA score in Section 3.1; based on a small, single-source sample, and the paper itself notes that domain-specific rates may be higher.
  • domain assumption Manually annotated presuppositions for Cancer-Myth TPQs are accurate gold labels for fact-checking evaluation.
    The annotations were produced by one author without reported inter-annotator agreement in Appendix I, yet they ground the fact-checking accuracy numbers in Table 1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Don't `Well, Actually' Me Unless You Know What You're Talking About: Weak Presupposition Verification Degrades General QA Performance." pith.science (2026). https://pith.science/paper/HPP3DKCA

@misc{pith2026260806539,
  author       = {Pith},
  title        = {Pith review of: Don't `Well, Actually' Me Unless You Know What You're Talking About: Weak Presupposition Verification Degrades General QA Performance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HPP3DKCA}},
  note         = {Machine review of arXiv:2608.06539}
}
read the original abstract

False-presupposition QA (FPQA) tests LLMs on their ability to identify false presuppositions in questions and abstain or correct them rather than reinforcing false assumptions. The common approach reduces the task to prompting LLMs to extract presuppositions and fact checking each presupposition. While the performance on dedicated benchmarks keeps improving, evaluation largely focuses on questions with false presuppositions (FPQs) while ignoring the performance on ``normal'' questions (TPQs). Since many benchmarks over-represent FPQs compared to their natural occurrence, the result is that performance on these benchmarks doesn't reflect real-world QA performance. Through extensive experiments across various model families, sizes, and benchmarks, we show that methods that perform better on FPQs tend to perform worse on TPQs. Our analysis reveals this is the result of weak fact checking modules that reject also true presuppositions. We hope our findings will help guide future work toward FPQA methods that generalize well to realistic settings.

Figures

Figures reproduced from arXiv: 2608.06539 by the authors.

Figure 1
Figure 1. The tradeoff between TPQ (x axis) and FPQ (y axis) accuracies on CancerMyth and SynQA2. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Annotation UI for the Alternative Annotator Test. [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗
Figure 3
Figure 3. The tradeoff between TPQ (x axis) and FPQ (y axis) accuracies on QA [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Annotation UI for round-1 WildChat annotation. [PITH_FULL_IMAGE:figures/full_fig_p024_4.png]
Figure 5
Figure 5. Figure 5: Annotation UI for round-2 WildChat annotation. [PITH_FULL_IMAGE:figures/full_fig_p025_5.png]
Figure 6
Figure 6. Figure 6: Annotation UI for annotating Cancer-Myth TPQ. [PITH_FULL_IMAGE:figures/full_fig_p027_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 13 canonical work pages

  1. [1]

    The answer entirely fails to recognize the false claim within the user's question and reinforces the misinformation in its response, or the answer ignores the false claim and diverts attention to other aspects of the question

    work page

  2. [2]

    The answer might recognize the false claim, but it fails to clearly and directly state the claim within the user's question is false. The response either lacks a clear correction (i.e., explanation for why the claim is false), or even provides an incorrect or misleading correction that fails to address the inaccuracy or align with the provided explanation...

    work page

  3. [3]

    Although it does not confidently debunk the claim, the answer might suggest that the user should verify the claim Figure 2: Annotation UI for the Alternative Annotator Test

    The answer refuses to answer or questions the truthfulness of the claim, indicating that it requires further verification. Although it does not confidently debunk the claim, the answer might suggest that the user should verify the claim Figure 2: Annotation UI for the Alternative Annotator Test. G Full Results Figure 3 presents the tradeoff plot for QA 2 ...

    work page

  4. [4]

    false presupposition

    KG-FPQ: Evaluating factuality hallucina- tion in LLMs with knowledge graph-based false premise questions. InProceedings of the 31st Inter- national Conference on Computational Linguistics, pages 10472–10490, Abu Dhabi, UAE. Association for Computational Linguistics. A Benchmarks Benchmark Description Source Train / Dev / Test Example Cancer-Myth Expert-ve...

    work page 2023

  5. [5]

    The answer is gibberish or unreadable content, or a reasoning trace without a final answer

    work page

  6. [11]

    Select No if the question has no false presupposition. When to discard Select Discard if: you cannot understand the datapoint; the datapoint is not a question; or WildChat annotation 0 / 100 valid · 0 discarded judging whether it contains a false presupposition is beyond your capabilities. Do not guess when one of these conditions applies. Figure 4: Annot...

    work page

  7. [12]

    Identify the claim or claims the question takes for granted

    work page

  8. [13]

    Select Yes if any required presupposition is false

    work page

Show all 16 references
  1. [14]

    We annotated this as

    Select No if the question has no false presupposition or does not presuppose anything. When to discard Select Discard if: you cannot understand the datapoint; the datapoint is not a question; or P r e v i o u s q u e s t i o n N e x t q u e s t i o n WildChat blind reannotatio...

  2. [15]

    Each presupposition should be a single line of text without additional context or explanation

  3. [16]

    presupposes that

    Please begin your presupposition with "presupposes that ..."

  4. [17]

    This presupposition is explicitly suggested by a doctor in the question

    Please check "This presupposition is explicitly suggested by a doctor in the question" only when applicable

  5. [18]

    I'm concerned about the potential for rapid metastasis post-surgery

    Use Add presupposition when more than one is needed Below are some examples: [Question] I've been diagnosed with early-stage hypopharyngeal cancer, and my doctor has suggested surgery as part of the treatment plan. I'm concerned about the potential for rapid metastasis post-su...

  6. [2024]

    Syn-qa2: Evaluating false assumptions in long- tail questions with synthetic qa datasets.Preprint, arXiv:2403.12145. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schel- ten, Alex Vaughan...

    2024 arXiv

  7. [2025]

    In Proceedings of the 63rd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 14956–14975, Vienna, Austria

    Can LLMs ground when they (don’t) know: A study on direct and loaded political questions. In Proceedings of the 63rd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 14956–14975, Vienna, Austria. Association for Computational Lin...

    2026 arXiv

  8. [2026]

    InProceedings of the 64th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16181– 16203, San Diego, California, United States

    Accommodation and epistemic vigilance: A pragmatic account of why LLMs fail to challenge harmful beliefs. InProceedings of the 64th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16181– 16203, San Diego, California, United Stat...

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.