Pith. sign in

REVIEW 3 major objections 6 minor 12 references

Answer correctness and question answerability are separate axes of LLM abstention that a single confidence score cannot tell apart.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 07:10 UTC pith:YFHEPNYG

load-bearing objection Solid empirical decomposition of abstention into two axes, with honest controls and dual-risk certificates that actually move the needle over single-threshold baselines. the 3 major comments →

arxiv 2607.08456 v1 pith:YFHEPNYG submitted 2026-07-09 cs.CL cs.AI

Two Axes of LLM Abstention: Answer Correctness and Question Answerability

Benedikt J. Wagner This is my paper
classification cs.CL cs.AI
keywords LLM abstentionanswerabilityanswer confidencehidden-state probesselective predictionfalse presuppositionsdual-risk certificationCREPE
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Selective answering is usually done by thresholding one confidence score. This paper shows that recipe confuses two different failures: emitting a wrong answer to an answerable question, and answering a question that should not be answered at all. Across five instruction-tuned models from 2B to 14B, ordinary answer-confidence tracks whether an attempted answer succeeded but is nearly blind to whether the question was admissible, while a linear probe on hidden states does the reverse. The blind spot does not shrink with scale and is worst on naturally occurring false-premise questions, where confidence, self-assessment, and even direct premise checks stay near chance while the internal probe still works. The authors turn the separation into a factorized policy that answers only when both scores clear independent thresholds, certifying separate risk budgets on unanswerable answers and wrong answers, often at higher coverage of correct answers than any single-score rule.

Core claim

Correct-answerable, wrong-answerable, and unanswerable questions form a crossed L-shaped geometry on the same decisions: ordinary answer-confidence separates successful from unsuccessful execution but barely separates wrong answers from unanswerable questions, while a hidden-state answerability readout separates admissibility from inadmissibility but collapses correct and wrong answerable items. The pattern holds from 2B to 14B and is sharpest on natural false-presupposition data.

What carries the argument

Factorized Abstention: answer only when both an answerability score and a correctness score clear independent thresholds, with dual-risk certification of separate budgets on the unanswerable-answer rate and the wrong-answer rate.

Load-bearing premise

The claim rests on a linear readout of final prompt-token hidden states giving a general enough answerability signal for risk-controlled gating and premise routing, even though that signal is only moderate on natural data and SelfAware answerability is largely available from surface features alone.

What would settle it

On a large held-out sample of natural false-premise questions, if trained answer-confidence or direct premise-check elicitation matched or beat the hidden-state probe, and a two-threshold factorized policy failed to improve certified dual-risk coverage over a single calibrated confidence threshold, the claimed axis separation would not hold.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Single-threshold confidence policies systematically over-abstain or fail to control unanswerable-answer rates because they cannot represent the open region of inadmissible questions.
  • Certified control of unanswerable answers is possible at every scale tested, while wrong-answer control is limited by model accuracy and only becomes useful as models get more accurate.
  • Prompting a model to check premises without an external discriminator causes it to challenge sound and false premises alike.
  • Routing the same premise-check instruction with a hidden-state probe roughly triples challenge precision.
  • At 14B in the tested range, only the factorized two-axis policy certifies both risk budgets at all.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Production systems that report only one confidence number will be systematically miscalibrated for open-world user questions even when they look well-calibrated on closed-world benchmarks.
  • The moderate CREPE signal suggests that stronger answerability representations, if they can be trained or steered, would unlock much higher certified coverage without losing per-axis risk attribution.
  • The same factorization could apply to other selective behaviors such as tool use, retrieval triggering, or multi-step planning, wherever “should I attempt” and “will the attempt succeed” diverge.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues that LLM abstention conflates two distinct failure modes—emitting a wrong answer to an answerable question (W) versus answering an unanswerable or false-premise question (U)—and that these map to separate score axes on the same items. Across five instruction-tuned models (2B–14B, three families), ordinary answer-confidence separates correct-answerable (C) from W/U but barely separates W from U, while a linear hidden-state answerability probe does the reverse (an L-shaped C/W/U geometry). The asymmetry is worst on naturally occurring false presuppositions (CREPE), where output confidence, P(IK), P(True), and direct premise-check elicitation stay near chance while internal readouts reach 0.69–0.78 AUROC. Instructed premise-checking contests sound and false premises indiscriminately; routing the same instruction with the probe roughly triples challenge precision. The authors formalize three-class selective acceptance with separate risk budgets (α_U=0.15, α_W=0.50) and show a factorized two-threshold policy certifies higher C-coverage than single-axis baselines under leak-free split-conformal certification, with a scale-dependent asymmetry: the unanswerable budget is controllable, while the wrong-answer budget is limited by model accuracy.

Significance. If the same-decision L-geometry and dual-risk certificates hold, the paper cleanly separates two requirements that the dominant single-threshold abstention recipe cannot represent, and turns that separation into an operational policy with per-axis risk attribution. Strengths that should be credited: equal-capacity trained readouts for all four source×axis cells (Table 1); a label-artifact control training the answerability direction on C-vs-U only that still places held-out W with C (Table 8); honest surface and elicitation bounds (bag-of-words 0.87 on SelfAware; P(IK)/premise-check on CREPE); SelfAware→CREPE transfer; leak-free four-way splits with Clopper–Pearson dual-risk certificates; and a 100-resplit end-to-end certificate audit (Appendix D). The premise-routing pilot is a concrete behavioral payoff beyond pure detection. The contribution is not that answerability is internally legible (already shown by Slobodkin et al. and Lavi et al.), but the controlled crossed geometry, its measurement under those bounds, and the certified two-budget composition.

major comments (3)
  1. [Abstract; §7; Tables 4 and 9] Abstract and §7 claim that at 14B the factorized policy is “the only policy that certifies at all.” Tables 4 and 9 contradict this: on Qwen2.5-14B, answerability-only also certifies (test C-coverage 0.35, UCB(R_U)=0.07, UCB(R_W)=0.45 ≤ 0.50), while answer-confidence-only and the learned joint scalar fail. Factorized is better (0.47) but not unique. The accompanying gloss that “both confidence-thresholding alternatives fail a budget” is true of those two baselines but does not justify the stronger uniqueness claim. Please correct the abstract, §7, and any parallel conclusion language so the 14B comparison matches the tables.
  2. [§7; Tables 4, 9, 10; Appendix B/D] Certification and test splits are very small for the dual-risk claim: n_C on the test split is 8–17 (Table 4), and certification splits contain only 18–22 W items (Table 9). Under those counts α_W=0.50 is a lenient budget; tighter W budgets never certify (Table 10), partly by sample limit rather than signal quality, as the paper notes. The 100-resplit audit (Appendix D) is the right mitigation and shows issued certificates are rarely violated, but issue rates themselves are low on smaller models (18/100 on Gemma 2B). The scale-dependent story (W budget becomes certifiable as accuracy rises) is plausible but currently rests on thin binomial counts. Either enlarge the SelfAware sample so that W and C denominators support tighter budgets, or move the small-n fragility and the α_W=0.50 choice into the main text of §7 rather than leaving them mostly in the appendix and Limitations.
  3. [Abstract; §5; §7; Tables 2 and 11] On CREPE the internal signal is only moderate (hidden readout 0.69–0.73; difference-of-means 0.74–0.78; Table 2), and the certified false-premise gate yields low admissible coverage (0.09–0.25 on models that certify; Table 11). The dual-risk factorized policy of §7 is evaluated only on SelfAware, where answerability is largely surface-recoverable (bag-of-words AUROC 0.87). The paper is careful about this, but the abstract’s policy claim is written as if the two-axis certificate transfers cleanly to the natural-data regime where the output blind spot is sharpest. Either run the factorized (or at least dual-signal) policy on a setting that has both axes without the SelfAware surface confound, or explicitly scope the dual-budget certificates to SelfAware and keep CREPE as an admissibility-only result.
minor comments (6)
  1. [Table 4] Table 4’s checkmark formatting is hard to parse in the preprint layout (checkmarks run into adjacent cells). Align with the clearer Table 9 layout or use an explicit “cert. yes/no” column.
  2. [§3; Figure 1; Table 1] SelfAware answerable accuracies are low (0.28–0.42). The paper flags caution on the correctness axis; a one-sentence reminder next to Figure 1 and Table 1 would help readers who skip the setup paragraph.
  3. [§1; §2; §7] The neuro-symbolic framing (open/closed-world boundary, Logic Tensor Networks) is interesting but optional; a shorter pointer in the introduction and a fuller paragraph in Related Work or Discussion would reduce the risk that readers treat it as a load-bearing formal result.
  4. [§6; Appendix E] Premise-routing pilot (Table 3) reports strict template detection and NLI validation bounds; releasing the manual audit sample is good. State the NLI model and decision rule in the main text or Appendix E so the 47%/61% “validated correct contest” numbers are reproducible from the paper alone.
  5. [Abstract] Minor wording: abstract sentence “each clear a separately certifies behave differently” appears truncated/garbled in the provided text; repair before camera-ready.
  6. [§3; Appendix H] Quantization validation is only on Qwen2.5-3B (Appendix H). A one-line note that 8-bit/4-bit results for 7B–14B are therefore pipeline-validated by transfer, not re-validated per model, would be clearer.

Circularity Check

1 steps flagged

No significant circularity: empirical geometry, probes, and dual-risk certificates are measured out-of-sample on external benchmarks; author self-citations supply only interpretive neuro-symbolic framing.

specific steps
  1. self citation load bearing [Section 2 (Related Work) and Section 7 (Factorized Abstention)]
    "Our C/W/U geometry is an empirical LLM instantiation of that boundary: the admissibility axis separates the model’s closed fragment... (Wagner & d’Avila Garcez, 2022)... the factorized policy of Section 7 is a thresholded conjunction of two named predicates, Answerable and Correct, grounded in hidden states... conceptual grounding (Wagner & d’Avila Garcez, 2021)."

    The author cites his own prior neuro-symbolic papers to frame the measured C/W/U geometry and the two-threshold policy as an open-world / conceptual-grounding construction. The citations are not used to derive AUROCs, means, or certificates (those rest on SelfAware/CREPE measurements and standard binomial bounds), so the step is only interpretive framing rather than a load-bearing reduction; it is recorded solely because it is the sole self-citation pattern present.

full rationale

The paper's load-bearing claims are empirical measurements (C/W/U L-geometry via standardized means and AUROCs on SelfAware and CREPE; out-of-sample linear/SAE probes; split-conformal dual-risk certificates under Learn-Then-Test) and a defined factorized policy (answer iff s_A ≥ τ_A and s_C ≥ τ_C). Probes and thresholds are fit exclusively on tuning/calibration splits and evaluated/certified on independent held-out splits (including a 100-resplit audit); surface, elicitation, and label-artifact controls are reported rather than assumed away. The Wagner & d'Avila Garcez (2021, 2022) citations appear only as post-hoc interpretive readings that map the observed geometry onto open-world/closed-world boundaries and conceptual grounding; they do not supply uniqueness theorems, force the choice of axes, or enter the AUROC or Clopper-Pearson calculations. No equation equates a fitted quantity to a claimed prediction by construction, no ansatz is smuggled via self-citation, and no known result is merely renamed. The work is therefore self-contained against external benchmarks; the single minor self-citation is non-load-bearing framing and warrants only a score of 1.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 2 invented entities

The paper is primarily empirical. Load-bearing choices are the risk budgets, the linear-probe / SAE feature construction, the three-class labeling of SelfAware/CREPE, and the exchangeability assumption required for Clopper–Pearson selective-risk certificates. No new physical entities are postulated; the ‘two axes’ and ‘Factorized Abstention’ are measurement and policy constructs grounded in observed scores.

free parameters (6)
  • α_U (unanswerable-answer risk budget) = 0.15
    Pre-declared target 0.15 used for all dual-risk certificates; sensitivity grid explored but not family-wise controlled.
  • α_W (wrong-answer risk budget) = 0.50
    Pre-declared lenient target 0.50; tighter values (0.20–0.30) fail to certify at these sample sizes.
  • δ (certificate failure probability) = 0.10
    Bonferroni-split confidence level 0.10 for simultaneous dual-risk bounds.
  • answerability / correctness thresholds (τ_A, τ_C) = model-specific, tuned
    Chosen on tuning split to maximize C-coverage subject to feasibility, then frozen for certification; discrete search on small splits produces non-monotone coverage.
  • L2-regularized logistic probe / SAE top-k battery = nested calibration
    Equal-capacity readouts and SAE feature selection (k=16) fitted on calibration/tuning data only; layer depth nested-selected.
  • premise-routing budget = 0.35
    Fixed 35% of questions routed to challenge prompt in the pilot.
axioms (5)
  • domain assumption SelfAware and CREPE gold labels correctly mark semantic answerability / false presupposition for the purpose of the geometry.
    Used throughout Sections 3–6 as ground truth for C/W/U and FP/sound classes; CREPE annotations noted to carry some ambiguity.
  • domain assumption Alias matching against benchmark gold answers defines correctness on answerable items.
    Section 3; small number of correct aliases and no human audit of matches.
  • domain assumption Exchangeability of items (or groups) across calibration/certification/test splits for Clopper–Pearson selective-risk bounds.
    Section 7 and Appendices B/G; required for the dual-risk certificates to be valid.
  • domain assumption Final-prompt-token hidden states (or selected SAE features) linearly encode a usable answerability direction.
    Core measurement assumption of Sections 3–5; supported by transfer but only moderate on natural data.
  • standard math Standard binomial / conformal risk-control mathematics (Clopper–Pearson, Learn-Then-Test).
    Used for all certificates; not re-derived.
invented entities (2)
  • Factorized Abstention policy (thresholded conjunction of Answerable and Correct predicates) no independent evidence
    purpose: Operational composition of the two axes into a dual-risk selective-acceptance rule with per-axis certificates.
    Defined in Section 7 as the high-answerability, high-confidence corner; simplest logical composition, not claimed as unique optimum.
  • C/W/U three-state geometry as empirical open-world boundary no independent evidence
    purpose: Frame the two axes as an LLM instantiation of closed vs open regions from neuro-symbolic open-world reasoning.
    Interpretive link to Wagner & d’Avila Garcez (2022); the geometry itself is measured, the open-world reading is framing.

pith-pipeline@v1.1.0-grok45 · 22802 in / 3814 out tokens · 48057 ms · 2026-07-10T07:10:42.927479+00:00 · methodology

0 comments
read the original abstract

A model should refuse two different things: answers it would get wrong, and questions it should not answer at all, such as unanswerable ones or ones resting on a false premise. The usual recipe thresholds a single confidence score, which cannot tell these apart. Across five instruction-tuned models from three families (2B to 14B), we find they are separate axes. Ordinary answer-confidence tracks whether an answer is right but is nearly blind to whether the question is answerable; a linear probe on hidden states does the reverse. The blind spot does not shrink with scale. It is worst on naturally occurring false-premise questions (CREPE). There, answer-confidence, P(IK), P(True), and even asking the model outright whether a premise is false all stay near chance, while a hidden-state probe reaches 0.69 to 0.77 AUROC: the model represents a problem it will not report. This turns out to be fixable. Instructing a model to check premises backfires, because it then disputes sound and false premises alike (57% false challenges), unable to tell them apart; routing the same instruction with the probe roughly triples challenge precision. We turn the two axes into a calibrated policy that answers only when an answerability score and a correctness score each clear a separately certifies behave differently: the unanswerable-answer rate is controllable at every scale, while the wrong-answer rate is capped by model accuracy, so the guarantee tightens as threshold policy certifies both budgets at 0.75 coverage of correct answers, against 0.31 for a single threshold; at 14B it is the only policy that certifies at all.

Figures

Figures reproduced from arXiv: 2607.08456 by Benedikt J. Wagner.

Figure 1
Figure 1. Figure 1: The crossed three-state geometry at 2B (left, full precision) and 8B (right, 8-bit). Correct [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 12 canonical work pages · 6 internal anchors

  1. [1]

    Mitigating LLM Hallucinations via Conformal Abstention

    Yasin Abbasi Yadkori, Ilja Kuzborskij, David Stutz, Andr´as Gy¨orgy, Adam Fisch, Arnaud Doucet, Iuliya Beloshapka, Wei-Hung Weng, Yao-Yuan Yang, Csaba Szepesv ´ari, Ali Taylan Cemgil, and Nenad Tomasev. Mitigating LLM hallucinations via conformal abstention.arXiv preprint arXiv:2405.01563,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control

    arXiv:2110.01052. Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it’s lying. InFindings of the Association for Computational Linguistics: EMNLP 2023, pp. 967–976,

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  3. [3]

    Detecting hallucinations in large language models using semantic entropy

    doi: 10.1038/s41586-024-07421-0. Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. InInternational Conference on Learning Representations (ICLR),

    work page doi:10.1038/s41586-024-07421-0

  4. [4]

    Language Models (Mostly) Know What They Know

    10 Preprint. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language mod- els (mostly) know what they know.arXiv preprint arXiv:2207.05221,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs

    Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth Malik, and Yarin Gal. Semantic entropy probes: Robust and cheap hallucination detection in LLMs.arXiv preprint arXiv:2406.15927,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    No answer needed: Predicting LLM answer accuracy from question-only linear probes, 2025

    Iv´an Vicente Moreno Cencerrado, Arnau Padr ´es Masdemont, Anton Gonzalvez Hawthorne, David Demitri Africa, and Lorenzo Pacchiardi. No answer needed: Predicting LLM answer accuracy from question-only linear probes.arXiv preprint arXiv:2509.10625,

    work page arXiv

  7. [7]

    B´alint Mucs´anyi, Michael Kirchhof, and Seong Joon Oh

    ICLR 2026 Workshop on Principled Design for Trustworthy AI. B´alint Mucs´anyi, Michael Kirchhof, and Seong Joon Oh. Benchmarking uncertainty disentangle- ment: Specialized uncertainties for specialized tasks. InAdvances in Neural Information Pro- cessing Systems (NeurIPS), Datasets and Benchmarks Track,

    work page 2026

  8. [8]

    Are LLM Uncertainty and Correctness Encoded by the Same Features? A Functional Dissociation via Sparse Autoencoders

    Het Patel, Tianyi Chen, Hua Wei, Evangelos E. Papalexakis, and Jiayu Chen. Are LLM uncertainty and correctness encoded by the same features? a functional dissociation via sparse autoencoders. arXiv preprint arXiv:2604.19974,

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    Gustafsson, Sean Wu, Anshul Thakur, and David A

    Edward Phillips, Fredrik K. Gustafsson, Sean Wu, Anshul Thakur, and David A. Clifton. Entropy alone is insufficient for safe selective prediction in LLMs.arXiv preprint arXiv:2603.21172,

    work page arXiv

  10. [10]

    The curious case of hallucinatory (un)answerability: Finding truths in the hidden states of over-confident large lan- guage models

    Aviv Slobodkin, Omer Goldman, Avi Caciularu, Ido Dagan, and Shauli Ravfogel. The curious case of hallucinatory (un)answerability: Finding truths in the hidden states of over-confident large lan- guage models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3607–3625,

    work page 2023

  11. [11]

    Neural-Symbolic Integration for Interactive Learning and Conceptual Grounding

    Benedikt Wagner and Artur d’Avila Garcez. Neural-symbolic integration for interactive learning and conceptual grounding.arXiv preprint arXiv:2112.11805,

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    Do large language models know what they don’t know? InFindings of the Association for Computational Linguistics: ACL 2023, pp

    Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. Do large language models know what they don’t know? InFindings of the Association for Computational Linguistics: ACL 2023, pp. 8653–8665,

    work page 2023