{"id":"ad686896-9f9a-4494-99e4-bcb88a4c7a9e","arxiv_id":"2505.21701","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Knowledge probing methods for LLMs are found to be highly inconsistent under minor prompt perturbations and across different probes.","lead":"This paper asks whether tools that find knowledge gaps in large language models are themselves reliable, and reports that they are not: small changes like shuffling answer choices flip their decisions, and different tools disagree with each other. If correct, this means current methods for deciding when a model knows or does not know an answer cannot be trusted on their own.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Intra-method inconsistency may be inherited from model output variability rather than from the probes themselves.","rationale":"The reader identified threshold recalibration as the weakest assumption, but that concern is narrower: it applies mainly to calibration-based methods and is partially addressed by the ASKCAL threshold correction in Section 5.5. The more load-bearing issue is that the paper's consistency metrics cannot attribute observed changes to the probing method because the model's outputs are allowed to vary across the compared conditions. Since every probe consumes those outputs, low consistency could simply reflect the model's own sensitivity to the perturbations, which the paper explicitly says it is not evaluating. The proposed test directly separates these two sources: by conditioning on identical model behavior, any remaining inconsistency must come from the probe's decision rule. If the condition reveals high consistency, the paper's central claim would need substantial revision; if low consistency remains, the claim is strengthened. Verdict should remain conditional pending this control, rather than accepting the current attribution as established.","tokens_in":48199,"tokens_out":6313,"duration_ms":73130,"concrete_test":"Using the released code, recompute intra-method IoUcons, IoUrej, and DecCons from Table 1 restricted to the subset of questions where the model's selected answer content and its correctness are identical across the original and variant prompts (map option letters to option text for the shuffled-options variant). If consistency rises substantially on this matched subset, the reported inconsistency is largely an artifact of model output variability; if low consistency persists on the matched subset, the probe-level instability is confirmed. Report the size of the matched subset to assess coverage.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that the probing tools themselves are inconsistent, but the intra-method metrics in Section 3 do not condition on the model's prediction or internal state. Each probe is a function of the model's output (token probabilities, hidden states, self-reported confidence), and Section 5.1 reports large changes in the model's answers under the same variants, e.g., Table 1 shows Agr. values near 0.66 for TOKPROB and 0.57 for NOTA under shuffled options. If a perturbation changes the model's confidence or correctness, a probe that is perfectly calibrated to the model's current behavior will change its accept/reject decision exactly as observed. The paper states it evaluates probe consistency, not LLM consistency, but IoUcons and DecCons compare decisions across variants while allowing the model's output to vary, so the two are not separated. This confound affects all six methods, not only calibration-based ones, and therefore undermines the headline conclusion that the inconsistency resides in the probes themselves.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a consistency-evaluation framework for knowledge-gap probing methods in LLMs. It defines four metrics (IoUcons, IoUacc, IoUrej, DecCons, plus Agreement) and applies them to six probing methods (TOKPROB, ASKCAL, EMBEDDING, NOTA, MOREINFO, SELFREF) under four prompt variants (inserted spaces, shuffled options, typos, and a one-shot exemplar) across seven models (Mistral-7B, LLaMA-3.2-1B/3B, LLaMA-3.1-8B/70B, OLMo-2-7B) on MMLU and HellaSwag. The main findings are that intra-method accept/reject decisions change substantially under surface-level perturbations (e.g., shuffled options reduce IoUcons to about 0.4), that cross-method decision consistency can be as low as 7%, that scaling does not consistently improve probing consistency, and that aggregate metrics like Abstain F1 hide this instability. The paper also reports an analysis showing that ASKCAL's poor intra-method consistency is partly caused by unstable threshold selection, and proposes a threshold-correction safeguard.","tokens_in":48355,"tokens_out":8679,"duration_ms":92644,"significance":"If the central claims survive scrutiny, the paper makes a useful contribution: it demonstrates that aggregate abstention metrics can be stable while the specific items rejected change dramatically, and it documents large disagreement among probes on identical inputs. The experimental scale is a strength — six probes, multiple model families and sizes, two datasets, multiple seeds, and publicly released code — and the reported variance across seeds is near zero. The proposed metrics are simple and likely to be adopted. However, the headline paradox ('the tools themselves suffer from alarming inconsistencies') is currently supported by measurements that do not cleanly separate probe behavior from model-output instability or from threshold-recalibration artifacts. The cross-method inconsistency is the least confounded result; the intra-method results need additional conditioning to support the paper's interpretation.","major_comments":[{"comment":"The intra-method consistency metrics compare probe decisions across prompt variants while allowing the model's output to vary, so they conflate probe instability with the model's own answer instability. Table 1's Agr. column shows that under shuffled options the model's answer changes on a large fraction of commonly accepted questions (e.g., TOKPROB Agr.=0.66 and NOTA Agr.=0.57 on Mistral/MMLU). Because each probe is a function of these outputs, a low IoUcons may simply reflect the model's sensitivity rather than an internal defect of the probe. The paper acknowledges this in the 'Source of Inconsistency' paragraph of §5.1, but does not act on it; the conclusion in §5.3 that the inconsistency 'should be attributed to the knowledge probing methods themselves' is therefore not supported by the reported measurements. To support the central claim, the authors should report the consistency metrics restricted to items where the model's answer (and, ideally, its confidence) is unchanged across variants, or otherwise decompose the observed variance.","section":"§5.1, Table 1 and §3"},{"comment":"The threshold-recalibration confound is demonstrated inside the paper but not resolved in the main results. Section 5.5 describes a threshold correction for ASKCAL that raises HellaSwag IoUcons values from 0.24/0.05 to 0.53/0.48 (Table 3), yet the main intra-method results for ASKCAL in Table 8 still report the uncorrected values (0.243 and 0.049 for Mistral). No equivalent threshold-stability analysis is provided for TOKPROB, which also selects a threshold per prompt variant, and the correction is not applied uniformly to all calibration-based methods. Because a poorly chosen threshold alone can make a probe appear arbitrarily inconsistent across variants, the intra-method numbers for calibration-based probes cannot be interpreted as measuring intrinsic probe reliability unless threshold selection is controlled (e.g., by fixing one threshold or by reporting rejection sets under several thresholds). The same concern applies to the cross-method heatmaps in Figure 3, where ASKCAL's decisions are based on a per-variant threshold whose stability is not checked.","section":"§5.5, Table 3 vs. Table 8"},{"comment":"The scaling-consistency analysis inherits the same confounds and adds a format confound. Figure 2 averages IoUcons over variants that include one-shot prompts, which are not a minimal surface-level perturbation: they change the task format and, as §5.1 reports, can collapse a method's abstention rate (e.g., MOREINFO IoUcons 0.04 on Mistral/MMLU). The non-monotonic trend (e.g., NOTA peaking at 3B and declining at 70B) is therefore not a clean statement about probing consistency under semantically equivalent perturbations. The authors should either restrict the scaling analysis to the three zero-shot variants or show that the pattern is unchanged when the one-shot variant is removed.","section":"§5.3, Figure 2"}],"minor_comments":[{"comment":"The one-shot variant is described as a 'minimal non-semantic perturbation,' but it substantially changes the prompt structure and the instructions. The abstract and framing should be qualified so that the one-shot results are not grouped with the zero-shot perturbation results.","section":"Abstract and §3"},{"comment":"When interpreting a DecCons value as low as 0.07, the main text should explicitly reference the base abstention rates of the two methods (available in Appendix B), since large differences in abstention rates mechanically depress DecCons even when the underlying uncertainty signals are correlated.","section":"§5.2, Figure 3"},{"comment":"The threshold-correction safeguard sets the threshold to 0.5 without a stated justification; the paper should report sensitivity to this choice (e.g., 0.3 and 0.7) and should clarify whether the main results in Tables 1, 6, and 8 use corrected or uncorrected ASKCAL thresholds.","section":"§5.5, Table 3"},{"comment":"The heading 'Inconsistency is Consistent' is confusing; a clearer heading such as 'Variance Across Runs Is Low' would better convey the finding that the reported mean values have near-zero standard deviation.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern is largely valid and is actually evidenced inside the manuscript: Table 3 shows that ASKCAL's intra-method consistency changes dramatically under a simple threshold adjustment, yet Table 8 still reports the uncorrected values. The central interpretation that the inconsistency resides in the probing tools themselves is not yet supported. I would require a conditional analysis (e.g., items with identical model outputs) and a consistent treatment of threshold recalibration for all calibration-based methods before recommending acceptance. The cross-method result is the strongest part and could survive such a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this paper is the first systematic consistency evaluation of knowledge probing methods. It contributes new metrics, IoUcons and DecCons, applies them to six probes across five model sizes and two datasets, and reports near-zero variance across seeds and one-shot examples. The cross-method result is the clean part: same model, same prompt, same dataset, different probes disagree with each other, with decision consistency as low as 7%. Because the model outputs are literally identical in both columns, this is a genuine probe-level finding. Second, the intra-method claim--that shuffling options makes a probe contradict its own verdict about 40% of the time--is only partially supported. The metrics compare probe decisions under different prompts, but the model's outputs change under those prompts; Table 1's Agr. values drop to roughly 0.6 under shuffled options, and the calibration-based probes re-derive their thresholds on the dev set for each variant. A probe that is well-calibrated to a shifted model should shift its decisions. So part of the measured intra-method inconsistency is inherited from model variability or threshold instability, not intrinsic to the probe. The paper acknowledges this in Section 5.1 but never separates the sources.\n\nOn the plus side, the DecCons heatmaps are a useful measurement contribution, and Section 5.4 makes a point that should influence practice: aggregate Abstain F1 stays nearly constant while the actual set of rejected questions rotates. That means current evaluation metrics can hide the instability the field cares about. The code release and variance reporting are good.\n\nThe main soft spot is the threshold handling. Section 5.5 shows ASKCAL's thresholds on HellaSwag jumping from 0.98 to 0.01; applying a safeguard moves IoUcons from 0.05 to 0.48. The same safeguard is not applied to TOKPROB or the other calibration-based methods, so their intra-method numbers are suspect. A second, broader issue is that the abstract's 'tools themselves suffer from alarming inconsistencies' is too strong--the cross-method result supports it, but the intra-method part is confounded with model behavior. Also, only two MCQ datasets are tested, and the authors list that as a limitation.\n\nI think this deserves a serious referee. A revision that reports intra-method consistency conditioned on model output stability, and applies the threshold correction uniformly, would make a solid paper much stronger. The cross-method result alone is publishable. I'd bring it to a reading group to debate the confound.","headline":"Valuable measurement study with a clean cross-method result; the intra-method claim is confounded by model-output variability and threshold re-tuning.","tokens_in":48856,"tokens_out":5273,"would_cite":true,"duration_ms":51916,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Knowledge-probing methods for LLMs contradict themselves and each other: shuffling answer options drops intra-method agreement to about 40%, and different probes agree on as little as 7% of accept/reject decisions.","keywords":["knowledge probing","knowledge gaps","LLM hallucination","prompt sensitivity","consistency metrics","calibration","abstention","multiple-choice question answering"],"falsifier":"Run a calibration-based probe such as ASKCAL with a single fixed threshold chosen once on the original prompt, apply that same threshold to all shuffled-option variants, and compare IoUcons with the paper's per-variant re-calibration results; a large jump in consistency would show that the reported probe instability is threshold drift rather than intrinsic to the probe.","tokens_in":48022,"feed_emoji":"❓","tokens_out":9622,"duration_ms":89180,"temperature":0.7,"pith_summary":"This paper argues that methods designed to reveal what large language models do not know are themselves unreliable, and it demonstrates that unreliability with a systematic consistency test. The test applies minimal, meaning-preserving prompt changes (extra spaces, shuffled answer options, typos, and a one-shot example) and measures whether each probing method labels the same questions as known or unknown. The authors report that shuffling answer options lowers agreement with the original prompt to roughly 40%, and that different methods applied to the same model, dataset, and prompt agree on accept/reject decisions as little as 7% of the time. Their conclusion is that current knowledge-gap probes, and any refusal or abstention system built on them, cannot be trusted at the level of individual questions.","feed_headline":"LLM knowledge probes flip verdicts when options are shuffled","feed_subtitle":"Six probing methods disagree with themselves and each other, with decision consistency as low as 7 percent.","key_machinery":"The argument is carried by a consistency-evaluation procedure built from prompt variants and decision-set metrics. For each prompt variant (space insertion, option shuffling, typo, and an added one-shot example), the probe produces an accept set and a reject set; the metrics $IoU_{acc}$ and $IoU_{rej}$ measure intersection-over-union of those sets between two setups, $IoU_{cons}$ is their harmonic mean, and $DecCons$ measures the fraction of all questions with a matching accept/reject decision. These set comparisons turn the abstract question of probe reliability into concrete numbers that can be compared across methods, models, datasets, and perturbations.","core_discovery":"The central discovery is that knowledge probing outcomes are highly unstable in two ways: the same probe gives different verdicts under semantically irrelevant prompt changes, and different probes give conflicting verdicts under identical conditions. To quantify this, the paper defines set-based metrics over the questions each probe accepts (model knows the answer) and rejects (knowledge gap), with IoUcons as the harmonic mean of acceptance and rejection intersection-over-union and DecCons as the fraction of questions on which two setups make the same decision. Across six probing methods, two multiple-choice datasets, and models from 1B to 70B parameters, the authors observe intra-method IoUcons values as low as 0.27 under option shuffling and cross-method DecCons values as low as 0.07. They also show that the established aggregate metric Abstain F1 stays roughly flat across prompt variants even while the underlying per-question decisions change substantially, which they argue makes that metric insufficient for evaluating probe reliability.","pith_inferences":["A natural extension the paper does not test is consensus probing: aggregating verdicts across several prompt variants or several methods might recover accuracy, since the data imply no single-query probe can be trusted.","The low cross-method agreement may partly reflect that 'knows the answer' is not a single property: token probability, verbalized confidence, hidden-state training, and self-reflection measure different signals, so the 7% floor may indicate a measurement-construct mismatch as much as method failure.","The stability of Abstain F1 alongside unstable per-question decisions suggests that future probe evaluations should routinely report a per-question consistency statistic such as $IoU_{cons}$ under option shuffling, since aggregate scores can hide label flips.","Because option shuffling is the most destructive zero-shot variant and is trivial to apply, it could serve as a minimal stress test for any newly proposed knowledge-gap probe."],"forward_implications":["If shuffling answer options changes which questions a probe rejects, then any refusal or abstention system built on that probe will reject different questions depending on option order.","Aggregate metrics such as Abstain F1 cannot certify a probe's reliability, because they can stay stable while the specific questions marked as knowledge gaps change substantially.","Cross-method disagreements mean that published probing results on the same model, dataset, and prompt may be answering materially different questions about what the model knows.","Scaling to larger models does not by itself make probing consistent; some methods such as NOTA are less consistent at 70B parameters than at smaller sizes.","One-shot prompting, often expected to stabilize behavior, can shift probe decisions even more than typos or extra spaces, with MOREINFO's $IoU_{cons}$ dropping to values near 0.04 in some settings."],"supporting_citations":[{"why":"Supplies the four-way probe taxonomy, the TOKPROB and NOTA implementations, and the Abstain F1 metric that the paper shows to be insensitive.","marker":"Feng et al. (2024b)"},{"why":"Defines ASKCAL, the calibration method whose per-variant threshold selection the paper analyzes.","marker":"Tian et al. (2023)"},{"why":"Introduces the hidden-state signal used by the embedding-based probe.","marker":"Slobodkin et al. (2023)"},{"why":"Also grounds the embedding-based probe in internal model states.","marker":"Azaria and Mitchell (2023)"},{"why":"Provides SELFREF, the self-reflective prompting probe that asks the model to reassess its answer.","marker":"Kadavath et al. (2022)"},{"why":"Provides MOREINFO, the prompting probe that asks whether the model needs more information.","marker":"Liu et al. (2023)"},{"why":"Provides MMLU, one of the two multiple-choice datasets on which consistency is measured.","marker":"Hendrycks et al. (2021)"},{"why":"Provides HellaSwag, the second dataset, where the lowest cross-method consistency of 0.07 appears.","marker":"Zellers et al. (2019)"},{"why":"Documents option-order sensitivity in LLMs, the phenomenon the shuffled-options variant targets.","marker":"Pezeshkpour and Hruschka (2024)"},{"why":"Quantifies prompt-format sensitivity in LLM performance, motivating the use of surface-level variants in the evaluation.","marker":"Sclar et al. (2024)"}],"fun_headline_variants":["LLM knowledge probes agree on as few as 7% of cases","Knowledge probes for LLMs are deeply inconsistent","LLM knowledge probed? Results flip with minor tweaks","Probing LLM knowledge: methods clash, even on same prompt","Knowledge gap verdicts inconsistent across probing methods"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the observed inconsistency lives in the probing methods rather than in the way their accept/reject cutoffs are re-tuned for each prompt: the paper re-calibrates thresholds on the development set for every variant, reports values ranging from 0.98 to 0.01, and only applies a safeguard to one of the methods.","fun_headline_variants_meta":{"raw":{"variants":["LLM knowledge probes agree on as few as 7% of cases","Knowledge probes for LLMs are deeply inconsistent","LLM knowledge probed? Results flip with minor tweaks","Probing LLM knowledge: methods clash, even on same prompt","Knowledge gap verdicts inconsistent across probing methods"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000308,"raw_usage":{"total_tokens":1748,"prompt_tokens":922,"completion_tokens":826,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":745}},"tokens_in":538,"tokens_out":826,"duration_ms":9138,"temperature":1.0,"reasoning_tokens":745,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:24:01.544697+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a calibration-based probe such as ASKCAL with a single fixed threshold chosen once on the original prompt, apply that same threshold to all shuffled-option variants, and compare IoUcons with the paper's per-variant re-calibration results; a large jump in consistency would show that the reported probe instability is threshold drift rather than intrinsic to the probe.","supporting_citations":[],"review_version":1}