{"id":"fc4ba121-87c9-4f13-820e-c662d7962bf7","arxiv_id":"2505.16520","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Hidden-state factuality probes trained on synthetic statements do not generalize to LLM-generated factual statements, despite reproducing prior results on original datasets.","lead":"This paper tests whether the internal hidden states of large language models actually encode whether a statement is true or false, using new, more realistic datasets. It finds that earlier probing methods reproduce well on synthetic facts but fail to generalize to facts the models themselves generate.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Eq. 4 uncertainty filter (|p_i−0.5|<0.1) restricts the LLM-generated dataset to the model's own decision boundary; near-chance probe accuracy in Table 6 may be an expected artifact of this hard subset, not a general failure on LLM-generated facts.","rationale":"The reproduction of Azaria and Mitchell is convincing and independently valuable; the perplexity-based negative sampling is a reasonable methodological contribution. The reader's weakest assumption about GPT-4o mini oracle labels is legitimate, but I see it as secondary to the uncertainty filter. Label noise would attenuate probe accuracy, but it cannot easily explain a drop to chance unless the oracle is nearly random; moreover, the paper's Limitations already concede that manual verification is needed. The Eq. 4 filter is more directly load-bearing because it changes what the central claim is about. The paper intentionally studies questions where the model's generations are split roughly equally between true and false, which is the hardest possible test for a factuality probe and arguably outside the scope of 'does the model encode factuality in its hidden states' as originally posed. The abstract's phrase 'generalization to LLM-generated datasets remains challenging' is narrower than the actual experimental setting: the dataset is not a sample of LLM-generated facts, but a sample of facts from the model's uncertainty boundary. This does not invalidate the paper's results for that boundary, but it requires a qualification of the central claim and a control experiment to show the result is not a filter artifact. Since the recommended fix is a new experiment and a revised generalization statement, the appropriate verdict remains CONDITIONAL rather than ACCEPT or REJECT.","tokens_in":20958,"tokens_out":5118,"duration_ms":46752,"concrete_test":"Re-run the same SAPLMA experiments on TriviaQA-generated facts while varying the uncertainty filter: tau in {no filter, 0.1, 0.2, 0.3, 0.5}, or stratified by binned p_i, using the same GPT-4o mini oracle labels and 20 seeds per layer. If probe accuracy rises materially with tau (e.g., from about 0.59 to above 0.7 for tau = 0.5), the reported negative generalization is an artifact of selecting only maximally uncertain facts; if accuracy remains near chance across all tau, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The most load-bearing concern is the dataset filter in Eq. 4, not primarily the oracle labels. The LLM-generated dataset keeps only questions with |p_i − 0.5| < tau (tau = 0.1), i.e. items for which the target model produces roughly equal numbers of correct and incorrect answers across K = 10 generations. This intentionally selects the model's knowledge boundary: the exact regime where hidden-state factuality signals, if they encode confidence or knowledge strength, should be hardest to decode. The paper's central negative claim in Section 5.3 ('the trained probes are not capable of providing good generalization to an LLM-generated dataset') and the RQ2 answer are therefore statements about this deliberately ambiguous subset, not about LLM-generated facts in general. The paper states the p ≈ 0.5 assumption in Section 4.2, but the abstract and conclusion generalize the failure to 'real-world scenarios' and 'LLM-generated datasets' without qualification. If factuality is linearly decodable mainly when the model is confident, near-chance accuracy at p ≈ 0.5 is expected and does not contradict Azaria and Mitchell's original claim. A control experiment without this filter, or with progressively larger tau, is missing; without it, the headline negative result is not yet established for typical LLM-generated content.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper re-examines whether hidden states of LLMs linearly encode statement factuality, in the setting of the SAPLMA probe of Azaria and Mitchell (2023). It first reproduces the original experiments on Llama 2-7b and OPT-6.7b, reporting close agreement with the published accuracy values (Tables 1-3). It then proposes two dataset-construction strategies: a perplexity-based negative-sampling procedure that generates plausible false statements from tabular data (Section 4.1), and an LLM-generated fact set built by sampling answers to TriviaQA, SQuAD 2.0, and TruthfulQA questions, with GPT-4o mini as a veracity oracle (Section 4.2). The experiments address RQ1 (probes trained on the original versus the refined synthetic data transfer comparably to the refined test set) and RQ2 (probes trained and tested on LLM-generated facts stay near chance, with best average accuracy about 0.59 in Table 6). The paper concludes that prior findings are only partially validated and that generalization to LLM-generated datasets remains challenging. Code is released.","tokens_in":21231,"tokens_out":18739,"duration_ms":138435,"significance":"If established as stated, the main negative result would be a useful contribution to the factuality-probing literature, complementing recent findings by Orgad et al. (2024) and Levinstein and Herrmann (2024) that factual-signal decodability is dataset- and task-dependent. The paper's concrete strengths are the careful reproduction protocol (five layers, two models, 100 probes), the public code release, the matched-perplexity design of the refined synthetic dataset, and the consistency of the RQ2 outcome across three QA collections and two open models. The principal limitation is scope: the negative result is demonstrated only on the p approximately 0.5 subset selected by Eq. (4), and the missing control experiments mean the claim must currently be read as conditional rather than as a general failure of factuality probing on LLM-generated content.","major_comments":[{"comment":"The central negative claim of the paper is established only for the subset of questions selected by the filter in Eq. (4), |p_i - 0.5| < tau with tau = 0.1, i.e., questions for which the target model produces roughly as many correct as incorrect answers over K = 10 generations. This deliberately isolates the model's decision boundary, the regime in which a factuality signal that scales with knowledge strength or confidence is hardest to decode, and in the train-on-TriviaQA condition of Table 6 the probe is trained exclusively on such boundary items, so it never observes the input regime in which the original SAPLMA results were obtained. Near-chance accuracy on this subset is therefore at least partly confounded with the difficulty of the selected items and cannot, by itself, contradict the hypothesis that factuality is linearly decodable for confident model generations. Yet the abstract, Section 5.3, and the Conclusion state the failure as a claim about 'LLM-generated datasets' in general, without this qualification. Because the same filter is applied to SQuAD 2.0 and TruthfulQA in Appendix E, the cross-dataset consistency does not resolve the scope problem. A control experiment without the filter or with progressively larger tau (e.g., tau = 0.3, 0.5, no filtering, or a matched subset with p far from 0.5) is needed to determine whether the negative result extends to typical LLM-generated content; without it the headline claim is not yet established.","section":"Section 4.2, Eq. (4), and the claims of Section 5.3 and the Abstract."},{"comment":"The veracity labels that drive both the Eq. (4) filter and the probe targets are produced by a single oracle model, GPT-4o mini at temperature 1 with a 3-shot prompt, and no human validation is reported; the paper's Limitations section correctly asks for manual verification, but the analysis proceeds as if the labels were noise-free. Since the same labels determine which questions enter the dataset and what the probe is trained and evaluated on, systematic oracle bias (e.g., consistent mislabeling of a particular answer style) would distort both the composition of the p approximately 0.5 subset and the measured probe accuracy, making the near-chance result hard to interpret as evidence about hidden-state factuality encoding. Some labels in Table 9 are non-obvious (for instance, 'Anakin Skywalker is Darth Vader's son' being labeled false), which underscores the need for a verification step. Reporting the oracle's agreement with human annotations on a sample and checking the sensitivity of Table 6 to the oracle choice would substantially strengthen the RQ2 conclusion.","section":"Section 4.2, Eq. (3), Appendix C.2, and the Limitations section."},{"comment":"All accuracy figures in the paper are point estimates without variance, confidence intervals, or significance tests. This omission matters most for the central comparison, because the values in Table 6 hover near chance and individual cells fall below 0.5 (e.g., triviacountry OPT optimal-threshold 0.449, triviabug Llama last-layer 0.494), which is compatible with substantial run-to-run variation even if the averages are nominally above chance given the large aggregated sample sizes. The reproduction section already trains 20 probes per layer (Section 2.3), so reporting the dispersion of the probe accuracies (or at least bootstrap intervals) for the novel settings is a natural addition; without it, the statement in Section 5.3 that accuracy 'does not reach a noteworthy threshold' is not backed by a quantified comparison.","section":"The point estimates in Tables 5, 6, 13, and 16."},{"comment":"The RQ1 conclusion that the probes 'can generalize even when the train-test datasets have different perplexity' is not fully supported by the experimental design. Table 5 varies the training dataset (original versus refined) while keeping the test set fixed to the refined dataset, so it measures robustness to a change in training distribution, not generalization across test distributions; the complementary cell, training on the refined dataset and testing on the original dataset, is missing. The accompanying claim that results are 'mostly comparable' is also uneven: for Llama 2-7b the original-trained probes outperform the refined-trained probes by about 0.06 at layer 28 (0.666 versus 0.609) and 0.07 at layer 24 (0.708 versus 0.642), and the Cities topic drops from 0.706 to 0.509 at layer 28 and from 0.829 to 0.603 at layer 24, so the wording overstates the level of agreement and the dropped cells deserve discussion.","section":"Section 5.2 and the design of Table 5, in relation to RQ1."}],"minor_comments":[{"comment":"Section 5.1 points to Appendix B for details on excluding the Scientific Facts topic, but Appendix B discusses only the Cities-template restriction and does not describe that exclusion; the pointer should be corrected or the discussion added, since the exclusion changes the topic set in Table 5 from six to five topics and affects comparability with Tables 1 and 2.","section":"Section 5.1 and Appendix B."},{"comment":"The perplexity-based pipeline of Section 4.1 assumes that a low-perplexity true statement indicates model knowledge (the alpha-filter) and that perplexity is a valid plausibility proxy, an assumption the paper itself calls into question in the Limitations by citing Duan et al. (2024); one sentence in Section 5.2 acknowledging how this affects the interpretation of Table 5 would help calibrate the RQ1 answer.","section":"Section 4.1 and the Limitations section."},{"comment":"Table 6 reports per-source accuracies over very different sample sizes (97 to 1054 sentences per source in Table 8), and the reader must cross-reference the two tables to know which averages are reliable; adding the sentence counts to Table 6, or a note in the caption, would improve readability.","section":"Table 6 and Table 8."},{"comment":"The probability distribution in Eq. (2) is typeset ambiguously; writing the denominator explicitly as the sum over candidates in C* would remove the ambiguity, and the relationship between the sampling parameters k = 10 and p = 0.9 (Section 5.1) and the threshold parameters alpha and beta should be stated once in a single place.","section":"Eq. (2) and Section 5.1."},{"comment":"The source name 'wwwodquiz' in Table 8 appears as 'odquiz' in Table 6; the two should be aligned.","section":"Tables 6 and 8."},{"comment":"In Section 3.1 the comparison of the Table 2 baselines with SAPLMA requires cross-referencing Table 1, since the key point is that all baselines are near chance (0.537-0.559) while SAPLMA reaches about 0.65-0.83 on the same data; restating those two reference points in the text would make the reproduction claims easier to verify.","section":"Section 3.1 and Table 2."}],"recommendation":"major_revision","confidential_remarks":"The manuscript header indicates that this is the authors' version of a paper accepted at ACL 2025; this report evaluates the claims as they stand in this version. The reproduction study is careful and the code release is a genuine asset, but the published conclusion will be misleading unless the scope is matched to the Eq. (4) filter. I would urge the authors to add the tau-expansion control, report variance or significance, and either obtain human agreement on a sample of oracle labels or explicitly qualify the RQ2 claim. I have no concerns about citation practices, and the self-citations appear only as background context."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Before you cite this as evidence that hidden-state probes don't generalize to LLM-generated facts, know that the negative result is demonstrated only on a deliberately chosen subset: questions where the target model's correctness over 10 generations is near chance (Eq. 4, |p_i - 0.5| < 0.1). That is the hardest regime for any factuality signal, so the abstract's 'generalization to LLM-generated datasets remains challenging' is broader than what the data support.\n\nThe paper earns its keep in other ways. The reproduction of Azaria and Mitchell is careful and useful—20 probes per layer, consistent trends, small deviations. The two dataset construction strategies are genuinely new: perplexity-weighted negative sampling makes false statements much more plausible, and the LLM-generated fact pipeline from QA collections with GPT-4o mini as oracle is a reasonable, reusable recipe. The failure of SAPLMA on the filtered subsets is consistent across TriviaQA, SQuAD 2.0, and TruthfulQA, and code is provided.\n\nThe load-bearing soft spot is the filter itself. By keeping only items with p ≈ 0.5, the authors select the model's own knowledge boundary, where hidden-state signals, if they track confidence, should be weakest. Without a control with the filter relaxed (larger tau) or removed entirely, the paper's central negative claim is really about the boundary subset, not LLM-generated facts in general. The paper does state the assumption and the filter, but the abstract and conclusion don't carry the qualification. A second issue: every result is a point estimate, no error bars or significance tests, even though the reproduction uses 20 seeds. Third, the RQ1 'mostly comparable' verdict hides large per-topic drops (e.g., Llama Cities at layer 28: 0.706 to 0.509); the averages look fine because other topics improve. The oracle labels from GPT-4o mini are a defensible choice, and the paper's Limitations already flag the need for manual verification—that's the right instinct.\n\nThis is a paper for the interpretability and hallucination-detection subfield. It deserves a serious referee; the reproduction alone justifies a reading, and the new dataset recipes will be used by others. But the headline should be scoped to the ambiguous-subset result, or the missing control should be run. As it stands, I'd take the finding as 'probes fail where the model is maximally uncertain,' not as a blanket obituary for hidden-state factuality.","headline":"A mostly solid reproduction plus a genuinely new negative result, undercut by a dataset filter that selects only the model's hardest, most ambiguous facts.","tokens_in":21794,"tokens_out":3512,"would_cite":true,"duration_ms":28300,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hidden-state factuality probes trained on templated sentences fail to classify facts generated by LLMs themselves.","keywords":["factuality encoding","hidden states","probing classifiers","hallucination detection","self-evaluation","LLM-generated text","truthfulness","perplexity-based negative sampling"],"falsifier":"A replication in which the same probes are trained and tested on an LLM-generated fact dataset whose labels are verified by human annotators, and any layer reaches clearly above-chance accuracy (for example above 0.65), would contradict the paper's central conclusion.","tokens_in":20680,"feed_emoji":"🧠","tokens_out":8295,"duration_ms":56497,"temperature":0.7,"pith_summary":"The paper asks whether a language model's internal activations really encode whether a statement is true, as earlier work claimed, and whether that signal survives when statements come from the model's own generative distribution. It first reproduces the SAPLMA probing study on templated true-false sentences and confirms the original results. It then introduces two harder dataset-construction strategies: false statements sampled to be plausible to the model, and true-false facts generated by the model itself from question-answering collections. The central finding is that probes that do well on templated facts fall to near chance on LLM-generated facts from TriviaQA, SQuAD 2.0, and TruthfulQA. The reader should care because it defines the boundary of hidden-state truthfulness: internal factuality signals may exist, but they do not yet transfer to the realistic case of judging the model's own outputs.","feed_headline":"LLM truth probes go chance on model-generated facts","feed_subtitle":"Realistic new tests show hidden-state factuality signals do not transfer to a model's own outputs.","key_machinery":"The carrying object is the SAPLMA probe, a three-layer feedforward classifier trained on hidden-state activations at selected transformer layers and originally proposed by Azaria and Mitchell (2023). The paper builds two new dataset mechanisms around it. The first is perplexity-guided negative sampling, which selects false statements by weighting alternative property values through the target LLM's own token likelihoods, so false sentences are plausible to the model. The second is an LLM-generated fact pipeline that prompts the model multiple times with questions from QA collections, keeps only questions whose correct-answer ratio is near one half, and labels the generated answers with an LLM judge. These mechanisms isolate what changes when evaluation moves from templated statements to model-generated text.","core_discovery":"On the paper's own terms, the discovery is that the factuality-encoding result of Azaria and Mitchell is reproducible on synthetic templated statements but does not carry over to LLM-generated facts. Using a probe trained and evaluated on sentences produced by Llama 2-7b and OPT-6.7b answering TriviaQA questions, accuracy hovers near chance across layers even after threshold tuning, and the same pattern holds for SQuAD 2.0 and TruthfulQA. The authors interpret this as showing that hidden states carry some factuality signal, enough to separate easy true-false templates, but not a general task-independent signal that lets a probe judge whatever the model itself generates.","pith_inferences":["Editorial inference: if the result holds in larger models, it casts doubt on hidden-state probing as a standalone hallucination detector and points toward combining probes with output-level self-consistency or retrieval verification.","Editorial inference: the near-chance probe accuracy could partly reflect the LLM judge's labels rather than an absent factuality signal; a human-annotated subset would separate 'the signal is not in hidden states' from 'the labels do not match what the model was saying'.","Editorial inference: the perplexity-guided negative-sampling strategy is reusable as a benchmark generator that produces hard negatives plausible to the model, which could make other truthfulness evaluations more stringent."],"forward_implications":["SAPLMA-style probing cannot yet serve as a reliable factuality self-evaluation tool for model-generated text, because near-chance accuracy on open-domain QA facts would make abstention or self-correction driven by such probes no better than guessing.","High accuracy on templated true-false datasets is not evidence that an LLM knows when it is lying in general, so evaluations of factuality encoding must include statements drawn from the model's own generative distribution.","Perplexity-based negative sampling produces harder, more plausible false statements and lowers the average perplexity gap between true and false sentences, yet probe accuracy stays comparable, indicating the difficulty comes from the data distribution rather than statement plausibility alone.","Future factuality probes should be trained and tested on LLM-generated facts, and may need uncertainty-aware classification or other signals beyond a fixed hidden-state probe."],"supporting_citations":[{"why":"Supplies the SAPLMA probe architecture, the True-False dataset, and the original claim that hidden states encode factuality, which the paper reproduces and extends.","marker":"Azaria and Mitchell (2023)"},{"why":"Provides the TriviaQA question-answer collection used to elicit LLM-generated true and false facts for the main generalization test.","marker":"Joshi et al. (2017)"},{"why":"Provides SQuAD 2.0 as an additional QA source used to confirm that probes fail to generalize to LLM-generated facts.","marker":"Rajpurkar et al. (2018)"},{"why":"Provides TruthfulQA as a third QA collection whose LLM-generated facts reproduce the near-chance probe pattern.","marker":"Lin et al. (2022)"},{"why":"Supplies the top-k and nucleus sampling rules used in the perplexity-based negative sampling strategy.","marker":"Holtzman et al. (2020)"},{"why":"Defines Llama 2-7b, one of the two open-source LLMs whose hidden states are probed.","marker":"Touvron et al. (2023)"},{"why":"Defines OPT-6.7b, the second LLM whose hidden states are probed.","marker":"Zhang et al. (2022)"}],"fun_headline_variants":["LLM fact probes collapse on model's own claims","Hidden-state truth probes don't transfer to LLM output","Probes spot fake facts in templates but not in LLM text","Factuality encoding in LLMs fails generalization test","Truth signals vanish when LLMs generate the facts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the GPT-4o-mini judge's true and false labels for the LLM-generated answers are accurate enough to serve as ground truth; if those labels are biased or noisy, the near-chance probe results cannot be cleanly interpreted.","fun_headline_variants_meta":{"raw":{"variants":["LLM fact probes collapse on model's own claims","Hidden-state truth probes don't transfer to LLM output","Probes spot fake facts in templates but not in LLM text","Factuality encoding in LLMs fails generalization test","Truth signals vanish when LLMs generate the facts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000157,"raw_usage":{"total_tokens":1182,"prompt_tokens":867,"completion_tokens":315,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":237}},"tokens_in":483,"tokens_out":315,"duration_ms":3157,"temperature":1.0,"reasoning_tokens":237,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:58:46.340883+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A replication in which the same probes are trained and tested on an LLM-generated fact dataset whose labels are verified by human annotators, and any layer reaches clearly above-chance accuracy (for example above 0.65), would contradict the paper's central conclusion.","supporting_citations":[{"cited_title":"Mitchell","cited_arxiv_id":null,"evidence_quote":"Supplies the SAPLMA probe architecture, the True-False dataset, and the original claim that hidden states encode factuality, which the paper reproduces and extends."},{"cited_title":"Weld, and Luke Zettlemoyer","cited_arxiv_id":null,"evidence_quote":"Provides the TriviaQA question-answer collection used to elicit LLM-generated true and false facts for the main generalization test."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides SQuAD 2.0 as an additional QA source used to confirm that probes fail to generalize to LLM-generated facts."}],"review_version":1}