{"id":"2e094513-4989-40a0-8fb4-2d3b96f5786a","arxiv_id":"2505.18575","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LLM response uncertainty and linear probe performance are strongly negatively correlated across six fact-based datasets and six models, with high-uncertainty responses associated with more spread-out feature importance.","lead":"This paper finds that when a large language model answers the same factual question inconsistently, a simple linear probe also struggles to read that fact out of the model's internal representations, and it explains the link through feature sparsity. If the correlation holds, LLM response uncertainty could become a cheap diagnostic for whether a concept is linearly encoded, reducing the need for labeled probe datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central correlation may be an artifact of comparing R2/Spearman across bins whose ground-truth target variance differs; no control for within-bin target distribution is reported.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: bin-level probe performance is treated as a clean measure of concept encoding without controlling for within-bin target variance, entity frequency, or dataset difficulty. This is the right place to apply pressure. The paper provides real evidence for a robust negative correlation across many datasets, models, temperatures, prompts, and window settings, and it honestly discloses exceptions such as uniformly low uncertainty in the USA dataset and the top-20k selection on World. However, all of that evidence is filtered through R2 and Spearman rank coefficients computed separately within each sliding window. Because both metrics depend on the distribution of the target variable inside the segment, a confound can generate the headline correlation without any genuine relationship between response uncertainty and linear decodability. The proposed check is cheap and decisive: normalize RMSE by a global target standard deviation and compute partial correlations controlling for within-segment std(Y). If the correlation survives, the empirical claim is substantially supported; if it collapses, the central claim should be reframed as a bin-composition effect. The mechanistic story in Sec. 5 and Appendix G is also under-derived—the Lasso oracle inequality does not directly bound ridge regression R2—but this is secondary because the empirical correlation is the paper's main contribution. The appropriate verdict is conditional acceptance: the conclusion is promising but not yet established until the confound check is run.","tokens_in":18124,"tokens_out":5343,"duration_ms":50407,"concrete_test":"Recompute the main correlation using a probe metric that is invariant to the within-segment distribution of Y. For each segment, keep the same trained probe, but report (a) test RMSE normalized by the standard deviation of Y over the full dataset (not the segment), and (b) the partial Spearman/Kendall correlation between uncertainty and probe performance with log within-segment std(Y) included as a covariate. If the negative correlation largely survives both checks, the confound does not explain the result. If it attenuates to near zero or flips sign, the headline claim is a segment-composition artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim (Sec. 3.1, Table 1, Fig. 2) is that lower response uncertainty is correlated with higher probe performance across sliding-window bins. The load-bearing assumption is that bin-level R2 and Spearman rank correlations are comparable measures of how linearly decodable the concept is in each bin. This is not controlled for. R2 is defined as 1 - SSE/SST, so it is normalized by the segment's own total variance in the target Y; Spearman is similarly sensitive to the range and tie structure of Y in the segment. If high-uncertainty bins happen to have narrower Y distributions (e.g., spatial entities clustered in a small region, or temporal entities from a short interval), the same level of absolute prediction error yields a lower R2 and a lower rank correlation, even if the linear encoding quality is identical. The paper does not report within-segment std(Y), nor does it include it as a covariate in Table 1. A plausible common cause—obscure/atypical entities produce both variable LLM responses and less structured embeddings—would also produce the observed negative correlation without the proposed feature-importance mechanism. The mechanistic step in Appendix G does not rescue this: Eq. (3) is a Lasso oracle-inequality risk bound applied to ridge regression, and the step from 'more important features' to 'lower probe R2' is asserted rather than derived. The correlation could therefore be a bin-composition artifact rather than evidence about response uncertainty and linear representability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript investigates whether the uncertainty of an LLM's generated responses is correlated with how accurately a linear probe can recover a target concept from the LLM's internal representations. The authors measure response uncertainty as the variance (or entropy) across repeated generations, sort samples by uncertainty, partition them into overlapping sliding-window segments, train a ridge-regression probe on each segment, and report the rank correlation between average segment uncertainty and probe performance (R2 and Spearman). Across six time/space datasets and six public LLMs, they report strong negative correlations. They then propose a mechanistic explanation based on AttnLRP feature attribution: high-uncertainty responses draw on a larger set of important features, making the probe harder to fit, and they support this with masking experiments and a theoretical argument in Appendix G. The paper also presents case studies in which low-uncertainty concepts produce interpretable embedding structure. The central claim is framed as establishing a link between response uncertainty and probe performance that could enable cheaper probing diagnostics.","tokens_in":18349,"tokens_out":5786,"duration_ms":53873,"significance":"If the empirical correlation is genuine and survives appropriate controls, the result would be practically useful: response uncertainty is cheap to measure, and could serve as a proxy for whether a linear probe will succeed on a given sample set, connecting two currently separate literatures. The paper has clear strengths: the correlation is measured directly rather than fitted, the experiment spans six datasets and six LLM families, both variance and entropy uncertainty estimators are considered, and sensitivity analyses are reported for temperature, sliding-window parameters, and prompt variation. The mechanistic story is plausible and potentially interesting, and the case studies are illustrative. However, the central empirical claim depends on bin-level R2/Spearman comparability, which is confounded by within-bin target variance, and the theoretical support in Appendix G is not derived for the actual ridge probe. These issues are local and fixable, but they are load-bearing for the paper's main conclusion, so the manuscript needs substantial revision.","major_comments":[{"comment":"The central correlation is computed between average response uncertainty and bin-level probe performance measured by R2 and Spearman rank. R2 is normalized by the segment's own total variance of the ground-truth target, and Spearman is sensitive to the range, tie structure, and distribution of distinct target values within the segment. The paper does not report within-segment standard deviations of Y or include them as covariates. If high-uncertainty bins happen to contain targets that are concentrated over a short time interval or small spatial region, the same absolute prediction error would yield a lower R2 and a lower rank correlation even if the linear encoding quality were identical. The authors should control for segment target variance (for example, by reporting RMSE evaluated under a global variance normalization, or by computing partial correlations with within-bin std(Y) as a covariate), or otherwise demonstrate that the correlation is not a bin-composition artifact. This is necessary to support the headline claim that lower response uncertainty implies more accurately probeable representations.","section":"Section 3.1, Table 1, Figure 2"},{"comment":"The mechanistic claim that high-uncertainty responses are associated with a larger set of important features rests on comparing probe performance across Low/Mid/High uncertainty subsets, but the method described in the text trains the probe on the entire dataset and then evaluates per-subset performance after masking features selected by AttnLRP with respect to the LLM response, not the probe. The sentence stating that \"the probe trained on the high-uncertainty subset requires a larger number of features\" is inconsistent with this reported procedure. The observed differences could reflect distribution shift between subsets rather than the size of the feature set needed for probing. The authors should train independent probes on each subset, use consistent Remove-and-Retrain methodology for each subset, provide error bars across random seeds, and report results for more than the two datasets currently shown in the main text and appendix.","section":"Section 5.1, Figure 6"},{"comment":"The theoretical argument applies a Lasso oracle inequality to what is, in the paper's experiments, a ridge-regression probe (Eq. (2)). The bound in Eq. (3) controls the excess risk of Lasso in a high-dimensional sparse linear model; it does not apply to the ridge objective, and it does not imply that a larger number of important features causes lower probe R2. In addition, the premise that higher response uncertainty implies a larger truly important feature set is asserted rather than derived. The authors should either derive a relevant bound for ridge/linear regression in the exact setting used (or for the probe metric actually reported), or remove the claim that the mechanism is theoretically supported. As written, the theoretical analysis does not rescue the empirical correlation from the bin-composition confound.","section":"Appendix G, Eq. (3)"}],"minor_comments":[{"comment":"Table 1 reports correlations without confidence intervals or significance tests; because sliding-window segments overlap, the effective number of independent observations is smaller than the number of segments. The authors should report bootstrap or other valid uncertainty estimates, or at least state the number of segments and their dependence structure.","section":"Table 1 and Section 4.1"},{"comment":"The decision to compute World-dataset correlations on the top 20,000 most uncertain samples is a data-dependent choice that is not accompanied by a sensitivity analysis on the cutoff. The paper should justify this cutoff and show that the conclusion is stable across reasonable cutoff choices, especially since the World dataset is one of the six datasets supporting the central claim.","section":"Table 1 footnote"},{"comment":"The text refers to \"Section 5.3\" in the Table 7 caption and in Appendix H, but the case-study section is numbered Section 5.2; the cross-references should be fixed.","section":"Section 5.2 / Table 7"},{"comment":"The cited reference is written as \"Templeton Adly, Conerly Tom, ...\" but the author is conventionally cited as Adly Templeton et al.; the entry should be corrected for consistency with the in-text citation \"Adly et al.\".","section":"Reference [5]"},{"comment":"The caption reads \"Std R2 score Spearman rank correlation\"; the intended labels are \"R2 score\" and \"Spearman rank correlation\" with a separator. Please clarify the caption and legend.","section":"Figure 2 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a useful empirical observation, but the central correlation could be an artifact of within-bin target variance, and the theoretical mechanism is not yet sound. I do not see this as a reject: the confound can be addressed with additional controls or alternative metrics, and the mechanistic claims can be revised or softened. If the authors are willing to add those analyses, the result could become solid enough for publication. I would also encourage the editor to ask for full reproducibility details, since the paper does not include code or exact response-generation configurations beyond model names, temperatures, and prompts."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful core of this paper is the direct measurement: across six datasets and six LLMs, lower response uncertainty in repeated generations tracks with higher linear-probe performance on the same concept, whether uncertainty is variance or entropy, and whether probe performance is R2 or Spearman. That correlation is new relative to the cited work—Kossen et al. go the other direction, predicting semantic entropy from probes—and the finding is plausibly useful as a cheap screening signal for where probing will work. The sensitivity runs (temperature, window/stride, prompt) are more thorough than most interpretability papers, and the authors are honest about exceptions like USA and World.\n\nThe soft spots are in the framing, not the headline measurement. The stress-test concern about bin-level R2 is legitimate: R2 normalizes by the segment's own target variance, and Spearman responds to tie structure, so if high-uncertainty bins happen to contain less spread in death years or coordinates, the correlation could be inflated. The authors do not report within-segment std(Y) or control for entity difficulty. I do not think this kills the claim—the pattern is consistent across two uncertainty measures and two probe metrics, and the common-cause story (obscure entities are both uncertain and less linearly encodable) still leaves the correlation as a useful empirical regularity—but it is a control they need to run.\n\nThe mechanism is the weakest part. Section 5 shows that high-uncertainty subsets require more features to maintain probe performance, which is suggestive. But the Appendix G step from 'more important features' to 'lower R2' is asserted, and the Lasso oracle inequality is invoked for a ridge probe without derivation. That is post-hoc rationalization, not a theory. The paper would be stronger if it presented the correlation as the result and the feature-importance analysis as a preliminary mechanistic observation.\n\nOther nits: no code or data released, no confidence intervals on the correlations, and the World dataset's top-20k cutoff is disclosed but not justified.\n\nWho should read it: people building interpretability pipelines who want a cheap proxy for probe trainability, and hallucination researchers interested in the inverse direction. It deserves a serious referee; the central empirical claim is testable and the methodological gaps are addressable. I would recommend acceptance conditional on the within-bin variance control and a modest rewrite of the mechanism section.","headline":"A genuinely new empirical correlation between response uncertainty and probe performance, with a mechanistic story that outruns the evidence.","tokens_in":18910,"tokens_out":2556,"would_cite":true,"duration_ms":22932,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper reports a strong negative correlation between how consistently an LLM answers a factual question and how accurately a linear probe can decode the underlying concept from the model's hidden representations.","keywords":["LLM interpretability","probing","response uncertainty","linear representation hypothesis","feature attribution","AttnLRP","hallucination detection","semantic entropy"],"falsifier":"Re-run the uncertainty-sorted bin analysis while controlling for the variance of ground-truth labels within each bin (e.g., matching bins to have similar spread of death years or coordinates). If the negative correlation between response uncertainty and probe performance largely disappears under this control, the claimed relationship is an artifact of bin composition rather than the proposed feature-sharing mechanism. A second check: test the correlation on a base (non-instruction-tuned) model; the paper's own observation predicts it should be much weaker, and a null result there would refine the scope of the claim.","tokens_in":17871,"feed_emoji":"🧠","tokens_out":7683,"duration_ms":59619,"temperature":0.7,"pith_summary":"The paper sets out to show that two seemingly separate diagnostics of large language models are actually two views of the same internal fact: the consistency of a model's repeated answers to the same question (response uncertainty) and how accurately a simple linear probe can decode the underlying concept from the model's hidden activations. Across six time- and space-related datasets and six instruction-tuned LLMs from 8 to 72 billion parameters, it reports a strong negative correlation: when responses are less variable, probe performance is higher, with Kendall and Spearman rank correlations frequently below -0.8. It then offers a mechanistic account: high response uncertainty corresponds to relevance spread across a broader set of features, which makes a low-dimensional linear probe a worse fit, a claim supported by feature-attribution experiments and a sparsity-based generalization bound. If the connection holds, response uncertainty can act as a cheap, label-free indicator of whether a concept is linearly decodable, and can guide where to look for interpretable representations.","feed_headline":"When LLM answers waver, concepts resist linear probes","feed_subtitle":"Repeated-question consistency flags which hidden representations linear probes can decode.","key_machinery":"The load-bearing machinery is the uncertainty-binned probe analysis: query the model repeatedly for the same factual prompt, estimate per-sample response uncertainty (variance for continuous values, with entropy as a check), sort samples by uncertainty, divide them into overlapping sliding-window segments, train a separate ridge-regression probe on the hidden-activation representation of each segment, and correlate segment mean uncertainty with probe performance. The supporting mechanism is AttnLRP feature attribution, which assigns importance scores to latent features with respect to the generated response; combined with Remove-and-Retrain masking (RoAR), it shows the probe and the generation process rely on the same concentrated feature set. The theoretical piece is a sparsity-based argument: if responses vary more, the union of important features across response variants is larger, and the Lasso oracle inequality (a high-probability bound on the gap between empirical and true risk for sparse regression) makes the ridge probe's fit harder as the number of relevant features grows.","core_discovery":"The central claim is that 'when an LLM exhibits lower response uncertainty (i.e., less variability in its outputs), its corresponding internal representations can be more accurately modeled by a linear probe,' and conversely that high uncertainty signals a larger set of important features that degrade probe training. The paper demonstrates this by sorting samples by per-sample response variance or entropy, splitting the sorted data into overlapping sliding-window segments, training a ridge-regression probe on each segment, and correlating segment-level mean uncertainty with probe $R^2$ and Spearman rank scores; the correlation is strongly negative across six datasets and six models. Feature-attribution experiments using AttnLRP (attention-aware layer-wise relevance propagation) show that keeping only the top 20-30% of response-relevant features preserves probe performance, and that high-uncertainty subsets require a larger fraction of features to maintain performance, consistent with the proposed mechanism. The paper also shows that low-uncertainty examples yield interpretable visualizations, such as brand categories, a circular clock-time embedding, and a birth-year probe.","pith_inferences":["An untested extension is whether the correlation is causal: a controlled experiment that intervenes on representation sparsity (e.g., by pruning or steering features) and measures both probe performance and response uncertainty would distinguish a shared mechanism from a common confound.","The paper leaves open whether bin-level correlations are distorted by the spread of ground-truth values within a bin; a follow-up that stratifies by ground-truth variance or entity frequency could settle whether the correlation is an artifact of sample composition.","The observation that the correlation appears mainly in instruction-tuned models suggests a testable prediction: base models should show a much weaker relationship, and instruction-tuning should strengthen it; this is grounded in the paper's own remark but not experimentally quantified.","If the sparsity mechanism is right, techniques that increase activation sparsity or concentrate feature importance should improve both linear probeability and answer consistency, which would open a practical route to more interpretable and more reliable LLMs."],"forward_implications":["Response uncertainty can serve as a lightweight, label-free diagnostic: datasets or samples with limited uncertainty are likely to yield probeable concepts, reducing the need for exhaustive data labeling and trial-and-error probing.","Probe performance and generation consistency reflect the same representational substance: both degrade or improve together as the number of important features in the latent space grows or shrinks.","Low-uncertainty subsets can be mined for interpretable structure without training probes first, as demonstrated by the brand, clock-time, and birth-year examples.","The finding links the probing and hallucination research threads, suggesting that uncertainty measurements and probe analyses can be traded off depending on which is practically available."],"supporting_citations":[{"why":"Supplies the six time/space datasets, the data collection methodology, and the ridge-regression probe training and evaluation setup that the paper adapts.","marker":"[21]"},{"why":"Provides AttnLRP, the attention-aware layer-wise relevance propagation method used to compute per-feature importance scores for the response.","marker":"[3]"},{"why":"Defines the Remove-and-Retrain (RoAR) paradigm used to test whether the probe relies on the same important features as generation.","marker":"[4]"},{"why":"Supplies the Lasso oracle inequality that the paper invokes to bound the ridge probe's empirical risk gap by the number of important features.","marker":"[11]"},{"why":"Provides the semantic entropy method for detecting hallucination that motivates using response variability as an uncertainty signal.","marker":"[20]"},{"why":"Furnishes the semantic-entropy approach to response uncertainty, adapted here for entropy-based uncertainty estimation on temporal concepts.","marker":"[29]"},{"why":"Cites evidence that only a small subset of features is highly important for LLM predictions, the sparsity premise of the mechanistic argument.","marker":"[31]"},{"why":"Supports the sparsity premise by showing that attention heads and features can be pruned without losing performance, reinforcing that few features matter.","marker":"[45]"}],"fun_headline_variants":["Low uncertainty predicts linear probe success","Probe accuracy falls when LLM responses waver","Stable LLM answers map to simpler probe features","Uncertain outputs inflate feature sets, hurting probes","Linear probing works best on confident LLM outputs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that probe performance measured on a bin of samples cleanly reflects how well the concept is encoded in the representations of those samples, independent of confounds such as how spread out the ground-truth values are within the bin, how familiar the entities are, or how hard the subset is; if these differ systematically between low- and high-uncertainty bins, the correlation could arise without the proposed feature-sparsity mechanism.","fun_headline_variants_meta":{"raw":{"variants":["Low uncertainty predicts linear probe success","Probe accuracy falls when LLM responses waver","Stable LLM answers map to simpler probe features","Uncertain outputs inflate feature sets, hurting probes","Linear probing works best on confident LLM outputs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001003,"raw_usage":{"total_tokens":4231,"prompt_tokens":924,"completion_tokens":3307,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":3236}},"tokens_in":540,"tokens_out":3307,"duration_ms":19868,"temperature":1.0,"reasoning_tokens":3236,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:29:25.571945+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the uncertainty-sorted bin analysis while controlling for the variance of ground-truth labels within each bin (e.g., matching bins to have similar spread of death years or coordinates). If the negative correlation between response uncertainty and probe performance largely disappears under this control, the claimed relationship is an artifact of bin composition rather than the proposed feature-sharing mechanism. A second check: test the correlation on a base (non-instruction-tuned) model; the paper's own observation predicts it should be much weaker, and a null result there would refine the scope of the claim.","supporting_citations":[{"cited_title":"Language models represent space and time","cited_arxiv_id":null,"evidence_quote":"Supplies the six time/space datasets, the data collection methodology, and the ridge-regression probe training and evaluation setup that the paper adapts."},{"cited_title":"Attnlrp: attention-aware layer-wise relevance propagation for transformers","cited_arxiv_id":null,"evidence_quote":"Provides AttnLRP, the attention-aware layer-wise relevance propagation method used to compute per-feature importance scores for the response."},{"cited_title":"Springer Science & Business Media, 2011","cited_arxiv_id":null,"evidence_quote":"Supplies the Lasso oracle inequality that the paper invokes to bound the ridge probe's empirical risk gap by the number of important features."}],"review_version":1}