{"id":"2c581bc9-f18e-4c57-bff3-36bd913bca75","arxiv_id":"2507.00239","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Linear probes recover jailbreak-only answers from aligned models' hidden states, sometimes transfer from base models, and correlate with pairwise preference rankings.","lead":"When a chatbot refuses to answer, the refused answer can still be read from the model's internal states with a simple linear probe. These traces survive safety fine-tuning and can nudge later comparisons, suggesting alignment suppresses answers rather than erasing them.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract's 'actively used' and 'indirectly influential' claim outruns the evidence: Section 5 only correlates probe readouts with jailbroken pairwise comparisons from the same model, so causal/downstream influence is not established.","rationale":"The reader's conditional verdict is appropriate. Section 3 shows strong within-model linear decodability, and Section 4's base-to-instruct transfer is a concrete empirical result; the paper provides code, and the comparative analysis (Figure 6) shows positive cross-task correlations, which are useful internal consistency checks. The weakest load-bearing point is the leap from Section 5's correlation to the abstract's causal-sounding 'actively used' and 'indirectly influential' language. The pairwise comparisons were jailbroken with the same ICL prompt family used to create probe labels, so the two measures share prompt and model sources; a third variable could drive both. The authors themselves flag the lack of non-jailbroken downstream tasks in Limitations, which directly undercuts the abstract's generality. Reported correlations are also maxima over layers chosen on the evaluation set, so point estimates are optimistically biased; this reinforces the need for conditional acceptance. I would not move the verdict from conditional: the empirical core is plausible, but the central claim needs either softer wording ('correlates with' rather than 'actively uses') or an intervention/control establishing behavioral relevance. This is a partial agreement with the reader: I share the call for controls and de-overclaiming, but I locate the highest-risk assumption in Section 5, not in the Section 4 coordinate-frame assumption.","tokens_in":25853,"tokens_out":8852,"duration_ms":110488,"concrete_test":"For the two flagship attributes (Occupations IQ and Percent Women) on gemma-2-9b-it, take the probe and layer used in Section 5 and regenerate the 15,000 pairwise comparisons with the top probe direction removed from the residual stream (subtract w w^T r from the hidden states at that layer during generation). Then re-estimate Bradley-Terry scores and recompute Spearman correlation with the original probe predictions. If the ranking and correlation survive ablation, 'actively used' is not supported; if they collapse, the readout is behaviorally implicated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has three load-bearing parts: (i) refused content is linearly decodable (Section 3), (ii) it persists across instruction-tuning (Section 4), and (iii) it is 'actively used' and 'indirectly influential in downstream behavior' (abstract, Section 5). Part (iii) is the least secure. Section 5 computes Spearman correlations between probe predictions from Section 3 and Bradley-Terry scores estimated from pairwise comparisons. Both quantities are generated by the same model answering the same attribute questions under the same ICL jailbreak, so the correlation may reflect a shared underlying entity-attribute association (for example a broad status/development dimension) rather than the model's decision process reading the specific linear direction probed. The paper's own Limitations section concedes that only pairwise comparisons were tested and that non-jailbroken downstream tasks are left to future work, yet the abstract asserts active use and downstream influence. No control for an unrelated or permuted attribute probe, no partial correlation, and no intervention (e.g., ablating the probe direction) is provided. The reader's alternative concern about Section 4 coordinate alignment is real but less damaging: base and instruct checkpoints share pretrained weights, so linear transfer is a plausible empirical consequence rather than an unexplained coordinate miracle. The causal/downstream overreach is the more load-bearing gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether information that instruction-tuned language models refuse to generate is nevertheless linearly decodable from their hidden states. Using linear probes trained on innocuous prompts such as \"This document describes [entity]\", the authors report that jailbroken responses about attributes like country IQ or occupation substance-abuse rates can be predicted with Pearson correlations often above 0.6 and sometimes above 0.8. They further report that probes trained on base (non-instruction-tuned) models sometimes transfer to instruction-tuned models, suggesting that instruction-tuning preserves linear representations of refused content. Finally, they show that probe predictions correlate with Bradley-Terry scores estimated from jailbroken pairwise comparisons, and interpret this as evidence that the decoded information is actively used in downstream behavior. The experiments use three open-weight models (gemma-2-9b-it, gemma-2-2b-it, Yi-6B-Chat), four entity types, and two jailbreak prompts.","tokens_in":26109,"tokens_out":3969,"duration_ms":44391,"significance":"If the central claim holds, the paper would be a useful bridge between two usually separate literatures: jailbreak attacks on aligned LMs and linear-probing analyses of internal representations. It would also strengthen the practical concern that open-weight models retain linearly accessible traces of refused information after instruction-tuning. The paper has several concrete strengths: the measurement pipeline is straightforward, code is released, the probing targets are specified in detail, and the pairwise-comparison experiment is an attempt to move beyond the probe-fitting loop toward behavioral evidence. The headline claims, however, are currently stronger than the evidence: the \"actively used\" and \"indirectly influential\" language in the abstract goes beyond what a correlational study can establish, and the statistical reporting (best-layer selection without confidence intervals or multiple-comparison correction) makes the quantitative headline numbers less solid than they appear. With appropriate controls and tempered claims, the paper would be a useful empirical contribution to the alignment and interpretability communities.","major_comments":[{"comment":"The claim that refused information is 'actively used' and 'indirectly influential in downstream behavior' outruns the evidence. Section 5 correlates probe predictions (from probes trained on ICL-jailbroken responses, Section 3) with Bradley-Terry scores estimated from pairwise comparisons that are also elicited under the same ICL jailbreak from the same model. This correlation could reflect a shared entity-attribute association or a common prompt-induced response style rather than the model's decision process reading the probed linear direction. The paper provides no control probe (e.g., permuted labels or an unrelated attribute), no partial correlation controlling for a broad status/development dimension, and no intervention such as ablating the probed direction. The full results in Figures 10-12 also contain many near-zero and negative Spearman correlations, and the cross-task correlations in Figure 13 are modest (0.463 for gemma-2-9b-it between main and Bradley-Terry results). The Limitations section concedes that only pairwise comparisons were tested and that non-jailbroken downstream tasks are left to future work, yet the abstract asserts active use and downstream influence. I request either tempering these claims to correlational language or adding controls/interventions that support the causal reading.","section":"Abstract and Section 5"},{"comment":"The headline correlations (e.g., >0.8 for country IQ) are reported after selecting the best layer across all layers, with no confidence intervals and no correction for multiple comparisons. Because the layer is chosen on the same held-out set used to evaluate the probe, the reported performance is an optimistic estimate; leave-one-out cross-validation tunes lambda only, not the layer. I ask for confidence intervals or nested cross-validation over layer selection, and a shuffled-label or random-direction baseline to calibrate the expected correlation under the null. This is load-bearing for the paper's core claim that refused content is linearly decodable from innocuous hidden states.","section":"Section 3.1, Figures 2 and 7"},{"comment":"The base-to-instruct transfer experiment applies probes trained on base model hidden states directly to instruction-tuned hidden states, but the paper does not justify that the two models' residual streams share a common linear coordinate system or that a single linear readout remains valid after fine-tuning. Because base and instruct checkpoints share pretrained weights, transfer is plausible, but controls are needed: for example, probes trained on shuffled labels, random directions, or an attribute that should not transfer. Without such controls, the observed transfer correlations could reflect generic shared structure rather than persistence of the specific refused attribute. Please add these controls or explicitly discuss the coordinate-alignment assumption and its failure modes.","section":"Sections 4.1 and 4.2"}],"minor_comments":[{"comment":"The sentence 'above shown initially refused information indicates can be linearly decodable' contains a grammatical error; 'indicates' should be removed or the sentence restructured.","section":"Section 5 opening"},{"comment":"The caption says 'These two entities were the same...' but IQ and Percent Women are attributes, not entities; please use 'attributes' throughout the discussion.","section":"Figure 4 caption"},{"comment":"The word 'langauge' should be 'language' in the synthetic-names construction paragraph.","section":"Appendix A"},{"comment":"The phrase 'remaining largely in-tact' should be 'remaining largely intact'.","section":"Section 6"},{"comment":"The phrase 'The goal is the assess whether' should be 'The goal is to assess whether'.","section":"Section 4.1"},{"comment":"The description of ICL prompt construction says answers were 'randomly generated ... within a reasonable range' but does not specify the ranges or random seed; please provide the exact ranges or a reference to the code for reproducibility.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a machine-learning or NLP venue and the empirical setup is generally sound, but the abstract and conclusion currently overclaim what a correlational analysis can show. I would advise the editor that the revision should be judged primarily on whether the authors either add the requested controls (random-label baselines, confidence intervals, and ideally a probe-direction intervention) or substantially temper the causal/downstream language. The paper's contribution is real even without the causal claim, and I do not see grounds for rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"If you care about interpretability or safety, give this a serious read. The paper does something concrete: it takes jailbroken responses from three open-weight models, trains linear probes on innocuous hidden states, and shows that a lot of refused content (country IQ, occupation attributes) is linearly decodable. It also shows that probes trained on base models transfer to instruction-tuned versions on several attributes. That is a real and useful empirical contribution, and the Section 5 pairwise-comparison setup is a good idea—it gives a behavioral anchor that wasn't used to fit the probes. Code is linked, the methodology is transparent, and the authors are honest about what they did and did not test. Credit where earned. The soft spots are substantial but not fatal. The abstract says the probed information is 'actively used' and 'indirectly influential in downstream behavior.' What Section 5 actually shows is a Spearman correlation between probe predictions and Bradley-Terry scores from the same model answering pairwise comparison questions under the same jailbreak. Both quantities can plausibly be driven by a shared underlying entity-attribute association (say, a broad status/development dimension) rather than by the model reading the specific probed direction. The authors' own Limitations section concedes that only pairwise comparisons were tested and non-jailbroken downstream tasks are left to future work—so the abstract overclaims relative to the evidence. The missing controls matter too: no permutation tests, no partial correlations, no intervention like ablating the probe direction. Also, the headline Pearson correlations come from best-layer selection across many layers without confidence intervals or multiple-comparison correction, so the 0.8+ country-IQ number is likely inflated. The reader's worry about Section 4 coordinate alignment is less damaging: base and instruct checkpoints share pretrained weights, so linear transfer is a reasonable empirical consequence rather than an unexplained coordinate miracle. Three small models (2B–9B) is a thin base, but for a first pass that's acceptable. Overall, the central measurement is sound in its own terms; the problem is the gap between the data and the causal language in the abstract. This deserves peer review, not a desk reject. A serious referee should ask for the controls, confidence intervals, and a more careful framing of what correlation does and does not imply. I'd bring it to a reading group if someone is working on representation probing, and I'd cite it with a caveat about the overclaim.","headline":"Worth a serious look: the decodability and transfer results are real, but the abstract's 'actively used' claim outruns the correlation evidence and should be tempered before publication.","tokens_in":26590,"tokens_out":1522,"would_cite":true,"duration_ms":19959,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Instruction-tuning suppresses refused answers in outputs but leaves the underlying values linearly readable in hidden states, where those values continue to line up with the models' downstream judgments.","keywords":["linear probing","jailbreak prompts","refusal behavior","instruction tuning","hidden state representations","alignment safety","Bradley-Terry model","linear representation hypothesis"],"falsifier":"Run the Section 4 transfer pipeline with base-model labels scrambled, or with random probe directions matched in norm: if shuffled-label or random probes reach transfer correlations comparable to the real ones, the persistence result is an artifact of marginal statistics rather than a preserved linear direction. A complementary check compares the best base-model probe weight vector with the best instruction-tuned probe weight vector at the same layer via cosine similarity: high similarity would confirm one shared geometry, while near-orthogonal weights would show the transfer rides on something other than a common direction.","tokens_in":25650,"feed_emoji":"🔓","tokens_out":13797,"duration_ms":133217,"temperature":0.7,"pith_summary":"This paper asks whether instruction-tuning, the stage at which language models learn to refuse harmful requests, actually removes the underlying knowledge or merely hides its expression. The authors show that the numbers a jailbroken model will produce—the average IQ of a country, an occupation's substance-abuse rate, a political figure's net worth—can be recovered by a linear probe trained on hidden states from the innocuous prompt 'This document describes [entity],' with Pearson correlations above 0.8 in the strongest cases. Probes trained on base models that never refuse sometimes transfer to the instruction-tuned versions, indicating the refused attributes persist in much the same linear form through alignment. The sharpest claim is that these traces are not inert: probe-predicted values correlate with the model's own pairwise-comparison rankings, which suggests the suppressed information still influences how the model makes judgments. If true, alignment training changes what models are allowed to say more than what they represent and believe.","feed_headline":"Refused knowledge stays linearly readable in aligned language models","feed_subtitle":"Base-model probes transfer to safety-tuned models, and the decoded values track the model's own comparative judgments.","key_machinery":"The argument is carried by three components: linear ridge-regression probes on residual-stream hidden states, $\\hat{w} = (A^\\top A + \\lambda I)^{-1}A^\\top y$, which test whether a scalar attribute lies along a single direction in representation space; the cross-model transfer design, in which a probe trained on base-model hidden states and base-model labels is applied unchanged to instruction-tuned hidden states, with the tuned model's jailbroken responses as the held-out test set; and a Bradley-Terry paired-comparison model that converts the tuned model's pairwise generations into a latent ordinal score per entity. The transfer step is the operational definition of persistence: if a readout learned on the base model keeps predicting after alignment, the linear geometry of the attribute survived. The innocuous probe prompt 'This document describes [entity]' is equally load-bearing, because it extracts representations without ever mentioning the attribute, so high decodability shows the attribute is spontaneously present in the entity's representation.","core_discovery":"The paper's central claim is that instruction-tuning does not wholly eliminate, or even relocate, harmful information in representation space: it merely suppresses its direct expression. Concretely, the authors fit ridge-regression probes to last-token hidden states from 'This document describes [entity]' and show that the resulting readouts predict, often with Pearson correlation above 0.8, the numerical answers a jailbroken model gives for attributes the tuned model refuses to state, across the three instruction-tuned models studied (gemma-2-9b-it, gemma-2-2b-it, and Yi-6B-Chat). Because that prompt never mentions the attribute, high decodability means the attribute is spontaneously present in the entity's representation rather than induced by the question. Transfer experiments then show that probes trained on the base, non-refusing model predict the instruction-tuned model's jailbroken responses on many attribute pairs, which the authors read as evidence that these linear representations survive instruction-tuning largely unchanged. Finally, using a Bradley-Terry model fit to pairwise comparisons, they find Spearman correlations up to 0.824 between probe predictions and the model's latent rankings, concluding that the decoded representations align with, and may steer, suppressed generative behavior expressed more subtly elsewhere.","pith_inferences":["A causal test the paper leaves implicit: editing or ablating the probe direction in the tuned model's activations should move the Bradley-Terry rankings if the representation actually drives the comparisons, turning a correlational result into an interventional one.","The base-to-tuned transfer suggests a cheap audit procedure: train probes once on an open base model and apply them to any fine-tuned variant of the same weights, scanning for refused attributes without jailbreaking the target.","The entity set itself may carry confounds: if probes are reading stereotype statistics of the name or category rather than the attribute, then re-running the comparison task with randomized entity labels would tell whether the probe-model alignment survives the loss of real-world correlates.","For safety engineering, the results point toward interventions in representation space, such as steering or erasing specific directions, rather than output filtering, since the expressive channel can be suppressed without touching the underlying information."],"forward_implications":["Refusal becomes a behavioral gate over unchanged representations: a linear probe trained on innocuous hidden states can read the refused answer without any jailbreak prompt.","Alignment evaluations that inspect only generated text will miss this channel; hidden-state probing offers a complementary audit that catches what suppression leaves behind.","Because base-model probes transfer to tuned models, alignment data need not mention an attribute for its representation to persist, so safety tuning cannot be assumed to scrub any particular fact.","The correlation between probe values and pairwise-comparison rankings implies that a refused attribute can leak into decisions that never name it, such as recommending a course of study based on the model's unstated association between an occupation and IQ.","Probes trained on the jailbreak prompts themselves improve only modestly over innocuous-prompt probes, suggesting the information is present before jailbreaking and is not confabulated on demand."],"supporting_citations":[{"why":"This work introduces linear classifier probes, the methodology adapted here to predict numeric jailbroken responses from hidden states.","marker":"[3]"},{"why":"This work shows that refusal is mediated by a single direction in representation space, grounding the premise that alignment adds refusal behavior without erasing underlying knowledge.","marker":"[5]"},{"why":"This work supplies the paired-comparison model that converts pairwise generations into the latent ordinal rankings correlated with probe predictions in Section 5.","marker":"[11]"},{"why":"This work demonstrates that LMs represent space and time linearly, motivating linear probes as the right readout for high-level attributes.","marker":"[23]"},{"why":"This work provides the ridge-regression estimator and leave-one-out cross-validation used to fit and tune the probes.","marker":"[24]"},{"why":"This work articulates the linear representation hypothesis, that high-level concepts are encoded as linear directions; it is the core assumption of the probing method.","marker":"[42]"},{"why":"This work proposes the Superficial Alignment Hypothesis, that knowledge is learned in pre-training and post-training only changes style; the transfer experiments build on this frame.","marker":"[66]"}],"fun_headline_variants":["Linear probes unmask refused answers in aligned LMs","Refused knowledge stays linearly readable after safety tuning","Base-model probes reveal what aligned LMs refuse to say","Probes decode refused knowledge that alignment only suppresses"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The transfer experiment applies a probe trained on base-model hidden states directly to instruction-tuned hidden states, which assumes the two models' residual streams at corresponding layers share a coordinate frame in which a single linear readout stays meaningful; if instruction-tuning rotates or rescales that subspace, the reported transfer correlations could be an artifact rather than evidence that the refused knowledge persisted.","fun_headline_variants_meta":{"raw":{"variants":["Linear probes unmask refused answers in aligned LMs","Refused knowledge stays linearly readable after safety tuning","Base-model probes reveal what aligned LMs refuse to say","Probes decode refused knowledge that alignment only suppresses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00135,"raw_usage":{"total_tokens":5541,"prompt_tokens":1062,"completion_tokens":4479,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":678,"completion_tokens_details":{"reasoning_tokens":4417}},"tokens_in":678,"tokens_out":4479,"duration_ms":33561,"temperature":1.0,"reasoning_tokens":4417,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:20:27.668927+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Section 4 transfer pipeline with base-model labels scrambled, or with random probe directions matched in norm: if shuffled-label or random probes reach transfer correlations comparable to the real ones, the persistence result is an artifact of marginal statistics rather than a preserved linear direction. A complementary check compares the best base-model probe weight vector with the best instruction-tuned probe weight vector at the same layer via cosine similarity: high similarity would confirm one shared geometry, while near-orthogonal weights would show the transfer rides on something other than a common direction.","supporting_citations":[{"cited_title":"Refusal in language models is mediated by a single direction","cited_arxiv_id":null,"evidence_quote":"This work shows that refusal is mediated by a single direction in representation space, grounding the premise that alignment adds refusal behavior without erasing underlying knowledge."},{"cited_title":"Language models represent space and time","cited_arxiv_id":null,"evidence_quote":"This work demonstrates that LMs represent space and time linearly, motivating linear probes as the right readout for high-level attributes."},{"cited_title":"The Elements of Statistical Learning: Data Mining, Inference, and Prediction, volume 2","cited_arxiv_id":null,"evidence_quote":"This work provides the ridge-regression estimator and leave-one-out cross-validation used to fit and tune the probes."},{"cited_title":"The linear representation hypothesis and the geometry of large language models","cited_arxiv_id":null,"evidence_quote":"This work articulates the linear representation hypothesis, that high-level concepts are encoded as linear directions; it is the core assumption of the probing method."}],"review_version":1}