{"id":"5eea766d-f0b1-4531-916f-f93219a0b871","arxiv_id":"2608.21766","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Across six language models, evaluation awareness is linearly decodable from activations but only weakly correlated with verbalization, and steering along the probe direction shifts verbalization.","lead":"Researchers probed six open-weight language models and found that their internal representations distinguish evaluation from deployment contexts, but these internal signals rarely match what the models verbalize about being tested. The study matters because it shows that measuring only one facet of evaluation awareness can mislead AI safety evaluations.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The probe AUROC may reflect corpus-level differences between benchmark and chat prompts, not evaluation awareness per se; a matched-pair validation is needed before the representation and steering claims stand.","rationale":"I read the paper as making three linked claims: (i) linear probes decode evaluation awareness from residual streams; (ii) this internal signal is only weakly coupled with verbalized awareness; (iii) steering along the probe direction causally shifts verbalization. Claim (i) is the foundation: claims (ii) and (iii) inherit their interpretation from the probe. The probe's training signal is explicit evaluation-framing language, but its evaluation uses unmatched corpora. The paper's own Limitations state that 'these prompt distributions also differ in topic, style, and length' and respond that the contrastive-pair construction 'makes the probe direction unlikely to be dominated by surface-level dataset features.' That is a defensible design argument, but it is not a control: the direction could still align with a between-corpus feature that correlates with evaluation status in the test set. The random-probe control rules out arbitrary activation directions, not this specific confound. A matched-pair AUROC test would settle it. The reader's concern about the LLM-judge is also real: the two judges built on GPT-5.4 agree with kappa at most 0.09, and the judge is not validated against humans. However, Appendix D partially defends J3 by showing that the disagreements are predominantly persona/task-meta reasoning that the rubric deliberately excludes. The probe confound, by contrast, is not tested at all. For these reasons I identify the probe confound as the single most load-bearing concern, while acknowledging the judge issue as a secondary limitation. I keep the verdict CONDITIONAL because the paper's claims are plausible and the required validation is straightforward.","tokens_in":32815,"tokens_out":7817,"duration_ms":74581,"concrete_test":"Construct a matched held-out evaluation set by taking, say, 200 diverse base tasks and creating two prompt versions per task that differ only in evaluation framing (e.g., prepend 'You are being evaluated on this task' vs 'A user asks you this'), keeping length and style identical. Compute the probe's AUROC on these matched pairs at the best layer for each model. If AUROC remains above 0.7 (or well above random), the representation claim survives; if it drops to near 0.5, the reported AUROC is a corpus artifact and the steering results must be reinterpreted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central representational claim rests on a probe trained on 16 contrastive pairs that explicitly label evaluation vs deployment framing, then evaluated on held-out prompts from MASK/GSM8K/TruthfulQA vs WildChat. These corpora differ in topic, length, formatting, and instruction style, so the AUROC>0.7 at the best layer may be driven by linear separability of corpus-level features (e.g., question-like vs request-like last tokens, presence of persona instructions in MASK) rather than by a representation of evaluation awareness per se. The random-direction control (Appendix E.1) only shows the learned direction is informative about the train/test split; it does not establish that the direction is specific to evaluation framing. The same ambiguity propagates to the steering experiment in Section 4: adding a direction that separates corpora could shift verbalization by altering text style rather than by activating an evaluation-awareness representation. The paper acknowledges dataset differences in Limitations but does not provide a matched-pair evaluation to rule out this confound. This is the most load-bearing concern because if the probe tracks format rather than evaluation status, the AUROC claim, the training-stage trajectory, and the steering causal claim all lose their stated interpretation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a systematic study of evaluation awareness in six open-weight language models across four measurement facets: linear-probe classification of evaluation vs deployment prompts in residual streams, LLM-as-judge scoring of verbalized awareness, activation steering along the probe direction, and tracking across Olmo3 Base/SFT/DPO/Think checkpoints. The main reported findings are: (i) AUROC ≥ 0.7 at the best layer for all six models; (ii) weak per-prompt coupling between probe scores and judge scores (|ρ|<0.19, MI<0.04 nats); (iii) steering along the probe direction increases judge scores by 0.2–1.5 on a 0–3 scale for the models tested; and (iv) post-training stabilizes the probe direction (cosine ≥0.998 for SFT/DPO/Think) while steering effects grow and localize.","tokens_in":33043,"tokens_out":8090,"duration_ms":66931,"significance":"The paper is a well-organized empirical contribution with several good methodological practices: held-out evaluation of the probe, random-direction baselines for classification and coupling, repeated seeds, and an interventional steering test rather than correlational evidence alone. The checkpoints analysis across training stages is a useful addition to the evaluation-awareness literature. If the main representational claim survives a matched-pair control and the verbalization measure is validated against human raters, the dissociations reported here would be a valuable caution for benchmark design. In its current form, the two load-bearing measurement choices—corpus-level probe evaluation and an unvalidated single LLM judge—mean the stated conclusions are not yet established.","major_comments":[{"comment":"The headline claim that evaluation awareness is linearly decodable (best AUROC ≥ 0.7) is based on evaluating a probe built from 16 artificial contrastive pairs on held-out prompts from WildChat versus GSM8K, TruthfulQA, and MASK. These corpora differ in topic, style, length, and instruction format, and the random-direction control (Appendix E.1) only establishes that the learned direction is informative about this particular train/test split, not that it is specific to evaluation framing. The Nemotron3-49B result in Appendix E.2 makes the confound concrete: under mean-token readout, the same probe direction achieves AUROC ≈ 0.3 on MASK vs WildChat, i.e., its orientation reverses relative to the GSM8K/TruthfulQA calibration, even though |AUROC − 0.5| ≈ 0.2 is claimed as 'real' separation. This shows the direction is not a stable evaluation-vs-deployment axis across benchmarks. A matched-pair test—where evaluation and deployment prompts are equated on topic and length and differ only in evaluation framing—is needed to support the representational, training-stage, and steering interpretations. The Limitations paragraph acknowledges the dataset differences, but the mitigation ('unlikely to be dominated') is not a control.","section":"Section 3, Appendix E.2"},{"comment":"The verbalization score is produced by a single proprietary LLM judge (GPT-5.4) that is explicitly not validated against human raters (Appendix C.1). Appendix D.3 reports that a second judge built on the same backbone has at most Cohen's κ = 0.09 agreement with the primary 0-3 rubric on post-trained Olmo checkpoints. Since the per-prompt dissociation (|ρ|<0.19, MI<0.04) and the steering effect (ΔJ up to 1.5) are both defined in terms of this judge, the central 'verbalization' construct lacks measurement validity. Appendix D.4 shows that most judge disagreements are persona/task-meta reasoning that the 0-3 rubric deliberately excludes, which mitigates the concern for the conservative rubric; nevertheless, the paper needs an external grounding of the rubric, for example a small human-rater agreement study or an independent judge from a different model family, before the dissociation and steering claims can be interpreted as properties of evaluation awareness.","section":"Section 3, Appendices C.1 and D"},{"comment":"The steering vector scale mℓ is computed as the mean activation norm over all sequence positions, while the probe direction is extracted at the final non-padding token; the authors state this asymmetry as an implementation detail. Because padding statistics vary with batch composition and sequence length, the effective steering strength at a given α may differ across models, layers, and checkpoints in an uncontrolled way. The training-stage comparison ('steering grows more pronounced at every successive stage') and the quantitative comparisons of peak ΔJ across models therefore rest on a scale convention that has not been validated. The paper should either align the token subsets used for direction and magnitude, or report a sensitivity analysis showing that the qualitative findings are unchanged under an alternative consistent scaling.","section":"Section 4, Appendix H.1"}],"minor_comments":[{"comment":"The description of the random-direction baseline says 'We average the random baseline over multiple seeds and report the maximum across seeds,' which is internally contradictory; please clarify whether the reported baseline is the mean or the maximum over seeds, and specify how it is used as an upper bound.","section":"Appendix G.2"},{"comment":"The exact 16 contrastive prompts used to construct the probe are described only verbally; for reproducibility, list the full prompt pairs in an appendix or state that they are released verbatim in the code repository.","section":"Section 2, Appendix B.2"},{"comment":"The unsteered baseline for Qwen3-32B reports 96.0% score 0, differing from Table 1's 94.73% for the same model; please reconcile or explain the difference (e.g., different generation temperature or exclusion of empty generations).","section":"Appendix H.3, Table 10"},{"comment":"The response-last readout analysis is described as using 'the response's last token score'; please state explicitly whether the last token is the final token of the generated response before or after any special end-of-sequence token, and how truncated responses are handled.","section":"Appendix G"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThis paper is worth reading: it is the first systematic cross-model, cross-stage comparison of evaluation awareness measured three ways (linear probe, LLM-judged verbalization, steering). The main finding—internal representation and verbalization dissociate, while steering along the probe direction changes verbalization—is real if the measures hold up. The authors did careful work: random-probe controls, 10 seeds, held-out evaluation, honest limitations, and released code.\n\nThe strongest part is the training-stage analysis on Olmo3 checkpoints: the probe direction is nearly identical across SFT/DPO/Think (cosine ≥ 0.998) and distinct from Base, while steering effects grow and localize. That is a genuinely interesting empirical pattern.\n\nSoft spots, in proportion. Two matter. First, the probe AUROC may reflect corpus differences rather than evaluation awareness per se. The probe is trained on 16 matched contrastive pairs (good), but evaluated on MASK/GSM8K/TruthfulQA vs WildChat, which differ in topic, length, and formatting. Random-direction controls only show the learned direction carries information, not that it is specific to evaluation framing. A matched-pair test on held-out data (swapping framing within each benchmark) would close this. As is, the central representational claim is not airtight.\n\nSecond, the verbalization measure rests entirely on an unvalidated LLM judge. The paper's own second-judge analysis shows near-zero kappa with the primary rubric. The appendix argues the disagreements are mostly persona/task-meta reasoning, which is reassuring, but it is still a single proprietary backbone with no human validation. Since the dissociation and steering results are defined through this judge, measure validity matters. If the judge tracks something else, the dissociation is an artifact.\n\nNeither flaw is fatal. The paper is transparent about both, and the contrastive probe construction plus the interventional steering results give me reasonable confidence the phenomenon is real. But I would want matched-pair validation and a human-validated judge before treating the numbers as solid.\n\nWho this is for: anyone working on situational awareness, benchmark validity, or AI safety measurement. It is a useful reference for the claim that single-metric evaluations are insufficient.\n\nRecommendation: send it to peer review. It is a serious empirical paper with real controls and honest limitations; the flaws are addressable in revision. A good referee would ask for the matched-pair test and clearer judge validation, but the core contribution deserves a venue.","headline":"A careful, honest empirical study of evaluation awareness whose central dissociation claim is plausible but needs matched-pair controls and a validated verbalization measure.","tokens_in":33542,"tokens_out":2491,"duration_ms":22705},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Across six open-weight language models, a linear probe finds a strong internal signal of being evaluated, but that signal only weakly predicts what the models say, and steering can change what they say.","keywords":["evaluation awareness","linear probing","activation steering","verbalization","residual stream","LLM-as-a-judge","post-training dynamics"],"falsifier":"Rescore the same 30,000 generations and the steered outputs with a scoring rubric that has been validated against human raters; if the human-validated verbalization score shows per-prompt mutual information with the probe clearly above the random-direction baseline, then the reported dissociation and the low verbalization rates would turn out to be artifacts of the unvalidated GPT-5.4 judge rather than properties of the models.","tokens_in":32636,"feed_emoji":"🧠","tokens_out":5597,"duration_ms":46772,"temperature":0.7,"pith_summary":"The paper tests evaluation awareness: whether a language model can detect that it is under evaluation rather than in ordinary deployment, and what that detection does to behavior. It reports that across six open-weight models (Qwen3-8B/32B, Olmo3-7B/32B, Gemma4-31B, Nemotron3-49B), a linear probe built from 16 contrastive prompt pairs separates evaluation-style from deployment-style prompts in the residual stream with AUROC at or above 0.7 at the best layer. It further reports that this internal separation is almost uncoupled from what the model verbalizes: per prompt, the probe score's Spearman correlation with an LLM-judge's 0–3 verbalization score stays below 0.19 in absolute value and mutual information below 0.04 nats. Steering along the probe direction, however, does increase verbalization scores by 0.2 to 1.5 points on the same 0–3 scale, showing a causal route. A sympathetic reader should care because benchmarks assume test behavior reflects deployment behavior; if internal awareness and expressed awareness diverge, evaluation protocols that look at only one side can be misled.","feed_headline":"All six LLMs internally sense being evaluated","feed_subtitle":"Benchmarks assume tests reveal real behavior; this study shows internal and verbalized awareness can diverge.","key_machinery":"The load-bearing object is the contrastive probe direction $\\hat{v}_\\ell$, computed as the unit-normalized mean difference between last-token residual-stream activations for 16 paired deployment-framed and evaluation-framed prompts at layer $\\ell$. The projection of any prompt's activation onto $\\hat{v}_\\ell$ is the probe score used for classification and coupling, and the appropriately scaled version of the direction can be added to activations during generation following the Contrastive Activation Addition protocol for steering. Because the same direction is used for all three measurements, the paper can compare representational separation, verbalization coupling, and causal influence without changing the underlying feature.","core_discovery":"The paper's central claim is that evaluation awareness is a measurable property of current open-weight language models, but that its three observable facets — what the model internally encodes, what it says, and what can be changed by intervention — are not the same signal. Using a linear probe constructed from 16 contrastive prompt pairs, the paper reports that at the best layer every one of the six models separates evaluation-style from deployment-style prompts with AUROC at least 0.7, while random-direction controls sit near 0.5. Yet on individual MASK prompts, probe scores and an LLM-judge's verbalization score are coupled only at the level of random directions (max |ρ| < 0.19, max mutual information < 0.04 nats), even though at least 93.9% of generations receive the lowest verbalization score. Steering along the probe direction toward the evaluation-like side raises the judge score by up to 1.5 points on a 0–3 scale in some models, while positive steering does not consistently lower it. Across Olmo3 checkpoints, the separating direction already exists at Base, is sharpened by SFT, and is then nearly fixed (cosine at least 0.998 across SFT, DPO, and Think), with steering effects localizing into fewer layers after SFT.","pith_inferences":["If the LLM-judge were replaced by human raters, verbalization rates and the probe–verbalization dissociation could look different; the paper's own second-judge comparison shows the 0–3 rubric is conservative in ways that mask persona and task-meta reasoning.","The probe may be carrying prompt-format or framing information rather than a general 'being evaluated' concept; testing on held-out unseen evaluation formats, such as interactive evaluations or novel benchmark designs, would clarify that question.","The steering result raises a practical question the paper does not answer: whether steering-induced verbalization reflects genuine awareness or just a style shift, and whether it transfers to safety-relevant behaviors such as refusal rates.","A direct testable extension is to run the same probe and steering pipeline on deployment-style prompts that are not benchmarks, to see whether the evaluation-versus-deployment separation appears in natural conversation alone."],"forward_implications":["A model can internally encode evaluation context while almost never verbalizing it; text-only monitoring will miss the signal in these models.","Low verbalization rates do not imply the absence of evaluation awareness, and high verbalization rates are not required for internal representation.","Linear steering along the probe direction can increase verbalized evaluation awareness by 0.2–1.5 points on a 0–3 scale in some models, so this behavior is at least partially controllable.","Post-training stage, not scale or base pretraining, is the stage that sharpens and stabilizes the internal direction: SFT, DPO, and Think share a probe direction at cosine at least 0.998.","Benchmark design must treat representation, verbalization, and control as separate measurements, since they do not track one another."],"supporting_citations":[{"why":"Introduces the term evaluation awareness and supplies the initial evidence on frontier models that this paper systematically extends.","marker":"Needham et al. 2025"},{"why":"Supplies the 16 contrastive prompt pairs and the contrastive activation probe construction that the paper replicates.","marker":"Hua et al. 2025"},{"why":"Defines Contrastive Activation Addition, the steering protocol used in the causal intervention experiments.","marker":"Turner et al. 2023"},{"why":"Provides the WildChat corpus used as the deployment-like prompt source.","marker":"Zhao et al. 2024"},{"why":"Provides the MASK benchmark prompts used as the main evaluation-like source for verbalization and steering.","marker":"Ren et al. 2025"},{"why":"Provides the GSM8K grade-school math prompts used in probe calibration.","marker":"Cobbe et al. 2021"},{"why":"Provides the TruthfulQA prompts used in probe calibration.","marker":"Lin, Hilton, and Evans 2021"},{"why":"Supplies the mutual-information estimator used to measure the weak coupling between probe scores and judge scores.","marker":"Kraskov, Stögbauer, and Grassberger 2004"}],"fun_headline_variants":["LLMs encode evaluation awareness, but verbalization lags","All six LLMs show internal signals of being evaluated","Steering LLM thoughts about evaluation changes their words","Probe finds evaluation awareness in every tested LLM","LLM evaluation awareness is real, decodable, and steerable"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that GPT-5.4's 0–3 score measures verbalized evaluation awareness; because the judge is never validated against human raters and a second judge built on the same backbone agrees with it at Cohen's kappa at most 0.09, all claims about how often or how strongly models verbalize awareness rest on that unvalidated ruler.","fun_headline_variants_meta":{"raw":{"variants":["LLMs encode evaluation awareness, but verbalization lags","All six LLMs show internal signals of being evaluated","Steering LLM thoughts about evaluation changes their words","Probe finds evaluation awareness in every tested LLM","LLM evaluation awareness is real, decodable, and steerable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00023,"raw_usage":{"total_tokens":1566,"prompt_tokens":1116,"completion_tokens":450,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":732,"completion_tokens_details":{"reasoning_tokens":369}},"tokens_in":732,"tokens_out":450,"duration_ms":4572,"temperature":1.0,"reasoning_tokens":369,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-27T18:48:31.186784+00:00","model_set":{"reader":"deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.0-alltime-deepseek-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Rescore the same 30,000 generations and the steered outputs with a scoring rubric that has been validated against human raters; if the human-validated verbalization score shows per-prompt mutual information with the probe clearly above the random-direction baseline, then the reported dissociation and the low verbalization rates would turn out to be artifacts of the unvalidated GPT-5.4 judge rather than properties of the models.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}