Pith. sign in

REVIEW 4 major objections 4 minor 31 references

Do All LLMs Know When They're Being Harmful? A Reproducibility Study of Latent-Space Safety Probes Across Model Families

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A cheap latent-space safety probe works across four LLM families and is seed-invariant in this setup.

desk verdict Useful cross-family probe numbers and a clean determinism check, but the main tables contradict the appendix and the first author is reproducing his own work without disclosure. read the letter →

arxiv 2608.08029 v1 pith:45Z5HPR5 submitted 2026-08-08 cs.LG cs.AIcs.CR

Alizishaan Khatri , Dun Li Chan This is my paper · ORCID
classification cs.LGcs.AIcs.CR
keywords latentsafetyprobesharmfulnessdetectionLLMinterpretabilityreproducibilitystudyMLPprobefinal-tokenactivationscross-architecturegeneralizationseedinvariance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is a reproducibility study of the claim that a small MLP probe on the final-layer, last-token activation of an LLM can detect harmful prompts about as well as guard models thousands of times larger. The authors re-implement the original single-model pipeline on LLaMA-3.1-8B and match its F1 scores to within 0.37 percentage points. They then train identical probes on Gemma-4-E4B, Mistral-7B-v0.3, and Qwen2-7B, and find F1 scores within roughly one point of the original on WildJailbreak, BeaverTails, and AEGIS 2.0. Repeating extraction under five random seeds shows the final-token latent vectors are byte-for-byte identical across seeds, so probe F1 variance is zero in this setup.

What carries the argument

The carrying mechanism is the final-layer, last-token hidden state after prompt prefill, i.e. the vector $h_i = M(x_i)[-1] \in \mathbb{R}^d$ for prompt $x_i$ and backbone $M$, followed by a fixed six-layer MLP probe with hidden sizes 2048, 512, 64 and a two-way softmax output. The MLP's input dimension is the only quantity that changes across backbones. This vector is the entire input to the probe; the empirical claim is that it already contains linearly decodable harmfulness information, and the determinism check (SHA-256 hashes of the extracted tensor files) shows the staging step is a pure function of weights and tokens in this setup.

What would settle it

Take a jailbreak set whose unsafe content is distributed across earlier user-token representations, run the probe and a large guard model on identical prompts, and compare F1; the paper's Section 7 cites evidence that final-token probes can miss such prompts, so a clear F1 collapse on that set would falsify the generality of the high benchmark scores. Separately, rerunning activation extraction through a different inference implementation and comparing SHA-256 digests of the tensor files would settle whether the seed-invariance is a property of the models or of the current serving stack.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is twofold: the latent-space probing recipe is portable, and the extraction step is deterministic under the tested settings. Identical MLP probes trained on final-layer, last-token activations reach F1 scores of 99.47 on WildJailbreak, 82.90 on BeaverTails, and 83.42 on AEGIS 2.0 for the reproduced LLaMA-3.1-8B backbone, and stay within roughly a point of those values for Gemma-4-E4B, Mistral-7B-v0.3, and Qwen2-7B. Across five random seeds, the serialized activation tensors from Stage 1 are byte-for-byte identical (SHA-256 digests match), and F1, precision, and recall are constant to at least two decimal places. The paper reads this as evidence that harmfulness is linearly represented in the final token across current aligned open-weight chat models, while cautioning that the determinism result is protocol-specific.

Load-bearing premise

The whole result rests on the assumption that the final-layer activation at the last token after prompt processing carries enough harmfulness signal for detection, so prompts whose unsafe evidence is spread across earlier tokens would be invisible to this probe.

Editorial extensions

If this is right

  • The same probe hyperparameters transfer across Llama, Gemma, Mistral, and Qwen backbones without per-family tuning, which makes the pipeline a plausible drop-in replacement for a guard model.
  • Because activation extraction is seed-invariant here, a single extraction run is enough to judge a model's probe performance; multi-seed averaging adds nothing for these backbones and datasets.
  • The F1 spread between the best and worst backbone is about one point, so alignment recipe differences among open-weight chat models do not materially change the harmfulness information carried by the final token.
  • If the result holds beyond the 4B-8B open-weight range, latent probing could replace expensive external classifiers in production moderation pipelines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the byte-for-byte seed invariance implies the extraction pipeline is a pure function of weights and tokens in this setup; if that holds on other serving stacks, probe training results become fully deterministic and reproducibility artifacts need only be produced once.
  • The paper's cited limitation suggests final-token probes can miss jailbreaks whose evidence is spread across earlier tokens, so one testable extension is to pool activations across all user-token positions and check whether F1 rises on adversarial jailbreak sets while staying stable on the standard benchmarks.
  • A further extension is to measure whether the zero-variance result survives when the same model is run through a different inference backend, a different GPU generation, or with chunked prefills enabled; any digest mismatch would delimit the determinism claim.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper reproduces the latent-space safety-probe pipeline of Khatri et al. (2026), in which a lightweight MLP is trained on final-layer, last-token activations of an LLM to classify harmful prompts, and extends it in two directions: (1) applying the identical pipeline to three additional backbones (Gemma-4-E4B, Mistral-7B-v0.3, Qwen2-7B) on WildJailbreak, BeaverTails, and AEGIS 2.0, and (2) repeating extraction under five seeds to measure variance in F1 and in the extracted activation tensors. The reported reproduction is within 0.37 F1 points of the original LLaMA numbers, the cross-architecture F1 values are within about a point of the original model, and the per-seed F1 values are reported as constant, with SHA-256 digests of the serialized activation tensors identical across seeds. The authors position the work as a reproducibility study with multi-seed statistical controls and release code, activations, and probe checkpoints.

Significance. If the numerical record is reliable, this is a useful and appropriately cautious reproducibility contribution: it provides evidence that linear-probe safety detection transfers across several open-weight model families, and it introduces a concrete SHA-256 diagnostic for distinguishing seed-invariant activations from downstream probe stability. The multi-seed protocol and the explicit discussion of what the zero-variance result does and does not claim are good practices. The main caveat is that the central empirical claims currently rest on an inconsistent set of F1 tables, and several claims in the abstract and contributions are broader than the executed protocol. With those issues resolved, the paper would be a solid contribution to the reproducibility literature, though its incremental nature means it should be judged as a workshop-style reproducibility study rather than a large conceptual advance.

major comments (4)
  1. [§5.3/Table 6 and Appendix B/Table 11] The per-seed F1 values in Table 6 and Appendix B are irreconcilable for the same (model, dataset) pairs. For example, LLaMA/BeaverTails appears as 82.90 in Table 6 but 82.49 in Table 11 for seeds 75/456/789/1024; Gemma/BeaverTails appears as 83.35 versus 82.97; Mistral/BeaverTails as 83.32 versus 82.92; Qwen/BeaverTails as 83.39 versus 83.01; and LLaMA/AEGIS appears as 83.42 versus 83.48. The Appendix B note that these runs 'might not use the most efficient probe model checkpoint' does not reconcile the discrepancy, because checkpoint selection is part of the pipeline and the main text asserts a single constant value per pair across seeds. Either Table 6 or Table 11 is not reporting the actual per-seed test F1, so the zero-variance observation and the cross-architecture deltas are not supported by a consistent numerical record. This must be fixed before the headline claims can be accepted.
  2. [§1, Contributions] The contribution bullet stating that the paper quantifies 'what fraction of the original paper's reported margin over baseline guard models survives this noise' is not delivered anywhere in Sections 5 or 6. The manuscript reports probe F1 values and seed variance, but it does not report guard-model F1 values, margin calculations, or any fraction of the margin surviving noise. The authors should either add the missing guard-model comparison or remove this bullet from the contributions.
  3. [§3.3 and §5.4] The non-determinism conclusion is broader than the protocol supports. The protocol varies seed and batch order under a single inference stack (nnsight, batch size 5, no chunked prefill), and Section 6 explicitly acknowledges that these settings may be inadequate to exercise non-deterministic prefill behavior. In addition, Table 7 checks only one dataset (BeaverTails) for each non-LLaMA backbone, plus LLaMA/AEGIS, so the abstract's statement that 'final token latent vectors remained the same for all tested architectures irrespective of the seed values used' should be qualified to 'under the tested extraction configuration and datasets.'
  4. [§1 and reference list] The paper does not disclose that the reproduced work, Khatri et al. (2026), shares its first author with this manuscript. Since one contribution is described as a 'faithful reproduction' used to calibrate the extension, the author overlap should be stated explicitly, and the text should clarify that the LLaMA-3.1-8B calibration is an author-run reproduction rather than an independent replication by a separate group.
minor comments (4)
  1. [Table 5 caption] The caption says 'First row reproduces the original paper's setting,' but the LLaMA-3.1-8B row shows values of 99.1/82.9/83.5 rather than the reproduced values of 99.47/82.90/83.42 from Table 4. This should be labeled as the original reported values, not the reproduction values, to avoid confusion in the cross-architecture deltas.
  2. [Appendix B, Table 11] Appendix B omits seed 42 for all pairs and one Qwen seed without explanation; the authors should provide the complete per-seed log or state why those runs are excluded.
  3. [§5.3] The text says 'standard deviation is 0 throughout,' but the values are reported to two decimal places; the claim should be phrased as 'zero to the reported precision,' since unrounded F1 values could still have small nonzero variance.
  4. [References] The reference 'Arney Agrawal' should be 'Amey Agrawal' for the chunked-prefill paper; please check spelling against the published version.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the new cross-architecture and per-seed variance results are independent measurements, and the self-citation to Khatri et al. (2026) is the reproduction target rather than the evidence.

full rationale

The paper's two load-bearing new claims are empirical and self-contained: (1) probe F1 on three additional backbones (Gemma-4-E4B, Mistral-7B-v0.3, Qwen2-7B) is within roughly a point of the original LLaMA-3.1-8B values, and (2) per-seed F1 and SHA-256 digests of extracted activation tensors are constant across five seeds. Neither claim is derived from the cited Khatri et al. (2026) paper: the reproduced Table 4 numbers are produced by the authors' own implementation, and the original numbers serve only as a comparison baseline, not as inputs to the calculation. The MLP pipeline and hyperparameters are inherited from the original study by design because this is a reproducibility study, but inheritance is not circularity when the fidelity check is an independent computation and the extension axes (new backbones, repeated seeds, hash comparisons) are not fitted to the target outcomes. The same-first-author self-citation is a transparency concern but is not load-bearing: no claim is justified solely by citing that paper, and the new empirical results would stand even if the original paper were ignored. The Appendix B note that those per-seed F1 values 'might not use the most efficient probe model checkpoint' explains the small numerical difference from Table 6; this is a reporting consistency question, not a circular reduction. No equation defines the predicted quantity in terms of the input, and no parameter is fitted to the claimed result. The low score reflects only the minor, non-load-bearing self-citation issue.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities and fits no new constants; it inherits the original probe pipeline, benchmark labels, and extraction configuration as given. The main unstated premises are that final-token activations are the right substrate and that the chosen benchmarks and splits capture generalizable harmfulness.

free parameters (2)
  • Probe training hyperparameters = LR 2.5e-4, wd 1e-2, dropout 0.1, batch 1024, 50 epochs, 6-layer MLP
    Adopted unchanged from Khatri et al. (2026); the transfer claim is established only under this configuration.
  • Activation extraction configuration = Final layer, last prefill token, batch 5, max len 512, base seed 42
    Hand-chosen settings inherited from the original study; they define what 'the latent state' means and are load-bearing for the determinism claim.
assumptions (4)
  • domain assumption Safety-relevant information about prompt harmfulness is linearly decodable from the final-layer hidden state at the last token after prefill.
    Section 3.1, stage 1; underlying hypothesis inherited from Khatri et al. (2026). The reproducibility study does not test earlier-layer or multi-token representations.
  • domain assumption Binary labels in WildJailbreak, BeaverTails, and AEGIS 2.0 are a faithful ground truth for the harmfulness construct.
    Section 4 datasets; all evaluation and F1 comparisons depend on this.
  • domain assumption Fixed data splits match the original paper's protocol and are comparable across backbones.
    Section 4 and the Evaluation section state matching protocol; no statistical test of split comparability is provided.
  • ad hoc to paper The current inference stack (nnsight, batch size 5, no chunked prefill) exercises the nondeterminism sources relevant to real deployment.
    Sections 5.3 and 6; the authors themselves caution that their settings may not recreate nondeterministic prefill behavior, so the zero-variance result is conditional.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do All LLMs Know When They're Being Harmful? A Reproducibility Study of Latent-Space Safety Probes Across Model Families." pith.science (2026). https://pith.science/paper/45Z5HPR5

@misc{pith2026260808029,
  author       = {Pith},
  title        = {Pith review of: Do All LLMs Know When They're Being Harmful? A Reproducibility Study of Latent-Space Safety Probes Across Model Families},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/45Z5HPR5}},
  note         = {Machine review of arXiv:2608.08029}
}
read the original abstract

Khatri et al. (2026) [DOI: 10.1109/DSN-W70714.2026.00027] show that lightweight MLP probes on final-layer activations of a single 8B model (LLaMA-3.1-8B) detect harmful prompts at F1 competitive with guard models 1000x larger, using one probe per benchmark. We reproduce this pipeline end-to-end and extend it along two axes the original study leaves open. First, we test whether the result generalizes across other model architecture and scale by training identical probes on activations from models like Gemma-4-E4B, Mistral-7B-v0.3, and Qwen2-7B, using the three benchmarks (WildJailbreak, BeaverTails, AEGIS 2.0). Second, we test how much of the reported performance is affected by non-determinism during inference by repeating extraction under five random seeds and measuring the variance of F1 scores. Our results reproduce the original LLaMA model benchmarks within 0.37 percentage points of the original F1 scores (and within 0.2 points on BeaverTails). We find that the original MLP probe architecture extends to other model families with F1 scores within a point of the values reported for LLaMA-3.1-8B. Our experiments varying seed values reveal an interesting observation: final token latent vectors remained the same for all tested architectures irrespective of the seed values used.

Figures

Figures reproduced from arXiv: 2608.08029 by the authors.

Figure 1
Figure 1. Reproduced extraction-and-probing pipeline. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 19 canonical work pages

  1. [1]

    Safety Beyond the Interface: Detecting Harm via Latent States in Large Language Models , year=

    Khatri, Alizishaan and Prabhu, Chiquita and Neogi, Omkar , booktitle=. Safety Beyond the Interface: Detecting Harm via Latent States in Large Language Models , year=

    work page

  2. [2]

    Large Language Models Encode Semantics and Alignment in Linearly Separable Representations , author=. Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics , pages=

    work page

  3. [3]

    Representation Engineering: A Top-Down Approach to

    Zou, Andy and Phan, Long and Chen, Sarah and Campbell, James and Guo, Phillip and Ren, Richard and Pan, Alexander and Yin, Xuwang and Mazeika, Mantas and Dombrowski, Ann-Kathrin and others , journal=. Representation Engineering: A Top-Down Approach to

  4. [4]

    The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

    Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=

  5. [5]

    Llama Guard:

    Inan, Hakan and Upasani, Kartikeya and Chi, Jianfeng and Rungta, Rashi and Iyer, Krithika and Mao, Yuning and Tontchev, Michael and Hu, Qing and Fuller, Brian and Testuggine, Davide and Khabsa, Madian , journal=. Llama Guard:

    work page

  6. [6]

    ShieldGemma: Generative

    Zeng, Wenjun and Liu, Yuchi and Mullins, Ryan and Peran, Ludovic and Fernandez, Joe and Harkous, Hamza and Narasimhan, Karthik and Proud, Drew and Kumar, Piyush and Radharapu, Bhaktipriya and others , journal=. ShieldGemma: Generative

    work page

  7. [7]

    WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of

    Han, Seungju and Rao, Kavel and Ettinger, Allyson and Jiang, Liwei and Lin, Bill Yuchen and Lambert, Nathan and Choi, Yejin and Dziri, Nouha , booktitle=. WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of

    work page

  8. [8]

    BeaverTails: Towards Improved Safety Alignment of

    Ji, Jiaming and Liu, Mickel and Dai, Juntao and Pan, Xuehai and Zhang, Chi and Bian, Ce and Chen, Boyuan and Sun, Ruiyang and Wang, Yizhou and Yang, Yaodong , booktitle=. BeaverTails: Towards Improved Safety Alignment of

    work page

Show all 31 references
  1. [9]

    Advances in Neural Information Processing Systems , volume=

    WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models , author=. Advances in Neural Information Processing Systems , volume=

  2. [10]

    Ghosh, Shaona and Varshney, Prasoon and Sreedhar, Makesh Narsimhan and Padmakumar, Aishwarya and Rebedea, Traian and Varghese, Jibin Rajan and Parisien, Christopher , booktitle=

  3. [11]

    arXiv preprint arXiv:2601.04603 , year=

    Constitutional Classifiers++: Efficient Production-Grade Defenses Against Universal Jailbreaks , author=. arXiv preprint arXiv:2601.04603 , year=

  4. [12]

    Advances in Neural Information Processing Systems , volume=

    Training Language Models to Follow Instructions with Human Feedback , author=. Advances in Neural Information Processing Systems , volume=

  5. [13]

    International Conference on Learning Representations , year=

    Decoupled Weight Decay Regularization , author=. International Conference on Learning Representations , year=

  6. [14]

    LLMScan: Causal Scan for

    Zhang, Mengdi and Goh, Kai Kiat and Zhang, Peixin and Sun, Jun and Xin, Rose Lin and Zhang, Hongyu , journal=. LLMScan: Causal Scan for

  7. [15]

    Lightweight Safety Guardrails Using Fine-Tuned

    Zheng, Aaron and Rana, Mansi and Stolcke, Andreas , journal=. Lightweight Safety Guardrails Using Fine-Tuned

  8. [16]

    Jailbroken: How Does

    Wei, Alexander and Haghtalab, Nika and Steinhardt, Jacob , booktitle=. Jailbroken: How Does

  9. [17]

    arXiv preprint arXiv:2307.15043 , year=

    Universal and Transferable Adversarial Attacks on Aligned Language Models , author=. arXiv preprint arXiv:2307.15043 , year=

    arXiv

  10. [18]

    arXiv preprint arXiv:2412.14093 , year=

    Alignment Faking in Large Language Models , author=. arXiv preprint arXiv:2412.14093 , year=

    arXiv

  11. [19]

    arXiv preprint arXiv:2402.01822 , year=

    Building Guardrails for Large Language Models , author=. arXiv preprint arXiv:2402.01822 , year=

    arXiv

  12. [20]

    Attention Tracker: Detecting Prompt Injection Attacks in

    Hung, Kuo-Han and Ko, Ching-Yun and Rawat, Ambrish and Chung, I-Hsin and Hsu, Winston H and Chen, Pin-Yu , booktitle=. Attention Tracker: Detecting Prompt Injection Attacks in

  13. [21]

    Probing Latent Subspaces in

    Chia, Xin Wei and Wong, Swee Liang and Pan, Jonathan , journal=. Probing Latent Subspaces in

  14. [22]

    DeepContext: Stateful Real-Time Detection of Multi-Turn Adversarial Intent Drift in

    Albrethsen, Justin and Datta, Yash and Kumar, Kunal and Rajasekar, Sharath , journal=. DeepContext: Stateful Real-Time Detection of Multi-Turn Adversarial Intent Drift in

  15. [23]

    Dubey, Abhimanyu and Jauhri, Abhinav and Pandey, Abhinav and Kadian, Abhishek and Al-Dahle, Ahmad and Letman, Aiesha and Mathur, Akhil and Schelten, Alan and Yang, Amy and Fan, Angela and others , journal=. The

  16. [24]

    arXiv preprint arXiv:2407.10671 , year=

    Qwen2 Technical Report , author=. arXiv preprint arXiv:2407.10671 , year=

    arXiv

  17. [25]

    arXiv preprint arXiv:2408.00118 , year=

    Gemma 2: Improving Open Language Models at a Practical Size , author=. arXiv preprint arXiv:2408.00118 , year=

    arXiv

  18. [26]

    Zhao, Jiachen and Huang, Jing and Wu, Zhengxuan and Bau, David and Shi, Weiyan , journal=

  19. [27]

    Harmful Intent as a Geometrically Recoverable Feature of

    Isaac Llorente-Saguer , journal=. Harmful Intent as a Geometrically Recoverable Feature of

  20. [28]

    arXiv preprint arXiv:2605.12726 , year=

    Before the Last Token: Diagnosing Final-Token Safety Probe Failures , author=. arXiv preprint arXiv:2605.12726 , year=

    arXiv

  21. [29]

    Jaden Fiotto-Kaufman and Alexander R. Loftus and Eric Todd and Jannik Brinkmann and Koyena Pal and Dmitrii Troitskii and Michael Ripa and Adam Belfki and Can Rager and Caden Juang and Aaron Mueller and Samuel Marks and Arnab Sen Sharma and Francesca Lucchetti and Nikhil Prakas...

  22. [30]

    ACM SIGOPS Operating Systems Review , volume=

    Efficient llm inference via chunked prefills , author=. ACM SIGOPS Operating Systems Review , volume=. 2025 , publisher=

    2025

  23. [31]

    Advances in Neural Information Processing Systems , volume=

    Understanding and mitigating numerical sources of nondeterminism in llm inference , author=. Advances in Neural Information Processing Systems , volume=

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.