Pith. sign in

REVIEW 3 major objections 5 minor 26 references

Activation Probes Surface Code-Security Signals that the Model's Output Misses

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Pith's one-line read A linear probe on five open-weight code-reviewer models' activations ranks the vulnerable function above its fix on 61–67% of unseen real-world CVEs, beating the same models' prompted yes/no answers under every prompt tried.

desk verdict Solid OOD transfer result for activation probes, but the security-specificity claim needs control probes on non-security edits before I'd trust the triage use case. read the letter →

arxiv 2608.09643 v1 pith:VHSAKT2J submitted 2026-08-10 cs.CR cs.LG

classification cs.CRcs.LG
keywords activationprobeslinearresidualstreamcodesecurityvulnerabilitydetectionlargelanguagemodelsout-of-distributiontransferCWE
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the hidden internal states of an open-weight code-reviewer language model contain a security signal that the same model's prompted text output does not express. If true, a team that deploys an opaque AI coding agent could self-host a lightweight reviewer probe, read that reviewer's activations, and obtain a ranking of which generated functions deserve human security review, for weakness types the probe never saw in training. The evidence is a linear probe per model, trained on paired vulnerable-and-fixed Python functions and evaluated without retraining on 147 unseen single-function real-world CVEs, where it scores the vulnerable function above its fix on 61–67% of cases across five open-weight models. The same models' prompted yes/no answers, read from their logits, trail that win-rate under every prompt tried, and their written verdicts tie on most pairs and so cannot rank at all.

What carries the argument

The carrying mechanism is a logistic linear probe applied to the residual stream at the input of a chosen transformer layer's input layer-norm, producing a continuous vulnerability score for each function. Per model, the layer, token-pooling strategy (mean, max, last, or sliding-window-max), and L2 strength are selected by 5-fold GroupKFold AUC, with folds grouped by project to prevent code-duplication leakage. The probe's verdict is the difference in score between the vulnerable and fixed function in each CVE pair, summarized as a paired win-rate; the comparator is the same model's prompted YES/NO probability read from the answer-token logits. The central comparison is between a continuous read of the hidden state and a continuous read of the output distribution, with the written token verdict as the collapse case.

What would settle it

A decisive control would be to train the same probe on paired vulnerable-fixed functions where the fix is a non-security edit, such as a variable rename, comment addition, or unrelated library call, and measure the win-rate; if it stays near 61–67%, the probe is keying on surface differences rather than the security weakness.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a single linear probe fitted to the residual-stream activations of an open-weight reviewer model recovers a code-security signal that the same model's prompted output misses. Trained on four Python CWE types and tested zero-shot on 234 disclosed vulnerabilities whose weakness labels do not overlap training, the probe ranks the vulnerable function above its post-fix counterpart on 61–67% of the 147 single-function cases, with Wilson intervals above chance and sign-test p < 0.05 for all five models. Read from logits, the same models' prompted YES/NO answers do worse under every prompt tested, and the written verdict ties on 72–97% of pairs. The paper concludes that activations carry a code-security signal prompting misses, while cautioning that on unseen weakness types the paired win-rate is a usable ranking signal, not a calibrated detector.

Load-bearing premise

The load-bearing premise is that the only systematic difference between the paired vulnerable and fixed functions is the security weakness, so the probe's win-rate reflects security content rather than correlated surface properties such as code length, comment density, or API-token presence; no control on non-security edits is reported.

Editorial extensions

If this is right

  • An open-weight reviewer's activations can rank vulnerable over fixed code on weakness types absent from training, across five models and four architecture families.
  • Prompting the same model for a binary verdict is not just fragile; the written verdict ties on most pairs, so it is unusable as a ranking signal, while the probe always produces a rankable score.
  • The probe's advantage over no-shot and few-shot prompting is broad and often significant; its edge over chain-of-thought logit readout is consistent but narrow, so the immediate practical win is over cheap prompting.
  • Wiring the probe into a post-write hook of a coding agent could provide a self-hosted triage flag for human security review, though it would need threshold calibration to become a detector rather than a ranker.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the signal were shown to be security-specific, the probe would offer a distillation target: a tiny classifier that captures what the large model knows but does not say, usable on models whose internals cannot be inspected directly by deploying teams.
  • Because the same method transfers across four architecture families, the security signal likely lives in a low-dimensional direction of the residual stream that is shared across code LLMs, which would make cross-model probes feasible.
  • A natural test the paper leaves open is whether the probe detects stylistic or spurious differences rather than the weakness itself; running it on paired non-security commits would settle whether the 61–67% win-rate is a floor or an artifact.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper fits a linear probe on the residual-stream activations of five open-weight code LLMs, training on paired vulnerable/fixed Python functions from SVEN (four CWEs) and testing zero-shot on PatchEval CVEs whose CWE labels do not overlap the training set. The probe scores the vulnerable function above its paired fix on 61–67% of 147 single-function unseen CVEs, with Wilson intervals above 50% and sign-test significance for all five models. The same models' prompted yes/no verdicts, read from logits, achieve lower win-rates, and when read as written verdicts they tie on 72–97% of pairs. The paper concludes that model activations carry a code-security signal that prompting the same model misses.

Significance. If the result holds, the paper provides an inexpensive, self-hostable way to surface potentially vulnerable code changes from open-weight reviewer models: a linear probe over activations outperforms the same model's prompted output on previously unseen weakness types. The design is unusually careful in several respects: the OOD evaluation excludes CWE-label overlap, the train/eval split and cross-validation folds are grouped by project, the comparator is the strongest prompting configuration per model (conservative for the probe), and paired sign tests with Wilson intervals are used. The empirical pattern is consistent across four architecture families. The main weakness is that the attribution to 'code-security' as opposed to a generic before/after code-edit signal is not tested, and the prompted-comparison sample consistency is not fully clear.

major comments (3)
  1. [Section 3 and Section 4 (Limitations)] The paper's central claim is that activations carry a 'code-security signal'. However, the paired design only shows that the probe ranks the vulnerable function above its fix on 61–67% of unseen cases. The probe is a linear classifier on residual-stream features and could exploit any systematic surface difference between the two sides of each pair, such as function length, comment density, or specific API tokens that differ between vulnerable and fixed versions. Because the same kind of before/after difference exists in the SVEN training pairs and in the PatchEval pairs, the observed transfer may reflect a learned 'code-difference' direction rather than a security-specific direction. No control probes on paired non-security edits (e.g., refactors, feature additions, formatting changes) are run. This is load-bearing for the conclusion and for the proposed triage use case: if the probe also ranks the 'before' side above the 'after' side on benign paired edits at above 50%, it would flag routine code changes as security-relevant. I request a control experiment on non-security paired edits, and a correspondingly revised claim if the control win-rate is also above chance.
  2. [Section 2 (Data) and Section 3 (Results)] The authors carefully exclude PatchEval CVEs whose CWE labels overlap the four SVEN training CWEs, and they group the SVEN train/eval split by project. However, they do not report whether the SVEN training repositories overlap with the PatchEval test repositories. Since SVEN injects vulnerabilities into real-world projects and PatchEval contains real-world CVEs, shared repositories are plausible. If a project appears in both, the probe could exploit project-specific features (e.g., code style, file structure, naming conventions) rather than weakness-type-generalizable signals, inflating the reported 'unseen' win-rates. This is a load-bearing check for the out-of-distribution generalization claim. Please measure the project-level overlap between SVEN-Python training and PatchEval, and if any overlap exists, rerun the headline analysis on a version with those projects removed or report the overlap and its impact.
  3. [Section 2 (Prompted comparator) and Appendix A.3] There is an inconsistency about the sample used for the paired McNemar tests. The main text says 'We compare the probe against the strongest prompting per model with a paired McNemar test on the same CVEs.' Appendix A.3, however, states that DeepSeek-Coder 33B has a no-verdict rate of 16% (no-shot), 9% (few-shot), and 2% (CoT), so its prompted win-rates are computed on a smaller compliant set. If Table 3's prompting percentages for DeepSeek are on a subset while the probe's 60.8% is on the full 147, the comparison is not paired, and the McNemar p-values would be invalid. Please clarify the exact n for each model and prompt, restrict the McNemar tests to the common subset of CVEs with both a probe score and a prompting verdict, and report those n's in Table 3 or its caption.
minor comments (5)
  1. [Section 3, Table 3] The caption and column header 'Prompting (%)' do not indicate which readout (logit or token) is used for the reported percentages. Please state explicitly that these are the logit readout (or whichever readout is used), and clarify whether the 'best prompting' selection is over prompt mode only or over the prompt×readout grid.
  2. [Appendix A.1, Table 4] The abbreviation 'swim' appears in the table caption but is not defined there; define it in the caption (sliding-window-max with window 16) for self-containment.
  3. [Abstract and Section 3] The abstract says the probe 'beats' the prompted logit win-rate 'under every prompt we try.' Table 3 shows that the point estimate is higher for every model and every prompt, but the advantage over chain-of-thought is not always statistically significant (only one of five models). Please add a qualifier such as 'in point estimate' or 'not always significant' to avoid overstating the comparison.
  4. [Section 4 (Limitations)] The phrase 'limited set data' should be 'limited dataset' or 'limited set of data'; also, the limitations paragraph could more explicitly name the confound of non-security code edits, which is currently only implied by 'usable ranking signal.'
  5. [Section 2 (Probe fitting)] The sentence 'hooking the input of layers[L].input_layernorm' is ambiguous about whether the probe reads the input to the layer norm or the normalized output. Please specify the exact hook point.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the probe is trained on SVEN and evaluated zero-shot on PATCHEVAL CVEs with non-overlapping CWE labels, so the reported win-rate is a genuine out-of-distribution measurement.

full rationale

The paper's central claim is an external-benchmark generalization result, not a derivation that assumes its conclusion. The probe is fit on SVEN paired vulnerable/fixed Python functions (Section 2: 'For training we use the four SVEN ... CWEs'), with hyperparameters selected by 5-fold GroupKFold AUC on the training set only. Evaluation is zero-shot on PATCHEVAL CVEs whose CWE labels do not overlap training: 'no CWE in the CVE's full label set overlaps the four SVEN-training CWEs.' The paired win-rate is a measurement on held-out data, not a quantity determined by the fitting procedure. The prompted comparator is an independent baseline; picking the comparator's best configuration on the test set ('picking the comparator’s best configuration only makes the test harder for the probe') only makes the comparison conservative. No load-bearing step reduces to the paper's own inputs by construction, and there are no self-citations in the load-bearing argument. The main weakness is construct validity—no control probes on paired non-security edits—which is a correctness risk, not circularity. The paper is self-contained against an external benchmark, so the appropriate circularity score is 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities or theoretical constructs. Its empirical claim rests on standard ML assumptions about label quality, leakage control, and the meaning of activation representations, plus per-model hyperparameters fitted on the training set.

free parameters (1)
  • Per-model probe configuration (layer, pooling, L2 C) = Qwen7B:L11,max; Qwen14B:L21,swim; DS33B:L7,max; Devstral:L21,max; Llama8B:L17,max; C=1e-2 all
    Selected by sweeping layer, pooling, and L2 strength on SVEN to maximize GroupKFold AUC. These choices affect the probe's OOD win-rate, but the headline result holds across all five models regardless of exact config.
assumptions (4)
  • domain assumption CWE labels on SVEN and PatchEval are accurate and complete enough for the 'unseen' split.
    The unseen-bug-types evaluation excludes CVEs whose CWE label set overlaps the four SVEN CWEs. Noisy or incomplete CWE labels could violate the 'unseen' claim and inflate or deflate the transfer result. Section 2, Data.
  • domain assumption A linear probe on residual-stream activations at input_layernorm captures task-relevant security representations.
    The method relies on the standard interpretability assumption that these activations linearly encode useful information. Section 2 and Section 4.
  • domain assumption Project-grouped splits prevent data leakage between SVEN training and PatchEval evaluation.
    The generalization claim depends on no code overlap between train and eval; the paper asserts grouping by project but provides no similarity check. Section 2.
  • domain assumption Paired vulnerable/fixed functions differ mainly in the security weakness, not in confounds like length or API usage.
    The win-rate design scores the vulnerable function against its own fix; if surface features drive the score, the 'security signal' interpretation weakens. Section 3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Activation Probes Surface Code-Security Signals that the Model's Output Misses." pith.science (2026). https://pith.science/paper/VHSAKT2J

@misc{pith2026260809643,
  author       = {Pith},
  title        = {Pith review of: Activation Probes Surface Code-Security Signals that the Model's Output Misses},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VHSAKT2J}},
  note         = {Machine review of arXiv:2608.09643}
}
read the original abstract

AI coding agents now write a growing share of production code, and human security review does not scale at the rate code is generated. The agents in widest use are closed-weight, so a deploying team cannot read their internals. It can instead run an open-weight model as a reviewer over the agent's output. That reviewer's activations are readable. We ask whether reading those activations recovers a security signal that simply asking the same reviewer misses. We fit a single linear probe per model on a corpus of paired vulnerable-and-fixed Python functions, then test it without retraining on real disclosed vulnerabilities whose weakness type the probe never saw in training, across five open-weight reviewer models. On the vulnerabilities fixed by changing a single function, the probe scores the vulnerable function above its fix on 61-67% of cases for every model, beating the 50% chance line. It also beats the same model's prompted YES/NO win-rate read from its logits, under every prompt we try. Asking the model for a written verdict, even with chain-of-thought, returns the same answer on the vulnerable and fixed function most of the time and so cannot tell them apart. Model activations carry a code-security signal that prompting the same model misses.

Figures

Figures reproduced from arXiv: 2608.09643 by the authors.

Figure 1
Figure 1. Paired win-rate on 147 unseen real-world vulnerabilities: probe versus the same model’s best prompting, per reviewer model. A star marks a significant probe advantage over that model’s best prompt-and-readout (paired McNemar, p < 0.05); [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 7 canonical work pages

  1. [1]

    and Bengio, Y

    Alain, G. and Bengio, Y. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016. URL https://arxiv.org/abs/1610.01644. ICLR 2017 Workshop

  2. [2]

    The adverse effects of code duplication in machine learning models of code

    Allamanis, M. The adverse effects of code duplication in machine learning models of code. In Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) , pp.\ 143--153. ACM, 2019. doi:10.1145/3359591.3359735. URL https://arxiv.org/abs/1812.06469

  3. [3]

    and Mitchell, T

    Azaria, A. and Mitchell, T. The internal state of an LLM knows when it's lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp.\ 967--976, 2023. URL https://aclanthology.org/2023.findings-emnlp.68/

  4. [4]

    Ding, Y., Fu, Y., Ibrahim, O., Sitawarin, C., Chen, X., Alomair, B., Wagner, D., Ray, B., and Chen, Y. Vulnerability detection with code language models: How far are we? In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering ( ICSE ) , pp.\ 1729--1741, 2025. doi:10.1109/ICSE55347.2025.00038. URL https://arxiv.org/abs/2403.18624

  5. [5]

    A., Horv \'a t, K

    Dubniczky, R. A., Horv \'a t, K. Z., Bisztray, T., Ferrag, M. A., Cordeiro, L. C., and Tihanyi, N. CASTLE : Benchmarking dataset for static code analyzers and LLMs towards CWE detection. arXiv preprint arXiv:2503.09433, 2025. URL https://arxiv.org/abs/2503.09433

  6. [6]

    and Clark, J

    Favaro, M. and Clark, J. When AI builds itself. Anthropic Institute report, June 2026. URL https://www.anthropic.com/institute/recursive-self-improvement. Published 2026-06-04. As of May 2026, more than 80\

  7. [7]

    R., Todd, E., Brinkmann, J., Juang, C., Pal, K., Rager, C., Mueller, A., Marks, S., Sharma, A

    Fiotto-Kaufman, J., Loftus, A. R., Todd, E., Brinkmann, J., Juang, C., Pal, K., Rager, C., Mueller, A., Marks, S., Sharma, A. S., Lucchetti, F., Prakash, N., Brodley, C., Guha, A., Bell, J., Wallace, B. C., and Bau, D. NNsight and NDIF : Democratizing access to open-weight foundation model internals. arXiv preprint arXiv:2407.14561, 2024. URL https://arxi...

  8. [8]

    Security weaknesses of Copilot -generated code in GitHub projects: An empirical study

    Fu, Y., Liang, P., Tahir, A., et al. Security weaknesses of Copilot -generated code in GitHub projects: An empirical study. ACM Transactions on Software Engineering and Methodology, 34, 2025. doi:10.1145/3716848. URL https://arxiv.org/abs/2310.02059

Show all 26 references
  1. [9]

    Grattafiori, A. et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. URL https://arxiv.org/abs/2407.21783

  2. [10]

    K., Luo, F., Xiong, Y., and Liang, W

    Guo, D., Zhu, Q., Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y. K., Luo, F., Xiong, Y., and Liang, W. DeepSeek-Coder : When the large language model meets programming -- the rise of code intelligence. arXiv preprint arXiv:2401.14196, 2024. URL https:...

  3. [11]

    and Vechev, M

    He, J. and Vechev, M. Large language models for code: S ecurity hardening and adversarial testing. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS), 2023. URL https://arxiv.org/abs/2302.05319

  4. [12]

    Qwen2.5-Coder technical report

    Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Lu, K., et al. Qwen2.5-Coder technical report. arXiv preprint arXiv:2409.12186, 2024. URL https://arxiv.org/abs/2409.12186

  5. [13]

    M., and Anciaux, N

    Ibanez-Lissen, L., Gonzalez-Manzano, L., de Fuentes, J. M., and Anciaux, N. LPASS : Linear probes as stepping stones for vulnerability detection using compressed LLMs . arXiv preprint arXiv:2505.24451, 2025. URL https://arxiv.org/abs/2505.24451

  6. [14]

    H., Gonzalez, J

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with PagedAttention . In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP), 2023. URL h...

  7. [15]

    and Tegmark, M

    Marks, S. and Tegmark, M. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. In Conference on Language Modeling (COLM), 2024. URL https://arxiv.org/abs/2310.06824

  8. [16]

    S., and Krasheninnikov, D

    McKenzie, A., Pawar, U., Blandfort, P., Bankes, W., Krueger, D., Lubana, E. S., and Krasheninnikov, D. Detecting high-stakes interactions with activation probes. arXiv preprint arXiv:2506.10805, 2025. URL https://arxiv.org/abs/2506.10805. NeurIPS 2025

  9. [17]

    Devstral-Small-2505 , May 2025

    Mistral AI and All Hands AI . Devstral-Small-2505 , May 2025. URL https://huggingface.co/mistralai/Devstral-Small-2505. Model release, 2025-05-21

  10. [18]

    LLMs know more than they show: On the intrinsic representation of LLM hallucinations

    Orgad, H., Toker, M., Gekhman, Z., Reichart, R., Szpektor, I., Kotek, H., and Belinkov, Y. LLMs know more than they show: On the intrinsic representation of LLM hallucinations. In International Conference on Learning Representations (ICLR), 2025. URL https://arxiv.org/abs/2410.02707

  11. [19]

    PyTorch : An imperative style, high-performance deep learning library

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. PyTorch : An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 (NeurIPS), 2019. URL h...

  12. [20]

    Scikit-learn: Machine learning in P ython

    Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, \'E . Scikit-learn: Machine learning in P ython. Journal ...

  13. [21]

    Y C ombinator startups are fastest growing, most profitable in fund history because of AI

    Rooney, K. Y C ombinator startups are fastest growing, most profitable in fund history because of AI . CNBC, March 2025. URL https://www.cnbc.com/2025/03/15/y-combinator-startups-are-fastest-growing-in-fund-history-because-of-ai.html. Reports Y Combinator CEO Garry Tan: for 25\

  14. [22]

    LLMs cannot reliably identify and reason about security vulnerabilities (yet?): A comprehensive evaluation, framework, and benchmarks

    Ullah, S., Han, M., Pujar, S., Pearce, H., Coskun, A., and Stringhini, G. LLMs cannot reliably identify and reason about security vulnerabilities (yet?): A comprehensive evaluation, framework, and benchmarks. In 2024 IEEE Symposium on Security and Privacy ( SP ) . IEEE, 2024. ...

  15. [23]

    E., Haberland, M., Reddy, T., Cournapeau, D., et al

    Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., et al. SciPy 1.0: Fundamental algorithms for scientific computing in P ython. Nature Methods, 17 0 (3): 0 261--272, 2020. doi:10.1038/s41592-019-0686-2. URL https://doi.org/10.1038/s41592-019-0686-2

  16. [24]

    AI code in the wild: Measuring security risks and ecosystem shifts of AI -generated code in modern software

    Wang, B., Yu, W., Zhong, Y., Yu, H., Lian, K., Lu, C., Zheng, H., Zhang, D., and Li, H. AI code in the wild: Measuring security risks and ecosystem shifts of AI -generated code in modern software. arXiv preprint arXiv:2512.18567, 2025. URL https://arxiv.org/abs/2512.18567

  17. [25]

    PATCHEVAL : A new benchmark for evaluating LLM s on patching real-world vulnerabilities

    Wei, Z., Zeng, J., Wen, M., Yu, Z., Cheng, K., Zhu, Y., Guo, J., Zhou, S., Yin, L., Su, X., and Ma, Z. PATCHEVAL : A new benchmark for evaluating LLM s on patching real-world vulnerabilities. arXiv preprint arXiv:2511.11019, 2025. URL https://arxiv.org/abs/2511.11019

  18. [26]

    Transformers: State-of-the-art natural language processing

    Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Sys...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.