REVIEW 3 major objections 5 minor 26 references
Activation Probes Surface Code-Security Signals that the Model's Output Misses
T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read A linear probe on five open-weight code-reviewer models' activations ranks the vulnerable function above its fix on 61–67% of unseen real-world CVEs, beating the same models' prompted yes/no answers under every prompt tried.
desk verdict Solid OOD transfer result for activation probes, but the security-specificity claim needs control probes on non-security edits before I'd trust the triage use case. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is a logistic linear probe applied to the residual stream at the input of a chosen transformer layer's input layer-norm, producing a continuous vulnerability score for each function. Per model, the layer, token-pooling strategy (mean, max, last, or sliding-window-max), and L2 strength are selected by 5-fold GroupKFold AUC, with folds grouped by project to prevent code-duplication leakage. The probe's verdict is the difference in score between the vulnerable and fixed function in each CVE pair, summarized as a paired win-rate; the comparator is the same model's prompted YES/NO probability read from the answer-token logits. The central comparison is between a continuous read of the hidden state and a continuous read of the output distribution, with the written token verdict as the collapse case.
What would settle it
A decisive control would be to train the same probe on paired vulnerable-fixed functions where the fix is a non-security edit, such as a variable rename, comment addition, or unrelated library call, and measure the win-rate; if it stays near 61–67%, the probe is keying on surface differences rather than the security weakness.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that a single linear probe fitted to the residual-stream activations of an open-weight reviewer model recovers a code-security signal that the same model's prompted output misses. Trained on four Python CWE types and tested zero-shot on 234 disclosed vulnerabilities whose weakness labels do not overlap training, the probe ranks the vulnerable function above its post-fix counterpart on 61–67% of the 147 single-function cases, with Wilson intervals above chance and sign-test p < 0.05 for all five models. Read from logits, the same models' prompted YES/NO answers do worse under every prompt tested, and the written verdict ties on 72–97% of pairs. The paper concludes that activations carry a code-security signal prompting misses, while cautioning that on unseen weakness types the paired win-rate is a usable ranking signal, not a calibrated detector.
Load-bearing premise
The load-bearing premise is that the only systematic difference between the paired vulnerable and fixed functions is the security weakness, so the probe's win-rate reflects security content rather than correlated surface properties such as code length, comment density, or API-token presence; no control on non-security edits is reported.
Editorial extensions
If this is right
- An open-weight reviewer's activations can rank vulnerable over fixed code on weakness types absent from training, across five models and four architecture families.
- Prompting the same model for a binary verdict is not just fragile; the written verdict ties on most pairs, so it is unusable as a ranking signal, while the probe always produces a rankable score.
- The probe's advantage over no-shot and few-shot prompting is broad and often significant; its edge over chain-of-thought logit readout is consistent but narrow, so the immediate practical win is over cheap prompting.
- Wiring the probe into a post-write hook of a coding agent could provide a self-hosted triage flag for human security review, though it would need threshold calibration to become a detector rather than a ranker.
Reading between the lines
- If the signal were shown to be security-specific, the probe would offer a distillation target: a tiny classifier that captures what the large model knows but does not say, usable on models whose internals cannot be inspected directly by deploying teams.
- Because the same method transfers across four architecture families, the security signal likely lives in a low-dimensional direction of the residual stream that is shared across code LLMs, which would make cross-model probes feasible.
- A natural test the paper leaves open is whether the probe detects stylistic or spurious differences rather than the weakness itself; running it on paired non-security commits would settle whether the 61–67% win-rate is a floor or an artifact.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper fits a linear probe on the residual-stream activations of five open-weight code LLMs, training on paired vulnerable/fixed Python functions from SVEN (four CWEs) and testing zero-shot on PatchEval CVEs whose CWE labels do not overlap the training set. The probe scores the vulnerable function above its paired fix on 61–67% of 147 single-function unseen CVEs, with Wilson intervals above 50% and sign-test significance for all five models. The same models' prompted yes/no verdicts, read from logits, achieve lower win-rates, and when read as written verdicts they tie on 72–97% of pairs. The paper concludes that model activations carry a code-security signal that prompting the same model misses.
Significance. If the result holds, the paper provides an inexpensive, self-hostable way to surface potentially vulnerable code changes from open-weight reviewer models: a linear probe over activations outperforms the same model's prompted output on previously unseen weakness types. The design is unusually careful in several respects: the OOD evaluation excludes CWE-label overlap, the train/eval split and cross-validation folds are grouped by project, the comparator is the strongest prompting configuration per model (conservative for the probe), and paired sign tests with Wilson intervals are used. The empirical pattern is consistent across four architecture families. The main weakness is that the attribution to 'code-security' as opposed to a generic before/after code-edit signal is not tested, and the prompted-comparison sample consistency is not fully clear.
major comments (3)
- [Section 3 and Section 4 (Limitations)] The paper's central claim is that activations carry a 'code-security signal'. However, the paired design only shows that the probe ranks the vulnerable function above its fix on 61–67% of unseen cases. The probe is a linear classifier on residual-stream features and could exploit any systematic surface difference between the two sides of each pair, such as function length, comment density, or specific API tokens that differ between vulnerable and fixed versions. Because the same kind of before/after difference exists in the SVEN training pairs and in the PatchEval pairs, the observed transfer may reflect a learned 'code-difference' direction rather than a security-specific direction. No control probes on paired non-security edits (e.g., refactors, feature additions, formatting changes) are run. This is load-bearing for the conclusion and for the proposed triage use case: if the probe also ranks the 'before' side above the 'after' side on benign paired edits at above 50%, it would flag routine code changes as security-relevant. I request a control experiment on non-security paired edits, and a correspondingly revised claim if the control win-rate is also above chance.
- [Section 2 (Data) and Section 3 (Results)] The authors carefully exclude PatchEval CVEs whose CWE labels overlap the four SVEN training CWEs, and they group the SVEN train/eval split by project. However, they do not report whether the SVEN training repositories overlap with the PatchEval test repositories. Since SVEN injects vulnerabilities into real-world projects and PatchEval contains real-world CVEs, shared repositories are plausible. If a project appears in both, the probe could exploit project-specific features (e.g., code style, file structure, naming conventions) rather than weakness-type-generalizable signals, inflating the reported 'unseen' win-rates. This is a load-bearing check for the out-of-distribution generalization claim. Please measure the project-level overlap between SVEN-Python training and PatchEval, and if any overlap exists, rerun the headline analysis on a version with those projects removed or report the overlap and its impact.
- [Section 2 (Prompted comparator) and Appendix A.3] There is an inconsistency about the sample used for the paired McNemar tests. The main text says 'We compare the probe against the strongest prompting per model with a paired McNemar test on the same CVEs.' Appendix A.3, however, states that DeepSeek-Coder 33B has a no-verdict rate of 16% (no-shot), 9% (few-shot), and 2% (CoT), so its prompted win-rates are computed on a smaller compliant set. If Table 3's prompting percentages for DeepSeek are on a subset while the probe's 60.8% is on the full 147, the comparison is not paired, and the McNemar p-values would be invalid. Please clarify the exact n for each model and prompt, restrict the McNemar tests to the common subset of CVEs with both a probe score and a prompting verdict, and report those n's in Table 3 or its caption.
minor comments (5)
- [Section 3, Table 3] The caption and column header 'Prompting (%)' do not indicate which readout (logit or token) is used for the reported percentages. Please state explicitly that these are the logit readout (or whichever readout is used), and clarify whether the 'best prompting' selection is over prompt mode only or over the prompt×readout grid.
- [Appendix A.1, Table 4] The abbreviation 'swim' appears in the table caption but is not defined there; define it in the caption (sliding-window-max with window 16) for self-containment.
- [Abstract and Section 3] The abstract says the probe 'beats' the prompted logit win-rate 'under every prompt we try.' Table 3 shows that the point estimate is higher for every model and every prompt, but the advantage over chain-of-thought is not always statistically significant (only one of five models). Please add a qualifier such as 'in point estimate' or 'not always significant' to avoid overstating the comparison.
- [Section 4 (Limitations)] The phrase 'limited set data' should be 'limited dataset' or 'limited set of data'; also, the limitations paragraph could more explicitly name the confound of non-security code edits, which is currently only implied by 'usable ranking signal.'
- [Section 2 (Probe fitting)] The sentence 'hooking the input of layers[L].input_layernorm' is ambiguous about whether the probe reads the input to the layer norm or the normalized output. Please specify the exact hook point.
Circularity Check
No significant circularity: the probe is trained on SVEN and evaluated zero-shot on PATCHEVAL CVEs with non-overlapping CWE labels, so the reported win-rate is a genuine out-of-distribution measurement.
full rationale
The paper's central claim is an external-benchmark generalization result, not a derivation that assumes its conclusion. The probe is fit on SVEN paired vulnerable/fixed Python functions (Section 2: 'For training we use the four SVEN ... CWEs'), with hyperparameters selected by 5-fold GroupKFold AUC on the training set only. Evaluation is zero-shot on PATCHEVAL CVEs whose CWE labels do not overlap training: 'no CWE in the CVE's full label set overlaps the four SVEN-training CWEs.' The paired win-rate is a measurement on held-out data, not a quantity determined by the fitting procedure. The prompted comparator is an independent baseline; picking the comparator's best configuration on the test set ('picking the comparator’s best configuration only makes the test harder for the probe') only makes the comparison conservative. No load-bearing step reduces to the paper's own inputs by construction, and there are no self-citations in the load-bearing argument. The main weakness is construct validity—no control probes on paired non-security edits—which is a correctness risk, not circularity. The paper is self-contained against an external benchmark, so the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (1)
- Per-model probe configuration (layer, pooling, L2 C) =
Qwen7B:L11,max; Qwen14B:L21,swim; DS33B:L7,max; Devstral:L21,max; Llama8B:L17,max; C=1e-2 all
assumptions (4)
- domain assumption CWE labels on SVEN and PatchEval are accurate and complete enough for the 'unseen' split.
- domain assumption A linear probe on residual-stream activations at input_layernorm captures task-relevant security representations.
- domain assumption Project-grouped splits prevent data leakage between SVEN training and PatchEval evaluation.
- domain assumption Paired vulnerable/fixed functions differ mainly in the security weakness, not in confounds like length or API usage.
Cite this review
Pith. "Pith review of Activation Probes Surface Code-Security Signals that the Model's Output Misses." pith.science (2026). https://pith.science/paper/VHSAKT2J
@misc{pith2026260809643,
author = {Pith},
title = {Pith review of: Activation Probes Surface Code-Security Signals that the Model's Output Misses},
year = {2026},
howpublished = {\url{https://pith.science/paper/VHSAKT2J}},
note = {Machine review of arXiv:2608.09643}
}
read the original abstract
AI coding agents now write a growing share of production code, and human security review does not scale at the rate code is generated. The agents in widest use are closed-weight, so a deploying team cannot read their internals. It can instead run an open-weight model as a reviewer over the agent's output. That reviewer's activations are readable. We ask whether reading those activations recovers a security signal that simply asking the same reviewer misses. We fit a single linear probe per model on a corpus of paired vulnerable-and-fixed Python functions, then test it without retraining on real disclosed vulnerabilities whose weakness type the probe never saw in training, across five open-weight reviewer models. On the vulnerabilities fixed by changing a single function, the probe scores the vulnerable function above its fix on 61-67% of cases for every model, beating the 50% chance line. It also beats the same model's prompted YES/NO win-rate read from its logits, under every prompt we try. Asking the model for a written verdict, even with chain-of-thought, returns the same answer on the vulnerable and fixed function most of the time and so cannot tell them apart. Model activations carry a code-security signal that prompting the same model misses.
Figures
Reference graph
Works this paper leans on
-
[1]
Alain, G. and Bengio, Y. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016. URL https://arxiv.org/abs/1610.01644. ICLR 2017 Workshop
arXiv 2016
-
[2]
The adverse effects of code duplication in machine learning models of code
Allamanis, M. The adverse effects of code duplication in machine learning models of code. In Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) , pp.\ 143--153. ACM, 2019. doi:10.1145/3359591.3359735. URL https://arxiv.org/abs/1812.06469
arXiv 2019
-
[3]
Azaria, A. and Mitchell, T. The internal state of an LLM knows when it's lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp.\ 967--976, 2023. URL https://aclanthology.org/2023.findings-emnlp.68/
work page 2023
-
[4]
Ding, Y., Fu, Y., Ibrahim, O., Sitawarin, C., Chen, X., Alomair, B., Wagner, D., Ray, B., and Chen, Y. Vulnerability detection with code language models: How far are we? In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering ( ICSE ) , pp.\ 1729--1741, 2025. doi:10.1109/ICSE55347.2025.00038. URL https://arxiv.org/abs/2403.18624
arXiv 2025
-
[5]
Dubniczky, R. A., Horv \'a t, K. Z., Bisztray, T., Ferrag, M. A., Cordeiro, L. C., and Tihanyi, N. CASTLE : Benchmarking dataset for static code analyzers and LLMs towards CWE detection. arXiv preprint arXiv:2503.09433, 2025. URL https://arxiv.org/abs/2503.09433
arXiv 2025
-
[6]
Favaro, M. and Clark, J. When AI builds itself. Anthropic Institute report, June 2026. URL https://www.anthropic.com/institute/recursive-self-improvement. Published 2026-06-04. As of May 2026, more than 80\
work page 2026
-
[7]
R., Todd, E., Brinkmann, J., Juang, C., Pal, K., Rager, C., Mueller, A., Marks, S., Sharma, A
Fiotto-Kaufman, J., Loftus, A. R., Todd, E., Brinkmann, J., Juang, C., Pal, K., Rager, C., Mueller, A., Marks, S., Sharma, A. S., Lucchetti, F., Prakash, N., Brodley, C., Guha, A., Bell, J., Wallace, B. C., and Bau, D. NNsight and NDIF : Democratizing access to open-weight foundation model internals. arXiv preprint arXiv:2407.14561, 2024. URL https://arxi...
arXiv 2024
-
[8]
Security weaknesses of Copilot -generated code in GitHub projects: An empirical study
Fu, Y., Liang, P., Tahir, A., et al. Security weaknesses of Copilot -generated code in GitHub projects: An empirical study. ACM Transactions on Software Engineering and Methodology, 34, 2025. doi:10.1145/3716848. URL https://arxiv.org/abs/2310.02059
arXiv 2025
Show all 26 references
-
[9]
Grattafiori, A. et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. URL https://arxiv.org/abs/2407.21783
2024 arXiv
-
[10]
K., Luo, F., Xiong, Y., and Liang, W
Guo, D., Zhu, Q., Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y. K., Luo, F., Xiong, Y., and Liang, W. DeepSeek-Coder : When the large language model meets programming -- the rise of code intelligence. arXiv preprint arXiv:2401.14196, 2024. URL https:...
2024 arXiv
-
[11]
and Vechev, M
He, J. and Vechev, M. Large language models for code: S ecurity hardening and adversarial testing. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS), 2023. URL https://arxiv.org/abs/2302.05319
2023 arXiv
-
[12]
Qwen2.5-Coder technical report
Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Lu, K., et al. Qwen2.5-Coder technical report. arXiv preprint arXiv:2409.12186, 2024. URL https://arxiv.org/abs/2409.12186
2024 arXiv
-
[13]
M., and Anciaux, N
Ibanez-Lissen, L., Gonzalez-Manzano, L., de Fuentes, J. M., and Anciaux, N. LPASS : Linear probes as stepping stones for vulnerability detection using compressed LLMs . arXiv preprint arXiv:2505.24451, 2025. URL https://arxiv.org/abs/2505.24451
2025 arXiv
-
[14]
H., Gonzalez, J
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with PagedAttention . In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP), 2023. URL h...
2023 arXiv
-
[15]
and Tegmark, M
Marks, S. and Tegmark, M. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. In Conference on Language Modeling (COLM), 2024. URL https://arxiv.org/abs/2310.06824
2024 arXiv
-
[16]
S., and Krasheninnikov, D
McKenzie, A., Pawar, U., Blandfort, P., Bankes, W., Krueger, D., Lubana, E. S., and Krasheninnikov, D. Detecting high-stakes interactions with activation probes. arXiv preprint arXiv:2506.10805, 2025. URL https://arxiv.org/abs/2506.10805. NeurIPS 2025
2025
-
[17]
Devstral-Small-2505 , May 2025
Mistral AI and All Hands AI . Devstral-Small-2505 , May 2025. URL https://huggingface.co/mistralai/Devstral-Small-2505. Model release, 2025-05-21
2025
-
[18]
LLMs know more than they show: On the intrinsic representation of LLM hallucinations
Orgad, H., Toker, M., Gekhman, Z., Reichart, R., Szpektor, I., Kotek, H., and Belinkov, Y. LLMs know more than they show: On the intrinsic representation of LLM hallucinations. In International Conference on Learning Representations (ICLR), 2025. URL https://arxiv.org/abs/2410.02707
2025 arXiv
-
[19]
PyTorch : An imperative style, high-performance deep learning library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. PyTorch : An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 (NeurIPS), 2019. URL h...
2019 arXiv
-
[20]
Scikit-learn: Machine learning in P ython
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, \'E . Scikit-learn: Machine learning in P ython. Journal ...
2011
-
[21]
Y C ombinator startups are fastest growing, most profitable in fund history because of AI
Rooney, K. Y C ombinator startups are fastest growing, most profitable in fund history because of AI . CNBC, March 2025. URL https://www.cnbc.com/2025/03/15/y-combinator-startups-are-fastest-growing-in-fund-history-because-of-ai.html. Reports Y Combinator CEO Garry Tan: for 25\
2025
-
[22]
LLMs cannot reliably identify and reason about security vulnerabilities (yet?): A comprehensive evaluation, framework, and benchmarks
Ullah, S., Han, M., Pujar, S., Pearce, H., Coskun, A., and Stringhini, G. LLMs cannot reliably identify and reason about security vulnerabilities (yet?): A comprehensive evaluation, framework, and benchmarks. In 2024 IEEE Symposium on Security and Privacy ( SP ) . IEEE, 2024. ...
2024 arXiv
-
[23]
E., Haberland, M., Reddy, T., Cournapeau, D., et al
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., et al. SciPy 1.0: Fundamental algorithms for scientific computing in P ython. Nature Methods, 17 0 (3): 0 261--272, 2020. doi:10.1038/s41592-019-0686-2. URL https://doi.org/10.1038/s41592-019-0686-2
2020 doi
-
[24]
AI code in the wild: Measuring security risks and ecosystem shifts of AI -generated code in modern software
Wang, B., Yu, W., Zhong, Y., Yu, H., Lian, K., Lu, C., Zheng, H., Zhang, D., and Li, H. AI code in the wild: Measuring security risks and ecosystem shifts of AI -generated code in modern software. arXiv preprint arXiv:2512.18567, 2025. URL https://arxiv.org/abs/2512.18567
2025
-
[25]
PATCHEVAL : A new benchmark for evaluating LLM s on patching real-world vulnerabilities
Wei, Z., Zeng, J., Wen, M., Yu, Z., Cheng, K., Zhu, Y., Guo, J., Zhou, S., Yin, L., Su, X., and Ma, Z. PATCHEVAL : A new benchmark for evaluating LLM s on patching real-world vulnerabilities. arXiv preprint arXiv:2511.11019, 2025. URL https://arxiv.org/abs/2511.11019
2025
-
[26]
Transformers: State-of-the-art natural language processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Sys...
2020
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.