Refusal Before Decoding: Detecting and Exploiting Refusal Signals in Intermediate LLM Activations

Alberto Giaretta; Denis Kleyko; Matteo Gioele Collu; Matteo Zavatteri; Mauro Conti; Riccardo Conte; Roberto Confalonieri

REVIEW 2 major objections 2 minor 7 references

Refusal Before Decoding: Detecting and Exploiting Refusal Signals in Intermediate LLM Activations

T0 review · 2 major / 2 minor · reviewed 2026-06-29 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Refusal signals in LLMs are linearly decodable from intermediate activations well before the final layer, allowing probe-guided searches to find jailbreaks faster.

desk verdict The paper shows refusal is linearly decodable from intermediate activations and uses that to speed up AutoDAN search, but the probe's reliability as a fitness proxy on evolving adversarial prompts is the key untested spot. read the letter →

arxiv 2605.28553 v1 pith:5V7E47YJ submitted 2026-05-27 cs.AI cs.CR

Matteo Gioele Collu , Riccardo Conte , Alberto Giaretta , Denis Kleyko , Mauro Conti , Matteo Zavatteri , Roberto Confalonieri This is my paper

classification cs.AIcs.CR

keywords refusaldetectionintermediateactivationslinearprobesjailbreakattacksmechanisticinterpretabilitygeneticsearchLLMsafety

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper establishes that refusal behavior is represented as a structured signal in the residual stream activations at transformer blocks, detectable via linear probes before any output is generated. A sympathetic reader would care because this moves safety mechanisms from opaque final outputs into an earlier, accessible computational stage that can be measured and acted upon during prompt optimization. The authors test actionability by replacing full forward passes in a genetic search algorithm with partial passes and probe scores, yielding attack success rates that match or approach the baseline while cutting per-iteration time by up to 72 percent. The benefit grows with model size, and the resulting prompts retain cross-model transfer performance.

What carries the argument

Linear probes trained on residual stream activations at each block, used inside a genetic search loop to score refusal likelihood from partial forward passes.

What would settle it

A held-out set of prompts where probe scores show no correlation with actual model refusals, or where probe-guided searches produce lower success rates than full-model AutoDAN on the same models.

Watch

Extended reading notes

Core claim

Refusal is linearly decodable well before the final layer from residual stream activations at each transformer block; this signal can substitute for full-model fitness evaluation inside a genetic prompt search, producing Mechanistic AutoDAN that achieves competitive attack success rates with substantially lower compute per iteration.

Load-bearing premise

The linear probes generalize to unseen prompts so that their scores can replace full-model evaluations inside the genetic search without lowering attack quality.

Editorial extensions

If this is right

Probe guidance reduces per-iteration search time by up to 72 percent while preserving attack success rates.
The advantage of probe guidance increases as model scale grows.
Prompts discovered with probe guidance match or exceed baseline cross-model transfer in several tested configurations.
Safety-relevant behavior is represented in intermediate activations before output generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the signal is this early and linear, targeted interventions at specific blocks could suppress or enhance refusal without retraining the whole model.
Real-time monitoring of these probes during normal generation might allow early detection of impending refusals.
The same early-layer representation could be tested on other alignment-related behaviors such as sycophancy or topic avoidance.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

read the letter

The core claim is that refusal shows up as a linear signal in residual stream activations well before the final layer, and that this signal can replace full forward passes inside a genetic jailbreak search without much loss in attack success rate. They call the resulting method Mechanistic AutoDAN and report up to 72% per-iteration time savings plus competitive or better cross-model transfer on the models they tested.

What is actually new is the closed-loop use of the probes inside the evolutionary search itself. Prior work has probed for refusal directions, and AutoDAN already exists, but wiring the two together so the genetic algorithm only runs partial forwards is a practical step that had not been reported.

The engineering result looks useful on its face: time savings that grow with model scale, and final prompts that still transfer. If the numbers hold under proper controls, this is the kind of incremental efficiency gain that labs running large-scale red-teaming would notice.

The soft spot is exactly the one the stress-test note flags. The probes are trained on held-out data that is presumably not adversarial. Once the genetic loop starts mutating prompts, the activations move out of distribution. Nothing in the abstract shows they measured whether probe scores remain correlated with true refusal probability on those mutated prompts, or whether the search ends up exploiting probe artifacts instead of real refusal directions. Competitive final ASR only tells us the endpoint worked; it does not rule out systematic bias in the fitness estimates along the way.

This is for groups working on mechanistic safety or on making adversarial search cheaper. The idea is concrete enough that a serious referee should see the full experiments, including probe training details, statistical tests, and any checks on probe generalization during search. I would send it to review rather than desk-reject.

Referee Report

2 major / 2 minor

Summary. The paper claims that refusal behavior in LLMs is linearly decodable from residual-stream activations at intermediate transformer blocks well before the final layer. It introduces Mechanistic AutoDAN, a variant of the AutoDAN genetic prompt search that substitutes full-model fitness evaluation with partial forward passes plus linear probe scores; across models this yields attack success rates competitive with vanilla AutoDAN, per-iteration time reductions up to 72 %, and in several configurations improved cross-model transfer. The utility of probe guidance is reported to increase with model scale.

Significance. If the central empirical claims are substantiated, the work supplies concrete evidence that safety-relevant refusal circuitry is represented and actionable in intermediate activations, together with a practical efficiency gain for adversarial red-teaming. The scaling trend and transfer results are additional points of interest for mechanistic interpretability and AI safety research.

major comments (2)

[Mechanistic AutoDAN description and experimental results] The actionability claim (that probe scores on residual-stream activations can serve as a faithful proxy inside the genetic loop) rests on the untested assumption that the probes, trained on held-out non-adversarial data, remain accurate on the out-of-distribution adversarial prompts generated during search. No ablation is described that measures probe–full-model agreement on the evolving prompt population or that checks whether the search is optimizing for superficial correlates rather than true refusal direction.
[Abstract and §4 (Experiments)] The abstract and results sections report competitive ASR and up to 72 % time savings, yet the provided text supplies no information on the number of independent runs, variance across seeds, statistical tests, or explicit baseline comparisons that would allow readers to judge whether the efficiency claim is robust or whether any observed ASR parity is within noise.

minor comments (2)

[Methods] Notation for the probe training objective and the precise layer/block indices used for scoring should be stated explicitly rather than left to the supplementary material.
[Figures] Figure captions should include the exact models, prompt counts, and probe training split sizes so that each panel is self-contained.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on the mechanistic claims and experimental reporting. We address each major comment below and outline revisions to strengthen the manuscript.

read point-by-point responses

Referee: The actionability claim rests on the untested assumption that probes trained on held-out non-adversarial data remain accurate on out-of-distribution adversarial prompts. No ablation measures probe–full-model agreement on the evolving prompt population or checks for superficial correlates.

Authors: We agree this is an important untested assumption. While competitive ASR with the probe-guided method provides indirect support that the signal is actionable, we did not explicitly ablate probe agreement on the adversarial prompts generated during search. In revision we will add an analysis comparing probe scores to full-model refusal labels on a sample of prompts from the genetic search trajectory, and we will report whether the search appears to optimize the intended refusal direction. revision: yes
Referee: The abstract and results report competitive ASR and up to 72% time savings, yet supply no information on the number of independent runs, variance across seeds, statistical tests, or explicit baseline comparisons.

Authors: We acknowledge the lack of these details limits assessment of robustness. The original experiments were run with a fixed seed for reproducibility, but variance and statistical comparisons were not reported. In the revised version we will rerun key experiments across multiple random seeds, report means and standard deviations, include statistical tests against the vanilla AutoDAN baseline, and add these metrics to both the abstract and §4. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity; external full-model evaluation anchors results

full rationale

The paper trains linear probes on residual-stream activations from held-out data to predict refusal, then substitutes probe scores for full-model fitness inside a genetic search loop. Final attack success rates are measured by running the evolved prompts through the original model, which is independent of the probe values. No self-definitional equations, fitted inputs renamed as predictions, or load-bearing self-citations appear in the derivation. The method is therefore self-contained against an external benchmark (vanilla AutoDAN ASR).

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

Review performed on abstract only; no explicit free parameters, axioms, or invented entities are stated in the provided text.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Refusal Before Decoding: Detecting and Exploiting Refusal Signals in Intermediate LLM Activations." pith.science (2026). https://pith.science/paper/5V7E47YJ

@misc{pith2026260528553,
  author       = {Pith},
  title        = {Pith review of: Refusal Before Decoding: Detecting and Exploiting Refusal Signals in Intermediate LLM Activations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5V7E47YJ}},
  note         = {Machine review of arXiv:2605.28553}
}

read the original abstract

In this paper, we investigate whether refusal behavior can be predicted from LLM intermediate activations before decoding using linear probes trained on residual stream activations at each transformer block. We find that refusal is linearly decodable well before the final layer, indicating that safety-relevant behavior is represented in intermediate activations before output generation. To test whether this signal is actionable, we introduce Mechanistic AutoDAN, a probe-guided variant of AutoDAN that replaces full-model fitness evaluation with partial forward passes and probe-based scoring inside a genetic prompt search loop. Across the evaluated models, our method achieves attack success rates competitive with vanilla AutoDAN while reducing per-iteration search time by up to 72%, and probe-guided prompts match or exceed AutoDAN's cross-model transfer in several configurations. We further find that the usefulness of probe guidance increases with model scale. Our results show that refusal is not only observable at the output level, but is encoded as a structured and actionable signal in intermediate LLM activations.

Figures

Figures reproduced from arXiv: 2605.28553 by the authors.

**Figure 3.** Pipeline for the creation of the Activation Datasets [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗

**Figure 4.** Example of an augmentation procedure 4. Split the dataset into training, validation, and test sets with a 70%/15%/15% ratio. To reduce leakage across these splits, we cluster similar sentences based on their embedding cosine similarity before assigning them to each set. We obtain the final Clustered Prompt Dataset. The clustering pipeline is the following: we transform the prompts into embedding vectors; we cluste… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

7 extracted references · 3 canonical work pages

[1]

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

Refusal in language models is mediated by a single direction.Advances in Neural Information Processing Systems, 37:136037–136083. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, and 1 others. 2022. Training a helpful and harmless assis- tant with reinforcement learning...

work page Pith review arXiv 2022
[2]

Qwen3Guard Technical Report

Qwen3guard technical report.arXiv preprint arXiv:2510.14276. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, and 1 others. 2023a. Representation engineering: A top-down approach to ai transparency.arXiv preprint arXiv:2310.01405. Andy Zou, Zifan Wang, Nicholas Ca...

work page Pith review arXiv
[3]

For these, we assign the re- spective labels of 1 forrefusableand 0 for compliant

AssembleanInitial Prompt Datasetcollect- ing both harmful and benign sentences from other datasets. For these, we assign the re- spective labels of 1 forrefusableand 0 for compliant. We collectrefusableprompts from these datasets: 520 fromADVBench(Zou et al., 2023b) - MIT License, 390 fromDAN for- bidden questions(Shen et al., 2024) - MIT License, 1960 fr...

2024
[4]

Ad- ditionally, we remove the borderline cases that are considered harmful according to other works, but do not really induce refusal behav- ior

Filterartifacts and remove duplicates. Ad- ditionally, we remove the borderline cases that are considered harmful according to other works, but do not really induce refusal behav- ior. We prompt an aligned LLM to evaluated whether or not it actually refuses the harmful requests. Then, we evaluate the reply with an external LLM Judge; the prompt is reporte...
[5]

In order to do this, we utilize the LLM described in Appendix A.2

Augmentthe dataset to improve coverage and diversity and obtain theAugmented Prompt Dataset. In order to do this, we utilize the LLM described in Appendix A.2. We con- tinue this process until we get a dataset of 10,000 prompts. An example of this proce- dure is shown in Figure 4. Llama 3.2 3B Instruct – Denial Attacks Attack ASR Search Time (s) Attack Ti...
[6]

To re- duce leakage across these splits, we cluster similar sentences based on their embedding cosine similarity before assigning them to each set

Splitthe dataset into training, validation, and test sets with a 70%/15%/15% ratio. To re- duce leakage across these splits, we cluster similar sentences based on their embedding cosine similarity before assigning them to each set. We obtain the finalClustered Prompt Dataset. The clustering pipeline is the following: we transform the prompts into embeddin...

2021
[7]

These are the Activation Datasetsused for the target classi- fier training (as described in Section 4)

Extractthe intermediate activations for each prompt at every transformer block, thereby obtaining for every layer D(l). These are the Activation Datasetsused for the target classi- fier training (as described in Section 4). We will release the dataset under Apache 2.0 License. D Classifiers We report the accuracies on the test set of the LR and MLP probes...

work page arXiv

Pith tools

Reviewed June 29, 2026 · model on record in the stance chip above.

[1] [1]

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

Refusal in language models is mediated by a single direction.Advances in Neural Information Processing Systems, 37:136037–136083. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, and 1 others. 2022. Training a helpful and harmless assis- tant with reinforcement learning...

work page Pith review arXiv 2022

[2] [2]

Qwen3Guard Technical Report

Qwen3guard technical report.arXiv preprint arXiv:2510.14276. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, and 1 others. 2023a. Representation engineering: A top-down approach to ai transparency.arXiv preprint arXiv:2310.01405. Andy Zou, Zifan Wang, Nicholas Ca...

work page Pith review arXiv

[3] [3]

For these, we assign the re- spective labels of 1 forrefusableand 0 for compliant

AssembleanInitial Prompt Datasetcollect- ing both harmful and benign sentences from other datasets. For these, we assign the re- spective labels of 1 forrefusableand 0 for compliant. We collectrefusableprompts from these datasets: 520 fromADVBench(Zou et al., 2023b) - MIT License, 390 fromDAN for- bidden questions(Shen et al., 2024) - MIT License, 1960 fr...

2024

[4] [4]

Ad- ditionally, we remove the borderline cases that are considered harmful according to other works, but do not really induce refusal behav- ior

Filterartifacts and remove duplicates. Ad- ditionally, we remove the borderline cases that are considered harmful according to other works, but do not really induce refusal behav- ior. We prompt an aligned LLM to evaluated whether or not it actually refuses the harmful requests. Then, we evaluate the reply with an external LLM Judge; the prompt is reporte...

[5] [5]

In order to do this, we utilize the LLM described in Appendix A.2

Augmentthe dataset to improve coverage and diversity and obtain theAugmented Prompt Dataset. In order to do this, we utilize the LLM described in Appendix A.2. We con- tinue this process until we get a dataset of 10,000 prompts. An example of this proce- dure is shown in Figure 4. Llama 3.2 3B Instruct – Denial Attacks Attack ASR Search Time (s) Attack Ti...

[6] [6]

To re- duce leakage across these splits, we cluster similar sentences based on their embedding cosine similarity before assigning them to each set

Splitthe dataset into training, validation, and test sets with a 70%/15%/15% ratio. To re- duce leakage across these splits, we cluster similar sentences based on their embedding cosine similarity before assigning them to each set. We obtain the finalClustered Prompt Dataset. The clustering pipeline is the following: we transform the prompts into embeddin...

2021

[7] [7]

These are the Activation Datasetsused for the target classi- fier training (as described in Section 4)

Extractthe intermediate activations for each prompt at every transformer block, thereby obtaining for every layer D(l). These are the Activation Datasetsused for the target classi- fier training (as described in Section 4). We will release the dataset under Apache 2.0 License. D Classifiers We report the accuracies on the test set of the LR and MLP probes...

work page arXiv