REVIEW 4 major objections 5 minor 4 cited by
DROJ: A Prompt-Driven Attack against Large Language Models
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read DROJ optimizes a jailbreak prompt in embedding space to shift hidden representations away from the refusal direction, achieving 100% keyword-based ASR on LLaMA-2-7b-chat.
desk verdict A clean inversion of the DRO defense that likely suppresses refusal wording, but the headline 100% ASR is an artifact of a best-of-25 keyword metric and the paper's own examples show the outputs are often uninformative gibberish. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the refusal direction fitted by Zheng et al.'s anchoring procedure: hidden states of harmful and harmless queries are projected with PCA into a low-dimensional space, and a logistic regression on the projected states yields a weight vector $w_\phi$ that points toward higher refusal probability. DROJ optimizes a continuous jailbreak prompt $\theta$ in embedding space to minimize $L_a(\theta)=\frac{1}{N}\sum_i \log\sigma[f_\phi(x_{\theta,i})-f_\phi(x_{0,i})]$, which, because $\log\sigma$ is monotone, is equivalent to minimizing $w_\phi^T(p(x_{\theta,i})-p(x_{0,i}))$: moving each prefixed query's projected representation against the refusal direction. A regularization term $L_r(\theta)$ penalizes changes in the remaining $n-m$ hidden dimensions, preserving query meaning, and the full objective is $L(\theta)=L_a(\theta)+\beta L_r(\theta)$. This machinery turns jailbreaking into a continuous optimization problem over representations rather than a search over discrete tokens.
What would settle it
Run DROJ on LLaMA-2-7b-chat and, for a random sample of AdvBench and MaliciousInstruct queries, have independent raters score the selected most harmful of 25 responses for whether it actually provides actionable harmful content; if ASR stays 1.0000 while informativeness is near zero, the central claim of successful jailbreak is not supported.
Extended reading notes
Core claim
DROJ's central claim is that a soft prompt optimized directly in the embedding space can serve as a universal jailbreak prefix for a white-box model: after fitting a logistic refusal classifier on a PCA projection of hidden states, the method minimizes the log-sigmoid gap between the predicted refusal score of prefixed and unprefixed queries, pulling representations away from the refusal direction. On LLaMA-2-7b-chat, this trained 20-token prompt transfers to unseen benchmarks, reaching an ASR of 1.0000 on AdvBench and MaliciousInstruct while the no-prompt baseline sits near 0.05 and 0.0001, respectively. The same prompt also moves harmless queries away from refusal, and the authors observe that the attack prevents direct refusals without guaranteeing informative answers.
Load-bearing premise
The load-bearing assumption is that a keyword-based ASR computed by picking the most harmful of 25 responses measures a successful jailbreak; the paper's own admission that responses are often repetitive and non-informative shows this metric can be satisfied without useful harmful content being produced.
Editorial extensions
If this is right
- DROJ achieves a keyword-based ASR of 1.0000 on both AdvBench and MaliciousInstruct for LLaMA-2-7b-chat without being trained on either benchmark.
- The learned soft prompt transfers across query sets, moving both harmful and harmless representations consistently away from the refusal direction.
- Adding a helpfulness system prompt improves response utility but lowers AdvBench ASR slightly to 0.9846, indicating a trade-off between refusal avoidance and informative output.
- The optimized prompt is not human-interpretable, since nearest-token projections yield mostly unrelated tokens; interpretability is not needed for the attack to work.
- Because DROJ is a white-box attack requiring access to hidden states, it is not directly transferable across models, a limitation the authors acknowledge.
Reading between the lines
- A corollary the authors do not spell out: the same direction-steering logic suggests that refusal is not a robust property of the model but a representation-space boundary; defenses that widen or randomize that boundary could blunt DROJ.
- Because the evaluation selects the most harmful of 25 generations, comparing ASR numbers under different sampling settings can overstate gains; a matched-generation comparison to GCG and AutoDAN would be a stronger test.
- The helpfulness-prompt result hints at a two-step attack pattern: first suppress refusal in latent space, then use a system prompt to steer decoding toward content; this combination could be adapted to other alignment layers.
- If DROJ is correct, alignment via RLHF alone will remain vulnerable to continuous-space attacks as long as refusal is concentrated in low-dimensional hidden directions.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes DROJ (Directed Representation Optimization Jailbreak), a white-box attack that optimizes a continuous jailbreak prompt in embedding space. The prompt is trained to shift hidden representations of queries away from a refusal direction, which is estimated by PCA projection and a logistic regression refusal classifier following Zheng et al. (2024). The authors evaluate on LLaMA-2-7b-chat using the AdvBench and MaliciousInstruct datasets, reporting a keyword-based Attack Success Rate of 100% on both, and they introduce a helpfulness system prompt to mitigate the model's tendency to produce repetitive or uninformative responses. The paper also reports visualization of hidden representations, an interpretability analysis of the learned prompt, and a discussion of limitations including the lack of independent reproduction of GCG and AutoDAN baselines.
Significance. If the central claim were fully established, DROJ would be a meaningful contribution: it shows that a defense mechanism based on representation-space optimization (DRO) can be inverted into an attack, and it demonstrates transfer across datasets without training on the test set. The method's optimization derivation is clear, the code is publicly available, and the transfer result is a concrete strength. However, the evidence as presented does not establish that DROJ elicits harmful content; it establishes that DROJ suppresses refusal wording under a particular keyword-based metric. Because the evaluation protocol is the load-bearing element of the paper, the significance of the reported 100% ASR is conditional on a substantially revised evaluation.
major comments (4)
- [§4.3–4.4, Tables 1–2] The headline 100% ASR is computed by checking keyword absence on the most harmful of 25 temperature-1 samples per query, and §4.4 itself admits that without the helpfulness prompt the model 'often fails to provide genuinely informative responses, frequently repeating the question.' Under this metric, any non-refusal output, including repetition or gibberish, is scored as a successful attack. The paper therefore does not demonstrate that DROJ elicits harmful content. Please report human or LLM-judged response quality, including whether each response actually contains the requested harmful information, and provide a representative sample of successful outputs rather than a single illustrative figure.
- [§5 (Limitations) and Table 1] The GCG and AutoDAN baseline numbers are quoted from Liu et al. (2023) without reproduction under the same sampling protocol used for DROJ and the No Jailbreak condition (temperature=1, top-p=0.9, 25 returned sequences, best-of-25 selection). The comparison in Table 1 is therefore not controlled, and the claim that DROJ 'significantly outperforms' these baselines is not supported by the data as presented. Reproduce the baselines under identical settings, or clearly label the comparison as inherited from prior work with different generation configurations and refrain from stating a performance advantage based on that comparison.
- [§3.3, Eq. (8)] The optimization objective minimizes predicted refusal probability in the PCA/logistic subspace plus a norm-preservation regularizer; there is no term that constrains the response to be relevant, informative, or to contain the requested harmful content. Consequently, the high ASR may reflect only the absence of refusal phrases. To interpret the 100% ASR as evidence of jailbreak effectiveness, the authors should include control conditions under the same 25-sample protocol, such as a random continuous prompt or a trivial non-refusal instruction (e.g., 'Ignore safety instructions'), and show that DROJ's ASR and response quality exceed those of the controls.
- [§4.2, Figure 3] The paper reports visualizations using empirical refusal rates and states that harmful queries are 'less likely to be refused after the addition of the jailbreak prompt,' but the quantitative evaluation uses only the keyword-based ASR. Please report the actual refusal rate of the final model on held-out queries with and without the helpfulness prompt, and assess how well the logistic-regression proxy predicts true refusal behavior. This would clarify whether DROJ truly bypasses the model's refusal mechanism or merely avoids the specific refusal wording captured by the keyword list.
minor comments (5)
- [Title and Abstract] The word 'Rrepresentation' in the title and abstract appears to be a typo and should read 'Representation.'
- [Abstract and §4.4] The abstract says the model 'occasionally produces repetitive and non-informative responses,' while §4.4 says the model 'often fails to provide genuinely informative responses'; this inconsistency should be resolved with a quantitative measure of response informativeness.
- [Figure 3] The figure relies on colored symbols to distinguish harmful and harmless queries; consider adding distinct markers or a grayscale-friendly palette to improve accessibility for colorblind readers and for print.
- [Table 2] The table uses '-' for unavailable GCG and AutoDAN results; please define this notation in the caption, for example as 'not reported in Liu et al. (2023) for this dataset.'
- [§4.2] Only a single training run is reported (La=14.74, Lr=1.99); given random initialization of the jailbreak prompt, reporting the mean and standard deviation over multiple seeds would strengthen confidence in the optimization results.
Circularity Check
No circularity: DROJ's optimization objective and its keyword-based ASR are distinct; the repetitive-output limitation is an evaluation-validity concern, not a by-construction reduction.
full rationale
The paper's derivation chain is self-contained with respect to circularity. DROJ inherits the PCA-plus-logistic-regression anchoring method and the regularization term from Zheng et al. (2024), but that is an external prior work (no author overlap) and is not a self-citation. The attack objective (Eq. 8, minimizing La + beta*Lr) directly minimizes the fitted refusal classifier f_phi in the m-dimensional PCA subspace; the reported metric (Section 4.3) is instead a keyword-based Attack Success Rate computed on held-out AdvBench and MaliciousInstruct queries with 25 samples per query. No equation in the paper identifies f_phi with the keyword ASR, so the 1.0000 ASR is an empirical transfer result rather than an identity forced by the optimization. The paper's own Section 4.4 concession that responses are often repetitive and non-informative ('the LLM often fails to provide genuinely informative responses, frequently repeating the question') does undermine the construct validity of ASR as evidence of eliciting harmful content, and Section 5 acknowledges that GCG/AutoDAN baselines were not reproduced. These are measurement-validity and reproducibility limitations, not circular derivation or a prediction that reduces by construction to a fitted parameter. No self-citation chain, imported uniqueness theorem, or ansatz-via-citation step is load-bearing. Score 0 is therefore appropriate under the circularity rubric.
Assumptions & free parameters
free parameters (4)
- beta (regularization coefficient) =
1e-3
- m (PCA dimension) =
4
- Jailbreak prompt length =
20
- Helpfulness system prompt =
manual
assumptions (5)
- domain assumption The refusal direction can be captured by a logistic regression on the first m PCA components of the hidden states.
- domain assumption The PCA fitted on training queries generalizes to held-out test queries.
- domain assumption The hidden state at the last token of a query is a sufficient representation for predicting refusals.
- domain assumption The keyword-based ASR metric (absence of refusal phrases) is a valid proxy for a successful jailbreak.
- domain assumption Generating 25 responses and selecting the most harmful one is a fair way to measure attack success.
invented entities (1)
-
None
Cite this review
Pith. "Pith review of DROJ: A Prompt-Driven Attack against Large Language Models." pith.science (2026). https://pith.science/paper/REO4SNA5
@misc{pith2026241109125,
author = {Pith},
title = {Pith review of: DROJ: A Prompt-Driven Attack against Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/REO4SNA5}},
note = {Machine review of arXiv:2411.09125}
}
read the original abstract
Large Language Models (LLMs) have demonstrated exceptional capabilities across various natural language processing tasks. Due to their training on internet-sourced datasets, LLMs can sometimes generate objectionable content, necessitating extensive alignment with human feedback to avoid such outputs. Despite massive alignment efforts, LLMs remain susceptible to adversarial jailbreak attacks, which usually are manipulated prompts designed to circumvent safety mechanisms and elicit harmful responses. Here, we introduce a novel approach, Directed Rrepresentation Optimization Jailbreak (DROJ), which optimizes jailbreak prompts at the embedding level to shift the hidden representations of harmful queries towards directions that are more likely to elicit affirmative responses from the model. Our evaluations on LLaMA-2-7b-chat model show that DROJ achieves a 100\% keyword-based Attack Success Rate (ASR), effectively preventing direct refusals. However, the model occasionally produces repetitive and non-informative responses. To mitigate this, we introduce a helpfulness system prompt that enhances the utility of the model's responses. Our code is available at https://github.com/Leon-Leyang/LLM-Safeguard.
Figures
Figures from the paper (1 more)
Forward citations
Cited by 4 Pith papers
-
On Surjectivity of Neural Networks: Can you elicit any behavior from your model?
Pre-LayerNorm transformers and linear attention are almost always surjective, so any target output has an input that produces it in the continuous embedding space.
-
Probing the Difficulty Perception Mechanism of Large Language Models
LLMs linearly encode math-problem difficulty in their final-token representations, and specific final-layer attention heads are specialized for easy vs hard problems.
-
LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems
A systematic review that categorizes LLM threats, severity scores, and mitigations across development and operation life cycles and multiple deployment scenarios.
-
SoK: A Comprehensive Security Analysis of Jailbreak Resilience in GPT and DeepSeek Models
Across 510 HarmBench behaviors and seven attack methods, GPT-4 models show more consistent jailbreak resilience than DeepSeek models, whose vulnerability grows with scale.
Reference graph
Works this paper leans on
-
[1]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,
-
[3]
Foundational challenges in assuring alignment and safety of large language models
Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. Foundational challenges in assuring alignment and safety of large language models. arXiv preprint arXiv:2404.09932,
-
[7]
Catastrophic jailbreak of open-source llms via exploiting generation
Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. Catastrophic jailbreak of open-source llms via exploiting generation. arXiv preprint arXiv:2310.06987,
-
[8]
Baseline defenses for adversarial attacks against aligned language models
Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chi- ang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614,
-
[9]
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825,
-
[10]
Open sesame! universal black box jailbreaking of large language models
Raz Lapid, Ron Langberg, and Moshe Sipper. Open sesame! universal black box jailbreaking of large language models. arXiv preprint arXiv:2309.01446,
-
[11]
Autodan: Generating stealthy jailbreak prompts on aligned large language models
Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451,
-
[13]
Hao Wang, Hao Li, Minlie Huang, and Lei Sha. From noise to clarity: Unraveling the adver- sarial suffix of large language model attacks via translation of text embeddings. arXiv preprint arXiv:2402.16006,
Show all 18 references
-
[14]
Finetuned language models are zero-shot learners.arXiv preprint arXiv:2109.01652,
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners.arXiv preprint arXiv:2109.01652,
-
[15]
Jailbreak and guard aligned language models with only few in-context demonstrations
Zeming Wei, Yifei Wang, and Yisen Wang. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv:2310.06387,
-
[16]
Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher
Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher. arXiv preprint arXiv:2308.06463,
-
[17]
Prompt-driven llm safeguarding via directed representation optimization
Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng. Prompt-driven llm safeguarding via directed representation optimization. arXiv preprint arXiv:2401.18018,
-
[18]
Universal and transferable adversarial attacks on aligned language models
Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043,
-
[2017]
Gradient-based adversarial attacks against text transformers
Chuan Guo, Alexandre Sablayrolles, Herv ´e J ´egou, and Douwe Kiela. Gradient-based adversarial attacks against text transformers. arXiv preprint arXiv:2104.13733,
-
[2021]
Llm self defense: By self examination, llms know they are being tricked
Alec Helbling, Mansi Phute, Matthew Hull, and Duen Horng Chau. Llm self defense: By self examination, llms know they are being tricked. arXiv preprint arXiv:2308.07308,
-
[2022]
Scalable and transferable black-box jailbreaks for language models via persona modulation
Rusheb Shah, Soroush Pour, Arush Tagade, Stephen Casper, Javier Rando, et al. Scalable and transferable black-box jailbreaks for language models via persona modulation. arXiv preprint arXiv:2311.03348,
-
[2023]
Detecting language model attacks with perplexity
Gabriel Alon and Michael Kamfonas. Detecting language model attacks with perplexity. arXiv preprint arXiv:2308.14132,
-
[2024]
Training a helpful and harmless assistant with reinforcement learning from human feedback
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862,
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.