{"id":"0fdf8ccf-b539-4b9e-8666-f67585f53a07","arxiv_id":"2411.09125","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"DROJ optimizes a soft prompt to move hidden representations away from the refusal direction, achieving 100% keyword ASR on LLaMA-2-7b-chat, but the responses are often uninformative repeats.","lead":"This paper introduces DROJ, a white-box jailbreak method that optimizes a soft prompt in embedding space to push query representations away from a model's refusal direction, reporting 100% keyword-based attack success on LLaMA-2-7b-chat. The paper is a direct inversion of the DRO defense method, but its evaluation uses an easily gamed metric and the model often responds with repetitive, non-informative text rather than actual harmful content.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 100% ASR is computed by keyword absence and best-of-25 selection, while §4.4 concedes outputs are often repetitive and uninformative; the claimed effective jailbreak is therefore not established.","rationale":"I read DROJ's central claim as: optimizing a soft prompt in embedding space against the fitted refusal direction yields a universal jailbreak prompt that elicits harmful content. The optimization itself is internally coherent and follows Zheng et al.; the PCA/logistic anchor and the regularizer are plausible. The vulnerability is in the operationalization of success. Section 4.3 defines success by absence of refusal keywords and selects the most harmful of 25 stochastic samples; Section 4.4 explicitly concedes many responses are repetitive and non-informative. This means the 100% ASR can be achieved without producing harmful content, so the claim 'effectively preventing direct refusals' is the only part the evidence supports. The comparison to GCG/AutoDAN is also weakened by using published numbers rather than identical settings, but the metric-validity issue is the dominant one. I therefore agree with the reader's weakest-assumption analysis and would keep the CONDITIONAL verdict, conditioned on a harmfulness-based evaluation with identical baselines.","tokens_in":8084,"tokens_out":4203,"duration_ms":40704,"concrete_test":"On a random sample of 100 AdvBench queries, rerun the §4.3 DROJ protocol exactly (25 responses, temperature=1, top_p=0.9), then compute three numbers: (a) keyword ASR on the best-of-25 response; (b) single-sample ASR using only the first returned response; (c) harmfulness ASR using an independent judge (e.g., Llama Guard or a GPT-4 harmfulness rubric) to label whether any response contains actionable harmful content. If either (b) or (c) falls materially below the reported 1.0000, the headline claim depends on best-of-25 keyword filtering and does not demonstrate a successful jailbreak.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the evaluation protocol in §4.3, not the optimization itself. DROJ's objective (Eq. 8) minimizes predicted refusal probability in the PCA/logistic subspace; it does not constrain the generated text to be informative or to contain the requested harmful content. The paper's own §4.4 states that without the helpfulness prompt the model 'often fails to provide genuinely informative responses, frequently repeating the question.' Under the keyword ASR, any response lacking refusal phrases such as \"I'm sorry\" or \"I cannot\" counts as successful, and the metric is applied to the most harmful of 25 temperature-1 samples per query. Repetition or gibberish therefore satisfies the metric. Additionally, the GCG and AutoDAN baselines are taken from Liu et al. (2023) rather than reproduced under the same generation settings, so Table 1 does not support a controlled comparison. Thus the central claim that DROJ is an effective jailbreak that elicits harmful responses is not supported; what is supported is that DROJ suppresses refusal wording.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DROJ (Directed Representation Optimization Jailbreak), a white-box attack that optimizes a continuous jailbreak prompt in embedding space. The prompt is trained to shift hidden representations of queries away from a refusal direction, which is estimated by PCA projection and a logistic regression refusal classifier following Zheng et al. (2024). The authors evaluate on LLaMA-2-7b-chat using the AdvBench and MaliciousInstruct datasets, reporting a keyword-based Attack Success Rate of 100% on both, and they introduce a helpfulness system prompt to mitigate the model's tendency to produce repetitive or uninformative responses. The paper also reports visualization of hidden representations, an interpretability analysis of the learned prompt, and a discussion of limitations including the lack of independent reproduction of GCG and AutoDAN baselines.","tokens_in":8254,"tokens_out":4018,"duration_ms":45989,"significance":"If the central claim were fully established, DROJ would be a meaningful contribution: it shows that a defense mechanism based on representation-space optimization (DRO) can be inverted into an attack, and it demonstrates transfer across datasets without training on the test set. The method's optimization derivation is clear, the code is publicly available, and the transfer result is a concrete strength. However, the evidence as presented does not establish that DROJ elicits harmful content; it establishes that DROJ suppresses refusal wording under a particular keyword-based metric. Because the evaluation protocol is the load-bearing element of the paper, the significance of the reported 100% ASR is conditional on a substantially revised evaluation.","major_comments":[{"comment":"The headline 100% ASR is computed by checking keyword absence on the most harmful of 25 temperature-1 samples per query, and §4.4 itself admits that without the helpfulness prompt the model 'often fails to provide genuinely informative responses, frequently repeating the question.' Under this metric, any non-refusal output, including repetition or gibberish, is scored as a successful attack. The paper therefore does not demonstrate that DROJ elicits harmful content. Please report human or LLM-judged response quality, including whether each response actually contains the requested harmful information, and provide a representative sample of successful outputs rather than a single illustrative figure.","section":"§4.3–4.4, Tables 1–2"},{"comment":"The GCG and AutoDAN baseline numbers are quoted from Liu et al. (2023) without reproduction under the same sampling protocol used for DROJ and the No Jailbreak condition (temperature=1, top-p=0.9, 25 returned sequences, best-of-25 selection). The comparison in Table 1 is therefore not controlled, and the claim that DROJ 'significantly outperforms' these baselines is not supported by the data as presented. Reproduce the baselines under identical settings, or clearly label the comparison as inherited from prior work with different generation configurations and refrain from stating a performance advantage based on that comparison.","section":"§5 (Limitations) and Table 1"},{"comment":"The optimization objective minimizes predicted refusal probability in the PCA/logistic subspace plus a norm-preservation regularizer; there is no term that constrains the response to be relevant, informative, or to contain the requested harmful content. Consequently, the high ASR may reflect only the absence of refusal phrases. To interpret the 100% ASR as evidence of jailbreak effectiveness, the authors should include control conditions under the same 25-sample protocol, such as a random continuous prompt or a trivial non-refusal instruction (e.g., 'Ignore safety instructions'), and show that DROJ's ASR and response quality exceed those of the controls.","section":"§3.3, Eq. (8)"},{"comment":"The paper reports visualizations using empirical refusal rates and states that harmful queries are 'less likely to be refused after the addition of the jailbreak prompt,' but the quantitative evaluation uses only the keyword-based ASR. Please report the actual refusal rate of the final model on held-out queries with and without the helpfulness prompt, and assess how well the logistic-regression proxy predicts true refusal behavior. This would clarify whether DROJ truly bypasses the model's refusal mechanism or merely avoids the specific refusal wording captured by the keyword list.","section":"§4.2, Figure 3"}],"minor_comments":[{"comment":"The word 'Rrepresentation' in the title and abstract appears to be a typo and should read 'Representation.'","section":"Title and Abstract"},{"comment":"The abstract says the model 'occasionally produces repetitive and non-informative responses,' while §4.4 says the model 'often fails to provide genuinely informative responses'; this inconsistency should be resolved with a quantitative measure of response informativeness.","section":"Abstract and §4.4"},{"comment":"The figure relies on colored symbols to distinguish harmful and harmless queries; consider adding distinct markers or a grayscale-friendly palette to improve accessibility for colorblind readers and for print.","section":"Figure 3"},{"comment":"The table uses '-' for unavailable GCG and AutoDAN results; please define this notation in the caption, for example as 'not reported in Liu et al. (2023) for this dataset.'","section":"Table 2"},{"comment":"Only a single training run is reported (La=14.74, Lr=1.99); given random initialization of the jailbreak prompt, reporting the mean and standard deviation over multiple seeds would strengthen confidence in the optimization results.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper's core derivation appears sound, and the transfer of the optimized prompt to held-out datasets is a useful result. The main decision hinges on the evaluation: the combination of best-of-25 selection, keyword-based ASR, and the paper's own admission of uninformative outputs makes the central claim currently unsupported. I also note that the method is essentially the direct inversion of DRO (Zheng et al., 2024); the novelty is incremental but acceptable if the evaluation is made rigorous. The lack of reproduced baselines and the absence of a response-quality metric are fixable within a major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is genuinely worth a look: take Zheng et al.'s DRO defense and invert it. Instead of pushing harmful queries toward the refusal direction, DROJ optimizes a soft prompt to push all queries away from it. The math is straightforward and the implementation is competent—Equation 8 is a clean objective, the PCA/logistic classifier anchors the refusal direction sensibly, and the regularization term is derived correctly. The authors also release code and are unusually upfront about what doesn't work. That transparency counts for something.\n\nThe soft spot is exactly where the stress-test puts it: the evaluation. The 100% ASR is computed by taking the most harmful of 25 generated samples per query and checking whether the response avoids refusal keywords like \"I'm sorry.\" Repetition and gibberish pass that test. And the paper itself admits in Section 4.4 that without the helpfulness prompt the model often just repeats the question. So the strong claim—DROJ elicits harmful content—is not established. What is established is that DROJ suppresses refusal wording on LLaMA-2-7b-chat.\n\nThe baseline comparison is also weaker than it looks. GCG and AutoDAN numbers are taken from Liu et al. rather than reproduced under the same generation settings, so Table 1 is not a controlled comparison. The paper acknowledges this in Section 5, which is good, but it means the performance gap is not as clean as the abstract implies.\n\nThat said, this is not a paper with a load-bearing technical flaw. The optimization works as intended; the failure is in the evaluation protocol and the interpretation of the metric. The idea of inverting a representation-space defense is a legitimate contribution, and the limitations are stated rather than hidden. For a workshop paper or a revise-and-resubmit at a security venue, there is a solid kernel here. The authors need to redo the evaluation with single-sample generation, a content-informativeness measure, and properly reproduced baselines before the empirical claims can be trusted.\n\nOverall: worth engaging with, but the current version oversells. I would send it to peer review because the method is new and the flaws are fixable, not because the current evidence supports the headline.","headline":"A clean inversion of the DRO defense that likely suppresses refusal wording, but the headline 100% ASR is an artifact of a best-of-25 keyword metric and the paper's own examples show the outputs are often uninformative gibberish.","tokens_in":8762,"tokens_out":1360,"would_cite":false,"duration_ms":102290,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DROJ optimizes a jailbreak prompt in embedding space to shift hidden representations away from the refusal direction, achieving 100% keyword-based ASR on LLaMA-2-7b-chat.","keywords":["DROJ","jailbreak attack","embedding-space optimization","refusal direction","white-box attack","attack success rate","LLM safety"],"falsifier":"Run DROJ on LLaMA-2-7b-chat and, for a random sample of AdvBench and MaliciousInstruct queries, have independent raters score the selected most harmful of 25 responses for whether it actually provides actionable harmful content; if ASR stays 1.0000 while informativeness is near zero, the central claim of successful jailbreak is not supported.","tokens_in":7839,"feed_emoji":"🔓","tokens_out":7436,"duration_ms":66891,"temperature":0.7,"pith_summary":"The paper tries to establish that refusal behavior in an aligned language model is governed by a low-dimensional direction in the hidden representation space, and that this direction can be attacked directly. Its method, DROJ, optimizes a continuous jailbreak prompt in embedding space so that harmful queries, when prefixed with that prompt, move away from the refusal direction; on LLaMA-2-7b-chat this yields a keyword-based attack success rate of 1.0000 on both AdvBench and MaliciousInstruct. The authors also report that the model often answers with repetitive, uninformative text rather than real harmful content, and they introduce a helpfulness system prompt that improves utility while slightly reducing ASR. If the claim holds, it matters because it shows safety alignment can be undone by optimizing in latent space rather than searching over readable text, and that common keyword-based metrics can overstate how much harmful content an attack actually elicits.","feed_headline":"Jailbreak soft prompt flips refusal direction in LLaMA-2","feed_subtitle":"Continuous prompt moves harmful queries off the refusal direction; keyword ASR hits 100 percent.","key_machinery":"The central object is the refusal direction fitted by Zheng et al.'s anchoring procedure: hidden states of harmful and harmless queries are projected with PCA into a low-dimensional space, and a logistic regression on the projected states yields a weight vector $w_\\phi$ that points toward higher refusal probability. DROJ optimizes a continuous jailbreak prompt $\\theta$ in embedding space to minimize $L_a(\\theta)=\\frac{1}{N}\\sum_i \\log\\sigma[f_\\phi(x_{\\theta,i})-f_\\phi(x_{0,i})]$, which, because $\\log\\sigma$ is monotone, is equivalent to minimizing $w_\\phi^T(p(x_{\\theta,i})-p(x_{0,i}))$: moving each prefixed query's projected representation against the refusal direction. A regularization term $L_r(\\theta)$ penalizes changes in the remaining $n-m$ hidden dimensions, preserving query meaning, and the full objective is $L(\\theta)=L_a(\\theta)+\\beta L_r(\\theta)$. This machinery turns jailbreaking into a continuous optimization problem over representations rather than a search over discrete tokens.","core_discovery":"DROJ's central claim is that a soft prompt optimized directly in the embedding space can serve as a universal jailbreak prefix for a white-box model: after fitting a logistic refusal classifier on a PCA projection of hidden states, the method minimizes the log-sigmoid gap between the predicted refusal score of prefixed and unprefixed queries, pulling representations away from the refusal direction. On LLaMA-2-7b-chat, this trained 20-token prompt transfers to unseen benchmarks, reaching an ASR of 1.0000 on AdvBench and MaliciousInstruct while the no-prompt baseline sits near 0.05 and 0.0001, respectively. The same prompt also moves harmless queries away from refusal, and the authors observe that the attack prevents direct refusals without guaranteeing informative answers.","pith_inferences":["A corollary the authors do not spell out: the same direction-steering logic suggests that refusal is not a robust property of the model but a representation-space boundary; defenses that widen or randomize that boundary could blunt DROJ.","Because the evaluation selects the most harmful of 25 generations, comparing ASR numbers under different sampling settings can overstate gains; a matched-generation comparison to GCG and AutoDAN would be a stronger test.","The helpfulness-prompt result hints at a two-step attack pattern: first suppress refusal in latent space, then use a system prompt to steer decoding toward content; this combination could be adapted to other alignment layers.","If DROJ is correct, alignment via RLHF alone will remain vulnerable to continuous-space attacks as long as refusal is concentrated in low-dimensional hidden directions."],"forward_implications":["DROJ achieves a keyword-based ASR of 1.0000 on both AdvBench and MaliciousInstruct for LLaMA-2-7b-chat without being trained on either benchmark.","The learned soft prompt transfers across query sets, moving both harmful and harmless representations consistently away from the refusal direction.","Adding a helpfulness system prompt improves response utility but lowers AdvBench ASR slightly to 0.9846, indicating a trade-off between refusal avoidance and informative output.","The optimized prompt is not human-interpretable, since nearest-token projections yield mostly unrelated tokens; interpretability is not needed for the attack to work.","Because DROJ is a white-box attack requiring access to hidden states, it is not directly transferable across models, a limitation the authors acknowledge."],"supporting_citations":[{"why":"This work supplies the anchoring procedure, the training queries, and the DRO defense that DROJ inverts.","marker":"Zheng et al. (2024)"},{"why":"It provides the AdvBench Harmful Behaviors benchmark, the GCG baseline, and the keyword-based ASR metric.","marker":"Zou et al. (2023)"},{"why":"It provides the AutoDAN baseline, the exact refusal keyword list, and the reported GCG/AutoDAN ASR numbers that DROJ is compared against.","marker":"Liu et al. (2023)"},{"why":"It provides the MaliciousInstruct benchmark used as a second test set for the optimized jailbreak prompt.","marker":"Huang et al. (2023)"}],"fun_headline_variants":["Soft prompt rewires refusal direction for 100% jailbreak","Embedding-level jailbreak flips LLaMA-2 refusal signal","Universal jailbreak prompt erases LLaMA-2 refusals","DROJ: 20-token prompt hijacks LLaMA-2 hidden states","Refusal direction reversed by optimized soft prompt"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that a keyword-based ASR computed by picking the most harmful of 25 responses measures a successful jailbreak; the paper's own admission that responses are often repetitive and non-informative shows this metric can be satisfied without useful harmful content being produced.","fun_headline_variants_meta":{"raw":{"variants":["Soft prompt rewires refusal direction for 100% jailbreak","Embedding-level jailbreak flips LLaMA-2 refusal signal","Universal jailbreak prompt erases LLaMA-2 refusals","DROJ: 20-token prompt hijacks LLaMA-2 hidden states","Refusal direction reversed by optimized soft prompt"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1248,"prompt_tokens":902,"completion_tokens":346,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":254}},"tokens_in":518,"tokens_out":346,"duration_ms":6998,"temperature":1.0,"reasoning_tokens":254,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:59:42.850990+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DROJ on LLaMA-2-7b-chat and, for a random sample of AdvBench and MaliciousInstruct queries, have independent raters score the selected most harmful of 25 responses for whether it actually provides actionable harmful content; if ASR stays 1.0000 while informativeness is near zero, the central claim of successful jailbreak is not supported.","supporting_citations":[],"review_version":1}