{"id":"5dca10c8-3dc3-4af1-8fea-64b6b4232312","arxiv_id":"2507.06043","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A GAN learns to shift malicious prompts into the safe region of an LLM's internal embedding space, and its discriminator is reused as a no-fine-tuning defense filter.","lead":"A new framework trains a GAN on the hidden layers of an LLM to both craft jailbreak attacks and filter unsafe queries. The authors report high attack and defense rates across several LLMs, though the reported defense average drops when all three tested models are counted.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract's 84.17% defense average is not in Table 3; the three-row mean is 81.57%, so the headline defense claim needs correction.","rationale":"The reader's weakest_assumption is the linear separability premise, which is important but partially supported by the high transfer DSR on SafeEdit. I focus instead on a more concrete and immediately checkable flaw: the abstract's 84.17% defense average cannot be reproduced from Table 3. The table lists three models; the mean of all three is 81.57, while the average of the first two is exactly 84.17. The text in Section 5.2 says 'two LLMs' even though the table has three rows, so either the table is misleading or the abstract drops a model. This internal inconsistency directly affects the headline quantitative claim of the paper and should be settled before any further evaluation. The missing threshold p0 is a related reproducibility gap: Eq. (7) defines the defense decision rule, but its operating point is never given, so the reported DSR/BAR balance cannot be independently verified. My proposed test resolves the arithmetic issue decisively; the p0 disclosure is a necessary companion. The reader's verdict of CONDITIONAL remains appropriate, since these are fixable reporting issues rather than evidence of a fundamentally broken method.","tokens_in":15886,"tokens_out":12873,"duration_ms":129480,"concrete_test":"Recompute the mean of the DSR column in Table 3 across all three listed models. If the mean is 81.57 rather than 84.17, revise the Abstract and conclusion to report the full-table average, or remove the Mistral-8B row if the defense evaluation was actually run on only Qwen2.5-7B and Llama3.1-8B. Also report the numerical value of threshold p0 used in Eq. (7) and, if possible, error bars across random seeds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the defense 'reaches an average of 84.17%' (Abstract) is not derivable from the paper's own results. Table 3 reports DSR for Qwen2.5-7B (91.12), Llama3.1-8B (77.22), and Mistral-8B (76.37); the mean over all three is 81.57, while (91.12+77.22)/2 = 84.17. Section 5.2 states the defense is applied to two LLMs, yet the table includes Mistral-8B. The abstract's average silently excludes the worst-performing model. This is not a matter of interpretation: the headline number is inconsistent with the reported table. Additionally, Eq. (7) depends on a threshold p0 that is never specified anywhere in the paper; without p0, the defense operating point and the DSR/BAR trade-off cannot be reproduced or compared with baselines. Because the defense success rate is a cornerstone of the 'unified attack and defense' claim, the paper must correct this reporting and disclose p0 before the headline result can be accepted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CAVGAN, a GAN trained on LLM intermediate-layer embeddings to simultaneously generate jailbreak perturbations (generator) and detect malicious or jailbroken inputs (discriminator). The attack injects the generator output into a selected hidden layer of a white-box LLM; the defense uses the discriminator to flag unsafe embeddings and reruns the model with a handcrafted safety prefix. Experiments report jailbreak success on AdvBench and StrongREJECT across three 7-8B models plus two larger Qwen models, and defense results on SafeEdit jailbreak templates compared with SmoothLLM and RA-LLM. The paper claims a unified attack-defense framework and provides code and data.","tokens_in":16027,"tokens_out":7404,"duration_ms":77325,"significance":"If the results hold, the framework is a valuable proof-of-concept that a single representation-space GAN can serve both white-box jailbreak attacks and inference-time defense without fine-tuning. The multi-model evaluation, the availability of code and data, and the attempt to connect attack and defense mechanisms are strengths. However, the headline defense average is inconsistent with Table 3, the decision threshold p0 is never reported, the generator loss direction is ambiguous, and the defense's safety prefix confounds attribution of the defense success. These issues must be resolved before the quantitative claims can be accepted.","major_comments":[{"comment":"The abstract's claim of an average defense success rate of 84.17% is not derivable from Table 3. The table reports DSR values of 91.12 for Qwen2.5-7B, 77.22 for Llama3.1-8B, and 76.37 for Mistral-8B; the mean over all three rows is 81.57, while the mean over the first two rows is 84.17. Section 5.2 states that the defense is applied to only two LLMs, yet Table 3 includes a Mistral-8B row and the results text also mentions this row. Please clarify which models and metrics are used for the reported average and correct the abstract, the text, and the table caption accordingly.","section":"Abstract; §5.2, Table 3"},{"comment":"The generator objective is underspecified and appears inconsistent with the discriminator's output convention. In Eq. (4), D(h) is trained to be high for benign embeddings and low for malicious embeddings, so D outputs the probability of the benign class. Eq. (3) gives LG = E[log D(h + G(h))]. If the generator minimizes this loss, it drives the perturbed embedding toward the malicious side, which is the opposite of the stated goal of moving malicious queries into the safe area. If the generator instead maximizes LG, the paper must state this explicitly, and the sign in Eq. (3) should be corrected or the notation clarified.","section":"§4.2, Eqs. (3)-(6)"},{"comment":"The norm constraint ||δ|| ≤ ε is stated in Eq. (2) but is never enforced. The paper says this is 'indirectly achieved by normalizing the weights of the parameters of G,' but weight normalization does not bound the magnitude of the generator's output, so the actual perturbation can have arbitrary norm. Please enforce the constraint explicitly (e.g., by projection or a penalty term) and report the value of ε used in the experiments. Without this, the claim that perturbations stay in the semantic space is unsupported.","section":"§4.2, Eq. (2)"},{"comment":"The defense success rate is potentially confounded by the handcrafted safety prefix P_safe. When the discriminator flags an input, the model is rerun with a prefix that explicitly instructs the model to refuse ('Please clearly reject the answer...'). The reported DSR therefore measures the combination of the discriminator trigger and a strong rule-based refusal prompt, not the learned boundary alone. Please include ablations that apply P_safe unconditionally or at a random rate matched to the discriminator's trigger rate, and report the decision threshold p0, which is never specified anywhere in the paper. Without p0, the DSR/BAR operating point cannot be reproduced or compared with the baselines.","section":"§4.3, Eq. (7); Appendix B, P-SAFE"},{"comment":"The two geometric premises—that malicious and benign embeddings are strongly linearly separable and that successful jailbreak attacks move malicious embeddings from the insecure region to the safe region—are cited from prior work (Zhou et al. 2024; Lin et al. 2024) but are not verified on the models and datasets used here. Since the entire GAN design rests on these premises, please provide a concrete check, such as the accuracy of a simple linear probe on the actual victim models (Qwen2.5-7B, Llama3.1-8B, Mistral-8B) or an analysis of the direction of the learned perturbations, to confirm that the assumed geometry holds in the evaluated setting.","section":"§3.2-3.3"}],"minor_comments":[{"comment":"Please specify which metric is averaged to obtain the 88.85% jailbreak success rate in the abstract; the values in Table 1 do not transparently yield this number.","section":"Abstract; Table 1"},{"comment":"The symbol G is used for the classifier in Eq. (1) and for the generator in §4.2, and Eq. (7) uses G(hQ) where the discriminator D is presumably meant; please fix the notation to avoid confusion.","section":"§4.1 and §4.2"},{"comment":"The sentence 'our defense method has achieved a high jailbreak success rate' should read 'our attack method'; it appears in the attack results paragraph and is misleading.","section":"§5.1, Results and Analysis"},{"comment":"The text contains a typo, 'Figue 2', and Eq. (2) refers to a constraint on 'A' instead of δ; please correct these.","section":"Figure 2 and §4.2"},{"comment":"The label 'ASR-Repetiton' should be 'ASR-Repetition'.","section":"Appendix A, Figure 7"},{"comment":"The table is formatted as 'table4' in the text and would benefit from a proper caption and a statement of whether these numbers are ASR-kw or another metric.","section":"Table 4, §5.4"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising core idea and the experiments show nontrivial attack and defense results, but the reporting inconsistencies (defense average, unreported p0, ambiguous generator loss) and the confounding safety prefix need to be fixed before the headline claims can be trusted. The issues are local and fixable, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is better than the packaging. Training a GAN on LLM internal embeddings so the generator produces jailbreak perturbations and the discriminator becomes a no-fine-tuning safety filter is a real step beyond SCAV and JRE, which treat the perturbation as a one-off optimization or a difference of examples. The attack is measured by actual LLM outputs, and the defense is tested on SafeEdit templates that were not in the GAN's training set, so there is no damaging circularity. They also report quality metrics rather than keyword matches alone, which is more honest than most attack papers.\n\nThe soft spots are real but mostly fixable. The stress-test note is correct: the abstract claims an average defense success rate of 84.17%, but Table 3 shows 91.12, 77.22, and 76.37; the mean is 81.57, and 84.17 is just the average of the first two rows. The text in Section 5.2 says the defense was applied to two LLMs, yet the table includes Mistral-8B. That mismatch needs to be corrected before anything else. The other load-bearing problem is that Eq. (7) depends on a threshold p0 that is never reported; without it, the defense operating point and the DSR/BAR trade-off cannot be reproduced or compared to SmoothLLM and RA-LLM. The generator loss in Eq. (3) also lacks an explicit optimization direction, and the norm constraint in Eq. (2) is only loosely enforced by weight normalization. None of these are fatal, but they make the paper harder to trust than the ideas deserve.\n\nSmaller issues: no error bars or multiple runs, only one hundred training samples, and the linear-separability premise is imported from prior work rather than verified on these particular models. The Limitations section openly admits the validation-set tuning and the simple MLP structure, which is honest but undercuts the generality claims.\n\nWho should read this: anyone working on white-box attacks or lightweight defenses. It is a solid workshop-to-conference-level contribution with a novel mechanism, not a paradigm shift. A serious referee should engage with it, but the authors need to fix the headline number, report p0, and add variance estimates before the central claims are fully supported.","headline":"Genuinely useful GAN-based unification of jailbreak attack and defense, but the abstract's defense average is contradicted by the paper's own Table 3 and the unreported threshold p0 blocks reproduction.","tokens_in":16654,"tokens_out":2374,"would_cite":true,"duration_ms":27185,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One GAN learns the LLM's safety boundary and uses it for both jailbreak attacks and defense.","keywords":["LLM jailbreak","adversarial defense","generative adversarial network","concept activation vector","internal representations","linear separability","security alignment"],"falsifier":"Find one jailbreak prompt that succeeds against the target LLM but whose middle-layer embedding is classified on the malicious side of a linear probe trained on clean malicious and benign prompts; such an example would directly contradict the paper's core premise that jailbreak works by moving embeddings into the safe region.","tokens_in":15616,"feed_emoji":"🛡️","tokens_out":6584,"duration_ms":63555,"temperature":0.7,"pith_summary":"The paper claims that a single generative adversarial network (GAN) trained on the internal embeddings of a large language model can do two jobs at once: craft jailbreak perturbations that move malicious queries into the model's 'safe' embedding region, and detect such disguised queries so the model can refuse them. This unifies attack and defense in one framework rather than treating them separately, as prior work does. If the claim holds, the same learned security boundary would give a cheap, no-fine-tuning defense and a high-success attack, reported at an average jailbreak success rate of 88.85% and an average defense success rate of 84.17% on standard benchmarks. The broader point is that the model's own internal geometry is load-bearing: whether harmful and benign prompts are separable inside the model decides whether the GAN's boundary is real.","feed_headline":"One GAN both jailbreaks LLMs and defends them","feed_subtitle":"The same internal security boundary powers high-success attacks and a no-fine-tuning filter.","key_machinery":"The central object is the security judgment boundary in the LLM's intermediate-layer embedding space, formalized as a probabilistic classifier over layer embeddings with a threshold $p_0$. A generator network takes a malicious embedding $h$ as input and emits a perturbation $G(h)$ whose addition is meant to make the discriminator classify the modified embedding as safe, with the norm of $G(h)$ loosely controlled by weight normalization. The discriminator is trained on three classes of embeddings — benign, malicious, and jailbroken — so its output $D(h_Q)$ can be reused as the defense flag that triggers regeneration with a safety prefix $P_{\\mathrm{safe}}$. Layer choice matters: the paper finds middle layers give the best attack quality, because perturbing late layers degrades text quality and perturbing early layers barely bypasses security.","core_discovery":"CAVGAN treats the concept activation vector — the direction along which a concept is encoded inside the model — not as something extracted by optimization but as something a generator produces. Trained adversarially, the generator learns a security concept activation vector whose addition to a malicious query's decoding-layer embedding pushes that embedding out of the malicious region and into the safe region, so the LLM no longer treats the query as harmful. The discriminator learns to tell benign, malicious, and jailbroken embeddings apart, and in defense mode it acts as a filter: if a query's embedding is flagged malicious, the model regenerates its answer with a safety-warning prefix. The paper reports a jailbreak success rate averaging 88.85% across three LLMs, with defense success averaging over 84% on the SafeEdit benchmark while keeping the benign answering rate high. On the attack side it beats the JRE baseline, is close to SCAV, and leads on Mistral-8B; on the defense side it outperforms both SmoothLLM and RA-LLM.","pith_inferences":["A direct test of the paper's premise would be to train a single linear probe on the same layer and ask whether it achieves comparable detection; if it does, the GAN's boundary is essentially the linear separator and the costly adversarial training may not be the source of the gains.","The defense depends on the discriminator recognizing a jailbreak from embeddings alone, so adaptive attackers who craft embeddings that look benign to the discriminator but still fool the LLM may be able to bypass it, since the defense adds no training-time robustness against attacks that target the discriminator itself.","The same 'generated concept vector' trick could generalize beyond safety — for example, steering style, factuality, or harmlessness by generating concept directions on demand for other control tasks.","The drop in attack success after 80 training samples hints at a capacity ceiling in the simple MLP generator; more expressive generators may either improve or destabilize the boundary, so the reported numbers are tied to architecture choices."],"forward_implications":["If the security boundary is learnable by a GAN, defense can be updated against new attacks simply by adversarial training on embeddings, without fine-tuning the LLM.","The same boundary that attacks exploit can be turned into an input filter, so attack research directly produces defense capacity rather than remaining a separate arms race.","Because the method needs only the embedding of one decoding layer, it may transfer to larger models of the same family; the paper reports sustained attack success on Qwen2.5-14B and Qwen2.5-32B.","Layer selection becomes a design knob: middle-layer embeddings preserve text quality after perturbation, giving a concrete place to intervene for both safety and utility."],"supporting_citations":[{"why":"Shows hidden-layer activations distinguish malicious from benign prompts almost everywhere in the network, supplying the linear-separability premise.","marker":"Zhou et al., 2024"},{"why":"Observes that successful jailbreaks move malicious embeddings from the insecure to the safe region, motivating the generator's objective.","marker":"Lin et al., 2024"},{"why":"JRE, the representation-engineering baseline that perturbs internal embeddings, used as comparison and prior art.","marker":"Li et al., 2025"},{"why":"SCAV, the optimization-based concept-activation-vector attack that supplies the baseline, training data split, and evaluation prompts.","marker":"Xu et al., 2024"},{"why":"Introduces concept activation vectors and TCAV, the interpretability tool the paper generalizes into a generative process.","marker":"Kim et al., 2017"},{"why":"Demonstrates steering LLM behavior with residual-stream activation differences, establishing the CAV-style direction idea on LLMs.","marker":"Rimsky et al., 2024"},{"why":"Provides evidence of a shared representation space for malicious queries and the adversarial-suffix attack family the paper situates itself against.","marker":"Zou et al., 2023b"},{"why":"SmoothLLM, the no-fine-tuning defense baseline whose defense success rate CAVGAN is compared against.","marker":"Robey et al., 2024"},{"why":"RA-LLM, the robustly-aligned defense baseline that CAVGAN outperforms on defense success.","marker":"Cao et al., 2024"},{"why":"StrongREJECT, the jailbreak evaluation dataset used alongside AdvBench.","marker":"Souly et al., 2024"}],"fun_headline_variants":["GAN turns LLM safety boundary into jailbreak and defense","One GAN jailbreaks LLMs and filters attacks at 84%","Concept-vector GAN unifies LLM attack and defense","CAVGAN: single GAN for LLM jailbreak and defense","GAN learns LLM security line for attack and defense"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method assumes that harmful and harmless prompts occupy neatly separable regions in the model's internal embedding space, and that every jailbreak works by pushing harmful prompts into the harmless region; if either fails for a new attack, the GAN's boundary is wrong and both attack and defense fail.","fun_headline_variants_meta":{"raw":{"variants":["GAN turns LLM safety boundary into jailbreak and defense","One GAN jailbreaks LLMs and filters attacks at 84%","Concept-vector GAN unifies LLM attack and defense","CAVGAN: single GAN for LLM jailbreak and defense","GAN learns LLM security line for attack and defense"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000257,"raw_usage":{"total_tokens":1586,"prompt_tokens":963,"completion_tokens":623,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":536}},"tokens_in":579,"tokens_out":623,"duration_ms":6384,"temperature":1.0,"reasoning_tokens":536,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:12:21.056566+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Find one jailbreak prompt that succeeds against the target LLM but whose middle-layer embedding is classified on the malicious side of a linear probe trained on clean malicious and benign prompts; such an example would directly contradict the paper's core premise that jailbreak works by moving embeddings into the safe region.","supporting_citations":[{"cited_title":"Cai, James Wexler, Fernanda B","cited_arxiv_id":null,"evidence_quote":"Introduces concept activation vectors and TCAV, the interpretability tool the paper generalizes into a generative process."}],"review_version":1}