{"id":"b5e601fd-b771-48a1-8b13-ffa18d1c802b","arxiv_id":"2412.03235","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new attack pipeline, ReG-QA, generates natural, semantically related questions from a toxic seed and jailbreaks aligned LLMs at rates up to 93% on GPT-3.5 and 82% on GPT-4.","lead":"This paper shows that safety-aligned chatbots like GPT-4 can be jailbroken by natural questions that are paraphrases or answer-guided rewrites of a toxic prompt, with no explicit jailbreak instruction. The authors propose ReG-QA, a pipeline that turns a toxic question into answers and then back into new questions, achieving attack success rates comparable to or better than leading adversarial attacks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ReG-QA's headline ASR rests on a single GPT-4o-mini judge used both to filter toxic answers and to score final responses; without human validation of final jailbreaks, the 82%/93% figures may be inflated.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the evaluation judge is used both to filter toxic answers and to score final attack success, with no human validation of the final jailbreaks. This is not a general objection to LLM-based judges; it is a specific non-independence that can systematically inflate ASR if the judge is biased toward labeling answer-cue responses unsafe. Since the paper's central contribution is a quantitative ASR comparison, this concern directly affects the strength of the main claim. The concern is addressable with a human-annotation study, so the conditional verdict is appropriate; no change to the reader's verdict is needed.","tokens_in":19915,"tokens_out":6412,"duration_ms":63744,"concrete_test":"Sample 100 final ReG-QA jailbreak instances (seed question, generated question, target response) that the GPT-4o-mini judge labels unsafe, for each of GPT-4 and GPT-3.5. Have three independent human annotators label these responses as safe or unsafe using the same JailbreakBench rubric, and compute human-validated ASR with a 95% confidence interval. If the human-validated ASR is more than 10 percentage points below the reported 82%/93%, or if the judge's false-positive rate on these answer-cue prompts exceeds 20%, the headline ASR is materially inflated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim depends on a single judge model, gpt-4o-mini-2024-07-18 with the JailbreakBench prompt, selected in Section 6.1. The same judge is used inside the ReG-QA pipeline (Algorithm 1, line 3) to filter which generated answers are toxic, and then again to score whether the target model's final response is unsafe. This creates a non-independence: the attack is built around what this judge labels toxic, and the reported ASR is measured by the same labeler. Appendix Table 5 reports FPR 10.5% and FNR 22.7% on the general Judge Comparison set, but no human evaluation of the final ReG-QA prompt-response pairs is provided. Because ReG-QA questions deliberately embed answer-derived details, the judge may classify a response as unsafe when it merely echoes or discusses those cues, even if a human would read it as a refusal or a benign explanation. The headline figures for GPT-4 (82%), GPT-3.5 (93%), and the defense robustness numbers in Table 3 all inherit this proxy. The claim that safety training fails to generalize to semantically related natural prompts is only as strong as the judge's accuracy on these in-distribution, answer-guided prompts.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether safety alignment generalizes to natural, semantically related prompts. It proposes ReG-QA: starting from a toxic seed question, an unaligned LLM generates answers; a judge filters answers that are toxic and sufficiently long; then a safety-aligned LLM (GPT-4o) generates questions that would elicit each selected answer. The resulting question augmentations are evaluated on several API-accessible LLMs. The authors report attack success rates of 93% on GPT-3.5 and 82% on GPT-4 on JailbreakBench, outperforming a paraphrasing baseline and remaining effective against Smooth-LLM, synonym substitution, and removal of non-dictionary words. They conclude that safety training generalizes poorly to semantically related natural prompts and that forward (Q-to-A) safety does not imply reverse (A-to-Q) safety.","tokens_in":20197,"tokens_out":9074,"duration_ms":83362,"significance":"If the reported results hold, the paper is significant: it demonstrates that current alignment can be bypassed by prompts that are natural and not optimized against the target, and it identifies a concrete asymmetry in safety training. The experimental breadth is a strength: eight target models, several defenses, ablations over generation LLMs, and a judge-selection procedure validated against human majority labels (Appendix Table 5). The paper also provides naturalness evidence via log-likelihood comparisons and a human-judge-like preference study. The main weakness is that the judge used to construct the attack is the same judge used to measure attack success, and no human validation of the final ReG-QA prompt–response pairs is reported; this directly affects the headline ASR numbers and the defense-robustness claims.","major_comments":[{"comment":"The load-bearing ASR numbers are produced by the same judge model and prompt used inside the attack-generation pipeline. The paper selects gpt-4o-mini-2024-07-18 with the JailbreakBench judge prompt as Mjudge in §6.1, uses it to filter toxic answers in Algorithm 1 (line 3), and then uses it again to decide whether the target model’s final response is unsafe. Because ReG-QA questions deliberately embed details from answers that this same judge labeled toxic, the judge may systematically classify responses that echo or discuss those cues as unsafe even when human readers would call them refusals or benign explanations. Appendix Table 5 reports FNR 22.7% and FPR 10.5% on the general Judge Comparison set, but no human evaluation of the final ReG-QA prompt–response pairs is reported. The headline figures (82% on GPT-4 and 93% on GPT-3.5 in Table 1) and the defense numbers in Table 3 therefore inherit a potential evaluation bias. The paper should either report human-annotated ASR on a random sample of final responses or recompute ASR with an independent judge that was not used in generation, with per-category agreement statistics.","section":"§6.1, Algorithm 1 line 3, Table 5"},{"comment":"The defense comparison is not fully specified. The caption states that “our approach replaces the default Llama based models with alternate LLMs in both defense implementation and judge LLM,” but the paper does not identify the alternate LLMs or state whether the baseline rows (Prompt and Random Search, PAIR, GCG) were re-run under the same replacements or copied from the JailbreakBench leaderboard. If the baselines were not re-run under identical judge and defense settings, the large differences in Table 3 could reflect evaluation protocol rather than inherent robustness of ReG-QA. Please clarify the exact protocol for each row, and ideally re-run all methods under the same judge and defense configuration so the comparison is controlled.","section":"§6.2, Table 3 caption"},{"comment":"The claim that ReG-QA produces “natural, in-distribution” prompts rests on indirect proxies. The threat model in §4 defines Dnat as prompts naturally occurring in the training distribution, but the generation pipeline does not enforce this constraint: prompts are not filtered by the embedding radius ε before evaluation. The post-hoc analysis in §6.2 (Figure 3b) reports ASR as a function of cosine similarity, but it does not state what fraction of successful jailbreaks lie within a plausible ε of the seed prompt. Since the paper’s central conclusion is specifically about semantically related natural prompts, the authors should provide the distribution of embedding distances for successful jailbreaks and quantify ASR within the declared threat model.","section":"§6.2, §4"}],"minor_comments":[{"comment":"The title contains a typo: “LLM S” should be “LLMs.”","section":"Title"},{"comment":"The algorithm header reads “Reponse Guided Question Augmentation”; “Reponse” should be “Response.”","section":"Algorithm 1 header"},{"comment":"The judge model is listed as “gpt-4o-mini-2024-07-1” in one place; it should be “gpt-4o-mini-2024-07-18” to match the appendix.","section":"§6.1"},{"comment":"The paper says the number of unique answers from the Q→A model was 25.9 ± 20.9 and that answers were “re-sampled” to obtain 1000 unique questions per seed. This is unclear: if answers are re-sampled, duplicates would likely produce duplicate questions; please specify the deduplication procedure and how “unique” is defined.","section":"§5.2"},{"comment":"The statement that “ASR with temperature 0 is always higher than ASR with default (higher) temperatures” is only demonstrated for GPT-3.5 in Table 6; please state the model and criterion this observation is based on, or provide similar data for GPT-4 and the open models.","section":"§6.2, Table 6"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid empirical jailbreak paper with a genuinely new attack loop (Q→A→Q) and a real finding: safety-aligned models generate unsafe questions from toxic answers, and those questions jailbreak other aligned models better than paraphrases. The headline ASRs are probably somewhat inflated because the same GPT-4o-mini judge filters the answers and scores the final responses, but the qualitative conclusion survives that concern.\n\nWhat's new: the response-guided question augmentation pipeline. The paper shows that answer-derived cues matter: asking GPT-4o to produce questions that would elicit a given toxic answer creates prompts that beat paraphrase baselines by large margins across GPT-3.5, GPT-4, Gemma, Qwen, Mistral, and Mixtral. The A→Q asymmetry (forward safety doesn't give reverse safety) is a nice observation, and it's supported by ablations using different Q→A models and by the defense-robustness results. The evaluation is more careful than most: judge selection against human labels (Table 5), multiple success criteria, temperature-0 verification, and per-category breakdowns. The naturalness evidence (log-likelihood, judge preference) is credible.\n\nThe soft spots: the judge issue is real, though not fatal. The same judge family (gpt-4o-mini with the JailbreakBench prompt) is used both to filter toxic answers inside ReG-QA and to decide whether the target response is unsafe. That means the attack is tuned to what this judge calls toxic, and the ASR is measured with the same labeler. FPR 10.5% and FNR 22.7% on the general judge set don't tell us how the judge behaves on answer-guided prompts specifically, and the paper doesn't provide human evaluation of the final jailbreak pairs. So the absolute ASRs (82% GPT-4, 93% GPT-3.5) should be read as upper bounds. The relative comparison to Para-QA is fair since both use the same judge, and the qualitative vulnerability is corroborated by the Appendix D examples, which are clearly harmful. Also minor: no code/data release, and the \"not crafted with jailbreak objective\" framing is a bit strong—prompting for questions that produce a toxic answer is implicitly adversarial, even without target-model optimization.\n\nBottom line: for someone working on safety evaluation or red-teaming, this is worth reading and citing. It deserves a serious referee; the right outcome would be acceptance with a request for human-validated final ASR on a sample and a code release.","headline":"A genuinely new jailbreak loop (answer-guided question augmentation) with strong experiments, but the headline ASRs are upper bounds because the same judge filters and scores.","tokens_in":20703,"tokens_out":1976,"would_cite":true,"duration_ms":19312,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Safety-aligned LLMs can be jailbroken by natural, semantically related questions that are never optimized to attack.","keywords":["safety alignment","jailbreak","natural prompts","question augmentation","distribution shift","attack success rate","LLM safety evaluation","reversal curse"],"falsifier":"Take the seed questions, regenerate the ReG-QA question sets, have independent human raters label each target-model response, and compare the human-unsafe rate with the judge-unsafe rate; if human agreement is materially below the reported judge agreement (85% overall, 22.7% false-negative rate on the paper's own comparison set), the central claim of 82% and 93% attack success is not supported.","tokens_in":19743,"feed_emoji":"🔓","tokens_out":7048,"duration_ms":62020,"temperature":0.7,"pith_summary":"The paper sets out to test whether safety-aligned language models stay safe for natural prompts that are semantically close to a toxic seed question, not just for adversarially optimized or handcrafted jailbreaks. It claims they do not: plain paraphrases of toxic questions—and especially questions built from details of a toxic answer—routinely elicit unsafe content from models such as GPT-4 and GPT-3.5. The proposed pipeline, ReG-QA, first asks an unaligned model to answer a seed question, then asks an aligned model to produce questions that would yield those answers, yielding natural jailbreak prompts without any attack objective. On a standard jailbreak benchmark the method reports 82% attack success on GPT-4 and 93% on GPT-3.5, and it remains effective against defenses that perturb prompts. If true, the result means current safety training generalizes poorly across the semantic neighborhood of a prompt, and forward question-to-answer safety does not imply reverse answer-to-question safety.","feed_headline":"Natural questions jailbreak GPT-4 at 82 percent success","feed_subtitle":"Plain answer-derived prompts beat optimized attacks and survive Smooth-LLM defenses.","key_machinery":"The engine of the method is the ReG-QA (Response Guided Question Augmentation) loop: an unaligned LLM turns a toxic seed question into multiple detailed answers, a judge selects toxic and sufficiently long answers, and a second LLM—often a safety-aligned model like GPT-4o—turns each answer into several questions that would elicit it. The A-to-Q direction is the load-bearing step. Because safety training is asymmetric, the aligned model refuses to answer harmful questions but does not refuse to write questions from harmful answers, and the answer-derived details act as cues that push the target model past its refusal. The pipeline deliberately avoids any jailbreaking objective, so the resulting prompts stay close to the distribution of natural text, which is why perplexity filters and perturbation defenses do not catch them.","core_discovery":"The central discovery is that safety fine-tuning does not generalize to natural, semantically related prompts: aligned models can be jailbroken by questions that are only one paraphrase or one answer-detail away from a refused seed, with no optimization against the target. The paper demonstrates this through ReG-QA, which generates diverse questions by converting a seed question into toxic answers with an unaligned LLM and then converting those answers back into questions with an aligned LLM. It reports attack success rates of 82% on GPT-4 and 93% on GPT-3.5, comparable to or better than optimized attacks, and, in contrast to those attacks, the generated prompts survive perturbation-based defenses because they look and behave like ordinary natural text. The paper also identifies an asymmetry in safety alignment: models trained to refuse harmful questions will still produce harmful questions when prompted with harmful answers, which is the property ReG-QA exploits.","pith_inferences":["The answer-cue mechanism suggests a concrete fix that the paper does not develop: safety training should include refusal on questions reconstructed from toxic answers, not just on the original questions, and ReG-QA could generate that training data.","The same two-step loop is likely to transfer to other safety domains, such as misinformation, regulated advice, or policy violations, whenever an aligned model will still write questions from an unsafe answer; that transfer is testable directly by swapping the seed set.","If the 82% and 93% rates depend on using the same judge family for both answer selection and final scoring, then human-annotated subsets or a judge from a different model family would separate the method's true effect from judge leniency; the paper's own judge-agreement statistics (85% agreement, 22.7% false-negative rate) make this check feasible.","The method maintains high attack success at embedding similarity around 0.7 to the seed, which suggests that meaningful safety guarantees, if they are to exist, would need to be certified over continuous semantic neighborhoods rather than at individual prompts."],"forward_implications":["If safety training does not generalize across paraphrases, then evaluating a model only on its seed refusal set overstates its safety; evaluation should sample the semantic neighborhood of each refusal.","Defenses that flag unnaturalness or instability—perplexity thresholds, Smooth-LLM, synonym substitution—cannot be relied on once the attack is natural; the reported attack success rate stays at or above 82% under those defenses.","The asymmetry between question-to-answer and answer-to-question safety means aligned models can be used as jailbreak generators: simply asking them for questions that produce a toxic answer yields effective attacks on other models.","Because no target-model queries are used during generation, the method is a transferable, black-box assessment that can be applied to any newly released model.","ReG-QA's generated questions can double as safety-training augmentations; the paper frames the method as a tool for assessing and ultimately improving alignment."],"supporting_citations":[{"why":"Provides the GCG gradient-based attack baseline and part of the AdvBench seed prompts used in the benchmark evaluation.","marker":"Zou et al. (2023)"},{"why":"Supplies the PAIR iterative attacker baseline and one of the judge-prompt candidates compared in the judge selection study.","marker":"Chao et al. (2023)"},{"why":"Introduces the JailbreakBench dataset, judge prompt, and leaderboard that define the evaluation and the comparison targets.","marker":"Chao et al. (2024)"},{"why":"Provides the leading random-search attack baseline whose 78% attack success rate the paper compares against its own 82% on GPT-4.","marker":"Andriushchenko et al. (2024)"},{"why":"Contributes the Smooth-LLM defense that ReG-QA is shown to survive while existing attacks collapse under it.","marker":"Robey et al. (2023)"},{"why":"Underlies the Synonym Substitution defense from the leaderboard that the paper tests against.","marker":"Speer (2022)"},{"why":"Supplies the reversal-curse concept that the paper invokes to explain why forward safety training does not imply reverse safety.","marker":"Berglund et al. (2023)"},{"why":"Provides the distribution-shift and competing-objectives explanation for jailbreaks that the paper builds on for its naturalness analysis.","marker":"Wei et al. (2024)"},{"why":"Supplies the Gecko embeddings used to define the threat-model radius and to measure diversity and relevance of generated questions.","marker":"Lee et al. (2024)"},{"why":"Contributes a portion of the seed behaviors in JailbreakBench that the evaluation draws on.","marker":"Mazeika et al. (2024)"}],"fun_headline_variants":["Natural questions jailbreak GPT-4: 82% success, no optimization","Safety training fails on semantic twin prompts: 82% jailbreak","Answer-derived prompts bypass LLM safety and resist defenses","ReG-QA: 82% jailbreak rate with plain natural questions","Semantic relatives of refused prompts crack aligned LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results assume the automated safety judge used both to pick toxic answers and to score responses is a trustworthy measure of unsafe content; if that judge is systematically lenient toward answer-guided questions, the headline success rates would not survive human review.","fun_headline_variants_meta":{"raw":{"variants":["Natural questions jailbreak GPT-4: 82% success, no optimization","Safety training fails on semantic twin prompts: 82% jailbreak","Answer-derived prompts bypass LLM safety and resist defenses","ReG-QA: 82% jailbreak rate with plain natural questions","Semantic relatives of refused prompts crack aligned LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000619,"raw_usage":{"total_tokens":2917,"prompt_tokens":1034,"completion_tokens":1883,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":650,"completion_tokens_details":{"reasoning_tokens":1794}},"tokens_in":650,"tokens_out":1883,"duration_ms":12813,"temperature":1.0,"reasoning_tokens":1794,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:36:47.052196+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the seed questions, regenerate the ReG-QA question sets, have independent human raters label each target-model response, and compare the human-unsafe rate with the judge-unsafe rate; if human agreement is materially below the reported judge agreement (85% overall, 22.7% false-negative rate on the paper's own comparison set), the central claim of 82% and 93% attack success is not supported.","supporting_citations":[{"cited_title":"Harmbench: A standardized evaluation framework for automated red teaming and robust refusal","cited_arxiv_id":null,"evidence_quote":"Contributes a portion of the seed behaviors in JailbreakBench that the evaluation draws on."}],"review_version":1}