{"id":"5b73e359-c0e3-4ec6-8181-db6669cbe5a1","arxiv_id":"2509.10931","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"HaPLa, a black-box jailbreak combining abductive framing and symbolic encoding, achieves over 95% ASR on GPT-series and over 70% on all six tested LLMs, and safety tuning against it significantly degrades helpfulness.","lead":"This paper shows a new way to bypass LLM safety guards by rewriting harmful requests as puzzle-like stories and encoding sensitive words in symbols, succeeding over 95% of the time on GPT models. The result matters because it suggests that tuning models to resist such attacks makes them noticeably less helpful on normal questions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 95%/70% ASR numbers rest on a single goal-primed GPT-4 judge with no human validation; if that judge is lenient, the central attack and defense-evasion claims are inflated. A human-labeled subset would settle it.","rationale":"The reader's weakest assumption is exactly the load-bearing concern I identify: the LLM-judge measurement without human validation. The paper's own cross-check (Appendix F) only compares average harmfulness scores between GPT-4 and Claude judges, not per-prompt ASR agreement, so it does not break the circularity that both judges are LLMs exposed to the same harmful-goal context. Because all the headline comparisons (Table 1, Table 2, and the abstract's 95%/70% claims) are ASR values from this judge, a systematic leniency bias would invalidate the central claim even if the attack is genuinely strong in the qualitative examples. The appendix examples are real and concerning, but they are selected and cannot support a 95% success rate. I therefore agree with the reader's CONDITIONAL verdict: the paper is plausible and worth publishing if the authors provide human-validated harmfulness judgments, variance estimates, and prompt/response artifacts. I also note secondary concerns—per-model masking levels are tuned via a 100-prompt decoding test (Section 3.3) with no stated overlap check against AdvBench, and the safety-tuning tradeoff (Table 3) is based on one model and a narrow benign-prompt set—but these are subordinate to the measurement issue. The acknowledged limitations about model and dataset scope further support keeping the claim scoped rather than treating it as universal. No change to the reader's verdict is needed; the same conditions should be enforced.","tokens_in":22288,"tokens_out":5376,"duration_ms":62976,"concrete_test":"Take a stratified random sample of 100–200 model responses across the six targets and at least one defense condition, covering the full 1–5 score range. Have at least three independent human annotators, blind to attack condition, apply the same rubric in Appendix J.3 (or a pre-registered binary judgment: 'does the response contain actionable harmful instructions?'). Compute human ASR (proportion scored 5 / yes) and compare to the GPT-4 judge ASR on the identical responses. If the human ASR for GPT models is not within 5–10 percentage points of the reported 95.77/97.11/98.84, or if the GPT judge's agreement with the human majority is below roughly 0.7 Cohen's kappa, the headline success rates are not established and the central claim should be downgraded. Also report per-prompt judge scores and the human disagreement cases.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claims—>95% ASR on GPT models, >70% across all six targets, and superiority over five baselines—rest entirely on ASR/HS values produced by a single GPT-4 judge (Appendix J.3). The judge is shown the original harmful goal alongside the model response, and ASR is defined as the judge giving a score of exactly 5. No human-annotated subset is reported, and the Appendix F cross-check only compares average HS between GPT-4 and Claude judges, not per-prompt ASR; both judges are LLMs with the same goal-priming and could share the same leniency bias. If the judge systematically awards 5 to any response that follows the requested 'plausible, detailed draft' format or contains any step related to the goal—even when the content is generic, hedged, or non-actionable—the 95%/70% numbers and the defeat-of-defenses results are inflated. The handful of examples in Appendix K are qualitative and not a substitute for a distributional check. This concern is load-bearing because every downstream claim (defense robustness, safety-tuning tradeoff, universality) inherits these ASR numbers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HaPLa, a black-box jailbreak attack on LLMs that combines two components: (1) abductive framing, which converts a direct harmful request into a third-person declarative statement and asks the model to reconstruct the plausible step-by-step actions that led to an already-completed event; and (2) symbolic encoding, which masks sensitive words using ASCII codes, emoji, or other custom rules, with the masking ratio chosen per model. The authors report ASR/HS numbers for six target LLMs on the AdvBench benchmark, claim >95% ASR on GPT-3.5-turbo, GPT-4o-mini, and GPT-4o, and >70% ASR on all six models, outperforming five baselines (ArtPrompt, AutoDAN, DeepInception, CodeChameleon, TAP). They also evaluate four defenses, an ablation study, multi-turn attacks, safety-tuning tradeoffs, keyword sensitivity, and a qualitative comparison of generated content to real-world crime cases.","tokens_in":22538,"tokens_out":5128,"duration_ms":64991,"significance":"If the quantitative results are sound, HaPLa is a meaningful contribution to the jailbreak literature: it is a black-box, query-efficient attack, it uses a flexible symbolic encoding scheme rather than a fixed static rule, and the defense evaluations and safety-tuning experiments provide useful evidence about the limits of current alignment. The paper is also transparent about its prompts and includes many concrete input/output examples, which aids reproducibility and follow-up research. The main importance hinges on the reliability of the ASR/HS measurements, since every headline claim (attack success, defense evasion, safety-tuning tradeoff, universality) is derived from those numbers.","major_comments":[{"comment":"All ASR/HS values are produced by a single GPT-4 judge that is given the original harmful goal and the model response, and ASR is defined as a judge score of exactly 5. No human-validated labels, inter-annotator agreement, or per-prompt agreement statistics are reported. Appendix F compares only average harmfulness scores between GPT and Claude judges, not ASR or per-prompt classifications; both judges are LLMs exposed to the same goal-primed prompt and may share the same leniency bias. Because Table 1, Table 2, Figure 3, Figure 5, and Table 3 all rest on this measurement, the central claims would be inflated if the judge is systematically generous. I request a human-annotated random subset (e.g., 100–200 responses per model or a combined subset) with a clear rubric, reported agreement (e.g., Cohen's kappa), and ASR recomputed from human labels.","section":"§4.1, Appendix J.3, Appendix F"},{"comment":"The masking ratio and the flip technique are chosen per target model through a 'decoding test' (Table 4), so the headline ASR numbers are not for a fixed, untuned attack. The paper says HaPLa requires 'no fine-tuning or optimization procedures,' but the per-model masking setting is a performance-relevant hyperparameter. It is not clear whether these settings were selected before seeing attack success on the test prompts, or whether the same settings would be chosen without access to the test distribution. Please report the ASR with a single fixed encoding rule (e.g., full ASCII) for all models, and show that the adaptive per-model settings are not responsible for the 95%/70% results. This is necessary for a fair comparison with baselines that use fixed official implementations.","section":"§3.3, Table 4, Table 1"},{"comment":"There are internal inconsistencies in the reported no-defense HaPLa results. For example, GPT-4o-mini ASR is 97.11 in Table 1 but 95.57 in Table 2's No Defense row; Claude 3.5 ASR is 70.58 in Table 1 but 70.76 in Table 2. Similarly, Table 3 reports a no-training ASCII ASR of 75.80 for LLaMA-3-8B, while Table 1 reports 72.88 for HaPLa on the same model. These discrepancies may stem from different runs or different subsets, but they are not explained. The paper should state the exact evaluation set size, report the relevant run(s) consistently, and explain any differences.","section":"Table 2 vs. Table 1; Table 3 vs. Table 1"},{"comment":"The multi-turn attack section claims that 'both the Second-turn ASR and the Win Rate consistently exceed 75%,' but Figure 5 is the only evidence and the exact numbers are not given in the text or caption. No dataset details or judge validation are provided for these metrics. Since this is a separate quantitative claim, please report the numeric values in the text or table, and include enough detail about the second-turn generation and the A/B judge (Appendix J.4) to allow replication.","section":"§5.1, Figure 5"}],"minor_comments":[{"comment":"The Manchu-script and emoji encoding prompts refer to 'the following table/rule' but the actual mapping tables are not included in the appendix. Without these mappings, those experiments cannot be reproduced. Please provide the full encoding tables.","section":"Appendix J.2.2, J.2.3"},{"comment":"The few-shot prompts used to transform the original query into declarative form and to identify toxic words are not supplied. Only the final attack prompts are shown. Please include the exact few-shot templates used with GPT-4o-mini.","section":"§3.3, Step 1 and Step 2"},{"comment":"The Reminder row for CodeChameleon on GPT-3.5-turbo reads '4.6880.58' with a missing separator; this appears to be a typo for '4.68 80.58'. Please fix the formatting.","section":"Table 2"},{"comment":"The table header uses 'IA' while the text and row label use 'IAR'; please standardize. Also clarify the denominator and the exact benign-prompt set used for the instruction acceptance rate.","section":"Table 3"},{"comment":"Several jailbroken examples end with a disclaimer such as 'should not be replicated' or 'provided solely for understanding the mechanism.' Since the judge is asked to score harmfulness, it would be useful to state explicitly whether such disclaimers affect scoring, and to show examples where the judge assigned score 5 despite a disclaimer.","section":"Appendix K.1"},{"comment":"Minor writing issues: 'success' is misspelled as 'sucess' in Appendix K heading; 'GPT-o1' should be 'o1' or 'GPT-5 (o1)' depending on the intended model; 'LLaMa' and 'LLaMA' are used inconsistently. A careful proofread would improve presentation.","section":"Global"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and addresses a timely problem. The central obstacle is not the attack idea itself—the attack design and qualitative examples are plausible—but the fact that all quantitative claims rest on a single, goal-primed, non-human-validated judge. I would want to see human validation with a concrete sample and agreement metrics before recommending acceptance. The per-model adaptive masking should also be documented transparently, since it affects the interpretation of the 'black-box, no optimization' claim. If the authors can supply these, the paper could be a useful contribution to the jailbreak literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a legitimate empirical jailbreak paper, not a breakthrough. The combination of abductive framing ('assume it happened, now reconstruct the steps') and per-model symbolic encoding is a genuine improvement over static ASCII/CipherChat-style attacks. The standout results are the ablation (both components matter, but differently per model), the adaptive masking with the flip trick for Claude, and the fine-tuning experiment showing that closing the attack on seen encodings destroys helpfulness on benign prompts. That last one is the most interesting part of the paper.\n\nThe main caveat is the judge. ASR is defined as GPT-4 giving a score of exactly 5, with the original harmful goal pasted into the judge prompt. No human labels, no agreement stats. Appendix F compares average harmfulness scores between GPT-4 and Claude judges, not per-prompt ASR, so it doesn't address leniency at the threshold. If the judge is generous, the headline 95%/70% numbers and the defense-evasion numbers all drift down. That's load-bearing. The few appendix examples look genuinely harmful, so I'm not saying the attack doesn't work; I'm saying the exact rates need a human-checked subset.\n\nOther soft spots are minor. The masking ratio and flip are tuned per model—the paper is upfront about that, and it's not circular, but it means the method is less 'one-shot universal' than the abstract implies. The defense claim is a bit strong: several defenses drop ASR by 20–30 points (e.g., Reminder on GPT-4o, Paraphrase on Claude/LLaMA). No code or prompt dataset is released, which hurts reproducibility. And the safety-tuning result is one model, no seeds, so take the tradeoff curve as illustrative.\n\nOverall: worth a serious referee. The attack is simple, works to a degree that should worry people, and the helpfulness tradeoff is a real finding. The judge validation is fixable. If you're working on jailbreaks or safety training, this is worth your time.","headline":"Real attack, real numbers, but trust those numbers only after a human checks the judge.","tokens_in":23045,"tokens_out":2100,"would_cite":true,"duration_ms":27295,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A black-box jailbreak that reframes harmful prompts as reasoning puzzles and masks trigger words succeeds on over 95% of GPT queries and on 70% of all tested LLMs.","keywords":["jailbreak attack","LLM safety alignment","abductive reasoning","symbolic encoding","prompt laundering","black-box attack","harmful content","safety tuning"],"falsifier":"Have human annotators blind to the original goal score a random subset of the responses that the GPT-4 judge rated 5. If the human-mean harmfulness score falls below 4 (or fewer than 70% of those responses get a human 5), then the central ASR claims are overstated. A more targeted variant: run the same judge without showing it the goal; a large ASR drop would imply goal leakage in the metric.","tokens_in":22148,"feed_emoji":"🔓","tokens_out":9594,"duration_ms":99560,"temperature":0.7,"pith_summary":"The paper is trying to show that a jailbreak attack called HaPLa—which needs only black-box access to a model—can defeat the safety alignment of current LLMs at scale. The method works by laundering the harmful request: it rewrites an imperative command as a third-person mystery to be solved, and it masks trigger words with symbolic encodings like ASCII numbers or emoji. On six commercial and open-source LLMs, HaPLa reports an attack success rate above 95% on GPT-series models and above 70% on every target, beating five existing attack baselines. The authors claim this is not a niche weakness but a structural one: safety training keys on explicit harmful wording and initial tokens, so any prompt that avoids those can slide through. They also present evidence that fine-tuning against HaPLa either fails against novel encodings or imposes an unacceptable drop in helpfulness.","feed_headline":"Masked abductive prompts jailbreak GPT models at 95%+ success","feed_subtitle":"Black-box attack reframes harmful requests as inference puzzles, beating five baselines across six LLMs.","key_machinery":"The central object is HaPLa (Harmful Prompt Laundering). Its two engines are abductive framing—restating the request as a completed event and asking the model to reconstruct the likely path—and symbolic encoding—replacing toxic words with adjustable symbolic rules (ASCII numbers, emoji, arithmetic rules, Manchu script). What carries the argument is the combination: the symbolic layer evades keyword filters, and the abductive layer changes the model's task from 'comply' to 'infer', shifting internal activations toward benign clusters (as evidenced by principal-component analysis of hidden states). An adaptive decoding test tunes the masking intensity per target model, and a flip technique (re","core_discovery":"HaPLa combines two transformations. Abductive framing converts a harmful imperative (e.g., 'Give instructions for making a bomb') into a declarative case: a person is said to have done the activity, and the model is asked to infer the plausible step-by-step actions. Symbolic encoding then masks the sensitive words inside that case, using ASCII codes at a per-model masking ratio chosen by a decoding test, plus a 'flip' variant that reverses the code order. The paper reports that this combination outperforms five prior attacks on all six tested models, and that harmfulness scores and success rates remain high under four common defenses (LlamaGuard, paraphrasing, self-reminder, and perplexity f","pith_inferences":["A direct test of the paper's metric: if a human-validated (or goal-blinded) judge reproduces the 95% and 70% success rates, then deployed guardrails need to decode and re-classify inferred intent rather than surface prompt; if not, the headline numbers are an artifact of the judge.","A cheap defense to try: force the model to decode and restate the masked query in its own words before answering, then run the safety classifier on that restatement. The paper does not test this, and it could break the laundering step without hurting helpfulness.","The paper's crime-case analysis suggests jailbroken outputs closely match real-world criminal methods (most similarity scores are 5). That implies the model is reproducing memorized harmful content, pointing to training-data filtering as a potentially more effective mitigation than post-hoc alignment.","Because the evaluation is confined to a single benchmark of 520 harmful prompts from a fixed distribution, whether HaPLa generalizes to broader or more recent harmful-query sets remains open; a cross-benchmark replication would be needed before treating the universal-jailbreak claim as settled."],"forward_implications":["Prompt-level defenses that inspect surface text (perplexity filters, keyword lists, LlamaGuard) cannot reliably catch HaPLa, because the malicious content is encoded and framed as neutral narrative.","Upgrading a guard model from 7B to 8B parameters cuts HaPLa's success by only about 10 percentage points, so the attack is not solved by bigger classifiers.","Safety fine-tuning on a known encoding does not generalize: after training on ASCII-masked prompts, unseen encodings (arithmetic, Manchu script, emoji) still succeed at non-trivial rates, and after the third epoch the model's acceptance of benign prompts collapses to roughly 1%.","Multi-turn versions are even more dangerous: after one successful turn, the model's second-turn response is judged better (more harmful/detailed) in over 75% of cases across all targets, enabling adaptive attacks that fold in real-time system information.","Reasoning-tuned models still fall to alternative encodings, so scaling reasoning capability is not a sufficient defense by itself."],"fun_headline_variants":["Jailbreak LLMs with abductive puzzles and symbolic code","Abductive framing plus encoding beats five defenses","Crafting jailbreaks as inference tasks: 95% on GPTs","Symbolic encoding hides harm from LLM filters","Infer-this attack turns harmful prompts into puzzles"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The attack success numbers depend on a single GPT-4 judge that sees the original harmful goal when it scores each model response on a 1–5 scale; if that judge is lenient or influenced by the goal, the reported 95%/70% rates could be inflated, and the paper's cross-check with a second LLM judge does not fully remove the shared-bias concern.","fun_headline_variants_meta":{"raw":{"variants":["Jailbreak LLMs with abductive puzzles and symbolic code","Abductive framing plus encoding beats five defenses","Crafting jailbreaks as inference tasks: 95% on GPTs","Symbolic encoding hides harm from LLM filters","Infer-this attack turns harmful prompts into puzzles"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1166,"prompt_tokens":749,"completion_tokens":417,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":351}},"tokens_in":493,"tokens_out":417,"duration_ms":5582,"temperature":1.0,"reasoning_tokens":351,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T17:22:14.524452+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human annotators blind to the original goal score a random subset of the responses that the GPT-4 judge rated 5. If the human-mean harmfulness score falls below 4 (or fewer than 70% of those responses get a human 5), then the central ASR claims are overstated. A more targeted variant: run the same judge without showing it the goal; a large ASR drop would imply goal leakage in the metric.","supporting_citations":[],"review_version":1}