Pith. sign in

REVIEW 4 major objections 3 minor 25 references

PUZZLED: Jailbreaking LLMs through Word-Based Puzzles

T0 review · 4 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By masking harmful keywords as word puzzles, PUZZLED gets state-of-the-art LLMs to comply 88.8% of the time.

desk verdict Plausible new jailbreak variant; the headline ASR numbers are unverifiable from this abstract because the evaluation protocol—judge, trials, dataset—is absent. read the letter →

arxiv 2508.01306 v1 pith:QQ4NGASH submitted 2025-08-02 cs.AI cs.CR

Yelim Ahn , Jaejin Lee This is my paper
classification cs.AIcs.CR
keywords LLMsafetyjailbreakattackswordpuzzlespromptengineeringadversarialpromptsreasoningGPT-4.1Claude3.7Sonnet
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PUZZLED tries to show that a jailbreak can be built from familiar word puzzles: mask the sensitive words in a harmful instruction, present the instruction as a word search, anagram, or crossword, and let the LLM's reasoning reconstruct it and then comply. The authors report the method succeeds on five state-of-the-art models, averaging an 88.8% attack success rate, with 96.5% on GPT-4.1 and 92.3% on Claude 3.7 Sonnet. If true, this matters because it means safety training can be sidestepped by a cognitive formatting trick rather than by semantic rewording, and it makes puzzle-style inputs a concrete threat surface.

What carries the argument

The load-bearing mechanism is the puzzle-masked instruction. A harmful prompt is transformed by masking selected keywords and embedding them in one of three puzzle formats: word search, anagram, or crossword. The model's task is split: first solve the puzzle to recover the masked words, then answer the reconstructed harmful instruction. The paper's argument is that this two-step, reasoning-based format separates the surface prompt from the harmful request, so safety training that recognizes explicit harmful wording never sees it in one piece.

What would settle it

Re-run the PUZZLED prompts on the same models and have two independent human annotators score each output as harmful, benign, or a refusal; if human-flagged harmful outputs fall well below 88.8%, the claim fails as an artifact of the automated judge.

Watch

Extended reading notes

Core claim

PUZZLED is a jailbreak method that takes a harmful instruction, removes the words that carry its harmful meaning, and presents the masked result as a word search, anagram, or crossword puzzle. The authors claim that because these puzzle formats are familiar and solvable, the LLM's reasoning completes the missing words and then produces the response the original instruction asked for. They report an average attack success rate of 88.8% across five state-of-the-art LLMs, with 96.5% on GPT-4.1 and 92.3% on Claude 3.7 Sonnet. The intended point is that puzzle-solving indirection is itself enough to bypass safety alignment, without iterative prompt engineering or semantic paraphrase.

Load-bearing premise

The whole result depends on the automatic evaluation that decides what counts as a successful attack being accurate, so if puzzle-solving outputs are labeled harmful when they are actually neutral puzzle answers, the reported 88.8% success rate is inflated.

Editorial extensions

If this is right

  • If PUZZLED's reported success rates hold, current safety training does not recognize harmful intent once the instruction is split into a puzzle-solving task and a reconstructed request.
  • Any LLM deployment that accepts word games or interactive puzzle inputs inherits the same attack surface, so filtering or restricting puzzle-encoded instruction formats becomes a necessary defense.
  • The numbers on GPT-4.1 and Claude 3.7 Sonnet imply stronger alignment alone does not eliminate this jailbreak class; defenses need to check for reconstructed harmful instructions before execution.
  • Safety evaluators should include puzzle-encoded versions of standard harm benchmarks, or they will underestimate real-world vulnerability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the reported ASR is likely judge-based; a human-annotation study of the same outputs could show whether puzzle-solving responses are genuinely harmful or merely misclassified, and that distinction is not reported in the abstract.
  • Editorial inference: the same masking-by-puzzle idea likely generalizes to non-textual puzzles, such as visual word searches, audio anagrams, or CAPTCHA-like reconstructions, so the attack class may be broader than the three text formats tested.
  • Editorial inference: one testable defense the paper does not explore is forcing the model to restate the user's underlying request before answering; this may reduce ASR by surfacing the harmful intent in plain language.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper introduces PUZZLED, a jailbreak attack that masks keywords in harmful instructions and presents them to LLMs as word puzzles (word search, anagram, crossword). The abstract claims that the model solves the puzzle, reconstructs the harmful instruction, and then complies, with an average attack success rate of 88.8% on five state-of-the-art LLMs, including 96.5% on GPT-4.1 and 92.3% on Claude 3.7 Sonnet. The manuscript as provided consists of the abstract only; no details of the puzzle construction, the evaluation dataset, the success criterion, the judge, or the baseline are included.

Significance. If the reported results are substantiated, the work is significant for AI safety research: it identifies a new family of jailbreak attacks that exploit LLMs' puzzle-solving abilities, which is both novel and practically relevant for understanding safety boundaries. The abstract's numbers are striking but currently unverifiable because the evaluation methodology is entirely absent. The paper does not appear to involve circular reasoning, since the ASR is measured, not derived from the target models; the central concern is the lack of evidence underpinning the central quantitative claim.

major comments (4)
  1. [Abstract] The headline ASR values (88.8%, 96.5%, 92.3%) are uninterpretable without a precise definition of 'attack success.' Please specify the judge (human or automated classifier), the scoring rule, and whether mere puzzle-solving behavior, such as listing the masked words without producing any harmful content, counts as success. If the judge is a permissive string matcher or an unreliable classifier, the reported ASR may be inflated and would not measure actual safety bypasses.
  2. [Abstract] The evaluation setup is not described: what harmful-prompt dataset was used, how many prompts, how many independent runs per model, what decoding settings (temperature, top-p) were used, and what system prompt or safety guardrails were active. Without these details, the reported numbers cannot be reproduced or compared with existing jailbreak benchmarks, so the central claim of high ASR is not auditable.
  3. [Abstract] No baseline or control condition is reported. To support the claim that the puzzle mechanism is effective, the authors should compare PUZZLED against direct (unmodified) harmful instructions and against existing jailbreak methods under identical evaluation conditions. Without such baselines, the high ASR may reflect a general vulnerability of the models rather than a property of the puzzle transformation.
  4. [Entire submission] The manuscript as provided contains only the abstract; the main text, figures, tables, and appendices are missing. Because the central claim is an empirical result, the absence of the full methodology and evaluation section is a load-bearing gap. The authors must provide a complete paper that includes a detailed method description and a reproducible evaluation protocol before the work can be assessed.
minor comments (3)
  1. [Abstract] The abbreviation 'ASR' is used without being defined; please spell out 'attack success rate' on first use, in both the abstract and the main text.
  2. [Abstract] The characterization of the puzzles as 'cognitively demanding for LLMs' is an empirical claim that is not substantiated; either cite relevant studies or report task difficulty measurements.
  3. [Abstract] The abstract does not mention any ethical considerations or responsible-disclosure statement for the jailbreak method; many venues require such a statement in the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PUZZLED is an empirical attack study whose reported ASR is a direct measurement, not a derived quantity.

full rationale

The paper reports an empirical evaluation of a jailbreak method. There is no mathematical derivation, no fitted parameter later called a prediction, no self-citation, and no uniqueness argument. The central claim (an average ASR of 88.8%, with 96.5% on GPT-4.1 and 92.3% on Claude 3.7 Sonnet) is a measurement of target LLM behavior under a defined attack protocol, not a consequence of the method by construction. The abstract omits details of the harmfulness judge and trial counts, which is a reproducibility concern, not circularity. Under the hard rules, circularity requires quoting a specific reduction; none is present. Therefore the paper is self-contained as an empirical study and the score is 0.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

No free parameters or invented entities are mentioned in the abstract. The main assumptions are about the reliability of the evaluation and the standard safety configuration of the tested models.

assumptions (2)
  • domain assumption The evaluation judge used to label responses as harmful is reliable and matches standard jailbreak benchmarks.
    The high ASR is only meaningful if the harmfulness labels are accurate; the abstract does not identify the judge.
  • domain assumption The target models are evaluated in their default safety postures without additional per-query filtering.
    If an external guardrail blocked puzzle prompts, the ASR would drop; the abstract does not specify deployment context.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PUZZLED: Jailbreaking LLMs through Word-Based Puzzles." pith.science (2026). https://pith.science/paper/QQ4NGASH

@misc{pith2026250801306,
  author       = {Pith},
  title        = {Pith review of: PUZZLED: Jailbreaking LLMs through Word-Based Puzzles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QQ4NGASH}},
  note         = {Machine review of arXiv:2508.01306}
}
read the original abstract

As large language models (LLMs) are increasingly deployed across diverse domains, ensuring their safety has become a critical concern. In response, studies on jailbreak attacks have been actively growing. Existing approaches typically rely on iterative prompt engineering or semantic transformations of harmful instructions to evade detection. In this work, we introduce PUZZLED, a novel jailbreak method that leverages the LLM's reasoning capabilities. It masks keywords in a harmful instruction and presents them as word puzzles for the LLM to solve. We design three puzzle types-word search, anagram, and crossword-that are familiar to humans but cognitively demanding for LLMs. The model must solve the puzzle to uncover the masked words and then proceed to generate responses to the reconstructed harmful instruction. We evaluate PUZZLED on five state-of-the-art LLMs and observe a high average attack success rate (ASR) of 88.8%, specifically 96.5% on GPT-4.1 and 92.3% on Claude 3.7 Sonnet. PUZZLED is a simple yet powerful attack that transforms familiar puzzles into an effective jailbreak strategy by harnessing LLMs' reasoning capabilities.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

25 extracted references · 4 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al

    Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    Anthropic. 2024. Claude 3 Model Card. Accessed: 2025-06-16

    work page 2024

  5. [5]

    J.; Tramer, F.; et al

    Chao, P.; Debenedetti, E.; Robey, A.; Andriushchenko, M.; Croce, F.; Sehwag, V.; Dobriban, E.; Flammarion, N.; Pappas, G. J.; Tramer, F.; et al. 2024. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. Advances in Neural Information Processing Systems, 37: 55005--55029

    2024

  6. [6]

    Ding, P.; Kuang, J.; Ma, D.; Cao, X.; Xian, Y.; Chen, J.; and Huang, S. 2023. A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily. arXiv preprint arXiv:2311.08268

    arXiv 2023

  7. [7]

    Dong, X.; Hu, W.; Xu, W.; and He, T. 2024. SATA: A Paradigm for LLM Jailbreak via Simple Assistive Task Linkage. arXiv preprint arXiv:2412.15289

    work page arXiv 2024

  8. [8]

    Grattafiori, A.; Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Vaughan, A.; et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

Show all 25 references
  1. [9]

    Helbling, A.; Phute, M.; Hull, M.; and Chau, D. H. 2023. Llm self defense: By self examination, llms know they are being tricked. arXiv e-prints, arXiv--2308

    2023

  2. [10]

    Honnibal, M.; and Montani, I. 2017. spaCy 2 : Natural language understanding with B loom embeddings, convolutional neural networks and incremental parsing. To appear

    2017

  3. [11]

    Jain, N.; Schwarzschild, A.; Wen, Y.; Somepalli, G.; Kirchenbauer, J.; Chiang, P.-y.; Goldblum, M.; Saha, A.; Geiping, J.; and Goldstein, T. 2023. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614

    2023 arXiv

  4. [12]

    Jiang, F.; Xu, Z.; Niu, L.; Xiang, Z.; Ramasubramanian, B.; Li, B.; and Poovendran, R. 2024. Artprompt: Ascii art-based jailbreak attacks against aligned llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1...

    2024

  5. [13]

    Li, X.; Zhou, Z.; Zhu, J.; Yao, J.; Liu, T.; and Han, B. 2023 a . Deepinception: Hypnotize large language model to be jailbreaker. arXiv preprint arXiv:2311.03191

    2023 arXiv

  6. [14]

    Li, Y.; Wei, F.; Zhao, J.; Zhang, C.; and Zhang, H. 2023 b . Rain: Your language models can align themselves without finetuning. arXiv preprint arXiv:2309.07124

    2023 arXiv

  7. [15]

    Liu, X.; Xu, N.; Chen, M.; and Xiao, C. 2023. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451

    2023 arXiv

  8. [16]

    Liu, Y.; He, X.; Xiong, M.; Fu, J.; Deng, S.; and Hooi, B. 2024. Flipattack: Jailbreak llms via flipping. arXiv preprint arXiv:2410.02832

    2024 arXiv

  9. [17]

    Lv, H.; Wang, X.; Zhang, Y.; Huang, C.; Dou, S.; Ye, J.; Gui, T.; Zhang, Q.; and Huang, X. 2024. Codechameleon: Personalized encryption framework for jailbreaking large language models. arXiv preprint arXiv:2402.16717

    2024 arXiv

  10. [18]

    Robey, A.; Wong, E.; Hassani, H.; and Pappas, G. J. 2023. Smoothllm: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684

    2023 arXiv

  11. [19]

    M.; Hauth, A.; Millican, K.; et al

    Team, G.; Anil, R.; Borgeaud, S.; Alayrac, J.-B.; Yu, J.; Soricut, R.; Schalkwyk, J.; Dai, A. M.; Hauth, A.; Millican, K.; et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  12. [20]

    Wei, Z.; Wang, Y.; Li, A.; Mo, Y.; and Wang, Y. 2023. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv:2310.06387

    2023 arXiv

  13. [21]

    Y.; and Poovendran, R

    Xu, Z.; Jiang, F.; Niu, L.; Jia, J.; Lin, B. Y.; and Poovendran, R. 2024. Safedecoding: Defending against jailbreak attacks via safety-aware decoding. arXiv preprint arXiv:2402.08983

    2024 arXiv

  14. [22]

    Yuan, Y.; Jiao, W.; Wang, W.; Huang, J.-t.; He, P.; Shi, S.; and Tu, Z. 2023. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher. arXiv preprint arXiv:2308.06463

    2023 arXiv

  15. [23]

    Zhao, W.; Li, Z.; Li, Y.; Zhang, Y.; and Sun, J. 2024. Defending large language models against jailbreak attacks via layer-specific editing. arXiv preprint arXiv:2405.18166

    2024 arXiv

  16. [24]

    Zhou, W.; Wang, X.; Xiong, L.; Xia, H.; Gu, Y.; Chai, M.; Zhu, F.; Huang, C.; Dou, S.; Xi, Z.; et al. 2024. Easyjailbreak: A unified framework for jailbreaking large language models. arXiv preprint arXiv:2403.12171

    2024 arXiv

  17. [25]

    Z.; and Fredrikson, M

    Zou, A.; Wang, Z.; Carlini, N.; Nasr, M.; Kolter, J. Z.; and Fredrikson, M. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.