REVIEW 4 major objections 4 minor 3 cited by
The TIP of the Iceberg: Revealing a Hidden Class of Task-in-Prompt Adversarial Attacks on LLMs
T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Task-in-Prompt attacks embed forbidden requests inside benign transformation tasks and circumvent safety safeguards in all six tested LLMs, including GPT-4o and LLaMA 3.2.
desk verdict A useful benchmark and a plausible reframing of ArtPrompt as a broad class, but the headline ASRs rest on a lightly validated LLM judge and best-of-grid selection. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the TIP prompt template $x^* = x_{\text{task}} + E(u)$, where $E$ is an encoding function (Caesar, Morse, Base64, riddles, Python code, etc.) that maps an unsafe prompt $u$ into a seemingly benign string, and $x_{\text{task}}$ instructs the model to process that string. The attack relies on implicit decoding: the model is not asked to output the decoded word step by step; instead it reconstructs the meaning through attention and context and uses it in a downstream request. Difficulty tiers modulate how transparent the encoding is, which determines whether a model can solve it and whether its safeguards register the danger.
What would settle it
A human expert panel independently labels a random, condition-stratified sample of target-model outputs and compares its labels with the LLaMA-3.1-70B judge; if human agreement falls well below 92 percent or systematic disagreement appears in any condition, the reported attack success rates are unreliable.
Extended reading notes
Core claim
The central claim is that encoding unsafe content with an arbitrary sequence-to-sequence transformation and embedding that transformation as an instruction in the prompt is sufficient to make aligned LLMs produce prohibited output with high probability. The paper demonstrates this across six state-of-the-art models, including GPT-4o and LLaMA 3.2, with per-task attack success rates reaching 1.0. The mechanism, according to the paper, is that the model performs the decoding internally and implicitly, integrating the reconstructed unsafe meaning into its response without an explicit chain of thought, thereby bypassing safety filters that are triggered by direct forbidden wording. The paper frames TIP as a general class of vulnerabilities of which the prior ArtPrompt ASCII-art attack is one instance, and shows in JailbreakBench comparisons that TIP outperforms several established jailbreaks on the newest models.
Load-bearing premise
The reported attack success rates all depend on an automated judge model, LLaMA-3.1-70B, whose classifications were checked on only 100 sampled responses with 92 percent accuracy; if that judge is biased, the central claim about universal vulnerability is not supported.
Editorial extensions
If this is right
- Any safety alignment that relies on detecting trigger words, known attack patterns, or prompt-format rules will miss TIP prompts, because the harmful term never appears in the input.
- The attack transfers across the encoding space: at least one of the ten tested transformations succeeded on every tested model, so the vulnerability is not specific to one cipher or format.
- Defenses and red-teaming must assume that task-solving capability itself is an attack surface; models that get better at decryption, riddles, and code will become more, not less, vulnerable.
- Depersonalisation, attributing the request to a third party, raises success rates for most models but is not necessary; TIP works without it on all models.
- Existing filter and classifier defenses, including Llama Guard 3 and keyword filtering, have near-zero detection rates for several TIP encodings.
Reading between the lines
- If TIP is as general as claimed, then model capability increases and safety tuning may be in a direct arms race: any benchmark that measures task-solving skill is, indirectly, a measure of the surface available for this attack class.
- The PHRYGE benchmark could be extended to multimodal encodings, such as images or audio steganography, and to retrieval-augmented systems; the paper lists these as future work but does not test them.
- A natural follow-up experiment is to test whether fine-tuning on explicit decoding tasks amplifies TIP success, since the model's task-solving ability is the attack's engine.
- Because the ASR numbers rest on an LLM judge, they should be treated as upper bounds until a human-rated subset across all conditions confirms them.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Task-in-Prompt (TIP) attacks, a proposed class of jailbreak in which an unsafe concept is encoded via a sequence-to-sequence task (cipher, riddle, code, etc.) embedded in the prompt, and the model is asked to perform the task rather than directly produce the prohibited content. The authors formalize the attack, introduce the PHRYGE benchmark (10 encodings x 4 objectives x 3 difficulty levels), evaluate six LLMs, report attack success rates (ASR), compare against JailbreakBench attacks, and test several defenses. They report that all six models are vulnerable, with best ASR reaching 1.0 on several tasks, and that TIP attacks outperform prior jailbreaks on GPT-4o and LLaMA 3.2. The central quantitative claim rests on an LLM-as-judge protocol (LLaMA-3.1-70B) validated on only 100 manually inspected samples at 92% accuracy.
Significance. If the empirical claims hold, the paper is significant: it identifies a general attack class rather than a single encoding trick, provides a reusable benchmark, and makes falsifiable predictions about the vulnerability of instruction-following models. The formal definition in Section 2.1 is clear, the PHRYGE benchmark is a useful artifact, and the paper includes sanity checks, a comparison with established attacks on JailbreakBench, and reproducible details such as GPU hours and sample prompts. The main weakness is that the headline ASR numbers are only as credible as the automated judge, and the current validation is too thin to support per-model rankings and best-cell maxima.
major comments (4)
- [Section 4.3] The only validation of the automated judge is manual inspection of 100 randomly selected samples, reported as 92% accuracy. This is insufficient to support the headline ASR values in Tables 1 and 2, because Table 1 selects the best cell from the 10 encodings x 3 difficulty levels x 2 depersonalisation settings per objective. A judge false-positive rate concentrated in a few cells would directly inflate the reported maxima. Please provide per-task and per-model confusion matrices, human agreement on the selected high-ASR cells, and confidence intervals on the ASR estimates.
- [Sections 4 and 4.3] The judge is LLaMA-3.1-70B, and LLaMA-3.1-70B-Instruct is one of the six attacked models. Using the same model as both victim and measurement instrument creates a systematic-bias risk: a judge that over-applies its instruction to flag 'detailed procedural information, even if disguised as hypothetical, indirect, or within a role-playing context' may mark refusals or generic warnings as successful attacks, and this bias is inseparable from the model's own safety behavior. Please use a judge that is not among the attacked models, or at minimum report per-cell agreement between the judge and human labels for each victim model.
- [Section 5.1, Table 1] ASR values such as 0.94, 0.97, and 1.00 are reported without sample sizes, confidence intervals, or any measure of run-to-run variability; only the sanity check in Section 4.1 states 100 runs per query. Without the number of runs per PHRYGE cell, differences such as GPT-4o's 0.94 vs 0.79 in the same column are not statistically interpretable, and the 'best attack' selection in Table 1 has no grounding. Please state how many runs were executed per cell and report uncertainty for every ASR estimate, including the JailbreakBench comparisons in Table 2.
- [Table 3 and Section 6.5] The 'GPT-4o*' column defines detection rate as 1 minus ASR, which is a refusal rate of the target model, not the performance of a separate detection system. This conflates the target model's alignment with the effectiveness of an actual defense and weakens the claim in Section 6.5 that TIP attacks bypass modern detection models, at least for that column. Please either evaluate a real guard model (e.g., an available content-moderation API) or remove this column and qualify the defense comparison accordingly.
minor comments (4)
- [Table 2] The header 'Llama3' appears in Table 2, but Section 4 lists four LLaMA variants; please clarify whether this is LLaMA-3.1-70B, LLaMA-3.2-3B, or another variant, and use consistent naming across tables.
- [Figure 2] The red shading is described only qualitatively as 'darker red shades indicate higher ASR'; please add a numeric color bar or label values so the heatmaps are quantitatively readable.
- [Sections 3 and 4.5] The benchmark is described as 10 encodings x 4 objectives x 3 difficulty levels generating 120 prompts, while Section 4.5 says scenarios were tested with and without depersonalisation; please clarify whether the 120 count includes depersonalisation variants, and if not, state the total number of prompt configurations actually evaluated.
- [Section 8] The Limitations section already acknowledges that the automated evaluation function 'may introduce biases or fail to capture nuanced instances of unsafe content generation'; the main text in Section 4.3 should not soften this caveat, and the experimental section should operationalize the needed validation rather than leaving it as a deferred limitation.
Circularity Check
No circularity: TIP attack claims are empirical, benchmarked externally, and the only self-citation is non-load-bearing.
full rationale
The paper's central claim—that Task-in-Prompt attacks circumvent safeguards across six LLMs—is an empirical result, not a derivation from a fitted parameter or a definitional identity. The formal definition in Section 2.1 merely formalizes what a TIP attack is and when it succeeds; it does not assert that such attacks exist. The PHRYGE benchmark is a new evaluation harness, and the reported ASRs are measurements, not predictions obtained from a fitted model. The selection of each model's best-performing prompt in Table 1 is an explicit best-of-grid reporting convention common in jailbreak benchmarks, not a fitted input disguised as a prediction; the averaged ASR results in Figure 2 and Appendix B provide independent support independent of that selection. The comparison against JailbreakBench uses the external JailbreakBench suite and external baselines (TAP, DAN, PTA, ArtPrompt), so the headline effectiveness claims are not reducible to the paper's own construction. The only self-citation is Berezin et al. (2024), used in Section 1.3 to argue that ArtPrompt's success is not due to ASCII-art-specific spatial reasoning; this point is corroborated by the paper's own data showing high ASR for non-ASCII encodings such as riddles, phonetic encoding, and Python code, so the citation is not load-bearing. The LLM-as-judge validation concern (Section 4.3, 92% accuracy on 100 samples) is a measurement-quality limitation, and the paper itself acknowledges this in Section 8; it is not circularity. No uniqueness theorem is imported, no ansatz is smuggled via citation, and no known result is merely renamed in a way that drives the conclusions. The derivation chain is self-contained and empirically grounded.
Assumptions & free parameters
assumptions (4)
- domain assumption Known unsafe prompts have negligible probability of producing unsafe output under normal conditions, PM(y in S | u) < epsilon.
- domain assumption The LLM-as-judge, LLaMA-3.1-70B, correctly maps model outputs to safe or unsafe labels with negligible error.
- domain assumption The hand-chosen PHRYGE tasks, difficulty levels, and depersonalisation variants are representative of real attacker choices.
- domain assumption Models can reconstruct encoded content internally without being asked to output the decoded word.
Cite this review
Pith. "Pith review of The TIP of the Iceberg: Revealing a Hidden Class of Task-in-Prompt Adversarial Attacks on LLMs." pith.science (2026). https://pith.science/paper/QWAGRCG4
@misc{pith2026250118626,
author = {Pith},
title = {Pith review of: The TIP of the Iceberg: Revealing a Hidden Class of Task-in-Prompt Adversarial Attacks on LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/QWAGRCG4}},
note = {Machine review of arXiv:2501.18626}
}
read the original abstract
We present a novel class of jailbreak adversarial attacks on LLMs, termed Task-in-Prompt (TIP) attacks. Our approach embeds sequence-to-sequence tasks (e.g., cipher decoding, riddles, code execution) into the model's prompt to indirectly generate prohibited inputs. To systematically assess the effectiveness of these attacks, we introduce the PHRYGE benchmark. We demonstrate that our techniques successfully circumvent safeguards in six state-of-the-art language models, including GPT-4o and LLaMA 3.2. Our findings highlight critical weaknesses in current LLM safety alignments and underscore the urgent need for more sophisticated defence strategies. Warning: this paper contains examples of unethical inquiries used solely for research purposes.
Figures
Forward citations
Cited by 3 Pith papers
-
LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems
A systematic review that categorizes LLM threats, severity scores, and mitigations across development and operation life cycles and multiple deployment scenarios.
-
POT: Inducing Overthinking in LLMs via Black-Box Iterative Optimization
POT automatically finds natural fluent prompt phrases, via iterative LLM optimization, that inflate reasoning token counts in frontier LLMs by up to 8x while keeping answers correct.
-
TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law
Trident-Bench provides 2,652 professionally validated harmful prompts across finance, law, and medicine, and shows that domain-specialized LLMs often comply with unethical requests more than generalist models.
Reference graph
Works this paper leans on
-
[1]
Maksym Andriushchenko and Nicolas Flammarion. 2024. https://arxiv.org/abs/2407.11969 Does refusal training in llms generalize to the past tense? Preprint, arXiv:2407.11969
arXiv 2024
-
[2]
Sergey Berezin, Reza Farahbakhsh, and Noel Crespi. 2024. https://arxiv.org/abs/2409.18708 Read over the lines: Attacking llms and toxicity detection systems with ascii art to mask profanity . Preprint, arXiv:2409.18708
arXiv 2024
-
[3]
Pappas, Florian Tramer, Hamed Hassani, and Eric Wong
Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramer, Hamed Hassani, and Eric Wong. 2024. https://arxiv.org/abs/2404.01318 Jailbreakbench: An open robustness benchmark for jailbreaking large language models . Preprint, arXiv:2404.01318
arXiv 2024
-
[4]
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. 2023. https://arxiv.org/abs/2310.12773 Safe rlhf: Safe reinforcement learning from human feedback . Preprint, arXiv:2310.12773
arXiv 2023
-
[5]
Yi Dong, Ronghui Mu, Gaojie Jin, Yi Qi, Jinwei Hu, Xingyu Zhao, Jie Meng, Wenjie Ruan, and Xiaowei Huang. 2024 a . https://arxiv.org/abs/2402.01822 Building guardrails for large language models . Preprint, arXiv:2402.01822
arXiv 2024
-
[6]
Yi Dong, Ronghui Mu, Yanghao Zhang, Siqi Sun, Tianle Zhang, Changshun Wu, Gaojie Jin, Yi Qi, Jinwei Hu, Jie Meng, Saddek Bensalem, and Xiaowei Huang. 2024 b . https://arxiv.org/abs/2406.02622 Safeguarding large language models: A survey . Preprint, arXiv:2406.02622
arXiv 2024
-
[7]
Europol. 2023. https://www.europol.europa.eu/publications-events/publications/chatgpt-impact-of-large-language-models-law-enforcement Chatgpt – the impact of large language models on law enforcement . Accessed: 2024-12-12
work page 2023
-
[8]
Gemma Team . 2024. https://arxiv.org/abs/2408.00118 Gemma 2: Improving open language models at a practical size . Preprint, arXiv:2408.00118
arXiv 2024
Show all 31 references
-
[9]
Fengqing Jiang, Zhangchen Xu, Luyao Niu, Zhen Xiang, Bhaskar Ramasubramanian, Bo Li, and Radha Poovendran. 2024. https://aclanthology.org/2024.acl-long.809 A rt P rompt: ASCII art-based jailbreak attacks against aligned LLM s . In Proceedings of the 62nd Annual Meeting of the ...
2024
-
[10]
Yige Li, Hanxun Huang, Yunhan Zhao, Xingjun Ma, and Jun Sun. 2024. https://arxiv.org/abs/2408.12798 Backdoorllm: A comprehensive benchmark for backdoor attacks on large language models . Preprint, arXiv:2408.12798
2024 arXiv
-
[11]
Leon Lin, Hannah Brown, Kenji Kawaguchi, and Michael Shieh. 2024. https://arxiv.org/abs/2407.03232 Single character perturbations break llm alignment . Preprint, arXiv:2407.03232
2024 arXiv
-
[12]
AI @ Meta Llama Team. 2024. https://llama.meta.com/ The llama 3 herd of models . A detailed contributor list can be found in the appendix of this paper
2024
-
[13]
Marah Abdin et al. 2024. https://arxiv.org/abs/2404.14219 Phi-3 technical report: A highly capable language model locally on your phone . Preprint, arXiv:2404.14219
2024 arXiv
-
[14]
Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. 2024. https://arxiv.org/abs/2312.02119 Tree of attacks: Jailbreaking black-box llms automatically . Preprint, arXiv:2312.02119
2024 arXiv
-
[15]
Meta AI . 2024. Llama 3.2: Advancing vision and edge ai for mobile devices. https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/. Accessed: 2024-12-11
2024
-
[16]
Mistral AI . 2024. https://mistral.ai/news/mistral-nemo/ Mistral-nemo: A new generation of open foundation models . Accessed: 2024-12-14
2024
-
[17]
OpenAI. 2023. https://cdn.openai.com/rule-based-rewards-for-language-model-safety.pdf Rule-based rewards for language model safety . OpenAI
2023
-
[18]
OpenAI. 2024 a . https://cdn.openai.com/gpt-4o-system-card.pdf Gpt-4o system card . Technical report, OpenAI. Accessed: 2024-04-27
2024
-
[19]
OpenAI. 2024 b . Gpt-4o system card. https://cdn.openai.com/gpt-4o-system-card.pdf. Accessed: 2024-12-11
2024
-
[20]
Damian Ruck and Matthew Sutton. 2024. https://cetas.turing.ac.uk/publications/indirect-prompt-injection-generative-ais-greatest-security-flaw Indirect prompt injection: Generative ai’s greatest security flaw . CETaS Expert Analysis
2024
-
[21]
Erfan Shayegani, Md Abdullah Al Mamun, Yu Fu, Pedram Zaree, Yue Dong, and Nael Abu-Ghazaleh. 2023. https://arxiv.org/abs/2310.10844 Survey of vulnerabilities in large language models revealed by adversarial attacks . Preprint, arXiv:2310.10844
2023 arXiv
-
[22]
do anything now
Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. https://doi.org/10.1145/3658644.3670388 "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models . In Proceedings of the 2024 on ACM SIGSAC Conference on Co...
2024
-
[23]
unslothai . 2024. https://github.com/unslothai unslothai: Thai natural language processing library . Accessed: 2024-09-16
2024
-
[24]
Neeraj Varshney, Pavel Dolin, Agastya Seth, and Chitta Baral. 2023. The art of defending: A systematic evaluation and analysis of llm defense strategies on safety and over-defensiveness. arXiv preprint arXiv:2401.00287
2023 arXiv
-
[25]
Shengye Wan, Cyrus Nikolaidis, Daniel Song, David Molnar, James Crnkovich, Jayson Grace, Manish Bhatt, Sahana Chennabasappa, Spencer Whitman, Stephanie Ding, Vlad Ionescu, Yue Li, and Joshua Saxe. 2024. https://arxiv.org/abs/2408.01605 Cyberseceval 3: Advancing the evaluation ...
2024 arXiv
-
[26]
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. https://arxiv.org/abs/2307.02483 Jailbroken: How does llm safety training fail? Preprint, arXiv:2307.02483
2023 arXiv
-
[27]
Zhen Xiang, Fengqing Jiang, Zidi Xiong, Bhaskar Ramasubramanian, Radha Poovendran, and Bo Li. 2024. https://openreview.net/forum?id=S4cYxINzjp Badchain: Backdoor chain-of-thought prompting for large language models . In NeurIPS 2023 Workshop on Backdoors in Deep Learning - The...
2024
-
[28]
Xilie Xu, Keyi Kong, Ning Liu, Lizhen Cui, Di Wang, Jingfeng Zhang, and Mohan Kankanhalli. 2023. https://arxiv.org/abs/2310.13345 An llm can fool itself: A prompt-based adversarial attack . Preprint, arXiv:2310.13345
2023 arXiv
-
[29]
Jingwei Yi, Yueqi Xie, Bin Zhu, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. 2024. https://arxiv.org/abs/2312.14197 Benchmarking and defending against indirect prompt injection attacks on large language models . Preprint, arXiv:2312.14197
2024 arXiv
-
[30]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[31]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.