Should LLM Safety Be More Than Refusing Harmful Instructions?

Mark Dras; Usman Naseem; Utsav Maskey

REVIEW 5 major objections 5 minor 48 references

Should LLM Safety Be More Than Refusing Harmful Instructions?

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Encrypted prompts make aligned LLMs fail on at least one safety dimension.

desk verdict Useful two-dimensional evaluation framework for encrypted-input safety, but the headline claim rests on unmatched text sets and a safety signal that is never directly measured. read the letter →

arxiv 2506.02442 v2 pith:GF5IUTBP submitted 2025-06-03 cs.CL

Utsav Maskey , Mark Dras , Usman Naseem This is my paper

classification cs.CL

keywords LLMsafetyjailbreakingmismatchedgeneralizationciphertextcryptanalysisover-refusalgenerationdefensemechanisms

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that refusing harmful instructions is only half of LLM safety: when inputs arrive in long-tail formats such as encrypted text, safety splits into two separable demands, instruction refusal and generation safety. It builds an encrypted benchmark from nine ciphers and measures how well five LLMs decrypt harmful versus benign texts. The central result is that models with measurable decryption ability on the easy classical ciphers (Caesar, Atbash, Morse) exhibit mismatched generalization: on encrypted harmful inputs their safety mechanisms fail on at least one dimension, producing either unsafe responses or over-refusal of benign decryption requests. The paper then tests six defenses and concludes that none keeps both safety dimensions intact while preserving ordinary decryption utility.

What carries the argument

The central object is a two-dimensional safety-evaluation framework built on decryption-performance gaps. For each cipher, plaintext pairs $(x, x')$ with $x' = e(x)$, and the model's decryption $\hat{x} = d_M(x')$ is scored by exact match, BLEU, and normalized Levenshtein. The argument works through two deltas: $\Delta_{IR}$ = decryption performance on benign texts minus performance on encrypted harmful instructions, which measures instruction-refusal over-alignment, and $\Delta_{resp}$ = benign minus encrypted harmful-response performance, which measures generation suppression. The decisive mechanism is the comparison of these deltas: a large positive $\Delta_{IR}$ means safety suppresses decryption of harmful instructions (over-refusal), a large positive $\Delta_{resp}$ means safety suppresses harmful response generation, and the paper claims that no tested model or defense achieves both while keeping utility loss $\Delta_{utility}$ low.

What would settle it

A matched-pair experiment using the same cipher would settle it: encrypt harmful prompts and benign prompts of equal length, similar vocabulary, and similar ciphertext perplexity, then measure the decryption gaps; if matching the texts on difficulty makes the gaps vanish, the safety interpretation is not supported.

Watch

Extended reading notes

Core claim

The paper's central claim is that an LLM with measurable decryption ability for a cipher seen in pretraining is vulnerable to mismatched-generalization attacks, meaning safety behavior learned on normal text does not transfer to ciphertext inputs. Its two-dimensional measurement separates the failure modes: dimension one is instruction refusal (the model should decline to decrypt and act on a harmful instruction, but often either complies or over-refuses), and dimension two is generation safety (the model should suppress a harmful response even when the input itself is benign). The measurements show that aligned models do not maintain both dimensions at once on encrypted content: some models suppress harmful instructions more than harmful responses, others show the reverse, and every defense evaluated sacrifices at least one side. The paper therefore claims that safety for long-tail text is not achievable by a single refusal gate and needs coordinated pre- and post-generation mechanisms.

Load-bearing premise

The argument relies on the assumption that harmful and benign encrypted texts are equally hard to decrypt, so that a decryption gap between them reflects safety behavior rather than text difficulty.

Editorial extensions

If this is right

Safety claims about a model must be scoped to input format: a model that refuses ordinary harmful prompts can still fail on encrypted or otherwise long-tail versions of the same prompts.
Decryption quality on benign ciphertexts is not enough to certify safety; the meaningful quantity is the gap between decryption quality on benign and harmful texts.
Exact-match suppression being larger than BLEU-based suppression implies that current alignment is better at preventing verbatim reproductions than at preventing partial or loosely rephrased harmful content.
Pre-LLM filters and post-LLM response checks pull in opposite directions, so a workable defense must combine an input-side detector that understands ciphertext semantics with an output-side gate that suppresses harmful responses without rejecting the decryption task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the two-dimensional pattern is a general property of mismatched generalization, the same lopsided safety should appear for other formats the model can process, such as ASCII art, leetspeak, or constructed languages; that is testable with the paper's delta methodology.
A practical audit could score a model's safety profile by the ratio $\Delta_{resp} / \Delta_{IR}$ on one easy cipher, flagging models that suppress one dimension far more than the other.
The larger exact-match drop suggests a specific mechanism worth testing directly: safety training may be suppressing token sequences that match harmful strings rather than the semantics of harm, which would predict that paraphrased or lightly mistranscribed harmful responses survive at higher rates.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the D1/D2 split is a real addition to cipher-jailbreak evaluation, and the defense comparison is worth having. But the headline claim—safety mechanisms fail on at least one dimension—is not yet established by the measurements. The benign and harmful text sets are not matched, and the paper never reports the refusal/unsafety rate it defines.

The paper does some things well. The two-dimensional framing (instruction refusal vs. generation suppression) measured through decryption accuracy is genuinely new relative to prior cipher-attack work, which focused on attack success. Decryption accuracy as an objective, continuous probe is a clever idea. The defense survey on encrypted inputs is useful: post-hoc filters like Self-Examination and Llama Guard suppress harmful outputs but over-refuse, while perplexity filtering degrades utility. That's a practical, credible comparison. Finding 1 (models decode only ciphers that appear in pretraining) is consistent with prior work and backed by Table 2.

The soft spots are real, and the stress-test note hits the main one. ΔIR and Δresp in Section 5.2 are differences between decryption performance on JailbreakBench texts and a small hand-built benign set that varies in length, style, and domain. If the harmful texts are harder to decrypt under the same cipher—due to length, vocabulary, or syntax—the deltas appear with no safety behavior at all. The paper offers no matched control and no output-level labels (refusal vs. correct vs. wrong decryption). It defines ASR in Eq. 5 but never reports it, so 'unsafe responses or over-refusal' is inferred from string-similarity drops, not directly measured. Finding 4's 'statistically significant' has no statistical test behind it. The defense comparison runs on GPT-4o only, and the perplexity filter's threshold is a free parameter. These are addressable, but they mean the paper currently supports a weaker claim: safety-relevant deltas exist for some models on some ciphers, with the mechanism unconfirmed.

This is a paper for people working on jailbreak evaluation and safety metrics. It deserves a serious referee—the framework is valuable enough that the methodology should be given a chance to be fixed. I'd send it out, expecting major revision: a matched benign control or direct refusal/unsafety labels would turn a plausible framework into a solid result.

Referee Report

5 major / 5 minor

Summary. The paper proposes a two-dimensional safety evaluation for LLMs on encrypted (long-tail) inputs: D1, instruction refusal (whether models refuse to decrypt harmful instructions), and D2, generation safety (whether models suppress harmful decrypted responses). Using a new benchmark of 2,502 plaintext-ciphertext pairs over nine ciphers and five LLMs, the authors compute decryption performance (EM, BLEU, NL) on benign versus harmful texts and define deltas Δ_IR and Δ_resp as proxies for the two safety dimensions. They then evaluate several pre- and post-LLM defenses (perplexity filtering, re-tokenization, Llama Guard, Self-Reminder, Self-Examination) on GPT-4o. The central claim is that models able to decrypt ciphers can exhibit mismatched-generalization failures: safety mechanisms fail on at least one dimension, causing either unsafe responses or over-refusal, and existing defenses trade off one dimension against the other.

Significance. If the central claim were fully supported, the two-dimensional framework would be a useful corrective to ASR-only jailbreak evaluation, and the benchmark could support future work on long-tail-distribution safety. The paper also gives a concrete, falsifiable prediction about the trade-off between instruction refusal and generation safety, and it tests a range of published defenses. The main weaknesses are construct validity: the safety deltas rest on an unmatched benign-versus-harmful text comparison, and the paper never directly measures the safety outcomes (ASR or refusal/unsafety labels) that the framework is about. As it stands, the evidence is consistent with an alternative explanation in terms of decryption difficulty rather than safety behavior.

major comments (5)

[§5.2, Eqs. (3)–(4), Table 3] The central measurements are confounded by unmatched text sets. The benign set is 78 LLM-generated texts deliberately varied in domain, style, and length, while the harmful set is 100 JailbreakBench prompts/responses. These sets are not matched on length, vocabulary frequency, or syntactic complexity; harmful prompts often contain uncommon technical terms and longer clauses, making ciphertext harder to decrypt regardless of safety behavior. Claude's headline ΔEM_IR = 0.25 and the analogous deltas in Table 3 and Table 4 could therefore be lexical-difficulty artifacts rather than refusal or suppression. To support the central claim, the authors should either match benign and harmful texts on length and difficulty, or directly label outputs as refusals, unsafe generations, or successful decryptions.
[§3.2, Eq. (5)] Attack Success Rate (ASR) is defined in Eq. (5), but it is never computed or reported anywhere in the paper. The abstract and conclusion claim that safety mechanisms 'fail on at least one safety dimension, leading to unsafe responses or over-refusal,' yet the only evidence is a drop in decryption performance on harmful relative to benign texts. A drop in EM/BLEU is not the same as a refusal or an unsafe response: it could mean the model simply failed to decrypt. The paper should report ASR on encrypted harmful instructions and, ideally, sample-level labels of refusal versus unsafe generation versus incorrect decryption.
[§5.2, Finding 4] Finding 4 states that there is 'a statistically significant disparity (ΔEM ≫ ΔBLEU)' indicating that safety mechanisms suppress exact reproductions more than partial outputs, but the paper reports no statistical tests, no error bars, and no per-sample variance. The averages in Table 3 are over only four easy ciphers, one of which (Bacon) is at floor for all models, so the claim rests on very few effective observations. The authors should provide per-cipher or per-sample confidence intervals and a significance test, or temper the claim to a descriptive observation.
[§5.3, Table 4] The defense evaluation is conducted on a single model (GPT-4o) with no repeated trials and no significance testing, so the comparative claims about defenses (Findings 5 and 6) are not quantitatively supported. In addition, some cells in Table 4 are difficult to interpret without error bars: for Re-tokenization on Morse, Δresp = −0.61, which the text summarizes only as 'counterproductively facilitates harmful response generation,' and the utility drop for Perplexity Filter (0.22–0.27) is larger than most of the safety deltas it is supposed to produce. A sensitivity analysis for the perplexity threshold (Appendix A.7.1) is also needed, since that threshold is a free parameter that can strongly affect all Perplexity Filter results.
[Table 2, Table 3, Appendix A.5] The inclusion of Bacon in the 'Easy' group and in the average deltas is not justified by the results. Table 2 shows Bacon decryption performance near 0.05–0.07 for all models, so for this cipher there is no measurable decryption capability on either benign or harmful texts. Including Bacon in the Table 3 averages compresses all deltas toward zero (e.g., Claude's harmful-instruction EM average includes 0.00 for Bacon alongside 0.64, 0.58, and 0.61 for the other three ciphers), making the average deltas hard to interpret. The paper should either analyze each easy cipher separately when testing safety or justify why a cipher the models cannot decrypt should contribute to a safety-behavior average.

minor comments (5)

[Abstract and §1] The phrase 'susceptible tomismatched-generalization attacks' contains a spacing typo ('tomismatched'); similarly, Figure 2 contains 'Cased 3' for 'Case 3', and §6 has 'Safey' for 'Safety'.
[§1 and Table 1] The paper alternates between 'JailbreakBench' and 'JailbreakBench Dataset'; the two references (Chao et al., 2024a and 2024b) appear to be the same resource, and the text should disambiguate which version is used.
[Appendix A.7.1] The perplexity filter threshold is set to the maximum perplexity of 'legitimate decryption instructions,' but the exact set of instructions used to set this threshold is not described; without this detail the defense results are not reproducible.
[Appendix A.6, Eq. (9)] The Normalized Levenshtein formula is written with L(x̂, x) in the numerator but the text says 'with higher values indicating greater similarity'; this is inconsistent with standard Levenshtein distance, and the direction should be clarified (or the metric should be defined as 1 − normalized distance).
[§5.2 and Table 3] The colors used to mark positive and negative deltas in Table 3 are described in the caption but are not visible in black-and-white print; adding explicit plus/minus signs or arrows would improve readability.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central results are direct measurements against external benchmarks, with only a peripheral self-citation.

full rationale

The paper's derivation chain is empirical rather than definitional. The two safety dimensions are defined by direct decryption-performance comparisons: ΔIR = Perf(benign) − Perf(harmful instructions) and Δresp = Perf(benign) − Perf(harmful responses), measured against the external JailbreakBench dataset (Chao et al., 2024a) and previously published defense methods. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' own prior work, and no ansatz is smuggled in by self-citation. The only self-citation, Maskey et al. (2025), supplies a supporting explanation for Bacon-cipher failure (token inflation) and is not load-bearing for the main findings. The substantive weakness—that the benign and harmful text sets are not matched in length, vocabulary, or complexity, so the reported deltas could partly reflect decryption difficulty rather than safety behavior—is a construct-validity threat, not a circularity in the derivation. Thus the circularity score is near zero.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No fitted parameters enter the central D1/D2 derivation; the only data-dependent knob is the perplexity filter threshold. The load-bearing assumptions are about what the measured deltas mean. No new entities are postulated.

free parameters (1)

Perplexity filter detection threshold = Not reported
Set to the maximum perplexity observed on legitimate decryption instructions (Appendix A.7.1). This data-dependent threshold affects the pre-LLM defense results in Table 4, but not the main D1/D2 claim.

assumptions (4)

domain assumption Decryption performance gaps between benign and harmful encrypted texts are interpretable as safety refusal or suppression.
Used in Eqs. 3 and 4 to define D1 and D2. Confounded because harmful and benign text sets come from different sources and are not matched in style, length, or domain.
domain assumption Ciphertext preserves plaintext semantics exactly for the tested ciphers.
Decryption of harmful instructions and harmful responses is assumed to present the same harm as the plaintext. This is plausible for Caesar, Atbash, and Morse, but not justified for all long-tail formats.
domain assumption JailbreakBench harmful instructions and responses are representative harmful content.
Used for Dharmful and Dharmful_resp. The paper itself notes that harmful responses are short leads and that more diverse harmful response data would be useful.
ad hoc to paper Bacon cipher belongs in the Easy group and in average deltas.
Bacon is labeled Easy in Table 1 and included in averages despite near-zero decryption performance across all models (Table 2). Including it lowers average scores without adding evidence of safety behavior.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Should LLM Safety Be More Than Refusing Harmful Instructions?." pith.science (2026). https://pith.science/paper/GF5IUTBP

@misc{pith2026250602442,
  author       = {Pith},
  title        = {Pith review of: Should LLM Safety Be More Than Refusing Harmful Instructions?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GF5IUTBP}},
  note         = {Machine review of arXiv:2506.02442}
}

read the original abstract

This paper presents a systematic evaluation of Large Language Models' (LLMs) behavior on long-tail distributed (encrypted) texts and their safety implications. We introduce a two-dimensional framework for assessing LLM safety: (1) instruction refusal-the ability to reject harmful obfuscated instructions, and (2) generation safety-the suppression of generating harmful responses. Through comprehensive experiments, we demonstrate that models that possess capabilities to decrypt ciphers may be susceptible to mismatched-generalization attacks: their safety mechanisms fail on at least one safety dimension, leading to unsafe responses or over-refusal. Based on these findings, we evaluate a number of pre-LLM and post-LLM safeguards and discuss their strengths and limitations. This work contributes to understanding the safety of LLM in long-tail text scenarios and provides directions for developing robust safety mechanisms.

Figures

Figures reproduced from arXiv: 2506.02442 by the authors.

**Figure 2.** Figure illustrates a two-dimensional long-tail based safety evaluation of LLMs: D1 (Pre-LLM Early [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 12 canonical work pages

[1]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023
[2]

Ezat Ahmadzadeh, Hyunil Kim, Ongee Jeong, Namki Kim, and Inkyu Moon. 2022. A deep bidirectional lstm-gru network model for automated ciphertext classification. IEEE access, 10:3228--3237

work page 2022
[3]

Gabriel Alon and Michael Kamfonas. 2023. Detecting language model attacks with perplexity. arXiv preprint arXiv:2308.14132

arXiv 2023
[4]

Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 1

arXiv 2023
[5]

Adnan Benamira et al. 2021. https://eprint.iacr.org/2021/287 Revisiting neural distinguishers and their relation to differential distribution tables . IACR ePrint Archive

work page 2021
[6]

Emet Bethany, Mazal Bethany, Juan Arturo Nolazco Flores, Sumit Kumar Jha, and Peyman Najafirad. 2024. Jailbreaking large language models with symbolic mathematics. arXiv preprint arXiv:2409.11445

arXiv 2024
[7]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

2020
[8]

Pappas, Florian Tram\` e r, Hamed Hassani, and Eric Wong

Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tram\` e r, Hamed Hassani, and Eric Wong. 2024 a . https://proceedings.neurips.cc/paper_files/paper/2024/file/63092d79154adebd7305dfd498cbff70-Paper-Datasets_and_Benchmarks_Track.pdf Jailb...

work page 2024

Show all 48 references

[9]

Pappas, Florian Tram \`e r, Hamed Hassani, and Eric Wong

Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tram \`e r, Hamed Hassani, and Eric Wong. 2024 b . https://openreview.net/forum?id=urjPCYZt0I Jailbreakbench...

2024
[10]

Jing Cui, Yishi Xu, Zhewei Huang, Shuchang Zhou, Jianbin Jiao, and Junge Zhang. 2024. Recent advances in attack and defense approaches of large language models. arXiv preprint arXiv:2409.03274

2024 arXiv
[11]

Zhichen Dong, Zhanhui Zhou, Chao Yang, Jing Shao, and Yu Qiao. 2024. https://arxiv.org/abs/2402.09283 Attacks, defenses and evaluations for llm conversation safety: A survey . Preprint, arXiv:2402.09283

2024 arXiv
[12]

John Dooley. 2018. https://doi.org/10.1007/978-3-031-67485-3 History of Cryptography and Cryptanalysis: Codes, Ciphers, and Their Algorithms

2018 doi
[13]

Albrecht Gohr. 2019. Improving attacks on round-reduced speck32/64 using deep learning. IACR Transactions on Symmetric Cryptology, 2019(1):163--181

2019
[14]

Rodrigo Gomez et al. 2018. https://arxiv.org/abs/1801.04883 Ciphergan: Unsupervised cipher cracking using gans . arXiv preprint arXiv:1801.04883

2018 arXiv
[15]

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

2025 arXiv
[16]

Roger A Hallman. 2022. Poster evegan: Using generative deep learning for cryptanalysis. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pages 3355--3357

2022
[17]

competency

Divij Handa, Zehua Zhang, Amir Saeidi, Shrinidhi Kumbhar, and Chitta Baral. 2024. When" competency" in reasoning opens the door to vulnerability: Jailbreaking llms via novel complex ciphers. arXiv preprint arXiv:2402.10601

2024
[18]

Jeremy Howard and Sebastian Ruder. 2018. https://doi.org/10.18653/v1/P18-1031 Universal language model fine-tuning for text classification . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328--339, Melb...

2018 doi
[19]

Brian RY Huang, Maximilian Li, and Leonard Tang. 2024. Endless jailbreaks with bijection learning. arXiv preprint arXiv:2410.01294

2024 arXiv
[20]

Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. 2023. https://arxiv.org/abs/2312.06674 Llama guard: Llm-based input-output safeguard for human-ai conversations ...

2023 arXiv
[21]

Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614

2023 arXiv
[22]

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

2023 arXiv
[23]

Fengqing Jiang, Zhangchen Xu, Luyao Niu, Zhen Xiang, Bhaskar Ramasubramanian, Bo Li, and Radha Poovendran. 2024. Artprompt: Ascii art-based jailbreak attacks against aligned llms. arXiv preprint arXiv:2402.11753

2024 arXiv
[24]

Shubhra Kanti Karmaker Santu and Dongji Feng. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.946 TEL e R : A general taxonomy of LLM prompts for benchmarking complex tasks . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14197--14203, Si...

2023 doi
[25]

Yu Li, Qizhi Pei, Mengyuan Sun, Honglin Lin, Chenlin Ming, Xin Gao, Jiang Wu, Conghui He, and Lijun Wu. 2025. https://arxiv.org/abs/2504.19093 Cipherbank: Exploring the boundary of llm reasoning capabilities through cryptography challenges . Preprint, arXiv:2504.19093

2025 arXiv
[26]

Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2024. https://arxiv.org/abs/2310.04451 Autodan: Generating stealthy jailbreak prompts on aligned large language models . Preprint, arXiv:2310.04451

2024 arXiv
[27]

Huijie Lv, Xiao Wang, Yuansen Zhang, Caishuang Huang, Shihan Dou, Junjie Ye, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. https://arxiv.org/abs/2402.16717 Codechameleon: Personalized encryption framework for jailbreaking large language models . Preprint, arXiv:2402.16717

2024 arXiv
[28]

Utsav Maskey, Chencheng Zhu, and Usman Naseem. 2025. https://arxiv.org/abs/2505.24621 Benchmarking large language models for cryptanalysis and mismatched-generalization . Preprint, arXiv:2505.24621

2025 arXiv
[29]

Yutao Mou, Yuxiao Luo, Shikun Zhang, and Wei Ye. 2025. https://arxiv.org/abs/2504.09420 Saro: Enhancing llm safety through reasoning-based alignment . Preprint, arXiv:2504.09420

2025 arXiv
[30]

D. Noever. 2023. https://doi.org/10.5121/ijaia.2023.14301 Large language models for ciphers . International Journal of Artificial Intelligence & Applications, 14:1--20

2023
[31]

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...

2022
[32]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 Bleu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, page 311...

2002
[33]

Mansi Phute, Alec Helbling, Matthew Daniel Hull, ShengYun Peng, Sebastian Szyller, Cory Cornelius, and Duen Horng Chau. 2024. https://openreview.net/forum?id=YoqgcIA19o LLM self defense: By self examination, LLM s know they are being tricked . In The Second Tiny Papers Track a...

2024
[34]

Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2020. https://doi.org/10.18653/v1/2020.acl-main.170 BPE -dropout: Simple and effective subword regularization . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1882--1892, O...

2020 doi
[35]

Jitendrakumar Radadiya and Hb Tank. 2023. A review paper on cryptography algorithms

2023
[36]

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

2023 arXiv
[37]

Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36:80079--80110

2023
[38]

Dai, and Quoc V Le

Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022 a . https://openreview.net/forum?id=gEZrGCozdqR Finetuned language models are zero-shot learners . In International Conference on Learning Representations

2022
[39]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022 b . Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

2022
[40]

Ellington Wenger et al. 2022. https://eprint.iacr.org/2022/935 Applying neural networks to solve learning with errors problem in cryptography . IACR ePrint Archive

2022
[41]

Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. 2023. https://doi.org/10.1038/s42256-023-00765-8 Defending chatgpt against jailbreak attack via self-reminders . Nature Machine Intelligence, 5:1486--1496

2023 doi
[42]

Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Poovendran. 2024. https://doi.org/10.18653/v1/2024.acl-long.303 S afe D ecoding: Defending against jailbreak attacks via safety-aware decoding . In Proceedings of the 62nd Annual Meeting of the As...

2024 doi
[43]

Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. 2024. https://doi.org/10.1016/j.hcc.2024.100211 A survey on large language model (llm) security and privacy: The good, the bad, and the ugly . High-Confidence Computing, 4(2):100211

2024
[44]

Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. 2024. https://openreview.net/forum?id=MbfAK4s61A GPT -4 is too smart to be safe: Stealthy chat with LLM s via cipher . In The Twelfth International Conference on Learning Repres...

2024
[45]

Li Yujian and Liu Bo. 2007. https://doi.org/10.1109/TPAMI.2007.1078 A normalized levenshtein distance metric . IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(6):1091--1095

2007
[46]

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675

2019 arXiv
[47]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[48]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023

[2] [2]

Ezat Ahmadzadeh, Hyunil Kim, Ongee Jeong, Namki Kim, and Inkyu Moon. 2022. A deep bidirectional lstm-gru network model for automated ciphertext classification. IEEE access, 10:3228--3237

work page 2022

[3] [3]

Gabriel Alon and Michael Kamfonas. 2023. Detecting language model attacks with perplexity. arXiv preprint arXiv:2308.14132

arXiv 2023

[4] [4]

Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 1

arXiv 2023

[5] [5]

Adnan Benamira et al. 2021. https://eprint.iacr.org/2021/287 Revisiting neural distinguishers and their relation to differential distribution tables . IACR ePrint Archive

work page 2021

[6] [6]

Emet Bethany, Mazal Bethany, Juan Arturo Nolazco Flores, Sumit Kumar Jha, and Peyman Najafirad. 2024. Jailbreaking large language models with symbolic mathematics. arXiv preprint arXiv:2409.11445

arXiv 2024

[7] [7]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

2020

[8] [8]

Pappas, Florian Tram\` e r, Hamed Hassani, and Eric Wong

Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tram\` e r, Hamed Hassani, and Eric Wong. 2024 a . https://proceedings.neurips.cc/paper_files/paper/2024/file/63092d79154adebd7305dfd498cbff70-Paper-Datasets_and_Benchmarks_Track.pdf Jailb...

work page 2024