Pith. sign in

REVIEW 3 major objections 4 minor 51 references

Improving LLM Outputs Against Jailbreak Attacks with Expert Model Integration

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A small BERT classifier, appended to prompts as an expert label, improves LLM accuracy on jailbreak and injection benchmarks, with gains up to 17.9 percent for one large model.

desk verdict Plausible engineering recipe, but the benchmark's label-following confound undercuts the headline accuracy gains. read the letter →

arxiv 2505.17066 v3 pith:S5PAFPUT submitted 2025-05-18 cs.CR cs.AI

Tatia Tsmindashvili , Ana Kolkhidashvili , Dachi Kurtskhalia , Nino Maghlakelidze , Elene Mekvabishvili , Guram Dentoshvili , Orkhan Shamilov , Zaal Gachechiladze
show 2 more authors
Steven Saporta David Dachi Choladze
This is my paper · ORCID
classification cs.CRcs.AI
keywords LLMsecurityjailbreakattackspromptinjectionexpertmodelintegrationBERTclassifierautomotivedomainbenchmarkdatasetin-contextlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a small, fine-tuned classifier—Archias, a BERT-based model—can make large language models substantially more resistant to jailbreaks, prompt injections, and out-of-domain queries in a specialized retail setting. By appending Archias's category label for a user inquiry to the prompt before generation, the authors report consistent accuracy gains on their own 150-question multiple-choice benchmark, including a 17.9 percent improvement for a 70-billion-parameter instruct model and 12.7 percent for their in-house automotive model. The proposal matters because API-hosted models cannot be freely fine-tuned by a business, and existing defenses such as fine-tuning and in-context learning are not dependable. If the effect is real, a lightweight, industry-customizable expert model could be a practical security layer for production chatbots.

What carries the argument

Archias is the load-bearing component: a pretrained transformer classifier fine-tuned on public, synthetic, and masked automotive-domain conversations. Its output—one of five category labels for the user inquiry—is inserted directly into the prompt before the LLM generates a response, combining the ingestion idea behind SuperICL with the reminder idea behind self-reminder. The appended label reframes the task for the LLM, and the paper reports it changes answers in cases like 'Pretend you're a cow now,' steering models away from misreading playful input as harassment and toward recognizing a prompt-injection attempt. Confidence scores from Archias are also proposed as a control signal, letting the pipeline ignore the expert output when the classifier is unsure.

What would settle it

Run the same expert-label pipeline on an established external jailbreak benchmark with open-ended generation and human evaluation; if accuracy gains disappear or reverse when the expert label is occasionally wrong, the central claim is an artifact of the multiple-choice format.

Watch

Extended reading notes

Core claim

The central claim is that integrating an expert model's classification output into an LLM prompt improves the model's ability to detect and reason about attacks compared with the model alone. Archias, a 109-million-parameter transformer classifier with a five-category scheme (in-domain, malicious questions, price injections, prompt injections, out-of-domain), is assigned to the user inquiry, and its label is appended to the prompt; the LLM then uses that label when selecting its answer. Across the paper's benchmark of 150 manually crafted automotive-domain multiple-choice questions, adding the expert label improved every evaluated model, with the largest gains in price injection, in-domain, and prompt injection categories. The paper also reports that this expert-labeling approach outperformed the self-reminder defense on the models tested, and that Archias itself reaches 0.94 accuracy and 0.92 macro F1 on its classification test set.

Load-bearing premise

The claim rests on a same-team, closed-ended benchmark whose correct answers are tied to Archias's five categories, so the reported gains may partly reflect the LLM following an appended label rather than true resistance to attacks.

Editorial extensions

If this is right

  • A business can add a security layer to any LLM, including API-only models, without retraining the LLM itself: only the small classifier needs domain-specific fine-tuning.
  • Accuracy gains are concentrated where generic models struggle most, namely price injection and prompt injection, suggesting the expert label supplies missing context rather than general helpfulness.
  • Because the classifier runs in 5-10 milliseconds on a GPU and under 500 megabytes of memory, the defense is cheap enough for real-time production chatbots.
  • Extending Archias to another industry, such as healthcare or finance, would require only new labeled examples rather than a new architecture, if the method transfers.
  • The released 150-question benchmark gives other teams a public test for jailbreak, price-injection, and prompt-injection resistance in retail contexts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The benchmark's correct answers are defined by Archias's five categories and the expert label is appended verbatim, so part of the measured gain may be the LLM complying with the injected label rather than independently recognizing an attack; an independent attack-identification test would separate these.
  • An untested adversarial scenario is an attacker who imitates the expert label, for example phrasing a price injection as an in-domain request, which could turn the appended label into a second injection channel.
  • Open-ended generation, where a model must produce an answer rather than select one, is acknowledged as future work; the 88 percent human score on the closed-ended benchmark suggests real interactions are harder, so gains may shrink in deployment.
  • A testable extension is to apply the same method with the same classifier in other domains and on established external jailbreak benchmarks to verify the gains are not specific to the team's own dataset.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Archias, a small BERT-based expert model that classifies user inquiries into five categories (in-domain, malicious questions, price injections, prompt injections, out-of-domain) and appends its output to prompts before an LLM generates a response. The method is evaluated on a new 150-question multiple-choice benchmark in the automotive domain, reporting consistent accuracy improvements across a range of open-source, API-based, and in-house models, including +17.9% for Llama-3-70B-Instruct and +12.7% for Impel-LLM. The authors also compare against a 'self-reminder' baseline and release the benchmark dataset.

Significance. If the reported effect is real, the approach would offer a lightweight, inexpensive, and industry-customizable defense layer for domain-specific conversational AI, with execution in tens of milliseconds and sub-500 MB memory. The paper also contributes a publicly released benchmark and a reproducible evaluation harness, which supports community follow-up. The practical relevance is high and the cross-model generality is plausible, but the empirical claim is currently compromised by a design that lets the appended expert label partially reveal the correct answer, making it uncertain whether the gains reflect improved security reasoning or simple label-following.

major comments (3)
  1. [III-B, Fig. 2] The evaluation design creates an information leak: the benchmark's correct answers are built on the same five-category taxonomy that Archias outputs, and the expert opinion appended to the prompt is exactly the predicted category label. An LLM facing a multiple-choice question can therefore improve by picking the option consistent with the appended label, without any additional security reasoning. The paper reports no control conditions in Section IV (e.g., appending a random, fixed, or deliberately wrong label), so the +17.9% for Llama-3-70B-Instruct and +12.7% for Impel-LLM cannot be attributed to expert-guided reasoning as opposed to label-following. Please add such controls, or redesign the task so that the expert output requires the model to reason about the classification rather than simply being handed it.
  2. [IV, Tables 2 and 4] The central accuracy claims rest on 150 manually crafted examples, with category sizes as small as 25-41 items, and the results are reported as point accuracies with no error bars, confidence intervals, or significance tests. In the smallest categories, a single example shifts the measured accuracy by several percentage points, so the per-category improvement patterns in Fig. 4 may not be reliable. Please report bootstrap confidence intervals or exact paired tests for the with-expert versus without-expert comparisons, and preferably also evaluate on an additional benchmark that was not authored by the same team.
  3. [III-A, IV] Archias is reported to have an F1 score of 0.92 and accuracy of 0.94 on its own test set, but the paper never analyzes what happens when Archias's label is incorrect. Conditioning the Table 2 results on the correctness of Archias's prediction would clarify the mechanism: if gains occur mainly when the label is correct, the method is essentially 'a reasonably accurate classifier followed by an LLM that obeys the hint,' whereas gains even for wrong labels would demonstrate genuine robustness. Such an analysis is necessary to address the label-leakage concern directly, and it can be computed from the released benchmark at negligible cost.
minor comments (4)
  1. [III-A] The claim of '~5-10 ms' GPU and '~50-100 ms' CPU latency and '$35 per month' cost is presented without measurement or pricing details; please either provide a short methodology for these numbers or characterize them as rough estimates.
  2. [Throughout] Model names are written inconsistently ('LLama 3 70B Instruct' vs. 'Llama-3-70B-Instruct', 'GPT 3.5 Turbo' vs. 'GPT-3.5'); unify the naming and ensure the exact model versions used in the evaluation are stated.
  3. [References, [51]] The released benchmark is a central reproducibility claim, but the reference to the Hugging Face dataset omits a URL or persistent identifier; include the full URL (and dataset version) so readers can actually access it.
  4. [Fig. 2 and Appendix] The example prompt templates used for the with-expert and without-expert conditions should be shown verbatim, ideally in a ready-to-run format; the appendix examples (Figs. 5-6) do not clearly indicate which prompt variant produced each quoted output.

Circularity Check

1 steps flagged · score 6.0 of 10

The benchmark's correct answers are keyed to the same five categories that Archias outputs, so the reported accuracy gains partly measure the LLM following an injected category label rather than improved attack resistance.

  1. fitted input called prediction [Abstract (Archias categories); Section III-B (benchmark construction, Fig. 2); Section IV (reported result)]
    "Archias classifies user inquiries into several categories: in-domain (specifically for the automotive industry), malicious questions, price injections, prompt injections, and out-of-domain examples. ... We manually crafted a total of 150 examples, each designed to challenge the models on various aspects of the five topics in Table 1. ... Fig. 2 shows an example of our benchmark dataset: A) without and B) with use of the expert model’s output."

    Archias is a fitted classifier over exactly these five categories, and the benchmark's items are explicitly 'designed to challenge the models on various aspects of the five topics in Table 1'—the same five categories. The method then inserts Archias's category output into the prompt before the LLM answers the multiple-choice question (Fig. 2B). Because each item's correct answer is determined by which of these categories it belongs to, an accurate Archias label is a near-answer hint: the LLM can choose the option matching the injected label without any independent security reasoning.

full rationale

Score 6 reflects a partial but real circularity in the central empirical claim. The paper's headline result—appending Archias's output improves LLM accuracy on attack scenarios—is evaluated entirely on a 150-item multiple-choice benchmark whose items are deliberately built around the same five categories that Archias outputs. The pipeline (Fig. 2B) puts the expert's category label into the prompt before the LLM chooses among options, so an accurate label is a strong hint about the correct answer. Archias is reported to have 0.94 test accuracy, so most labels are correct; the improvement from 63% to 75.7% for Impel-LLM and similar jumps for Llama-3-70B-Instruct can be explained by label-following alone. The paper provides no control condition with wrong, random, or constant labels, and the benchmark is the authors' own, not an external or open-ended attack suite; the paper itself concedes that real-world interactions are open-ended. This makes the measured improvement partly an artifact of the evaluation design. It is not a full equivalence: the LLM still has to map the label to the correct option, and the 'with expert' condition contains additional text that may focus attention, but the decisive claim of 'detecting and reasoning about jailbreaks, price injections, and out-of-domain examples' is tied to the taxonomy the expert was trained on. No load-bearing self-citation or imported uniqueness theorem is involved; the self-citation [51] is to the released benchmark and is not used to justify the method. Hence 6, not 0 or 2.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on a hand-built five-category taxonomy, a synthetic and masked training set, a small same-team benchmark, and an evaluation design in which the fitted classifier's label is injected into the prompt. These are all author-controlled choices rather than externally established benchmarks, so the ledger is dominated by domain assumptions and design choices.

free parameters (2)
  • Archias fine-tuning hyperparameters = learning rate 1e-5, batch size 16, 3 epochs, weight decay 0.1
    Selected as the best configuration (F1 0.92, accuracy 0.94) in Section IV; no sensitivity analysis is provided, and the defense's value depends on this classifier being accurate.
  • Benchmark composition = 150 examples: 41 malicious, 31 prompt injections, 27 out-of-domain, 26 price injections, 25 in-domain
    The empirical claim is measured only on this handcrafted set; small per-category counts and the same-team design make the reported improvements fragile.
assumptions (3)
  • domain assumption The five categories (in-domain, malicious, price injection, prompt injection, out-of-domain) are exhaustive and mutually exclusive for real user inquiries.
    Section III-A defines the taxonomy and builds both training data and benchmark around it; no evidence covers classes outside this scheme.
  • domain assumption A 150-item multiple-choice benchmark is a valid proxy for jailbreak and prompt-injection resistance.
    Section III-B concedes that real interactions are open-ended and calls alternative evaluation setups future work.
  • domain assumption Synthetic templates and masked Impel data are representative of real-world attacks.
    Section III-A describes generating 20-30 templates per category and manual review, but no external attack corpus is used to validate representativeness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving LLM Outputs Against Jailbreak Attacks with Expert Model Integration." pith.science (2026). https://pith.science/paper/S5PAFPUT

@misc{pith2026250517066,
  author       = {Pith},
  title        = {Pith review of: Improving LLM Outputs Against Jailbreak Attacks with Expert Model Integration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S5PAFPUT}},
  note         = {Machine review of arXiv:2505.17066}
}
read the original abstract

Using LLMs in a production environment presents security challenges that include vulnerabilities to jailbreaks and prompt injections, which can result in harmful outputs for humans or the enterprise. The challenge is amplified when working within a specific domain, as topics generally accepted for LLMs to address may be irrelevant to that field. These problems can be mitigated, for example, by fine-tuning large language models with domain-specific and security-focused data. However, these alone are insufficient, as jailbreak techniques evolve. Additionally, API-accessed models do not offer the flexibility needed to tailor behavior to industry-specific objectives, and in-context learning is not always sufficient or reliable. In response to these challenges, we introduce Archias, an expert model adept at distinguishing between in-domain and out-of-domain communications. Archias classifies user inquiries into several categories: in-domain (specifically for the automotive industry), malicious questions, price injections, prompt injections, and out-of-domain examples. Our methodology integrates outputs from the expert model (Archias) into prompts, which are then processed by the LLM to generate responses. This method increases the model's ability to understand the user's intention and give appropriate answers. Archias can be adjusted, fine-tuned, and used for many different purposes due to its small size. Therefore, it can be easily customized to the needs of any industry. To validate our approach, we created a benchmark dataset for the automotive industry. Furthermore, in the interest of advancing research and development, we release our benchmark dataset to the community.

Figures

Figures reproduced from arXiv: 2505.17066 by the authors.

Figure 1
Figure 1. FIGURE 1 [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. FIGURE 2 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. FIGURE 3 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: FIGURE 4 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: FIGURE 5 [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: FIGURE 6 [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: FIGURE 7 [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 12 canonical work pages

  1. [1]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, ‘‘BERT: Pre-training of deep bidirectional transformers for language understanding,’’ 2018, arXiv:1810.04805

    arXiv 2018

  2. [2]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P . Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P . Welinder, P . F. Christiano, J. Leike, and R. Lowe, ‘‘Training language models to follow instructions with human feedback,’’ in Proc. Advances in Neural Information Processing S...

    work page 2022

  3. [3]

    Touvron, T

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Roziére, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, ‘‘LLaMA: Open and efficient foundation language models,’’ 2023, arXiv:2302.13971

    arXiv 2023

  4. [4]

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P . Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed, ‘‘Mistral 7B,’’ 2023, arXiv:2310.06825

    arXiv 2023

  5. [5]

    A. Q. Jiang et al., ‘‘Mixtral of experts,’’ 2024, arXiv:2401.04088

    arXiv 2024

  6. [6]

    Z. Ji, N. Lee, R. Frieske, T. Y u, D. Su, Y . Xu, E. Ishii, Y . J. Bang, A. Madotto, and P . Fung, ‘‘Survey of hallucination in natural language generation,’’ ACM Comput. Surveys, vol. 55, no. 12, pp. 1–38, Mar. 2023

    work page 2023

  7. [7]

    H. Qiu, S. Zhang, A. Li, H. He, and Z. Lan, ‘‘Latent jailbreak: A benchmark for evaluating text safety and output robustness of large language models,’’ 2023, arXiv:2307.08487

    arXiv 2023

  8. [8]

    Y . Xie, J. Yi, J. Shao, J. Curl, L. Lyu, Q. Chen, X. Xie, and F. Wu, ‘‘Defending ChatGPT against jailbreak attack via self-reminders,’’ Nature Mach. Intell., vol. 5, no. 12, pp. 1486–1496, Dec. 2023

    work page 2023

Show all 51 references
  1. [9]

    J. Li, Y . Liu, C. Liu, L. Shi, X. Ren, Y . Zheng, Y . Liu, and Y . Xue, ‘‘A cross- language investigation into jailbreak attacks in large language models,’’ 2024, arXiv:2401.16765

    2024 arXiv

  2. [10]

    Takemoto, ‘‘All in how you ask for it: Simple black-box method for jailbreak attacks,’’ 2024, arXiv:2401.09798

    K. Takemoto, ‘‘All in how you ask for it: Simple black-box method for jailbreak attacks,’’ 2024, arXiv:2401.09798

    2024 arXiv

  3. [11]

    Gupta, C

    M. Gupta, C. Akiri, K. Aryal, E. Parker, and L. Praharaj, ‘‘From ChatGPT to ThreatGPT: Impact of generative AI in cybersecurity and privacy,’’IEEE Access, vol. 11, pp. 80218–80245, 2023

    2023

  4. [12]

    T. Li, Z. Wang, W. Liu, M. Wu, S. Dou, C. Lv, X. Wang, X. Zheng, and X. Huang, ‘‘Revisiting jailbreaking for large language models: A representation engineering perspective,’’ 2024, arXiv:2401.06824

    2024 arXiv

  5. [13]

    Y . Zeng, H. Lin, J. Zhang, D. Y ang, R. Jia, and W. Shi, ‘‘How Johnny can persuade LLMs to jailbreak them: Rethinking persuasion to challenge AI safety by humanizing LLMs,’’ 2024, arXiv:2401.06373

    2024 arXiv

  6. [14]

    X. Shen, Z. Chen, M. Backes, Y . Shen, and Y . Zhang, ‘‘‘Do anything Now’: Characterizing and evaluating In-The-Wild jailbreak prompts on large language models,’’ 2023, arXiv:2308.03825

    2023 arXiv

  7. [15]

    X. Li, Z. Zhou, J. Zhu, J. Y ao, T. Liu, and B. Han, ‘‘DeepInception: Hyp- notize large language model to be jailbreaker,’’ 2023, arXiv:2311.03191

    2023 arXiv

  8. [16]

    Phute, A

    M. Phute, A. Helbling, M. Hull, S. Peng, S. Szyller, C. Cornelius, and D. H. Chau, ‘‘LLM self defense: By self examination, LLMs know they are being tricked,’’ 2023, arXiv:2308.07308

    2023 arXiv

  9. [17]

    N. Jain, A. Schwarzschild, Y . Wen, G. Somepalli, J. Kirchenbauer, P .-Y . Chiang, M. Goldblum, A. Saha, J. Geiping, and T. Goldstein, ‘‘Baseline defenses for adversarial attacks against aligned language models,’’ 2023, arXiv:2309.00614

    2023 arXiv

  10. [18]

    Y . Deng, W. Zhang, S. Jialin Pan, and L. Bing, ‘‘Multilingual jailbreak challenges in large language models,’’ 2023, arXiv:2310.06474

    2023 arXiv

  11. [19]

    Ganguli et al., ‘‘Red teaming language models to reduce harms: Meth- ods, scaling behaviors, and lessons learned,’’ 2022, arXiv:2209.07858

    D. Ganguli et al., ‘‘Red teaming language models to reduce harms: Meth- ods, scaling behaviors, and lessons learned,’’ 2022, arXiv:2209.07858

    2022 arXiv

  12. [20]

    K. Zhu, J. Wang, J. Zhou, Z. Wang, H. Chen, Y . Wang, L. Y ang, W. Y e, Y . Zhang, N. Zhenqiang Gong, and X. Xie, ‘‘PromptRobust: Towards evaluating the robustness of large language models on adversarial prompts,’’ 2023, arXiv:2306.04528

    2023 arXiv

  13. [21]

    Robey, E

    A. Robey, E. Wong, H. Hassani, and G. J. Pappas, ‘‘SmoothLLM: Defending large language models against jailbreaking attacks,’’ 2023, arXiv:2310.03684

    2023 arXiv

  14. [22]

    S. Zhu, R. Zhang, B. An, G. Wu, J. Barrow, Z. Wang, F. Huang, A. Nenkova, and T. Sun, ‘‘AutoDAN: Interpretable gradient-based adver- sarial attacks on large language models,’’ 2023, arXiv:2310.15140

    2023 arXiv

  15. [23]

    A. Rao, S. V ashistha, A. Naik, S. Aditya, and M. Choudhury, ‘‘Tricking LLMs into disobedience: Formalizing, analyzing, and detecting jail- breaks,’’ 2023, arXiv:2305.14965

    2023 arXiv

  16. [24]

    C. Anil, E. Durmus, M. Sharma, J. Benton, S. Kundu, J. Batson, N. Rimsky, M. Tong, J. Mu, and D. Ford, ‘‘Many-shot jailbreaking,’’ in Proc. Adv. Neural Inf. Process. Syst., 2023, pp. 1–47

    2023

  17. [25]

    T. Kim, S. Kotha, and A. Raghunathan, ‘‘Testing the limits of jailbreaking defenses with the purple problem,’’ 2024, arXiv:2403.14725

    2024 arXiv

  18. [26]

    Gehman, S

    S. Gehman, S. Gururangan, M. Sap, Y . Choi, and N. A. Smith, ‘‘RealToxicityPrompts: Evaluating neural toxic degeneration in language models,’’ 2020, arXiv:2009.11462

    2020 arXiv

  19. [27]

    Nangia, C

    N. Nangia, C. V ania, R. Bhalerao, and S. R. Bowman, ‘‘CrowS-pairs: A challenge dataset for measuring social biases in masked language models,’’ 2020, arXiv:2010.00133

    2020 arXiv

  20. [28]

    Carlini, F

    N. Carlini, F. Tramèr, E. Wallace, M. Jagielski, A. Herbert-V oss, K. Lee, A. Roberts, T. Brown, D. Song, Ú. Erlingsson, A. Oprea, and C. Raffel, ‘‘Extracting training data from large language models,’’ in Proc. 30th USENIX Secur . Symp., Aug. 2021, pp. 2633–2650

    2021

  21. [29]

    L. He, M. Xia, and P . Henderson, ‘‘What is in your safe data? Identifying benign data that breaks safety,’’ 2024, arXiv:2404.01099

    2024 arXiv

  22. [30]

    Z. Xu, F. Jiang, L. Niu, J. Jia, B. Y uchen Lin, and R. Poovendran, ‘‘SafeDecoding: Defending against jailbreak attacks via safety-aware decoding,’’ 2024, arXiv:2402.08983

    2024 arXiv

  23. [31]

    Perez and I

    F. Perez and I. Ribeiro, ‘‘Ignore previous prompt: Attack techniques for language models,’’ 2022, arXiv:2211.09527

    2022 arXiv

  24. [32]

    Wang and Y

    Y .-S. Wang and Y . Chang, ‘‘Toxicity detection with generative prompt- based inference,’’ 2022, arXiv:2205.12390

    2022 arXiv

  25. [33]

    A. Wei, N. Haghtalab, and J. Steinhardt, ‘‘Jailbroken: How does LLM safety training fail?’’ in Proc. Adv. Neural Inf. Process. Syst., vol. 36, 2024, pp. 1–12

    2024

  26. [34]

    Perez, S

    E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving, ‘‘Red teaming language models with language models,’’ 2022, arXiv:2202.03286

    2022 arXiv

  27. [35]

    Ganguli et al., ‘‘The capacity for moral self-correction in large language models,’’ 2023, arXiv:2302.07459

    D. Ganguli et al., ‘‘The capacity for moral self-correction in large language models,’’ 2023, arXiv:2302.07459

    2023 arXiv

  28. [36]

    P . C. R. Puttaparthi, S. S. Deo, H. Gul, Y . Tang, W. Shang, and Z. Y u, ‘‘Comprehensive evaluation of ChatGPT reliability through multilingual inquiries,’’ 2023, arXiv:2312.10524

    2023 arXiv

  29. [37]

    Y ong, C

    Z.-X. Y ong, C. Menghini, and S. H. Bach, ‘‘Low-resource languages jailbreak GPT-4,’’ 2023, arXiv:2310.02446

    2023 arXiv

  30. [38]

    Alon and M

    G. Alon and M. Kamfonas, ‘‘Detecting language model attacks with perplexity,’’ 2023,arXiv:2308.14132

    2023 arXiv

  31. [39]

    P . Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong, ‘‘Jailbreaking black box large language models in twenty queries,’’ 2023, arXiv:2310.08419

    2023 arXiv

  32. [40]

    G. Deng, Y . Liu, Y . Li, K. Wang, Y . Zhang, Z. Li, H. Wang, T. Zhang, and Y . Liu, ‘‘MASTERKEY: Automated jailbreaking of large language model chatbots,’’ in Proc. Netw. Distrib. Syst. Secur . Symp., 2024

    2024

  33. [41]

    C. Xu, Y . Xu, S. Wang, Y . Liu, C. Zhu, and J. McAuley, ‘‘Small models are valuable plug-ins for large language models,’’ 2023, arXiv:2305.08848

    2023 arXiv

  34. [42]

    H. Li, D. Guo, W. Fan, M. Xu, J. Huang, F. Meng, and Y . Song, ‘‘Multi-step jailbreaking privacy attacks on ChatGPT,’’ 2023, arXiv:2304.05197

    2023 arXiv

  35. [43]

    Shayegani, M

    E. Shayegani, M. Abdullah Al Mamun, Y . Fu, P . Zaree, Y . Dong, and N. Abu-Ghazaleh, ‘‘Survey of vulnerabilities in large language models revealed by adversarial attacks,’’ 2023, arXiv:2310.10844

    2023 arXiv

  36. [44]

    A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Zico Kolter, and M. Fredrikson, ‘‘Universal and transferable adversarial attacks on aligned language models,’’ 2023, arXiv:2307.15043

    2023 arXiv

  37. [45]

    Gao et al., ‘‘A framework for few-shot language model evaluation,’’ Zenodo, Dec

    L. Gao et al., ‘‘A framework for few-shot language model evaluation,’’ Zenodo, Dec. 2023, doi: 10.5281/zenodo.10256836

    2023 doi

  38. [46]

    Reddy, D

    S. Reddy, D. Chen, and C. D. Manning, ‘‘CoQA: A conversational question answering challenge,’’ Trans. Assoc. for Comput. Linguistics, vol. 7, pp. 249–266, Nov. 2019

    2019

  39. [47]

    Zellers, A

    R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi, ‘‘HellaSwag: Can a machine really finish your sentence?’’ 2019, arXiv:1905.07830

    2019 arXiv

  40. [48]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, ‘‘Measuring massive multitask language understanding,’’ 2020, arXiv:2009.03300

    2020 arXiv

  41. [49]

    Y . Wang, Y . Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, ‘‘Self-instruct: Aligning language models with self-generated instructions,’’ 2022, arXiv:2212.10560. 134986 VOLUME 13, 2025 T. Tsmindashvili et al.: Improving LLM Outputs Against Jailbreak Attacks

    2022 arXiv

  42. [50]

    Longpre, L

    S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung, Y . Tay, D. Zhou, Q. V . Le, B. Zoph, J. Wei, and A. Roberts, ‘‘The flan collection: Designing data and methods for effective instruction tuning,’’ in Proc. 40th Int. Conf. Mach. Learn., Jul. 2023, pp. 22631–22648

    2023

  43. [51]

    Impel Research Team. (2024). Jailbreak Benchmark: Evaluating LLM Robustness Against Prompt Injection and Malicious Queries. [Online]. Available: https://huggingface.co/datasets/Impel/Jailbreak_ Benchmark TATIA TSMINDASHVILI received the B.Sc. degree in electrical and computer ...

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.