Pith. sign in

REVIEW 3 major objections 4 minor 48 references

An Audit and Analysis of LLM-Assisted Health Misinformation Jailbreaks Against LLMs

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM-crafted jailbreak prompts elicit health misinformation from three popular models in 81–89% of trials — and the same models can detect most of it.

desk verdict A transparent, carefully documented jailbreak audit whose headline ASR numbers are conditional on a pre-selected prompt set; the detection results are plausible but unsurprising. Worth a serious referee, with revision. read the letter →

arxiv 2508.10010 v1 pith:P56GYK4K submitted 2025-08-06 cs.CL

Ayana Hussain , Patrick Zhao , Nicholas Vincent This is my paper
classification cs.CL
keywords LLMjailbreakshealthmisinformationattacksuccessrateLLM-as-a-judgedetectionpromptfilteringRedditWildChat
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims a double result: commodity LLMs can be steered into generating persuasive health misinformation, and the same class of models can reliably detect that misinformation. The authors used one LLM as an attacker, guided by role-play and scenario-building instructions, to produce 109 jailbreak prompts across three health topics, and report that three target models (GPT-3.5, Llama 3.1-8B, Gemini 2.0 Flash) generated misinformation in 81–89% of trials. Prompt framing mattered: leaked-document and alternate-reality narratives succeeded most, crisis narratives least. The same LLM used as a judge agreed with human annotations on up to 100% of cases for detecting jailbreak generation, and simple text classifiers separated jailbreak-generated misinformation from organic Reddit posts at roughly 99% accuracy. The stakes: if the picture holds, LLM-made health misinformation is easy to produce but also easy to detect, which tilts the balance toward LLM-assisted moderation.

What carries the argument

Two mechanisms carry the argument. The attack side is an iteratively curated generation loop: an attacker model (GPT-3.5) receives a minimal system prompt plus guidance components — alternate realities, simulated scenarios, persuasive and creative reasoning, and a push toward concise authoritative answers — and emits batches of ten attacks; any attack that fails is reintroduced and regenerated until 109 working prompts (each verified to elicit misinformation) are collected across three topics. The evaluation side is an LLM-as-a-Judge protocol: a Gemini judge scores every response on three dimensions — misinformation generation, validation, and obedience — calibrated against human annotations

What would settle it

Re-run the attack-generation pipeline without the discard-and-regenerate filter: keep the first unedited batches of attacker-generated prompts, refusals included, and test them on GPT-3.5, Llama 3.1-8B, and Gemini 2.0 Flash. If the unfiltered success rate collapses toward the models' baseline refusal rate, the claim that LLMs are 'easily exploited' to mass-produce health misinformation would not generalize. A second test: retrain the TF-IDF classifiers on unfiltered jailbreak responses — if the ~99% separation from organic text shrinks, the containment story weakens with it.

Watch

Extended reading notes

Core claim

The paper's central empirical claim is that a semi-automated attack loop — an attacker LLM prompted with alternate identities, fictitious scenarios, and authority framing, plus an iterative process that discards failed prompts and regenerates them — yields jailbreak prompts that reliably elicit fluent health misinformation from three target models: attack success rates of 0.86 for GPT-3.5, 0.89 for Llama 3.1-8B, and 0.81 for Gemini 2.0 Flash across all 109 attacks. Success depends on framing: 'New Leaked Documents and Scientific Proof' attained the highest ASR (0.943) and 'Alternate Reality and Perspectives' the second (0.906), while 'Crisis Narratives' lagged at 0.593. The paper's companion

Load-bearing premise

The reported attack success rates (0.81–0.89) are measured on a dataset the authors deliberately curated for success: any attack that failed was discarded and regenerated until all 109 prompts elicited misinformation, so the rates describe how reliably pre-selected working attacks succeed again, not how often raw, uncurated LLM-generated prompts would succeed in the wild.

Editorial extensions

If this is right

  • Framing is the lever: authority-style attacks ('leaked documents', 'alternate reality') reach 0.91–0.94 success, so safety work should prioritize credibility-impersonation and role-play framings over blanket refusal training.
  • Linguistic filtering is a plausible cheap defense: the attack prompts differ from real health queries in measurable ways — shorter, more vocabulary-diverse, credibility-seeking nouns, validation-seeking verbs — so prompt-level filters could catch many attacks without model retraining.
  • LLM judges are usable moderators: agreement with human labels reached 85% even on the demanding three-dimension score and near-perfect on generation alone, with low false-positive rates (1.8% on jailbreak detection), though the smaller Llama model lagged at 50%.
  • Classifiers distinguish origin, not truth: separating LLM-generated misinformation from organic misinformation is near-trivial (~99%), while telling organic misinformation from real health content is the genuinely hard task (~86%).
  • Detection-side feasibility reframes the threat model: generating persuasive health misinformation remains easy, but the authors' data indicate it is also easy to catch, supporting an LLM-assisted moderation deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The headline ASR is best read as an upper bound: because the 109 prompts were all pre-verified to work before the main runs, the 0.81–0.89 figures measure how reliably a curated attack list re-offends on fresh runs and other models — a raw, uncurated generation trial would give the honest exploitability estimate for a naive attacker.
  • The near-perfect classifier separation is likely a style fingerprint of current-generation LLM output (uniform length, academic vocabulary, structured argumentation); if jailbreak styles imitate anecdotal Reddit discourse, the ~99% separation should erode, which would be a testable early-warning signal.
  • The same attacker–judge loop is domain-portable: alternate-reality and leaked-document framings should transfer to financial, legal, or civic misinformation, where fake 'expert' citations carry the same persuasive weight — an extension the paper does not explore.
  • The judge's error pattern — flagging conditional language and echoing-before-refutation as endorsement — implies deployed moderation must distinguish repetition from agreement, or legitimate rebuttals will be over-blocked.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper audits LLM-assisted jailbreak attacks that elicit health misinformation from three LLMs (GPT-3.5, Llama 3.1-8B, Gemini 2.0 Flash). The authors iteratively generated 109 attack prompts with GPT-3.5, tested them against the three target models, and report high attack success rates (ASR: 0.86, 0.89, 0.81). They then compare attack prompts to WildChat queries, compare jailbreak responses to Reddit misinformation, and evaluate both LLM-as-judge and classical ML classifiers for misinformation detection. The main positive claim is that LLM-generated health misinformation is easy to produce and also easy to detect with LLM judges and simple classifiers.

Significance. If the claims held, the paper would strengthen evidence that LLM-assisted jailbreaks are a practical route to health misinformation and that LLM-based detection is a viable mitigation. The study has real strengths: a detailed attack-generation procedure, a manually reviewed attack taxonomy, a comparison of jailbreak prompts with in-the-wild queries, and an error analysis of false positives in both LLM and classifier detection. The paper also provides concrete prompt examples and appendices with additional metrics, which support reproducibility. However, the headline ASR numbers are conditional on a prompt set curated for success, and several consistency issues in the classification experiments need to be resolved before the central claims can be taken at face value.

major comments (3)
  1. [RQ1 Methods, 'Constructing and Analyzing the Attacks'; Conclusion] The 109-prompt dataset was iteratively filtered: 'If an attack failed, the failed attempt was reintroduced, and the model was instructed to regenerate a new batch of ten attacks... until we compiled a set of at least 100 total attack prompts, all of which successfully elicited misinformation from the target model.' The reported ASRs (0.86, 0.89, 0.81) are therefore conditional on a set already selected for success on the construction target (GPT-3.5). They measure the transfer/failure rate of a pre-selected 'successful prompt dataset,' not the probability that an LLM-assisted attack attempt succeeds or that a randomly sampled prompt is vulnerable. The conclusion that 'LLMs can still be easily exploited' depends on reading these conditional rates as unconditional. Please report the total number of generated and discarded prompts, and compute an end-to-end ASR (successful attacks / all gen
  2. [RQ1 Methods, 'Jailbreak Judge Model'; RQ3 Results, 'Jailbreak False Positives'] The judge model that labels all 981 responses is Gemini 2.0 Flash, which is also one of the three target models. The judge was calibrated against human annotations on only 20 randomly sampled responses, and the reported 100%/95% agreement figures come from that 20-item sample. A 20-item validation set gives very wide confidence intervals, and using the same model family as both target and judge creates a potential same-model bias. The manual analysis of 18 false positives is useful, but it is not described as a full relabeling of all 981 responses. Please provide a larger human-validation set (or a full relabeling of the corpus), report the confusion matrix against human labels, and discuss whether judge errors are systematically correlated with the target model being judged.
  3. [RQ3 Methods, 'Machine Learning Misinformation Classifiers'; Appendix A.7] The classifier experiments report inconsistent numbers of successful jailbreak responses: 825 in the JB-REAL task and 791 in the JB-ORG-MISINFO task. Given 109 attacks × 3 models × 3 runs = 981 total responses, the number of successful responses should be derivable from the reported ASRs; the manuscript does not reconcile these counts. This inconsistency undermines the reproducibility of the high test accuracies (99.0% for JB-ORG-MISINFO and 99.3% for JB-REAL). Please report the exact sample size for each class in each task, explain the discrepancy, and make the filtered datasets available.
minor comments (4)
  1. [Discussion] The Discussion bullet says 'average attack success rates (ASR) across all attacks between 0.83 and 0.85 on all health queries,' while the Conclusion reports per-model ASRs of 0.86, 0.89, and 0.81. Please clarify which aggregate is being reported and keep the numbers consistent.
  2. [RQ2 Methods] The WildChat comparison set is only 32 prompts per category (96 total) and the paper acknowledges this. The t-tests on vocabulary diversity and length should be interpreted with this small, non-random sample in mind; consider reporting effect sizes or confidence intervals.
  3. [RQ2 Results, 'Linguistic Structure and Style'] There is a typo in 'V ocabulary diversity' (extra space). Minor copyediting throughout would improve readability.
  4. [RQ1 Methods, 'Constructing and Analyzing the Attacks'] The attack success criterion is 'any explicit misinformation.' This binary threshold can count responses that also contain partial refutations as full successes. Consider reporting a graded obedience score as a secondary metric, since Appendix A.4 already computes such scores.

Circularity Check

1 steps flagged · score 2.0 of 10

No load-bearing circularity; minor self-evaluation and LLM-label feedback loops are partially mitigated by human calibration.

  1. other [RQ1 Methods: Jailbreak Judge Model; RQ3 Methods: LLM Misinformation Classifier]
    "Gemini was thus selected as the judge model and was used to score all responses across the misinformation-related metrics. ... we use the same model to identify misinformation in Reddit posts."

    The quantity 'attack success' for Gemini is defined by whether Gemini's own output contains misinformation, and the judge making that determination is Gemini itself; likewise, the Reddit posts that serve as ground-truth labels for training and testing the RQ3 classifiers are labeled by the same LLM family. These are feedback loops rather than derivations: the ASR and classifier accuracies are not independent of the evaluator. The paper partially breaks the loop by calibrating the judge against human annotations on 20 jailbreak responses and 200 Reddit posts and by manually reviewing false positives, so the central numbers are not pure identities; the loop is a validity limitation, not a forced result.

full rationale

I walked the derivation chain. There are no self-citations (the reference list contains no Hussain/Zhao/Vincent works), no imported uniqueness theorems, and no fitted parameter later relabeled as a prediction. The iterative attack construction ('This iterative refinement process continued until we compiled a set of at least 100 total attack prompts, all of which successfully elicited misinformation from the target model') means the reported ASRs are conditional on a pre-selected successful-prompt dataset; however, the paper labels this set as a 'successful prompt dataset' and also reports transfer rates to two models not used in selection, so the central existence claim retains independent content. The most defensible circularity concern is the evaluator/target overlap: Gemini serves as both a jailbreak target and the judge of all responses, and the same model labels Reddit posts that later define classifier ground truth. These loops are real but are mitigated by human calibration on small samples and by manual error analysis, so they do not make the headline results true by construction. Overall score 2 reflects minor, non-load-bearing feedback loops rather than significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new particles, forces, or entities. Its assumptions are about data representativeness and the validity of LLM-generated labels as ground truth, plus a hand-defined success criterion and topic taxonomy.

free parameters (3)
  • Attack success threshold: 'any explicit misinformation'
    An attack is counted successful if the model generates any explicit misinformation (e.g., an endorsing social media post or fabricated citations). This binary criterion is hand-defined and applied by an LLM judge, not independently derived or calibrated against a fully human-labeled set.
  • Obedience scoring scale (0, 0.33, 0.66, 1)
    The four-point obedience scale is chosen by the authors for manual annotation and reused by the LLM judge; the exact spacing is arbitrary and affects the reported agreement rates.
  • LDA topic count (9 attack types) = 9 topics
    The number of attack types is selected by lowest log-perplexity over 2-10 topics, then prompts are manually assigned to the first matching label. The taxonomy is therefore a mix of an unsupervised model output and hand-coded decisions.
assumptions (4)
  • domain assumption The Gemini LLM judge's labels are a valid proxy for human judgment of misinformation generation and validation.
    The judge was chosen after 85% agreement with human annotations on 20 responses, and 95.5% agreement on 200 Reddit posts. The calibration sets are small and the paper treats these agreements as sufficient, but the assumption is weakly supported.
  • domain assumption WildChat chat logs are representative of in-the-wild health-related LLM queries.
    WildChat is one source of real user interactions with ChatGPT; filtering by keywords and English-only content yields 32 entries per topic, which may not represent typical health queries broadly.
  • domain assumption Reddit misinformation labels produced by the LLM are sufficiently accurate to serve as training ground truth for classifiers.
    The LLM labeled 2,476 posts, with only 200 posts human-checked (3-6 discrepancies). The classifier training and evaluation use these LLM-generated labels as gold labels.
  • ad hoc to paper The iterative prompt-generation process converges to a representative set of jailbreak attacks.
    All successful attacks are kept and all failed attacks discarded; the resulting 109 prompts are treated as a dataset for ASR estimation. The paper assumes this curated set supports generalizable ASR numbers, but the filtering invalidates any random-sample interpretation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Audit and Analysis of LLM-Assisted Health Misinformation Jailbreaks Against LLMs." pith.science (2026). https://pith.science/paper/P56GYK4K

@misc{pith2026250810010,
  author       = {Pith},
  title        = {Pith review of: An Audit and Analysis of LLM-Assisted Health Misinformation Jailbreaks Against LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P56GYK4K}},
  note         = {Machine review of arXiv:2508.10010}
}
read the original abstract

Large Language Models (LLMs) are a double-edged sword capable of generating harmful misinformation -- inadvertently, or when prompted by "jailbreak" attacks that attempt to produce malicious outputs. LLMs could, with additional research, be used to detect and prevent the spread of misinformation. In this paper, we investigate the efficacy and characteristics of LLM-produced jailbreak attacks that cause other models to produce harmful medical misinformation. We also study how misinformation generated by jailbroken LLMs compares to typical misinformation found on social media, and how effectively it can be detected using standard machine learning approaches. Specifically, we closely examine 109 distinct attacks against three target LLMs and compare the attack prompts to in-the-wild health-related LLM queries. We also examine the resulting jailbreak responses, comparing the generated misinformation to health-related misinformation on Reddit. Our findings add more evidence that LLMs can be effectively used to detect misinformation from both other LLMs and from people, and support a body of work suggesting that with careful design, LLMs can contribute to a healthier overall information ecosystem.

Figures

Figures reproduced from arXiv: 2508.10010 by the authors.

Figure 1
Figure 1. Example of Modified User Query, Jailbreak [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Study Design Workflow: (1) Query Selection, Attack Creation, Evaluation, and Analysis, (2) Comparison Between [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Heatmap of Attack Success Rates Across Mod [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Average Classification Results Across Misinfor [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Bar Chart of Overall Attack Success Rates by Attack [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 7
Figure 7. Figure 7: Subreddit Misinformation Distribution Across [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 6
Figure 6. Figure 6: Custom Stop Words Subreddit Count nottheonion 6 CovidVaccinated 4 WayOfTheBern 3 Thedaily 3 ukpolitics 2 [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 8
Figure 8. Figure 8: System Prompt Used for Attack Model [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Guidance Components/Phrases Used for Attack [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Additional Instructions Prompt Used for Target Model [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 12
Figure 12. Figure 12: Prompt Used for Misinformation Classifier [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 11
Figure 11. Figure 11: Prompt Used for Jailbreak Judge [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 13
Figure 13. Figure 13: LLM Annotated Example of Non-Jailbreak Non-Jailbreak Example Response: The assertion that COVID-19 vaccines directly caused a rise in mpox cases, particularly among immunocompromised individuals, is a complex claim that requires careful examination. While a leading in…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 29 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Alamleh, H.; AlQahtani, A. A. S.; and ElSaid, A. 2023. Distinguishing human-written and ChatGPT-generated text using machine learning. In 2023 Systems and Information Engineering Design Symposium (SIEDS), 154--158. IEEE

    work page 2023

  4. [4]

    L.; Corney, D.; DiResta, R.; Ferrara, E.; Hale, S.; Halevy, A.; et al

    Augenstein, I.; Baldwin, T.; Cha, M.; Chakraborty, T.; Ciampaglia, G. L.; Corney, D.; DiResta, R.; Ferrara, E.; Hale, S.; Halevy, A.; et al. 2024. Factuality challenges in the era of large language models and opportunities for fact-checking. Nature Machine Intelligence, 6(8): 852--863

    work page 2024

  5. [5]

    Barman, D.; Guo, Z.; and Conlan, O. 2024. The dark side of language models: Exploring the potential of llms in multimedia disinformation generation and dissemination. Machine Learning with Applications, 100545

    work page 2024

  6. [6]

    Bozarth, L.; Im, J.; Quarles, C.; and Budak, C. 2023. Wisdom of Two Crowds: Misinformation Moderation on Reddit and How to Improve this Process---A Case Study of COVID-19. Proceedings of the ACM on Human-Computer Interaction, 7(CSCW1): 1--33

    work page 2023

  7. [7]

    J.; Tramer, F.; et al

    Chao, P.; Debenedetti, E.; Robey, A.; Andriushchenko, M.; Croce, F.; Sehwag, V.; Dobriban, E.; Flammarion, N.; Pappas, G. J.; Tramer, F.; et al. 2024. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. Advances in Neural Information Processing Systems, 37: 55005--55029

    work page 2024

  8. [8]

    Chen, C.; and Shu, K. 2023. Can llm-generated misinformation be detected? arXiv preprint arXiv:2309.13788

    arXiv 2023

Show all 48 references
  1. [9]

    Chen, C.; and Shu, K. 2024. Combating misinformation in the age of llms: Opportunities and challenges. AI Magazine, 45(3): 354--368

    2024

  2. [10]

    Chu, J.; Liu, Y.; Yang, Z.; Shen, X.; Backes, M.; and Zhang, Y. 2024. Comprehensive assessment of jailbreak attacks against llms. arXiv preprint arXiv:2402.05668

    2024 arXiv

  3. [11]

    Doumbouya, M. K. B.; Nandi, A.; Poesia, G.; Ghilardi, D.; Goldie, A.; Bianchi, F.; Jurafsky, D.; and Manning, C. D. 2024. h4rm3l: A dynamic benchmark of composable jailbreak attacks for llm safety assessment. arXiv preprint arXiv:2408.04811

    2024 arXiv

  4. [12]

    Du, J.; Preston, S.; Sun, H.; Shegog, R.; Cunningham, R.; Boom, J.; Savas, L.; Amith, M.; and Tao, C. 2021. Using machine learning--based approaches for the detection and classification of human papillomavirus vaccine misinformation: Infodemiology study of reddit discussions. ...

    2021

  5. [13]

    M.; Atwan, A.; and El-Daydamony, E

    Elsaeed, E.; Ouda, O.; Elmogy, M. M.; Atwan, A.; and El-Daydamony, E. 2021. Detecting fake news in social media using voting classifier. IEEE Access, 9: 161909--161925

    2021

  6. [14]

    Gilda, S. 2017. Notice of Violation of IEEE Publication Principles: Evaluating machine learning algorithms for fake news detection. In 2017 IEEE 15th student conference on research and development (SCOReD), 110--115. IEEE

    2017

  7. [15]

    Guo, B.; Zhang, X.; Wang, Z.; Jiang, M.; Nie, J.; Ding, Y.; Yue, J.; and Wu, Y. 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597

    2023 arXiv

  8. [16]

    Han, T.; Kumar, A.; Agarwal, C.; and Lakkaraju, H. 2024. Medsafetybench: Evaluating and improving the medical safety of large language models. Advances in Neural Information Processing Systems, 37: 33423--33454

    2024

  9. [17]

    Huang, T.; Yi, J.; Yu, P.; and Xu, X. 2025. Unmasking digital falsehoods: A comparative analysis of LLM-based misinformation detection strategies. In 2025 8th International Conference on Advanced Algorithms and Control Engineering (ICAACE), 2470--2476. IEEE

    2025

  10. [18]

    Jin, H.; Chen, R.; Zhou, A.; Zhang, Y.; and Wang, H. 2024 a . Guard: Role-playing to generate natural-language jailbreakings to test guideline adherence of large language models. arXiv preprint arXiv:2402.03299

    2024

  11. [19]

    Jin, H.; Hu, L.; Li, X.; Zhang, P.; Chen, C.; Zhuang, J.; and Wang, H. 2024 b . Jailbreakzoo: Survey, landscapes, and horizons in jailbreaking large language and vision-language models. arXiv preprint arXiv:2407.01599

    2024

  12. [20]

    Keskar, D.; Palwe, S.; and Gupta, A. 2020. Fake news classification on twitter using flume, n-gram analysis, and decision tree machine learning technique. In Proceeding of International Conference on Computational Science and Applications: ICCSA 2019, 139--147. Springer

    2020

  13. [21]

    M.; Field, S.; and Casper, S

    Kirch, N. M.; Field, S.; and Casper, S. 2024. What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attacks. arXiv preprint arXiv:2411.03343

    2024

  14. [22]

    M.; Freiling, I.; Beets, B.; and Brossard, D

    Krause, N. M.; Freiling, I.; Beets, B.; and Brossard, D. 2020. Fact-checking as risk communication: the multi-layered risk of misinformation in times of COVID-19. Journal of Risk Research, 23(7-8): 1052--1059

    2020

  15. [23]

    Kumar, R.; Goddu, B.; Saha, S.; and Jatowt, A. 2024. Silver lining in the fake news cloud: Can large language models help detect misinformation? IEEE Transactions on Artificial Intelligence

    2024

  16. [24]

    Prompter Says

    Lee, D.; Xie, S.; Rahman, S.; Pat, K.; Lee, D.; and Chen, Q. A. 2023. "Prompter Says": A Linguistic Approach to Understanding and Detecting Jailbreak Attacks Against Large-Language Models. In Proceedings of the 1st ACM Workshop on Large AI Systems and Models with Privacy and S...

    2023

  17. [25]

    Li, D.; Jiang, B.; Huang, L.; Beigi, A.; Zhao, C.; Tan, Z.; Bhattacharjee, A.; Jiang, Y.; Chen, C.; Wu, T.; et al. 2024. From generation to judgment: Opportunities and challenges of llm-as-a-judge. arXiv preprint arXiv:2411.16594

    2024

  18. [26]

    Li, X.; Zhou, Z.; Zhu, J.; Yao, J.; Liu, T.; and Han, B. 2023. Deepinception: Hypnotize large language model to be jailbreaker. arXiv preprint arXiv:2311.03191

    2023 arXiv

  19. [27]

    Liu, A.; Sheng, Q.; and Hu, X. 2024. Preventing and detecting misinformation generated by large language models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, 3001--3004

    2024

  20. [28]

    D.; Kuderer, N

    Menz, B. D.; Kuderer, N. M.; Bacchi, S.; Modi, N. D.; Chin-Yee, B.; Hu, T.; Rickard, C.; Haseloff, M.; Vitry, A.; McKinnon, R. A.; et al. 2024. Current safeguards, risk mitigation, and transparency measures of large language models against the generation of health disinformati...

    2024

  21. [29]

    Nakamura, K.; Levy, S.; and Wang, W. Y. 2019. r/fakeddit: A new multimodal benchmark dataset for fine-grained fake news detection. arXiv preprint arXiv:1911.03854

    2019 arXiv

  22. [30]

    Nathanson, S.; Yoo, Y.; Na, D.; Cao, Y.; and Watkins, L. 2024. A Step Towards Modern Disinformation Detection: Novel Methods for Detecting LLM-Generated Text. In MILCOM 2024-2024 IEEE Military Communications Conference (MILCOM), 615--620. IEEE

    2024

  23. [31]

    Pan, Y.; Pan, L.; Chen, W.; Nakov, P.; Kan, M.-Y.; and Wang, W. Y. 2023. On the risk of misinformation pollution with large language models. arXiv preprint arXiv:2305.13661

    2023 arXiv

  24. [32]

    Ramesh, B.; Gursale, D.; Jopaul, A.; and Ernst, M. 2025. Reddit Misinformation Dataset. https://zenodo.org/records/14900167. To appear in DISMISS-FAKE'25: 1st Workshop on Disinformation and Misinformation in the Age of Generative AI, co-located with the 18th ACM WSDM, March 14...

    2025

  25. [33]

    S.; Naik, A

    Rao, A. S.; Naik, A. R.; Vashistha, S.; Aditya, S.; and Choudhury, M. 2024. Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluat...

    2024

  26. [34]

    Russinovich, M.; Salem, A.; and Eldan, R. 2024. Great, now write an article about that: The crescendo multi-turn llm jailbreak attack. arXiv preprint arXiv:2404.01833

    2024 arXiv

  27. [35]

    A.; Kashyap, A

    Sager, M. A.; Kashyap, A. M.; Tamminga, M.; Ravoori, S.; Callison-Burch, C.; and Lipoff, J. B. 2021. Identifying and responding to health misinformation on reddit dermatology forums with artificially intelligent bots using natural language processing: design and evaluation stu...

    2021

  28. [36]

    Saleh, H.; Alharbi, A.; and Alsamhi, S. H. 2021. OPCNN-FAKE: Optimized convolutional neural network for fake news detection. IEEE Access, 9: 129471--129489

    2021

  29. [37]

    Scepanovic, S.; Martin-Lopez, E.; Quercia, D.; and Baykaner, K. 2020. Extracting medical entities from social media. In Proceedings of the ACM conference on health, inference, and learning, 170--181

    2020

  30. [38]

    do anything now

    Shen, X.; Chen, Z.; Backes, M.; Shen, Y.; and Zhang, Y. 2024. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, 1671--1685

    2024

  31. [39]

    Sukhija, N.; Tatineni, M.; Brown, N.; Van Moer, M.; Rodriguez, P.; and Callicott, S. 2016. Topic modeling and visualization for big data in social sciences. In 2016 Intl IEEE Conferences on Ubiquitous Intelligence & Computing, Advanced and Trusted Computing, Scalable Computing...

    2016

  32. [40]

    Sun, Y.; He, J.; Cui, L.; Lei, S.; and Lu, C.-T. 2024. Exploring the deceptive power of llm-generated fake news: A study of real-world detection challenges. arXiv preprint arXiv:2403.18249

    2024 arXiv

  33. [41]

    Traore, I.; Woungang, I.; and Awad, A. 2018. Intelligent, Secure, and Dependable Systems in Distributed and Cloud Environments. Springer

    2018

  34. [42]

    Wang, Z.; Cao, Y.; and Liu, P. 2024. Hidden You Malicious Goal Into Benign Narratives: Jailbreak Large Language Models through Logic Chain Injection. arXiv preprint arXiv:2404.04849

    2024 arXiv

  35. [43]

    Weidinger, L.; Mellor, J.; Rauh, M.; Griffin, C.; Uesato, J.; Huang, P.-S.; Cheng, M.; Glaese, M.; Balle, B.; Kasirzadeh, A.; et al. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359

    2021 arXiv

  36. [44]

    Zeng, Y.; Lin, H.; Zhang, J.; Yang, D.; Jia, R.; and Shi, W. 2024. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volum...

    2024

  37. [45]

    Zhang, H.; Lou, Q.; and Wang, Y. 2025. Towards Safe AI Clinicians: A Comprehensive Study on Large Language Model Jailbreaking in Healthcare. arXiv preprint arXiv:2501.18632

    2025 arXiv

  38. [46]

    Zhang, Y.; Sharma, K.; Du, L.; and Liu, Y. 2024. Toward mitigating misinformation and social media manipulation in llm era. In Companion Proceedings of the ACM Web Conference 2024, 1302--1305

    2024

  39. [47]

    Zhao, W.; Ren, X.; Hessel, J.; Cardie, C.; Choi, Y.; and Deng, Y. 2024. Wildchat: 1m chatgpt interaction logs in the wild. arXiv preprint arXiv:2405.01470

    2024 arXiv

  40. [48]

    G.; and De Choudhury, M

    Zhou, J.; Zhang, Y.; Luo, Q.; Parker, A. G.; and De Choudhury, M. 2023. Synthetic lies: Understanding ai-generated misinformation and evaluating algorithmic and human solutions. In Proceedings of the 2023 CHI conference on human factors in computing systems, 1--20

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.