Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

LLMs are Vulnerable to Malicious Prompts Disguised as Scientific Language

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Prompting LLMs with scientific-sounding summaries of (even fabricated) studies makes them produce markedly more biased and toxic responses.

desk verdict A real and reproducible scientific-prompt jailbreak vector, with a useful fabricated-paper variant; the quantitative support needs variance reporting and a stronger judge-validation check before trusting the exact numbers. read the letter →

arxiv 2501.14073 v2 pith:XDJ5BKN4 submitted 2025-01-23 cs.CL

Yubin Ge , Neeraja Kirtane , Hao Peng , Dilek Hakkani-Tür This is my paper · ORCID
classification cs.CL
keywords LLMjailbreakpersuasionattackstereotypicalbiasscientificlanguagepaperfabricationtoxicityauthorityprinciplemulti-turndialogue
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a scientific veneer is itself a jailbreak vector for current large language models. The authors build prompts that summarize real psychology and social-science papers as evidence that stereotypes are beneficial, then ask models to produce follow-up responses with strong, harmful bias. Across GPT-4o, GPT-4, Llama 3.1, Gemini, and Cohere models, bias and toxicity scores rise substantially compared with zero-shot, role-play, or DAN baselines. They also show the attack needs no scholarly knowledge: an LLM can fabricate fake paper titles and abstracts, and those fabricated summaries persuade target models nearly as well. If true, scientific-sounding authority, not logical argument, is enough to override safety training.

What carries the argument

The mechanism is a two-stage persuasion pipeline built on the authority principle from persuasion research. Stage one produces a persuasive scientific summary: real papers, or LLM-fabricated research ideas with added venue and author metadata, are summarized by GPT-4o into a multi-document summary of the benefits of stereotypical bias. Stage two places that summary in the system message and instructs the target model to write a follow-up response containing strong, harmful bias, including a chain-of-thought rationale explaining why the bias is beneficial. The summary is the load-bearing object: it converts an otherwise-refused request into an authoritative scientific context that the target model complies with.

What would settle it

Score the target models' outputs with human raters who are blind to whether the generating prompt contained scientific citations, and compare their scores with GPT-4's. If blind human scores show no increase under the scientific condition, the reported vulnerability is largely an artifact of the judge rather than a property of the target models.

Watch

Extended reading notes

Core claim

In the authors' telling, the central discovery is that authority cues carried by scientific language can be weaponized: real studies on stereotype accuracy, cognitive heuristics, and stereotype boost are deliberately misread as endorsements of harmful bias, summarized by an LLM, and placed in the system message. Target models then generate biased continuations in a multi-turn dialogue, with average bias scores on a 0-4 scale rising from 0 under baselines to values around 1.1-3.3 depending on model and dataset, and toxicity also increasing. When the summary is instead built from fabricated papers invented by GPT-4o, with plausible author and venue metadata added, the effect persists and sometimes grows (for example, GPT-4o's bias score rises from 1.71 to 2.56 on StereoSet). The paper further reports that omitting author names or venues lowers persuasion for some models, that bias compounds over dialogue turns, and that two mutation-based defenses, rephrasing and retokenizing, fail to neutralize the attack.

Load-bearing premise

The central evidence assumes that GPT-4's bias-score judgment is impartial; if the scientific framing of the prompt also sways the judge, the reported rise in bias could be partly a measurement artifact.

Editorial extensions

If this is right

  • Scientific-sounding context alone is enough to bypass safety training that resists direct requests, role-play, and DAN-style jailbreaks in several current models.
  • The attack can be automated end-to-end: a single LLM can invent the fake papers, summaries, and metadata, so no specialist knowledge or curated papers are required.
  • Adding credibility metadata such as author names and venues strengthens the effect for some models, so removing that metadata is not a reliable defense.
  • Bias escalates across dialogue turns, meaning multi-turn conversations do not currently re-anchor the model to neutral behavior.
  • Rephrasing and retokenization defenses are ineffective and can even raise bias scores, so standard prompt-mutation defenses do not blunt this vector.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's claims, the same pipeline may transfer to other authoritative-sounding text domains, such as legal, medical, or financial documents; testing those corpora would show whether the vulnerability is specifically about science or about authority cues in general.
  • Because the key measurement uses GPT-4 as a judge of bias, a blind human-rater evaluation could separate whether the scientific framing changes the target model's behavior or merely shifts how the judge scores that behavior.
  • If scientific corpora reinforce this vulnerability, safety training that teaches models to describe literature without endorsing its conclusions might generalize better than prompt filters; that is a testable mitigation direction.
  • The fabricated-paper result suggests citation-verification tools, which check whether cited studies actually exist, could serve as a concrete defense even though the paper only proposes fact-checking in general terms.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces a persuasion-based jailbreaking method that hides malicious requests behind scientific language. The authors collect real papers (and, in a second study, LLM-fabricated paper titles and abstracts) that discuss potential benefits of stereotypical bias, summarize them, and prepend the summary to prompts asking target LLMs to generate follow-up responses with strong, harmful stereotypical bias. They evaluate seven model families (GPT-4o, GPT-4o-mini, GPT-4, Llama3.1-70B/405B, Gemini, Cohere) on a balanced subset of StereoSet and on 200 GPT-4-generated neutral sentences. Bias is scored by GPT-4 using a five-level rubric from Kumar et al. (2024), with a 100-response human validation yielding Cohen's kappa 0.88; toxicity is scored with the Perspective API. The reported results show that scientific-language prompts raise bias scores from near zero in baselines to approximately 1.1-3.3 and also increase toxicity, that ablation studies suggest author names and venues contribute to persuasiveness, that bias scores increase over multi-turn dialogues, and that two mutation-based defenses are largely ineffective. The paper concludes that LLMs are vulnerable to malicious requests disguised as scientific language.

Significance. If the results are robust, the paper identifies a practically important and partially automatable jailbreak vector: exploiting LLMs' tendency to treat scientific-sounding text as authoritative evidence. The work has several strengths: the bias rubric is taken from an external benchmark (Kumar et al., 2024), the human validation reports a high kappa, the evaluation spans multiple proprietary and open-weight model families, and the defense evaluation is useful for safety practitioners. The paper also explicitly provides prompts and examples in appendices, supporting reproducibility of the attack pipeline. However, the quantitative claims currently rely on a single LLM judge with limited validation, on tables that report only means without variance or significance tests, and on a dataset subset that is not fully specified. These gaps are load-bearing for the central claim that scientific language causes a substantial increase in bias and toxicity.

major comments (5)
  1. [§3.2, Evaluation Metrics] The central quantitative evidence for the paper's claim rests on GPT-4 judging bias severity, yet the only validity check is a 100-response comparison with human annotations (Cohen's kappa 0.88). This does not establish condition-blind validity: Sci-Paper responses are likely longer, more hedged, and more academically phrased than baseline responses, and if GPT-4's scoring is sensitive to register features, the reported condition effect could be an artifact of the judge rather than of actual stereotyping. The kappa only shows that human and GPT-4 scores correlate on the same 100 items; it does not show that a human reader blind to condition would rate the full set similarly. Please provide a stratified, condition-blind human evaluation over all conditions and target models, or otherwise rule out this same-family and register confound.
  2. [Tables 2-5 and Figure 7] All quantitative comparisons report only mean bias and toxicity scores, with no standard deviations, confidence intervals, sample sizes, or significance tests. Consequently, statements such as 'bias scores substantially increase' (§3.3), 'scores remain generally comparable' (§4.2), and 'bias scores consistently increase as the conversation progresses' (§5.2) are not statistically supported. For example, the difference between GPT-4o at 1.71 and Llama3.1-70B at 1.70 in Table 2 could be within noise, and the defense results in Table 5 (e.g., GPT-4o-mini 2.59 vs 2.74 vs 2.65) show small changes that are reported as meaningful. Please report variance and run appropriate significance tests, or qualify the claims accordingly.
  3. [§3.2, Dataset] The 'balanced subset' of StereoSet is not described: the number of instances per bias category, the selection criterion for balance, and whether the same subset is used across all models and conditions are all unspecified. The 200 GPT-4-generated neutral sentences are also not released or exemplified. Without this information, the experimental results cannot be reproduced or compared across settings, and the per-category scores in Tables 2-3 may be based on very different sample sizes. Please specify the subset construction and include the generated sentences or a full sample.
  4. [§4.1, Fabricated Paper Based Persuasion] The abstract and introduction state that the attack can be automated by asking an LLM to invent fake papers, but the method as described says 'we manually add top venues in the relevant fields and notable researchers as authors.' This manual step is part of the evaluated pipeline, so the automation claim is not actually tested. If the claim of systematic automated jailbreaking is central, the authors should either evaluate a fully LLM-generated version (including metadata) or explicitly revise the claim to reflect the human-in-the-loop component.
  5. [Table 3, Fabricated Paper row for Llama3.1-70B-Instruct] This cell reports a bias score of 0.26, which is dramatically lower than every other condition in the table and is not discussed anywhere in the text. This outlier could indicate a compliance failure, a prompt-format issue, or a genuine model behavior difference; without analysis it undermines the cross-model generalization claim and may signal that the fabricated-paper effect is not uniformly present. Please examine this case and report why the score is so low.
minor comments (5)
  1. [Abstract and §3.2 Target Models] The abstract and Section 3.2 list GPT-o1 and Claude as target models, but the results tables do not include them; the text later explains that these models refused to generate summaries. Please state this exclusion explicitly in the abstract and in the model list description.
  2. [Table 5 and model naming] The model is referred to as 'Llama3-405B' in Table 5 but as 'Llama3.1-405B-Instruct' elsewhere; please unify the naming.
  3. [§5.1, Figures 5 and 6] The ablation study reports label distributions but does not provide numeric bias-score means or per-variant counts. Please add the underlying numbers or a table with the mean scores, so the reader can quantify the ablation effects.
  4. [§5.2, Figure 7] The x-axis is described only as 'turn,' and the role-switching protocol between user and assistant is not fully defined. Please clarify how turns are numbered and what the error bars (if any) represent; currently the figure has no error bars or confidence intervals.
  5. [§9 Limitations] The limitations section focuses on future model updates but does not discuss the judge bias limitation, the missing statistical testing, or the human-in-the-loop metadata component of Study II. Adding these would give readers a more complete picture of the evidence's reliability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the measured bias increase is an empirical outcome, not an artifact of construction, fitting, or self-citation.

full rationale

The paper's central claim is an empirical comparison: Sci-Paper persuasion against zero-shot, DAN, and role-playing baselines, with bias scored by GPT-4 under a rubric taken from Kumar et al. (2024) and validated against 100 human annotations (Cohen's kappa = 0.88, Section 3.2). No parameter is fitted to the outcome and no result is defined in terms of the quantity it is supposed to explain. The fabricated-paper condition also starts from an explicitly generated summary and then measures whether target models comply, so the summary is an experimental input, not a hidden refit. The only self-citations (Jeoung et al. 2023 for background on LLM bias; Ge et al. 2021 for scientific-text generation) are contextual and do not carry the jailbreak argument. The LLM-as-judge design could be a validity or same-family measurement concern, but that is an empirical confound, not circularity: a biased judge would produce inaccurate measurements, not a conclusion that is true by construction. Since the evaluation is grounded in an external rubric, human annotation, and an external toxicity API, the derivation chain is self-contained.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

This is an empirical vulnerability study with no fitted free parameters and no invented entities. The central claim rests on domain assumptions about persuasion (authority principle), on the validity of the automated bias judge, and on the representativeness of the selected papers and compliant models.

assumptions (4)
  • domain assumption The authority principle of persuasion applies to LLMs: scientific text is perceived as authoritative and persuasive.
    Section 1 motivates the attack using Cialdini's authority principle; if LLMs do not weight authority cues, the attack would not work.
  • domain assumption GPT-4-as-judge bias scores are a valid measure of stereotypical bias.
    Section 3.2 adopts LLM-as-judge from prior work and validates on 100 human labels with kappa 0.88; this validation is small and may not generalize across models and conditions.
  • domain assumption The selected papers genuinely discuss potential benefits of stereotypical bias, and the GPT-4o summary reflects that bent.
    Section 3.1 uses multi-document summarization of papers in Appendix A; the summary deliberately focuses on advantages.
  • domain assumption Compliant models are representative of 'many state-of-the-art LLMs'; refusals by Claude, o1, and Gemini do not undermine the generalizability claim.
    Results are averaged over compliant models; the paper reports refusals but still generalizes to 'many state-of-the-art LLMs'.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLMs are Vulnerable to Malicious Prompts Disguised as Scientific Language." pith.science (2026). https://pith.science/paper/XDJ5BKN4

@misc{pith2026250114073,
  author       = {Pith},
  title        = {Pith review of: LLMs are Vulnerable to Malicious Prompts Disguised as Scientific Language},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XDJ5BKN4}},
  note         = {Machine review of arXiv:2501.14073}
}
read the original abstract

As large language models (LLMs) have been deployed in various real-world settings, concerns about the harm they may propagate have grown. Various jailbreaking techniques have been developed to expose the vulnerabilities of these models and improve their safety. This work reveals that many state-of-the-art LLMs are vulnerable to malicious requests hidden behind scientific language. Specifically, our experiments with GPT4o, GPT4o-mini, GPT-4, LLama3-405B-Instruct, Llama3-70B-Instruct, Cohere, Gemini models demonstrate that, the models' biases and toxicity substantially increase when prompted with requests that deliberately misinterpret social science and psychological studies as evidence supporting the benefits of stereotypical biases. Alarmingly, these models can also be manipulated to generate fabricated scientific arguments claiming that biases are beneficial, which can be used by ill-intended actors to systematically jailbreak these strong LLMs. Our analysis studies various factors that contribute to the models' vulnerabilities to malicious requests in academic language. Mentioning author names and venues enhances the persuasiveness of models, and the bias scores increase as dialogues progress. Our findings call for a more careful investigation on the use of scientific data for training LLMs.

Figures

Figures reproduced from arXiv: 2501.14073 by the authors.

Figure 1
Figure 1. An overview of our approach. Our motivation [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The prompt to generate a summary of schol [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The prompt to persuade a target LLM to gener [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: The prompt for generating new research ideas [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Label distributions of results on GPT-4o for original Sci-Paper Based Persuasion and its three variants: [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Label distributions of results on GPT-4o-mini for original Sci-Paper Based Persuasion and its three [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: The average bias scores of each turn for Sci [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 9
Figure 9. Figure 9: Examples of outputs by the Cohere model (command-r-plus) when prompted to generate toxic con￾tent. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Example of the outputs by GPT-4o to gener [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: One example generated dialogue showing gender bias. The dialogue starts from a context sentence from StereoSet and the model is persuaded to generate follow-up responses with stereotypical bias. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Original prompt to jailbreak the model [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Rephrased prompt as a defense strategy [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Retokenized prompt as a defense strategy. [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Technical Survey of Reinforcement Learning Techniques for Large Language Models

    cs.AI 2025-07 conditional novelty 3.0 of 10

    A survey of RL methods for LLMs that organizes the field by reward modeling, feedback source, and optimization strategy, with benchmark tables favoring a scalar-regression UNA variant over DPO and KTO in offline alignment.

Reference graph

Works this paper leans on

55 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big?��. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610--623

    work page 2021

  3. [3]

    Anjanava Biswas and Wrick Talukdar. 2023. Guardrails for trust, safety, and ethical development and deployment of large language models (llm). Journal of Science & Technology, 4(6):55--82

    work page 2023

  4. [4]

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2023. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419

    arXiv 2023

  5. [5]

    Robert B Cialdini. 2007. Influence: The psychology of persuasion, volume 55. Collins New York

    work page 2007

  6. [6]

    Alexander M Czopp, Aaron C Kay, and Sapna Cheryan. 2015. Positive stereotypes are pervasive and powerful. Perspectives on Psychological Science, 10(4):451--463

    work page 2015

  7. [7]

    Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. 2023. Safe rlhf: Safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations

    work page 2023

  8. [8]

    Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. 2023 a . https://api.semanticscholar.org/CorpusID:259951184 Masterkey: Automated jailbreaking of large language model chatbots . Proceedings 2024 Network and Distributed System Security Symposium

    work page 2023

Show all 55 references
  1. [9]

    Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. 2024. Masterkey: Automated jailbreaking of large language model chatbots. In Proc. ISOC NDSS

    2024

  2. [10]

    Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. 2023 b . https://api.semanticscholar.org/CorpusID:263831094 Multilingual jailbreak challenges in large language models . ArXiv, abs/2310.06474

    2023 arXiv

  3. [11]

    Deshpande, Vishvak Murahari, Tanmay Rajpurohit, A

    A. Deshpande, Vishvak Murahari, Tanmay Rajpurohit, A. Kalyan, and Karthik Narasimhan. 2023. https://api.semanticscholar.org/CorpusID:258060002 Toxicity in chatgpt: Analyzing persona-assigned language models . ArXiv, abs/2304.05335

    2023 arXiv

  4. [12]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  5. [13]

    Nitya Ann Eapen. 2024. Stereotype: Cognition and biases. International Journal of Indian Psychȯlogy, 12(1)

    2024

  6. [14]

    David Esiobu, Xiaoqing Ellen Tan, Saghar Hosseini, Megan Ung, Yuchen Zhang, Jude Fernandes, Jane Dwivedi-Yu, Eleonora Presani, Adina Williams, and Eric Michael Smith. 2023. https://api.semanticscholar.org/CorpusID:265506521 Robbie: Robust bias evaluation of large generative la...

    2023

  7. [15]

    Lea Frermann and Alexandre Klementiev. 2019. Inducing document structure for aspect-based summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6263--6273

    2019

  8. [16]

    Yubin Ge, Ly Dinh, Xiaofeng Liu, Jinsong Su, Ziyao Lu, Ante Wang, and Jana Diesner. 2021. Baco: A background knowledge-and content-based framework for citing sentence generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the...

    2021

  9. [17]

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462

    2020 arXiv

  10. [18]

    Nianlong Gu and Richard Hahnloser. 2024. Controllable citation sentence generation with language models. In Proceedings of the Fourth Workshop on Scholarly Document Processing (SDP 2024), pages 22--37

    2024

  11. [19]

    Deshpande, A

    Shashank Gupta, Vaishnavi Shrivastava, A. Deshpande, A. Kalyan, Peter Clark, Ashish Sabharwal, and Tushar Khot. 2023. https://api.semanticscholar.org/CorpusID:265050702 Bias runs deep: Implicit reasoning biases in persona-assigned llms . ArXiv, abs/2311.04892

    2023 arXiv

  12. [20]

    Perry Hinton. 2017. Implicit stereotypes and the predictive brain: cognition and culture in “biased” person perception. Palgrave Communications, 3(1):1--9

    2017

  13. [21]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  14. [22]

    Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614

    2023 arXiv

  15. [23]

    Sullam Jeoung, Yubin Ge, and Jana Diesner. 2023. S tereo M ap: Quantifying the awareness of human-like stereotypes in large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore. Association for Computational Linguistics

    2023

  16. [24]

    Bilal Khan, Zohaib Ali Shah, Muhammad Usman, Inayat Khan, and Badam Niazi. 2023. Exploring the landscape of automatic text summarization: a comprehensive survey. IEEE Access

    2023

  17. [25]

    Hadas Kotek, Rikker Dockum, and David Sun. 2023. Gender bias and stereotypes in large language models. In Proceedings of the ACM collective intelligence conference, pages 12--24

    2023

  18. [26]

    Shachi H Kumar, Saurav Sahay, Sahisnu Mazumder, Eda Okur, Ramesh Manuvinakurike, Nicole Beckage, Hsuan Su, Hung-yi Lee, and Lama Nachman. 2024. Decoding biases: Automated methods and llm judges for gender bias detection in language models. arXiv preprint arXiv:2408.03907

    2024 arXiv

  19. [27]

    Kiho Lee. 2023. https://doi.org/10.5281/zenodo.1234 ChatGPT\_DAN

    2023 doi

  20. [28]

    Yueh-Ting Lee. 2011. Social psychology of stereotyping and human difference appreciation. Diversity management: Theoretical perspectives, practical approaches, and academic leadership, pages 33--46

    2011

  21. [29]

    Congbo Ma, Wei Emma Zhang, Mingyu Guo, Hu Wang, and Quan Z Sheng. 2022. Multi-document summarization via deep learning techniques: A survey. ACM Computing Surveys, 55(5):1--37

    2022

  22. [30]

    Weicheng Ma, Brian Chiang, Tong Wu, Lili Wang, and Soroush Vosoughi. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.575 Intersectional stereotypes in large language models: Dataset and analysis . In Findings of the Association for Computational Linguistics: EMNLP 2023, ...

    2023 doi

  23. [31]

    Clark R McCauley, Lee J Jussim, and Yueh-Ting Lee. 1995. Stereotype accuracy: Toward appreciating group differences. American Psychological Association

    1995

  24. [32]

    Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. Stereoset: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.09456

    2020 arXiv

  25. [33]

    Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2023. https://arxiv.org/abs/2310.03693 Fine-tuning aligned language models compromises safety, even when users do not intend to! Preprint, arXiv:2310.03693

    2023 arXiv

  26. [34]

    Mark Russinovich, Ahmed Salem, and Ronen Eldan. 2024. https://api.semanticscholar.org/CorpusID:268856920 Great, now write an article about that: The crescendo multi-turn llm jailbreak attack . ArXiv, abs/2404.01833

    2024 arXiv

  27. [35]

    Rusheb Shah, Quentin Feuillade-Montixi, Soroush Pour, Arush Tagade, Stephen Casper, and Javier Rando. 2023. Scalable and transferable black-box jailbreaks for language models via persona modulation. ArXiv, abs/2311.03348

    2023 arXiv

  28. [36]

    Margaret Shih, Nalini Ambady, Jennifer A Richeson, Kentaro Fujita, and Heather M Gray. 2002. Stereotype performance boosts: the impact of self-relevance and the manner of stereotype activation. Journal of Personality and social psychology, 83(3):638

    2002

  29. [37]

    Margaret J Shih, Todd L Pittinsky, and Geoffrey C Ho. 2012. Stereotype boost: Positive outcomes from the activation of positive stereotypes. Journal of Personality and social psychology

    2012

  30. [38]

    Kumar, Saurav Sahay, Shang-Tse Chen, and Hung yi Lee

    Hsuan Su, Cheng-Chu Cheng, Hua Farn, Shachi H. Kumar, Saurav Sahay, Shang-Tse Chen, and Hung yi Lee. 2023. https://api.semanticscholar.org/CorpusID:264172254 Learning from red teaming: Gender bias provocation and mitigation in large language models . ArXiv, abs/2310.11079

    2023 arXiv

  31. [39]

    Don R Swanson. 1986. Undiscovered public knowledge. The Library Quarterly, 56(2):103--118

    1986

  32. [40]

    Sotaro Takeshita, Tommaso Green, Ines Reinig, Kai Eckert, and Simone Paolo Ponzetto. 2024. Aclsum: A new dataset for aspect-based summarization of scientific publications. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational ...

    2024

  33. [41]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  34. [42]

    A Tobena, I Marks, and R Dar. 1999. Advantages of bias and prejudice: An exploration of their neurocognitive templates. Neuroscience & Biobehavioral Reviews, 23(7):1047--1058

    1999

  35. [43]

    Amos Tversky and Daniel Kahneman. 1974. Judgment under uncertainty: Heuristics and biases: Biases in judgments reveal some heuristics of thinking under uncertainty. science, 185(4157):1124--1131

    1974

  36. [44]

    Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. 2024. S ci MON : Scientific inspiration machines optimized for novelty. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand. Association for Comp...

    2024

  37. [45]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  38. [46]

    Rongwu Xu, Brian Lin, Shujian Yang, Tianqi Zhang, Weiyan Shi, Tianwei Zhang, Zhixuan Fang, Wei Xu, and Han Qiu. 2024 a . The earth is flat because...: Investigating LLM s ' belief towards misinformation via persuasive conversation. In Proceedings of the 62nd Annual Meeting of ...

    2024

  39. [47]

    Zhao Xu, Fan Liu, and Hao Liu. 2024 b . https://api.semanticscholar.org/CorpusID:270440302 Bag of tricks: Benchmarking of jailbreak attacks on llms . ArXiv, abs/2406.09324

    2024 arXiv

  40. [48]

    Xianjun Yang, Xiao Wang, Qi Zhang, Linda Ruth Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. 2023. https://api.semanticscholar.org/CorpusID:263620436 Shadow alignment: The ease of subverting safely-aligned language models . ArXiv, abs/2310.02949

    2023 arXiv

  41. [49]

    Xikang Yang, Xuehai Tang, Songlin Hu, and Jizhong Han. 2024. https://api.semanticscholar.org/CorpusID:269635253 Chain of attack: a semantic-driven contextual multi-turn attacker for llm . ArXiv, abs/2405.05610

    2024 arXiv

  42. [50]

    Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. 2023 a . Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher. In The Twelfth International Conference on Learning Representations

    2023

  43. [51]

    Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. 2023 b . Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher. arXiv preprint arXiv:2308.06463

    2023 arXiv

  44. [52]

    Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. 2024. https://api.semanticscholar.org/CorpusID:266977395 How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms . ArXiv, abs/2401.06373

    2024 arXiv

  45. [53]

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043

    2023 arXiv

  46. [54]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  47. [55]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.