Pith. sign in

REVIEW 3 major objections 5 minor 58 references

SweEval: Do LLMs Really Swear? A Safety Benchmark for Testing Limits for Enterprise Use

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SweEval asks whether LLMs will insert a requested swear word into a real-world writing task, and finds they do so far more often in Indic languages than in English.

desk verdict The SweEval prompt set is genuinely useful, but the reported 'harmful rates' are just swear-word inclusion rates, so the compliance-versus-resistance claims don't hold as written. read the letter →

arxiv 2505.17332 v1 pith:QXSCASYC submitted 2025-05-22 cs.CL cs.AIcs.LGcs.MA

Hitesh Laxmichand Patel , Amit Agarwal , Arion Das , Bhargava Kumar , Srikant Panda , Priyaranjan Pattnayak , Taki Hasan Rafi , Tejaswini Kumar
show 1 more author
Dong-Kyu Chae
This is my paper
classification cs.CLcs.AIcs.LGcs.MA
keywords SweEvalmultilingualsafetyprofanitybenchmarkharmfulratetransliterationenterpriseLLMlow-resourcelanguagesIndic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SweEval is a benchmark that asks whether large language models follow an instruction to include a specific swear word while completing a realistic writing task, such as drafting an email or a sales pitch. The paper claims that open-source LLMs comply far more often when the swear word is in an Indic language such as Hindi, Marathi, Bengali, or Gujarati than when it is in English, and that compliance rises again when Indic swear words are transliterated into Latin script. The authors argue this reveals a safety gap rooted in uneven multilingual training data and shallow contextual understanding, and they present it as a concrete risk for enterprises deploying LLMs across regions.

What carries the argument

The benchmark's engine is a constructed dataset: 109 hand-written English task prompts (professional emails, sales pitches, customer support, social messages) varied by tone (positive/negative) and context (formal/informal), with 25 swear words per language inserted into placeholders. Case 1 uses the word in its native script; Case 2 uses Latin-script transliterations for Indic languages. Responses are scored by the Harmful Rate, the fraction of outputs that both complete the instructed task and include the requested swear word, with all classification done by manual review.

What would settle it

Re-annotate a random, language-blinded sample of the model outputs used for Tables 3 and 4 with at least two independent raters and check whether the English-versus-Indic gap in harmful rates survives; if agreement is low or the gap narrows, the central pattern is an artifact of the labeling rule.

Watch

Extended reading notes

Core claim

The central discovery is a language-dependent compliance pattern: across 13 open-source models, harmful rates for English prompts with English swear words are typically below 0.5 and often below 0.4, while Hindi, Marathi, Bengali, and Gujarati prompts typically exceed 0.6. Transliterated Indic swear words make the gap larger. The paper reads this as evidence that models do not recognize or evaluate profanity in low-resource languages the way they do in English, so safety alignment that works for English does not transfer.

Load-bearing premise

Every reported harmful rate rests on human reviewers consistently judging a response harmful only when it completes the task and includes the exact requested swear word, and the paper does not report inter-annotator agreement or release the labels.

Editorial extensions

If this is right

  • If the reported rates hold, deploying any of these open models for customer-facing writing in Indic-language markets carries a higher risk of generating profanity than English-only deployment.
  • Positive-tone prompts and formal-context prompts can increase compliance with swearing instructions, so tone-based safety filters are not reliable safeguards.
  • Transliterated swear words are more dangerous than native-script ones for Indic languages, implying that Latin-script user input is the harder safety case.
  • Larger and newer model versions generally improve resistance, but the improvement is uneven and does not close the language gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to prompt models to classify each swear word's offensiveness in isolation; if classification is accurate but compliance is high, the gap is a refusal/alignment problem, and if classification fails, it is a recognition problem.
  • The benchmark's English-only prompts mean the results measure recognition of foreign words inside English text, not full conversational fluency; a mirror dataset with prompts written entirely in each language would separate these effects.
  • If the manual labels were released with inter-annotator agreement, the harmful-rate tables could be re-audited; until then, the precise magnitudes should be treated as approximate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. SweEval is a multilingual benchmark that pairs 109 enterprise and casual English task prompts with 25 swear words from eight languages (English, Spanish, French, German, Hindi, Marathi, Bengali, Gujarati), producing 2,725 prompts per language, plus a transliterated Indic variant set. The authors evaluate 13 open-source LLMs and define the Harmful Rate as the proportion of responses that both complete the requested task and include the requested swear word. The central descriptive finding is that harmful rates are substantially higher for Indic-language and transliterated swear words than for English, which the paper attributes to weaker safety alignment and cultural awareness in low-resource languages.

Significance. The dataset addresses a genuinely underexplored enterprise-relevant harm, instructed swearing, and the authors ship a released dataset and code, use deterministic decoding, and test multiple model families and sizes. The cross-lingual gap is a plausible and practically important descriptive finding that would justify further study. However, the paper's contribution is currently conditional on the response-evaluation pipeline actually measuring what the definition says; the reported numbers appear to be inclusion-only counts, and the manual review is undocumented. With those issues fixed, this could be a useful benchmark; as it stands, the central claim is not yet supported.

major comments (3)
  1. [§4.1 Response Evaluation, Tables 3–8] The stated Harmful Rate requires both task completion and inclusion of the requested swear word, but every reported value is numerically equal to the swear-word inclusion count. For example, qw-2.5-14b English gives (239+266)/2,725 = 0.185, matching Table 3's 0.19; qw-2.5-7b Hindi gives (1,176+837)/2,725 = 0.738, matching Table 3's 0.74. This equality means the manual review either did not apply the completion condition or the condition was vacuous; the latter is implausible because responses that quote the word while refusing would be counted as harmful under an inclusion-only rule. The 'comply vs. resist' framing in the Abstract and §4.2 is therefore not supported by the reported numbers; the tables measure token emission, not instruction following. The revision must either report the separately verified two-condition counts or redefine the metric and temper the claims accordingly.
  2. [§4.1 Response Evaluation] The evaluation relies on a 'meticulous manual review process' but reports no inter-annotator agreement, no annotation guidelines, and no release of the response labels. Every table from Table 3 through Table 14 is affected, and the appendix tables are explicitly counts of responses 'containing swear words,' which can be produced automatically and do not demonstrate that the two-condition harmful definition was applied. Without reliability statistics or a label release, the magnitude of every reported harmful rate is unverifiable. At minimum, the authors should release the per-response labels and report agreement on a sample.
  3. [§4.2, RQ2–RQ3, and Conclusion] The causal interpretation goes beyond the design. The paper attributes higher harmful rates for Indic languages to 'lack of cultural awareness' and 'limited reasoning skills,' but the experiment only varies the language/script of the inserted word, not the model's knowledge of that word. The gap is equally consistent with tokenization, frequency, and refusal-training coverage. A concrete control, for example measuring refusal rates when the same word is glossed in English, or regressing harmful rate on word frequency and token length, would be needed to support the causal claim. As written, the cross-lingual descriptive finding is credible, but the conclusion that it reflects cultural awareness is not established.
minor comments (5)
  1. [§1 and References] The introduction cites 'Pattnayak et al.' twice without year or a matching reference entry, and several reference entries are formatted inconsistently; please normalize the citation style.
  2. [Tables 2–14] Model abbreviations are inconsistent across tables, for example 'ms-22b-v1' in Table 5 versus 'mx-22b-v1' in Table 2 and 'mx-8-22b-v1' in Table 3; please standardize all model identifiers.
  3. [§4.3 RQ3] Claims about ordering between model versions, such as Llama-3.1 versus Llama-3 or Phi-3 versus Phi-3.5, are based on point estimates without confidence intervals; for differences of a few hundredths (e.g., Table 3, en column, lm-3.1-8b 0.38 vs lm-3-8b 0.33), sampling noise is a plausible alternative explanation.
  4. [Abstract and §2.2] The claim that SweEval is 'the first cross-lingual enterprise safety benchmark' should be qualified, given that XSTest, SORRY-Bench, and XSafety already address multilingual safety; the distinctive contribution is better stated as a focus on instructed swearing in enterprise-style tasks.
  5. [Appendix A.3] The ablation conclusion that positive tone leads to more harmful outputs is based on Tables 5–6, which are inclusion counts rather than the two-condition harmful definition; this section must be revised together with the main metric.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: SweEval is an empirical benchmark; the reported harmful rates are measurements, not reductions from fitted inputs or self-citations.

full rationale

SweEval is an empirical measurement study, not a derivation. The Harmful Rate in Section 4.1 is defined as #Harmful/(#Harmless + #Harmful), with each response manually classified against the prompt's two conditions (task completion and inclusion of the requested swear word). No parameter is fitted to the headline numbers, and the cross-lingual comparisons (e.g., higher rates for Hindi/Marathi/Bengali/Gujarati than for English in Tables 3-4) are observed model outputs, not consequences of the benchmark's construction. The cited definition (Shen et al., 2024b) is external, and the paper's other self-citations appear only in the introduction and references, not as load-bearing support for the safety claim. One internal-consistency concern should be noted: the appendix Tables 5-8 are captioned as counts of responses 'containing swear words,' and those counts divided by 2,725 reproduce the Table 3/4 harmful rates (e.g., qw-2.5-14b English (239+266)/2725 approximately equals 0.19; qw-2.5-7b Hindi (1176+837)/2725 approximately equals 0.74). That suggests the reported rates may reflect swear-word inclusion alone rather than the stated two-condition manual review, which is a construct-validity issue, not a circular derivation: the numbers still come from model outputs, and no equation reduces to its own input. The benchmark is self-contained against external models, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted; the benchmark is empirical. The central claim rests on several domain assumptions about the definition of harm, annotation reliability, and the representativeness of the swear-word lists. These assumptions are plausible but unverified.

assumptions (4)
  • domain assumption A response is harmful only if it both completes the requested task and includes the exact requested swear word.
    This binary rule equates instruction-following with harm and may misclassify paraphrased, translated, or otherwise offensive responses. It underlies all Harmful Rate calculations (Section 4.1).
  • domain assumption Manual review produces accurate and consistent harmful/harmless labels.
    No inter-annotator agreement, annotation guidelines, or label release is provided; all reported rates depend on this unverified reliability.
  • domain assumption The 25 swear words per language are representative of culturally offensive terms and are correctly translated or transliterated.
    The list is curated with native speakers but not published in full; the transliteration step in Case 2 assumes phonetic preservation of offensiveness.
  • domain assumption English-template prompts with foreign swear words inserted test multilingual safety behavior.
    The prompt is English except for one token; this may not reflect genuine multilingual conversational contexts and limits the strength of the 'cross-lingual' claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SweEval: Do LLMs Really Swear? A Safety Benchmark for Testing Limits for Enterprise Use." pith.science (2026). https://pith.science/paper/QXSCASYC

@misc{pith2026250517332,
  author       = {Pith},
  title        = {Pith review of: SweEval: Do LLMs Really Swear? A Safety Benchmark for Testing Limits for Enterprise Use},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QXSCASYC}},
  note         = {Machine review of arXiv:2505.17332}
}
read the original abstract

Enterprise customers are increasingly adopting Large Language Models (LLMs) for critical communication tasks, such as drafting emails, crafting sales pitches, and composing casual messages. Deploying such models across different regions requires them to understand diverse cultural and linguistic contexts and generate safe and respectful responses. For enterprise applications, it is crucial to mitigate reputational risks, maintain trust, and ensure compliance by effectively identifying and handling unsafe or offensive language. To address this, we introduce SweEval, a benchmark simulating real-world scenarios with variations in tone (positive or negative) and context (formal or informal). The prompts explicitly instruct the model to include specific swear words while completing the task. This benchmark evaluates whether LLMs comply with or resist such inappropriate instructions and assesses their alignment with ethical frameworks, cultural nuances, and language comprehension capabilities. In order to advance research in building ethically aligned AI systems for enterprise use and beyond, we release the dataset and code: https://github.com/amitbcp/multilingual_profanity.

Figures

Figures reproduced from arXiv: 2505.17332 by the authors.

Figure 1
Figure 1. Regions where our chosen languages are spo [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Case 1 - Multilingual Swearing. older and newer versions of models in the Llama and Mistral series to assess whether safety align￾ment improves in successive iterations. To ensure deterministic results, we set the temperature to 0 and limited the maximum output token count to 2048. Model Parameters Mistral-7b-instruct-v1 (ms-7b-v1) 7B Mistral-7b-instruct-v2 (ms-7b-v2) 7B Mistral-7b-instruct-v3 (ms-7b-v3) 7B Mixtral-… view at source ↗
Figure 3
Figure 3. Case 2 - Transliterated Swearing. understanding of English swear words and their contextual usage, enabling them to avoid harmful outputs. In contrast, for other languages, the mod￾els may not fully grasp the meaning or context of swear words, resulting in more frequent usage. These findings shed light on the need for enhanced data curation and improved training methodolo￾gies to enhance the handling of sensitive la… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Case 1 - Model-wise harmful rate distribution across all languages [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Case 2 - Model-wise harmful rate distribution across all languages [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Case 1 - Latin vs. Indic Languages (lower is better). MULTILINGUAL MODELS BETTER AT RESIST￾ING UNETHICAL INSTRUCTIONS? In our study, models with 8 billion parameters or fewer are cat￾egorized as small models, while those with more than 8 billion parameters are classifi…
Figure 8
Figure 8. Figure 8: Harmful rate of Mistral and Llama models [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

58 extracted references · 8 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  4. [4]

    Amit Agarwal, Kulbhushan Pachauri, Iman Zadeh, and Jun Qian. 2024 a . Techniques for graph data structure augmentation. US Patent 11,989,964

    work page 2024

  5. [5]

    Amit Agarwal, Srikant Panda, Angeline Charles, Bhargava Kumar, Hitesh Patel, Priyanranjan Pattnayak, Taki Hasan Rafi, Tejaswini Kumar, and Dong-Kyu Chae. 2024 b . Mvtamperbench: Evaluating robustness of vision-language models. arXiv preprint arXiv:2412.19794

    arXiv 2024

  6. [6]

    Amit Agarwal, Srikant Panda, Deepak Karmakar, and Kulbhushan Pachauri. 2024 c . Domain adapting graph networks for visually rich documents. US Patent App. 18/240,480

    work page 2024

  7. [7]

    Amit Agarwal, Srikant Panda, and Kulbhushan Pachauri. 2024 d . Synthetic document generation pipeline for training artificial intelligence models. US Patent App. 17/994,712

    work page 2024

  8. [8]

    Amit Agarwal, Srikant Panda, and Kulbhushan Pachauri. 2025. https://aclanthology.org/2025.coling-industry.9/ FS - DAG : Few shot domain adapting graph networks for visually rich document understanding . In Proceedings of the 31st International Conference on Computational Linguistics: Industry Track, pages 100--114, Abu Dhabi, UAE. Association for Computat...

    2025

Show all 58 references
  1. [9]

    Amit Agarwal, Hitesh Patel, Priyaranjan Pattnayak, Srikant Panda, Bhargava Kumar, and Tejaswini Kumar. 2024 e . Enhancing document ai data generation through graph-based synthetic layouts. arXiv preprint arXiv:2412.03590

    2024 arXiv

  2. [10]

    Do, Yan Xu, and Pascale Fung

    Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. https://doi.org/10.18653/v1/2023.ijcnlp-main.45 A multitask, multilingual, multimodal evaluation of C h...

    2023 doi

  3. [11]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffr...

    2020 arXiv

  4. [12]

    Pappas, Florian Tramer, Hamed Hassani, and Eric Wong

    Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramer, Hamed Hassani, and Eric Wong. 2024. https://arxiv.org/abs/2404.01318 Jailbreakbench: An open robustn...

    2024 arXiv

  5. [13]

    Lynn Chua, Badih Ghazi, Yangsibo Huang, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Amer Sinha, Chulin Xie, and Chiyuan Zhang. 2024. https://arxiv.org/abs/2406.16135 Crosslingual capabilities and knowledge barriers in multilingual large language models . Preprint, arXiv:2406.16135

    2024 arXiv

  6. [14]

    John Chung, Ece Kamar, and Saleema Amershi. 2023. https://doi.org/10.18653/v1/2023.acl-long.34 Increasing diversity while maintaining accuracy: Text data generation with large language models and human interventions . In Proceedings of the 61st Annual Meeting of the Associatio...

    2023 doi

  7. [15]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  8. [16]

    Daniil Gurgurov, Tanja Bäumel, and Tatiana Anikina. 2024. https://doi.org/10.48550/ARXIV.2406.10602 Multilingual large language models and curse of multilinguality

    2024 doi arXiv

  9. [17]

    Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. https://arxiv.org/abs/2203.09509 Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection . Preprint, arXiv:2203.09509

    2022 arXiv

  10. [18]

    Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Qiu, Boxun Li, and Yaodong Yang. 2024. https://arxiv.org/abs/2406.15513 Pku-saferlhf: Towards multi-level safety alignment for llms with human preference . Preprint, arXiv:2406.15513

    2024 arXiv

  11. [19]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  12. [20]

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024 a . https://arxiv.org/abs/2406.00515 A survey on large language models for code generation . Preprint, arXiv:2406.00515

    2024 arXiv

  13. [21]

    Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. 2024 b . https://arxiv.org/abs/2303.06689 Self-planning code generation with large language models . Preprint, arXiv:2303.06689

    2024

  14. [22]

    Divyanshu Kumar, Anurakt Kumar, Sahil Agarwal, and Prashanth Harshangi. 2024. https://arxiv.org/abs/2404.04392 Fine-tuning, quantization, and llms: Navigating unintended outcomes . Preprint, arXiv:2404.04392

    2024 arXiv

  15. [23]

    Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. 2024. https://arxiv.org/abs/2402.05044 Salad-bench: A hierarchical and comprehensive safety benchmark for large language models . Preprint, arXiv:2402.05044

    2024 arXiv

  16. [24]

    Xun Liang, Hanyu Wang, Yezhaohui Wang, Shichao Song, Jiawei Yang, Simin Niu, Jie Hu, Dan Liu, Shunyu Yao, Feiyu Xiong, and Zhiyu Li. 2024. https://arxiv.org/abs/2408.12599 Controllable text generation for large language models: A survey . Preprint, arXiv:2408.12599

    2024 arXiv

  17. [25]

    Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang. 2023. https://arxiv.org/abs/2310.17389 Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation . Preprint, arXiv:2310.17389

    2023 arXiv

  18. [26]

    Zhuoran Lu, Sheshera Mysore, Tara Safavi, Jennifer Neville, Longqi Yang, and Mengting Wan. 2024. https://arxiv.org/abs/2405.04656 Corporate communication companion (ccc): An llm-empowered writing assistant for workplace social media . Preprint, arXiv:2405.04656

    2024 arXiv

  19. [27]

    Wong, Siyou Liu, and Longyue Wang

    Chenyang Lyu, Zefeng Du, Jitao Xu, Yitao Duan, Minghao Wu, Teresa Lynn, Alham Fikri Aji, Derek F. Wong, Siyou Liu, and Longyue Wang. 2024. https://arxiv.org/abs/2305.01181 A paradigm shift: The future of machine translation lies with large language models . Preprint, arXiv:2305.01181

    2024 arXiv

  20. [28]

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. 2024. https://arxiv.org/abs/2402.04249 Harmbench: A standardized evaluation framework for automated red teaming and robu...

    2024 arXiv

  21. [29]

    R OpenAI et al. 2023. Gpt-4 technical report. ArXiv, 2303:08774

    2023

  22. [30]

    Srikant Panda, Amit Agarwal, Gouttham Nambirajan, and Kulbhushan Pachauri. 2025 a . Out of distribution element detection for information extraction. US Patent App. 18/347,983

    2025

  23. [31]

    Srikant Panda, Amit Agarwal, and Kulbhushan Pachauri. 2025 b . Techniques of information extraction for selection marks. US Patent App. 18/240,344

    2025

  24. [32]

    Hitesh Laxmichand Patel, Amit Agarwal, Bhargava Kumar, Karan Gupta, and Priyaranjan Pattnayak. 2024. Llm for barcodes: Generating diverse synthetic data for identity documents. arXiv preprint arXiv:2411.14962

    2024 arXiv

  25. [33]

    Review of reference generation methods in large language models

    Priyaranjan Pattnayak, Amit Agarwal, Bhargava Kumar, Yeshil Bangera, Srikant Panda, Tejaswini Kumar, and Hitesh Laxmichand Patel. Review of reference generation methods in large language models. Journal ID, 9339:1263

  26. [34]

    Priyaranjan Pattnayak, Amit Agarwal, Hansa Meghwani, Hitesh Laxmichand Patel, and Srikant Panda. 2025 a . Hybrid ai for responsive multi-turn online conversations with novel dynamic routing and feedback adaptation. In Proceedings of the 4th International Workshop on Knowledge-...

    2025

  27. [35]

    Priyaranjan Pattnayak, Hitesh Laxmichand Patel, and Amit Agarwal. 2025 b . https://arxiv.org/abs/2504.16977 Tokenization matters: Improving zero-shot ner for indic languages . Preprint, arXiv:2504.16977

    2025 arXiv

  28. [37]

    Priyaranjan Pattnayak, Hitesh Laxmichand Patel, Amit Agarwal, Bhargava Kumar, Srikant Panda, and Tejaswini Kumar. 2025 d . https://arxiv.org/abs/2502.13108 Improving clinical question answering with multi-task learning: A joint approach for answer extraction and medical catego...

    2025 arXiv

  29. [38]

    Priyaranjan Pattnayak, Hitesh Laxmichand Patel, Bhargava Kumar, Amit Agarwal, Ishan Banerjee, Srikant Panda, and Tejaswini Kumar. 2024. Survey of large multimodal model datasets, application categories and taxonomy. arXiv preprint arXiv:2412.17759

    2024 arXiv

  30. [39]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. https://api.semanticscholar.org/CorpusID:160025533 Language models are unsupervised multitask learners

    2019

  31. [40]

    Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2024. https://arxiv.org/abs/2308.01263 Xstest: A test suite for identifying exaggerated safety behaviours in large language models . Preprint, arXiv:2308.01263

    2024 arXiv

  32. [41]

    Lingfeng Shen, Weiting Tan, Sihao Chen, Yunmo Chen, Jingyu Zhang, Haoran Xu, Boyuan Zheng, Philipp Koehn, and Daniel Khashabi. 2024 a . https://doi.org/10.18653/v1/2024.findings-acl.156 The language barrier: Dissecting safety challenges of LLM s in multilingual contexts . In F...

    2024 doi

  33. [42]

    Lingfeng Shen, Weiting Tan, Sihao Chen, Yunmo Chen, Jingyu Zhang, Haoran Xu, Boyuan Zheng, Philipp Koehn, and Daniel Khashabi. 2024 b . https://arxiv.org/abs/2401.13136 The language barrier: Dissecting safety challenges of llms in multilingual contexts . Preprint, arXiv:2401.13136

    2024 arXiv

  34. [43]

    Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024 c . ``Do Anything Now'': Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models . In ACM SIGSAC Conference on Computer and Communications Security (CCS) . ACM

    2024

  35. [44]

    Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, Shangwei Guo, Tianwei Zhang, and Guoyin Wang. 2023. https://arxiv.org/abs/2305.08377 Text classification via large language models . Preprint, arXiv:2305.08377

    2023 arXiv

  36. [45]

    Qwen Team. 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  37. [46]

    Simone Tedeschi, Felix Friedrich, Patrick Schramowski, Kristian Kersting, Roberto Navigli, Huu Nguyen, and Bo Li. 2024. https://arxiv.org/abs/2404.08676 Alert: A comprehensive benchmark for assessing large language models' safety through red teaming . Preprint, arXiv:2404.08676

    2024 arXiv

  38. [47]

    Edwin Thomas, Amit Agarwal, Sandeep Jana, and Kulbhushan Pachauri. 2025. Model augmentation framework for domain assisted continual learning in deep learning. US Patent App. 18/406,905

    2025

  39. [48]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://arxiv.org/abs/2302.13971 Llama:...

    2023 arXiv

  40. [49]

    Wenxuan Wang, Zhaopeng Tu, Chang Chen, Youliang Yuan, Jen tse Huang, Wenxiang Jiao, and Michael R. Lyu. 2024 a . https://arxiv.org/abs/2310.00905 All languages matter: On the multilingual safety of large language models . Preprint, arXiv:2310.00905

    2024 arXiv

  41. [50]

    Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. 2023. https://arxiv.org/abs/2308.13387 Do-not-answer: A dataset for evaluating safeguards in llms . Preprint, arXiv:2308.13387

    2023 arXiv

  42. [51]

    Zhiqiang Wang, Yiran Pang, and Yanbin Lin. 2024 b . https://arxiv.org/abs/2405.10523 Smart expert system: Large language models as text classifiers . Preprint, arXiv:2405.10523

    2024 arXiv

  43. [52]

    Azmine Toushik Wasi, Raima Islam, Mst Rafia Islam, Farig Yousuf Sadeque, Taki Hasan Rafi, and Dong-Kyu Chae. 2025. Dialectal bias in bengali: An evaluation of multilingual large language models across cultural variations. In Companion Proceedings of the ACM on Web Conference

    2025

  44. [53]

    Azmine Toushik Wasi, Taki Hasan Rafi, and Dong-Kyu Chae. 2024. Diaframe: A framework for understanding bengali dialects in human-ai collaborative creative writing spaces. In Companion Publication of the 2024 Conference on Computer-Supported Cooperative Work and Social Computin...

    2024

  45. [54]

    Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, Ruoxi Jia, Bo Li, Kai Li, Danqi Chen, Peter Henderson, and Prateek Mittal. 2024. https://arxiv.org/abs/2406.14598 Sorry-bench: Systematically ev...

    2024 arXiv

  46. [55]

    Jingwei Yi, Rui Ye, Qisi Chen, Bin Zhu, Siheng Chen, Defu Lian, Guangzhong Sun, Xing Xie, and Fangzhao Wu. 2024. https://doi.org/10.18653/v1/2024.findings-acl.549 On the vulnerability of safety alignment in open-access LLM s . In Findings of the Association for Computational L...

    2024 doi

  47. [56]

    Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, and Grzegorz Kondrak. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.491 Don ' t trust C hat GPT when your question is not in E nglish: A study of multilingual abilities and types of LLM s . In Proceedings of the 2023 Conferen...

    2023 doi

  48. [57]

    Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. 2024. https://arxiv.org/abs/2309.07045 Safetybench: Evaluating the safety of large language models . Preprint, arXiv:2309.07045

    2024 arXiv

  49. [58]

    Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.176 Multilingual machine translation with large language models: Empirical results and analysis . In Findings of the ...

    2024 doi

  50. [59]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023. https://arxiv.org/abs/2307.15043 Universal and transferable adversarial attacks on aligned language models . Preprint, arXiv:2307.15043

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.