Pith. sign in

REVIEW 4 major objections 5 minor 75 references

Towards Safer Chatbots: Automated Policy Compliance Evaluation of Custom GPTs

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A fully automated, black-box audit of 782 Custom GPTs finds that 58.7% produce at least one response violating OpenAI's usage policies, and that most of these violations trace back to the base models rather than user customization.

desk verdict A novel, cheap, and useful automated audit pipeline for Custom GPTs, but the headline 58.7% violation rate rests on the authors' own policy operationalization and should be read as conditional. read the letter →

arxiv 2502.01436 v3 pith:BGE75D3G submitted 2025-02-03 cs.CL cs.AI

David Rodriguez , William Seymour , Jose M. Del Alamo , Jose Such This is my paper · ORCID
classification cs.CLcs.AI
keywords CustomGPTspolicycomplianceLLM-as-a-judgered-teamingGPTStoreusagepoliciesautomatedevaluationsafety
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that policy compliance in a marketplace of user-customized chatbots can be measured automatically and at scale, without any access to the chatbots' internal instructions. It builds a pipeline that discovers Custom GPTs, generates policy-targeted test prompts from each GPT's public description, submits them as direct questions, and uses a language model judge to decide whether each response violates one of three OpenAI usage policies. On 782 GPTs from the Romantic, Cybersecurity, and Academic corners of the GPT Store, the pipeline classifies 58.7% as non-compliant. The result matters because manual review cannot keep pace with thousands of custom chatbots, and because the paper's comparison with base models suggests most violations are already present in the foundation model before any customization.

What carries the argument

The machinery that carries the argument is the compliance-assessment module: an LLM-as-a-judge setup in which a language model rates another system's output by receiving the red-teaming prompt, the Custom GPT's response, and an operationalized policy, then returning a binary label and a rationale. The operationalized policies are the hand-written bridge that turns OpenAI's short clauses into testable criteria; for example, the romantic-companionship policy is expanded to cover acting as a boyfriend or girlfriend, romantic role-play, emotional dependency, and sustained affectionate language. Around that judge, the pipeline's other moving parts are keyword-driven discovery of GPTs, metadata-tailored prompt generation, browser-based interaction with rate-limit handling, and conservative aggregation in which one violating response marks the whole GPT non-compliant. The orchestration of these parts is what lets the paper scale to hundreds of GPTs at low cost.

What would settle it

Compare the paper's 40 human-annotated prompt-response pairs against OpenAI's internal enforcement decisions on the same pairs; meaningful disagreement would show the reported F1 of 0.975 measures agreement with the paper's operationalized criteria, not with the platform's actual interpretation. A second decisive test is to run the pipeline on a set of GPTs that OpenAI has explicitly cleared as compliant, and check whether the judge flags more than the validation rate predicts.

Watch

Extended reading notes

Core claim

The paper's central claim is that a fully automated, black-box evaluation pipeline can reliably identify policy-violating behavior in Custom GPTs, and that applying it to 782 GPTs from the store shows 58.7% produce at least one violating response. The pipeline operationalizes three OpenAI usage-policy clauses, romantic companionship, privacy and cybersecurity, and academic dishonesty, into detailed criteria, generates direct and deceptive red-teaming prompts from each GPT's title and description, submits five direct prompts per GPT through the web interface, and labels each response using GPT-4o as an LLM judge. Validation against 40 human-annotated direct prompt-response pairs yields an F1 score, accuracy, and recall of 0.9752 and precision of 0.976. Re-running the same prompts on GPT-4 and GPT-4o gives matching classifications for 93.02% and 92.73% of prompt sets, which the paper interprets as evidence that most violations are inherited from base models rather than created by customization.

Load-bearing premise

The measurement stands on the paper's hand-written operationalized policies in Appendix C, and if OpenAI enforces its concise usage clauses more narrowly than those criteria, the reported violation rates, especially the 98% for Romantic GPTs, would be substantially lower.

Editorial extensions

If this is right

  • Continuous platform screening becomes affordable: the paper reports evaluating 782 GPTs end-to-end for $10.06 total, about $0.0125 per GPT, which is a small fraction of the estimated cost of a five-minute human review.
  • Because 92–93% of prompt-level classifications match the base-model evaluations, the paper implies that aligning GPT-4 and GPT-4o themselves would remove the majority of observed violations, while customization-specific harms are concentrated in a smaller set.
  • The conservative one-violation-flags-the-GPT rule, combined with the paper's reliance on five direct prompts per GPT, means 58.7% is best read as a lower-bound estimate of policy non-compliance in the sampled categories.
  • The 7.4% non-compliance rate for Cybersecurity GPTs and roughly two-thirds for Academic GPTs show that violation rates vary by domain, so store review and mitigation should be targeted per policy area rather than uniform.
  • The black-box design means the same pipeline can be applied to any LLM-based chatbot ecosystem with written policies, not just the GPT Store, so the method transfers to enterprise and regulatory auditing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 58.7% figure should not be read as OpenAI's own enforcement verdict: it measures agreement with the paper's operationalized criteria, and a different reading of 'romantic companionship' could shrink the Romantic category's 98% rate dramatically.
  • The paper's exclusion of deceptive prompts, justified by near-chance annotator agreement, suggests that policy ambiguity is a measurement problem as much as a content problem; future work should focus on making deceptive-prompt adjudication reliable before claiming full-coverage audits.
  • If base-model alignment is the dominant lever, platform governance should push increasingly toward pre-deployment policy alignment of foundation models, a consequence the paper notes but does not develop: closed platforms could test base models against every policy before opening them to customization.
  • A testable extension is to run the pipeline on medical, financial, or legal GPTs using third-party experts rather than paper-written operationalized policies; if the judge's F1 holds outside paper-written criteria, the method generalizes, and if it drops, the operationalization step is the real bottleneck.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents a fully automated black-box method for evaluating whether Custom GPTs in OpenAI's GPT Store comply with selected OpenAI usage policies (Romantic, Cybersecurity, Academic). The method combines GPT Store crawling, tailored red-teaming prompt generation by GPT-4o, and an LLM-as-a-judge compliance classifier that uses hand-written operationalized policies. The judge is validated against a human-annotated ground truth of 40 direct-prompt duets, reporting precision 0.976 and accuracy/recall/F1 0.9752. Applied to 782 Custom GPTs, the pipeline flags 58.7% as non-compliant, with 98.0% non-compliance among Romantic GPTs, and a base-model comparison suggesting most violations are inherited from GPT-4/GPT-4o rather than introduced by customization. The authors disclose seven removals after reporting to OpenAI and commit to releasing code and data upon acceptance.

Significance. If the central claims hold, the paper would make a useful contribution: it demonstrates a scalable, black-box audit approach for a rapidly growing ecosystem, quantifies an apparently high violation rate, and provides an empirical attribution of violations to base-model behavior versus customization. The design choices are largely sensible: the pipeline is staged, the aggregation rule is conservative, and the external validity limits are partially acknowledged. The inclusion of a concrete cost estimate and a responsible disclosure account are also valuable. However, the load-bearing evidence for the central claim—the validation of the automated judge—is currently too thin and internally inconsistent, and the measurement's validity is tied to the authors' own operationalized policy interpretation, which limits the strength of the headline violation rates.

major comments (4)
  1. [§5.2] The reported validation metrics cannot all be correct. On a 40-instance binary ground truth, accuracy, recall, and F1 cannot all equal 0.9752 because accuracy would require a non-integer number of correct classifications, and the harmonic mean of precision 0.976 and recall 0.9752 is approximately 0.9756, not 0.9752. Please provide the actual confusion matrix, the numbers of true/false positives and negatives, and recompute all metrics; this is a load-bearing part of the claim that the automated judge is reliable.
  2. [§4.2.3, §5.1, Appendix C] The validation measures agreement between GPT-4o and the authors' own operationalized policy criteria, not agreement with OpenAI's enforcement interpretation. The same team wrote the operationalized policies in Appendix C, used them as input to the LLM judge, and used them as the annotation criteria for the ground truth in §5.1. The F1 of 0.975 therefore shows that the judge reproduces the authors' reading of the policies; it does not establish that the 58.7% overall non-compliance rate or the 98.0% Romantic non-compliance rate corresponds to actual violations under OpenAI's enforcement. Section 10 concedes that final judgments depend on OpenAI's interpretation, but this caveat is absent from the abstract and §6.2, where the rates are presented as findings.
  3. [§5.1, §6.1] The ground truth is restricted to 40 duets annotated by a single co-author, and all deceptive prompts are excluded from the large-scale evaluation because inter-annotator agreement on them was near chance (α = 0.126). This substantially narrows the claim of a 'fully automated' compliance evaluation: the system is only validated for simple direct requests, and the paper itself notes that multi-turn or deceptive interactions may reveal additional violations. The §10 statement that the non-compliance rates are a lower bound is appropriate, but the abstract and §6.2 present the 58.7% figure without that qualification, which is misleading.
  4. [§6.3, Table 2] The interpretation of the popularity analyses is internally inconsistent: the Mann–Whitney U test and Kendall's rank correlation reject their null hypotheses, yet the text concludes there is 'no substantial correlation' and attributes the findings to 'data variability or the presence of outliers.' The effect sizes are small, but the narrative should either state that the correlations are statistically significant but practically negligible, or justify why the tests are not meaningful for this sample. As written, the section both rejects and dismisses the same evidence.
minor comments (5)
  1. [Title header] The running title on page 1 reads 'TOWARDSSAFERCHATBOTS' with a duplicated letter; this should be corrected.
  2. [§5.3] The sentence 'we manually inspected more than 2' appears truncated; the intended number of additional prompt-response pairs should be stated.
  3. [§6.2, Figures 3–4] The text in §6.2 refers to 'Figure 4' when summarizing chat counts and ratings, but those plots appear in Figure 3; the figure references should be checked and renumbered consistently.
  4. [References] References [68] and [69] are the same paper (Zheng et al., 'Judging LLM-as-a-judge with MT-Bench and Chatbot Arena') and should be merged.
  5. [§4.2.2] The prompt generator is described as configurable with a settable direct/deceptive ratio, but §6.1 reports that only five direct prompts per GPT were used in the large-scale study; the relationship between the generator's ten prompts and the five used should be clarified earlier.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported F1 validates the automated judge against the authors' own explicitly stated rubric, and the 58.7% violation rate is an independent black-box measurement under that stated rubric.

full rationale

The paper's central derivation is an empirical pipeline rather than a mathematical one. Custom GPTs are retrieved (Section 4.2.1), red-teaming prompts are generated (Section 4.2.2), responses are judged by GPT-4o against explicitly operationalized policies (Section 4.2.3), the judge is compared with human annotations (Section 5), and the pipeline is applied to 782 GPTs (Section 6.2). No parameter is fitted and then renamed as a prediction: the LLM judge is prompted with the operationalized policies and is not trained on the ground-truth labels, so the F1 score of 0.9752 is not forced by construction. The human annotators and the judge did share the same operationalized criteria, which means the validation establishes inter-rater agreement on the authors' rubric rather than external validation of that rubric against OpenAI's enforcement; however, the paper explicitly discloses this limitation in Section 10: 'final judgments of compliance ultimately depend on OpenAI's interpretation.' That is a construct-validity caveat, not a circular reduction: the violation classifications still depend on the actual black-box responses, and the rubric is stated in full in Appendix C. There are no load-bearing self-citations: the authors' prior work (e.g., [4], [28]) is contextual and is not used to justify the compliance method or to rule out alternatives. The base-model comparison in Section 7.3 is an independent empirical check. One internal support gap exists: Section 6.1 attributes to Section 5.3 a load-testing result about deceptive prompts that Section 5.3 does not actually report, but this is an evidentiary inconsistency rather than a circularity. Overall, the reported F1 and the 58.7% violation rate are measurements under a stated operationalization, not restatements of the paper's inputs.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claims rest on hand-chosen design decisions rather than fitted numeric parameters: the operationalized policies, the retrieval keywords, the five-direct-prompt protocol, and the any-violation aggregation rule. The evaluation assumes these choices faithfully represent OpenAI's usage policies and that GPT-4o's judgments generalize beyond the 40-duet validation set. No numeric free parameters were fitted to optimize results.

free parameters (2)
  • Non-compliance aggregation threshold = 1 or more violating duets out of 5.
    A GPT is flagged non-compliant if any of its five direct responses violates policy (Sections 2.2.1 and 4.2.3). This conservative rule maximizes sensitivity and directly shapes the reported 58.7% non-compliance rate.
  • Number of direct red-teaming prompts per GPT = 5.
    Chosen in Section 6.1 to balance execution budget and population coverage. More prompts would likely surface additional violations, which the authors acknowledge by framing their rate as a lower bound.
assumptions (5)
  • domain assumption The operationalized policies in Appendix C faithfully represent OpenAI's usage policies.
    Section 4.2.3 converts concise OpenAI clauses into detailed criteria by hand; the human ground truth was annotated with these same criteria, so the validity of the compliance labels depends on this faithfulness, which OpenAI's actual enforcement has not confirmed.
  • domain assumption Keyword-based retrieval yields a representative sample of GPTs for each policy domain.
    Section 6.1 queries the GPT Store with terms like 'girlfriend', 'boyfriend', 'hacking', and 'homework'; this likely oversamples policy-violating Romantic GPTs and is not a random sample of the Store.
  • domain assumption GPT-4o as an LLM-as-a-judge produces valid compliance labels.
    Section 5.2 validates this on 40 human-annotated duets, which is a small sample and uses the authors' own operationalized criteria, so the strength of the validation is limited.
  • domain assumption Direct, single-turn red-teaming prompts suffice to detect policy-violating behavior.
    Deceptive prompts were excluded from the large-scale evaluation due to near-chance inter-annotator agreement (alpha=0.126, Section 5.1); the paper acknowledges this as a lower-bound limitation.
  • domain assumption The 40-duet human ground truth is a reliable benchmark.
    Section 5.1 shows the initial 30 duets annotated by three co-authors and the final 40 direct-prompt duets annotated by a single co-author; the small scale and annotator overlap with the policy authors limit the strength of the F1 metric.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Safer Chatbots: Automated Policy Compliance Evaluation of Custom GPTs." pith.science (2026). https://pith.science/paper/BGE75D3G

@misc{pith2026250201436,
  author       = {Pith},
  title        = {Pith review of: Towards Safer Chatbots: Automated Policy Compliance Evaluation of Custom GPTs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BGE75D3G}},
  note         = {Machine review of arXiv:2502.01436}
}
read the original abstract

User-configured chatbots built on top of large language models are increasingly available through centralized marketplaces such as OpenAI's GPT Store. While these platforms enforce usage policies intended to prevent harmful or inappropriate behavior, the scale and opacity of customized chatbots make systematic policy enforcement challenging. As a result, policy-violating chatbots continue to remain publicly accessible despite existing review processes. This paper presents a fully automated method for evaluating the compliance of Custom GPTs with its marketplace usage policy using black-box interaction. The method combines large-scale GPT discovery, policy-driven red-teaming prompts, and automated compliance assessment using an LLM-as-a-judge. We focus on three policy-relevant domains explicitly addressed in OpenAI's usage policies: Romantic, Cybersecurity, and Academic GPTs. We validate our compliance assessment component against a human-annotated ground-truth dataset, achieving an F1 score of 0.975 for binary policy violation detection. We then apply the method in a large-scale empirical study of 782 Custom GPTs retrieved from the GPT Store. The results show that 58.7% of the evaluated GPTs exhibit at least one policy-violating response, with substantial variation across policy domains. A comparison with the base models (GPT-4 and GPT-4o) indicates that most violations originate from model-level behavior, while customization tends to amplify these tendencies rather than create new failure modes. Our findings reveal limitations in current review mechanisms for user-configured chatbots and demonstrate the feasibility of scalable, behavior-based policy compliance evaluation.

Figures

Figures reproduced from arXiv: 2502.01436 by the authors.

Figure 1
Figure 1. Search of “girlfriend” keyword in the GPT store, showcasing the proliferation of models that violate OpenAI’s [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of the policy compliance evaluation method for Custom GPTs. The process spans five stages, from [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Characteristics of Evaluated Custom GPTs. Note that the total count of chats and ratings does not align with [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Compliance and Non-Compliance Ratios Across GPT Categories. This figure illustrates the compliance [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 62 canonical work pages

  1. [1]

    Gpt in sheep’s clothing: The risk of customized gpts, 2024

    Sagiv Antebi, Noam Azulay, Edan Habler, Ben Ganon, Asaf Shabtai, and Yuval Elovici. Gpt in sheep’s clothing: The risk of customized gpts, 2024

    work page 2024

  2. [2]

    Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions, 2024

    Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions, 2024

    2024

  3. [3]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Lit...

    2020

  4. [4]

    Personal data flows and privacy policy traceability in third-party llm apps in the gpt ecosystem

    Juan-Carlos Carrillo, Jose Luis Martin-Navarro, Rongjun Ma, and Jose Such. Personal data flows and privacy policy traceability in third-party llm apps in the gpt ecosystem. InProceedings on Privacy Enhancing Technologies (PETS), pages 1–23, 2026

    work page 2026

  5. [5]

    From representational harms to quality-of-service harms: A case study on llama 2 safety safeguards, 2024

    Khaoula Chehbouni, Megha Roshan, Emmanuel Ma, Futian Andrew Wei, Afaf Taik, Jackie CK Cheung, and Golnoosh Farnadi. From representational harms to quality-of-service harms: A case study on llama 2 safety safeguards, 2024

    work page 2024

  6. [6]

    Avoiding social engineering and phishing attacks, 2021

    Cybersecurity and Infrastructure Security Agency. Avoiding social engineering and phishing attacks, 2021. Accessed: 2025-12-12

    work page 2021

  7. [7]

    A security risk taxonomy for prompt-based interaction with large language models.IEEE Access, 12:126176–126187, 2024

    Erik Derner, Kristina Batistiˇc, Jan Zahálka, and Robert Babuška. A security risk taxonomy for prompt-based interaction with large language models.IEEE Access, 12:126176–126187, 2024

    work page 2024

  8. [8]

    Transferable adversarial distribution learning: Query-efficient adversarial attack against large language models.Computers & Security, 135:103482, 2023

    Huoyuan Dong, Jialiang Dong, Shaohua Wan, Shuai Yuan, and Zhitao Guan. Transferable adversarial distribution learning: Query-efficient adversarial attack against large language models.Computers & Security, 135:103482, 2023

    work page 2023

Show all 75 references
  1. [9]

    European network for academic integrity

    European Network for Academic Integrity. European network for academic integrity. https:// www.academicintegrity.eu/wp/, 2025. Accessed: 2025-12-12

    2025

  2. [10]

    Lessons from an app update at replika ai: Identity discontinuity in human-ai relationships

    Julian De Freitas, Noah Castelo, Ahmet Uguralp, and Zeliha Uguralp. Lessons from an app update at replika ai: Identity discontinuity in human-ai relationships. https://arxiv.org/abs/2412.14190, 2024. arXiv:2412.14190 [cs.HC]

    2024 arXiv

  3. [11]

    A brief survey on safety of large language models

    Zhengjie Gao, Xuanzi Liu, Yuanshuai Lan, and Zheng Yang. A brief survey on safety of large language models. Journal of computing and information technology, 32(1):47–64, 2024

    2024

  4. [12]

    A survey on llm-as-a-judge, 2025

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo. A survey on llm-as-a-judge, 2025. 17 Automated Policy Compliance Evaluation of Custom GPTs

    2025

  5. [13]

    Shin, and Karl Aberer

    Hamza Harkous, Kassem Fawaz, Rémi Lebret, Florian Schaub, Kang G. Shin, and Karl Aberer. Polisis: Automated analysis and presentation of privacy policies using deep learning. In27th USENIX Security Symposium (USENIX Security 18), pages 531–548, Baltimore, MD, 2018. USENIX Association

    2018

  6. [14]

    Safe lora: the silver lining of reducing safety risks when fine-tuning large language models, 2025

    Chia-Yi Hsu, Yu-Lin Tsai, Chih-Hsun Lin, Pin-Yu Chen, Chia-Mu Yu, and Chun-Ying Huang. Safe lora: the silver lining of reducing safety risks when fine-tuning large language models, 2025

    2025

  7. [15]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. https://arxiv.org/abs/2106.09685, 2021. arXiv:2106.09685 [cs.LG]

    2021 arXiv

  8. [16]

    Chatgpt sets record for fastest-growing user base - analyst note

    Krystal Hu. Chatgpt sets record for fastest-growing user base - analyst note. https://www.reuters.com/ technology/chatgpt-sets-record-fastest-growing-user-base-analyst-note-2023-02-01/ , February 2023. Accessed: 2025-12-12

    2023

  9. [17]

    An empirical study of llm-as-a-judge for llm evaluation: Fine-tuned judge model is not a general substitute for gpt-4, 2024

    Hui Huang, Yingqi Qu, Xingyuan Bu, Hongli Zhou, Jing Liu, Muyun Yang, Bing Xu, and Tiejun Zhao. An empirical study of llm-as-a-judge for llm evaluation: Fine-tuned judge model is not a general substitute for gpt-4, 2024

    2024

  10. [18]

    Lisa: Lazy safety alignment for large language models against harmful fine-tuning attack, 2024

    Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Lisa: Lazy safety alignment for large language models against harmful fine-tuning attack, 2024

    2024

  11. [19]

    International center for academic integrity, 2025

    International Center for Academic Integrity. International center for academic integrity, 2025. Accessed: 2025- 12-12

    2025

  12. [20]

    Social engineering scams, 2025

    INTERPOL. Social engineering scams, 2025. Accessed: 2025-12-12

    2025

  13. [21]

    Knowledge sanitization of large language models

    Yoichi Ishibashi and Hidetoshi Shimodaira. Knowledge sanitization of large language models. https:// arxiv.org/abs/2309.11852, 2023. arXiv:2309.11852 [cs.CL]

    2023 arXiv

  14. [22]

    Unleashing offensive artificial intelligence: Automated attack technique code generation.Computers & Security, 147:104077, 2024

    Eider Iturbe, Oscar Llorente-Vazquez, Angel Rego, Erkuden Rios, and Nerea Toledo. Unleashing offensive artificial intelligence: Automated attack technique code generation.Computers & Security, 147:104077, 2024

    2024

  15. [23]

    Springer Nature, Cham, 2024

    Andrei Kucharavy, Octave Plancherel, Valentin Mulder, Alain Mermoud, and Vincent Lenders, editors.Large Language Models in Cybersecurity: Threats, Exposure and Mitigation. Springer Nature, Cham, 2024. Open Access under CC BY 4.0

    2024

  16. [24]

    Fine-tuning, quantization, and llms: Navigating unintended outcomes, 2024

    Divyanshu Kumar, Anurakt Kumar, Sahil Agarwal, and Prashanth Harshangi. Fine-tuning, quantization, and llms: Navigating unintended outcomes, 2024

    2024

  17. [25]

    From generation to judgment: Opportunities and challenges of llm-as-a-judge, 2025

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu. From generation to judgment: Opportunities and challenges of llm-as-a-judge, 2025

    2025

  18. [26]

    Safety layers in aligned large language models: The key to llm security.https://arxiv.org/abs/2408.17003, 2024

    Shen Li, Liuyi Yao, Lan Zhang, and Yaliang Li. Safety layers in aligned large language models: The key to llm security.https://arxiv.org/abs/2408.17003, 2024. arXiv:2408.17003 [cs.CR]

    2024 arXiv

  19. [27]

    A hitchhiker’s guide to jailbreaking chatgpt via prompt engineering

    Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Kailong Wang. A hitchhiker’s guide to jailbreaking chatgpt via prompt engineering. InProceedings of the 4th International Workshop on Software Engineering and AI for Data Qualit...

    2024

  20. [28]

    Privacy perceptions of custom gpts by users and creators

    Rongjun Ma, Caterina Maidhof, Juan Carlos Carrillo, Janne Lindqvist, and Jose Such. Privacy perceptions of custom gpts by users and creators. InProceedings of the Conference on Human Factors in Computing Systems (CHI), pages 237:1–237:18. ACM, 2025

    2025

  21. [29]

    The trauma floor: The secret lives of facebook moderators in america, 2019

    Casey Newton. The trauma floor: The secret lives of facebook moderators in america, 2019. The Verge, February 25

    2019

  22. [30]

    Academic integrity policy

    Northeastern University, Office of Student Conduct and Conflict Resolution. Academic integrity policy. https: //osccr.sites.northeastern.edu/academic-integrity-policy/, 2025. Accessed: 2025-12-12

    2025

  23. [31]

    Introducing gpts: Custom versions of chatgpt for specific purposes

    OpenAI. Introducing gpts: Custom versions of chatgpt for specific purposes. https://openai.com/index/ introducing-gpts/, November 2023. Accessed: 2025-12-12

    2023

  24. [32]

    Openai red teaming network

    OpenAI. Openai red teaming network. https://openai.com/index/red-teaming-network/ , September

  25. [33]

    Usage policies

    OpenAI. Usage policies. https://openai.com/policies/usage-policies/, January 2024. Accessed: 2025-12-12

    2024

  26. [34]

    Openai safety.https://openai.com/safety/, 2025

    OpenAI. Openai safety.https://openai.com/safety/, 2025. Accessed: 2025-12-12. 18 Automated Policy Compliance Evaluation of Custom GPTs

    2025

  27. [35]

    Llm self defense: By self examination, llms know they are being tricked

    Mansi Phute, Alec Helbling, Matthew Hull, ShengYun Peng, Sebastian Szyller, Cory Cornelius, and Duen Horng Chau. Llm self defense: By self examination, llms know they are being tricked. https://arxiv.org/abs/ 2308.07308, 2023. arXiv:2308.07308 [cs.CL]

    2023 arXiv

  28. [36]

    Puppeteer documentation.https://pptr.dev/, 2025

    Puppeteer. Puppeteer documentation.https://pptr.dev/, 2025. Accessed: 2025-12-12

    2025

  29. [37]

    Fine-tuning aligned language models compromises safety, even when users do not intend to!, 2023

    Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to!, 2023

    2023

  30. [38]

    Improving language understanding by gen- erative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by gen- erative pre-training. https://cdn.openai.com/research-covers/language-unsupervised/language_ understanding_paper.pdf, 2018. OpenAI technical report

    2018

  31. [39]

    Language models are un- supervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are un- supervised multitask learners. https://cdn.openai.com/better-language-models/language_models_ are_unsupervised_multitask_learners.pdf, 2019. OpenAI preprint

    2019

  32. [40]

    Safetyprompts: a systematic review of open datasets for evaluating and improving large language model safety, 2025

    Paul Röttger, Fabio Pernisi, Bertie Vidgen, and Dirk Hovy. Safetyprompts: a systematic review of open datasets for evaluating and improving large language model safety, 2025

    2025

  33. [41]

    Iqbal H. Sarker. Llm potentiality and awareness: A position paper from the perspective of trustworthy and responsible ai modeling.Discover Artificial Intelligence, 4(1):40, May 2024

    2024

  34. [42]

    Identifying the provision of choices in privacy policy text

    Kanthashree Mysore Sathyendra, Shomir Wilson, Florian Schaub, Sebastian Zimmeck, and Norman Sadeh. Identifying the provision of choices in privacy policy text. InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2774–2779, Copenhagen...

    2017

  35. [43]

    do anything now

    Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. InProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, CCS ’24, pag...

    2024

  36. [44]

    Can people experience romantic love for artificial intelligence? an empirical study of intelligent assistants.Information & Management, 59(2):103595, 2022

    Xia Song, Bo Xu, and Zhenzhen Zhao. Can people experience romantic love for artificial intelligence? an empirical study of intelligent assistants.Information & Management, 59(2):103595, 2022

    2022

  37. [45]

    Gpt store mining and analysis, 2024

    Dongxun Su, Yanjie Zhao, Xinyi Hou, Shenao Wang, and Haoyu Wang. Gpt store mining and analysis, 2024

    2024

  38. [46]

    Safety assessment of chinese large language models, 2023

    Hao Sun, Zhexin Zhang, Jiawen Deng, Jiale Cheng, and Minlie Huang. Safety assessment of chinese large language models, 2023

    2023

  39. [47]

    Moin Syed and Sarah C. Nelson. Guidelines for establishing reliability when coding narrative data.Emerging Adulthood, 3(6):375–387, 2015

    2015

  40. [48]

    Opening a pandora’s box: Things you should know in the era of custom gpts, 2023

    Guanhong Tao, Siyuan Cheng, Zhuo Zhang, Junmin Zhu, Guangyu Shen, and Xiangyu Zhang. Opening a pandora’s box: Things you should know in the era of custom gpts, 2023

    2023

  41. [49]

    Glossary for academic integrity

    Loreta Tauginien˙e, Inga Gaižauskait˙e, Irene Glendinning, Július Kravjar, Milan Ojsteršek, Laura Ribeiro, Tatjana Odin, eca, Franca Marino, Marco Cosentino, Shiva Sivasubramaniam, and Tomáš Foltýnek. Glossary for academic integrity. ENAI Report 3G, revised version, 2023. Acce...

    2023

  42. [50]

    Ehsan Ullah, Anil Parwani, Mirza Mansoor Baig, and Rajendra Singh. Challenges and barriers of using large language models (llm) such as chatgpt for diagnostic medicine with a focus on digital pathology – a recent scoping review.Diagnostic Pathology, 19(1):43, February 2024

    2024

  43. [51]

    Academic misconduct

    University of California, Berkeley, Center for Student Conduct. Academic misconduct. https:// conduct.berkeley.edu/academic-misconduct/, 2025. Accessed: 2025-12-12

    2025

  44. [52]

    Definition of academic dis- honesty

    University of Colorado Denver, College of Liberal Arts and Sciences. Definition of academic dis- honesty. https://clas.ucdenver.edu/faculty-staff/policies-procedures/handling-academic- dishonesty/definition-academic-dishonesty, 2025. Accessed: 2025-12-12

    2025

  45. [53]

    Academic integrity

    University of Manitoba. Academic integrity. https://umanitoba.ca/student-supports/academic- supports/academic-integrity, 2025. Accessed: 2025-12-12

    2025

  46. [54]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InProceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Ho...

    2017

  47. [55]

    Definitions of academic miscon- duct

    Virginia Tech, Office of Undergraduate Academic Integrity. Definitions of academic miscon- duct. https://honorsystem.vt.edu/honor_code_policy_test/policy-and-manual/definitions_ of_academic_misconduct.html, 2025. Accessed: 2025-12-12. 19 Automated Policy Compliance Evaluation ...

    2025

  48. [56]

    Learning from failure: Integrating negative examples when fine-tuning large language models as agents, 2024

    Renxi Wang, Haonan Li, Xudong Han, Yixuan Zhang, and Timothy Baldwin. Learning from failure: Integrating negative examples when fine-tuning large language models as agents, 2024

    2024

  49. [57]

    Taxonomy of risks posed by language models

    Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, Courtney Biles, Sasha Brown, Zac Kenton, Will Hawkins, Tom Stepleton, Abeba Birhane, Lisa Anne Hendricks, Laura Rimell, William...

    2022

  50. [58]

    Shomir Wilson, Florian Schaub, F. Liu, K. M. Sathyendra, D. Smullen, S. Zimmeck, R. Ramanath, P. Story, F. Liu, N. Sadeh, and N. A. Smith. Analyzing privacy policies at scale: From crowdsourcing to automated annotations. ACM Transactions on the Web, 13(1):1–29, 2018

    2018

  51. [59]

    Sorry-bench: Systematically evaluating large language model safety refusal behaviors, 2024

    Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, Ruoxi Jia, Bo Li, Kai Li, Danqi Chen, Peter Henderson, and Prateek Mittal. Sorry-bench: Systematically evaluating large language model safety re...

    2024

  52. [60]

    Online safety analysis for llms: a benchmark, an assessment, and a path forward, 2024

    Xuan Xie, Jiayang Song, Zhehua Zhou, Yuheng Huang, Da Song, and Lei Ma. Online safety analysis for llms: a benchmark, an assessment, and a path forward, 2024

    2024

  53. [61]

    Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts, 2024

    Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts, 2024

    2024

  54. [62]

    Assessing prompt injection risks in 200+ custom gpts, 2024

    Jiahao Yu, Yuhang Wu, Dong Shu, Mingyu Jin, Sabrina Yang, and Xinyu Xing. Assessing prompt injection risks in 200+ custom gpts, 2024

    2024

  55. [63]

    Don’t listen to me: Understanding and exploring jailbreak prompts of large language models

    Zhiyuan Yu, Xiaogeng Liu, Shunning Liang, Zach Cameron, Chaowei Xiao, and Ning Zhang. Don’t listen to me: Understanding and exploring jailbreak prompts of large language models. In33rd USENIX Security Symposium (USENIX Security 24), pages 4675–4692, Philadelphia, PA, August 20...

    2024

  56. [64]

    S-eval: Automatic and adaptive test generation for benchmarking safety evaluation of large language models, 2024

    Xiaohan Yuan, Jinfeng Li, Dongxia Wang, Yuefeng Chen, Xiaofeng Mao, Longtao Huang, Hui Xue, Wenhai Wang, Kui Ren, and Jingyi Wang. S-eval: Automatic and adaptive test generation for benchmarking safety evaluation of large language models, 2024

    2024

  57. [65]

    Defending against neural fake news

    Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché Buc, E. Fox, and R. Garnett, editors,Advances in Neural Information Processin...

    2019

  58. [66]

    A first look at gpt apps: Landscape and vulnerability, 2024

    Zejun Zhang, Li Zhang, Xin Yuan, Anlan Zhang, Mengwei Xu, and Feng Qian. A first look at gpt apps: Landscape and vulnerability, 2024

    2024

  59. [67]

    Safetybench: Evaluating the safety of large language models, 2024

    Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. Safetybench: Evaluating the safety of large language models, 2024

    2024

  60. [68]

    Gonzalez, and Ion Stoica

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In A. Oh, T. Naumann, A. Globerson, K. Saenk...

    2023

  61. [69]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In A. Oh, T. Naumann, A. Globerson, K. Saenko...

    2023

  62. [70]

    Sadeh, Steven M

    Sebastian Zimmeck, Ziqi Wang, Lieyong Zou, Roger Iyengar, Bin Liu, Florian Schaub, Shomir Wilson, Norman M. Sadeh, Steven M. Bellovin, and Joel R. Reidenberg. Automated analysis of privacy requirements for mobile apps. InProceedings of the 2017 Network and Distributed System S...

    2017

  63. [74]

    Framework Source Code. The code of the framework is released to enable the community to build upon this work, facilitating further research and the development of new approaches to evaluating LLM compliance

  64. [75]

    boyfriends,

    Annotated Ground Truth Dataset. The manually crafted dataset used to validate theCompliance Assessment module is also shared, providing a benchmark for evaluating compliance detection systems and supporting revalidation efforts with future model updates or alternative LLMs, fa...

  65. [2017]

    Internet Society. A Open Science In line with principles of transparency, reproducibility, and fostering collaboration within the research community, we will publicly release the following resources upon acceptance of the paper: 20 Automated Policy Compliance Evaluation of Custom GPTs

  66. [2022]

    Association for Computing Machinery

  67. [2023]

    Accessed: 2025-12-12

    2025

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.