Pith. sign in

REVIEW 4 major objections 6 minor 28 references

Commercial AI system prompts often protect users only shallowly, and about 40% still contain instructions that work against them.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 02:02 UTC pith:YDN6QNZ3

load-bearing objection Solid first comparative audit of commercial system prompts: usable taxonomy, real org/time patterns, main caveat is leaked-corpus provenance already flagged by the authors. the 4 major comments →

arxiv 2607.28617 v1 pith:YDN6QNZ3 submitted 2026-07-30 cs.AI cs.CLcs.CYcs.HC

AISPA: User-Centric System Prompt Auditing for Large Language Model Applications

Xiangning Lin , Shenzhe Zhu , Shu Yang , Zhenyu Zhang , Haoqian Zhang , Yipeng Zhao , Chengxuan Qian , Tianwei Wang
show 18 more authors
Ziheng Zhang Zhenlong Yuan Dingcheng Wang Juncheng Wu Yuan Si Jiaxin Liu Baolong Bi Robert Mahari Tobin South Dazza Greenwood Zexue He Rishi Bommasani Sophia Kazinnik Andreas Haupt Samuele Marro Erik Brynjolfsson Alex Pentland Jiaxin Pei
This is my paper
classification cs.AI cs.CLcs.CYcs.HC
keywords system promptsAI auditinguser protectionLLM applicationstransparencyprompt governanceAISPAcommercial AI
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Hidden system prompts steer how commercial AI products behave before any user speaks, yet they are rarely disclosed or independently checked. This paper introduces AISPA, an eight-dimension, user-centered audit that labels individual instructions as protective or problematic. Applied to 3,249 instructions across 88 real products, the audit finds protection is nearly universal but thin: almost every product has at least one protective line, yet only about a quarter cover all eight dimensions. Prompts have grown longer and more protective over time, but roughly two in five products still include at least one instruction that undermines user interests, and protective and harmful directives often sit in the same prompt. Organizations differ sharply in how carefully they write these rules. The authors argue this layer needs transparency standards and third-party oversight without forcing full public disclosure.

Core claim

Across 88 commercial AI products, protective system-prompt instructions are near-universal (98.9% of products have at least one) but shallow (only about 24% cover all eight AISPA dimensions), while roughly 40% of products contain at least one problematic instruction that works against user interests, with large organization-level gaps and frequent coexistence of protective and problematic text in the same prompt.

What carries the argument

AISPA: a span-level audit taxonomy of eight user-rights dimensions (identity transparency, truthfulness, privacy, tool/action safety, user agency, unsafe-request handling, harm prevention, fairness/inclusion/neutrality), scoring each auditable instruction +1 protective or −1 problematic via a three-round human–LLM pipeline.

Load-bearing premise

The leaked or community-disclosed prompts from public repositories are authentic and representative enough of live commercial products to support product- and organization-level prevalence claims.

What would settle it

Obtain official current system prompts from a large, stratified sample of the same products and re-run the AISPA labeling; if comprehensive eight-dimension coverage rises well above ~24% and products with any problematic instruction fall well below ~40%, the prevalence claims fail.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Third-party pre-deployment prompt certification becomes a concrete accountability tool without requiring full public disclosure.
  • Product and organization rankings on protective versus problematic counts can become a public quality signal for users and regulators.
  • Prompt design standards can target the thinnest dimensions (notably privacy and unsafe-request handling) and the gray-area patterns of identity concealment and parasocial dependency.
  • Temporal growth in protective instructions can be tracked as an industry norm rather than left to ad hoc developer practice.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If certification status is public while prompt text stays private, market pressure may close the org-level gap faster than regulation alone.
  • Gray-area instructions (human mimicry, override keys, content-policy relaxations) may become the next contested frontier once binary problematic cases decline.
  • Specialized and agentic products may need dimension weights different from general chatbots, because autonomy–agency tradeoffs are structural there.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces AISPA, an eight-dimension, UDHR-anchored taxonomy and a three-round human–LLM workflow for span-level auditing of commercial LLM system prompts. Auditors label non-core-logic spans as protective (+1) or problematic (−1) along identity transparency, truthfulness, privacy, tool/action safety, user agency, unsafe-request handling, harm prevention, and fairness. Applying this protocol to leaked/disclosed prompts from 88 products, the authors report that protective instructions are near-universal but shallow (98.9% of products have ≥1; only ~24% cover all eight dimensions), that prompts have lengthened and grown more protective over 2024–2025, that organization-level protective density varies widely (e.g., Anthropic vs. weaker peers), and that roughly two-fifths of products still contain at least one user-adverse instruction, often coexisting with protective text. A separate gray-area analysis describes borderline design patterns (identity mimicry, parasocial cues, permission overrides, unrestricted content policies).

Significance. System prompts are a consequential and under-scrutinized control layer in deployed LLM products; a reusable user-centric audit codebook plus the first multi-product empirical map is a genuine contribution to AI governance and HCI/safety practice. Strengths include a clear unit of analysis (spans), explicit core vs. non-core scope rules, high calibration IAA (0.933), an asymmetric unanimous-expert bar for −1 labels, temporal and provider case studies, an honest limitations section, and Appendix B authenticity checks (maintainer contact and cross-repository Dice overlap). If the descriptive prevalence patterns hold under clearer sampling caveats, the work supplies concrete evidence for transparency, standardization, and third-party prompt review—without requiring full public disclosure of proprietary prompts.

major comments (4)
  1. [Abstract; §5.1] Abstract and opening claim “3,249 instructions,” but §5.1 reports a final dataset of 2,420 entries (2,346 protective + 74 problematic) from 1,818 spans, plus 44 gray-area entries. This is a load-bearing numerical inconsistency for the paper’s headline audit scale. Please reconcile the abstract, §1/§8, and §5.1 (e.g., candidate spans vs. retained entries vs. gray-area) and use one consistent accounting everywhere.
  2. [§5.2.2; Figure 1; Figure 7] Finding 1 and Figures 1/7 rank organizations by average protective/problematic counts, but many organizations appear to contribute a single product (or very few). With n_org often ≈1, “organization averages” are product point estimates and are sensitive to category mix (chatbot vs. coding agent). Report n per organization, avoid over-interpreting single-product ranks as institutional policy, and consider category-stratified or mixed-effects summaries before strong org-level claims.
  3. [Abstract; §5.1; Limitations; Appendix B] The central prevalence claims (98.9% ≥1 protective; ~38.6% ≥1 problematic; shallow eight-dimension coverage) are generalized to “commercial AI products,” yet the corpus is leaked/community-disclosed prompts with acknowledged selection toward extractable systems (Limitations; Appendix B: 38/88 lack cross-repo matches). The validation work is real but incomplete for product- and market-level inference. Tighten abstract/conclusion wording to “in this leaked corpus,” quantify how sensitive headline percentages are to excluding low-overlap or single-source prompts, and state what claims remain if the sample is biased toward longer or more jailbreak-exposed prompts.
  4. [§5.2.1; Figure 5] Figure 5’s temporal story (longer, more protective prompts; fluctuating problematic rates) bins heterogeneous products with small per-bin n (e.g., 2025-Q1 n=9) and does not control for shifting category composition or uncertain leak dates vs. deployment dates. Composition change could mimic “industry learning.” Either restrict the panel to repeated product lines (as in the Claude/GPT/Grok case study in Figure 8) or show category-adjusted trends and uncertainty; otherwise soften causal language about protection “becoming a more visible concern.”
minor comments (6)
  1. [Abstract; §5.2.1] Abstract says “only 24%” and “roughly 40%” while §5.2.1 gives 23.9% and 38.6%; keep one precise figure and use “approximately” consistently.
  2. [Figure 6; §5.2.1] Figure 6(b) percentages for problematic D5/D2 (18% / 15% in the plot text vs. 18.2% / 14.8% in prose) should be aligned and include absolute counts.
  3. [§4.1; §5.1] Define “entry” vs. “span” vs. “instruction” once early (footnote 8 helps) and use it uniformly in figures and the abstract.
  4. [Table 1; §9] Table 1 examples are effective; briefly note whether quotes are lightly redacted and whether products are identifiable from the full audit release plan.
  5. [§7] Related Work (§7) is thin on prompt-injection/hardening and industry model-spec literature; a short contrast paragraph would better situate AISPA’s user-protective (not system-defensive) stance.
  6. [§8; title page] Minor prose issues: “comprises of” → “comprises” (§8); ensure arXiv/author URL formatting is consistent in the header.

Circularity Check

0 steps flagged

No circular derivation: observational span-level audit with predefined taxonomy applied to external prompts.

full rationale

AISPA is an empirical content-analysis paper, not a first-principles derivation. The load-bearing claims are descriptive prevalence statistics (e.g., 98.9% of products have ≥1 protective entry; 23.9% cover all eight dimensions; ~38.6% have ≥1 problematic entry; org-level averages such as Anthropic 62.3 protective / 0.1 problematic) obtained by applying a pre-specified eight-dimension codebook and +1/−1 polarity rules to leaked system-prompt text from 88 products. The taxonomy is defined before the audit (Section 3; Table 1; UDHR anchors), spans are external artifacts, labels come from a three-round human–LLM protocol with reported IAA (0.933) and unanimous-expert threshold for −1, and time/org trends are counts over that labeled corpus—not fitted parameters re-exported as predictions. Related-work self-citations (e.g., co-author safety/agent papers) are background and do not force the prevalence results. No self-definitional loop, fitted-input-as-prediction, uniqueness import, or ansatz-via-citation chain appears in the claimed findings. Theory-ladenness of ‘protective’ vs ‘problematic’ is a construct-validity issue, not circularity by construction. Score 0.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 3 invented entities

Load-bearing structure is normative and methodological rather than parametric. Claims rest on (i) treating system-prompt text as a primary behavioral governor, (ii) UDHR-mapped user-interest dimensions as the right audit axes, (iii) span-level ±1 coding as a valid proxy for user protection, and (iv) leaked corpus authenticity/representativeness. No physical constants or curve-fit parameters drive the headlines; free choices are taxonomy design and adjudication thresholds.

free parameters (3)
  • Unanimous three-expert threshold for retaining −1 labels = 3/3 experts must agree on −1
    Asymmetric adjudication rule that directly shapes the reported problematic prevalence; chosen for reputational caution, not estimated from external outcome data.
  • Eight-dimension AISPA partition and span merge rules = 8 dimensions; sentence-default spans
    Hand-designed taxonomy and ‘same intent → one span’ rules determine what counts as coverage of ‘all eight dimensions’ and entry counts per product.
  • Temporal binning (2024 aggregate vs 2025 quarters; n=4 in 2026 dropped) = 2026 excluded (n=4)
    Binning and exclusion choices affect the reported growth narrative in Figure 5.
axioms (5)
  • domain assumption Developer system prompts are a primary, persistent control layer that can override or reshape aligned base-model behavior in deployed products.
    Stated throughout §§1–2; underpins why prompt text audit is treated as user-protection evidence without paired behavioral A/B tests.
  • ad hoc to paper User-relevant prompt quality is adequately captured by eight UDHR-anchored dimensions with binary protective/problematic polarity on non-core-logic spans.
    §3 taxonomy design; alternative dimensions or graded severity would change coverage and prevalence statistics.
  • domain assumption Leaked or community-disclosed prompts, after maintainer and cross-repo checks, are valid objects for product- and organization-level claims.
    §5.1 and Appendix B; Limitations explicitly flags imperfect production fidelity.
  • domain assumption Presence of a protective instruction indicates meaningful adoption of that safeguard dimension at the prompt layer (without requiring proof of runtime enforcement).
    Used when reporting 98.9% ≥1 protective and per-dimension coverage in Figure 6.
  • standard math Standard qualitative reliability practices (calibration, IAA, expert adjudication) suffice to treat labels as stable enough for aggregate statistics.
    §5.1 reports pairwise IAA 0.933 on 20 spans; conventional content-analysis assumption.
invented entities (3)
  • AISPA eight-dimension span-level audit taxonomy no independent evidence
    purpose: Provide a unified protective/problematic coding scheme for system prompts tied to UDHR articles.
    Core methodological invention; dimensions remix known safety themes but the operational package is paper-specific.
  • Three-round LLM→annotator→expert prompt auditing protocol no independent evidence
    purpose: Scale span discovery while controlling false problematic labels.
    Procedural construct; validated only via internal IAA/adjudication, not external outcome benchmarks.
  • Gray-area / risky span category (four patterns) no independent evidence
    purpose: Hold borderline design trade-offs outside the ±1 main tallies.
    Post-hoc expert typology on 29 spans; useful descriptively but not independently measured against user harm.

pith-pipeline@v1.2.0-daily-grok45 · 24683 in / 3820 out tokens · 81074 ms · 2026-07-31T02:02:21.384378+00:00 · methodology

0 comments
read the original abstract

System prompts are instructions configured by developers to govern the behaviors of foundation models in AI applications. They are used throughout commercial AI products, but are rarely disclosed to the public or regulators, creating a serious trust and accountability gap in the wide deployment of AI systems. In this paper, we introduce Artificial Intelligence System Prompt Assurance (AISPA), a user-centric framework for systematically auditing system prompts in AI systems. AISPA examines specific parts of a system prompt and evaluates them along eight dimensions that matter to users. We then use this framework to review 3,249 instructions from system prompts in 88 commercial AI products, classifying each instruction as either protective (of users) or problematic. Our audit surfaces four core findings. First, system prompt design varies substantially across products and developers, with some organizations averaging over 60 protective instructions per product while others average fewer than 5. Second, protective instructions are widely adopted but shallow in scope: 98.9% of products contain at least one, yet only 24% cover all eight dimensions of the AISPA taxonomy. Third, system prompts have grown steadily longer and more protective of users, suggesting that user protection is becoming a more visible concern in commercial prompt design. Fourth, despite this progress, problematic instructions remain pervasive: roughly 40% of products contain at least one instruction that works against user interests, and protective and problematic instructions frequently coexist within the same prompt. Our findings highlight the need for greater transparency, standardization, and independent oversight for system prompts in commercial AI products.

Figures

Figures reproduced from arXiv: 2607.28617 by Alex Pentland, Andreas Haupt, Baolong Bi, Chengxuan Qian, Dazza Greenwood, Dingcheng Wang, Erik Brynjolfsson, Haoqian Zhang, Jiaxin Liu, Jiaxin Pei, Juncheng Wu, Rishi Bommasani, Robert Mahari, Samuele Marro, Shenzhe Zhu, Shu Yang, Sophia Kazinnik, Tianwei Wang, Tobin South, Xiangning Lin, Yipeng Zhao, Yuan Si, Zexue He, Zhenlong Yuan, Zhenyu Zhang, Ziheng Zhang.

Figure 1
Figure 1. Figure 1: Overview of system prompt quality across organizations. Bars show the average number of [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The eight auditing dimensions in AISPA. Each dimension targets a distinct aspect of responsible system prompt design. Our taxonomy comprises eight dimensions, each designed to capture both protective and problematic instructions within a single evaluative framework. For example, Identity Transparency includes both explicit disclosure that the system is AI (protective) and deliberate concealment of its AI n… view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of span-level prompt auditing. A fictitious system prompt is segmented into spans. Each highlighted span is assigned a dimension and polarity under the taxonomy: the blue-toned span denotes a protective span (+1), while the red-toned span denotes a problematic span (−1). Unhighlighted sentences are not flagged by the auditor. 4.1 Auditing Guidelines Unit of analysis: Prompt Span. The basic uni… view at source ↗
Figure 4
Figure 4. Figure 4: Three-round collaborative audit protocol. In Round 1, an LLM pre-annotator identifies candidate spans with high recall. In Round 2, trained annotators screen these proposals to improve precision. In Round 3, three domain experts review the remaining cases to ensure consistency, with unanimous agreement required for all problematic (−1) labels. Round 2: Trained annotator screening. Before this round, all an… view at source ↗
Figure 5
Figure 5. Figure 5: Temporal trends in system prompt evolution from 2024 to 2025. (a) Average number of +1 (protective) entries per product. (b) Average system prompt length in characters. (c) Percentage of products containing at least one −1 (problematic) entry. To ensure balanced representation across time periods, products from 2024 are aggregated into a single bin, while products from 2025 are grouped by quarter. Products… view at source ↗
Figure 6
Figure 6. Figure 6: Prevalence of user protection and problematic entries. (a) Product-level coverage: percent￾age of the 88 products that contain at least one +1 entry, at least one −1 entry, or +1 entries across all eight dimensions. (b) Dimension-level coverage: for each dimension, the percentage of products with at least one +1 (protective, blue) or −1 (problematic, pink) entry. Problematic instructions have been declinin… view at source ↗
Figure 7
Figure 7. Figure 7: Organization-level ranking based on the rating of system prompts. (a) Average protective spans per product. (b) Average problematic spans per product. Amazon and Cline follow with strong protective counts of 42.0 and 39.5, respectively, while having few problematic instructions. At the other extreme, Venice is the only organization whose average problematic count (3.0) exceeds its average protective count … view at source ↗
Figure 8
Figure 8. Figure 8: The evolution of system prompts for frontier models released by Anthropic, OpenAI and xAI. For Claude, GPT, and Grok model series, the system prompts gradually become more protective. Case study: version evolution across three providers [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Distribution of 88 products across five categories. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Cross-repository content overlap for each matched product. For each of the 88 prompts, we compute the Sørensen–Dice overlap with all same-product files in the other five repositories and retain the highest score. 50 prompts have at least one cross-repository match; 38 are unique to their source repository. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Annotation platform example [PITH_FULL_IMAGE:figures/full_fig_p022_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Annotation platform example with human audited results. E Supplementary Analyses We report additional descriptive analyses that provide context for the main findings. These analyses characterize the structure of the audit dataset but do not constitute independent findings. E.1 Category-Level and Prompt-Level Patterns Short prompts are disproportionately vulnerable [PITH_FULL_IMAGE:figures/full_fig_p022_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Prompt size vs. user protection balance. Each bubble represents a product; size encodes the number of annotated spans and color encodes the protective rate. Products with the lowest protection balance are consistently those with short prompts. E.2 Dimension-Level Analyses Dimension Co-occurrence. Nearly a quarter of all spans (441 of 1,818) address two or more dimensions simultaneously, indicating that ma… view at source ↗
Figure 14
Figure 14. Figure 14: Word clouds for the eight auditing dimensions. Each panel shows the most frequent terms in annotated spans for that dimension, after removing domain-generic stopwords. Word Clouds show Dimension-Specific Vocabularies [PITH_FULL_IMAGE:figures/full_fig_p023_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

28 extracted references · 14 linked inside Pith

  1. [4]

    Dated data: Tracing knowledge cutoffs in large language models.arXiv preprint arXiv:2403.12958,

    Jeffrey Cheng, Marc Marone, Orion Weller, Dawn Lawrie, Daniel Khashabi, and Benjamin Van Durme. Dated data: Tracing knowledge cutoffs in large language models.arXiv preprint arXiv:2403.12958,

    Pith/arXiv arXiv

  2. [5]

    Who audits the auditors? recommendations from a field scan of the algorithmic auditing ecosystem

    Sasha Costanza-Chock, Inioluwa Deborah Raji, and Joy Buolamwini. Who audits the auditors? recommendations from a field scan of the algorithmic auditing ecosystem. InProceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1571–1583,

    2022

  3. [6]

    Wesley Hanwen Deng, Wang Claire, Howard Ziyu Han, Jason I Hong, Kenneth Holstein, and Motahhare Eslami

    URL https://arxiv.org/abs/2310.00907. Wesley Hanwen Deng, Wang Claire, Howard Ziyu Han, Jason I Hong, Kenneth Holstein, and Motahhare Eslami. Weaudit: Scaffolding user auditors and ai practitioners in auditing generative ai.Proceedings of the ACM on Human-Computer Interaction, 9(7):1–35,

    Pith/arXiv arXiv

  4. [7]

    Regulation (EU) 2024/1689

    URL https://eur-lex.europa.eu/eli/reg/202 4/1689/oj. Regulation (EU) 2024/1689. Edwin Farley. Ai auditing: First steps towards the effective regulation of artificial intelligence systems. Available at SSRN 4676184,

    2024

  5. [11]

    Scalable extraction of training data from (production) language models.arXiv preprint arXiv:2311.17035,

    Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models.arXiv preprint arXiv:2311.17035,

    Pith/arXiv arXiv

  6. [12]

    Anna Neumann, Yulu Pi, and Jatinder Singh

    URL https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf. Anna Neumann, Yulu Pi, and Jatinder Singh. Who controls the conversation? user perspectives on generative ai (llm) system prompts. InProceedings of the 2026 CHI Conference on Human Factors in Computing Systems, pages 1–37,

    2026

  7. [13]

    Jaildam: Jailbreak detection with adaptive memory for vision-language model.arXiv preprint arXiv:2504.03770,

    Yi Nian, Shenzhe Zhu, Yuehan Qin, Li Li, Ziyi Wang, Chaowei Xiao, and Yue Zhao. Jaildam: Jailbreak detection with adaptive memory for vision-language model.arXiv preprint arXiv:2504.03770,

    Pith/arXiv arXiv

  8. [14]

    Outsider oversight: Designing a third party audit ecosystem for ai governance

    Inioluwa Deborah Raji, Peggy Xu, Colleen Honigsberg, and Daniel Ho. Outsider oversight: Designing a third party audit ecosystem for ai governance. InProceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pages 557–571,

    2022

  9. [15]

    Alex Tamkin, Miles Brundage, Jack Clark, and Deep Ganguli

    URL https://arxiv.org/abs/2204.10814. Alex Tamkin, Miles Brundage, Jack Clark, and Deep Ganguli. Understanding the capabilities, limitations, and societal impact of large language models.arXiv preprint arXiv:2102.02503,

    Pith/arXiv arXiv

  10. [17]

    Law and the emerging political economy of algorithmic audits

    Petros Terzis, Michael Veale, and Noëlle Gaumann. Law and the emerging political economy of algorithmic audits. InProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 1255–1267,

    2024

  11. [18]

    Openagentsafety: A comprehensive framework for evaluating real-world ai agent safety.arXiv preprint arXiv:2507.06134,

    17 Sanidhya Vijayvargiya, Aditya Bharat Soni, Xuhui Zhou, Zora Zhiruo Wang, Nouha Dziri, Graham Neubig, and Maarten Sap. Openagentsafety: A comprehensive framework for evaluating real-world ai agent safety.arXiv preprint arXiv:2507.06134,

    arXiv

  12. [19]

    Ethical and social risks of harm from language models.arXiv preprint arXiv:2112.04359,

    Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models.arXiv preprint arXiv:2112.04359,

    Pith/arXiv arXiv

  13. [20]

    Sorry-bench: Systematically evaluating large language model safety refusal.arXiv preprint arXiv:2406.14598,

    Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, et al. Sorry-bench: Systematically evaluating large language model safety refusal.arXiv preprint arXiv:2406.14598,

    Pith/arXiv arXiv

  14. [21]

    Toolsafety: A comprehensive dataset for enhancing safety in llm-based agent tool invocations

    Yuejin Xie, Youliang Yuan, Wenxuan Wang, Fan Mo, Jianmin Guo, and Pinjia He. Toolsafety: A comprehensive dataset for enhancing safety in llm-based agent tool invocations. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 14146–14167,

    2025

  15. [22]

    Exploring the personality traits of llms through latent features steering.arXiv preprint arXiv:2410.10863,

    Shu Yang, Shenzhe Zhu, Liang Liu, Lijie Hu, Mengdi Li, and Di Wang. Exploring the personality traits of llms through latent features steering.arXiv preprint arXiv:2410.10863,

    Pith/arXiv arXiv

  16. [23]

    Fraud-r1: A multi-round benchmark for assessing the robustness of llm against augmented fraud and phishing inducements.arXiv preprint arXiv:2502.12904,

    Shu Yang, Shenzhe Zhu, Zeyu Wu, Keyu Wang, Junchi Yao, Junchao Wu, Lijie Hu, Mengdi Li, Derek F Wong, and Di Wang. Fraud-r1: A multi-round benchmark for assessing the robustness of llm against augmented fraud and phishing inducements.arXiv preprint arXiv:2502.12904,

    Pith/arXiv arXiv

  17. [24]

    Is your llm-based multi-agent a reliable real-world planner? exploring fraud detection in travel planning

    Junchi Yao, Jianhua Xu, Tianyu Xin, Ziyi Wang, Shenzhe Zhu, Shu Yang, and Di Wang. Is your llm-based multi-agent a reliable real-world planner? exploring fraud detection in travel planning. arXiv preprint arXiv:2505.16557,

    Pith/arXiv arXiv

  18. [25]

    Cognitive mirage: A review of hallucinations in large language models.arXiv preprint arXiv:2309.06794,

    Hongbin Ye, Tong Liu, Aijia Zhang, Wei Hua, and Weiqiang Jia. Cognitive mirage: A review of hallucinations in large language models.arXiv preprint arXiv:2309.06794,

    Pith/arXiv arXiv

  19. [26]

    Evaluating interfaced llm bias

    Kai-Ching Yeh, Jou-An Chi, Da-Chen Lian, and Shu-Kai Hsieh. Evaluating interfaced llm bias. In Proceedings of the 35th Conference on Computational Linguistics and Speech Processing (ROCLING 2023), pages 292–299,

    2023

  20. [27]

    Harmtransform: Transforming explicit harmful queries into stealthy via multi-agent debate.arXiv preprint arXiv:2512.23717,

    Shenzhe Zhu. Harmtransform: Transforming explicit harmful queries into stealthy via multi-agent debate.arXiv preprint arXiv:2512.23717,

    arXiv

  21. [28]

    The automated but risky game: Modeling and benchmarking agent-to-agent negotiations and transactions in consumer markets.arXiv preprint arXiv:2506.00073,

    Shenzhe Zhu, Jiao Sun, Yi Nian, Tobin South, Alex Pentland, and Jiaxin Pei. The automated but risky game: Modeling and benchmarking agent-to-agent negotiations and transactions in consumer markets.arXiv preprint arXiv:2506.00073,

    arXiv

  22. [2019]

    Accessed: 2025; provides foundational principles for human-centric and trustworthy AI development and deployment

    URL https://digital-strateg y.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai . Accessed: 2025; provides foundational principles for human-centric and trustworthy AI development and deployment. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A surv...

    2025

  23. [2021]

    Alert: A comprehensive benchmark for assessing large language models’ safety through red teaming.arXiv preprint arXiv:2404.08676,

    Simone Tedeschi, Felix Friedrich, Patrick Schramowski, Kristian Kersting, Roberto Navigli, Huu Nguyen, and Bo Li. Alert: A comprehensive benchmark for assessing large language models’ safety through red teaming.arXiv preprint arXiv:2404.08676,

    Pith/arXiv arXiv

  24. [2022]

    doi: 10.18653/v1/2022.acl-long.229

    Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.229. URLhttps://aclanthology.org/2022.acl-long.229/. Xiaoou Liu, Tiejin Chen, Longchao Da, Chacha Chen, Zhen Lin, and Hua Wei. Uncertainty quantification and confidence calibration in large language models: A survey. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Disco...

    doi:10.18653/v1/2022.acl-long.229 2022

  25. [2023]

    Black-box access is insufficient for rigorous ai audits

    Stephen Casper, Carson Ezell, Charlotte Siegmann, Noam Kolt, Taylor Lynn Curtis, Benjamin Bucknall, Andreas Haupt, Kevin Wei, Jérémy Scheurer, Marius Hobbhahn, et al. Black-box access is insufficient for rigorous ai audits. InProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 2254–2272,

    2024

  26. [2024]

    Frontier ai auditing: Toward rigorous third-party assessment of safety and security practices at leading ai companies.arXiv preprint arXiv:2601.11699,

    Miles Brundage, Noemi Dreksler, Aidan Homewood, Sean McGregor, Patricia Paskov, Conrad Stosz, Girish Sastry, A Feder Cooper, George Balston, Steven Adler, et al. Frontier ai auditing: Toward rigorous third-party assessment of safety and security practices at leading ai companies.arXiv preprint arXiv:2601.11699,

    arXiv

  27. [2025]

    doi: 10.1145/3703155

    ISSN 1558-2868. doi: 10.1145/3703155. URL http://dx.doi.org/10.1145/3703155. Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Ying Wang. A survey on fairness in large language models.arXiv preprint arXiv:2308.10149,

    Pith/arXiv arXiv

  28. [2026]

    Luca Beurer-Kellner, Beat Buesser, Ana-Maria Cre¸ tu, Edoardo Debenedetti, Daniel Dobos, Daniel Fabian, Marc Fischer, David Froelicher, Kathrin Grosse, Daniel Naeff, et al

    URLhttps://www-cdn.anthropic.com/6a5fa276ac68b9aeb0c 8b6af5fa36326e0e166dd.pdf. Luca Beurer-Kellner, Beat Buesser, Ana-Maria Cre¸ tu, Edoardo Debenedetti, Daniel Dobos, Daniel Fabian, Marc Fischer, David Froelicher, Kathrin Grosse, Daniel Naeff, et al. Design patterns for securing llm agents against prompt injections.arXiv preprint arXiv:2506.08837,

    Pith/arXiv arXiv