Pith. sign in

REVIEW 5 major objections 6 minor 43 references

How Good LLM-Generated Password Policies Are?

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper uses parameter-by-parameter comparison to show that LLM-generated pwquality.conf password policies are frequently inconsistent, hallucinated, incomplete, or incorrect, and therefore not sound for direct deployment in Linux PAM.

desk verdict A useful warning buried under an unreproducible evaluation: the qualitative finding is credible, but the published algorithms crash on hallucinated parameters and no data or code are released. read the letter →

arxiv 2506.08320 v2 pith:XGVI3XTZ submitted 2025-06-10 cs.CR cs.AI

Vivek Vaidya , Aditya Patwardhan , Ashish Kundu This is my paper · ORCID
classification cs.CRcs.AI
keywords LLM-generatedconfigurationfilespwquality.confLinuxPAMpasswordpolicyconsistencyevaluationhallucinationcorrectnessbenchmarkaccesscontrol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a large language model can be trusted to translate a natural-language password policy into a pwquality.conf file that Linux's PAM password-quality module will actually enforce. The paper defines a generated configuration as sound only if repeated generations are functionally equivalent, every parameter matches a human-built gold-standard file, no fabricated parameters appear, and no required parameter is missing. In tests of several LLMs, with and without the official pwquality.conf documentation supplied in the prompt, output quality varied widely: some models were largely consistent, while others hallucinated parameters, omitted assignments, or emitted files that PAM would reject or silently override. The paper concludes that LLM-generated password policies are not ready for direct deployment and that a validation step is needed before the files are placed into system configuration.

What carries the argument

The load-bearing object is pwquality.conf, the configuration file read by the pam_pwquality.so PAM module, whose parameters such as minlen and difok govern password complexity and which silently falls back to built-in defaults when the file is absent, malformed, or contains an invalid assignment. The argument is carried by three comparison algorithms: one generates several responses per prompt and averages pairwise comparisons; a second parses each generated file into a parameter-value hashmap, overlays documented defaults, and counts matching, differing, and hallucinated parameters; a third compares each generated file against a manually constructed gold-standard benchmark to score correctness and incompleteness. These algorithms turn the question of whether an LLM output is trustworthy into countable differences in parameter assignments.

What would settle it

Take the same natural-language policy prompts, have independent Linux administrators write their own pwquality.conf files by hand, and compare those files to the paper's benchmarks; if the independent translations disagree substantially, the reported correctness numbers are not a stable measure of LLM accuracy. A complementary runtime check is to feed each generated file through an actual pam_pwquality.so test stack and compare the enforced password rules to the intended policy, since malformed assignments can make PAM fall back to weaker defaults without any error being shown.

Watch

Extended reading notes

Core claim

The paper's central claim is that current LLMs cannot yet be relied on to produce sound password-policy configuration files, and that their failures take specific, measurable forms. Consistency is defined as functional equivalence: two generated files are consistent when they enforce the same effective parameter values, even if one writes minlen=8 and the other omits it because PAM supplies the same default. Correctness is scored parameter-by-parameter against a manually written benchmark pwquality.conf for each natural-language prompt, hallucination is the presence of parameters the real file does not define, and incompleteness is any missing parameter that silently falls back to a default. Across the models tested, the two best-scoring models generally scored highest on accuracy and lowest on hallucination, one model that hallucinated the most was nonetheless the most functionally consistent, and one model that refused to generate content scored well only because blank files inherit the defaults. Supplying the official documentation did not reliably help, and for some models it increased hallucination. The paper concludes that some LLMs show high consistency while others show discrepancies and hallucinations that could lead to inconsistent access control enforcement across a network.

Load-bearing premise

The manual translation of each natural-language password policy into a gold-standard pwquality.conf file is correct; if that translation is wrong, the correctness and soundness scores do not measure what they claim to measure.

Editorial extensions

If this is right

  • An administrator who copies an LLM-generated pwquality.conf into a Linux system without validation risks a password policy that is weaker than the one requested, because missing or invalid parameters make PAM fall back to built-in defaults.
  • Consistency evaluations should compare effective enforced behavior rather than raw text, since hallucinated parameters PAM ignores do not change enforcement, while a real parameter set to an invalid value can disable the whole file.
  • Supplying official documentation is not a reliable corrective: the paper finds the effect is model-dependent, sometimes reducing hallucination and sometimes increasing it.
  • The same parameter-by-parameter comparison method can be carried over to other security-sensitive configuration files, such as login.defs or sshd_config, where a misconfiguration could open a vulnerability.
  • Wide variance between models means an organization cannot adopt 'LLM-generated configs' as a category; each model needs individual validation before use.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A runtime validation harness that actually calls pam_pwquality.so would be a stronger check than parameter comparison alone, because the paper's own outlier examples show that a file can look correct yet be rejected or overridden by the PAM module.
  • The blank-file model scoring near-perfect accuracy exposes a scoring hazard: prompts whose intended settings are close to PAM defaults reward empty output, so future benchmarks should force at least one parameter away from its default to distinguish real generation from non-generation.
  • If LLM-generated configuration files become common, prompt injection becomes a plausible attack: a poisoned prompt could make a model emit a hallucinated parameter that causes PAM to fall back to weak defaults, a direction the paper names as future work but does not test.
  • Because the documentation effect varies by model, a single recipe for adding reference documentation to prompts is unlikely to work for every model; per-model grounding strategies would be a natural next step.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper asks how well LLMs translate natural-language password policies into Linux pwquality.conf files. It defines four properties — consistency, correctness, hallucination, and incompleteness — and aggregates them into a notion of "soundness." The experiments prompt seven LLMs (GPT-4o-mini, GPT-o3-mini, Cohere, Gemini, DeepSeek, Llama-3, Bloom) five times per policy prompt, both with and without the official pwquality.conf documentation, and compare outputs pairwise and against manually constructed gold-standard files using three algorithms. The reported results indicate that models such as Bloom return blank files, Cohere hallucinates parameters such as check_userpass, and only the OpenAI models approach high consistency and correctness; the paper concludes that LLM-generated policies are not yet sound for direct deployment. The qualitative observation that LLMs are inconsistent and hallucinate on this task is credible from the examples, but the quantitative pipeline as published is not reproducible because key algorithmic operations are undefined and no raw data or benchmark files are provided.

Significance. If the results are made reproducible, this is a useful and timely contribution: it studies a concrete, security-relevant configuration task rather than abstract consistency, and the two-framework design (with and without official documentation) provides a practical estimate of whether supplying documentation helps or hurts. The paper's strengths include concrete failure examples (missing '=' signs, bracketed headers, blank responses), explicit consideration of PAM failsafe behavior, and the attempt to define soundness for generated configurations. However, the paper currently ships no code, no benchmark files, and no raw outputs; the central metric algorithms have undefined cases; and the statistical support consists of point averages over five generations without variance, confidence intervals, or significance tests. These issues must be fixed before the quantitative claims can be relied upon.

major comments (5)
  1. [Algorithm 2, lines 12–14] The comparison `resps1[k] == resps2[k]` is undefined for hallucinated parameters that occur in only one of the two responses, because the dictionaries are initialized from the default parameter list and contain no entry for the missing key. A hallucinated parameter such as Cohere's `check_userpass` (Section V-A and V-D) will be present in one response but absent from the other, so this line raises a KeyError unless the implementation silently deviates from the pseudocode. The paper does not state whether missing keys are treated as absent, as a sentinel, or as unequal, and each choice changes `num_hal`, `num_same`, and the denominator `len`. Since the same procedure is reused by Algorithm 3 with the benchmark as Response2, this issue propagates to the correctness and incompleteness metrics in Figures 3–6. The published algorithms are therefore not reproducible for the central quantitative claims.
  2. [Algorithms 1 and 3, accumulation and return statements] The notation `total += ResponseComparison(...)` is ambiguous. If `total` is a Python list, `+=` concatenates, and then `return total[2]/iterations` in Algorithm 3 returns the third component of the first iteration only, not an average over iterations. If element-wise vector addition is intended, the pseudocode must say so explicitly. The same ambiguity affects Algorithm 1, where `total[0]/sum`, `total[1]/sum`, and `total[2]/sum` are returned as averages. This is load-bearing because every reported consistency, hallucination, and correctness score is computed from these accumulations.
  3. [Section V-B and V-C, statistical reporting] All quantitative claims in Section V are based on five generations per model and prompt, yet the paper reports only point averages without error bars, confidence intervals, significance tests, or raw output data. With n = 5, statements such as "o3-mini outscoring 4o-mini" in Section V-C and "Cohere is actually the stronger performer" in Section V-B may reflect sampling noise. At minimum, the paper should show per-generation distributions or bootstrapped intervals and release the raw generated files for independent analysis. This is not a presentation nicety: the ordering of models by consistency and correctness is the paper's main quantitative result.
  4. [Section III-A.2 and Figure 5, benchmark validity] The gold-standard benchmark files are manually constructed by the authors, are not released, and are not independently validated. Correctness is therefore a measure of agreement with one team's translation of natural-language policies into pwquality.conf parameters; if that translation is wrong or idiosyncratic, the correctness numbers in Figure 5 do not measure what they claim. The paper should release the benchmark files, the mapping from each policy sentence to the chosen parameters, and ideally inter-annotator agreement on the translation step.
  5. [Section III-C vs Section IV-A, definition of soundness] Section III-C defines an LLM-generated policy as sound "if it generates functionally equivalent files," while Section IV-A defines a configuration as sound if it meets all four criteria (consistency, correctness, absence of hallucination, and completeness). These are different definitions, and the Conclusion uses the four-criteria version. The paper should explicitly reconcile the two definitions or state that they refer to different objects (policy soundness versus configuration soundness); otherwise the aggregate soundness claim is ambiguous.
minor comments (6)
  1. [Algorithm 2, line 12] The expression `resps1.keys() + resps2.keys()` is not valid Python and is unclear as pseudocode; write it as the union of the two key sets.
  2. [Section V-E and Table II] The text and Figures 7–9 refer to "Prompt 4 vs Prompt 5," but Table II labels the two outlier policies as P1 and P2. Please align the numbering.
  3. [References] Reference [4] is cited as "A. L. et. al." and reference [35] as "B. Workshop, :, and T. L. S. et. al."; both author lists are corrupted and need to be completed.
  4. [Section V-A and Figure 5] The text notes that Bloom returns blank files and that this "still doesn't score badly on accuracy," but Figure 5 shows Bloom at or near 100% for the outlier prompts. Please clarify that blank files are counted as fully correct by construction because missing parameters are mapped to defaults, and discuss whether this conflates incompleteness with correctness.
  5. [Algorithm 1, line 9] Write the number of pairs as `iterations * (iterations - 1) / 2` to avoid any ambiguity about operator precedence.
  6. [Throughout Section V] Model names are spelled inconsistently: "Deepseek" vs "DeepSeek" and "GPT o3-mini" vs "o3-mini." Standardize the naming.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical evaluation compares LLM outputs against external pwquality.conf documentation and human-constructed benchmarks, and the self-citation [22] is motivational, not load-bearing.

full rationale

The paper's central claims (LLM-generated pwquality.conf files are inconsistent, hallucinated, and often incorrect relative to official documentation) are obtained by generating LLM responses and comparing them parameter-by-parameter with the official pwquality.conf defaults and with manually constructed benchmark files. No parameter is fitted from the model outputs and then reported as a prediction; the consistency, hallucination, and correctness numbers are computed directly from the generated files and the published man-page defaults (reference [10]). The human-authored benchmark is a ground-truth standard, not a function of the models' outputs; even if the benchmark is not released and the manual translation could be contested, the measurement is not circular. Reference [22] is authored by the current authors and is used only as prior motivation ('A more recent study [22] has also concluded that while LLMs have made significant strides in recent times, they are still inconsistent and tend to hallucinate'), not as the proof of any of this paper's measurements; the paper's own Algorithms 2 and 3 and Figures 3-6 stand independently. The undefined behavior of Algorithm 2 for hallucinated parameters present in only one of two compared files is a reproducibility issue, but it is a coding gap, not a logical reduction of the results to their inputs. Therefore no circular step is exhibited.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The evaluation introduces no fitted parameters and no invented entities. Its load-bearing assumptions are the authority of the chosen documentation, the correctness of the authors' manual benchmarks, and the adequacy of five samples per condition.

assumptions (3)
  • domain assumption The Arch Linux pwquality.conf man page [10] is the authoritative specification for valid parameters, defaults, and assignment rules.
    Hallucination and correctness scoring in Algorithms 2 and 3 use 'documented defaults' from this source. If the wrong documentation is used or the man page does not match the deployed PAM version, the scores would be wrong.
  • ad hoc to paper The manually created gold-standard benchmark files correctly translate each natural-language policy into pwquality.conf settings.
    Section III-A2 says the benchmark is 'manually defined.' Correctness scores are measured against it and no benchmark file or independent validation is provided.
  • ad hoc to paper Five generations per model and prompt are sufficient to estimate consistency and correctness.
    Algorithm 1 sets iterations to 5, and all reported figures are means over these generations. No statistical justification or power analysis is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How Good LLM-Generated Password Policies Are?." pith.science (2026). https://pith.science/paper/XGVI3XTZ

@misc{pith2026250608320,
  author       = {Pith},
  title        = {Pith review of: How Good LLM-Generated Password Policies Are?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XGVI3XTZ}},
  note         = {Machine review of arXiv:2506.08320}
}
abstract

Generative AI technologies, particularly Large Language Models (LLMs), are rapidly being adopted across industry, academia, and government sectors, owing to their remarkable capabilities in natural language processing. However, despite their strengths, the inconsistency and unpredictability of LLM outputs present substantial challenges, especially in security-critical domains such as access control. One critical issue that emerges prominently is the consistency of LLM-generated responses, which is paramount for ensuring secure and reliable operations. In this paper, we study the application of LLMs within the context of Cybersecurity Access Control Systems. Specifically, we investigate the consistency and accuracy of LLM-generated password policies, translating natural language prompts into executable pwquality$.$conf configuration files. Our experimental methodology adopts two distinct approaches: firstly, we utilize pre-trained LLMs to generate configuration files purely from natural language prompts without additional guidance. Secondly, we provide these models with official pwquality$.$conf documentation to serve as an informative baseline. We systematically assess the soundness, accuracy, and consistency of these AI-generated configurations. Our findings underscore significant challenges in the current generation of LLMs and contribute valuable insights into refining the deployment of LLMs in Access Control Systems.

Figures

Figures reproduced from arXiv: 2506.08320 by the authors.

Figure 1
Figure 1. PAM with configuration files in Linux SELinux [12] and AppArmor [39], define how users and processes interact with the system. This general framework not only regulates file and process permissions but also lays the groundwork for authenticating users—a process that directly influences how password management is conducted. Password Management is a critical component within this broader access control strategy. It fo… view at source ↗
Figure 2
Figure 2. Password Policy Generation and HICC Analysis [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Average LLM Consistency (Including Hallucinated [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Average LLM Consistency (Only Including Real Pa [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 6
Figure 6. Figure 6: Average Number of Hallucinated Parameters per LLM [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Average Number of Correctly Assigned Parameters per [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Average Number of Hallucinated Parameters per LLM, [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 28 canonical work pages

  1. [2]

    The impact of generative ai and llms on the cybersecurity profession,

    N. Capodieci, C. Sanchez-Adames, J. Harris, and U. Tatar, “The impact of generative ai and llms on the cybersecurity profession,” in 2024 Sys- tems and Information Engineering Design Symposium (SIEDS) , 2024, pp. 448–453

    work page 2024

  2. [1]

    Cohere api documentation,

    “Cohere api documentation,” https://cohere.ai/, 2023

    work page 2023

  3. [3]

    Cis password policy guide: Passphrases, monitoring, and more,

    Center for Internet Security (CIS), “Cis password policy guide: Passphrases, monitoring, and more,” 2023, ac- cessed: 2025-03-30. [Online]. Available: https://learn.cisecurity.org/ cis-password-policy-guide-passphrases-monitoring-and-more

    work page 2023

  4. [4]

    Deepseek-v3 technical report,

    A. L. et. al, “Deepseek-v3 technical report,” 2025. [Online]. Available: https://arxiv.org/abs/2412.19437

    arXiv 2025

  5. [5]

    BECEL: Benchmark for consistency evaluation of language models,

    M. Jang, D. S. Kwon, and T. Lukasiewicz, “BECEL: Benchmark for consistency evaluation of language models,” in Proceedings of the 29th International Conference on Computational Linguistics . Gyeongju, Republic of Korea: International Committee on Computational Linguistics, Oct. 2022, pp. 3680–3696. [Online]. Available: https: //aclanthology.org/2022.coling-1.324/

    work page 2022

  6. [6]

    Consistency analysis of chatgpt,

    M. E. Jang and T. Lukasiewicz, “Consistency analysis of chatgpt,”

    work page

  7. [7]

    Trutheval: A dataset to evaluate llm truthfulness and reliability,

    A. Khatun and D. G. Brown, “Trutheval: A dataset to evaluate llm truthfulness and reliability,” 2024. [Online]. Available: https: //arxiv.org/abs/2406.01855

    arXiv 2024

  8. [8]

    Evaluating the consistency of llm evaluators,

    N. Lee, J. Hong, and J. Thorne, “Evaluating the consistency of llm evaluators,” 2024. [Online]. Available: https://arxiv.org/abs/2412.00543

    arXiv 2024

Show all 43 references
  1. [9]

    Retrieval-augmented generation for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W. tau Yih, T. Rockt ¨aschel, S. Riedel, and D. Kiela, “Retrieval-augmented generation for knowledge-intensive nlp tasks,” 2021. [Online]. Available: https://arxiv.org/abs/2005.11401

    2021 arXiv

  2. [10]

    Linux, pwquality.conf(5) — Arch Manual Pages, n.d., accessed: 2025- 03-28

    A. Linux, pwquality.conf(5) — Arch Manual Pages, n.d., accessed: 2025- 03-28. [Online]. Available: https://man.archlinux.org/man/pwquality. conf.5.en

    2025

  3. [11]

    Default password policy in linux systems (e.g., pam, /etc/login.defs),

    Linux Manual, “Default password policy in linux systems (e.g., pam, /etc/login.defs),” 2024, based on standard Linux configurations; no official central documentation

    2024

  4. [12]

    F. M. Love, K. M. Anderson, and D. Smalley, SELinux by Example: Using Security Enhanced Linux. Upper Saddle River, NJ, USA: Prentice Hall PTR, 2006

    2006

  5. [13]

    Successful password policies for organizations,

    Lumi Cyber, “Successful password policies for organizations,” 2023, accessed: 2025-03-30. [Online]. Available: https://www.lumificyber. com/blog/successful-password-policies-for-organizations/

    2023

  6. [14]

    Reducing llm hallucination using knowledge distillation: A case study with mistral large and mmlu benchmark,

    D. McDonald, R. Papadopoulos, and L. Benningfield, “Reducing llm hallucination using knowledge distillation: A case study with mistral large and mmlu benchmark,” Authorea Preprints, 2024

    2024

  7. [15]

    Analyzing and mitigating (with llms) the security misconfigurations of helm charts from artifact hub,

    F. Minna, F. Massacci, and K. Tuma, “Analyzing and mitigating (with llms) the security misconfigurations of helm charts from artifact hub,” arXiv preprint arXiv:2403.09537 , 2024

    2024 arXiv

  8. [16]

    Large language models in cybersecurity: State-of-the-art,

    F. N. Motlagh, M. Hajizadeh, M. Majd, P. Najafi, F. Cheng, and C. Meinel, “Large language models in cybersecurity: State-of-the-art,”

  9. [17]

    Digital identity guidelines: Authentication and lifecycle management (sp 800-63b),

    NIST, “Digital identity guidelines: Authentication and lifecycle management (sp 800-63b),” 2020, accessed: 2025-03-30. [Online]. Available: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST. SP.800-63b.pdf

    2020

  10. [18]

    Chatgpt: Optimizing language models for dialogue,

    OpenAI, “Chatgpt: Optimizing language models for dialogue,” https: //openai.com/blog/chatgpt, 2022, accessed: 2025-03-30

    2022

  11. [19]

    GPT-4o Mini: Advancing Cost-Efficient Intelli- gence,

    OpenAI, “GPT-4o Mini: Advancing Cost-Efficient Intelli- gence,” Jul. 2024. [Online]. Available: https://openai.com/index/ gpt-4o-mini-advancing-cost-efficient-intelligence/

    2024

  12. [20]

    o3-mini in chatgpt - faq,

    OpenAI, “o3-mini in chatgpt - faq,” 2025, accessed March 31, 2025. [Online]. Available: https://help.openai.com/en/articles/ 10491870-o3-mini-in-chatgpt-faq

    2025

  13. [21]

    GPT-3.5: Generative Pre-trained Transformer,

    OpenAI Team, “GPT-3.5: Generative Pre-trained Transformer,” OpenAI API, 2023, accessed: Aug. 4, 2024. [Online]. Available: https: //platform.openai.com/docs/models/gpt-3-5

    2023

  14. [22]

    Automated consistency analysis of llms,

    A. Patwardhan, V . Vaidya, and A. Kundu, “Automated consistency analysis of llms,” in 2024 IEEE 6th International Conference on Trust, Privacy and Security in Intelligent Systems, and Applications (TPS-ISA) . IEEE, 2024, pp. 118–127

    2024

  15. [23]

    Cyber security policy templates: Password security,

    Purplesec, “Cyber security policy templates: Password security,” 2020, accessed: 2025-03-30. [Online]. Available: https://purplesec.us/ resources/cyber-security-policy-templates/password-security/

    2020

  16. [24]

    Empowering hardware security with llm: The development of a vul- nerable hardware database,

    D. Saha, K. Yahyaei, S. K. Saha, M. Tehranipoor, and F. Farahmandi, “Empowering hardware security with llm: The development of a vul- nerable hardware database,” in 2024 IEEE International Symposium on Hardware Oriented Security and Trust (HOST) . IEEE, 2024, pp. 233– 243

    2024

  17. [25]

    Top 10 password policy best practices,

    Securden, “Top 10 password policy best practices,” 2023, accessed: 2025-03-30. [Online]. Available: https://www.securden.com/ blog/top-10-password-policies.html

    2023

  18. [26]

    Poster: Seccomp profiling with dy- namic analysis via chatgpt-assisted test code generation,

    S. Song, A. Kundu, and B. Tak, “Poster: Seccomp profiling with dy- namic analysis via chatgpt-assisted test code generation,” in Proceedings of the 19th ACM Asia Conference on Computer and Communications Security, 2024, pp. 1928–1930

    2024

  19. [27]

    Gemini: A family of highly capable multimodal models,

    G. Team and R. Anil et. al., “Gemini: A family of highly capable multimodal models,” 2024. [Online]. Available: https://arxiv.org/abs/ 2312.11805

    2024 arXiv

  20. [28]

    Gpt-4: Generative pre-trained transformer,

    O. Team, “Gpt-4: Generative pre-trained transformer,” OpenAI API , 2023, https://platform.openai.com/docs/models/gpt-4

    2023

  21. [29]

    Cybermetric: A benchmark dataset based on retrieval-augmented gen- eration for evaluating llms in cybersecurity knowledge,

    N. Tihanyi, M. A. Ferrag, R. Jain, T. Bisztray, and M. Debbah, “Cybermetric: A benchmark dataset based on retrieval-augmented gen- eration for evaluating llms in cybersecurity knowledge,” in 2024 IEEE International Conference on Cyber Security and Resilience (CSR) , 2024, pp. 296–302

    2024

  22. [30]

    Llms in web-development: Evalu- ating llm-generated php code unveiling vulnerabilities and limitations,

    R. T ´oth, T. Bisztray, and L. Erdodi, “Llms in web-development: Evalu- ating llm-generated php code unveiling vulnerabilities and limitations,” arXiv preprint arXiv:2404.14459 , 2024

    2024 arXiv

  23. [31]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” 2023. [Online]. Available: https: //arxiv.org/abs/2...

    2023 arXiv

  24. [32]

    A critical look at ai-generate software: Coding with the new ai tools is both irresistible and dangerous,

    J. Vaidya and H. Asif, “A critical look at ai-generate software: Coding with the new ai tools is both irresistible and dangerous,” Ieee Spectrum, vol. 60, no. 7, pp. 34–39, 2023

    2023

  25. [33]

    Self-consistency improves chain of thought reasoning in language models,

    X. Wang, J. Wei, D. Schuurmans, Q. V . Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” in The Eleventh International Conference on Learning Representations , 2023. [Online]. Available: https://open...

    2023

  26. [34]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2201.11903

    2023 arXiv

  27. [35]

    Bloom: A 176b-parameter open-access multilingual language model,

    B. Workshop, :, and T. L. S. et. al., “Bloom: A 176b-parameter open-access multilingual language model,” 2023. [Online]. Available: https://arxiv.org/abs/2211.05100

    2023 arXiv

  28. [36]

    FLASK: Fine-grained language model evaluation based on alignment skill sets,

    S. Ye, D. Kim, S. Kim, H. Hwang, S. Kim, Y . Jo, J. Thorne, J. Kim, and M. Seo, “FLASK: Fine-grained language model evaluation based on alignment skill sets,” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https://openreview.ne...

    2024

  29. [37]

    Primus: A pioneering collection of open-source datasets for cybersecurity llm training,

    Y .-C. Yu, T.-H. Chiang, C.-W. Tsai, C.-M. Huang, and W.-K. Tsao, “Primus: A pioneering collection of open-source datasets for cybersecurity llm training,” 2025. [Online]. Available: https: //arxiv.org/abs/2502.11191

    2025

  30. [38]

    Opt: Open pre-trained transformer language models,

    S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V . Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer, “Opt: Open pre-trained transformer language models,”

  31. [39]

    Apparmor profile generator as a cloud service,

    H. Zhu and C. Gehrmann, “Apparmor profile generator as a cloud service,” in Proceedings of the 11th International Conference on Cloud Computing and Services Science . SciTePress, 2021, pp. 45–55

    2021

  32. [40]

    Multi-model consistency for llms’ evaluation,

    Q. Zhu, D. Lyu, X. Fan, X. Wang, Q. Tu, Y . Zhan, and H. Chen, “Multi-model consistency for llms’ evaluation,” in 2024 International Joint Conference on Neural Networks (IJCNN) , 2024, pp. 1–8. APPENDIX TABLE II: Password Policy Requirements Policy Requirements P1 There must b...

    2024

  33. [2022]

    Available: https://arxiv.org/abs/2205.01068

    [Online]. Available: https://arxiv.org/abs/2205.01068

    arXiv

  34. [2023]

    Available: https://arxiv.org/abs/2303.06273

    [Online]. Available: https://arxiv.org/abs/2303.06273

    arXiv

  35. [2024]

    Available: https://arxiv.org/abs/2402.00891

    [Online]. Available: https://arxiv.org/abs/2402.00891

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.