Pith. sign in

REVIEW 5 major objections 6 minor 39 references

Automated Consistency Analysis of LLMs

T0 review · 5 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLMs give inconsistent answers to the same security questions, making them unreliable in cybersecurity.

desk verdict The paper's real finding is that string-similarity metrics split into lexical and contentful families, but the leap from 'lexically varied' to 'untrustworthy for cybersecurity' is unsupported, and the validation algorithms are circular. read the letter →

arxiv 2502.07036 v2 pith:HG2EI6QH submitted 2025-02-10 cs.CR cs.AIcs.LG

Aditya Patwardhan , Vivek Vaidya , Ashish Kundu This is my paper · ORCID
classification cs.CRcs.AIcs.LG
keywords consistencylargelanguagemodelscybersecuritytrustworthinesshallucinationself-validationcross-validation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper defines what it means for a large language model to be consistent in its responses and builds an automated framework to measure that consistency. Applying the framework to five LLMs on 40 cybersecurity questions, the paper finds that no model stays consistent under its strictest thresholds, and that consistency drops markedly for open-ended situational questions. The paper also proposes self-validation and cross-validation to check factual agreement, finding that models often agree with themselves but sometimes fail to catch each other's hallucinations. The authors conclude that current LLMs are frequently inconsistent and thus not yet trustworthy for enterprise-level cybersecurity operations.

What carries the argument

The central mechanism is a formal definition of semantic consistency: responses to semantically equivalent prompts are consistent if they are semantically equivalent, operationalized by pairwise similarity scores from four metrics (Jaccard Index, Cosine Similarity, SequenceMatcher, and Levenshtein distance) standardized to a 0-100 scale, with low, medium, and high thresholds determining passage. Two algorithms carry the validation: self-validation (Algorithm 2) asks the same LLM whether its own response is correct, and cross-validation (Algorithm 3) asks every other LLM whether a given response is correct, with majority-agreement cutoffs. The load-bearing theoretical result is the implication that accuracy implies consistency, whose contrapositive lets the authors infer that observed inconsistency indicates a lack of accuracy.

What would settle it

Take any LLM that the framework rates 'inconsistent' at the high threshold on the 33 information questions, have domain experts check its individual answers against the known correct answers, and show that it answers all or nearly all of them correctly despite low similarity scores; that would break the paper's inference from inconsistency to unreliability.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLMs are often inconsistent in their responses to the same or semantically identical prompts, and this inconsistency makes them untrustworthy and unreliable for cybersecurity tasks. To support this, the paper provides a formal definition of consistency, proves that accuracy implies consistency (so inconsistency implies inaccuracy), and introduces a validation framework with two methods: self-validation, where a model judges its own repeated responses, and cross-validation, where other LLMs judge a model's response. Experiments on GPT-4o Mini, GPT-3.5, Gemini, Cohere, and Llama3 show that consistency scores are frequently below reliable thresholds, especially when all four similarity metrics must pass or when questions are situational; cross-validation also reveals cases where most LLMs accept a demonstrably hallucinated answer. The paper concludes that unless consistency improves, these models cannot be trusted in security operations.

Load-bearing premise

The paper assumes that agreement among an LLM's own responses or among several LLMs is a valid proxy for factual accuracy, so that observed inconsistency genuinely signals untrustworthiness rather than mere stylistic variation.

Editorial extensions

If this is right

  • Security teams should not rely on a single LLM for critical operations without first checking response consistency, since inconsistency signals possible inaccuracy.
  • An automated consistency check could serve as a gatekeeping tool to flag LLMs that are too unpredictable for security tasks, especially for situational questions.
  • Deterministic models like Gemini and Bloom pass the consistency test perfectly, but the paper shows this provides no accuracy information, so consistency alone cannot justify trust.
  • Newer models (GPT-4o Mini, Llama3) outperform older ones (OPT, Bloom) in consistency, yet even they fall below the high-threshold bar, meaning the field has progress to make before enterprise-level security use.
  • The formal implication that accuracy implies consistency gives a general principle: any model that fails a consistency check should not be considered fully accurate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The framework could be embedded as a runtime monitor in security operations centers, flagging responses that deviate from a model's past answers for human review.
  • The heavy reliance on order-sensitive metrics like SequenceMatcher and Levenshtein may conflate stylistic wording differences with factual inconsistency, so the high-threshold failures may overstate how untrustworthy a model is if it rephrases correct answers.
  • Self-validation and cross-validation measure inter-model agreement rather than ground truth; pairing them with expert-verified answer keys would make the trustworthiness conclusion far stronger.
  • The same consistency-checking approach could be transferred to other high-stakes domains such as medical diagnosis or legal advice, where response uniformity is equally critical.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes a formal definition of consistency for LLM responses and an automated framework to evaluate it, targeting cybersecurity applications. It defines consistency in terms of semantic equivalence of responses to the same or semantically identical prompts, states that accuracy implies consistency, and presents two validation approaches: self-validation (an LLM judges its own answers) and cross-validation (other LLMs judge a given LLM's answers). The authors report experiments on five LLMs (GPT-4o Mini, GPT-3.5, Gemini, Cohere, Llama3) using 40 cybersecurity questions (33 informational, 7 situational). They conclude that these LLMs are often inconsistent and therefore untrustworthy and unreliable for cybersecurity.

Significance. If the central claim were supported, the paper would be a useful cautionary study for cybersecurity practitioners adopting LLMs. The attempt to formalize consistency and to provide an automated evaluation framework is a reasonable research direction, and the inclusion of multiple LLMs and both informational and situational questions is a strength. However, the significance is severely limited by the gap between the formal definition (semantic equivalence) and the operational metric (string-similarity thresholds), and by the absence of any external ground truth in the validation algorithms. The paper itself contains statements that directly undermine its headline conclusion, and the empirical results are presented without statistical support. With substantial redesign of the evaluation and a much more cautious interpretation, a version of this work could be of interest, but as it stands the central claim is not established.

major comments (5)
  1. [Abstract and Section II-B, IV-B] The abstract's conclusion that LLMs are 'often inconsistent... and thus are untrustworthy and unreliable' does not follow from the consistency metric defined in Algorithm 1. Algorithm 1 measures lexical string similarity (Jaccard, Cosine, SequenceMatcher, Levenshtein), not the semantic equivalence r1 ≡ r2 that Definition 1 in Section II-C requires. The paper itself acknowledges in Section IV-B that Cohere 'is not factually inconsistent but still inconsistent in its responses,' and Section II-B states that consistency does not imply accuracy (C ≠> A). The leap from measured lexical variation to untrustworthiness is therefore unsupported by both the formal framework and the paper's own observations.
  2. [Section III-A, Algorithm 1] Algorithm 1 is internally inconsistent with the surrounding text and with the reported experiments. The text in Section III-A says 'If the pair passes x out of 4 consistency score metrics, it is considered to pass overall,' and the figures report results for '1 or more, 2 or more, 3 or more, or all 4' metrics. However, the pseudocode in Algorithm 1 (lines 15-20) increments qcnt only when all four counters (SS_cnt, LS_cnt, JS_cnt, CS_cnt) are at least npt, there is no parameter x in the input list, and the condition 'if SS cnt, LScnt, J Scnt, CS cnt ≥ npt' is not well-formed pseudocode. This makes the experimental procedure ambiguous and the reported pass rates unreproducible.
  3. [Section III-B1, Algorithm 2] The self-validation algorithm is circular as a measure of factual accuracy: Algorithm 2 asks the same LLM whether its own answer is correct, so an LLM with a strong 'yes' bias, or one that systematically flatters its own outputs, will trivially pass. Section IV-C1 shows that most models agree with themselves over 80% of the time; the authors do not show that this self-agreement correlates with any ground truth. Consequently, the reported self-validation scores do not support the claim that the framework 'determine[s] if the LLMs agree on whether a certain answer is correct' in any objective sense.
  4. [Section III-B2, Algorithm 3 and Section IV-C3] Cross-validation measures inter-LLM agreement, not correctness, because the judging LLMs are themselves the subject of the same failure modes. The paper's own hallucination example in Section IV-C3 demonstrates this: for the e-card situational question, only GPT-4o Mini recognized Meta OPT's answer as incorrect, while four other LLMs endorsed it despite having given the correct answer themselves. This is direct evidence that the cross-validation algorithm can be wrong, and the authors acknowledge the 'weakness in using LLM agreement to double-check responses.' Without an external correctness standard, the agreement rates cannot support the conclusion that inconsistent or poorly validated LLMs are unreliable for cybersecurity.
  5. [Section IV-B, Section IV-A] The quantitative claim that LLMs are 'often inconsistent' rests on ad hoc thresholds (70/80/90% for Jaccard/Cosine and 20/40/60% for SequenceMatcher/Levenshtein) and a small, unlabeled benchmark of 40 questions from a single blog source. No sensitivity analysis, confidence intervals, or statistical tests are reported, so the paper does not establish that the observed pass rates are robust to threshold choices or representative of cybersecurity queries more generally. The threshold parameters are free parameters of the framework, and different choices could change the qualitative conclusion.
minor comments (6)
  1. [Section IV-B] The text states that Gemini and Bloom are deterministic and therefore 'aren't included in the plots,' but Figures 3-8 appear to include Gemini bars and Table I reports Gemini's scores; the model inclusion rules should be clarified.
  2. [Section IV-B] There are several figure-reference errors: the text refers to 'the situation questions (Figure 5)' when the corresponding figure for high-threshold situational questions is Figure 8, and the captions of Figures 6-8 are inconsistent in phrasing.
  3. [Section III-B2] The cross-validation description refers to 'the Cross-Validation Algorithm (Algorithm 2),' but the algorithm is presented as Algorithm 3; this should be corrected.
  4. [References] References [37] and [38] are the same Meta OPT paper and are duplicated; one should be removed and the remaining citation renumbered.
  5. [Section II-C] The formal notation contains several typos and undefined symbols, including 'Rv ← − Qv(Li, Sj, Pk, tl, Uw)' with the arrow misformatted, the use of both pj and pk for prompts, the undefined term 'w'th user entity,' and the condition 'xv ≡ yv' in Definition 2 with variables that are not introduced.
  6. [Section IV-A] The benchmark is described as 40 questions from a popular interview-question list, but no ground-truth answers are provided or used in the evaluation; this limitation should be explicitly acknowledged in the benchmark section.

Circularity Check

2 steps flagged · score 6.0 of 10

The paper defines 'factual accuracy' as an LLM agreeing with itself or with peer LLMs, so the trustworthiness conclusion reduces to agreement by construction.

  1. self definitional [Section III-B.1, Self-Validation (Algorithm 2)]
    "That same LLM is then asked whether the generated responses are the correct answer to the original query. If it agrees with enough of its own responses, it is considered factually consistent by self-validation. ... If the LLM responds ”yes” 80% of the time, that question is considered correct by this metric."

    The validation output called 'factually consistent' is computed solely from the same model's yes/no answer about its own response. A sycophantic or overconfident model that always answers 'yes' passes by construction, with no external ground truth entering the evaluation. The measured quantity is therefore self-agreement relabeled as accuracy, and the paper's later trustworthiness conclusion inherits this definitional collapse.

  2. self definitional [Section III-B.2, Cross-Validation (Algorithm 3)]
    "then every other LLM is asked whether that LLM’s response to the original prompt is correct. If there is enough agreement between LLMs, that LLM is considered factually consistent by cross-validation."

    Again, 'factual' consistency is defined as inter-LLM agreement rather than correspondence to an external standard. Models that share a common false belief can all approve the same wrong answer. The paper's own CIA-triad hallucination case shows this: Cohere answered 'Yes, you explained the CIA triad...' despite the prompt containing an incorrect explanation. Hence the cross-validation score is agreement by construction, not factual accuracy.

full rationale

Two load-bearing reductions make the trustworthiness conclusion partially definitional. Algorithm 2 operationalizes 'factually consistent' as the tested LLM answering 'yes' to its own response; Algorithm 3 operationalizes it as approval by other LLMs. In both cases the reported accuracy/trustworthiness metric is by construction an agreement metric: no external ground truth is consulted, so a model or panel that always says 'yes' (or shares a common error) passes. The paper itself supplies a failure example in Section IV-C3, where Cohere approves an incorrect CIA-triad explanation, showing the self/cross-validation score can diverge from factual correctness. The formal implication Accuracy implies Consistency in Section II-B does not rescue this, because the operational consistency measure is string-level similarity, not the semantic equivalence defined in Definition 1; the paper even separates 'factually inconsistent' from 'inconsistent in its responses' for Cohere. No self-citation chain or imported uniqueness theorem is present, and the consistency thresholds are not fitted parameters renamed as predictions. The circularity is therefore localized to the validation/accuracy pipeline: the conclusion that LLMs are 'untrustworthy' for cybersecurity is substantially supported by metrics that reduce to self/peer agreement, warranting a partial-circularity score of 6 rather than a non-circular finding.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims rest on several hand-picked thresholds and on the unvalidated assumption that self/peer agreement measures accuracy. No external ground truth is used, and the paper does not provide any independent evidence that its consistency metrics correspond to trustworthiness.

free parameters (5)
  • similarity thresholds (low/medium/high) = Jaccard/Cosine: 0.7/0.8/0.9; Sequence/Levenshtein: 0.2/0.4/0.6
    Chosen by hand to 'bring out the differences between the models' (Section III-A), not justified by any independent criterion. These thresholds directly determine which models pass or fail the consistency check.
  • question pass threshold (qthreshold) = 0.8
    The minimum fraction of questions that must pass consistency or validation for the model to be deemed consistent or accurate. Arbitrary; used in Algorithms 1, 2, and 3.
  • self-validation agreement threshold = 0.8
    Fraction of self-answers that must be 'yes' for a question to pass in Algorithm 2. Arbitrary and central to the circular self-validation.
  • cross-validation LLM agreement fraction = 0.66
    Fraction of other LLMs that must agree with a response for a question to pass in Algorithm 3. Arbitrary.
  • number of repetitions k = unspecified
    The number of times each prompt is repeated is a required input to all algorithms, but the paper never states its value. This is a critical parameter for reproducibility.
assumptions (4)
  • domain assumption Semantic equivalence of responses is transitive and can be decided by string similarity thresholds
    Section II-C and III-A assume that 'sufficiently similar' strings under Jaccard/cosine/sequence/Levenshtein correspond to semantic equivalence, which is not established. The metrics are syntactic and cannot capture meaning.
  • domain assumption The LLM remains stable over the duration of the experiment (delta t)
    Section II-A assumes model parameters do not change during testing. This is reasonable for a short experiment but unstated for black-box APIs, which may be updated server-side.
  • ad hoc to paper Accuracy implies consistency (A => C)
    Section II-B asserts this implication without proof. It is not generally true because an accurate model can produce differently-worded but correct answers that may not be string-similar.
  • ad hoc to paper An LLM's 'yes' to its own answer indicates correctness
    Algorithm 2 relies on this for self-validation. This is the paper's main circularity: the model is both the source of the answer and the judge of its correctness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automated Consistency Analysis of LLMs." pith.science (2026). https://pith.science/paper/HG2EI6QH

@misc{pith2026250207036,
  author       = {Pith},
  title        = {Pith review of: Automated Consistency Analysis of LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HG2EI6QH}},
  note         = {Machine review of arXiv:2502.07036}
}
read the original abstract

Generative AI (Gen AI) with large language models (LLMs) are being widely adopted across the industry, academia and government. Cybersecurity is one of the key sectors where LLMs can be and/or are already being used. There are a number of problems that inhibit the adoption of trustworthy Gen AI and LLMs in cybersecurity and such other critical areas. One of the key challenge to the trustworthiness and reliability of LLMs is: how consistent an LLM is in its responses? In this paper, we have analyzed and developed a formal definition of consistency of responses of LLMs. We have formally defined what is consistency of responses and then develop a framework for consistency evaluation. The paper proposes two approaches to validate consistency: self-validation, and validation across multiple LLMs. We have carried out extensive experiments for several LLMs such as GPT4oMini, GPT3.5, Gemini, Cohere, and Llama3, on a security benchmark consisting of several cybersecurity questions: informational and situational. Our experiments corroborate the fact that even though these LLMs are being considered and/or already being used for several cybersecurity tasks today, they are often inconsistent in their responses, and thus are untrustworthy and unreliable for cybersecurity.

Figures

Figures reproduced from arXiv: 2502.07036 by the authors.

Figure 1
Figure 1. Self-Validation Architecure collects k model responses and then calculates pairwise con￾sistency scores using the four metrics for every possible pair of responses, including consecutive responses. If the metric score is higher than a certain threshold, that pair passes for that metric. Therefore, while consecutive comparisons are part of the pairwise evaluation, the algorithm ensures a comprehensive assessment by c… view at source ↗
Figure 2
Figure 2. Cross-Validation Architecture and then every other LLM is asked whether that LLM’s response to the original prompt is correct. If there is enough agreement between LLMs, that LLM is considered factually consistent by cross-validation If an LLM has unreliable information caused by biased training data, it may not be able to recognize that in the self-validation step. To remedy that we propose the Cross￾Validation Alg… view at source ↗
Figure 3
Figure 3. Consistency Analysis for Low threshold [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Consistency Analysis for Medium threshold [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Consistency Analysis for High threshold regular questions (Table I) and the situation questions (Table I) GPT 4o Mini consistently scores higher in Sequence Matcher and Levenshtein Distance, while 3.5 consistently scores higher on Jaccard Index and Cosine similarity. S…
Figure 6
Figure 6. Figure 6: Low threshold for Situational Questions [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: for Medium threshold for Situational Questions [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: High threshold for Situational Questions [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: Self Validation for Information Questions [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: Self Validation for Situational Questions [PITH_FULL_IMAGE:figures/full_fig_p007_10.png]
Figure 11
Figure 11. Figure 11: Cross Validation for Information Questions [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]
Figure 12
Figure 12. Figure 12: Cross Validation for Situational Questions [PITH_FULL_IMAGE:figures/full_fig_p008_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 21 canonical work pages

  1. [8]

    Consistency analysis of chatgpt,

    M. Jang and T. Lukasiewicz, “Consistency analysis of chatgpt,” arXiv preprint arXiv:2303.06273, 2023

    arXiv 2023

  2. [1]

    Chatgpt: Open possibilities,

    M. Aljanabi, M. Ghazi, A. H. Ali, S. A. Abed et al. , “Chatgpt: Open possibilities,” Iraqi Journal For Computer Science and Mathematics , vol. 4, no. 1, pp. 62–64, 2023

    work page 2023

  3. [2]

    Education in the era of generative artificial intelligence (ai): Understanding the potential benefits of chatgpt in promoting teaching and learning,

    D. Baidoo-Anu and L. Owusu Ansah, “Education in the era of generative artificial intelligence (ai): Understanding the potential benefits of chatgpt in promoting teaching and learning,” Available at SSRN 4337484, 2023

    work page 2023

  4. [3]

    Mixture of experts for network optimization: A large language model- enabled approach,

    H. Du, G. Liu, Y . Lin, D. Niyato, J. Kang, Z. Xiong, and D. I. Kim, “Mixture of experts for network optimization: A large language model- enabled approach,” 2024

    work page 2024

  5. [4]

    Considerations for evaluating large language models for cybersecurity tasks,

    J. Gennari, S.-h. Lau, S. Perl, J. Parish, and G. Sastry, “Considerations for evaluating large language models for cybersecurity tasks,” 2024. [Online]. Available: https://www.cmu.edu/news/stories/archives/2024/april/ sei-and-openai-recommend-ways-to-evaluate-large-language-models \ protect\discretionary{\char\hyphenchar\font} {}{}for-cybersecurity-applications

    work page 2024

  6. [5]

    Quan- tifying prediction consistency under model multiplicity in tabular llms,

    F. Hamman, P. Dissanayake, S. Mishra, F. Lecue, and S. Dutta, “Quan- tifying prediction consistency under model multiplicity in tabular llms,” arXiv preprint arXiv:2407.04173 , 2024

    arXiv 2024

  7. [6]

    Top 50 cyber security interview questions and answers in 2024,

    O. S. Hiremath, “Top 50 cyber security interview questions and answers in 2024,” Aug 2024. [Online]. Available: https://www.edureka.co/blog/ interview-questions/cybersecurity-interview-questions/#Part-B%20

    work page 2024

  8. [7]

    Etude de la distribution florale dans une portion des alpes et du jura,

    P. Jaccard, “Etude de la distribution florale dans une portion des alpes et du jura,” Bulletin de la Societe Vaudoise des Sciences Naturelles , vol. 37, pp. 547–579, 01 1901

    work page 1901

Show all 39 references
  1. [9]

    Towards mitigating LLM hallucination via self reflection,

    Z. Ji, T. Yu, Y . Xu, N. Lee, E. Ishii, and P. Fung, “Towards mitigating LLM hallucination via self reflection,” in Findings of the Association for Computational Linguistics: EMNLP 2023 , Dec. 2023, pp. 1827–1843. [Online]. Available: https://aclanthology.org/2023.findings-emnlp.123

    2023

  2. [10]

    Is chatgpt a good translator? a preliminary study,

    W. Jiao, W. Wang, J.-t. Huang, X. Wang, and Z. Tu, “Is chatgpt a good translator? a preliminary study,” arXiv preprint arXiv:2301.08745, 2023

    2023 arXiv

  3. [11]

    How secure is code generated by chatgpt?

    R. Khoury, A. R. Avila, J. Brunelle, and B. M. Camara, “How secure is code generated by chatgpt?” 2023

    2023

  4. [12]

    Validating large language models with relm,

    M. Kuchnik, V . Smith, and G. Amvrosiadis, “Validating large language models with relm,” 2023

    2023

  5. [13]

    Binary codes capable of correcting deletions, in- sertions, and reversals,

    V . I. Levenshtein, “Binary codes capable of correcting deletions, in- sertions, and reversals,” Soviet Physics Doklady , vol. 10, pp. 707–710, 1966

    1966

  6. [14]

    Ex- ploring and evaluating hallucinations in llm-powered code generation,

    F. Liu, Y . Liu, L. Shi, H. Huang, R. Wang, Z. Yang, and L. Zhang, “Ex- ploring and evaluating hallucinations in llm-powered code generation,” 2024

    2024

  7. [15]

    Summary of chatgpt/gpt-4 research and per- spective towards the future of large language models,

    Y . Liu, T. Han, S. Ma, J. Zhang, Y . Yang, J. Tian, H. He, A. Li, M. He, Z. Liu et al. , “Summary of chatgpt/gpt-4 research and per- spective towards the future of large language models,” arXiv preprint arXiv:2304.01852, 2023

    2023 arXiv

  8. [16]

    Chatting about chatgpt: how may ai and gpt impact academia and libraries?

    B. D. Lund and T. Wang, “Chatting about chatgpt: how may ai and gpt impact academia and libraries?” Library Hi Tech News , vol. 40, no. 3, pp. 26–29, 2023

    2023

  9. [17]

    Reducing llm hallucination using knowledge distillation: A case study with mistral large and mmlu benchmark,

    D. McDonald, R. Papadopoulos, and L. Benningfield, “Reducing llm hallucination using knowledge distillation: A case study with mistral large and mmlu benchmark,” Authorea Preprints, 2024

    2024

  10. [18]

    Analyzing and mitigating (with llms) the security misconfigurations of helm charts from artifact hub,

    F. Minna, F. Massacci, and K. Tuma, “Analyzing and mitigating (with llms) the security misconfigurations of helm charts from artifact hub,” arXiv preprint arXiv:2403.09537 , 2024

    2024 arXiv

  11. [19]

    Gpt-3.5: Generative pre-trained transformer,

    OpenAI, “Gpt-3.5: Generative pre-trained transformer,” OpenAI API , 2023, https://platform.openai.com/docs/models/gpt-3-5

    2023

  12. [20]

    Check your facts and try again: Improving large language models with external knowledge and automated feedback,

    B. Peng, M. Galley, P. He, H. Cheng, Y . Xie, Y . Hu, Q. Huang, L. Liden, Z. Yu, W. Chen, and J. Gao, “Check your facts and try again: Improving large language models with external knowledge and automated feedback,” 2023

    2023

  13. [21]

    Python 3 Documentation,

    Python Software Foundation, “Python 3 Documentation,” https://docs. python.org/3/library/difflib.html, accessed: Insert Date Here

  14. [22]

    All your llms belong to us: Experiments with a new extortion phishing dataset,

    F. Z. Qachfar and R. M. Verma, “All your llms belong to us: Experiments with a new extortion phishing dataset,” in IFIP Annual Conference on Data and Applications Security and Privacy . Springer, 2024, pp. 241– 258

    2024

  15. [23]

    Chatgpt: Bullshit spewer or the end of traditional assessments in higher education?

    J. Rudolph, S. Tan, and S. Tan, “Chatgpt: Bullshit spewer or the end of traditional assessments in higher education?” Journal of Applied Learning and Teaching, vol. 6, no. 1, 2023

    2023

  16. [24]

    Llm for soc security: A paradigm shift,

    D. Saha, S. Tarek, K. Yahyaei, S. K. Saha, J. Zhou, M. Tehranipoor, and F. Farahmandi, “Llm for soc security: A paradigm shift,” IEEE Access, 2024

    2024

  17. [25]

    Empowering hardware security with llm: The development of a vul- nerable hardware database,

    D. Saha, K. Yahyaei, S. K. Saha, M. Tehranipoor, and F. Farahmandi, “Empowering hardware security with llm: The development of a vul- nerable hardware database,” in 2024 IEEE International Symposium on Hardware Oriented Security and Trust (HOST) . IEEE, 2024, pp. 233– 243

    2024

  18. [26]

    Introduction to modern information retrieval,

    G. Salton and M. J. McGill, “Introduction to modern information retrieval,” Computer Science Press , vol. 1, no. 2, pp. 165–192, 1986

    1986

  19. [27]

    Poster: Seccomp profiling with dy- namic analysis via chatgpt-assisted test code generation,

    S. Song, A. Kundu, and B. Tak, “Poster: Seccomp profiling with dy- namic analysis via chatgpt-assisted test code generation,” in Proceedings of the 19th ACM Asia Conference on Computer and Communications Security, 2024, pp. 1928–1930

    2024

  20. [28]

    Multi-agent collaboration: Harnessing the power of intelligent llm agents,

    Y . Talebirad and A. Nadiri, “Multi-agent collaboration: Harnessing the power of intelligent llm agents,” arXiv preprint arXiv:2306.03314, 2023

    2023 arXiv

  21. [29]

    Gemini: A family of highly capable multimodal models,

    G. Team and R. Anil et. al., “Gemini: A family of highly capable multimodal models,” 2024. [Online]. Available: https://arxiv.org/abs/ 2312.11805

    2024 arXiv

  22. [30]

    Gpt-4: Generative pre-trained transformer,

    O. Team, “Gpt-4: Generative pre-trained transformer,” OpenAI API , 2023, https://platform.openai.com/docs/models/gpt-4

    2023

  23. [31]

    Llms in web-development: Evalu- ating llm-generated php code unveiling vulnerabilities and limitations,

    R. T ´oth, T. Bisztray, and L. Erdodi, “Llms in web-development: Evalu- ating llm-generated php code unveiling vulnerabilities and limitations,” arXiv preprint arXiv:2404.14459 , 2024

    2024 arXiv

  24. [32]

    A critical look at ai-generate software: Coding with the new ai tools is both irresistible and dangerous,

    J. Vaidya and H. Asif, “A critical look at ai-generate software: Coding with the new ai tools is both irresistible and dangerous,” Ieee Spectrum, vol. 60, no. 7, pp. 34–39, 2023

    2023

  25. [33]

    Bloom: A 176b-parameter open-access multilingual language model,

    B. Workshop, :, and T. L. S. et. al., “Bloom: A 176b-parameter open-access multilingual language model,” 2023. [Online]. Available: https://arxiv.org/abs/2211.05100

    2023 arXiv

  26. [34]

    A new era in llm security: Exploring security concerns in real-world llm-based systems,

    F. Wu, N. Zhang, S. Jha, P. McDaniel, and C. Xiao, “A new era in llm security: Exploring security concerns in real-world llm-based systems,” arXiv preprint arXiv:2402.18649 , 2024

    2024 arXiv

  27. [35]

    Hallucination is inevitable: An innate limitation of large language models,

    Z. Xu, S. Jain, and M. Kankanhalli, “Hallucination is inevitable: An innate limitation of large language models,” arXiv preprint arXiv:2401.11817, 2024

    2024 arXiv

  28. [36]

    Distributed training of large language models,

    F. Zeng, W. Gan, Y . Wang, and P. S. Yu, “Distributed training of large language models,” in 2023 IEEE 29th International Conference on Parallel and Distributed Systems (ICPADS) , 2023, pp. 840–847

    2023

  29. [37]

    Opt: Open pre-trained transformer language models,

    S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V . Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer, “Opt: Open pre-trained transformer language models,”

  30. [38]

    Opt: Open pre-trained transformer language models,

    S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. Victoria Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. Singh Koura, A. Sridhar, T. Wang, and L. Zettlemoyer, “Opt: Open pre-trained transformer language models,” 2022. [...

    2022 arXiv

  31. [39]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xing et al. , “Judging llm-as-a-judge with mt-bench and chatbot arena,” Advances in Neural Information Processing Systems, vol. 36, 2024. APPENDIX TABLE IV: List of Cybersecurity Informat...

    2024

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.