Pith. sign in

REVIEW 4 major objections 5 minor 46 references

Does Johnny Get the Message? Evaluating Cybersecurity Notifications for Everyday Users

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces HCSAEF, a seven-dimension rubric for evaluating LLM-written cybersecurity notifications, and shows it can differentiate outputs by prompt, model, and run-to-run consistency.

desk verdict A useful evaluation framework for LLM-written security notifications, but the core claim of 'effectively differentiates' rests entirely on unaudited manual scoring; worth sending to review, not accepting as-is. read the letter →

arxiv 2505.22435 v1 pith:IHHCBSAT submitted 2025-05-28 cs.CR

Victor Jüttner , Erik Buchmann This is my paper
classification cs.CR
keywords Human-CenteredSecurityAlertEvaluationFrameworkHCSAEFcybersecuritynotificationslargelanguagemodelsusablepromptengineeringsmarthome
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the quality of LLM-written cybersecurity notifications for non-experts can be assessed systematically, not just by taste. It introduces HCSAEF, a rubric that scores a notification on seven dimensions—Consequences, Context, Countermeasures, Correctness, Intuitiveness, Personalization, and Urgency—each on a five-point scale. Through three case studies, the paper argues the framework separates good from poor outputs: it distinguishes two prompt designs, surfaces real differences among four LLMs (Grok 3, GPT-4o, GPT o1, DeepSeek R1), and exposes run-to-run inconsistency in GPT-4o's responses. If the framework holds up, researchers gain a structured way to compare, tune, and select models and prompts for generating security alerts that everyday users can act on.

What carries the argument

The load-bearing object is HCSAEF itself: a seven-dimension rating rubric with a 0–4 Likert scale and a written anchor for every level of every dimension. The dimensions are adapted from general LLM-evaluation criteria (Context, Correctness, and Intuitiveness from [7] and [8]) and from security-communication research (Consequences and Countermeasures from Protection Motivation Theory, Personalization from warning-relevance studies, and Urgency from work on fear appeals). The rubric works by forcing each assessment to separate seven distinct qualities, so a notification cannot be dismissed or praised as simply good or bad; the differentiated scores are what allow the prompt, model, and robustness comparisons in the case study.

What would settle it

Run each model on the same prompt, say 30 times, score every output with HCSAEF, and compare the spread within each model to the spread between models; if within-model variance rivals between-model differences, the claim that HCSAEF cleanly differentiates models fails. Likewise, have two independent raters score the same set of notifications; low inter-rater agreement would show the rubric needs sharper anchors.

Watch

Extended reading notes

Core claim

The central claim is that HCSAEF delivers a differentiated, reproducible picture of LLM-generated security notifications where generic quality judgments would not. Each of the seven dimensions has an explicit 0–4 anchor description, so a notification can be strong on urgency and countermeasures while weak on personalization, as happened with the Prompt 2 output. In the case study, HCSAEF shows Prompt 2 with impersonation and chain-of-thought beats a simpler prompt on most dimensions, but adding device details raises Context and Personalization at the cost of coarser countermeasures. Across models, Grok 3 scored outstanding on essentially all dimensions, GPT o1 gave correct but generic advice, and DeepSeek R1 produced countermeasures that the framework's own definitions mark as unsafe, such as resetting a camera password to a weak dictionary string. Repeating the same prompt three times, HCSAEF captured that Grok 3 stayed stable while GPT-4o varied noticeably, proving the framework can track output robustness as well as quality.

Load-bearing premise

The model comparisons rest on one generated notification per model, while the paper itself shows the same prompt can yield very different outputs across runs; if that run-to-run variation is large, the reported differences between models may not be stable.

Editorial extensions

If this is right

  • Researchers can use HCSAEF to compare prompting strategies and see trade-offs, e.g., richer context raises Context and Personalization scores while simpler prompts produce less precise countermeasures.
  • Model choice measurably changes notification quality: in the case study Grok 3 scored highest on personalization and clear urgency, while DeepSeek R1 produced unsafe countermeasure advice.
  • Running the same prompt repeatedly exposes output robustness: GPT-4o varied substantially across runs while Grok 3 was more consistent, so evaluations should account for sampling variability.
  • HCSAEF's seven dimensions give a shared vocabulary for improving LLM-generated alerts before deployment to non-expert users.
  • The framework is currently manual, and the authors state the next step is automating it with an LLM-as-a-Judge approach for larger experiments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: If HCSAEF scores were validated against actual user comprehension and follow-through in a lab study, the seven dimensions could serve as a predictive proxy for whether a notification changes behavior, not just whether it reads well.
  • Editorial inference: The paper's own robustness data implies that single-output model comparisons are underpowered; a fair benchmark would average HCSAEF scores over many repeated generations per prompt.
  • Editorial inference: The same rubric could extend beyond smart-home IDS alerts to phishing warnings, privacy breach notices, or medical device alerts, since the dimensions are domain-general enough to transfer.
  • Editorial inference: A measurable check on the framework itself is inter-rater agreement: if two trained raters disagree widely on the same notification, the scales would need tighter anchor examples.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes HCSAEF, a seven-dimension framework (Consequences, Context, Countermeasures, Correctness, Intuitiveness, Personalization, Urgency) for evaluating LLM-generated cybersecurity notifications aimed at everyday users. Each dimension is defined on a 0-4 Likert scale with concrete anchor examples. The authors demonstrate the framework in three case studies: comparing two prompting strategies with GPT-4o, comparing four LLMs (DeepSeek R1, Grok 3, GPT-4o, GPT-o1) on a single prompt, and probing output robustness via three repeated runs with Grok 3 and GPT-4o. The paper reports that HCSAEF differentiates notifications across prompts, models, and repeated runs, and concludes with plans to automate scoring using an LLM-as-a-judge approach and to validate against human user assessments.

Significance. If its reliability and validity can be established, HCSAEF would fill a concrete gap: there is currently no widely accepted, human-centered rubric for evaluating LLM-generated security notifications. The framework's dimensions are grounded in existing LLM evaluation work and in security communication theory, and the scale anchors are unusually concrete, which is a genuine strength. The case studies, though small, do illustrate that the rubric can produce nontrivial distinctions—e.g., DeepSeek R1's unsafe password advice versus Grok 3's more personalized output. The authors are also transparent about the current manual application and about their intention to validate and automate the framework. However, at present the central claim of effective differentiation rests entirely on unaudited manual ratings by the authors, with no inter-rater reliability, no blind scoring, and no comparison against user or expert judgments. That is the main gap between the claim and the evidence.

major comments (4)
  1. [Section IV-B and IV-C] The model comparison in Section IV-B assigns each LLM a single HCSAEF rating profile (Figure 3), yet Section IV-C shows that GPT-4o's responses vary substantially across three repeated runs of the same prompt with the same settings. With only one run per model, the reported differences—such as Grok 3 outperforming the others—cannot be distinguished from run-to-run variability. Please either provide multiple runs per model with distributions or significance tests, or explicitly frame the model comparison as an illustrative single-example demonstration.
  2. [Section IV (Tables IX, Figures 2-4)] All HCSAEF ratings in the case studies are assigned manually by the authors, with no inter-rater reliability, no blind rating procedure, and no independent audit of the scoring rationales. Every conclusion about prompts, models, and robustness depends on these unaudited ratings, so the observed differentiation could reflect rater expectations rather than stable properties of the notifications. Please report inter-rater reliability (e.g., weighted Cohen's kappa) from at least two independent raters who score the outputs blindly and in randomized order.
  3. [Section V, final paragraph] The paper explicitly defers validation of HCSAEF against human user assessment ('we plan to run comparative experiments to determine whether HCSAEF's evaluation is similar to the assessment of a human user'). This is an admitted missing support for the abstract's claim that HCSAEF 'effectively differentiates' generated notifications. Without at least a preliminary validity check—e.g., comparing HCSAEF scores to expert or end-user judgments on a sample of notifications—the framework's construct validity remains open. Please add such a check or clearly restrict the claim to 'differentiates according to the authors' manual application of the rubric.'
  4. [Section IV-C] The robustness analysis is based on only three repeated runs per model, and the speculation that GPT-4o's variability is due to 'different times of the day' and 'lower system load' is not supported by any load or timing data. Please report exact run conditions, increase the number of repetitions, and avoid causal claims about system load unless directly measured.
minor comments (5)
  1. [Figures 2-4] The figures contain 'Tabelle1 Seite 1' artifacts, apparently from the source document, which should be removed before publication.
  2. [Section IV-B] The model name 'Grok3' is inconsistent with 'Grok 3' used elsewhere in the text; please unify the spelling.
  3. [Figure 4] The caption for Figure 4 does not explain what the '#1', '#2', and '#3' series denote; please add a short explanation that these are the three repeated runs of Prompt 1.
  4. [Section II-C] The sentence 'LLMs increasingly influence many aspects of cybersecurity' has a missing citation placeholder '[?]'; please supply a reference.
  5. [Section IV-A, Table IX] Table IX evaluates the notification in Figure 1, which was generated with Prompt 2, yet the rationale for Consequences (rated 3) says the consequences are 'specific and detailed to the extent of the information provided in the prompt.' Since Prompt 2 does not provide device details, please clarify how a 3 (specific consequences for the home network) is justified relative to the scale anchor, which requires mentioning specific consequences and affected devices.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: HCSAEF's dimensions come from external sources, and the case studies are demonstrations rather than fitted predictions.

full rationale

HCSAEF's seven dimensions are adopted from external general LLM evaluation frameworks and from security-communication research (Section III, citing [7]-[16]), not from the paper's own prior results. The three use cases are worked demonstrations in which the authors apply the pre-defined rubric to a small number of LLM outputs; no parameter is fitted to a subset of data and then renamed a prediction, and no equation or construction reduces a derived quantity to an input. The only self-citation (ChatIDS, [5], by author Jüttner et al.) appears in related work as background on LLM-based alert rewriting and is not load-bearing for HCSAEF's definitions or for the use-case conclusions. The paper explicitly admits in Section V that 'For the time being, we have evaluated HCSAEF's dimensions manually' and that it 'plan[s] to run comparative experiments to determine whether HCSAEF's evaluation is similar to the assessment of a human user.' That is an acknowledged validation gap for the framework's empirical claims, but it is a correctness/support concern, not a circularity: the case-study ratings are not derived from the conclusions, and the framework's content does not depend on the outputs it evaluates. Accordingly, no circular step can be quoted, and the score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 1 invented entities

No free parameters are fitted; the framework is constructed from prior literature. The assumptions are that the chosen dimensions and Likert anchors capture notification quality, that manual expert scoring is reliable, and that single LLM outputs are representative for model comparison. HCSAEF itself is a new construct without external validation.

assumptions (3)
  • domain assumption The seven dimensions (Consequences, Context, Countermeasures, Correctness, Intuitiveness, Personalization, Urgency) and their Likert anchors are a valid operationalization of helpfulness for non-expert users.
    Dimensions are drawn from [7]-[16] but the paper does not validate them against user responses or outcomes; the future work section explicitly plans a human comparison study to build ground truth.
  • domain assumption Manual ratings by the authors are consistent and correspond to perceived notification quality.
    All case study ratings are performed manually; no inter-rater reliability or agreement metrics are reported (Sections III and IV).
  • domain assumption A single generated notification per model is representative enough to compare models.
    Section IV-B uses one run per model, while Section IV-C shows GPT-4o's outputs vary noticeably across three runs, so this assumption is load-bearing and currently unverified.
invented entities (1)
  • HCSAEF (seven-dimension evaluation framework)
    purpose: A manual scoring instrument for LLM-generated cybersecurity notifications, used to compare prompts, models, and response robustness.
    The framework is a new construct proposed in this paper. It makes no falsifiable predictions and has not been benchmarked against user behavior or other validated instruments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Does Johnny Get the Message? Evaluating Cybersecurity Notifications for Everyday Users." pith.science (2026). https://pith.science/paper/IHHCBSAT

@misc{pith2026250522435,
  author       = {Pith},
  title        = {Pith review of: Does Johnny Get the Message? Evaluating Cybersecurity Notifications for Everyday Users},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IHHCBSAT}},
  note         = {Machine review of arXiv:2505.22435}
}
read the original abstract

Due to the increasing presence of networked devices in everyday life, not only cybersecurity specialists but also end users benefit from security applications such as firewalls, vulnerability scanners, and intrusion detection systems. Recent approaches use large language models (LLMs) to rewrite brief, technical security alerts into intuitive language and suggest actionable measures, helping everyday users understand and respond appropriately to security risks. However, it remains an open question how well such alerts are explained to users. LLM outputs can also be hallucinated, inconsistent, or misleading. In this work, we introduce the Human-Centered Security Alert Evaluation Framework (HCSAEF). HCSAEF assesses LLM-generated cybersecurity notifications to support researchers who want to compare notifications generated for everyday users, improve them, or analyze the capabilities of different LLMs in explaining cybersecurity issues. We demonstrate HCSAEF through three use cases, which allow us to quantify the impact of prompt design, model selection, and output consistency. Our findings indicate that HCSAEF effectively differentiates generated notifications along dimensions such as intuitiveness, urgency, and correctness.

Figures

Figures reproduced from arXiv: 2505.22435 by the authors.

Figure 1
Figure 1. Example of a cybersecurity alert rewritten by GPT-4o into a detailed, user-friendly notification tailored for non-expert homeowners. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparing Prompt 1 and Prompt 2 with HCSAEF. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Comparing different LLMs with HCSAEF. device in the network performs a factory reset by pressing the reset button for 10 seconds. We already discussed the performance of GPT 4o in the last subsection. GPT o1 performed slightly better. Its extended reasoning provided a more elaborate list of consequences of ignoring the alert. It also did not need technical terms to explain the cybersecurity threat and related counte… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 42 canonical work pages

  1. [1]

    A history and survey of network firewalls,

    K. Ingham and S. Forrest, “A history and survey of network firewalls,” University of New Mexico, Tech. Rep, 2002

    work page 2002

  2. [2]

    Evolution of firewalls: Toward securer network using next generation firewall,

    J. Liang and Y . Kim, “Evolution of firewalls: Toward securer network using next generation firewall,” inIEEE 12th Annual Computing and Communication Workshop and Conference, 2022, pp. 752–759

    work page 2022

  3. [3]

    A review of network vulnerabilities scanning tools: types, capabilities and functioning,

    A. Tundis, W. Mazurczyk, and M. M ¨uhlh¨auser, “A review of network vulnerabilities scanning tools: types, capabilities and functioning,” in Proceedings of the 13th International Conference on Availability, Re- liability and Security, ser. ARES ’18. Association for Computing Machinery, 2018

    work page 2018

  4. [4]

    A survey of intrusion detection and prevention systems,

    A. Patel, Q. Qassim, and C. Wills, “A survey of intrusion detection and prevention systems,”Information Management & Computer Security, vol. 18, no. 4, pp. 277–290, 2010

    work page 2010

  5. [5]

    ChatIDS: Advancing explainable cybersecurity using generative ai,

    V . J ¨uttner, M. Grimmer, and E. Buchmann, “ChatIDS: Advancing explainable cybersecurity using generative ai,”International Journal On Advances in Security, vol. 17, no. 1,2, 2024

    work page 2024

  6. [6]

    Chatsec: Spicing up vulnerability scans with ai for heterogeneous university it - towards enhancing security vulnerability reports for non-experts,

    M. Hoffmann and E. Buchmann, “Chatsec: Spicing up vulnerability scans with ai for heterogeneous university it - towards enhancing security vulnerability reports for non-experts,” inProceedings of the Conference on AI-based Systems and Services (AISyS’24), 2024

    work page 2024

  7. [7]

    A survey on evaluation of large language models,

    Y . Chang, X. Wang, J. Wang, Y . Wuet al., “A survey on evaluation of large language models,”ACM Trans. Intell. Syst. Technol., vol. 15, no. 3, 2024

    work page 2024

  8. [8]

    Unsupervised evaluation of interactive dialog with DialoGPT,

    S. Mehri and M. Eskenazi, “Unsupervised evaluation of interactive dialog with DialoGPT,” inProceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, O. Pietquin, S. Muresanet al., Eds. 1st virtual meeting: Association for Computa- tional Linguistics, Jul. 2020, pp. 225–235

    work page 2020

Show all 46 references
  1. [9]

    Cognitive and physiological processes in fear appeals and attitude change: A revised theory of protection motivation,

    R. W. Rogers, “Cognitive and physiological processes in fear appeals and attitude change: A revised theory of protection motivation,”Social psychology: A source book, pp. 153–176, 1983

    1983

  2. [10]

    Bridging the gap in computer security warnings: A mental model approach,

    C. Bravo-Lillo, L. F. Cranor, J. Downs, and S. Komanduri, “Bridging the gap in computer security warnings: A mental model approach,”IEEE Security & Privacy, vol. 9, pp. 18–26, 2011

    2011

  3. [11]

    Communication-human information processing (c-hip) model,

    M. S. Wogalter, “Communication-human information processing (c-hip) model,” inForensic human factors and ergonomics. CRC Press, 2018

    2018

  4. [12]

    Con- textualized web warnings, and how they cause distrust,

    S. Bartsch, M. V olkamer, H. Theuerling, and F. Karayumak, “Con- textualized web warnings, and how they cause distrust,” inTrust and Trustworthy Computing: 6th International Conference. Springer, 2013, pp. 205–222

    2013

  5. [13]

    It is not about the design - it is about the content! making warnings more efficient by communicating risks appropriately,

    M. Kauer, T. Pfeiffer, M. V olkamer, H. Theuerlinget al., “It is not about the design - it is about the content! making warnings more efficient by communicating risks appropriately,” inSICHERHEIT 2012 – Sicherheit, Schutz und Zuverl ¨assigkeit, 2012

    2012

  6. [14]

    What do users feel? towards affective eeg correlates of cybersecurity notifications,

    C. Conrad, J. Aziz, N. Smith, and A. Newman, “What do users feel? towards affective eeg correlates of cybersecurity notifications,” in Information Systems and Neuroscience, F. D. Davis, R. Riedlet al., Eds. Cham: Springer International Publishing, 2020, pp. 153–162

    2020

  7. [15]

    Beyond fear and frustration - towards a holistic understanding of emotions in cybersecurity,

    A. V on Preuschen, M. C. Schuhmacher, and V . Zimmermann, “Beyond fear and frustration - towards a holistic understanding of emotions in cybersecurity,” inProceedings of the Twentieth USENIX Conference on Usable Privacy and Security. USENIX Association, 2024

    2024

  8. [16]

    Scaring and bullying people into security won’t work,

    A. Sasse, “Scaring and bullying people into security won’t work,”IEEE Security & Privacy, vol. 13, no. 3, pp. 80–83, 2015

    2015

  9. [17]

    Judging llm- as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuanget al., “Judging llm- as-a-judge with mt-bench and chatbot arena,” inAdvances in Neural Information Processing Systems, A. Oh, T. Naumannet al., Eds., vol. 36. Curran Associates, Inc., 2023, pp. 46 595–46 623

    2023

  10. [18]

    Identifying vulnerabilities in security and privacy of smart home devices,

    C. Chhetri and V . Motti, “Identifying vulnerabilities in security and privacy of smart home devices,” inNational Cyber Summit (NCS) Research Track 2020, K.-K. R. Choo, T. Morriset al., Eds. Cham: Springer International Publishing, 2021, pp. 211–231

    2020

  11. [19]

    Smart home security: challenges, issues and solutions at different iot layers,

    H. Touqeer, S. Zaman, R. Amin, M. Hussainet al., “Smart home security: challenges, issues and solutions at different iot layers,”J. Supercomput., vol. 77, no. 12, p. 14053–14089, dec 2021

    2021

  12. [20]

    A survey of user perspectives on security and privacy in a home networking environment,

    N. Pattnaik, S. Li, and J. R. C. Nurse, “A survey of user perspectives on security and privacy in a home networking environment,”ACM Computing Surveys, vol. 55, pp. 1 – 38, 2022

    2022

  13. [21]

    A supervised intrusion detection system for smart home iot devices,

    E. Anthi, L. Williams, M. Słowi ´nska, G. Theodorakopoulos, and P. Bur- nap, “A supervised intrusion detection system for smart home iot devices,”IEEE Internet of Things Journal, vol. 6, no. 5, pp. 9042–9053, 2019

    2019

  14. [22]

    Aegis+: A context-aware platform-independent security framework for smart home systems,

    A. K. Sikder, L. Babun, and A. S. Uluagac, “Aegis+: A context-aware platform-independent security framework for smart home systems,” Digital Threats, vol. 2, no. 1, 2021

    2021

  15. [23]

    Can i sleep safely in my smarthome? a novel framework on automating dynamic risk assessment in iot environments,

    A. Collen and N. A. Nijdam, “Can i sleep safely in my smarthome? a novel framework on automating dynamic risk assessment in iot environments,”Electronics, vol. 11, no. 7, 2022

    2022

  16. [24]

    Piti: Protecting internet of things via intrusion detection system on raspberry pi,

    V . Visoottiviseth, G. Chutaporn, S. Kungvanruttana, and J. Paisarnduang- jan, “Piti: Protecting internet of things via intrusion detection system on raspberry pi,” in2020 International Conference on Information and Communication Technology Convergence (ICTC), 2020, pp. 75–80

    2020

  17. [25]

    Purposes and scope of warnings,

    M. Wogalter, “Purposes and scope of warnings,”Handbook of Warnings, pp. 3–9, 01 2006

    2006

  18. [26]

    How do non experts think about cyber attack consequences?

    K. S. Jones, N. R. Lodinger, B. P. Widlus, A. Siami Namin, E. Maw, and M. E. Armstrong, “How do non experts think about cyber attack consequences?”Information & Computer Security, vol. 30, no. 4, pp. 473–489, 2022

    2022

  19. [27]

    Scaring people is not enough: An examination of fear appeals within the context of promoting good password hygiene,

    M. Dupuis, A. Jennings, and K. Renaud, “Scaring people is not enough: An examination of fear appeals within the context of promoting good password hygiene,” inProceedings of the 22nd Annual Conference on Information Technology Education. Association for Computing Machinery, 20...

    2021

  20. [28]

    Warning design guidelines,

    L. Bauer, C. Bravo-Lillo, L. Cranor, and E. Fragkaki, “Warning design guidelines,” CyLab, Carnegie Mellon University, Tech. Rep., 2013

    2013

  21. [29]

    A framework for reasoning about the human in the loop,

    L. F. Cranor, “A framework for reasoning about the human in the loop,” inProceedings of the Conference on Usability, Psychology, and Security, ser. UPSEC’08. USA: USENIX Association, 2008

    2008

  22. [30]

    Moving from a ‘human-as-problem

    V . Zimmermann and K. Renaud, “Moving from a ‘human-as-problem” to a ‘human-as-solution” cybersecurity mindset,”International Journal of Human-Computer Studies, vol. 131, pp. 169–187, 2019

    2019

  23. [31]

    Huntgpt: Integrating machine learning-based anomaly detection and explainable ai with large language models (llms),

    T. Ali and P. Kostakos, “Huntgpt: Integrating machine learning-based anomaly detection and explainable ai with large language models (llms),” 2023

    2023

  24. [32]

    Shield: Apt detection and intelligent explanation using llm,

    P. A. Gandhi, P. N. Wudali, Y . Amaru, Y . Elovici, and A. Shabtai, “Shield: Apt detection and intelligent explanation using llm,” 2025

    2025

  25. [33]

    The prompt report: A systematic survey of prompt engineering techniques,

    S. Schulhoff, M. Ilie, N. Balepur, K. Kahadzeet al., “The prompt report: A systematic survey of prompt engineering techniques,”arXiv preprint arXiv:2406.06608, 2024

    2024 arXiv

  26. [34]

    Deepseek-r1,

    DeepSeek, “Deepseek-r1,” 2025, accessed: 2025-04-10. [Online]. Available: https://github.com/deepseek-ai/DeepSeek-R1

    2025

  27. [35]

    Hello gpt-4o,

    OpenAI, “Hello gpt-4o,” 2024, accessed: 2025-04-10. [Online]. Available: https://openai.com/index/hello-gpt-4o/

    2024

  28. [36]

    Introducing openai o1-preview,

    ——, “Introducing openai o1-preview,” 2024, accessed: 2025-04-

    2024

  29. [37]

    Available: https://openai.com/index/introducing-openai- o1-preview/

    [Online]. Available: https://openai.com/index/introducing-openai- o1-preview/

  30. [38]

    Grok 3: The next generation of conversational ai,

    xAI, “Grok 3: The next generation of conversational ai,” 2024, accessed: 2025-04-10. [Online]. Available: https://x.ai/grok

    2024

  31. [39]

    Llm leaderboard,

    J. Chavez, “Llm leaderboard,” https://llm-stats.com/, 2025, accessed: 2025-04-10

    2025

  32. [40]

    Chatbot arena llm leaderboard: Community-driven evaluation for best llm and ai chatbots,

    UC Berkeley SkyLab and LMArena, “Chatbot arena llm leaderboard: Community-driven evaluation for best llm and ai chatbots,” https: //lmarena.ai/, 2025, accessed: 2025-04-10

    2025

  33. [41]

    Bertscore: Evaluating text generation with bert,

    T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi, “Bertscore: Evaluating text generation with bert,”arXiv:1904.09675, 2020

    1904 arXiv

  34. [42]

    MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance,

    W. Zhao, M. Peyrard, F. Liu, Y . Gaoet al., “MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance,” in9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), K. Inui, J. Jianget al., Eds. Association for Compu...

    2019

  35. [43]

    How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation,

    C.-W. Liu, R. Lowe, I. Serban, M. Noseworthyet al., “How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation,” inProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, J. ...

    2016

  36. [44]

    Openai evals: A framework for evaluating llms and llm systems,

    OpenAI, “Openai evals: A framework for evaluating llms and llm systems,” 2023, accessed: 2025-03-28. [Online]. Available: https://github.com/openai/evals

    2023

  37. [45]

    G-eval: Nlg evaluation using gpt-4 with better human alignment,

    Y . Liu, D. Iter, Y . Xu, S. Wang, R. Xu, and C. Zhu, “G-eval: Nlg evaluation using gpt-4 with better human alignment,”arXiv preprint arXiv:2303.16634, 2023

    2023 arXiv

  38. [46]

    Secure: Bench- marking generative large language models for cybersecurity advisory,

    D. Bhusal, M. T. Alam, L. Nguyen, A. Maharaet al., “Secure: Bench- marking generative large language models for cybersecurity advisory,” CoRR, vol. abs/2405.20441, 2024

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.