Pith. sign in

REVIEW 4 major objections 5 minor 29 references

Fine-tuning small LLMs for cybersecurity QA consistently degrades their vocabulary and parametric knowledge, and the effect depends on the tuning regime.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 14:30 UTC pith:FJGE7L7Q

load-bearing objection Honest pilot with a useful diagnostic framing, but the abstention story is inferred and the n=5 single-run data won't support the screening claims yet. the 4 major comments →

arxiv 2607.18725 v1 pith:FJGE7L7Q submitted 2026-07-21 cs.CL cs.AIcs.CR

Find Before You Fine-Tune: A Diagnostic Study of Small LLMs for Cybersecurity QA

Shaswata Mitra , Subash Neupane , Trisha Chakraborty , Himanshu Tripathi , Sudip Mittal , Aritran Piplai , Shahram Rahimi This is my paper
classification cs.CL cs.AIcs.CR
keywords large language modelsfine-tuningcybersecurityquestion answeringretrieval-augmented generationmodel selectionabstentionrank correlation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces FiT, a lightweight diagnostic that scores a small language model's cybersecurity QA suitability along three capabilities before any fine-tuning is done: vocabulary recognition, parametric knowledge, and contextualization of retrieved information. Using FiT on five 7-billion-parameter models, it finds that fine-tuning does not uniformly help: both a knowledge-focused and an instruction-focused regime degrade vocabulary and standalone knowledge scores in every model. The two regimes trade off differently, however. Knowledge-focused tuning causes moderate degradation that preserves the relative ranking of models, while instruction-focused tuning collapses measured knowledge through induced abstention and inverts the knowledge ranking even as retrieval-grounded answering stays intact. The paper argues that pre-adaptation FiT scores can therefore anticipate the direction of post-tuning change and help practitioners avoid unnecessary fine-tuning in dynamic, low-data domains.

Core claim

The central discovery is that the effect of fine-tuning on small LLMs for cybersecurity QA is regime-specific but directionally predictable, and that a pre-tuning diagnostic can map it. Across five 7B open-weight models, both fine-tuning regimes lowered keyword-recognition F1 and MCQ accuracy relative to the pre-trained baseline. Knowledge-focused tuning produced moderate, rank-preserving degradation, with a Spearman correlation of 0.70 for knowledge rankings. Instruction-focused tuning, which explicitly trained the model to abstain when uncertain, drove measured MCQ accuracy toward near zero and inverted knowledge rankings (Spearman -0.60), but the paper argues this collapse is abstention r

What carries the argument

The central mechanism is FiT, a task-oriented decomposition of QA suitability into three scores: keyword-recognition F1 (vocabulary), multiple-choice accuracy (parametric knowledge), and a retrieval-augmented completion task scored for answer correctness and semantic similarity (contextualization). The analysis device is Spearman rank correlation between pre-trained and post-tuned scores across the five models, which separates 'rank-preserving degradation' from 'ranking inversion' and thereby makes the regime-specific trade-off visible.

Load-bearing premise

The load-bearing assumption is that the observed differences between pre-trained and fine-tuned models are caused by the fine-tuning regime itself, not by run-to-run variance, distribution shift between training and evaluation data, or the modest dataset sizes, since each regime was run once per model and no significance tests were reported.

What would settle it

A concrete falsifier: run the same two fine-tuning regimes multiple times (e.g., five seeds) on the same five models with the same hyperparameters, and report the distribution of MCQ accuracy and Spearman correlations. If the between-seed variance is comparable to or larger than the observed drops (e.g., a 0.20 accuracy fall) or the Spearman values cross zero across seeds, the directional claims are not supported. Additionally, if an instruction-tuned model with the abstention prompt removed restores MCQ accuracy to pre-trained levels, the abstention interpretation is confirmed; if accuracy st

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Practitioners can use pre-tuning FiT scores as a screen: weak models on vocabulary stay weak after tuning, so expensive fine-tuning of those candidates can be avoided.
  • If instruction-focused tuning is planned, teams should expect closed-book knowledge scores to collapse through abstention and should evaluate such models in their retrieval-grounded configuration, not on standalone knowledge probes.
  • In dynamic, low-data domains like cybersecurity, a well-chosen pre-trained model paired with strong retrieval is often safer than fine-tuning, since fine-tuning degrades parametric capabilities without improving contextual correctness.
  • Retrieval-grounded contextualization is robust to fine-tuning, meaning investment in retrieval quality yields more than investment in fine-tuning for answer correctness.
  • The FiT diagnosis makes the direction of post-tuning change predictable enough to guide model selection before adaptation cost is paid.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the abstention interpretation is right, an instruction-tuned model's low closed-book MCQ score may actually be a safety feature in RAG deployments; a direct test would be comparing MCQ performance with and without an 'answer even if unsure' prompt.
  • The rank-inversion result suggests that leaderboard-style pre-tuning comparison is regime-dependent: a model that looks best before tuning can look worst after instruction tuning, so any model-selection benchmark should specify the downstream adaptation regime.
  • Because the semantic-similarity score is near-saturated across all models (around 0.85–0.93), the contextualization measure may need harder or more sensitive test items before it can discriminate among models; the same FiT design at larger scale or with adversarial distractors could test this.
  • The single-run, no-significance-test design means the numeric patterns are directional; a replication with multiple seeds and significance testing would establish whether the Spearman values are stable.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. FiT (Find before Fine-Tune) is a three-task diagnostic framework: keyword recognition (Task 1), MCQ knowledge (Task 2), and RAG-based contextualization (Task 3). The paper applies it to five 7B open-weight models before and after two QLoRA fine-tuning regimes: knowledge-focused (Finetuned-1, trained on Talos-derived QA pairs) and instruction-focused (Finetuned-2, trained on a split of the Task-3 corpus with abstain instructions). Table II reports consistent drops in Tasks 1 and 2 under both regimes; Finetuned-1 preserves pre-trained rankings (Table IV: ρ=0.60 Task 1, 0.70 Task 2) while Finetuned-2 inverts the knowledge ranking (ρ=-0.60), which the authors interpret as induced abstention. Task 3 scores remain stable across regimes. The paper concludes that pre-adaptation FiT scores can screen models and that a well-chosen pretrained model with RAG is often safer than fine-tuning. Limitations are candidly listed in Section VI.F, including single runs, n=50/100/50 datasets, and no significance tests.

Significance. The paper addresses a real operational problem—model selection before costly fine-tuning—and its task decomposition is sensible and more process-oriented than fixed leaderboards. The metrics in Eqs. (1)–(3) are measured independently of the fine-tuning outcomes, so the study is not circular in the strong sense. The SME validation and the explicit threat-to-validity section are commendable. If the empirical patterns survive additional scrutiny, the distinction between parametric knowledge and retrieval-grounded contextualization, and the regime-dependent trade-off, would be a useful contribution. However, the load-bearing claims currently rest on a small, single-run sample, and the abstention mechanism is inferred rather than observed. The practical guidance in Section VI.E is stronger than the evidence supports.

major comments (4)
  1. [§V.D, §VI.F, Table IV] Section V.D and Section VI.F state that each fine-tuning regime was run once per model and that no significance tests were performed. These are not just side limitations; they bear directly on RQ2 and RQ3. Section VI.A's 'both fine-tuning regimes degrade ... for every model' and Section VI.B's 'knowledge rankings invert' are causal/directional claims drawn from Table II and Table IV. With one run per condition and n=5 models for the Spearman correlations, the observed ρ values (0.60, 0.70, −0.60) cannot distinguish a stable regime effect from run-to-run variance; the paper itself calls them 'descriptive of direction rather than statistically significant.' Please provide repeated runs (at least 3–5 seeds) with mean±SD or confidence intervals, and either add a permutation test or explicitly withdraw the predictive 'anticipate the direction' conclusion in RQ3 and the abstract.
  2. [§VI.C, Eq. (2)] The claim that the Finetuned-2 Task-2 collapse is 'abstention, not knowledge loss' is underdetermined by the reported evidence. Eq. (2) scores any non-gold answer as an error; no output-level analysis, abstention counts, or examples are given. The two arguments offered—the training data instructed abstention and Task-3 correctness is retained—do not discriminate. A model that genuinely lost parametric knowledge could still pass Task 3 because the necessary CVE/version facts are supplied in the retrieved context, and Finetuned-2 was trained on a split of the same synthetic Task-3 corpus (Section V.A), so Task-3 retention partly reflects in-distribution overlap. Please report the distribution of Task-2 responses (correct vs. wrong option vs. abstain), and ideally test a Finetuned-2 variant without the abstain instruction. Until then, the 'abstention-driven inversion' and the associated dep
  3. [§VI.F, §V.A] The Finetuned-1 result is confounded by distribution shift. Task 2 is the MMLU computer-security subset, while Finetuned-1 is trained on QA pairs generated from Cisco Talos. The observed Task-2 drop could be caused by the shift in question style/topic between the training data and the MMLU probe, rather than by fine-tuning-induced degradation of parametric knowledge. To support the 'knowledge-focused tuning causes moderate degradation' claim, please add a knowledge probe matched to the fine-tuning domain (e.g., a held-out set of Talos-style questions) or a control model fine-tuned on unrelated domain data. The current disclosure of this limitation in Section VI.F does not remove the confound from the central claim.
  4. [§VI.B, Table IV, Conclusion] The RQ3 claim that 'pre-fine-tuning FiT scores anticipate the direction of post-tuning change' is stronger than the data allow. Task 3's row is explicitly not interpretable (the score range is 0.69–0.79), Task 1 correlations are modest, and the Task 2 correlation becomes negative under Finetuned-2. A high pre-trained knowledge score only 'anticipates' the direction if the regime is already known; it does not identify which model will be best. Please rephrase the contribution as a regime-dependent description rather than a predictive screen, or add out-of-sample prediction evidence.
minor comments (5)
  1. [Table II] The caption says 'TASK3: RAGAS SEMANTIC SIMILARITY (SIM) AND ANSWER CORRECTNESS (COR' with a missing closing parenthesis; also use consistent capitalization ('Task 1', 'Task 2', 'Task 3').
  2. [Figures 3–5] The captions contain stray subfigure labels and repeated parenthetical items, e.g., '(b) (c)' at the end of Fig. 3's caption and '(d) (b) (c)' in Fig. 4's caption. These appear to be formatting errors and should be cleaned up.
  3. [Reference [22]] Reference [22] is listed as 'National vulnerability database' but the source is Cisco Talos. The title appears to be a copy-paste error from the NIST reference and should be corrected.
  4. [§VI.D] The phrase 'range0.69–0.79' is missing a space after 'range'; this should read 'range 0.69–0.79'.
  5. [§I] The claim that FiT is 'the first framework' to diagnose small LLM suitability before adaptation is not fully supported by the related-work discussion. The authors should clarify what 'first' means relative to benchmarks like CYBERBENCH and how FiT differs beyond reporting static performance.

Circularity Check

0 steps flagged

No significant circularity; empirical diagnostics are independent of the post-tuning outcomes.

full rationale

The paper's derivation chain is empirical rather than definitional. FiT scores are defined independently (Eq. 1 keyword F1, Eq. 2 MCQ accuracy, Eq. 3 RAGAS against SME references) and measured before and after fine-tuning; no parameter is fitted to the outcome being predicted. The rank correlations in Table IV are descriptive in-sample statistics explicitly flagged as non-significant over five models ('the Spearman values in Table IV are computed over five models and are descriptive of direction rather than statistically significant'), so the RQ3 'anticipation' claim is not a fitted prediction forced by construction. The only overlapping-author citation is [5] (Ranade, Piplai, Mittal), used for background ('The application of pre-trained LLMs in specialized domains has been an active research area [5]'), not as load-bearing evidence. Section VI.C's abstention explanation is an interpretation of the Task-2 collapse supported by training-data composition and Task-3 retention; the paper itself concedes Task-2 conflates abstention with wrong answers and reports no significance tests. This is an evidential/statistical weakness, not circularity. No equation reduces to its own input and no claim is imported from a self-cited uniqueness or ansatz result.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 1 invented entities

No numeric free parameters are fitted in the mathematical sense, but the paper implicitly assumes stability of a small-n single-run experimental setup, the validity of proxy metrics (MCQ accuracy, RAGAS), and the representativeness of synthetic data. These are domain assumptions rather than fitted constants. The one invented entity, FiT, is a diagnostic procedure with no independent external validation provided in this paper.

axioms (5)
  • domain assumption RAGAS answer correctness and semantic similarity are valid proxies for contextualization quality in this setting.
    The paper relies on RAGAS scores (Eq. 3) as the Task-3 metric, and notes that semantic similarity is 'saturated and non-discriminative' and should not be read as an information-leakage measure (Section VI.D). The choice of RAGAS is justified by a single citation; its reliability for n=50 with synthetic context is an assumption.
  • domain assumption MCQ accuracy (Eq. 2) conflating wrong answers and abstentions is a usable 'parametric knowledge' proxy.
    The paper explicitly says 'this score conflates two distinct behaviors when a model declines to answer' (Section IV.B). The abstention interpretation of the Task-2 collapse is supported only indirectly via Task-3 performance and the training data composition (Section VI.C), and is not verified by inspecting the model's outputs (e.g., counting abstention tokens).
  • ad hoc to paper Single-run QLoRA fine-tuning with fixed hyperparameters represents the two 'regimes' faithfully.
    Section V.D: 'Each regime was run once per model.' The paper itself flags that regime effects are not separated from run-to-run variance (Section VI.F). The claim that knowledge-focused tuning is 'rank-preserving' and instruction-focused 'rank-inverting' depends on this assumption.
  • domain assumption The 50 Task-1 prompts, 100 MMLU items, 50 Task-3 questions, and 50 generated QA pairs are representative of cybersecurity QA.
    Section VI.F 'External' notes the datasets are modest and the threat-intelligence snapshot is fixed in time. The paper's guidance to practitioners is predicated on this small sample generalizing to real deployment settings.
  • ad hoc to paper Synthetic infrastructure-specific data (S_d) behaves like real sensitive infrastructure data for the contextualization task.
    Section V.A: 'since such infrastructure detail is sensitive, we substitute synthetic data of the same form.' The paper acknowledges this in VI.F 'External' but the Task-3 robustness claim rests on the synthetic data being a faithful stand-in.
invented entities (1)
  • FiT (Find before Fine-Tune) diagnostic framework no independent evidence
    purpose: A three-task scoring scheme to evaluate small LLMs' cybersecurity QA suitability before adaptation.
    FiT is the paper's own proposed framework. It is a measurement protocol, not a testable physical entity; its validity is only supported by the internal consistency of the study (e.g., SMEs' Fleiss κ, the abstention story) rather than by an external benchmark or formal guarantee. The paper frames it as a 'lens,' not a universal predictor, and explicitly says it is not a general predictive benchmark.

pith-pipeline@v1.3.0-alltime-deepseek · 10505 in / 9858 out tokens · 63560 ms · 2026-08-01T14:30:53.234458+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Find Before You Fine-Tune: A Diagnostic Study of Small LLMs for Cybersecurity QA." pith.science (2026). https://pith.science/paper/FJGE7L7Q

@misc{pith2026260718725,
  author       = {Pith},
  title        = {Pith review of: Find Before You Fine-Tune: A Diagnostic Study of Small LLMs for Cybersecurity QA},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FJGE7L7Q}},
  note         = {Machine review of arXiv:2607.18725}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large Language Models (LLMs) are increasingly fine-tuned for critical-domain Question-Answering (QA), yet choosing which small model to adapt, before paying the cost of adaptation, remains difficult. Fine-tuning can improve domain alignment, but it may also erode prior knowledge, weaken instruction-following, or increase hallucination, especially when labeled data are scarce or rapidly evolving as in cybersecurity. We present FiT (Find before Fine-Tune), a task-oriented diagnostic framework that characterizes small LLMs along three capabilities required for cybersecurity QA: vocabulary recognition, parametric knowledge, and contextualization of retrieved information. Using FiT, we conduct an empirical study of five open-weight 7-billion-parameter models under two fine-tuning regimes. We find that fine-tuning does not uniformly help: it consistently degrades vocabulary and parametric knowledge in small models, and the two regimes trade off differently. Knowledge-focused tuning causes moderate, rank-preserving degradation, whereas instruction-focused tuning collapses measured knowledge through induced abstention, inverting the knowledge ranking while leaving retrieval-grounded contextualization essentially intact. We quantify these regime-specific patterns with rank-correlation analysis and show that pre-fine-tuning FiT scores anticipate the direction of post-tuning change. Our results suggest that task-oriented diagnosis can screen out unsuitable models, avoid unnecessary fine-tuning, and support safer deployment of small LLMs in cybersecurity QA pipelines.

Figures

Figures reproduced from arXiv: 2607.18725 by Aritran Piplai, Himanshu Tripathi, Shahram Rahimi, Shaswata Mitra, Subash Neupane, Sudip Mittal, Trisha Chakraborty.

Figure 1
Figure 1. Figure 1: Implementation scope of FiT in the generative-AI life-cycle (colored area implies applicability) with an example of our three evaluation tasks. By [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Flowchart of the FiT evaluation process. The three tasks align with the deployment scope: Task 1 assesses vocabulary, Task 2 assesses domain [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance of pre-trained models on Tasks 1 and 3. (a) [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Performance of Finetuned-1 (knowledge-focused) models on Tasks 1 and 3. (a) [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Performance of Finetuned-2 (instruction-focused) models on Tasks 1 and 3. (a) [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 8 linked inside Pith

  1. [1]

    Hallucinat- ing law: Legal mistakes with large language models are per- vasive

    M. Dahl, V . Magesh, M. Suzgun, and D. E. Ho, “Hallucinat- ing law: Legal mistakes with large language models are per- vasive.” https://hai.stanford.edu/news/hallucinating-law-legal-mistakes- large-language-models-are-pervasive, 2024

    2024

  2. [2]

    Retrieval- augmented generation for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel,et al., “Retrieval- augmented generation for knowledge-intensive nlp tasks,”Advances in Neural Information Processing Systems, vol. 33, pp. 9459–9474, 2020

    2020

  3. [3]

    Cyberbench: A multi-task benchmark for evaluating large language models in cybersecurity,

    Z. Liu, J. Shi, and J. F. Buford, “Cyberbench: A multi-task benchmark for evaluating large language models in cybersecurity,” 2024

    2024

  4. [4]

    Securebert: A domain- specific language model for cybersecurity,

    E. Aghaei, X. Niu, W. Shadid, and E. Al-Shaer, “Securebert: A domain- specific language model for cybersecurity,” inInternational Conference on Security and Privacy in Communication Systems, pp. 39–56, Springer, 2022

    2022

  5. [5]

    Generating fake cyber threat intelligence using transformer-based models,

    P. Ranade, A. Piplai, S. Mittal, A. Joshi, and T. Finin, “Generating fake cyber threat intelligence using transformer-based models,” in2021 International Joint Conference on Neural Networks (IJCNN), pp. 1–9, IEEE, 2021

    2021

  6. [6]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in neural information processing systems, vol. 30, 2017

    2017

  7. [7]

    Recent advances in natural language processing via large pre-trained language models: A survey,

    B. Min, H. Ross, E. Sulem, A. P. B. Veyseh, T. H. Nguyen, O. Sainz, E. Agirre, I. Heintz, and D. Roth, “Recent advances in natural language processing via large pre-trained language models: A survey,”ACM Computing Surveys, vol. 56, no. 2, pp. 1–40, 2023

    2023

  8. [8]

    Survey on factuality in large language models: Knowledge, retrieval and domain-specificity,

    C. Wang, X. Liu, Y . Yue, X. Tang, T. Zhang, C. Jiayang, Y . Yao, W. Gao, X. Hu, Z. Qi,et al., “Survey on factuality in large language models: Knowledge, retrieval and domain-specificity,”arXiv preprint arXiv:2310.07521, 2023

    Pith/arXiv arXiv 2023

  9. [9]

    Overcoming catastrophic forgetting in neural networks,

    J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al., “Overcoming catastrophic forgetting in neural networks,”Pro- ceedings of the national academy of sciences, vol. 114, no. 13, pp. 3521– 3526, 2017

    2017

  10. [10]

    Parameter-efficient transfer learning for nlp,

    N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for nlp,” inInternational conference on machine learning, pp. 2790–2799, PMLR, 2019

    2019

  11. [11]

    The power of scale for parameter-efficient prompt tuning,

    B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,”arXiv preprint arXiv:2104.08691, 2021

    Pith/arXiv arXiv 2021

  12. [12]

    Dylora: Parameter efficient tuning of pre-trained models using dynamic search- free low-rank adaptation,

    M. Valipour, M. Rezagholizadeh, I. Kobyzev, and A. Ghodsi, “Dylora: Parameter efficient tuning of pre-trained models using dynamic search- free low-rank adaptation,”arXiv preprint arXiv:2210.07558, 2022

    Pith/arXiv arXiv 2022

  13. [13]

    Qlora: Efficient finetuning of quantized llms,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  14. [14]

    Unipelt: A unified framework for parameter-efficient language model tuning,

    Y . Mao, L. Mathias, R. Hou, A. Almahairi, H. Ma, J. Han, W.-t. Yih, and M. Khabsa, “Unipelt: A unified framework for parameter-efficient language model tuning,”arXiv preprint arXiv:2110.07577, 2021

    Pith/arXiv arXiv 2021

  15. [15]

    Glue-x: Evaluating natural language understanding models from an out-of-distribution generalization perspective,

    L. Yang, S. Zhang, L. Qin, Y . Li, Y . Wang, H. Liu, J. Wang, X. Xie, and Y . Zhang, “Glue-x: Evaluating natural language understanding models from an out-of-distribution generalization perspective,”arXiv preprint arXiv:2211.08073, 2022

    Pith/arXiv arXiv 2022

  16. [16]

    Promptbench: Towards evaluating the robustness of large language models on adversarial prompts,

    K. Zhu, J. Wang, J. Zhou, Z. Wang, H. Chen, Y . Wang, L. Yang, W. Ye, N. Z. Gong, Y . Zhang,et al., “Promptbench: Towards evaluating the robustness of large language models on adversarial prompts,”arXiv preprint arXiv:2306.04528, 2023

    Pith/arXiv arXiv 2023

  17. [17]

    Kola: Carefully benchmarking world knowledge of large language models,

    J. Yu, X. Wang, S. Tu, S. Cao, D. Zhang-Li, X. Lv, H. Peng, Z. Yao, X. Zhang, H. Li,et al., “Kola: Carefully benchmarking world knowledge of large language models,”arXiv preprint arXiv:2306.09296, 2023

    Pith/arXiv arXiv 2023

  18. [18]

    Large language models encode clinical knowledge,

    K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl,et al., “Large language models encode clinical knowledge,”Nature, vol. 620, no. 7972, pp. 172– 180, 2023

    2023

  19. [19]

    Not what you’ve signed up for: Compromising real-world llm- integrated applications with indirect prompt injection,

    K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real-world llm- integrated applications with indirect prompt injection,” inProceedings of the 16th ACM Workshop on Artificial Intelligence and Security, pp. 79– 90, 2023

    2023

  20. [20]

    Measuring massive multitask language understanding

    CAIS, “Measuring massive multitask language understanding.” hugging- face.co/datasets/cais/mmlu, 2024

    2024

  21. [21]

    National vulnerability database

    NIST, “National vulnerability database.” nist.gov, 2024

    2024

  22. [22]

    National vulnerability database

    Cisco, “National vulnerability database.” talosintelligence.com, 2024

    2024

  23. [23]

    Ragas: Au- tomated evaluation of retrieval augmented generation,

    S. Es, J. James, L. Espinosa-Anke, and S. Schockaert, “Ragas: Au- tomated evaluation of retrieval augmented generation,”arXiv preprint arXiv:2309.15217, 2023

    Pith/arXiv arXiv 2023

  24. [24]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” inProceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311–318, 2002

    2002

  25. [25]

    A package for automatic evaluation of summaries,

    L. C. Rouge, “A package for automatic evaluation of summaries,” in Proceedings of Workshop on Text Summarization of ACL, Spain, vol. 5, 2004

    2004

  26. [26]

    Likert scales and data analyses,

    I. E. Allen and C. A. Seaman, “Likert scales and data analyses,”Quality progress, vol. 40, no. 7, pp. 64–65, 2007

    2007

  27. [27]

    Interrater reliability: the kappa statistic,

    M. L. McHugh, “Interrater reliability: the kappa statistic,”Biochemia medica, vol. 22, no. 3, pp. 276–282, 2012

    2012

  28. [28]

    Parameter-efficient fine-tuning of large- scale pre-trained language models,

    N. Ding, Y . Qin, G. Yang, F. Wei, Z. Yang, Y . Su, S. Hu, Y . Chen, C.-M. Chan, W. Chen,et al., “Parameter-efficient fine-tuning of large- scale pre-trained language models,”Nature Machine Intelligence, vol. 5, no. 3, pp. 220–235, 2023

    2023

  29. [29]

    The art of abstention: Selective prediction and error regularization for natural language processing,

    J. Xin, R. Tang, Y . Yu, and J. Lin, “The art of abstention: Selective prediction and error regularization for natural language processing,” inProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1040– 1051, 2021. (a) ...

    2021