Pith. sign in

REVIEW 4 major objections 5 minor 52 references

Less Data, More Security: Advancing Cybersecurity LLMs Specialization via Resource-Efficient Domain-Adaptive Continuous Pre-training with Minimal Tokens

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A curated 118M-token corpus beats billion-token security LLMs

desk verdict Real training runs and an honest limitations section, but the headline data-efficiency claim is confounded by model scale, base alignment, and evaluation protocol. read the letter →

arxiv 2507.02964 v2 pith:FF2OFFRK submitted 2025-06-30 cs.CL cs.AIcs.CRcs.LG

Salahuddin Salahuddin , Ahmed Hussain , Jussi Löppönen , Toni Jutila This is my paper · ORCID
classification cs.CLcs.AIcs.CRcs.LG
keywords cybersecurityLLMsdomain-adaptivecontinuouspretrainingdata-efficientspecializationbenchmarkssustainableAIthreatintelligence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a large language model can be specialized for cybersecurity with a carefully curated corpus of 126 million words (118.8 million tokens), rather than the billions of tokens used by existing specialized models. The authors continuously pre-train three open decoder models on this corpus and report that all three improve on CTI-MCQ, CyberMetric, and SecEval, with the Llama-3.3-70B-Instruct variant reaching accuracies of 0.718, 0.933, and 0.864. Those scores are claimed to beat Llama-Primus-Base and Foundation-Sec-8B, which used 2.77 billion and 5 billion tokens, a 23-to-42-fold data reduction. If true, the result matters because security-specialized assistants could be built at a fraction of the compute, cost, and energy that the field currently assumes is necessary.

What carries the argument

The load-bearing object is the DAP pipeline: continued causal-language-model training on a domain corpus after initial pretraining, configured with frozen embeddings, a conservative learning rate around 1e-6, two to three epochs, and a distributed fully sharded data-parallel setup on multi-GPU nodes. The argument's other half is the corpus itself: 126 million words deliberately selected from standards, research papers, and technical books, cleaned by regular-expression filtering and segmented at paragraph level. What this machinery does is update the model's internal representations on domain text rather than adding task-specific adapters on top, which the paper argues is necessary for intrinsic cybersecurity understanding.

What would settle it

Run the DAP models and the two billion-token baselines on identical prompt formats, both zero-shot and 5-shot, across the three benchmarks; if matching the shot count erases the reported accuracy gaps, the data-efficiency claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that domain-adaptive continuous pretraining on a small, authoritative corpus transfers enough cybersecurity knowledge to beat much more data-hungry baselines. Using standards, academic literature, and technical books, the authors adapt Llama-3.1-8B (1 million tokens), DeepSeek-R1-Distill-Qwen-14B (50 million tokens), and Llama-3.3-70B-Instruct (118.8 million tokens). On the three benchmarks, the 70B adapted model achieves 0.718, 0.933, and 0.864, compared with 0.667, 0.866, and 0.500 for Llama-Primus-Base and 0.676 and 0.851 for Foundation-Sec-8B on the benchmarks those reports cover. The paper further claims that full continued pretraining outperforms a LoRA/PEFT baseline, and that the largest gain appears where the base model is weakest: SecEval jumps from 0.5081 to 0.8638 after adaptation.

Load-bearing premise

The data-efficiency claim treats published 5-shot scores of the 8B baselines as directly comparable to zero-shot scores of the new models, attributing any accuracy gap to training-data size.

Editorial extensions

If this is right

  • A reproducible, roughly 120-million-token corpus would be enough to specialize instruction-tuned LLMs for cybersecurity, cutting training cost and energy by one to two orders of magnitude.
  • Dataset size must be calibrated to model capacity: the 8B model trained on 1 million tokens showed almost no SecEval gain, while the 70B model trained on 118.8 million tokens gained sharply.
  • Frozen-backbone adapters (LoRA/PEFT) are claimed to be insufficient for deep domain specialization, so continued pretraining should remain the default when domain understanding is the goal.
  • Because the biggest jump came on the benchmark where the base 70B model scored lowest, the method may be most valuable for repairing weak, format-sensitive reasoning in otherwise capable models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors' comparison mixes zero-shot scores for their models with 5-shot scores for the cited baselines; a matched-shot head-to-head is the natural test of the 23-to-42-fold data-efficiency claim.
  • If the data-efficiency mechanism is really curation rather than model size, the same recipe should transfer to other expert domains, a consequence the paper does not claim.
  • The 70B SecEval leap from 0.508 to 0.864 could partly reflect improved adherence to the multi-answer output format rather than pure cybersecurity knowledge; a format-only control would separate these.
  • General language ability was not benchmarked after adaptation, so the practical upside of the method may be offset by hidden capability loss until such validation is run.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper investigates Domain-Adaptive Continuous Pretraining (DAP) as a resource-efficient way to specialize pretrained LLMs for cybersecurity. The authors curate a 126-million-word corpus, train three models (Llama-3.1-8B, DeepSeek-R1-Distill-Qwen-14B, Llama-3.3-70B-Instruct) on different token subsets, and evaluate on CTI-MCQ, CyberMetric, and SecEval. They report consistent accuracy gains and claim that the 70B model reaches state-of-the-art scores (0.718, 0.933, 0.864) using only 118.8 million tokens, a 23-to-42-fold reduction relative to Llama-Primus-Base and Foundation-Sec-8B. The paper also provides infrastructure details, cost estimates, and an explicit discussion of limitations.

Significance. If the data-efficiency claim were valid, the result would be practically important: a small curated corpus could replace billions of tokens for domain specialization, with major savings in compute, cost, and energy. The paper's strengths include transparent reporting of training configurations, cost accounting, and an honest limitations section that identifies several confounds. The evaluation covers three model scales and three benchmarks, and the authors are explicit about their zero-shot protocol. However, the significance is conditional on the central claim that token count, rather than model scale, base alignment, or evaluation protocol, drives the reported superiority. The current design does not support that causal attribution.

major comments (4)
  1. [§6, Table 4, Figure 10] The headline 23-to-42-fold data-efficiency comparison is confounded by model scale and base alignment. Llama-3.3-70B-Ins-DAP is a 70B instruction-tuned model, whereas Llama-Primus-Base and Foundation-Sec-8B are 8B models. The 70B base already scores 0.705, 0.926, and 0.508 on CTI-MCQ, CyberMetric, and SecEval (§5.4), beating Llama-Primus-Base's 0.667, 0.866, and 0.500 on two of three benchmarks before any DAP is applied. Consequently, the accuracy advantage cannot be attributed to the smaller token count, and the abstract's 'despite utilizing only 118.8 million tokens' asserts a causal relation that the experimental design does not isolate.
  2. [§6, Zero-Shot Generalization] The comparison mixes evaluation protocols: the new models are evaluated zero-shot, while the cited baselines are evaluated 5-shot, as the text acknowledges. The statement 'Despite this methodological difference (zero-shot vs. 5-shot evaluation), our DAP models demonstrate higher performance across all three benchmarks' assumes that zero-shot and 5-shot results are directly comparable. This assumption is load-bearing and untested; a 5-shot evaluation of the new models or a zero-shot evaluation of the baselines is required before any superiority claim can be made. Without such a control, the performance gaps in Table 4 and Figure 10 may be protocol artifacts rather than evidence of domain adaptation quality.
  3. [§5.3 and Table 4] The 70B model's CyberMetric score is based on 1000 of the 2000 questions, while the cited baselines are evaluated on the full set; Table 4 reports 0.933 without noting this asymmetry. The text states 'Due to computational constraints, we evaluated the 70B model on reduced samples for CyberMetric (1000 questions) while maintaining full evaluation sets for other models and datasets.' This makes the 0.933 entry not directly comparable to the baseline CyberMetric values, and the data-efficiency argument inherits that incomparability. Moreover, no error bars, variance estimates, or repeated-run statistics are reported for any of the accuracy numbers.
  4. [§4.1.1 and §6.2] Dataset size is co-varied with model identity: 1 million tokens for Llama-3.1-8B, 50 million for DSR1D-Qwen-14B, and 118.8 million for Llama-3.3-70B-Ins. No model is trained at two corpus sizes under identical hyperparameters, so the marginal effect of token count is never identified. The paper's own §6.2 sections 'Non-Uniform Dataset Allocation' and 'Architectural Heterogeneity' admit that these confounds constrain cross-architecture comparisons, yet the abstract and conclusion still phrase the result as a data-volume effect. This internal tension should be resolved either by removing the causal claim or by adding controlled experiments that vary corpus size within a fixed model family and evaluation protocol.
minor comments (5)
  1. [§2.5.4 vs. Table 4] The CTI-MCQ score for Foundation-Sec-8B is given as 0.662 in §2.5.4 and as 0.676 in Table 4, while §6 states 'Foundation-Sec-8B maintains a slight edge on CTI-MCQ (0.662 vs. 0.642)'; these values should be reconciled.
  2. [References [40] and [52]] Reference [40] is titled 'Low-Rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition,' but the LoRA method attributed to it is the original LoRA paper by Hu et al.; the citation should be corrected. Reference [52] is KMMLU, not MMLU as cited in the Future Research Directions.
  3. [§4.1 vs. Table 1] The abstract and §4.1 describe the corpus as 126 million words (132 million tokens using the LLaMA tokenizer), but Table 1 lists 118.8 million tokens for the 70B model; the relationship between these numbers should be clarified.
  4. [Table 4, Baseline row] The baseline row is labeled 'Llama-3-8B-Base' in Table 4, while §6.2 and the surrounding text refer to 'Llama-3.1-8B-Base'; the model identifier should be consistent.
  5. [Figure 11] Figure 11 is described as demonstrating 'exponential scaling' of training time with model size, but only three points are shown; either add a fitted curve or soften the claim to 'increasing scaling.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central evaluation is anchored to external benchmarks, and no parameter or conclusion reduces by construction to the training corpus or to a self-citation chain.

full rationale

The paper's headline claim is that 118.8 million tokens of domain-adaptive pretraining suffice to surpass specialized cybersecurity models trained on 2.77B and 5B tokens. That claim is evaluated on three external benchmarks (CTI-MCQ, CyberMetric, SecEval) that are independent of the curated training corpus; no accuracy number in the paper is defined as a function of the corpus size or the training loss, and no benchmark answer or label is included in the training data. No fitted parameter is later renamed as a prediction, and the comparisons to Llama-Primus-Base and Foundation-Sec-8B are citations of external prior work, not self-citations. The paper itself acknowledges the main threats to causal inference in Section 6.2: non-uniform dataset allocation across model scales, architectural heterogeneity, and the absence of controlled equivalence testing. Those are validity and attribution concerns rather than circularity: confounded comparisons can be wrong without being circular. There is no equation-level reduction, no uniqueness theorem imported from the authors, and no ansatz smuggled in via citation. Under the rule that circularity must be exhibited as a specific reduction or a fitted-input-renamed-as-prediction, the derivation chain is self-contained against its external benchmarks, so the honest finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

No invented entities are introduced. The central claim rests on hand-picked hyperparameters, unequal dataset allocations, and comparability assumptions across heterogeneous models and evaluation protocols. The free parameters are design choices, not derived from theory, and the most consequential one is the dataset-size allocation per model, which is not controlled.

free parameters (5)
  • Learning rate = 1e-6
    Chosen by hand in Section 4.2.1 to balance adaptation and forgetting; no ablation is reported.
  • Training epochs = 2-3
    Set in Section 4.2.1 to prevent overfitting; no ablation is reported.
  • Max sequence length = 1024 or 2048 tokens
    Section 4.2.1, limited by GPU memory; may affect long-context performance.
  • Dataset tokens per model = 1M, 50M, 118.8M
    Section 4.1.1 and Table 1; the non-uniform allocation confounds cross-model comparisons and is central to the data-efficiency claim.
  • LoRA rank, alpha, dropout = r=16, alpha=32, dropout=0.05
    Section 5.1 PEFT baseline; a single configuration without tuning or repeated runs.
assumptions (5)
  • domain assumption Pretrained tokenizers have comprehensive vocabulary coverage for cybersecurity terms.
    Section 3.4 states tokenizers are used unmodified and embeddings are frozen, based on the assumption that the pretrained tokenizer already covers domain vocabulary.
  • domain assumption The curated corpus is representative of the knowledge tested by CTI-MCQ, CyberMetric, and SecEval.
    Section 3.2 and Section 4.1 describe the corpus but provide no verification of overlap or distributional match with the benchmark questions.
  • ad hoc to paper Zero-shot and 5-shot evaluation results are directly comparable across papers.
    Section 6 and Figure 10 use zero-shot for the new models and 5-shot for cited baselines, while asserting that the comparison still supports state-of-the-art performance.
  • domain assumption Jaccard Index on extracted answer sets is a valid measure of SecEval accuracy.
    Section 5.4 defines the metric but does not detail the answer extraction procedure or its reliability across models.
  • domain assumption GPU-hours are a valid proxy for energy consumption.
    Section 6.2 uses GPU-hours to estimate carbon footprint without direct energy measurements.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Less Data, More Security: Advancing Cybersecurity LLMs Specialization via Resource-Efficient Domain-Adaptive Continuous Pre-training with Minimal Tokens." pith.science (2026). https://pith.science/paper/FF2OFFRK

@misc{pith2026250702964,
  author       = {Pith},
  title        = {Pith review of: Less Data, More Security: Advancing Cybersecurity LLMs Specialization via Resource-Efficient Domain-Adaptive Continuous Pre-training with Minimal Tokens},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FF2OFFRK}},
  note         = {Machine review of arXiv:2507.02964}
}
read the original abstract

The increasing scale of AI workloads demands High-Performance Computing (HPC) infrastructure and training methodologies that are both scalable and sustainable. While Large Language Models (LLMs) demonstrate exceptional natural language capabilities, general-purpose models often lack the specialized domain knowledge necessary for effective cybersecurity analysis. We investigate Domain-Adaptive Continuous Pretraining (DAP) as a scalable, resource-efficient methodology for enhancing cybersecurity understanding in pretrained LLMs, implemented through a distributed Fully Sharded Data Parallel (FSDP) pipeline across multi-node GPU clusters. We systematically adapted three decoder-based architectures -- Llama-3.1-8B, DeepSeek-R1-Distill-Qwen-14B, and Llama-3.3-70B-Instruct -- using a curated 126-million-word cybersecurity corpus from standards, academic literature, and technical documentation. Evaluation across three cybersecurity benchmarks -- CTI-MCQ, CyberMetric, and SecEval -- demonstrates consistent improvements post-adaptation. Notably, our Llama-3.3-70B-Ins-DAP model achieves state-of-the-art performance with accuracies of 0.718, 0.933, and 0.864, respectively, surpassing parameter-efficient baselines and specialized models including Llama-Primus-Base (trained on 2.77 billion tokens) and Foundation-Sec-8B (trained on 5 billion tokens), despite utilizing only 118.8 million tokens -- representing a 23-to-42-fold reduction in training data. Targeted continuous pretraining via scalable HPC infrastructure enables effective cybersecurity domain adaptation with a substantially reduced computational and energy footprint, supporting specialized AI assistants in threat analysis, vulnerability assessment, and security documentation, while advancing sustainable and responsible AI development.

Figures

Figures reproduced from arXiv: 2507.02964 by the authors.

Figure 1
Figure 1. Traditional reactive cybersecurity framework illustrating the conven￾tional workflow from threat detection through policy enforcement. Large Language Models (LLMs), i.e., Generative AI (GenAI), to achieve greater efficiency. These agents can process vast amounts of data instantaneously, comprehend complex queries, and provide precise responses that often resolve problems without requiring human intervention. The ver… view at source ↗
Figure 2
Figure 2. Our domain-adaptive continuous pretraining pipeline. The workflow illustrates the systematic transformation from raw cybersecurity corpus collection through preprocessing, JSONL formatting, and distributed training to produce specialized domain-adapted model weights. 3.2. Dataset Preparation The foundation of our DAP methodology rests upon a meticulously curated cybersecurity corpus that achieves an op￾timal balance… view at source ↗
Figure 4
Figure 4. Sample tokenized input IDs (LLaMA3 tokenizer). [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (8 more)
Figure 3
Figure 3. Figure 3: Sample JSONL structure. Context window constraints of LLMs necessitated sequence truncation to specified lengths (1024 or 2048 tokens), as illus￾trated in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png]
Figure 5
Figure 5. Figure 5: AWS reference architecture for distributed training and evaluation [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Performance comparison of base and domain-adapted models on CTI [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Performance comparison of base and domain-adapted models on Cy [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Performance comparison of base and domain-adapted models on [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Semantic similarity analysis comparing model explanations to cor [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: Comparative performance of domain-adapted models against Llama-Primus-Base across three cybersecurity benchmarks. Our zero-shot domain-adapted models consistently outperform the specialized 5-shot Llama￾Primus-Base and Foundation-Sec-8B, with Llama-3.3-70B-Ins-DAP ach…
Figure 11
Figure 11. Figure 11: Training time comparison across model architectures demonstrating [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 31 canonical work pages

  1. [1]

    M.-F. Wong, S. Guo, C.-N. Hang, et al., Natural Lan- guage Generation and Understanding of Big Code for AI- Assisted Programming: A Review, Entropy 25 (6) (2023). doi:10.3390/e25060888

    doi:10.3390/e25060888 2023

  2. [2]

    J. P. Inala, C. Wang, S. M. Drucker, et al., Data Analysis in the Era of Generative AI, ArXiv abs/2409.18475 (2024)

    arXiv 2024

  3. [3]

    Mitra, M

    M. Mitra, M. G. De V os, N. Cortinovis, et al., Gen- erative AI for Research Data Processing: Lessons Learnt From Three Use Cases, 2024.doi:10.1109/e- Science62913.2024.10678704

    arXiv 2024

  4. [4]

    Lewis, E

    P. Lewis, E. Perez, A. Piktus, et al., Retrieval-augmented generation for knowledge-intensive NLP tasks, in: Pro- ceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Curran Asso- ciates Inc., Red Hook, NY , USA, 2020

    work page 2020

  5. [5]

    Yu, T.-H

    Y .-C. Yu, T.-H. Chiang, C.-W. Tsai, et al., Primus: A Pi- oneering Collection of Open-Source Datasets for Cyber- security LLM Training, arXiv preprint arXiv:2502.11191 (2025)

    arXiv 2025

  6. [6]

    Kassianik et al., Llama-3.1-foundationai-securityllm- base-8b technical report, arXiv preprint arXiv:2504.21039 (2025)

    arXiv 2025

  7. [7]

    Yun-tao, G

    Z. Yun-tao, G. Ling, W. Yong-cheng, An improved TF- IDF approach for text classification, Journal of Zhejiang University-Science A 6 (1) (2005) 49–55

    work page 2005

  8. [8]

    Ramos, et al., Using tf-idf to determine word relevance in document queries, in: Proceedings of the first instruc- tional conference on machine learning, V ol

    J. Ramos, et al., Using tf-idf to determine word relevance in document queries, in: Proceedings of the first instruc- tional conference on machine learning, V ol. 242, Citeseer, 2003, pp. 29–48

    work page 2003

Show all 52 references
  1. [9]

    P. F. Brown, V . J. Della Pietra, P. V . deSouza, et al., Class- Basedn-gram Models of Natural Language, Computa- tional Linguistics 18 (4) (1992) 467–480. URLhttps://aclanthology.org/J92-4003/

    1992

  2. [10]

    J. L. Elman, Finding structure in time, Cognitive science 14 (2) (1990)

    1990

  3. [11]

    Hochreiter, J

    S. Hochreiter, J. Schmidhuber, Long Short-Term Mem- ory, Neural Computation 9 (8) (1997).doi:10.1162/ neco.1997.9.8.1735

    1997

  4. [12]

    Graves, A

    A. Graves, A. rahman Mohamed, G. E. Hinton, Speech recognition with deep recurrent neural networks, 2013, pp. 6645–6649. URLhttps://api.semanticscholar.org/ CorpusID:206741496

    2013

  5. [13]

    Sutskever, O

    I. Sutskever, O. Vinyals, Q. V . Le, Sequence to se- quence learning with neural networks, in: International Conference on Neural Information Processing Systems, NIPS’14, 2014, p. 3104–3112

    2014

  6. [15]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, et al., Attention is All you Need, in: Advances in Neural Information Processing Systems, V ol. 30, 2017. URLhttps://proceedings.neurips.cc/ paper_files/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

    2017

  7. [16]

    Y . Ding, L. L. Zhang, C. Zhang, et al., LongRoPE: ex- tending LLM context window beyond 2 million tokens, in: Proceedings of the 41st International Conference on Machine Learning, ICML’24, JMLR.org, 2024

    2024

  8. [17]

    Q. Wang, B. Li, T. Xiao, et al., Learning Deep Trans- former Models for Machine Translation, in: Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computa- tional Linguistics, Florence, Italy, 2019, pp. 1810–1822. doi:10...

    2019 doi

  9. [18]

    Raffel, N

    C. Raffel, N. Shazeer, A. Roberts, et al., Exploring the limits of transfer learning with a unified text-to-text trans- former, J. Mach. Learn. Res. (Jan. 2020)

    2020

  10. [19]

    Lewis, Bart: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension, arXiv preprint arXiv:1910.13461 (2019)

    M. Lewis, Bart: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension, arXiv preprint arXiv:1910.13461 (2019)

    2019 arXiv

  11. [20]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Under- standing, in: North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, V olume 1 (Long and Short Papers), Associatio...

    2019 doi

  12. [21]

    Liu, RoBERTa: A robustly optimized bert pretraining approach, arXiv preprint arXiv:1907.11692 364 (2019)

    Y . Liu, RoBERTa: A robustly optimized bert pretraining approach, arXiv preprint arXiv:1907.11692 364 (2019)

    2019 arXiv

  13. [22]

    K. L. Tan, C. P. Lee, K. S. M. Anbananthen, et al., RoBERTa-LSTM: A Hybrid Model for Sentiment Anal- ysis With Transformer and Recurrent Neural Network, IEEE Access 10 (2022) 21517–21525.doi:10.1109/ ACCESS.2022.3152828

    2022

  14. [23]

    Z. Lan, M. Chen, S. Goodman, et al., ALBERT: A Lite BERT for Self-Supervised Learning of Language Rep- resentations, in: International Conference on Learning Representations (ICLR), 2020. URLhttps://openreview.net/forum?id= H1eA7AEtvS

    2020

  15. [24]

    Touvron, T

    H. Touvron, T. Lavril, G. Izacard, et al., LLaMA: Open and Efficient Foundation Language Models, ArXiv abs/2302.13971 (2023)

    2023 arXiv

  16. [25]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, et al., Gpt-4 technical report, arXiv preprint arXiv:2303.08774 (2023)

    2023 arXiv

  17. [26]

    S. Zhu, L. Pan, D. Jian, et al., Overcoming language barriers via machine translation with sparse Mixture- of-Experts fusion of large language models, Inf. Pro- cess. Manage. 62 (3) (Apr. 2025).doi:10.1016/ j.ipm.2025.104078

    2025

  18. [27]

    Houlsby, A

    N. Houlsby, A. Giurgiu, S. Jastrzebski, et al., Parameter- Efficient Transfer Learning for NLP, in: International Conference on Machine Learning, 2019, pp. 2790–2799. URLhttps://proceedings.mlr.press/v97/ houlsby19a.html

    2019

  19. [28]

    W. Y . Lee, J. Saxe, R. Harang, SeqDroid: Obfuscated Android Malware Detection Using Stacked Convolutional and Recurrent Neural Networks, Springer International Publishing, Cham, 2019, pp. 197–210.doi:10.1007/ 978-3-030-13057-2_9

    2019

  20. [29]

    T. A. Tang, D. McLernon, L. Mhamdi, et al., Intrusion Detection in SDN-Based Networks: Deep Recurrent Neu- ral Network Approach, Springer International Publishing, Cham, 2019, pp. 175–195.doi:10.1007/978-3-030- 13057-2_8

    2019 doi

  21. [30]

    A. Diaf, A. A. Korba, N. E. Karabadji, et al., Beyond de- tection: Leveraging large language models for cyber at- tack prediction in iot networks, in: International Confer- ence on Distributed Computing in Smart Systems and the Internet of Things (DCOSS-IoT), IEEE, 2024, pp. 117– 123

    2024

  22. [31]

    Aghaei, X

    E. Aghaei, X. Niu, W. Shadid, et al., SecureBERT: A Domain-Specific Language Model for Cybersecurity, in: Security and Privacy in Communication Networks, 2023, pp. 39–56

    2023

  23. [32]

    Huang, Y

    H. Huang, Y . Wang, SecBERT: Privacy-preserving pre- training based neural network inference system, Neural Networks 172 (2024) 106135

    2024

  24. [33]

    Aghaei, S

    E. Aghaei, S. Jain, P. Arun, et al., SecureBERT 2.0: Ad- vanced Language Model for Cybersecurity Intelligence, arXiv preprint arXiv:2510.00240 (2025)

    2025

  25. [34]

    Bayer, P

    M. Bayer, P. Kuehn, R. Shanehsaz, et al., CySecBERT: A Domain-Adapted Language Model for the Cybersecu- rity Domain, ACM Trans. Priv. Secur. (Apr. 2024).doi: 10.1145/3652594

    2024 doi

  26. [35]

    Ranade, A

    P. Ranade, A. Piplai, A. Joshi, et al., CyBERT: Con- textualized Embeddings for the Cybersecurity Domain, in: 2021 IEEE International Conference on Big Data (Big Data), 2021, pp. 3334–3342.doi:10.1109/ BigData52589.2021.9671824

    2021

  27. [36]

    J. He, M. Vero, G. Krasnopolska, et al., Instruction tuning for secure code generation, in: Proceedings of the 41st International Conference on Machine Learning, ICML’24, JMLR.org, 2024. 18

    2024

  28. [37]

    Shestov, A

    A. Shestov, A. Cheshkov, R. Levichev, et al., Finetuning large language models for vulnerability detection, arXiv preprint arXiv:2401.17010 (2024)

    2024 arXiv

  29. [38]

    Z. Luo, C. Xu, P. Zhao, et al., WizardCoder: Empowering Code Large Language Models with Evol-Instruct 2024 (2024) 27168–27188. URLhttps://proceedings.iclr.cc/ paper_files/paper/2024/file/ 72eba29737f9c3a5a4ce8cdb7b667145-Paper- Conference.pdf

    2024

  30. [39]

    Fayyazi, R

    R. Fayyazi, R. Taghdimi, S. J. Yang, Advancing TTP Analysis: Harnessing the Power of Large Language Mod- els with Retrieval Augmented Generation, in: 2024 An- nual Computer Security Applications Conference Work- shops (ACSAC Workshops), 2024, pp. 255–261.doi: 10.1109/ACSACW652...

    2024

  31. [40]

    Yu et al., Low-Rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recog- nition, in: IEEE ASRU, 2023.doi:10.1109/ ASRU57964.2023.10389632

    2023

  32. [41]

    Tihanyi, M

    N. Tihanyi, M. A. Ferrag, R. Jain, et al., CyberMetric: a benchmark dataset based on retrieval-augmented genera- tion for evaluating LLMs in cybersecurity knowledge, in: 2024 IEEE International Conference on Cyber Security and Resilience (CSR), IEEE, 2024, pp. 296–302

    2024

  33. [42]

    URLhttps://aws.amazon.com/sagemaker/

    Amazon Sagemaker, accessed: 05 May 2025. URLhttps://aws.amazon.com/sagemaker/

    2025

  34. [43]

    URLhttps://www.anthropic.com/news/claude-3- 7-sonnet

    Anthropic Claude 3.7, accessed: 05 May 2025. URLhttps://www.anthropic.com/news/claude-3- 7-sonnet

    2025

  35. [44]

    URLhttps://medium.com/@furkangozukara/what- is-the-dif...d75ac7ec30fa

    Medium – Difference between float16 and bfloat16, ac- cessed: 05 May 2025. URLhttps://medium.com/@furkangozukara/what- is-the-dif...d75ac7ec30fa

    2025

  36. [45]

    Loshchilov, F

    I. Loshchilov, F. Hutter, Decoupled Weight Decay Regu- larization, in: International Conference on Learning Rep- resentations, 2017

    2017

  37. [46]

    Amazon Web Services, SageMaker AI Pricing,https:// aws.amazon.com/sagemaker-ai/pricing/, accessed: 2025-05-26 (n.d.)

    2025

  38. [47]

    M. T. Alam, L. Nguyen, D. Bhusal, et al., CTIBench: a benchmark for evaluating LLMs in cyber threat intel- ligence, in: Proceedings of the 38th International Con- ference on Neural Information Processing Systems, NIPS ’24, Curran Associates Inc., Red Hook, NY , USA, 2025

    2025

  39. [48]

    URLhttps://xuanwuai.github.io/SecEval/

    SecEval Benchmark - GitHub Repository, accessed: 05 May 2025. URLhttps://xuanwuai.github.io/SecEval/

    2025

  40. [49]

    G. Li, Y . Li, G. Wang, et al., SecEval Leaderboard: A Comprehensive Benchmark for Evaluating Cyber- security Knowledge of Foundation Models,https: //xuanwuai.github.io/SecEval/leaderboard.html (2024)

    2024

  41. [50]

    Patterson et al., Carbon emissions and large neural net- work training, arXiv preprint arXiv:2104.10350 (2021)

    2021 arXiv

  42. [51]

    A. Wang, Y . Pruksachatkun, N. Nangia, et al., Super- GLUE: a stickier benchmark for general-purpose lan- guage understanding systems, Curran Associates Inc., Red Hook, NY , USA, 2019

    2019

  43. [52]

    G. Son, H. Lee, S. Kim, et al., KMMLU: Measuring Massive Multitask Language Understanding in Korean, Association for Computational Linguistics, 2025.doi: 10.18653/v1/2025.naacl-long.206

    2025 doi

  44. [53]

    Zellers, A

    R. Zellers, A. Holtzman, Y . Bisk, et al., HellaSwag: Can a Machine Really Finish Your Sentence?, in: Pro- ceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computa- tional Linguistics, Florence, Italy, 2019.doi:10.18653/ v1/...

    2019

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.