Pith. sign in

REVIEW 3 major objections 8 minor 11 cited by

Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct Technical Report

T0 review · 3 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An 8B security model beats far larger ones at CVE-to-CWE mapping.

desk verdict Useful open-weight cyber model and a genuinely informative contamination audit, but the headline RCM SOTA claim is unverified because the authors never decontaminate their own SFT/DPO mixtures. read the letter →

arxiv 2508.01059 v1 pith:WP6QU6TE submitted 2025-08-01 cs.CR cs.AI

Sajana Weerawardhena , Paul Kassianik , Blaine Nelson , Baturay Saglam , Anu Vellore , Aman Priyanshu , Supriti Vijay , Massimo Aufiero
show 9 more authors
Arthur Goldblatt Fraser Burch Ed Li Jianliang He Dhruv Kedia Kojin Oshiba Zhouran Yang Yaron Singer Amin Karbasi
This is my paper
classification cs.CRcs.AI
keywords cybersecurityassistantinstructiontuningcyberthreatintelligenceCWEmappingbenchmarkcontaminationdirectpreferenceoptimizationdomain-adaptedLLMCTIBench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This technical report argues that a compact, domain-adapted language model can be a practical general-purpose cybersecurity assistant: Foundation-Sec-8B-Instruct, an 8B-parameter model, is claimed to outperform Llama 3.1-70B-Instruct and GPT-4o-mini on CTIBench-RCM, the vulnerability root-cause mapping benchmark, while roughly matching GPT-4o-mini on other security and instruction-following tasks. The authors attribute this to starting from a base model continually pre-trained on a large cybersecurity corpus, then applying supervised fine-tuning and direct preference optimization without trying to add new domain knowledge during post-training. They also present a layered contamination-detection analysis of open post-training datasets and show that one widely used dataset, Primus-Reasoning, contains substantial CTIBench items. If the results hold, the recipe points to an efficient path for building capable security assistants without relying on closed frontier models.

What carries the argument

The load-bearing object is the model itself: Foundation-Sec-8B-Instruct, an 8B-parameter chat model built by continual pretraining of Llama 3.1-8B on roughly five billion cybersecurity tokens, producing the Foundation-Sec-8B base, followed by supervised fine-tuning and direct preference optimization on a diverse mix. The mechanism carrying the argument is the post-training recipe's division of labor: pre-training supplies domain knowledge, while SFT and DPO add instruction-following and preference alignment, and deliberately avoiding new knowledge during post-training is what the authors say protects the model's cybersecurity evaluations from dilution and keeps the benchmark comparisons meaningful. The supporting machinery includes a layered decontamination pipeline, combining exact-match detection, 8-gram overlap, embedding similarity with an elevated cosine threshold, and LLM-as-a-judge verification, used to audit public post-training datasets and validate the benchmarks.

What would settle it

Run the paper's own contamination-finding tools on the proprietary instruction-tuning and preference data, and look specifically for CTIBench-RCM items; if any are found at the similarity thresholds the paper uses, the claimed 0.692 score is no longer a fair comparison. A cleaner version would train a fresh model on decontaminated data and see if the edge over GPT-4o-mini survives.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that instruction-tuning a domain-specialized base model, rather than scaling up parameters or injecting new facts during post-training, can put an 8B model at the top of a challenging cyber threat intelligence benchmark. Foundation-Sec-8B-Instruct scores 0.692 on CTIBench-RCM, above GPT-4o-mini (0.655) and Llama 3.1-70B-Instruct (0.623), and it is within error bars of the best models on CTIBench-MCQA. The same model matches or beats Llama 3.1-8B-Instruct on instruction following while retaining most of its base model's cybersecurity knowledge, with only mild degradation on knowledge tasks. A secondary finding is that its three-stage contamination pipeline correctly identifies a known leak in Primus-Reasoning (79–91% of CTIBench-RCM items) while finding low overlap in other public datasets, which the authors present as evidence that their evaluation setup and detection tools are sound.

Load-bearing premise

The reported scores rest on the assumption that none of the test questions appeared in the model's own training data, and the paper checks only public datasets, not the mixture it actually used.

Editorial extensions

If this is right

  • A relatively small open model can beat models 8–9 times its size on at least one real cyber threat intelligence task, so scaling parameters is not the only route to domain competence.
  • Domain-adapted models can gain instruction-following and preference alignment without catastrophic forgetting of cybersecurity knowledge, and the comparison to Foundation-Sec-8B shows negligible drops.
  • The CTIBench-RCM result has a direct practical meaning: an 8B model can reliably map CVE descriptions to CWE identifiers, a task that analysts currently often do by hand.
  • Public post-training corpora vary sharply in contamination level, and the Primus-Reasoning case shows that even widely used cybersecurity datasets can encode benchmark answers.
  • Deploying such a model for safety-sensitive security work will require an external guardrail, since the model without a filter rejects only 92% of harmful prompts and with LlamaGuard reaches 99%.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the central claim is right, the biggest remaining threat to it is unmeasured contamination in the proprietary SFT and DPO mixture: the paper audits only public datasets, and its own pipeline should be run on the exact data behind Foundation-Sec-8B-Instruct.
  • A testable prediction follows from the paper's division-of-labor thesis: a model trained with less domain pre-training or with heavy cybersecurity data injected during SFT should show worse RCM performance and more hallucination on knowledge benchmarks.
  • The contamination pipeline's sensitivity to thresholds suggests the reported low overlaps for other datasets are lower bounds rather than exact point estimates, because semantic-similarity metrics on security text can overcount when domain vocabulary overlaps by default.
  • The paper's comparison to a base model is clouded by format differences between few-shot and chat prompting, so the claim of negligible knowledge loss is best read as an upper bound on retention rather than a precise measurement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. The paper presents Foundation-Sec-8B-Instruct, an instruction-tuned cybersecurity LLM built on the authors' previously released Foundation-Sec-8B base model. The report describes a post-training pipeline combining supervised fine-tuning and direct preference optimization, evaluates the model on six security benchmarks and seven general-purpose benchmarks, and includes safety and persona-adaptation analyses. The headline claims are state-of-the-art performance on CTIBench-RCM (0.692 ± 0.005, beating GPT-4o-mini and Llama 3.1-70B-Instruct) and competitiveness with GPT-4o-mini on cyber threat intelligence and instruction-following tasks.

Significance. If the headline results hold, the paper demonstrates that a domain-adapted 8B-parameter model can outperform much larger general-purpose models on a specific cyber threat intelligence task, which is practically significant for efficient deployment in security workflows. The evaluation protocol is generally sound: security results are averaged over 10 trials with reported standard deviations, and general benchmarks use standard settings via OLMES. The paper also contributes a careful contamination analysis of public post-training datasets, a public model release, and a candid discussion of safety limitations. However, the central claim is currently undercut by the absence of a contamination audit of the model's own SFT/DPO mixtures, and by a base-model comparison that excludes the key benchmark. These issues must be addressed before the comparative claims can be considered verified.

major comments (3)
  1. [§4.3, §5.2, §6.2 (Table 4)] The state-of-the-art claim on CTIBench-RCM rests on the untested purity of the proprietary SFT/DPO data. In §4.3, the authors develop a layered contamination detector (exact match, 8-gram overlap, embedding similarity, LLM-as-judge) and validate it by detecting 78.8–91.5% contamination of CTIBench-RCM in Primus-Reasoning. Yet the same detector is never applied to the authors' own post-training mixtures described in §5.2. This is a direct internal-validity threat: §5.2 states that the synthetic data pipeline is built from web-scale cybersecurity corpora, which are the same kinds of sources from which CTIBench-RCM CVE/CWE pairs are drawn, and the paper shows in Table 3 that cybersecurity post-training datasets can contain large fractions of benchmark items. Without an audit of the actual training data (or an explicit verified statement that no CTIBench-like items are present), the 0.692±0.005 score in Table 4 cannot be interpreted as generalization rather than memorization. I ask the authors to run their own detector on the SFT and DPO mixtures and report the results, including the LLM-verified estimates.
  2. [§6.5, Figure 4] The comparison to Foundation-Sec-8B excludes exactly the benchmark for which the SOTA claim is made. §6.5 compares the instruct model to the base model only on CyberMetric-500, CTIBench-MCQA, and SecBench, explicitly excluding CTIBench-RCM because the base model cannot follow the short-answer format. Moreover, Table 4 does not list Foundation-Sec-8B at all. As a result, the reader cannot tell whether the RCM score reflects knowledge acquired in pre-training or contamination introduced during post-training. I acknowledge the authors' stated difficulty in few-shot evaluating a base model on RCM, but some control is needed—for example, reporting the base model's best-effort RCM score with a clear caveat, or evaluating the instruct model on a withheld, uncontaminated RCM split—before the SOTA assertion is supportable.
  3. [Abstract; §8 Conclusion; Table 4] The abstract and conclusion overstate the comparative results. Table 4 shows that Foundation-Sec-8B-Instruct outperforms Llama 3.1-8B-Instruct on only two of six security benchmarks (CTIBench-RCM +24.03%, CTIBench-MCQA +4.40%) and is worse on the remaining four (CTIBench-VSP −1.67%, CyberMetric-500 −2.01%, SecBench −5.21%, SecEval −2.50%). The abstract's claim that the model 'outperforms Llama 3.1-8B-Instruct on a range of cybersecurity tasks' is therefore inaccurate. Similarly, the conclusion's 'best-in-class' framing is not supported by the full table. The summary statements should be revised to present the mixed results honestly, and the paper should avoid cherry-picking the two positive benchmarks.
minor comments (8)
  1. [§6.3] The word 'demonstraing' should be 'demonstrating'.
  2. [§6.5] The word 'Therefor' should be 'Therefore'.
  3. [Throughout] The model is inconsistently named; the title uses 'Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct' while the body uses 'Foundation-Sec-8B-Instruct'. Please unify the naming.
  4. [§5.2] The SFT/DPO data composition, dataset sizes, and hyperparameters are not reported; this limits reproducibility and prevents the contamination audit from being repeated by others.
  5. [§4.3, Appendix E] The contamination detector is validated on a positive control (Primus-Reasoning) but not on a known-clean dataset, so the false-positive rate of the embedding-similarity stage is unknown; the authors' own caveat that thresholds are tuned makes this absence of a negative control important.
  6. [§F] PersonaGym results are reported on 50 of 200 personas with no description of how the subset was selected; the subset choice could bias the comparison against the published baselines.
  7. [Table 7] For MATH, the metric 'flex exact match' is not defined in the main text; please define it or provide a more precise citation to the Tülu 3 definition.
  8. [Tables 9, 10] The labels 'CTI-ATE', 'CTI-MCQA', 'CTI-RCM', 'CTI-TAA', and 'CTI-VSP' do not match the 'CTIBench' names used elsewhere; please unify the nomenclature for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central claims are empirical benchmark comparisons, not derivations from fitted inputs or self-citation chains.

full rationale

The paper's central claims are comparative benchmark results (e.g., CTIBench-RCM, CTIBench-MCQA, AlpacaEval 2) obtained by evaluating a released model against external baselines. No equation in the paper derives a prediction from a fitted parameter, and no claimed result is defined in terms of the quantity it purports to predict. The base model report [36] is cited for evaluation setup and design continuity, but this is standard self-citation and is not load-bearing for the empirical comparisons; the benchmarks themselves are external (CTIBench, CyberMetric, SecBench, SecEval, IFEval, etc.), and the baseline numbers are measured, not imported. The contamination analysis in Section 4.3 is a data-quality audit, not a derivation: it validates the detection framework against the known-contaminated Primus-Reasoning dataset and reports leakage levels in public post-training sets, but it does not rename a fitted quantity as a prediction. The absence of a contamination audit on the authors' proprietary SFT/DPO mixtures is a genuine internal-validity limitation, and Section 6.5's exclusion of CTIBench-RCM from the base-vs-instruct comparison weakens the interpretation of the SOTA claim, but these are validity concerns, not circularity. The paper does not invoke a uniqueness theorem, does not smuggle an ansatz in via self-citation, and does not rename a known result. Under the stated rules, a contamination risk without demonstrated leakage is not a circularity argument, so the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim is empirical benchmark performance, so there are no fitted parameters or invented entities. The main assumptions are that the benchmarks are valid proxies, the base model is sound, and the proprietary training data is free of benchmark contamination. The last assumption is the most fragile because it is not checked.

assumptions (3)
  • domain assumption Benchmark scores reported under the OLMES harness and the custom MMLU framework accurately reflect the underlying capabilities.
    The paper relies on these evaluations as ground truth for all comparative claims.
  • domain assumption The base model Foundation-Sec-8B was trained and evaluated as described in the prior technical report [36] and is a valid starting point.
    The instruct model's performance is inherited from this base; the paper does not re-verify the base's training details.
  • ad hoc to paper The proprietary SFT and DPO dataset used for instruction-tuning and preference alignment does not contain evaluation benchmark items.
    The contamination analysis in Section 4.3 covers only public datasets like Tülu, Primus-Reasoning, and Alpaca; the authors' own data mixture is never examined for leakage.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct Technical Report." pith.science (2026). https://pith.science/paper/WP6QU6TE

@misc{pith2026250801059,
  author       = {Pith},
  title        = {Pith review of: Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct Technical Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WP6QU6TE}},
  note         = {Machine review of arXiv:2508.01059}
}
read the original abstract

Large language models (LLMs) have shown remarkable success across many domains, yet their integration into cybersecurity applications remains limited due to a lack of general-purpose cybersecurity data, representational complexity, and safety and regulatory concerns. To address this gap, we previously introduced Foundation-Sec-8B, a cybersecurity-focused LLM suitable for fine-tuning on downstream tasks. That model, however, was not designed for chat-style interactions or instruction-following. In this report, we release Foundation-Sec-8B-Instruct: a model specifically trained for general-purpose cybersecurity dialogue. Built on Foundation-Sec-8B, it combines domain-specific knowledge with instruction-following, conversational capabilities, and alignment with human preferences to produce high-quality, relevant responses. Comprehensive evaluations show that Foundation-Sec-8B-Instruct outperforms Llama 3.1-8B-Instruct on a range of cybersecurity tasks while matching its instruction-following performance. It is also competitive with GPT-4o-mini on cyber threat intelligence and instruction-following tasks. We envision Foundation-Sec-8B-Instruct becoming an indispensable assistant in the daily workflows of cybersecurity professionals. We release the model publicly at https://huggingface.co/fdtn-ai/Foundation-Sec-8B-Instruct.

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. FAPO: Fully Automated Prompt Optimization of Multi-Step LLM Pipelines

    cs.SE 2026-06 unverdicted novelty 7.0 of 10

    FAPO automates LLM pipeline optimization via iterative diagnosis and prompt-or-structure edits, beating GEPA baseline by +14.1 pp mean across 18 comparisons and +33.8 pp when structural changes occur.

  2. Trust Me, Import This: Dependency Steering Attacks via Malicious Agent Skills

    cs.CR 2026-05 unverdicted novelty 7.0 of 10

    Malicious Skills induce coding agents to hallucinate and import attacker-controlled packages at high rates while evading detection.

  3. Antares: Foundation Models for Agentic Vulnerability Localization

    cs.CR 2026-08 conditional novelty 6.0 of 10

    Antares-3B, a 3B model trained with SFT plus GRPO, matches GPT-5.5 on repository-scale vulnerability localization at roughly 1/100th the inference cost.

  4. Supervised Fine-Tuning vs. In-Context Learning: An Equilibrium Analysis of LLM Personalization under Congestion

    cs.LG 2026-07 conditional novelty 6.0 of 10

    In a linear model of LLM personalization with shared compute, SFT beats ICL above a coverage-dependent signal-to-noise threshold, congestion can reverse that ranking, and adding SFT never reduces platform profit.

  5. Test-Time Detoxification without Training or Learning Anything

    cs.CL 2026-01 conditional novelty 6.0 of 10

    A few zeroth-order gradient steps on prompt embeddings, using only forward evaluations and a toxicity scorer, reduce toxic LLM outputs while preserving fluency.

  6. Uncovering Vulnerabilities of LLM-Assisted Cyber Threat Intelligence

    cs.CR 2025-09 conditional novelty 6.0 of 10

    LLMs assisting cyber threat intelligence fail mainly due to spurious correlations, contradictory knowledge, and constrained generalization that stem from the threat landscape itself.

  7. TACTIC-KG: Toward Small Agent Teams for Cyber Threat Intelligence Knowledge Graph Construction

    cs.CR 2026-07 conditional novelty 5.5 of 10

    Specialized 3B–8B LLM agents for extraction, typing, verification, and curation outperform much larger monolithic ICL models on CTI knowledge-graph construction.

  8. Inherited Circuits, Learned Semantics: How Fine-Tuning Creates Evasion Vulnerabilities Invisible to Standard Evaluation

    cs.CR 2026-06 unverdicted novelty 5.0 of 10

    Fine-tuning security LLMs specializes inherited classification circuits into token-level indicators that preserve canonical accuracy but fail under behavior-preserving transformations like aliasing and case mutation.

  9. LLM Evolution as an Industry-Scale Ecosystem: A Lifecycle Perspective on Continual Learning

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    The paper reformulates industrial continual learning for LLMs as a closed-loop ecosystem problem, identifies three core challenges, and organizes solutions around five lifecycle design principles.

  10. Toward Cybersecurity-Expert Small Language Models

    cs.CL 2025-10 conditional novelty 5.0 of 10

    A family of 4B–20B cybersecurity models fine-tuned on an enriched, expert-steered reasoning dataset matches or beats larger frontier models on core CTI benchmarks.

  11. Threat Modelling using Domain-Adapted Language Models: Empirical Evaluation and Insights

    cs.CR 2026-05 unverdicted novelty 3.0 of 10

    Domain-adapted LLMs and SLMs do not consistently outperform general models on STRIDE threat classification for 5G, with decoding strategies and model scale affecting validity but gains remaining insufficient for reliable use.

Reference graph

Works this paper leans on

94 extracted references · 32 canonical work pages · cited by 11 Pith papers

  1. [1]

    https://www.first.org/cvss/v3.1/specification-document

    Common vulnerability scoring system v3.1: Specification document. https://www.first.org/cvss/v3.1/specification-document

  2. [2]

    Phi-4 technical report

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905, 2024

    arXiv 2024

  3. [3]

    CTIBench: A benchmark for evaluating LLMs in cyber threat intelligence

    Md Tanvirul Alam, Dipkamal Bhusal, Le Nguyen, and Nidhi Rastogi. CTIBench: A benchmark for evaluating LLMs in cyber threat intelligence. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openrevi ew.net/forum?id=iJAOpsXo2I

    2024

  4. [4]

    Cysecbert: A domain- adapted language model for the cybersecurity domain, 2022

    Markus Bayer, Philipp Kuehn, Ramin Shanehsaz, and Christian Reuter. Cysecbert: A domain- adapted language model for the cybersecurity domain, 2022. URL https://arxiv.org/abs/2212 .02974

    2022

  5. [5]

    Llama-nemotron: Efficient reasoning models

    Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El-Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, et al. Llama-nemotron: Efficient reasoning models. arXiv preprint arXiv:2505.00949, 2025

    arXiv 2025

  6. [6]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  7. [7]

    Common attack pattern enumerations and classifications (capec)

    CAPEC. Common attack pattern enumerations and classifications (capec). https://capec.mitr e.org/, 2024. Available at https://capec.mitre.org/

    2024

  8. [8]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

Show all 94 references
  1. [9]

    Meditron-70b: Scaling medical pretraining for large language models

    Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Ma- toba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, et al. Meditron-70b: Scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16...

    2023 arXiv

  2. [10]

    Kenderdine, J

    Steve Christey, J. Kenderdine, J. Mazella, and B. Miles. Common weakness enumeration. Technical report, The MITRE Corporation, 2013. URL https://cwe.mitre.org/

    2013

  3. [11]

    Scaling instruction-finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53, 2024

    2024

  4. [12]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  5. [13]

    Saullm-54b & saullm-141b: Scaling up domain adaptation for the legal domain

    Pierre Colombo, Telmo Pires, Malik Boudiaf, Rui Melo, Dominic Culver, Etienne Malaboeuf, Gabriel Hautreux, Johanne Charpentier, and Michael Desa. Saullm-54b & saullm-141b: Scaling up domain adaptation for the legal domain. Advances in Neural Information Processing Systems, 37:...

    2024

  6. [14]

    Saullm-7b: A pioneering large language model for law

    Pierre Colombo, Telmo Pessoa Pires, Malik Boudiaf, Dominic Culver, Rui Melo, Caio Corro, Andre FT Martins, Fabrizio Esposito, Vera Lúcia Raposo, Sofia Morgado, et al. Saullm-7b: A pioneering large language model for law. arXiv preprint arXiv:2403.03883, 2024

    2024 arXiv

  7. [15]

    AI for DevSecOps, DeepHat

    DeepHat. AI for DevSecOps, DeepHat. https://www.deephat.ai/, 2025. Accessed: 2025-7-29

    2025

  8. [16]

    Attacker: to- wards enhancing cyber-attack attribution with a named entity recognition dataset

    Pritam Deka, Sampath Rajapaksha, Ruby Rani, Amirah Almutairi, and Erisa Karafili. Attacker: to- wards enhancing cyber-attack attribution with a named entity recognition dataset. InInternational Conference on Web Information Systems Engineering, pages 255–270. Springer, 2024

    2024

  9. [17]

    Investigating data contamination in modern benchmarks for large language models

    Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. Investigating data contamination in modern benchmarks for large language models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter o...

    2024 doi

  10. [18]

    Length-controlled alpacaeval: A simple way to debias automatic evaluators

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024

    2024 arXiv

  11. [19]

    General Data Protection Regulation (GDPR)

    European Union. General Data Protection Regulation (GDPR). https://eur-lex.europa.eu/l egal-content/EN/TXT/PDF/?uri=CELEX:32016R0679 , 2024. Available at https://eur-lex.e uropa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32016R0679

    2024

  12. [20]

    Does data contamination detection work (well) for LLMs? a survey and evaluation on detection assumptions

    Yujuan Fu, Ozlem Uzuner, Meliha Yetisgen, and Fei Xia. Does data contamination detection work (well) for LLMs? a survey and evaluation on detection assumptions. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors,Findings of the Association for Computational Linguistics: NAACL...

    2025 doi

  13. [21]

    Does fine-tuning llms on new knowledge encourage hallucinations? arXiv preprint arXiv:2405.05904, 2024

    Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning llms on new knowledge encourage hallucinations? arXiv preprint arXiv:2405.05904, 2024

    2024 arXiv

  14. [22]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 14

    2024 arXiv

  15. [23]

    Olmes: A standard for language model evaluations

    Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi. Olmes: A standard for language model evaluations. arXiv preprint arXiv:2406.08446, 2024

    2024 arXiv

  16. [24]

    Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models

    Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in Neural...

    2023

  17. [25]

    Generative active adaptation for drifting and imbalanced network intrusion detection

    Ragini Gupta, Shinan Liu, Ruixiao Zhang, Xinyue Hu, Pranav Kommaraju, Xiaoyang Wang, Hadjer Benkraouda, Nick Feamster, and Klara Nahrstedt. Generative active adaptation for drifting and imbalanced network intrusion detection. arXiv [cs.NI], March 2025

    2025

  18. [26]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=d7KBjmI3GmQ

    2021

  19. [27]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  20. [28]

    Universal language model fine-tuning for text classification

    Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146, 2018

    2018 arXiv

  21. [29]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  22. [30]

    Llama guard: Llm-based input-output safeguard for human-ai conversations

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674, 2023

    2023 arXiv

  23. [31]

    Smith, Iz Beltagy, and Hannaneh Hajishirzi

    Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. Camels in a changing climate: Enhancing lm adaptation with tulu 2. 2023

    2023

  24. [32]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Re- nard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thoma...

    2023 arXiv

  25. [33]

    Secbench: A comprehensive multi-dimensional benchmarking dataset for llms in cybersecurity

    Pengfei Jing, Mengyun Tang, Xiaorong Shi, Xing Zheng, Sen Nie, Shi Wu, Yong Yang, and Xiapu Luo. Secbench: A comprehensive multi-dimensional benchmarking dataset for llms in cybersecurity. arXiv preprint arXiv:2412.20787, 2024

    2024 arXiv

  26. [34]

    Guide to cyber threat information sharing

    Chris Johnson, Lee Badger, David Waltermire, Julie Snyder, and Clem Skorupka. Guide to cyber threat information sharing. Technical Report 800-150, National Institute of Standards and Technol- ogy (NIST), 2016. URL https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.S P....

    2016

  27. [35]

    Bag of tricks for efficient text classification

    Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759, 2016

    2016 arXiv

  28. [36]

    Llama-3.1-foundationai- securityllm-base-8b technical report

    Paul Kassianik, Baturay Saglam, Alexander Chen, Blaine Nelson, Anu Vellore, Massimo Aufiero, Fraser Burch, Dhruv Kedia, Avi Zohary, Sajana Weerawardhena, et al. Llama-3.1-foundationai- securityllm-base-8b technical report. arXiv preprint arXiv:2504.21039, 2025. 15

    2025 arXiv

  29. [37]

    T\" ulu 3: Pushing frontiers in open language model post-training

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T\" ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024

    2024 arXiv

  30. [38]

    Platypus: Quick, cheap, and powerful refinement of llms

    Ariel N Lee, Cole J Hunter, and Nataniel Ruiz. Platypus: Quick, cheap, and powerful refinement of llms. arXiv preprint arXiv:2308.07317, 2023

    2023 arXiv

  31. [39]

    Seceval: A comprehensive benchmark for evaluating cybersecurity knowledge of foundation models

    Guancheng Li, Yifeng Li, Wang Guannan, Haoyu Yang, and Yang Yu. Seceval: A comprehensive benchmark for evaluating cybersecurity knowledge of foundation models. https://github.com/XuanwuAI/SecEval, 2023

    2023

  32. [40]

    Openorca: An open dataset of gpt augmented flan reasoning traces

    Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces. https://https://huggingf ace.co/datasets/Open-Orca/OpenOrca, 2023

    2023

  33. [41]

    Secqa: A concise question-answering dataset for evaluating large language models in computer security

    Zefang Liu. Secqa: A concise question-answering dataset for evaluating large language models in computer security. arXiv preprint arXiv:2312.15838, 2023

    2023 arXiv

  34. [42]

    Harmbench: A standardized evaluation framework for automated red teaming and robust refusal

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249, 2024

    2024 arXiv

  35. [43]

    Innocent Mbona and Jan H. P. Eloff. Data sets for cyber security machine learning models: A methodological approach. In Proceedings of the 9th International Conference on Internet of Things, Big Data and Security (IoTBDS), pages 149–156. SCITEPRESS, 2024

    2024

  36. [44]

    Reducing conversational agents’ overconfidence through linguistic calibration

    Sabrina J Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing conversational agents’ overconfidence through linguistic calibration. Transactions of the Association for Computational Linguistics, 10:857–872, 2022

    2022

  37. [45]

    Large language models in cybersecurity: State-of-the-art

    Farzad Nourmohammadzadeh Motlagh, Mehrdad Hajizadeh, Mehryar Majd, Pejman Najafi, Feng Cheng, and Christoph Meinel. Large language models in cybersecurity: State-of-the-art. arXiv preprint arXiv:2402.00891, 2024

    2024 arXiv

  38. [46]

    Learning word representation for the cyber security vulnerability domain

    Sara Mumtaz, Carlos Rodriguez, Boualem Benatallah, Mortada Al-Banna, and Shayan Zamanirad. Learning word representation for the cyber security vulnerability domain. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–8, 2020. doi: 10.1109/IJCNN48605.2020...

    2020

  39. [47]

    Cross-domain language modeling: An empirical investigation

    Vincent Nguyen, Sarvnaz Karimi, Maciej Rybinski, and Zhenchang Xing. Cross-domain language modeling: An empirical investigation. In Afshin Rahimi, William Lane, and Guido Zuccon, editors, Proceedings of the 19th Annual Workshop of the Australasian Language Technology Associati...

    2021

  40. [48]

    2 olmo 2 furious

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656, 2024

    2024 arXiv

  41. [49]

    ChatGPT [large language model]

    OpenAI. ChatGPT [large language model]. https://chat.openai.com/chat , 2023. Accessed: 2025-04-23

    2023

  42. [50]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, 16 Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, ...

    2022 arXiv

  43. [51]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:2773...

    2022

  44. [52]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    2023

  45. [53]

    Code llama: Open foundation models for code

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  46. [54]

    Personagym: Evaluating persona agents and llms

    Vinay Samuel, Henry Peng Zou, Yue Zhou, Shreyas Chaudhari, Ashwin Kalyan, Tanmay Rajpurohit, Ameet Deshpande, Karthik Narasimhan, and Vishvak Murahari. Personagym: Evaluating persona agents and llms. arXiv preprint arXiv:2407.18416, 2024

    2024 arXiv

  47. [55]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  48. [56]

    Lily-cybersecurity-7b-v0.2: A cybersecurity assistant

    segolilylabs. Lily-cybersecurity-7b-v0.2: A cybersecurity assistant. Hugging Face model repository,

  49. [57]

    Toward expert-level medical question answering with large language models

    Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. Toward expert-level medical question answering with large language models. Nature Medicine, pages 1–8, 2025

    2025

  50. [58]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.Transactions on machine learning research, 2023

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.Transactions on machi...

    2023

  51. [59]

    Strom, Andy Applebaum, Doug P

    Blake E. Strom, Andy Applebaum, Doug P. Miller, Kathryn C. Nickels, Adam G. Pennington, and Cody B. Thomas. MITRE ATT&CK: Design and Philosophy. Technical report, The MITRE Corporation, 2018. URL https://attack.mitre.org/resources/enterprise-introduction/

    2018

  52. [60]

    Bertscore is unfair: On social bias in language model-based metrics for text generation, 2022

    Tianxiang Sun, Junliang He, Xipeng Qiu, and Xuanjing Huang. Bertscore is unfair: On social bias in language model-based metrics for text generation, 2022. URL https://arxiv.org/abs/2210 .07626

    2022

  53. [61]

    Challenging big-bench tasks and whether chain-of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022

    arXiv

  54. [62]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023. 17

    2023

  55. [63]

    Gemma 3 technical report

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025

    2025 arXiv

  56. [64]

    Cve ® – common vulnerabilities and exposures program

    The MITRE Corporation. Cve ® – common vulnerabilities and exposures program. https: //cve.mitre.org/, 2025

    2025

  57. [65]

    Cybermetric: A benchmark dataset based on retrieval-augmented generation for evaluating llms in cybersecurity knowledge

    Norbert Tihanyi, Mohamed Amine Ferrag, Ridhi Jain, Tamas Bisztray, and Merouane Debbah. Cybermetric: A benchmark dataset based on retrieval-augmented generation for evaluating llms in cybersecurity knowledge. In 2024 IEEE International Conference on Cyber Security and Resilien...

    2024

  58. [66]

    Cybermetric: a benchmark dataset based on retrieval-augmented generation for evaluating llms in cybersecurity knowledge

    Norbert Tihanyi, Mohamed Amine Ferrag, Ridhi Jain, Tamas Bisztray, and Merouane Debbah. Cybermetric: a benchmark dataset based on retrieval-augmented generation for evaluating llms in cybersecurity knowledge. In 2024 IEEE International Conference on Cyber Security and Resilien...

    2024

  59. [67]

    A comprehensive survey of hallucination mitigation techniques in large language models

    S M Towhidul Islam Tonmoy, S M Mehedi Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. A comprehensive survey of hallucination mitigation techniques in large language models. arXiv [cs.CL], January 2024

    2024

  60. [68]

    Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks

    Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. arXiv pre...

    2022 arXiv

  61. [69]

    Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference

    Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. Smarter, better, faster, longer: A modern bidirectional e...

    2024

  62. [70]

    Wizardlm: Empowering large language models to follow complex instructions

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023

    2023 arXiv

  63. [71]

    Benchmark data contamination of large language models: A survey, 2024

    Cheng Xu, Shuhao Guan, Derek Greene, and M-Tahar Kechadi. Benchmark data contamination of large language models: A survey, 2024. URL https://arxiv.org/abs/2406.04244

    2024 arXiv

  64. [72]

    Large language models for cyber security: A systematic literature review

    HanXiang Xu, ShenAo Wang, Ningke Li, Kailong Wang, Yanjie Zhao, Kai Chen, Ting Yu, Yang Liu, and HaoYu Wang. Large language models for cyber security: A systematic literature review. arXiv preprint arXiv:2405.04760, 2024

    2024

  65. [73]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  66. [74]

    Gonzalez, and Ion Stoica

    Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and Ion Stoica. Rethinking benchmark and contamination for language models with rephrased samples, 2023. URL https: //arxiv.org/abs/2311.04850

    2023 arXiv

  67. [75]

    Primus: A pioneering collection of open-source datasets for cybersecurity llm training

    Yao-Ching Yu, Tsun-Han Chiang, Cheng-Wei Tsai, Chien-Ming Huang, and Wen-Kwang Tsao. Primus: A pioneering collection of open-source datasets for cybersecurity llm training. arXiv preprint arXiv:2502.11191, 2025. 18

    2025

  68. [76]

    Cybench: A framework for evaluating cybersecurity capabilities and risks of language models

    Andy K Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin W Lin, Eliot Jones, Gashon Hussein, Samantha Liu, Donovan Jasper, et al. Cybench: A framework for evaluating cybersecurity capabilities and risks of language models. arXiv preprint arXiv:2408.08926, 2024

    2024 arXiv

  69. [77]

    When llms meet cybersecurity: A systematic literature review, 2024

    Jie Zhang, Haoyu Bu, Hui Wen, Yongji Liu, Haiqiang Fei, Rongrong Xi, Lun Li, Yun Yang, Hongsong Zhu, and Dan Meng. When llms meet cybersecurity: A systematic literature review, 2024. URL https://arxiv.org/abs/2405.03644

    2024 arXiv

  70. [78]

    When llms meet cybersecurity: A systematic literature review.Cybersecurity, 8(1):55, 2025

    Jie Zhang, Haoyu Bu, Hui Wen, Yongji Liu, Haiqiang Fei, Rongrong Xi, Lun Li, Yun Yang, Hongsong Zhu, and Dan Meng. When llms meet cybersecurity: A systematic literature review.Cybersecurity, 8(1):55, 2025

    2025

  71. [79]

    Lima: Less is more for alignment

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006–55021, 2023

    2023

  72. [80]

    Instruction-following evaluation for large language models

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023

    2023 arXiv

  73. [81]

    output as a JSON object

    Qin Zhu, Qinyuan Cheng, Runyu Peng, Xiaonan Li, Ru Peng, Tengxiao Liu, Xipeng Qiu, and Xuanjing Huang. Inference-time decontamination: Reusing leaked benchmarks for large language model evaluation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Findings of the A...

    2024 doi

  74. [83]

    Governance, Risk, and Compliance (GRC) • Risk Management & Security Strategy • Compliance and Regulations (e.g., GDPR, HIPAA) • Security Frameworks (e.g., NIST CSF, ISO 27001) • Security Policies & Architecture

  75. [84]

    Network, Infrastructure, and Endpoint Security • Perimeter and Network Security (Firewalls, VPNs, Wireless) • Endpoint Protection & MDM • IoT and OT/ICS Security • Mobile Security

  76. [85]

    Application and Software Security • Secure Software Development (DevSecOps) • Application & API Security • Vulnerability Management & Penetration Testing • Software Supply Chain Security (SBOM, third-party risk) 23

  77. [86]

    Cloud and Data Security • Cloud Security Architecture & Tools • Identity and Access Management (IAM, PAM) • Data Loss Prevention & Privacy (DLP, encryption) • Cloud Compliance & Shared Responsibility Model

  78. [87]

    Identity, Access, and Zero Trust • Authentication & Authorization (MFA, SSO, RBAC) • Identity Governance & Lifecycle • Zero Trust Architecture • Privileged Access Controls

  79. [88]

    Security Operations and Monitoring (SecOps) • SIEM, SOC, and Log Management • Security Automation & SOAR • Detection Engineering • Operational Resilience & Monitoring

  80. [89]

    Threat Intelligence and Incident Response • Threat Detection, Analysis & Hunting • Threat Intelligence Platforms & IOCs • Advanced Persistent Threats (APTs) • Malware Techniques • Incident Response, Recovery & Digital Forensics

  81. [90]

    Cryptography and Secure Communications • Cryptographic Algorithms & PKI • Key Management • Post-Quantum Cryptography • Secure Protocols and Encryption Practices

  82. [91]

    Security Awareness and Human Risk • Social Engineering Techniques (Phishing, Pretexting) • Insider Threat Management • Security Awareness Training • Behavioral Risk Analysis

  83. [92]

    sandbox”, “policy

    Emerging Technologies and Future Threats • AI/ML & LLM Security (adversarial ML, prompt injection) • Quantum Security Threats • Deepfakes & Synthetic Media • Nation-State Threats and Geopolitical Risk D. Post-Training Data Analysis Here we present further details into our esti...

  84. [93]

    content":

    Average Matches and Average Density are only computed for prompts above the keyword threshold. 25 D.2. Examples of Cybersecurity Content Identified Example Cybersecurity Training Sample from Tülu 3 Source: tulu-3-sft-mixture ID: ai2-adapt-dev/evol_codealpaca_heval_decontaminat...

  85. [94]

    Respond to both names and maintain the identity at all times

    This allows you to be the best cybersecurity assistant in the world. Respond to both names and maintain the identity at all times. The user is a cybersecurity professional trying to accomplish some cybersecurity task. You must help them accomplish their tasks in the most effic...

  86. [2025]

    Fine-tuned on Mistral-7B-Instruct-v0.2 with 22,000 hand-crafted cybersecurity/hacking pairs

    URL https://huggingface.co/segolilylabs/Lily-Cybersecurity-7B-v0.2 . Fine-tuned on Mistral-7B-Instruct-v0.2 with 22,000 hand-crafted cybersecurity/hacking pairs. 7.24B parameters, trained over 5 epochs on an A100 GPU. Apache 2.0 license

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.