Pith. sign in

REVIEW 4 major objections 5 minor 5 cited by

Fine-tuning small language models on an expert-grounded, evidence-enriched cybersecurity dataset lets a 20-billion-parameter model beat frontier rivals on core threat-investigation benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 09:36 UTC pith:SXH72I4Q

load-bearing objection A solid, carefully-reported training pipeline and model family, but the headline claim of frontier-level RCM performance is not yet supported because the paper never checks overlap between SecKnowledge 2.0 and CTIBench-RCM. the 4 major comments →

arxiv 2510.14113 v2 pith:SXH72I4Q submitted 2025-10-15 cs.CL cs.AIcs.CR

Toward Cybersecurity-Expert Small Language Models

Matan Levi , Daniel Ohayon , Ariel Blobstein , Ravid Sagi , Ian Molloy , Yair Allouche This is my paper
classification cs.CL cs.AIcs.CR
keywords small language modelscybersecuritychain-of-thoughtinstruction tuningcyber threat intelligenceroot cause mappingexpert-in-the-loopdata enrichment
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that cybersecurity expertise can be distilled into small language models (4B–20B parameters) without giving up accuracy. It introduces CyberPal 2.0, a model family trained on SecKnowledge 2.0, a dataset built by reformatting and enriching an existing expert-created instruction set with structured reasoning formats and evidence retrieved from security documents and the web. On threat-intelligence knowledge questions the models rank second only to the top closed security model, and on vulnerability-to-weakness root-cause mapping the 20B model ranks first, ahead of GPT-4o, o1, o3-mini, and Sec-Gemini v1. If the results stand, they matter because enterprises could deploy near-frontier security reasoning on-premises with a fraction of the computational cost and without sending sensitive telemetry to external APIs.

Core claim

CyberPal 2.0 demonstrates that large frontier models are not necessary for state-of-the-art performance on core cybersecurity threat-investigation tasks. Using SecKnowledge 2.0 — an enriched instruction dataset in which expert-specified reasoning formats are combined with LLM-guided evidence retrieval and multi-step grounding — the authors fine-tune open-weight base models from 4B to 20B parameters. Their best 20B model ranks first on the root-cause-mapping benchmark, which requires correlating vulnerability records and bug tickets with the correct weakness entries, outperforming GPT-4o, o1, o3-mini, and Sec-Gemini v1; the smallest 4B variant ranks second. Across nine cybersecurity benchmark

What carries the argument

SecKnowledge 2.0 is a three-stage reformatting and enrichment pipeline. First, an expert-in-the-loop framework semi-automatically derives task-specific output formats that specify the reasoning steps a security analyst should follow. Second, an LLM-guided search stage generates and filters queries, retrieves evidence from a vector database or the web, and grounds each reasoning step in fetched documents. Third, the original terse answers from the SecKnowledge seed set are rewritten into structured, evidence-anchored chain-of-thought traces, and an LLM-as-a-judge scores readability and factuality. Training then mixes these long chain-of-thought examples with short fast-response items from the

Load-bearing premise

The benchmark gains reflect transferable security reasoning rather than memorization of overlapping public-source training items, because the evaluation suites and the training dataset draw on the same authoritative security sources and the paper does not quantify that overlap.

What would settle it

Evaluate the 20B model on a held-out set of vulnerability records and bug reports published after the training data cutoff, with expert-verified weakness mappings, and compare against GPT-4o, o1, and Sec-Gemini v1. If its first-place root-cause-mapping ranking falls below frontier models on that out-of-distribution set, the claim that the small model learned transferable security reasoning rather than benchmark-shaped patterns is disconfirmed.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Enterprises can run near-frontier security reasoning on-premises with 4B–20B open-weight models, avoiding the compliance risk of sending sensitive security telemetry to external frontier APIs.
  • On the root-cause-mapping task, the 20B model's first-place result over frontier models suggests that for taxonomy-heavy security reasoning, specialized small models can be more cost-effective than general-purpose scaling.
  • The 8B variant outperforms recent open-source security models by a substantial margin, and 8-bit quantization costs less than 1% average accuracy, making practical deployment on constrained hardware viable.
  • Ablations attribute most of the gains to the SecKnowledge 2.0 enrichment pipeline rather than the base model choice or extra compute, pointing to data quality as the driving factor.
  • Even the 4B model ranks second on root-cause mapping and beats much larger general models on threat-intelligence knowledge, implying strong compression of domain capability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the evaluation benchmarks and the training dataset draw on the same authoritative security sources, a direct test is to measure performance on vulnerability-to-weakness mappings published after the training cut-off; if the ranking collapses there, the claim of transferable reasoning would need revision.
  • The expert-in-the-loop format-definition plus evidence-grounded rewriting recipe is domain-generic; a similar pipeline could plausibly lift small models to frontier-level performance in other hallucination-prone expert fields such as legal or medical reasoning, where structured formats and verifiable evidence matter equally.
  • The paper's choice to fine-tune from base rather than instruction-tuned checkpoints, and to mix long and short reasoning traces, suggests that domain specialization may be more effective when applied early in the post-training stack; one could test whether the same holds for other expert domains.
  • If the first-place root-cause-mapping result replicates across future versions of the benchmark, it would strengthen the case that security operations centers should prefer small specialized models over closed chatbots for structured threat triage, reserving frontier models for open-ended investigation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces CyberPal 2.0, a family of 4B–20B cybersecurity SLMs trained on SecKnowledge 2.0, an enriched chain-of-thought instruction dataset built from SecKnowledge via expert-in-the-loop format definition, LLM-guided search/document grounding, and LLM-as-a-judge screening. The authors report 7–14% average gains over base models, substantial gains over several open-source security models, and claim that on CTIBench-RCM their 20B model ranks first, ahead of GPT-4o, o1, o3-mini, and Sec-Gemini v1, while their 4B model ranks second. Ablations on a 4B model are used to attribute the gains to the SecKnowledge 2.0 enrichment pipeline rather than to the original SecKnowledge data or a vanilla reformatting pipeline. The paper includes additional analyses of base-vs-post-trained initialization, prompt loss, quantization, and an expert-grounded LLM-as-a-judge evaluation.

Significance. If the headline ranking claim is correct, the paper is practically significant: it would show that an open-weight, 20B model fine-tuned on a relatively small, publicly grounded instruction corpus can match or beat much larger closed frontier models on a difficult threat-investigation task. The training recipe is concrete and reproducible in principle, the ablation design is reasonable, the benchmarks are standard in the field, and the LLM-as-a-judge validation with expert-authored questions and grounded documents is a useful addition. The main contribution, however, is not yet established. The training data are built from the same MITRE/CWE/CVE/BRON resources that several evaluation benchmarks use, and no overlap audit is reported; the ablations show per-benchmark regressions that the prose glosses over; and all scores are single-run point estimates with no uncertainty quantification. These are load-bearing issues for the ranking claim, so the paper should not be accepted without substantial revision.

major comments (4)
  1. [§3.1, §4.2, §5.1] The central 'ranking first' claim on CTIBench-RCM is vulnerable to training/evaluation leakage. SecKnowledge 2.0 is generated from seed corpora including BRON, Sigma, ATT&CK, CWE, and CVE-derived resources (§3.1), while CTIBench-RCM asks for CVE/bug-ticket→CWE mapping built from the same public sources (§4.2). Because the models are trained for two epochs on SecKnowledge 2.0, CVE-CWE associations that also appear in the benchmark could be memorized, making the RCM gains an artifact of overlap rather than evidence of transferable threat-investigation reasoning. The paper applies exactly this logic when excluding PRIMUS-Reasoning from CTIBench because its training set was distilled from CTIBench (§5.1), but it never audits SecKnowledge 2.0 for overlap with CTIBench, SecEval, or CyberMetric. I request a quantitative overlap audit (e.g., n-gram or CVE/CWE-ID intersection between training and
  2. [§5.2, Table 6, Fig. 5] The statement that 'our model consistently outperforms' both the SecKnowledge-based model and Baseline Reformatting is contradicted by the ablations in Table 6. On Adversarial CTI, CyberPal-2.0-4B scores 68.03, while the original SecKnowledge model scores 79.86 and Baseline Reformatting scores 76.05; on CTI Detect & Mitigate it scores 64.03, below Baseline Reformatting's 65.51; on CISSP Exams it scores 80.80, below the original SecKnowledge model's 85.35. Even in Table 1, CyberPal-2.0-20B is below its own baseline on CyberMetric-2000 (89.05 vs 90.20). Since the paper attributes the observed gains primarily to SecKnowledge 2.0, these per-benchmark regressions must be reported and discussed rather than hidden behind an average or a 'consistently outperforms' claim in Figure 5. Please provide per-benchmark directional analysis, error bars, and an explanation of the tradeoffs.
  3. [§5.1, Fig. 1] The evidence for the frontier ranking consists of single-run point estimates without standard deviations, confidence intervals, or significance tests, and the exact scores of GPT-4o, o1, o3-mini, and Sec-Gemini v1 are not tabulated in the text; Figure 1 alone is not a sufficient basis for readers to verify a first-place claim. Please add a table with exact scores, model versions and API snapshot dates, the evaluation prompt used for each model, the output-extraction method, and the source of each frontier-model number (independently run vs. taken from a prior report). For the open models, report at least three training seeds or repeated evaluations at nonzero temperature; for closed models, report repeated API calls if the endpoint is nondeterministic.
  4. [§5.1, Table 2] Table 2 compares CyberPal-2.0-8B with open-source security models, but PRIMUS-reasoning is excluded from CTIBench-MCQ and CTIBench-RCM, and its average is computed over the remaining seven benchmarks. This is disclosed, but it makes the 'outperforms all open-source baselines by a substantial margin' claim a comparison over inconsistent benchmark sets. Please report the average both with and without CTIBench, or use a common subset, and state explicitly which benchmarks are included in each average.
minor comments (5)
  1. [§3.3.1] The formal set definition 'S= {T1, ..., TN | Sk=N k=1 Tk =D, Ti ∩T j =∅}' has broken notation; please rewrite it correctly.
  2. [§4.1] The abbreviation LLMaaJ is used without expansion at its first occurrence; please define it as LLM-as-a-Judge.
  3. [Appendix B, Table 3] The model naming is inconsistent: 'CyberPal2.0-8B' here differs from 'CyberPal-2.0-8B' used in Table 1 and elsewhere. Please standardize.
  4. [References] In the Introduction, 'In earlier work Levi et al.' is mentioned without a citation number or year; please add the proper reference at that point.
  5. [General] The paper would benefit from a data/model availability statement. No repository link, dataset release, or model checkpoint link is provided, which makes the 'openness' aspect of the contribution hard to verify.

Circularity Check

1 steps flagged

The headline benchmark claims are not circular as stated; one self-referential data-quality metric is the only concrete circular step.

specific steps
  1. self definitional [Appendix A.2, 'Factuality' criterion and following summary]
    "We prompt the judge with the original answer and the rewritten answer, emphasizing that the original answer is the ground truth, and ask the LLM to provide a score in a scale of 1-10 that determines how factual the rewritten answer with respect to the original answer. ... while maintaining the factuality, reflected by an average factuality score of 9.25."

    The factuality metric used to validate SecKnowledge 2.0 defines the original/seed answer as ground truth. A rewritten answer is therefore 'factual' to the extent that it agrees with the pipeline's input answer, not with an independently verified external source. The reported 9.25 score thus measures self-consistency with the seed data by construction and cannot independently establish that the enriched reasoning traces are more accurate or higher-fidelity. This is a concrete self-referential step, though it is confined to the dataset-quality assessment and is not the direct basis for the headline CTIBench-RCM ranking.

full rationale

The paper's central claim is a benchmark comparison: CyberPal-2.0-20B ranks first on CTIBench-RCM against GPT-4o, o1, o3-mini, and Sec-Gemini v1. That benchmark is external (Alam et al., 2024), and the paper does not state that CTIBench-RCM items appear verbatim in SecKnowledge 2.0. There is a legitimate contamination risk: SecKnowledge 2.0 is built from BRON, Sigma, ATT&CK, and CWE-based seeds, while CTIBench-RCM tests CVE/bug-ticket-to-CWE mapping, and the paper excludes PRIMUS-Reasoning from CTIBench because its training set was distilled from CTIBench without performing an equivalent overlap audit for its own corpus. However, the reviewer instructions require an exhibited specific reduction to score a step as circular; the paper provides no exact overlap count or item-level identity, so the contamination concern remains an unquantified risk rather than a proven circularity. Self-citations to Levi et al. supply SecKnowledge and several auxiliary benchmarks, but the load-bearing headline result is on an independent external benchmark, so the self-citation is not itself load-bearing. The one demonstrable circular step is the LLM-as-judge factuality criterion that treats the original answer as ground truth; this affects the internal data-quality argument but not the external ranking claim. Overall score: 2.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

No new physical or conceptual entities are introduced; the novel items are datasets and trained weights. The central claim rests mainly on the train/eval independence assumption, the evaluation protocol, and the reliability of LLM-as-judge—none of which are externally verified by released artifacts.

free parameters (3)
  • Training recipe hyperparameters = LR 4e-5, warmup 0.15, context 8192, batch 3072, 2 epochs
    Hand-selected in §4.1; not fitted to benchmarks but directly shape the trained models' scores and the claimed 7–14% gains.
  • Retrieval configuration = K=2, Rmax=8, R=2
    Chosen in §3.3.2 to bound context window and lost-in-the-middle effects; affects the quality of grounding and therefore the rewritten training data.
  • Fast-response mixing fraction = ~25% original short instructions
    An author-chosen balance between long CoT and concise responses (§4.1); affects adaptive reasoning and overfitting behavior.
axioms (4)
  • domain assumption Evaluation benchmarks and SecKnowledge 2.0 training data are effectively disjoint, so measured gains reflect transferable security reasoning rather than memorization of near-identical public-source items.
    SecKnowledge 2.0 is built from BRON, Sigma, ATT&CK, and CWE seeds (§3.1); CTIBench, SecEval, and CyberMetric are built from the same authoritative sources (§4.2). No overlap audit is provided, while PRIMUS-Reasoning is excluded from CTIBench for training-set distillation (§5.1).
  • domain assumption Zero-shot CoT prompting with regex answer extraction and zero temperature is a fair protocol across all compared models.
    The evaluation prompt (Figure 11) is tailored to the fine-tuned models' output format; baseline and frontier models with different reasoning styles may be disadvantaged by the fixed CoT wrapper and greedy decoding (§4.3).
  • domain assumption o3 as LLM-as-a-judge with expert grounding approximates human expert preference.
    The paper reports >90% agreement with 30 experts on 115 self-authored questions (Appendix F), but the judge, the questions, and the grounding documents are not released, and the factuality score uses the original answer as ground truth.
  • domain assumption Starting from Qwen3/gpt-oss base checkpoints provides sufficient pretraining for domain specialization.
    The recipe deliberately fine-tunes base models rather than post-trained models; this is supported only by a small internal experiment (Appendix B) and is not independently verified.

pith-pipeline@v1.3.0-alltime-deepseek · 18167 in / 12948 out tokens · 104441 ms · 2026-08-04T09:36:28.848041+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) are transforming everyday applications, yet deployment in cybersecurity lags due to a lack of high-quality, domain-specific models and training datasets. To address this gap, we present CyberPal 2.0, a family of cybersecurity-expert small language models (SLMs) ranging from 4B-20B parameters. To train CyberPal 2.0, we generate an enriched chain-of-thought cybersecurity instruction dataset built with our data enrichment and formatting pipeline, SecKnowledge 2.0, which integrates expert-in-the-loop steering of reasoning formats alongside LLM-driven multi-step grounding, yielding higher-fidelity, task-grounded reasoning traces for security tasks. Across diverse cybersecurity benchmarks, CyberPal 2.0 consistently outperforms its baselines and matches or surpasses various open and closed-source frontier models, while remaining a fraction of their size. On core cyber threat intelligence knowledge tasks, our models outperform almost all tested frontier models, ranking second only to Sec-Gemini v1. On core threat-investigation tasks, such as correlating vulnerabilities and bug tickets with weaknesses, our best 20B-parameter model outperforms GPT-4o, o1, o3-mini, and Sec-Gemini v1, ranking first, while our smallest 4B-parameter model ranks second.

Figures

Figures reproduced from arXiv: 2510.14113 by Ariel Blobstein, Daniel Ohayon, Ian Molloy, Matan Levi, Ravid Sagi, Yair Allouche.

Figure 1
Figure 1. Figure 1: Comparing our models (blue) against frontier models such as Sec-Gemini v1, o1, and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: 3-step reformatting and enrichment pipeline overview - experts define task formats for the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Q&A example from SecKnowledge (green), and our improved answer (orange). [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Domain composition of each data source in the evaluation benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Ablation studies (using 4B parameters models) comparing [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Pairwise comparison results from LLMaaJ (OpenAI’s o3) with expert provided [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: It is important to note that this framework is applicable only once the set of tasks covering [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 7
Figure 7. Figure 7: An example task, specifically the one illustrated in Figure 3. A task consists of a name, a [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 12
Figure 12. Figure 12: Evaluation process — We use OpenAI’s o3 (OpenAI, 2025) as the LLM-as-a-judge. The judge evaluates each answer pair along six dimensions —Contextual Accuracy (highest priority), Helpful￾ness, Relevance, Conciseness, Completeness, and length bias (Gu et al., 2024) then issues a verdict: A better than B, B better than A, tie, or both bad. To mitigate positional bias in LLM-as-a-judge settings (Wang et al., 2… view at source ↗
Figure 8
Figure 8. Figure 8: Screenshots from the UI designed for our format generation framework. First, the relevant [PITH_FULL_IMAGE:figures/full_fig_p022_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Data generation quality assessment scores for [PITH_FULL_IMAGE:figures/full_fig_p023_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: The prompt used to classify the examples in SecKnowledge 2.0 into cybersecurity topics [PITH_FULL_IMAGE:figures/full_fig_p024_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: The prompt used to guide the LLMs during the evaluation process. [PITH_FULL_IMAGE:figures/full_fig_p025_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: LLM-as-Judge prompt used for pairwise comparison. [PITH_FULL_IMAGE:figures/full_fig_p025_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: LLM-as-Judge pairwise comparison per category: [PITH_FULL_IMAGE:figures/full_fig_p026_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: LLM-as-Judge pairwise comparison per category: [PITH_FULL_IMAGE:figures/full_fig_p026_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: LLM-as-Judge pairwise comparison per category: [PITH_FULL_IMAGE:figures/full_fig_p026_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Cybersecurity AI (CAI) Dataset

    cs.CR 2026-05 unverdicted novelty 7.0

    CAI Dataset is presented as the largest described corpus of LLM-driven hacker trajectories, with the claim that operator data concentration in frontier-model providers creates a major security risk best addressed by o...

  2. Dynamic Cyber Ranges

    cs.CR 2026-04 unverdicted novelty 7.0

    Dynamic Cyber Ranges with LLM defender agents reduce attacker success to 0-55% and preserve evaluation headroom as models advance by using comparable capabilities on both sides.

  3. Minerva: Reinforcement Learning with Verifiable Rewards for Cyber Threat Intelligence LLMs

    cs.LG 2026-01 unverdicted novelty 7.0

    MinervaRL applies reinforcement learning with verifiable rewards from CTI standards to improve LLM structured output performance by 15.8 points over base models across 12 benchmarks.

  4. Small, Free, and Effective: Orchestrating Open-Weight Small Language Models to Outperform Single LLM for Malware Analysis

    cs.CR 2026-07 conditional novelty 6.0

    An evidence-grounded hybrid of 4B and 8B open-weight models reaches 35.30% on malware-report QA, slightly beating the best ungrounded frontier model (34.77%) while trailing a frontier model given the same evidence (38.22%).

  5. A Red Teaming Framework for Evaluating Robustness of AI-enabled Security Orchestration, Automation, and Response Systems

    cs.CR 2026-05 unverdicted novelty 6.0

    A hybrid LLM-RL red teaming framework generates adaptive attack campaigns in simulated enterprise networks to evaluate the robustness of AI-enabled SOAR systems.

Reference graph

Works this paper leans on

29 extracted references · 16 linked inside Pith · cited by 5 Pith papers

  1. [1]

    Phi-4 techni- cal report.arXiv preprint arXiv:2412.08905,

    Marah Abdin, Jyoti Aneja, Harkirat Behl, S ´ebastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 techni- cal report.arXiv preprint arXiv:2412.08905,

    Pith/arXiv arXiv

  2. [3]

    A task consists of a name, a description, a format, whether it requires search, and whether it requires a grounding document. 2https://www.gradio.app/ 14 IBM Research A.2 DATAGENERATIONQUALITYASSESSMENT After rewriting the original answer according to the format, the pipeline also incorporates evaluation in the form of LLM as a Judge. There are 2 criteria...

    2023

  3. [4]

    Reformatted alignment.arXiv preprint arXiv:2402.12219,

    Run-Ze Fan, Xuefeng Li, Haoyang Zou, Junlong Li, Shwai He, Ethan Chern, Jiewen Hu, and Pengfei Liu. Reformatted alignment.arXiv preprint arXiv:2402.12219,

    Pith/arXiv arXiv

  4. [8]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276,

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276,

    Pith/arXiv arXiv

  5. [9]

    Active retrieval augmented generation

    Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7969–7992,

    2023

  6. [10]

    Kimi k2: Open agentic intelligence.arXiv preprint arXiv:2507.20534,

    Kimi Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence.arXiv preprint arXiv:2507.20534,

    Pith/arXiv arXiv

  7. [11]

    Ircopilot: Automated incident response with large language models

    Xihuan Lin, Jie Zhang, Gelei Deng, Tianzhe Liu, Xiaolong Liu, Changcai Yang, Tianwei Zhang, Qing Guo, and Riqing Chen. Ircopilot: Automated incident response with large language models. arXiv preprint arXiv:2505.20945,

    arXiv

  8. [12]

    Lost in the middle: How language models use long contexts.arXiv preprint arXiv:2307.03172,

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.arXiv preprint arXiv:2307.03172,

    Pith/arXiv arXiv

  9. [13]

    Orca 2: Teaching small language models how to reason.arXiv preprint arXiv:2311.11045,

    Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agar- wal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, et al. Orca 2: Teaching small language models how to reason.arXiv preprint arXiv:2311.11045,

    Pith/arXiv arXiv

  10. [14]

    Large language models in cybersecurity: State-of-the-art.arXiv preprint arXiv:2402.00891,

    Farzad Nourmohammadzadeh Motlagh, Mehrdad Hajizadeh, Mehryar Majd, Pejman Najafi, Feng Cheng, and Christoph Meinel. Large language models in cybersecurity: State-of-the-art.arXiv preprint arXiv:2402.00891,

    Pith/arXiv arXiv

  11. [15]

    Recycling the web: A method to enhance pre-training data quality and quantity for language models.arXiv preprint arXiv:2506.04689,

    Thao Nguyen, Yang Li, Olga Golovneva, Luke Zettlemoyer, Sewoong Oh, Ludwig Schmidt, and Xian Li. Recycling the web: A method to enhance pre-training data quality and quantity for language models.arXiv preprint arXiv:2506.04689,

    arXiv

  12. [16]

    Model version: o3 (2025-04-16)

    URL https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/ o3-and-o4-mini-system-card.pdf. Model version: o3 (2025-04-16). Pragnya Ramjee, Bhuvan Sachdeva, Satvik Golechha, Shreyas Kulkarni, Geeta Fulari, Kaushik Murali, and Mohit Jain. Cataractbot: an llm-powered expert-in-the-loop chatbot for cataract patients.Proceedings of the ACM on Interac...

    2025

  13. [17]

    Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram´e, Morgane Rivi`ere, et al. Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

    Pith/arXiv arXiv

  14. [18]

    Cybermetric: A benchmark dataset for evaluating large language models knowledge in cybersecurity.arXiv preprint arXiv:2402.07688,

    Norbert Tihanyi, Mohamed Amine Ferrag, Ridhi Jain, and Merouane Debbah. Cybermetric: A benchmark dataset for evaluating large language models knowledge in cybersecurity.arXiv preprint arXiv:2402.07688,

    Pith/arXiv arXiv

  15. [19]

    Large language models are not fair evaluators.arXiv preprint arXiv:2305.17926,

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators.arXiv preprint arXiv:2305.17926,

    Pith/arXiv arXiv

  16. [20]

    Llama-3.1- foundationai-securityllm-8b-instruct technical report.arXiv preprint arXiv:2508.01059,

    12 IBM Research Sajana Weerawardhena, Paul Kassianik, Blaine Nelson, Baturay Saglam, Anu Vellore, Aman Priyanshu, Supriti Vijay, Massimo Aufiero, Arthur Goldblatt, Fraser Burch, et al. Llama-3.1- foundationai-securityllm-8b-instruct technical report.arXiv preprint arXiv:2508.01059,

    Pith/arXiv arXiv

  17. [21]

    Large language models for cyber security: A systematic literature review

    HanXiang Xu, ShenAo Wang, Ningke Li, Kailong Wang, Yanjie Zhao, Kai Chen, Ting Yu, Yang Liu, and HaoYu Wang. Large language models for cyber security: A systematic literature review. arXiv preprint arXiv:2405.04760,

    arXiv

  18. [22]

    Primus: A pioneering collection of open-source datasets for cybersecurity llm training.arXiv preprint arXiv:2502.11191,

    Yao-Ching Yu, Tsun-Han Chiang, Cheng-Wei Tsai, Chien-Ming Huang, and Wen-Kwang Tsao. Primus: A pioneering collection of open-source datasets for cybersecurity llm training.arXiv preprint arXiv:2502.11191,

    arXiv

  19. [23]

    When llms meet cybersecurity: a systematic literature review.Cyber- security, 8:55, 2025a

    Jie Zhang, Haoyu Bu, Hui Wen, Yongji Liu, Haiqiang Fei, Rongrong Xi, Lun Li, Yun Yang, Hong- song Zhu, and Dan Meng. When llms meet cybersecurity: a systematic literature review.Cyber- security, 8:55, 2025a. doi: 10.1186/s42400-025-00361-w. Jie Zhang, Haoyu Bu, Hui Wen, Yongji Liu, Haiqiang Fei, Rongrong Xi, Lun Li, Yun Yang, Hong- song Zhu, and Dan Meng....

    doi:10.1186/s42400-025-00361-w

  20. [25]

    The questions for the benchmark were created through a collaborative process, i.e., merging expert knowledge with LLMs

    is a benchmark dataset for evaluating LLMs’ knowledge in cybersecurity. The questions for the benchmark were created through a collaborative process, i.e., merging expert knowledge with LLMs. We used the 2000 questions dataset, veri- fied by human evaluators, which covers a wide range of topics within cyber-security, validated by security experts. As ques...

    2000

  21. [26]

    evaluates the ability to differentiate between false and correct relationships among CTI entities. For example, it presents the model with two entities (e.g., instances of CVE and CWE) and two possible expla- nations—one justifying why the entities are related and another explaining why they are not. The objective is for the model to reason and determine ...

    2025

  22. [27]

    Restricting classification to single-label outputs would fail to capture these cross-cutting concerns

    This choice was intentional: many real-world cybersecurity problems span across multiple domains (e.g., a phishing campaign may involve HumanSec, IAM ZT, and ThreatOps IR simultaneously). Restricting classification to single-label outputs would fail to capture these cross-cutting concerns. Table 4: Counts by dataset and taxonomy category. dataset GCR NetS...

    1995

  23. [28]

    To mitigate positional bias in LLM-as-a-judge settings (Wang et al., 2023; Zheng et al., 2023), we run the comparison twice with the answers swapped

    then issues a verdict: A better than B, B better than A, tie, or both bad. To mitigate positional bias in LLM-as-a-judge settings (Wang et al., 2023; Zheng et al., 2023), we run the comparison twice with the answers swapped. For each permutation, a model receives a score of 3 if its answer is preferred by the judge, 1 for tie, and 0 for loss; if the prefe...

    2023

  24. [29]

    20 IBM Research Across our evaluation suite, 8-bit loading resulted in a negligible drop of 0.36% for the 4B model and 0.84% for the 8B model — relative to full precision

    by loading models directly in 8-bit and 4-bit modes, without any calibration or advanced schemes which are shown to perform better than out-of-the-box quantization (Frantar et al., 2022). 20 IBM Research Across our evaluation suite, 8-bit loading resulted in a negligible drop of 0.36% for the 4B model and 0.84% for the 8B model — relative to full precisio...

    2022

  25. [2021]

    Instruction fine-tuning: Does prompt loss matter? arXiv preprint arXiv:2401.13586,

    Mathew Huerta-Enochian and Seung Yong Ko. Instruction fine-tuning: Does prompt loss matter? arXiv preprint arXiv:2401.13586,

    Pith/arXiv arXiv

  26. [2022]

    A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594,

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Ying- han Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594,

    Pith/arXiv arXiv

  27. [2023]

    Small language models are the future of agentic ai.arXiv preprint arXiv:2506.02153,

    Peter Belcak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Muralidharan, Yingyan Celine Lin, and Pavlo Molchanov. Small language models are the future of agentic ai.arXiv preprint arXiv:2506.02153,

    Pith/arXiv arXiv

  28. [2024]

    Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323,

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323,

    Pith/arXiv arXiv

  29. [2025]

    Google Online Security Blog

    URLhttps://security.googleblog.com/2025/04/ google-launches-sec-gemini-v1-new.html. Google Online Security Blog. Anwoy Chatterjee, HSVNS Kowndinya Renduchintala, Sumit Bhatia, and Tanmoy Chakraborty. On the effect of instruction tuning loss on generalization.arXiv preprint arXiv:2507.07817,

    Pith/arXiv arXiv 2025