{"id":"c2019797-8a9b-490e-b9a5-47866bfd0a0c","arxiv_id":"2508.01059","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Foundation-Sec-8B-Instruct, an instruction-tuned 8B cybersecurity LLM, is released and claimed to beat Llama 3.1-8B-Instruct on CTIBench-RCM and CTIBench-MCQA while remaining competitive on general instruction-following.","lead":"This paper releases Foundation-Sec-8B-Instruct, a cybersecurity-focused chat model built on a prior Llama-3.1-8B based base model. It reports strong results on two cyber threat intelligence benchmarks, beating Llama 3.1-8B-Instruct and rivaling GPT-4o-mini, while losing on several other security and general benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The CTIBench-RCM SOTA claim rests on the untested purity of proprietary SFT/DPO data; the paper's own audit flags >80% RCM contamination in a related public set but never checks its own mixtures.","rationale":"The reader's weakest assumption is that the proprietary post-training data do not contain CTIBench-RCM items. This is exactly the load-bearing condition for the paper's strongest claim. The paper itself demonstrates how severe such leakage can be: Primus-Reasoning, a public cybersecurity post-training dataset, has 83.1% exact-match contamination with CTIBench-RCM and 78-91% by other detectors, and the authors use that as a validation case. They then fail to apply their own detector to the SFT/DPO mixtures that produced Foundation-Sec-8B-Instruct. Given that the SFT data are synthetically generated from web-scale cybersecurity text, the prior probability of some overlap with CVE/CWE benchmark items is non-trivial. The issue is compounded by the exclusion of RCM from the base-model comparison in Section 6.5, which prevents the reader from determining whether the RCM capability is inherited from Foundation-Sec-8B (legitimate pretraining) or introduced during post-training (possible leakage). The abstract's broader claim that the model outperforms Llama 3.1-8B-Instruct on 'a range of cybersecurity tasks' is also not supported by Table 4, where SecBench and SecEval show lower scores; this overstatement is secondary but reinforces the need for caution. None of this impugns the authors; it is a standard reproducibility concern for model-released benchmarks. The proposed test directly applies the paper's own validated methodology to the missing dataset, and the conditional verdict already given by the reader remains appropriate.","tokens_in":25577,"tokens_out":4983,"duration_ms":59139,"concrete_test":"Run the paper's own layered contamination pipeline (n-gram, embedding cosine >= 0.8, LLM-as-judge; Section 4.3/E.1) on the proprietary SFT and DPO mixtures against every CTIBench-RCM sample, and report EM/NG/LLM-ES rates. If any LLM-verified overlap is non-zero, the Table 4 RCM score must be treated as potentially contaminated and the claim re-evaluated on a retrained or decontaminated model; if the audit is clean, the SOTA claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6.2 (Table 4) claims Foundation-Sec-8B-Instruct is state-of-the-art on CTIBench-RCM (0.692 ± 0.005), beating GPT-4o-mini (0.655) and Llama 3.1-70B (0.623). The central claim presumes that this score reflects generalization rather than memorization. The paper's own contamination analysis (Section 4.3, Tables 9-10) shows that a related public post-training set, Primus-Reasoning, contains 78-91% of CTIBench-RCM samples; the authors use that finding to validate their detector, yet they never run the detector on their proprietary SFT/DPO mixtures. Section 5.2 describes a synthetic data pipeline built from web-scale cybersecurity corpora, the same sources from which CTIBench-RCM CVE/CWE pairs are drawn. The paper also omits Foundation-Sec-8B (base) from Table 4 and explicitly excludes RCM from the base-vs-instruct comparison in Section 6.5, so there is no evidence distinguishing post-training knowledge gain from leakage. Without a decontamination audit of the actual training mixtures, the SOTA comparison against larger models is unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Foundation-Sec-8B-Instruct, an instruction-tuned cybersecurity LLM built on the authors' previously released Foundation-Sec-8B base model. The report describes a post-training pipeline combining supervised fine-tuning and direct preference optimization, evaluates the model on six security benchmarks and seven general-purpose benchmarks, and includes safety and persona-adaptation analyses. The headline claims are state-of-the-art performance on CTIBench-RCM (0.692 ± 0.005, beating GPT-4o-mini and Llama 3.1-70B-Instruct) and competitiveness with GPT-4o-mini on cyber threat intelligence and instruction-following tasks.","tokens_in":25770,"tokens_out":7406,"duration_ms":79352,"significance":"If the headline results hold, the paper demonstrates that a domain-adapted 8B-parameter model can outperform much larger general-purpose models on a specific cyber threat intelligence task, which is practically significant for efficient deployment in security workflows. The evaluation protocol is generally sound: security results are averaged over 10 trials with reported standard deviations, and general benchmarks use standard settings via OLMES. The paper also contributes a careful contamination analysis of public post-training datasets, a public model release, and a candid discussion of safety limitations. However, the central claim is currently undercut by the absence of a contamination audit of the model's own SFT/DPO mixtures, and by a base-model comparison that excludes the key benchmark. These issues must be addressed before the comparative claims can be considered verified.","major_comments":[{"comment":"The state-of-the-art claim on CTIBench-RCM rests on the untested purity of the proprietary SFT/DPO data. In §4.3, the authors develop a layered contamination detector (exact match, 8-gram overlap, embedding similarity, LLM-as-judge) and validate it by detecting 78.8–91.5% contamination of CTIBench-RCM in Primus-Reasoning. Yet the same detector is never applied to the authors' own post-training mixtures described in §5.2. This is a direct internal-validity threat: §5.2 states that the synthetic data pipeline is built from web-scale cybersecurity corpora, which are the same kinds of sources from which CTIBench-RCM CVE/CWE pairs are drawn, and the paper shows in Table 3 that cybersecurity post-training datasets can contain large fractions of benchmark items. Without an audit of the actual training data (or an explicit verified statement that no CTIBench-like items are present), the 0.692±0.005 score in Table 4 cannot be interpreted as generalization rather than memorization. I ask the authors to run their own detector on the SFT and DPO mixtures and report the results, including the LLM-verified estimates.","section":"§4.3, §5.2, §6.2 (Table 4)"},{"comment":"The comparison to Foundation-Sec-8B excludes exactly the benchmark for which the SOTA claim is made. §6.5 compares the instruct model to the base model only on CyberMetric-500, CTIBench-MCQA, and SecBench, explicitly excluding CTIBench-RCM because the base model cannot follow the short-answer format. Moreover, Table 4 does not list Foundation-Sec-8B at all. As a result, the reader cannot tell whether the RCM score reflects knowledge acquired in pre-training or contamination introduced during post-training. I acknowledge the authors' stated difficulty in few-shot evaluating a base model on RCM, but some control is needed—for example, reporting the base model's best-effort RCM score with a clear caveat, or evaluating the instruct model on a withheld, uncontaminated RCM split—before the SOTA assertion is supportable.","section":"§6.5, Figure 4"},{"comment":"The abstract and conclusion overstate the comparative results. Table 4 shows that Foundation-Sec-8B-Instruct outperforms Llama 3.1-8B-Instruct on only two of six security benchmarks (CTIBench-RCM +24.03%, CTIBench-MCQA +4.40%) and is worse on the remaining four (CTIBench-VSP −1.67%, CyberMetric-500 −2.01%, SecBench −5.21%, SecEval −2.50%). The abstract's claim that the model 'outperforms Llama 3.1-8B-Instruct on a range of cybersecurity tasks' is therefore inaccurate. Similarly, the conclusion's 'best-in-class' framing is not supported by the full table. The summary statements should be revised to present the mixed results honestly, and the paper should avoid cherry-picking the two positive benchmarks.","section":"Abstract; §8 Conclusion; Table 4"}],"minor_comments":[{"comment":"The word 'demonstraing' should be 'demonstrating'.","section":"§6.3"},{"comment":"The word 'Therefor' should be 'Therefore'.","section":"§6.5"},{"comment":"The model is inconsistently named; the title uses 'Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct' while the body uses 'Foundation-Sec-8B-Instruct'. Please unify the naming.","section":"Throughout"},{"comment":"The SFT/DPO data composition, dataset sizes, and hyperparameters are not reported; this limits reproducibility and prevents the contamination audit from being repeated by others.","section":"§5.2"},{"comment":"The contamination detector is validated on a positive control (Primus-Reasoning) but not on a known-clean dataset, so the false-positive rate of the embedding-similarity stage is unknown; the authors' own caveat that thresholds are tuned makes this absence of a negative control important.","section":"§4.3, Appendix E"},{"comment":"PersonaGym results are reported on 50 of 200 personas with no description of how the subset was selected; the subset choice could bias the comparison against the published baselines.","section":"§F"},{"comment":"For MATH, the metric 'flex exact match' is not defined in the main text; please define it or provide a more precise citation to the Tülu 3 definition.","section":"Table 7"},{"comment":"The labels 'CTI-ATE', 'CTI-MCQA', 'CTI-RCM', 'CTI-TAA', and 'CTI-VSP' do not match the 'CTIBench' names used elsewhere; please unify the nomenclature for clarity.","section":"Tables 9, 10"}],"recommendation":"major_revision","confidential_remarks":"The paper reads more like a technical report than a full journal article, and the central SOTA claim hinges on the proprietary data contamination audit. The missing audit is fixable within the manuscript's scope, so a major revision rather than rejection seems appropriate. I would also encourage the editor to require the base-model control for CTIBench-RCM, if feasible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version. The paper releases a useful open-weight cybersecurity instruct model and contributes a genuinely informative contamination analysis of public post-training datasets. But the stress-test concern is real: the headline claim—state-of-the-art on CTIBench-RCM, beating GPT-4o-mini and Llama 3.1-70B—rests on the assumption that their proprietary SFT/DPO mixtures are free of benchmark leakage, and they never check. That makes the central comparison unverified, not false.\n\nWhat is actually new: the model itself, and the detailed contamination audit. Tables 9 and 10 show a layered detector (exact match, n-gram, embedding similarity, LLM-verified) applied to common post-training sets. The framework correctly flags Primus-Reasoning as heavily contaminated with CTIBench-RCM (78-91%), which validates the method and is a useful community reference. The safety discussion is also honest: no dedicated safety alignment, recommend guardrails, HarmBench with and without LlamaGuard.\n\nWhere the paper is soft: the abstract and conclusion overstate. The abstract says the model 'outperforms Llama 3.1-8B-Instruct on a range of cybersecurity tasks' and is 'competitive with GPT-4o-mini.' Table 4 shows the opposite on most benchmarks—below Llama 3.1-8B on VSP, CyberMetric, SecBench, and SecEval, and below GPT-4o-mini on everything except RCM. The selective framing is easy to fix. More importantly, Section 5.2 describes a synthetic data pipeline built from web-scale cybersecurity corpora—the same sources CTIBench-RCM draws from. Combined with the exclusion of RCM from the base-vs-instruct comparison (Section 6.5), there is no evidence distinguishing post-training capability gain from leakage. This is a load-bearing gap, but it is addressable: they already have the contamination detector, so running it on their own SFT/DPO mixtures is the obvious next step. Minor issues: the PersonaGym table lists 'Llama 2-8B' and 'Llama 3-8B' which look like typos, and Section 3 has a duplicated sentence.\n\nWho should read this: practitioners benchmarking cyber LLMs and researchers working on data contamination. The model is open-weight and the contamination tables are a good reference. I would send it to peer review, with the expectation that the authors either decontaminate their own training data or scale back the abstract and conclusion. The engineering is solid; the verification isn't finished.","headline":"Useful open-weight cyber model and a genuinely informative contamination audit, but the headline RCM SOTA claim is unverified because the authors never decontaminate their own SFT/DPO mixtures.","tokens_in":26386,"tokens_out":3544,"would_cite":true,"duration_ms":37880,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An 8B security model beats far larger ones at CVE-to-CWE mapping.","keywords":["cybersecurity assistant","instruction tuning","cyber threat intelligence","CWE mapping","benchmark contamination","direct preference optimization","domain-adapted LLM","CTIBench"],"falsifier":"Run the paper's own contamination-finding tools on the proprietary instruction-tuning and preference data, and look specifically for CTIBench-RCM items; if any are found at the similarity thresholds the paper uses, the claimed 0.692 score is no longer a fair comparison. A cleaner version would train a fresh model on decontaminated data and see if the edge over GPT-4o-mini survives.","tokens_in":25346,"feed_emoji":"🛡️","tokens_out":6882,"duration_ms":80075,"temperature":0.7,"pith_summary":"This technical report argues that a compact, domain-adapted language model can be a practical general-purpose cybersecurity assistant: Foundation-Sec-8B-Instruct, an 8B-parameter model, is claimed to outperform Llama 3.1-70B-Instruct and GPT-4o-mini on CTIBench-RCM, the vulnerability root-cause mapping benchmark, while roughly matching GPT-4o-mini on other security and instruction-following tasks. The authors attribute this to starting from a base model continually pre-trained on a large cybersecurity corpus, then applying supervised fine-tuning and direct preference optimization without trying to add new domain knowledge during post-training. They also present a layered contamination-detection analysis of open post-training datasets and show that one widely used dataset, Primus-Reasoning, contains substantial CTIBench items. If the results hold, the recipe points to an efficient path for building capable security assistants without relying on closed frontier models.","feed_headline":"8B security model beats 70B rival at CVE-to-CWE mapping","feed_subtitle":"An 8-billion-parameter open model posts the top CTIBench-RCM score; the lingering question is whether its training data leaked the answers.","key_machinery":"The load-bearing object is the model itself: Foundation-Sec-8B-Instruct, an 8B-parameter chat model built by continual pretraining of Llama 3.1-8B on roughly five billion cybersecurity tokens, producing the Foundation-Sec-8B base, followed by supervised fine-tuning and direct preference optimization on a diverse mix. The mechanism carrying the argument is the post-training recipe's division of labor: pre-training supplies domain knowledge, while SFT and DPO add instruction-following and preference alignment, and deliberately avoiding new knowledge during post-training is what the authors say protects the model's cybersecurity evaluations from dilution and keeps the benchmark comparisons meaningful. The supporting machinery includes a layered decontamination pipeline, combining exact-match detection, 8-gram overlap, embedding similarity with an elevated cosine threshold, and LLM-as-a-judge verification, used to audit public post-training datasets and validate the benchmarks.","core_discovery":"On the paper's own terms, the central discovery is that instruction-tuning a domain-specialized base model, rather than scaling up parameters or injecting new facts during post-training, can put an 8B model at the top of a challenging cyber threat intelligence benchmark. Foundation-Sec-8B-Instruct scores 0.692 on CTIBench-RCM, above GPT-4o-mini (0.655) and Llama 3.1-70B-Instruct (0.623), and it is within error bars of the best models on CTIBench-MCQA. The same model matches or beats Llama 3.1-8B-Instruct on instruction following while retaining most of its base model's cybersecurity knowledge, with only mild degradation on knowledge tasks. A secondary finding is that its three-stage contamination pipeline correctly identifies a known leak in Primus-Reasoning (79–91% of CTIBench-RCM items) while finding low overlap in other public datasets, which the authors present as evidence that their evaluation setup and detection tools are sound.","pith_inferences":["If the central claim is right, the biggest remaining threat to it is unmeasured contamination in the proprietary SFT and DPO mixture: the paper audits only public datasets, and its own pipeline should be run on the exact data behind Foundation-Sec-8B-Instruct.","A testable prediction follows from the paper's division-of-labor thesis: a model trained with less domain pre-training or with heavy cybersecurity data injected during SFT should show worse RCM performance and more hallucination on knowledge benchmarks.","The contamination pipeline's sensitivity to thresholds suggests the reported low overlaps for other datasets are lower bounds rather than exact point estimates, because semantic-similarity metrics on security text can overcount when domain vocabulary overlaps by default.","The paper's comparison to a base model is clouded by format differences between few-shot and chat prompting, so the claim of negligible knowledge loss is best read as an upper bound on retention rather than a precise measurement."],"forward_implications":["A relatively small open model can beat models 8–9 times its size on at least one real cyber threat intelligence task, so scaling parameters is not the only route to domain competence.","Domain-adapted models can gain instruction-following and preference alignment without catastrophic forgetting of cybersecurity knowledge, and the comparison to Foundation-Sec-8B shows negligible drops.","The CTIBench-RCM result has a direct practical meaning: an 8B model can reliably map CVE descriptions to CWE identifiers, a task that analysts currently often do by hand.","Public post-training corpora vary sharply in contamination level, and the Primus-Reasoning case shows that even widely used cybersecurity datasets can encode benchmark answers.","Deploying such a model for safety-sensitive security work will require an external guardrail, since the model without a filter rejects only 92% of harmful prompts and with LlamaGuard reaches 99%."],"supporting_citations":[{"why":"Supplies Foundation-Sec-8B, the base model and the 5-billion-token cybersecurity pretraining recipe that the instruct model builds on.","marker":"[36]"},{"why":"Defines CTIBench-RCM, the benchmark on which the paper's strongest SOTA claim is made.","marker":"[3]"},{"why":"Defines the Llama 3.1-8B and Llama 3.1-70B-Instruct baselines that the paper compares against.","marker":"[22]"},{"why":"Supplies the GPT-4o-mini checkpoint used as the closed-model baseline on security and general benchmarks.","marker":"[29]"},{"why":"Provides the Primus baselines and the Primus-Reasoning dataset whose known CTIBench contamination the paper uses to validate its detection pipeline.","marker":"[75]"},{"why":"Supplies the direct preference optimization method used alongside SFT in the post-training recipe.","marker":"[52]"},{"why":"Provides the LlamaGuard filter that the paper recommends and that raises HarmBench refusal from 92% to 99%.","marker":"[30]"}],"fun_headline_variants":["8B security model tops GPT-4o-mini on cyber benchmark","Instruction-tuned 8B model outranks 70B on threat intelligence","Small security LLM beats larger rivals on CTIBench-RCM","Open 8B model leads CTI tasks with leak checks done"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported scores rest on the assumption that none of the test questions appeared in the model's own training data, and the paper checks only public datasets, not the mixture it actually used.","fun_headline_variants_meta":{"raw":{"variants":["8B security model tops GPT-4o-mini on cyber benchmark","Instruction-tuned 8B model outranks 70B on threat intelligence","Small security LLM beats larger rivals on CTIBench-RCM","Open 8B model leads CTI tasks with leak checks done"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1357,"prompt_tokens":1012,"completion_tokens":345,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":266}},"tokens_in":628,"tokens_out":345,"duration_ms":4621,"temperature":1.0,"reasoning_tokens":266,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T05:52:50.204462+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's own contamination-finding tools on the proprietary instruction-tuning and preference data, and look specifically for CTIBench-RCM items; if any are found at the similarity thresholds the paper uses, the claimed 0.692 score is no longer a fair comparison. A cleaner version would train a fresh model on decontaminated data and see if the edge over GPT-4o-mini survives.","supporting_citations":[],"review_version":1}