{"id":"b6079970-f25e-41b8-a41e-4736041f29ae","arxiv_id":"2501.06239","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"0-CTI is a modular system that extracts cybersecurity entities and relationships from threat reports in both supervised and zero-shot modes, reporting higher entity-extraction F1 than the STIXnet baseline.","lead":"This paper introduces 0-CTI, a framework that reads cyber threat reports and turns them into structured data in a standard threat-sharing format. It combines supervised and zero-shot AI models so organizations without labeled data or dedicated analysts can still extract entities and relationships.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'surpasses state-of-the-art' claim rests on an unfair comparison: 0-CTI CyBERT is fine-tuned on the STIXnet dataset and evaluated on the same data, with no documented held-out split, so the reported 0.98 F1 vs 0.77 gap may reflect information leakage rather than genuine superiority.","rationale":"The strongest claim in the abstract is the quantitative promise that the supervised Entity Extractor surpasses current state-of-the-art performance. The evidence for this is Table IV, produced by the experiment in Section V-A-2. The text explicitly says the model was fine-tuned on the STIXnet dataset, and no held-out evaluation is mentioned. This means the evaluation protocol allows training information to leak into the test set, giving the 0-CTI model a built-in advantage over the rule-based STIXnet baseline. A proper hold-out split or k-fold evaluation would settle whether the reported superiority is real. The comparison is also limited to a single baseline, STIXnet, so the label 'state-of-the-art' would require broader benchmarking even with a clean split. The reader's identified weakest assumption — the NLI cross-encoder zero-shot relation extraction — is a valid concern, but it supports a secondary claim that the paper itself labels qualitative. The supervised SOTA claim is presented as a hard result, so the invalid comparison is more immediately damaging to the central contribution. I agree with the overall CONDITIONAL verdict, but the load-bearing weakness is the supervised evaluation protocol, not the zero-shot entailment scoring. The proposed concrete test directly checks this by recomputing Table IV with an honest train/test split, which either confirms or refutes the headline SOTA claim.","tokens_in":12018,"tokens_out":5175,"duration_ms":51606,"concrete_test":"Re-run the Section V-A-2 comparison with a strict protocol: (1) split the STIXnet dataset into, e.g., 80% train / 20% test by report; (2) fine-tune 0-CTI CyBERT on the train split only; (3) run STIXnet on the same test split; (4) compute per-class and overall F1 for both systems on the identical held-out instances; (5) repeat with at least 5 random seeds and report mean ± standard deviation. If the held-out 0-CTI F1 does not beat STIXnet by more than the seed variance, the state-of-the-art claim fails. In addition, release the exact split and code so the result is reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section V-A-2, the authors state: 'Our approach involved fine-tuning the 0-CTI CyBERT model on the new dataset' (the STIXnet dataset), and Table IV then reports 0-CTI with an overall F1 of 0.98 versus STIXnet's 0.77 on that dataset. No train/test split, cross-validation, or seed variance is described. Because STIXnet is rule-based, it cannot 'see' evaluation instances during any training phase; the 0-CTI model, by contrast, is trained on the very reports on which it is scored. This gives the 0-CTI model an information advantage that can fully explain the reported gap. The abstract's strongest quantitative claim — that the supervised Entity Extractor 'surpasses current state-of-the-art performance' — is directly based on this table. A valid comparison requires a held-out test set or k-fold evaluation, with STIXnet scored on the same held-out instances and with multiple training runs to estimate variance. Without that, the state-of-the-art claim is not established. Additionally, the comparison includes only STIXnet as a baseline; other published cyber-NER systems are not benchmarked, so 'state-of-the-art' is under-supported even apart from the split issue.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces 0-CTI, a modular framework for Cyber Threat Intelligence Information Extraction that combines text preprocessing, an IOC-finder, supervised or zero-shot named entity recognition, and a cross-encoder-based relation extraction module, with outputs aligned to the STIX format. The system supports a fully zero-shot pipeline using GLiNER for entities and a cross-encoder NLI-style scorer for relations, and a supervised pipeline fine-tuned on the authors' annotated OpenCTI-derived dataset or on the STIXnet dataset. The paper claims that 0-CTI is the first modular CTI-IE framework supporting both supervised and zero-shot learning, that its supervised entity extractor surpasses state-of-the-art performance (0.98 F1 vs. STIXnet's 0.77 on the STIXnet dataset), and that zero-shot entity and relation extraction obtain LLM-as-a-judge scores of 0.91 and 0.83 on 120 CTI reports.","tokens_in":12301,"tokens_out":4858,"duration_ms":48155,"significance":"If the empirical claims were validated, 0-CTI would be a practically useful contribution: it combines existing transformer components into a single STIX-compliant pipeline, offers a dataless mode for relation extraction, and reports detailed entity distributions that could help future benchmark design. The architectural modularity and the use of a fixed STIX relation table are sensible design choices, and the paper gives a reasonably complete description of the pipeline. However, the central quantitative claims are not currently established: the supervised state-of-the-art comparison in Table IV is compromised by the lack of a documented held-out split, and the zero-shot evaluation in Section V-B relies on an unvalidated LLM judge instead of ground truth. The paper contains no machine-checked proofs, no released code, and no parameter-free derivations; its contribution is purely empirical and therefore depends directly on the soundness of the evaluation protocol.","major_comments":[{"comment":"The abstract's claim that the supervised Entity Extractor 'surpasses current state-of-the-art performance' rests on Table IV, but this comparison is not valid as reported. The authors state that they fine-tuned 0-CTI CyBERT on the STIXnet dataset, and the table then reports 0.98 overall F1 versus STIXnet's 0.77 on that dataset. No held-out split, cross-validation, or multiple-seed variance is documented. Because STIXnet is rule-based and does not train on the reports, while 0-CTI's fine-tuned model does, the reported gap may be fully explained by information leakage from training on the evaluation instances. A valid comparison requires a held-out test set or k-fold evaluation, with STIXnet scored on the same held-out instances and with repeated training runs to estimate variance.","section":"§V-A-2, Table IV"},{"comment":"The zero-shot evaluation uses ChatGPT-4 as an LLM judge rather than gold annotations, and no evidence is provided that the judge's scores correlate with human judgments or with downstream task performance. The reported average scores of 0.91 for zero-shot NER and 0.83 for zero-shot relation extraction are therefore not established as measures of extraction quality. The paper itself acknowledges in §V that no relation-annotated dataset is available and in §VI that acquiring annotated data is future work; at least a subset of the 120 reports should be scored against human annotations before the zero-shot claims are made.","section":"§V-B"},{"comment":"The supervised NER comparison reports single point estimates from one 3500/880 train/test split, with no standard deviations, confidence intervals, or significance tests. Given the large class-level F1 differences (for example, 0.61 versus 0.27 for IDENTITY ORGANIZATION), these differences may be within noise. The claim that the transformer cores outperform the LSTM baseline requires multiple seeds and, ideally, a statistical test over repeated runs, or at least error bars.","section":"§V-A-1, Table III"},{"comment":"The 'state-of-the-art' claim is under-supported even if the split issue is set aside: the only quantitative baseline in Table IV is STIXnet, while several other cyber-NER systems discussed in §III (CyNER, Li et al., Wang et al., Zhou et al.) are not evaluated under the same protocol. At minimum, the claim should be narrowed to 'outperforms STIXnet on this dataset,' or additional baselines should be included under identical training and evaluation conditions.","section":"§V-A and Abstract"}],"minor_comments":[{"comment":"The text says STIX comprises 'six distinct classes,' but the footnote and the rest of the paper refer to 18 SDOs; this inconsistency should be reconciled.","section":"§II"},{"comment":"The word 'sanification' should be 'sanitization,' and the term 'chunkizer' is used informally; please define it or use standard terminology.","section":"§IV-A"},{"comment":"The relation extraction algorithm relies on a 'predefined threshold' for accepting relations, but no threshold value, tuning procedure, or sensitivity analysis is reported, which hinders reproducibility.","section":"§IV-C-2"},{"comment":"The claim that the IOC-finder's accuracy is 'approaching 100%' is not supported by any reported evaluation; if its results are excluded from all metrics, that assumption should be stated explicitly.","section":"§IV-B"},{"comment":"The description of GLiNER as interpreting NER as a natural language inference problem is imprecise; GLiNER is a bidirectional transformer trained for span matching, so the mechanism should be described accurately.","section":"§IV-C-1"}],"recommendation":"major_revision","confidential_remarks":"The main obstacle is the evaluation methodology: the supervised state-of-the-art claim depends on an uncontrolled train/test setup, and the zero-shot evaluation lacks gold-standard validation. These are fixable within the manuscript's scope by adding held-out evaluation, additional baselines, and human annotation for at least a subset of reports. I therefore recommend major revision rather than rejection. The paper's own statements about ongoing work and missing relation-annotated data should be taken as an acknowledgment of the current limitations rather than as a substitute for addressing them."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the 0-CTI paper. The architecture is coherent and the zero-shot design is genuinely useful for small CTI teams who lack annotated data. Combining GLiNER, cross-encoders, and a STIX-aligned relation table into a single modular pipeline is a new integration, and the idea of scoring candidate relation sentences with an NLI-style cross-encoder is neat. The recursive chunking to capture cross-sentence relations is also reasonable.\n\nThe main quantitative claim, though, does not survive contact with the experimental section. In Section V-A-2, the authors say they fine-tuned 0-CTI CyBERT on the STIXnet dataset and then report an F1 of 0.98 against STIXnet's 0.77 on that same dataset. No train/test split, cross-validation, or seed variance is described. STIXnet is rule-based, so it never sees training instances; the 0-CTI model is trained on the very reports it is scored against. That information advantage alone can explain the gap. The abstract's \"surpasses current state-of-the-art\" claim is built on this table, so it is not established as written.\n\nThe zero-shot relation extraction is also validated only via ChatGPT-4 as a judge, not gold annotations. That is a weak substitute, and the authors admit relation evaluation is still ongoing. The \"first modular framework\" claim is overstated since STIXnet already exists,\nthough the particular supervised-plus-zero-shot combination may be new. No code or data is released, so even the more plausible supervised NER result on their own dataset cannot be checked.\n\nCredit where due: the paper is honest about its limitations in the conclusion, the citation pattern is fair, and the system design is clearly explained. The math is simple—no formal guarantees—but none are claimed.\n\nWho is this for? Practitioners building CTI tooling who want a survey of off-the-shelf components and a template for a STIX-compliant pipeline. It deserves a serious referee, but the referee should send it back for a re-run of the STIXnet comparison with a proper held-out split or k-fold, plus gold-labeled evaluation of the zero-shot modules.\n\nVerdict: worthwhile as a systems idea, not yet convincing as a state-of-the-art claim. I would engage with the work in a reading group, but I would not cite the headline numbers until the experimental setup is fixed.","headline":"The 0-CTI pipeline is a sensible, practically useful integration of existing components, but the paper's headline claim of surpassing state-of-the-art supervised entity extraction rests on a comparison that likely leaks training data and needs to be redone before it is credible.","tokens_in":12820,"tokens_out":1841,"would_cite":false,"duration_ms":20614,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that 0-CTI, a modular transformer-based pipeline, extracts STIX-compliant cyber threat entities and relations from unstructured reports in both supervised and fully zero-shot modes, with the supervised entity extractor…","keywords":["cyber threat intelligence","information extraction","named entity recognition","relation extraction","zero-shot learning","STIX","transformer models","cross-encoder entailment"],"falsifier":"Annotate all STIX relationships in the 120 CTI reports used in the LLM-as-a-Judge evaluation with two independent human analysts, then compute precision, recall, and F1 for the zero-shot relation extractor at the paper's reported threshold and across a sweep of thresholds. If no threshold separates true from spurious candidate relations, or if the LLM judge's scores do not track the human annotations, the central dataless relation-extraction claim is falsified.","tokens_in":11821,"feed_emoji":"🛡️","tokens_out":10342,"duration_ms":88777,"temperature":0.7,"pith_summary":"0-CTI is a modular transformer-based pipeline that converts unstructured Cyber Threat Intelligence reports into STIX-compliant knowledge graphs, and the paper's central claim is that this extraction works both with labeled data and with no labeled data at all. The supervised entity extractor, fine-tuned on the STIXnet dataset, achieves an overall F1 of 0.98 versus STIXnet's 0.77, which grounds the paper's assertion that it surpasses the state of the art in cyber entity extraction. The zero-shot side pairs a generalist NER model with a flat STIX taxonomy for entities, and uses a cross-encoder to score whether generated candidate relation sentences are entailed by the original report. If the claim holds, organizations without annotated corpora or dedicated CTI analysts could still automate threat-intelligence extraction and share the results in a standard STIX format.","feed_headline":"Threat-intel extraction works with zero labeled data","feed_subtitle":"A modular STIX pipeline pulls entities and relations from raw reports in supervised or fully zero-shot mode.","key_machinery":"The load-bearing mechanism is the cross-encoder entailment scorer for zero-shot relation extraction. A cross-encoder processes two sequences together and returns a score for whether the second is entailed by the first; here it decides whether a candidate '<entity1> <relation> <entity2>' sentence is supported by the original report, and thresholding plus directional disambiguation convert those scores into accepted STIX relationship objects. For entities, the zero-shot path uses a generalist NER model that frames extraction as natural-language inference and is steered by a hand-built flat taxonomy mapping child labels to STIX parent classes, while the supervised path fine-tunes a transformer on annotated entities. The pipeline's modularity lets the regex-based IOC finder, the supervised NER core, and the zero-shot NER core be combined or swapped, with entity outputs feeding the relation module.","core_discovery":"On its own terms, 0-CTI's discovery is that relation extraction can be made fully dataless by recasting it as textual entailment. After entities are extracted, the system looks up which STIX relationship objects are permitted between the two entity types and generates candidate sentences of the form '<entity1> <relation> <entity2>'; a cross-encoder scores whether the original report text entails each candidate, and a threshold plus a directional tie-break keep only the most plausible relations. The same zero-shot principle is applied to entities by combining a generalist NER model with a flat taxonomy whose fine-grained child labels roll up into STIX domain objects. In the supervised branch, a cybersecurity-domain BERT model fine-tuned for token classification is reported to reach F1 of 0.98 on the STIXnet dataset against STIXnet's 0.77. The paper states that the zero-shot components were evaluated qualitatively with an LLM-as-a-Judge method, yielding average scores of 0.91 for entities and 0.83 for relations, while a gold-annotated relation benchmark is listed as ongoing future work.","pith_inferences":["The generate-and-score entailment recipe is domain-agnostic: any field with a closed relation vocabulary and typed entity pairs could reuse it, provided the cross-encoder's language coverage fits the domain.","The LLM-as-a-Judge scores are a provisional substitute for gold labels; the reported 0.83 for relations could shift materially once human-annotated relation benchmarks are used, so the zero-shot relation extraction claim should be read as promising rather than settled.","The 'data-independent' label is best understood as annotation-free rather than tuning-free: the flat taxonomy, the relation candidate list, and the thresholds are hand-designed components that carry real engineering weight.","A natural extension is active learning, where the cross-encoder's confidence scores flag uncertain relations for analyst review, blending zero-shot coverage with minimal human labeling effort."],"forward_implications":["A team with no annotated corpus can run the full entity-plus-relation extraction pipeline in zero-shot mode and receive STIX-formatted knowledge graph output.","The supervised NER core is claimed to beat the prior STIXnet system, reaching F1 0.98 versus 0.77 on the STIXnet dataset.","Because the output uses STIX relationship objects, extracted intelligence can be shared across organizations and integrated into existing CTI tooling without reformatting.","The modular design allows the regex-based IOC finder, the supervised NER core, and the zero-shot NER core to be combined or swapped independently, so upgrades can target one submodule at a time.","Errors in entity extraction propagate into relation scoring, so any improvement in NER, supervised or zero-shot, directly improves relation extraction quality."],"supporting_citations":[{"why":"Supplies the zero-shot NER model that the framework steers with a flat STIX taxonomy.","marker":"[4]"},{"why":"Provides the cybersecurity-domain BERT embeddings used in the supervised NER core that reports the F1 gains.","marker":"[11]"},{"why":"Defines the STIXnet baseline and dataset against which the supervised entity extractor claims superiority.","marker":"[17]"},{"why":"Provides the alternative BGE transformer backbone used for supervised token classification.","marker":"[25]"},{"why":"Defines the STIX SDO/SCO/SRO taxonomy that the whole extraction pipeline is aligned to.","marker":"[5]"},{"why":"Supplies the cross-encoder zero-shot scoring method that the relation extraction algorithm adapts.","marker":"[26]"},{"why":"Provides the LLM-as-a-Judge methodology used to evaluate the zero-shot components.","marker":"[28]"}],"fun_headline_variants":["Zero-shot CTI relation extraction via entailment","Fully dataless CTI extraction, STIX-aligned output","Threat intel extraction without any labeled data","Modular CTI framework: SOTA supervised or zero-shot","CTI extraction goes zero-shot with textual entailment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a general-purpose cross-encoder trained on natural-language inference data can correctly judge whether a generated sentence like 'APT1 targets Microsoft' is entailed by a threat report, and that this entailment score reliably separates true relations from spurious ones; the paper tests this only with an LLM-as-a-Judge evaluation, not against gold relation annotations.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot CTI relation extraction via entailment","Fully dataless CTI extraction, STIX-aligned output","Threat intel extraction without any labeled data","Modular CTI framework: SOTA supervised or zero-shot","CTI extraction goes zero-shot with textual entailment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000313,"raw_usage":{"total_tokens":1812,"prompt_tokens":1014,"completion_tokens":798,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":718}},"tokens_in":630,"tokens_out":798,"duration_ms":8462,"temperature":1.0,"reasoning_tokens":718,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:31:46.830035+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Annotate all STIX relationships in the 120 CTI reports used in the LLM-as-a-Judge evaluation with two independent human analysts, then compute precision, recall, and F1 for the zero-shot relation extractor at the paper's reported threshold and across a sweep of thresholds. If no threshold separates true from spurious candidate relations, or if the LLM judge's scores do not track the human annotations, the central dataless relation-extraction claim is falsified.","supporting_citations":[{"cited_title":"SignalGP-Lite: Event Driven Genetic Programming Library for Large-Scale Artificial Life Applications","cited_arxiv_id":"2108.00382","evidence_quote":"Provides the cybersecurity-domain BERT embeddings used in the supervised NER core that reports the F1 gains."},{"cited_title":"STIXnet: A Novel and Modular Solution for Extracting All STIX Objects in CTI Reports","cited_arxiv_id":"2303.09999","evidence_quote":"Defines the STIXnet baseline and dataset against which the supervised entity extractor claims superiority."},{"cited_title":"C-Pack: Packaged Resources To Advance General Chinese Embedding,","cited_arxiv_id":null,"evidence_quote":"Provides the alternative BGE transformer backbone used for supervised token classification."},{"cited_title":"Standardizing Cyber Threat Intelligence Information with the Structured Threat Information eXpression (STIX),","cited_arxiv_id":null,"evidence_quote":"Defines the STIX SDO/SCO/SRO taxonomy that the whole extraction pipeline is aligned to."}],"review_version":1}