{"id":"4f8ae1c9-d046-4e5f-8aa0-437f3147b725","arxiv_id":"2507.16576","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fine-tuned LLMs extract STIX entities and relationships from threat reports with per-module F1 scores of 84.4%, 88.5%, 95.5%, and 84.6%, backed by a new 4,011-entity annotated dataset.","lead":"The paper introduces AZERG, a tool that fine-tunes open-weight language models to pull STIX-standard entities and relationships out of cyber threat reports, a task currently done mostly by hand. It also releases a manually annotated dataset of 4,011 entities and 2,075 relationships from 141 reports, and reports F1 scores of 84% to 95% across four extraction subtasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gold-standard reliability is the load-bearing assumption: with one primary annotator, documented label disputes, and no inter-annotator agreement reported, the reported F1 scores may measure agreement with a subjective reference rather than STIX correctness.","rationale":"The paper has genuine strengths: a new manually curated dataset, vendor- and campaign-level separation between train and test, transparent error analysis, and a clearly assistive system design with human verification after each task. I considered the per-module versus end-to-end evaluation gap as an alternative concern; it is real, but because AZERG is explicitly designed as a semi-automated tool with human verification at every stage, the model-only F1s are not the final product metric, and an end-to-end human-in-the-loop study would be a useful but different contribution. The gold-standard concern is more load-bearing because every headline number inherits it. The paper itself provides the raw material for the concern by listing the exact areas where its own annotators disagreed, yet it supplies no quantitative reliability measure. If the reference labels are not reproducible, then the 84-95% F1 figures and the 2-25% improvement margins may not describe correct STIX extraction; they may only describe agreement with one expert's interpretation. The proposed re-annotation test would settle this directly. This does not change the reader's CONDITIONAL verdict: the concern is substantial but addressable, and the paper already flags much of the relevant uncertainty, so UNCHANGED is appropriate.","tokens_in":19778,"tokens_out":6072,"duration_ms":71425,"concrete_test":"Have two additional STIX experts independently annotate the same 11 test reports (or a random sample of at least 50 test passages) using the paper's annotation guidelines and the original report text. Compute pairwise inter-annotator agreement for entity spans, entity types, and relationship types. Then recompute AZERG-MixTask F1 for T1-T4 against each expert's labels and against a majority-vote gold standard. If inter-annotator agreement is below roughly 0.8 or the model's F1 shifts by more than about 2 points relative to the published gold, the reported scores are not robust evidence for the stated extraction accuracy. If the labels are stable, the conditional verdict can stand.","verdict_should_be":"UNCHANGED","load_bearing_attack":"All four headline F1 scores (84.43%, 88.49%, 95.47%, 84.60%) are computed against a gold standard whose creation is described in Section V. The paper states that one offensive-security expert produced the annotations and that two others reviewed them, and it explicitly documents substantive disagreements: uses vs. exploits, downloads vs. drops, owns vs. hosts, and malware vs. threat-actor identities for ambiguous names. Despite this, no inter-annotator agreement statistic (e.g., Cohen's kappa or pairwise F1) is reported. Since the models are fine-tuned and evaluated against exactly these labels, the F1 scores conflate conformity to the STIX standard with agreement with one expert's judgment. If a second expert would label even a modest fraction of test entities or relationships differently, the absolute scores and the claimed 2-25% margins over GPT4o become unstable; T3's advantage is only 2.3% and is particularly vulnerable. The AnnoCTRPlus conversion (regex-added indicators, fuzzy string matching, removal of Concepts) is a related noise source for 82% of training passages, but because the test split is AZERG Data only, the more direct threat to the central claim is subjectivity in the test-set reference. Without quantitative reliability evidence, the 'real-world scenarios' claim rests on unverified label stability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AZERG, a four-stage pipeline that extracts STIX-aligned entities and relationships from unstructured threat analysis reports. The task is decomposed into entity detection (T1), entity type identification (T2), related-pair detection (T3), and relationship-type identification (T4). The authors contribute a manually annotated dataset of 4,011 entities and 2,075 relationships from 141 reports, fine-tune Mistral-7B-Instruct-v0.3 with LoRA on this dataset, and report F1 scores of 84.43%, 88.49%, 95.47%, and 84.60% for T1-T4, claiming 2-25% improvements over GPT4o, Mistral, and prior methods. The evaluation uses a vendor-disjoint train/test split to reduce contamination risk.","tokens_in":20075,"tokens_out":6180,"duration_ms":66478,"significance":"If the reported performance is reliable, AZERG would be a practically useful assistive tool for STIX report generation, and the curated dataset would be a valuable community resource. The paper's strengths include the dataset construction effort, the deliberate vendor-level train/test split, the module-level comparison against several open and closed models, the hyperparameter search, and a concrete error analysis with illustrative report examples. However, the central claim is currently supported only by module-level metrics computed against a gold standard whose reliability is not quantified, and no end-to-end evaluation is reported. These gaps are load-bearing for the abstract's 'real-world scenarios' claim and for the stated improvements over baselines.","major_comments":[{"comment":"The gold standard is the load-bearing reference for every reported F1 score. Section V states that one offensive-security expert annotated all entities and relationships, that two other experts reviewed them, and that disputes remained in exactly the areas where the models are evaluated (e.g., uses vs. exploits, downloads vs. drops, malware vs. threat actor). No inter-annotator agreement statistic is reported. As a result, the reported F1 values and the 2-25% margins over GPT4o conflate agreement with one expert's subjective judgment with correctness against the STIX standard. Please report a re-annotation study on a sample of the test split with Cohen's kappa or pairwise F1, separated for entity tasks (T1/T2) and relationship tasks (T3/T4), and state how disputed labels were adjudicated in the final gold standard.","section":"Section V (Annotation Process; Train and Test Splits)"},{"comment":"The tables report module-level scores. For T2, T3, and T4, precision equals recall for every model, which is the signature of single-label classification on a fixed instance set; this indicates the modules were evaluated on gold-standard entities, types, or pairs rather than on the output of the preceding module. The text confirms this by saying that AZERG-MixTask 'in combination with AZERG-S-T3 is expected to deliver optimal system performance.' Consequently, the reported F1 values do not measure end-to-end extraction on full reports, and the abstract's 'real-world scenarios' claim is not supported. Please state explicitly what inputs were used for each module in Tables III-VI and provide an end-to-end evaluation in which T1 predictions (with and without human verification) are propagated through T2, T3, and T4, reporting final entity and relationship F1.","section":"Section VII.B (Evaluation Protocol; Tables III-VI)"},{"comment":"No variance or significance measures accompany the F1 scores. The test set is 11 reports / 108 passages, and the smallest claimed advantage over GPT4o is 2.3 percentage points on T3 (95.47 vs. 93.15). With no confidence intervals, per-report breakdowns, or significance tests, the existence of the claimed 2-25% improvements is not established. Please add bootstrap confidence intervals or paired significance tests over reports.","section":"Section VII.B (Tables III-VI)"}],"minor_comments":[{"comment":"The text describing AnnoCTRPlus contains typos ('trough', 'AnnotCTR', 'enttities') and the base dataset name is spelled inconsistently; please normalize the spelling throughout.","section":"Section V"},{"comment":"The caption refers to 'misclassified relationship types in T2', but T2 is entity-type identification; the caption should say 'entity types'.","section":"Figure 7 caption"},{"comment":"Table V lists 'GliREL [46]' but reference [46] is GliNER; please clarify whether GliREL is a separate method and provide the correct citation.","section":"Table V"},{"comment":"The Index Terms line still contains the IEEE template placeholders 'component, formatting, style, styling, insert'; replace with actual keywords.","section":"Index Terms"},{"comment":"Footnotes 1 and 4 state that source code and models 'will be available', while the Conclusion states 'we made our collected datasets available'; align these statements with the actual release status.","section":"Conclusion and Footnotes"},{"comment":"The T3 error analysis reports '6 relationships were missed, while 47 were incorrectly identified'; given that the test set contains 565 relations, clarify whether these counts refer to relation pairs, sections, or something else, and how they relate to the reported recall of 0.9893.","section":"Section VII.C.3"}],"recommendation":"major_revision","confidential_remarks":"This is a solid systems and dataset paper whose evaluation currently overclaims. The two decisive fixes are inter-annotator agreement on a sample of the test set and an end-to-end pipeline evaluation with error propagation. If those are provided, the paper could be acceptable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know two things about this paper before reading it. First, the contribution is real: they built, to my knowledge, the largest STIX-aligned dataset for entity and relationship extraction (4,011 entities, 2,075 relationships across 141 reports) and a four-task LLM pipeline with task-specific and mixed-task fine-tuning. Second, the F1 scores in the abstract (84.43, 88.49, 95.47, 84.60) are per-module, not end-to-end. The paper never evaluates the full pipeline from raw report to STIX JSON, and its own error analysis shows errors amplify in later stages.\n\nWhat the paper does well: the dataset construction is careful, with vendor-level train/test separation and campaign-level decontamination. The task decomposition into detection, typing, pair finding, and relation labeling is sensible, and the comparison set is broad—GPT4o, Mistral, GliNER, EXTRACTOR, AttaKG. The error analysis is genuinely useful and honest, with concrete examples of entity confusion and relationship misclassification. The authors do not oversell full automation; they frame it as an assistive tool.\n\nThe soft spots: (1) No inter-annotator agreement is reported. The annotation process involved one primary expert and two reviewers, and the paper documents real disputes (uses vs. exploits, downloads vs. drops, malware vs. threat actor). Without a kappa or pairwise F1 on a subset, the absolute numbers could shift if a second expert labeled the test set. This is the load-bearing weakness. It doesn't sink the paper, but it tempers the 'real-world accuracy' claim. (2) No end-to-end metric. The four tasks are evaluated in isolation, and the paper's own error amplification examples show that T1 errors cascade into T3/T4. (3) The test set is small (108 passages from 11 reports) with no confidence intervals. (4) The code and models are linked on Hugging Face, but the text says 'will be available'—a minor clarity issue.\n\nThe gold-standard concern is real but not fatal. The authors disclose the ambiguity and provide confusion matrices and error examples, which gives the reader a sense of where the annotations are brittle. I would want a second-annotator reliability substudy or at least a subset double-annotated.\n\nFor peer review, send it out. The dataset alone is worth refereeing, and the pipeline is a reasonable baseline for future work. The revision should include inter-annotator agreement and an end-to-end evaluation, even on a handful of reports.","headline":"A genuinely useful STIX-aligned dataset and a sensible four-task LLM pipeline; the headline F1s are per-module, the gold standard lacks inter-annotator reliability data, but the core contribution deserves peer review.","tokens_in":20595,"tokens_out":2514,"would_cite":true,"duration_ms":25537,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned LLMs, organized into a four-stage pipeline called AZERG, extract STIX-standard entities and relationships from full threat reports at 84–95% F1, beating off-the-shelf models and prior pipelines by 2 to 25 percentage points.","keywords":["STIX","threat intelligence","large language models","named entity recognition","relationship extraction","cyber threat intelligence","fine-tuning","knowledge graph"],"falsifier":"Have two independent STIX experts re-annotate a random sample of the test passages or twenty full reports, measure inter-annotator agreement, and score AZERG against the resulting consensus; if the system's F1 against the new consensus falls below roughly 75% on T1 and T4, or if the experts disagree with the original labels as often as the model does, the central accuracy claim would not replicate.","tokens_in":19610,"feed_emoji":"🛡️","tokens_out":8555,"duration_ms":82466,"temperature":0.7,"pith_summary":"The paper argues that generating STIX-structured threat intelligence from unstructured security reports, a task now performed largely by hand, can be made fast and reliable enough for analyst-facing automation. To show this, it introduces AZERG, a pipeline that decomposes extraction into four subtasks—entity detection, entity type identification, related-pair detection, and relationship type identification—and fine-tunes a general-purpose large language model separately for each, with human verification between stages. Using a new dataset of 4,011 entities and 2,075 relationships annotated across 141 real APT reports, the system reaches F1 scores of 84.43%, 88.49%, 95.47%, and 84.60% on the four tasks, improvements of 2 to 25 percentage points over off-the-shelf LLMs and earlier pipelines. The payoff, if the claim holds, is that security analysts can produce STIX-compliant reports with far less manual effort and that relationship objects—currently all but absent from public STIX sharing—can be generated routinely.","feed_headline":"Fine-tuned LLMs turn threat reports into STIX data at 84%+ F1","feed_subtitle":"AZERG's four-task pipeline outdoes off-the-shelf LLMs and prior tools by 2 to 25 points.","key_machinery":"The load-bearing mechanism is task decomposition plus constrained prompting: each stage is a separately fine-tuned model whose prompt carries the STIX entity-type definitions (T1/T2) or the STIX relationship matrix with \"not related\" and \"not sure\" options (T3/T4), so the model decides only within the standard's allowed pairs. Indicators of compromise are caught first by regular-expression extractors, a section splitter preserves report context, and human verification gates each stage; the final compiled output is STIX-compliant JSON for TAXII sharing. The dataset itself—141 real reports annotated to 4,011 entities and 2,075 relationships—is the resource that makes the fine-tuning possible.","core_discovery":"The central discovery is that general-purpose instruction-tuned LLMs, which on their own score barely above 60% F1 on this material, become reliable extractors of STIX objects once they are continually fine-tuned on STIX-aligned annotations and prompted with the standard's definitions and relationship matrix. The resulting mixed-task model detects entities (T1), assigns their STIX types (T2: 88.49%), finds related pairs (T3: 95.47%), and chooses the correct relationship type (T4: 84.60%), with entity detection at 84.43% F1. Error analysis shows the remaining failures are concentrated in contextual ambiguity—malware-versus-threat-actor naming, tool-versus-infrastructure labeling, alias resolution—and in semantically overlapping relationship types such as \"uses\" versus \"communicates-with,\" the same distinctions that human annotators disputed. The paper therefore positions AZERG not as a replacement for analysts but as a verified assistive layer in STIX report generation.","pith_inferences":["Alias and coreference resolution is the most promising next step: names like Shuckworm/Gamaredon caused six missed relationships in a single passage, so linking aliases before or during extraction could lift T3/T4 recall without retraining the models.","The same four-task decomposition could be transferred to other threat-intelligence formats such as MISP or MAEC, since the bottleneck is not the format but the mapping from noisy expert prose to a controlled vocabulary.","A vision-language extension that reads charts, timelines, and code-snippet figures could recover entities the text-only pipeline misses, especially in reports where attack timelines are drawn rather than described.","Constrained decoding against the STIX relationship matrix, or a lightweight verifier that rejects invalid SRO pairs, would likely compress the T4 confusion between \"uses\" and \"communicates-with\" at inference time."],"forward_implications":["Security analysts can use AZERG as a semi-automated assistant: each stage's output can be added to, deleted from, or corrected before it feeds the next stage, so the final STIX JSON is verified at every step.","STIX relationship objects, which a nine-year sample of public STIX data found to be essentially unused, would become cheap to generate, improving both the coverage and timeliness of shared threat intelligence.","The released dataset of 4,011 STIX entities and 2,075 relationships gives the research community a large, STIX-aligned benchmark for entity and relation extraction.","Because the mixed-task model nearly matches the task-specific models at one-third the compute, deployment within an analyst workflow is practical with current hardware.","Error analysis directs future effort to alias resolution and the disambiguation of semantically overlapping relationship types, the two areas that most limit recall in T1 and precision in T4."],"supporting_citations":[{"why":"Supplies the STIX entity and relationship definitions (SDOs, SCOs, SROs, relationship types) that structure the annotations and prompts.","marker":"[3]"},{"why":"Defines TAXII, the sharing protocol that the generated STIX output is designed to feed.","marker":"[5]"},{"why":"Provides LoRA, the parameter-efficient fine-tuning method used to adapt the base LLM to each task.","marker":"[18]"},{"why":"Documents that public STIX data lacks relationship objects, motivating the need for automated SRO extraction.","marker":"[23]"},{"why":"Supplies the AnnoCTR dataset, which the authors revise and expand into AnnoCTRPlus for training and testing.","marker":"[26]"},{"why":"AttaKG, a prior technique-knowledge-graph extraction pipeline used as a comparable baseline for entity and relation extraction.","marker":"[30]"},{"why":"EXTRACTOR, a provenance-graph NLP pipeline used as a baseline that the four-task models must beat.","marker":"[39]"},{"why":"GliNER, a generalist transformer-based NER model used as a baseline for entity detection.","marker":"[46]"}],"fun_headline_variants":["LLMs auto-extract STIX entities and relations at 84%+ F1","AZERG: fine-tuned LLMs turn threat reports into STIX data","STIX extraction goes automatic: LLMs hit 84-95% F1","Four-step LLM pipeline beats prior tools by 2-25 points","From threat text to STIX: LLMs achieve 84%+ F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ground-truth annotations are treated as correct even though they were produced by one primary expert, reviewed by two others, and involved documented disagreements on exactly the boundary cases where the model errs; if those labels are wrong, the reported F1 scores measure agreement with one expert's reading rather than accurate STIX extraction.","fun_headline_variants_meta":{"raw":{"variants":["LLMs auto-extract STIX entities and relations at 84%+ F1","AZERG: fine-tuned LLMs turn threat reports into STIX data","STIX extraction goes automatic: LLMs hit 84-95% F1","Four-step LLM pipeline beats prior tools by 2-25 points","From threat text to STIX: LLMs achieve 84%+ F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1446,"prompt_tokens":1074,"completion_tokens":372,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":690,"completion_tokens_details":{"reasoning_tokens":282}},"tokens_in":690,"tokens_out":372,"duration_ms":3997,"temperature":1.0,"reasoning_tokens":282,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:07:00.882233+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two independent STIX experts re-annotate a random sample of the test passages or twenty full reports, measure inter-annotator agreement, and score AZERG against the resulting consensus; if the system's F1 against the new consensus falls below roughly 75% on T1 and T4, or if the experts disagree with the original labels as often as the model does, the central accuracy claim would not replicate.","supporting_citations":[{"cited_title":"Introduction to stix","cited_arxiv_id":null,"evidence_quote":"Supplies the STIX entity and relationship definitions (SDOs, SCOs, SROs, relationship types) that structure the annotations and prompts."},{"cited_title":"Taxii - trusted automated exchange of indicator information,","cited_arxiv_id":null,"evidence_quote":"Defines TAXII, the sharing protocol that the generated STIX output is designed to feed."},{"cited_title":"Sharing cyber threat intelligence: Does it really help?","cited_arxiv_id":null,"evidence_quote":"Documents that public STIX data lacks relationship objects, motivating the need for automated SRO extraction."},{"cited_title":"Attackg: Constructing technique knowledge graph from cyber threat intelligence reports,","cited_arxiv_id":null,"evidence_quote":"AttaKG, a prior technique-knowledge-graph extraction pipeline used as a comparable baseline for entity and relation extraction."},{"cited_title":"Extractor: Extracting attack behavior from threat reports,","cited_arxiv_id":null,"evidence_quote":"EXTRACTOR, a provenance-graph NLP pipeline used as a baseline that the four-task models must beat."},{"cited_title":"Gliner: Generalist model for named entity recognition using bidirectional transformer,","cited_arxiv_id":null,"evidence_quote":"GliNER, a generalist transformer-based NER model used as a baseline for entity detection."}],"review_version":1}