Pith. sign in

REVIEW 3 major objections 6 minor 47 references

From Text to Actionable Intelligence: Automating STIX Entity and Relationship Extraction

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuned LLMs, organized into a four-stage pipeline called AZERG, extract STIX-standard entities and relationships from full threat reports at 84–95% F1, beating off-the-shelf models and prior pipelines by 2 to 25 percentage points.

desk verdict A genuinely useful STIX-aligned dataset and a sensible four-task LLM pipeline; the headline F1s are per-module, the gold standard lacks inter-annotator reliability data, but the core contribution deserves peer review. read the letter →

arxiv 2507.16576 v1 pith:64WU6MDE submitted 2025-07-22 cs.CR

Ahmed Lekssays , Husrev Taha Sencar , Ting Yu This is my paper
classification cs.CR
keywords STIXthreatintelligencelargelanguagemodelsnamedentityrecognitionrelationshipextractioncyberfine-tuningknowledgegraph
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that generating STIX-structured threat intelligence from unstructured security reports, a task now performed largely by hand, can be made fast and reliable enough for analyst-facing automation. To show this, it introduces AZERG, a pipeline that decomposes extraction into four subtasks—entity detection, entity type identification, related-pair detection, and relationship type identification—and fine-tunes a general-purpose large language model separately for each, with human verification between stages. Using a new dataset of 4,011 entities and 2,075 relationships annotated across 141 real APT reports, the system reaches F1 scores of 84.43%, 88.49%, 95.47%, and 84.60% on the four tasks, improvements of 2 to 25 percentage points over off-the-shelf LLMs and earlier pipelines. The payoff, if the claim holds, is that security analysts can produce STIX-compliant reports with far less manual effort and that relationship objects—currently all but absent from public STIX sharing—can be generated routinely.

What carries the argument

The load-bearing mechanism is task decomposition plus constrained prompting: each stage is a separately fine-tuned model whose prompt carries the STIX entity-type definitions (T1/T2) or the STIX relationship matrix with "not related" and "not sure" options (T3/T4), so the model decides only within the standard's allowed pairs. Indicators of compromise are caught first by regular-expression extractors, a section splitter preserves report context, and human verification gates each stage; the final compiled output is STIX-compliant JSON for TAXII sharing. The dataset itself—141 real reports annotated to 4,011 entities and 2,075 relationships—is the resource that makes the fine-tuning possible.

What would settle it

Have two independent STIX experts re-annotate a random sample of the test passages or twenty full reports, measure inter-annotator agreement, and score AZERG against the resulting consensus; if the system's F1 against the new consensus falls below roughly 75% on T1 and T4, or if the experts disagree with the original labels as often as the model does, the central accuracy claim would not replicate.

Watch

Extended reading notes

Core claim

The central discovery is that general-purpose instruction-tuned LLMs, which on their own score barely above 60% F1 on this material, become reliable extractors of STIX objects once they are continually fine-tuned on STIX-aligned annotations and prompted with the standard's definitions and relationship matrix. The resulting mixed-task model detects entities (T1), assigns their STIX types (T2: 88.49%), finds related pairs (T3: 95.47%), and chooses the correct relationship type (T4: 84.60%), with entity detection at 84.43% F1. Error analysis shows the remaining failures are concentrated in contextual ambiguity—malware-versus-threat-actor naming, tool-versus-infrastructure labeling, alias resolution—and in semantically overlapping relationship types such as "uses" versus "communicates-with," the same distinctions that human annotators disputed. The paper therefore positions AZERG not as a replacement for analysts but as a verified assistive layer in STIX report generation.

Load-bearing premise

The ground-truth annotations are treated as correct even though they were produced by one primary expert, reviewed by two others, and involved documented disagreements on exactly the boundary cases where the model errs; if those labels are wrong, the reported F1 scores measure agreement with one expert's reading rather than accurate STIX extraction.

Editorial extensions

If this is right

  • Security analysts can use AZERG as a semi-automated assistant: each stage's output can be added to, deleted from, or corrected before it feeds the next stage, so the final STIX JSON is verified at every step.
  • STIX relationship objects, which a nine-year sample of public STIX data found to be essentially unused, would become cheap to generate, improving both the coverage and timeliness of shared threat intelligence.
  • The released dataset of 4,011 STIX entities and 2,075 relationships gives the research community a large, STIX-aligned benchmark for entity and relation extraction.
  • Because the mixed-task model nearly matches the task-specific models at one-third the compute, deployment within an analyst workflow is practical with current hardware.
  • Error analysis directs future effort to alias resolution and the disambiguation of semantically overlapping relationship types, the two areas that most limit recall in T1 and precision in T4.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Alias and coreference resolution is the most promising next step: names like Shuckworm/Gamaredon caused six missed relationships in a single passage, so linking aliases before or during extraction could lift T3/T4 recall without retraining the models.
  • The same four-task decomposition could be transferred to other threat-intelligence formats such as MISP or MAEC, since the bottleneck is not the format but the mapping from noisy expert prose to a controlled vocabulary.
  • A vision-language extension that reads charts, timelines, and code-snippet figures could recover entities the text-only pipeline misses, especially in reports where attack timelines are drawn rather than described.
  • Constrained decoding against the STIX relationship matrix, or a lightweight verifier that rejects invalid SRO pairs, would likely compress the T4 confusion between "uses" and "communicates-with" at inference time.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces AZERG, a four-stage pipeline that extracts STIX-aligned entities and relationships from unstructured threat analysis reports. The task is decomposed into entity detection (T1), entity type identification (T2), related-pair detection (T3), and relationship-type identification (T4). The authors contribute a manually annotated dataset of 4,011 entities and 2,075 relationships from 141 reports, fine-tune Mistral-7B-Instruct-v0.3 with LoRA on this dataset, and report F1 scores of 84.43%, 88.49%, 95.47%, and 84.60% for T1-T4, claiming 2-25% improvements over GPT4o, Mistral, and prior methods. The evaluation uses a vendor-disjoint train/test split to reduce contamination risk.

Significance. If the reported performance is reliable, AZERG would be a practically useful assistive tool for STIX report generation, and the curated dataset would be a valuable community resource. The paper's strengths include the dataset construction effort, the deliberate vendor-level train/test split, the module-level comparison against several open and closed models, the hyperparameter search, and a concrete error analysis with illustrative report examples. However, the central claim is currently supported only by module-level metrics computed against a gold standard whose reliability is not quantified, and no end-to-end evaluation is reported. These gaps are load-bearing for the abstract's 'real-world scenarios' claim and for the stated improvements over baselines.

major comments (3)
  1. [Section V (Annotation Process; Train and Test Splits)] The gold standard is the load-bearing reference for every reported F1 score. Section V states that one offensive-security expert annotated all entities and relationships, that two other experts reviewed them, and that disputes remained in exactly the areas where the models are evaluated (e.g., uses vs. exploits, downloads vs. drops, malware vs. threat actor). No inter-annotator agreement statistic is reported. As a result, the reported F1 values and the 2-25% margins over GPT4o conflate agreement with one expert's subjective judgment with correctness against the STIX standard. Please report a re-annotation study on a sample of the test split with Cohen's kappa or pairwise F1, separated for entity tasks (T1/T2) and relationship tasks (T3/T4), and state how disputed labels were adjudicated in the final gold standard.
  2. [Section VII.B (Evaluation Protocol; Tables III-VI)] The tables report module-level scores. For T2, T3, and T4, precision equals recall for every model, which is the signature of single-label classification on a fixed instance set; this indicates the modules were evaluated on gold-standard entities, types, or pairs rather than on the output of the preceding module. The text confirms this by saying that AZERG-MixTask 'in combination with AZERG-S-T3 is expected to deliver optimal system performance.' Consequently, the reported F1 values do not measure end-to-end extraction on full reports, and the abstract's 'real-world scenarios' claim is not supported. Please state explicitly what inputs were used for each module in Tables III-VI and provide an end-to-end evaluation in which T1 predictions (with and without human verification) are propagated through T2, T3, and T4, reporting final entity and relationship F1.
  3. [Section VII.B (Tables III-VI)] No variance or significance measures accompany the F1 scores. The test set is 11 reports / 108 passages, and the smallest claimed advantage over GPT4o is 2.3 percentage points on T3 (95.47 vs. 93.15). With no confidence intervals, per-report breakdowns, or significance tests, the existence of the claimed 2-25% improvements is not established. Please add bootstrap confidence intervals or paired significance tests over reports.
minor comments (6)
  1. [Section V] The text describing AnnoCTRPlus contains typos ('trough', 'AnnotCTR', 'enttities') and the base dataset name is spelled inconsistently; please normalize the spelling throughout.
  2. [Figure 7 caption] The caption refers to 'misclassified relationship types in T2', but T2 is entity-type identification; the caption should say 'entity types'.
  3. [Table V] Table V lists 'GliREL [46]' but reference [46] is GliNER; please clarify whether GliREL is a separate method and provide the correct citation.
  4. [Index Terms] The Index Terms line still contains the IEEE template placeholders 'component, formatting, style, styling, insert'; replace with actual keywords.
  5. [Conclusion and Footnotes] Footnotes 1 and 4 state that source code and models 'will be available', while the Conclusion states 'we made our collected datasets available'; align these statements with the actual release status.
  6. [Section VII.C.3] The T3 error analysis reports '6 relationships were missed, while 47 were incorrectly identified'; given that the test set contains 565 relations, clarify whether these counts refer to relation pairs, sections, or something else, and how they relate to the reported recall of 0.9893.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluation is anchored to manually annotated, report-disjoint test data, and the regex/IoC overlap does not make the reported F1 scores equivalent to their training inputs.

full rationale

AZERG's central claim is an empirical extraction pipeline: fine-tuned LLMs are evaluated on a held-out test split (11 AZERG Data reports, vendor-disjoint from training) against manually curated STIX-aligned annotations. No headline F1 score is derived from the model's own output by construction, and no fitted parameter is renamed as a prediction. The one candidate loop is the use of regexes: AnnoCTRPlus indicator entities were added automatically using regular expressions (Sec. V), and T1 also begins by identifying indicators with IoC extractors before applying the model (Sec. IV-A, VII-A). This is not a circular reduction, because the test split is AZERG Data, which was manually curated and expert-annotated, not AnnoCTRPlus; the regex pre-extraction is an external component applied uniformly to AZERG and all compared baselines. The paper's self-citations ([24], [25]) support related-work enumeration and a data-curation decision about explicitly mentioned ATT&CK techniques; they are not load-bearing for the extraction claim. The most significant validity threat is gold-standard subjectivity and the absence of reported inter-annotator agreement, but that is a measurement-quality concern, not circularity. No specific equation or construction in the paper equates a claimed prediction with its input, so the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The system's performance rests on tuned hyperparameters (learning rate, temperature, top-p, and unreported LoRA rank) and on domain assumptions about STIX applicability, context sufficiency, and the quality of the annotated gold standard. No new physical or formal entities are invoked.

free parameters (4)
  • learning rate = 1e-4
    Chosen via grid search over {1e-6, 1e-5, 1e-4, 1e-3} to maximize F1 across T1-T4 (Fig. 12, Appendix A).
  • temperature = 0.7
    Selected by grid search over {0.0, 0.4, 0.7, 1.0, 1.5}; used for all models and all tasks (Appendix A).
  • top-p = 0.1
    Selected by grid search over {0.1, 0.4, 0.7, 0.95, 1.0}; conservative sampling gave the best F1 (Appendix A).
  • LoRA rank and alpha = unspecified
    LoRA is used via LLaMA-Factory (Sec. VII-A) but the rank and alpha values are not reported, leaving an unreported design choice that affects performance.
assumptions (4)
  • domain assumption STIX entity type list, relationship matrix, and object definitions are correctly applied to the annotated data.
    The whole annotation and evaluation rests on the authors' interpretation of the STIX standard (Sec. II-B); disputes in annotation show the mapping is not mechanical.
  • domain assumption The four-task sequential decomposition (T1 to T4) preserves enough context and information to recover the full STIX graph.
    Each module consumes upstream outputs; the paper does not evaluate the joint pipeline or quantify error propagation end-to-end (Sec. IV-A, Sec. VII-B).
  • domain assumption Section-level segmentation of reports (test passages average 244 words) provides sufficient context for entity and relationship inference.
    Longer passages are split or overlapped when headings are absent; context length is not analyzed as a controlled variable (Sec. IV-A).
  • domain assumption AnnoCTR's existing annotations can be reliably mapped to STIX by removing 'Concept' entities, adding missing entities, and fuzzy-matching consolidated sentences.
    This revision step determines most of the training data (744 of 914 passages) and could inject label noise (Sec. V).

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Text to Actionable Intelligence: Automating STIX Entity and Relationship Extraction." pith.science (2026). https://pith.science/paper/64WU6MDE

@misc{pith2026250716576,
  author       = {Pith},
  title        = {Pith review of: From Text to Actionable Intelligence: Automating STIX Entity and Relationship Extraction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/64WU6MDE}},
  note         = {Machine review of arXiv:2507.16576}
}
read the original abstract

Sharing methods of attack and their effectiveness is a cornerstone of building robust defensive systems. Threat analysis reports, produced by various individuals and organizations, play a critical role in supporting security operations and combating emerging threats. To enhance the timeliness and automation of threat intelligence sharing, several standards have been established, with the Structured Threat Information Expression (STIX) framework emerging as one of the most widely adopted. However, generating STIX-compatible data from unstructured security text remains a largely manual, expert-driven process. To address this challenge, we introduce AZERG, a tool designed to assist security analysts in automatically generating structured STIX representations. To achieve this, we adapt general-purpose large language models for the specific task of extracting STIX-formatted threat data. To manage the complexity, the task is divided into four subtasks: entity detection (T1), entity type identification (T2), related pair detection (T3), and relationship type identification (T4). We apply task-specific fine-tuning to accurately extract relevant entities and infer their relationships in accordance with the STIX specification. To address the lack of training data, we compiled a comprehensive dataset with 4,011 entities and 2,075 relationships extracted from 141 full threat analysis reports, all annotated in alignment with the STIX standard. Our models achieved F1-scores of 84.43% for T1, 88.49% for T2, 95.47% for T3, and 84.60% for T4 in real-world scenarios. We validated their performance against a range of open- and closed-parameter models, as well as state-of-the-art methods, demonstrating improvements of 2-25% across tasks.

Figures

Figures reproduced from arXiv: 2507.16576 by the authors.

Figure 1
Figure 1. Overview of AZERG workflow. The example is taken from Buhti Report [1]. is utilized to represent threat information revealed that only 75% of the defined basic object types in the standard (i.e., domain objects) are utilized within their dataset. Moreover, indicator objects—containing basic indicators of compromise, such as malicious file hashes or URL strings—accounted for over 90% of the data. Notably, the analysi… view at source ↗
Figure 2
Figure 2. Example text from a real-world report [10] is shown [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The architecture of AZERG. The cyber espionage group known as “Shadow Dragon”, no￾torious for targeting financial institutions, recently conducted a campaign against ”Global Finance Corp.” This institution, pri￾marily located in United States, was targeted using a custom malware variant called “SerpentStealth”. The initial infection vec￾tor leveraged the “Spearphishing Attachment” technique (MITRE ATT&CK T1566.001),… view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Sample excerpt and its corresponding STIX graph [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Entity type distribution in our dataset. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Relationship type distribution in our dataset. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Confusion matrix for Relationship Type Identifica [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 7
Figure 7. Figure 7: Confusion matrix showing misclassified relationship [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 11
Figure 11. Figure 11: Example of an error from the V3G4 report [28] encountered during tasks T3 and T4. designed for their automated identification. For entities without standardized formats, like malware family names, adversary groups, and identities, neural-named entity recognition model…
Figure 10
Figure 10. Figure 10: Example of an error from the Shuckworm report [4] encountered during tasks T3 and T4. In [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 13
Figure 13. Figure 13: F1-score heatmap for T1 [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 14
Figure 14. Figure 14: F1-score heatmap for T2. We present the results of our grid search for the optimal setting for the temperature and top-p values [PITH_FULL_IMAGE:figures/full_fig_p014_14.png]
Figure 15
Figure 15. Figure 15: F1-score heatmap for T3 [PITH_FULL_IMAGE:figures/full_fig_p015_15.png]
Figure 17
Figure 17. Figure 17: The employed prompt for T1. # Instruction: You are a helpful threat intelligence analyst. Your task is to assign a STIX entity type to the given Entity in the input. To help you, here is a list of the possible STIX entity types. [STIX ENTITY TYPES] Choose STIX ENTITY …
Figure 18
Figure 18. Figure 18: The employed prompt for T2 [PITH_FULL_IMAGE:figures/full_fig_p015_18.png]
Figure 19
Figure 19. Figure 19: The employed prompt for T3. # Instruction: You are a helpful threat intelligence analyst. Your task is to identify the label of the relationship between the source entity and the target entity in the provided text passage. To help you, we provide all the possible rela…
Figure 20
Figure 20. Figure 20: The employed prompt for T4 [PITH_FULL_IMAGE:figures/full_fig_p016_20.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 34 canonical work pages

  1. [1]

    Buhti: New ransomware operation relies on repurposed payloads,

    “Buhti: New ransomware operation relies on repurposed payloads,” https://symantec-enterprise-blogs.security.com/blogs/threat-intelligence/ buhti-ransomware, [Accessed 05-04-2025]

    work page 2025

  2. [2]

    Cranefly: Threat actor uses previously unseen techniques and tools in stealthy campaign,

    “Cranefly: Threat actor uses previously unseen techniques and tools in stealthy campaign,” https://symantec-enterprise-blogs.security.com/ blogs/threat-intelligence/cranefly-new-tools-technique-geppei-danfuan, [Accessed 05-04-2025]

    work page 2025

  3. [3]

    Introduction to stix

    “Introduction to stix.” [Online]. Available: https://oasis-open.github.io/ cti-documentation/stix/intro

    work page

  4. [4]

    Shuckworm: Inside russia’s relentless cyber campaign against ukraine,

    “Shuckworm: Inside russia’s relentless cyber campaign against ukraine,” https://symantec-enterprise-blogs.security.com/blogs/threat-intelligence/ shuckworm-russia-ukraine-military, [Accessed 05-04-2025]

    work page 2025

  5. [5]

    Taxii - trusted automated exchange of indicator information,

    “Taxii - trusted automated exchange of indicator information,” https: //taxiiproject.github.io/, accessed: March 22, 2025

    work page 2025

  6. [6]

    Cyberentrel: Joint extraction of cyber entities and relations using deep learning,

    K. Ahmed, S. K. Khurshid, and S. Hina, “Cyberentrel: Joint extraction of cyber entities and relations using deep learning,” Computers & Security, vol. 136, p. 103579, 2024

    work page 2024

  7. [7]

    Looking beyond iocs: Automatically extracting attack patterns from external cti,

    M. T. Alam, D. Bhusal, Y . Park, and N. Rastogi, “Looking beyond iocs: Automatically extracting attack patterns from external cti,” in Pro- ceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses , 2023, pp. 92–108

    work page 2023

  8. [8]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language mod- els are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

Show all 47 references
  1. [9]

    Sparks of artificial general intelligence: Early experiments with gpt-4,

    S. Bubeck, V . Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Ka- mar, P. Lee, Y . T. Lee, Y . Li, S. Lundberg et al. , “Sparks of artificial general intelligence: Early experiments with gpt-4,” arXiv preprint arXiv:2303.12712, 2023

    2023 arXiv

  2. [10]

    Y . H. Chang and S. Singh. (2024, Jul.) Dodgebox: A deep dive into the updated arsenal of apt41 — part 1. Accessed: 2025-04-23. [Online]. Available: https://www.zscaler.com/blogs/security-research/ dodgebox-deep-dive-updated-arsenal-apt41-part-1

    2024

  3. [11]

    The use of large language models (llm) for cyber threat intelligence (cti) in cybercrime forums,

    V . Clairoux-Trepanier, I.-M. Beauchamp, E. Ruellan, M. Paquet- Clouston, S.-O. Paquette, and E. Clay, “The use of large language models (llm) for cyber threat intelligence (cti) in cybercrime forums,” arXiv preprint arXiv:2408.03354, 2024

    2024 arXiv

  4. [12]

    At- tacker: Towards enhancing cyber-attack attribution with a named entity recognition dataset,

    P. Deka, S. Rajapaksha, R. Rani, A. Almutairi, and E. Karafili, “At- tacker: Towards enhancing cyber-attack attribution with a named entity recognition dataset,” arXiv preprint arXiv:2408.05149 , 2024

    2024 arXiv

  5. [13]

    Cti-hal: A human-annotated dataset for cyber threat intelligence anal- ysis,

    S. Della Penna, R. Natella, V . Orbinato, L. Parracino, and L. Pianese, “Cti-hal: A human-annotated dataset for cyber threat intelligence anal- ysis,” arXiv preprint arXiv:2504.05866 , 2025

    2025 arXiv

  6. [14]

    Threatkg: A threat knowledge graph for automated open- source cyber threat intelligence gathering and management,

    P. Gao, X. Liu, E. Choi, S. Ma, X. Yang, Z. Ji, Z. Zhang, and D. Song, “Threatkg: A threat knowledge graph for automated open- source cyber threat intelligence gathering and management,” arXiv preprint arXiv:2212.10388, 2022

    2022 arXiv

  7. [15]

    A system for automated open-source threat intelligence gathering and management,

    P. Gao, X. Liu, E. Choi, B. Soman, C. Mishra, K. Farris, and D. Song, “A system for automated open-source threat intelligence gathering and management,” in Proceedings of the 2021 International Conference on Management of Data , 2021, pp. 2716–2720

    2021

  8. [16]

    Enabling efficient cyber threat hunting with cyber threat intelligence,

    P. Gao, F. Shao, X. Liu, X. Xiao, Z. Qin, F. Xu, P. Mittal, S. R. Kulkarni, and D. Song, “Enabling efficient cyber threat hunting with cyber threat intelligence,” in 2021 IEEE 37th International Conference on Data Engineering (ICDE) . IEEE, 2021, pp. 193–204

    2021

  9. [17]

    V oice of the soc analyst,

    E. Hinchy, “V oice of the soc analyst,” 2022

    2022

  10. [18]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685 , 2021

    2021 arXiv

  11. [19]

    Ctikg: Llm-powered knowledge graph construc- tion from cyber threat intelligence,

    L. Huang and X. Xiao, “Ctikg: Llm-powered knowledge graph construc- tion from cyber threat intelligence,” in First Conference on Language Modeling, 2024

    2024

  12. [20]

    Ttpdrill: Automatic and accurate extraction of threat actions from unstructured text of cti sources,

    G. Husari, E. Al-Shaer, M. Ahmed, B. Chu, and X. Niu, “Ttpdrill: Automatic and accurate extraction of threat actions from unstructured text of cti sources,” in Proceedings of the 33rd Annual Computer Security Applications Conference , ser. ACSAC ’17. New York, NY , USA: Associ...

    2017

  13. [21]

    Stixgen - a novel framework for automatic generation of structured cyber threat information,

    Z. Iqbal, Z. Anwar, and R. Mumtaz, “Stixgen - a novel framework for automatic generation of structured cyber threat information,” in 2018 International Conference on Frontiers of Information Technology (FIT) , 2018, pp. 241–246

    2018

  14. [22]

    A knowledge base question answering system for cyber threat knowledge acquisition,

    Z. Ji, E. Choi, and P. Gao, “A knowledge base question answering system for cyber threat knowledge acquisition,” in2022 IEEE 38th International Conference on Data Engineering (ICDE). IEEE, 2022, pp. 3158–3161

    2022

  15. [23]

    Sharing cyber threat intelligence: Does it really help?

    B. Jin, E. Kim, H. Lee, E. Bertino, D. Kim, and H. Kim, “Sharing cyber threat intelligence: Does it really help?” in The Network and Distributed System Security (NDSS) Symposium 2024 , 2024

    2024

  16. [24]

    Ttpxhunter: Actionable threat intelligence extraction as ttps form finished cyber threat reports,

    U. Kumarasinghe, Lekssays, Ahmed, H. T. Sencar, S. Boughorbel, and P. Nakov, “Ttpxhunter: Actionable threat intelligence extraction as ttps form finished cyber threat reports,” in The 19th ACM ASIA Conference on Computer and Communications Security (ACM AsiaCCS 2024) , 2024

    2024

  17. [25]

    Semantic ranking for automated adversarial technique annotation in security text,

    U. Kumarasinghe, A. Lekssays, H. T. Sencar, S. Boughorbel, C. Elvitigala, and P. Nakov, “Semantic ranking for automated adversarial technique annotation in security text,” in Proceedings of the 19th ACM Asia Conference on Computer and Communications Security, ser. ASIA CCS ’24...

    2024

  18. [26]

    Annoctr: A dataset for detecting and linking entities, tactics, and techniques in cyber threat reports,

    L. Lange, M. M ¨uller, G. H. Torbati, D. Milchevski, P. Grau, S. Pu- jari, and A. Friedrich, “Annoctr: A dataset for detecting and linking entities, tactics, and techniques in cyber threat reports,” arXiv preprint arXiv:2404.07765, 2024

    2024 arXiv

  19. [27]

    Automated retrieval of att&ck tactics and techniques for cyber threat reports,

    V . Legoy, M. Caselli, C. Seifert, and A. Peter, “Automated retrieval of att&ck tactics and techniques for cyber threat reports,” arXiv preprint arXiv:2004.14322, 2020

    2004 arXiv

  20. [28]

    Mirai variant v3g4 targets iot devices,

    C. Lei, “Mirai variant v3g4 targets iot devices,” https://unit42. paloaltonetworks.com/mirai-variant-v3g4/, [Accessed 05-04-2025]

    2025

  21. [29]

    Attackg: Constructing technique knowledge graph from cyber threat intelligence reports,

    Z. Li, J. Zeng, Y . Chen, and Z. Liang, “Attackg: Constructing technique knowledge graph from cyber threat intelligence reports,” in European Symposium on Research in Computer Security . Springer, 2022, pp. 589–609

    2022

  22. [30]

    Attackg: Constructing technique knowledge graph from cyber threat intelligence reports,

    ——, “Attackg: Constructing technique knowledge graph from cyber threat intelligence reports,” in Computer Security – ESORICS 2022: 27th European Symposium on Research in Computer Security, Copenhagen, Denmark, September 26–30, 2022, Proceedings, Part I. Berlin, Heidelberg: Spr...

    2022 doi

  23. [31]

    Large language models are un- reliable for cyber threat intelligence,

    E. Mezzi, F. Massacci, and K. Tuma, “Large language models are un- reliable for cyber threat intelligence,” arXiv preprint arXiv:2503.23175, 2025

    2025

  24. [32]

    Llms in biomedicine: A study on clinical named entity recognition,

    M. Monajatipoor, J. Yang, J. Stremmel, M. Emami, F. Mohaghegh, M. Rouhsedaghat, and K.-W. Chang, “Llms in biomedicine: A study on clinical named entity recognition,” arXiv preprint arXiv:2404.07376, 2024

    2024 arXiv

  25. [33]

    Noise contrastive estimation- based matching framework for low-resource security attack pattern recognition,

    T. Nguyen, N. ˇSrndi´c, and A. Neth, “Noise contrastive estimation- based matching framework for low-resource security attack pattern recognition,” in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics . Association for ...

    2024

  26. [34]

    Automatic genera- tion of maec and stix standards for android malware threat intelligence,

    J. Park, L. Nguyen Vu, G. Bencivengo, and S. Jung, “Automatic genera- tion of maec and stix standards for android malware threat intelligence,” KSII Transactions on Internet and Information Systems (TIIS) , vol. 14, no. 8, pp. 3420–3436, 2020

    2020

  27. [35]

    Project, MISP - Malware Information Sharing Platform & Threat Sharing , 2023, accessed: March 22, 2025

    M. Project, MISP - Malware Information Sharing Platform & Threat Sharing , 2023, accessed: March 22, 2025. [Online]. Available: https://www.misp-project.org/

    2023

  28. [36]

    Project, STIX - Structured Threat Information Expression , 2023, accessed: March 22, 2025

    S. Project, STIX - Structured Threat Information Expression , 2023, accessed: March 22, 2025. [Online]. Available: https://oasis-open. github.io/cti-documentation/

    2023

  29. [37]

    Ttpxhunter: Actionable threat intelligence extraction as ttps form finished cyber threat reports,

    N. Rani, B. Saha, V . Maurya, and S. K. Shukla, “Ttpxhunter: Actionable threat intelligence extraction as ttps form finished cyber threat reports,” arXiv preprint arXiv:2403.03267 , 2024

    2024 arXiv

  30. [38]

    Cskg4apt: A cy- bersecurity knowledge graph for advanced persistent threat organization attribution,

    Y . Ren, Y . Xiao, Y . Zhou, Z. Zhang, and Z. Tian, “Cskg4apt: A cy- bersecurity knowledge graph for advanced persistent threat organization attribution,” IEEE Transactions on Knowledge and Data Engineering , 2022

    2022

  31. [39]

    Extractor: Extracting attack behavior from threat reports,

    K. Satvat, R. Gjomemo, and V . Venkatakrishnan, “Extractor: Extracting attack behavior from threat reports,” in 2021 IEEE European Symposium on Security and Privacy (EuroS&P) . IEEE, 2021, pp. 598–615

    2021

  32. [40]

    Bert rediscovers the classical nlp pipeline,

    I. Tenney, D. Das, and E. Pavlick, “Bert rediscovers the classical nlp pipeline,” arXiv preprint arXiv:1905.05950 , 2019

    1905 arXiv

  33. [41]

    Cti ant: Hunting for chinese threat intelligence,

    C.-E. Tsai, C.-L. Yang, and C.-K. Chen, “Cti ant: Hunting for chinese threat intelligence,” in 2020 IEEE International Conference on Big Data (Big Data). IEEE, 2020, pp. 1847–1852

    2020

  34. [42]

    Knowcti: Knowledge-based cyber threat intelligence entity and relation extrac- tion,

    G. Wang, P. Liu, J. Huang, H. Bin, X. Wang, and H. Zhu, “Knowcti: Knowledge-based cyber threat intelligence entity and relation extrac- tion,” Computers & Security , vol. 141, p. 103824, 2024

    2024

  35. [43]

    Gpt-ner: Named entity recognition via large language models,

    S. Wang, X. Sun, X. Li, R. Ouyang, F. Wu, T. Zhang, J. Li, and G. Wang, “Gpt-ner: Named entity recognition via large language models,” arXiv preprint arXiv:2304.10428, 2023

    2023 arXiv

  36. [44]

    Intelex: A llm-driven attack-level threat intelligence extraction framework,

    M. Xu, H. Wang, J. Liu, Y . Lin, C. X. Y . Liu, H. W. Lim, and J. S. Dong, “Intelex: A llm-driven attack-level threat intelligence extraction framework,” arXiv preprint arXiv:2412.10872 , 2024

    2024

  37. [45]

    Tim: threat context-enhanced ttp intelligence mining on unstructured threat data,

    Y . You, J. Jiang, Z. Jiang, P. Yang, B. Liu, H. Feng, X. Wang, and N. Li, “Tim: threat context-enhanced ttp intelligence mining on unstructured threat data,” Cybersecurity, vol. 5, no. 1, p. 3, 2022

    2022

  38. [46]

    Gliner: Generalist model for named entity recognition using bidirectional transformer,

    U. Zaratiana, N. Tomeh, P. Holat, and T. Charnois, “Gliner: Generalist model for named entity recognition using bidirectional transformer,” 2023

    2023

  39. [47]

    Itirel: Joint entity and relation extraction for internet of things threat intelligence,

    F. Zhu, Z. Cheng, P. Li, and H. Xu, “Itirel: Joint entity and relation extraction for internet of things threat intelligence,” IEEE Internet of Things Journal, 2024. APPENDIX A IMPACT OF HYPERPARAMETERS ON PERFORMANCE We investigated the effects of varying learning rates, tem-...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.