Pith. sign in

REVIEW 4 major objections 5 minor 47 references

LRCTI: A Large Language Model-Based Framework for Multi-Step Evidence Retrieval and Reasoning in Cyber Threat Intelligence Credibility Verification

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a three-stage LLM pipeline—summarization, adaptive multi-step evidence retrieval, and prompt-based natural language inference—can verify cyber threat intelligence credibility with F1-Macro 90.9% and F1-Micro 93.6%…

desk verdict A plausible LLM pipeline for CTI credibility checks, but the headline F1 gains on the in-house benchmark are uninterpretable until the authors rule out corpus contamination. read the letter →

arxiv 2507.11310 v1 pith:WFCTK5YV submitted 2025-07-15 cs.CR cs.CL

Fengxiao Tang , Huan Li , Ming Zhao , Zongzong Wu , Shisong Peng , Tao Yin This is my paper
classification cs.CRcs.CL
keywords cyberthreatintelligencecredibilityverificationlargelanguagemodelsmulti-stepevidenceretrievalnaturalinferencefakenewsdetectionretrieval-augmentedgenerationexplainability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes LRCTI, a framework that automates credibility verification of cyber threat intelligence (CTI) reports by turning each report into a short claim, iteratively retrieving supporting evidence from a CTI-specific corpus, and using an LLM to judge whether the evidence entails, contradicts, or leaves uncertain the claim. The authors argue that this three-step design overcomes the limits of older methods that rely on handcrafted features or single-pass classification, which are brittle on noisy, heterogeneous intelligence and opaque in their reasoning. On the CTI-200 dataset, LRCTI reaches 90.9% F1-Macro and 93.6% F1-Micro, and on the PolitiFact fake-news benchmark it reaches 86.1% on both scores, beating ten baselines by a wide margin. If the results hold, the framework offers a scalable, explainable way to vet threat reports before they are trusted in defense operations.

What carries the argument

The load-bearing mechanism is the iterative retrieve-and-reason loop. The summarization module scores each sentence by $w(s_i) = \text{ROUGE1-F1}(s_i, S) + \alpha \cdot p(s_i) + \beta \cdot q(s_i)$, where $p(s_i)$ is positional weight and $q(s_i)$ is semantic similarity to other sentences, then feeds the top-$m$ sentences to an LLM prompt that produces a one-sentence claim. The retrieval module uses a dense retriever to get the top-$k$ paragraphs and a progressive LLM-guided selection that refines the paragraph set, then selects the top-$j$ evidence sentences by similarity to the claim. A re-retrieval trigger fires when the reasoner returns NEI or when the adjusted confidence $\lambda = \theta \times \text{Conf}$ falls below $\theta = 0.7$; the loop then incorporates previously established evidence and generates updated sub-claims to drive another retrieval round. The reasoning module is a prompt-based NLI step that labels the claim and writes an explanation. The whole pipeline runs on GPT-3.5-turbo with temperature 0.

What would settle it

Re-run LRCTI after deleting every CTI-200 document (and all documents with high lexical or semantic overlap with the test reports) from the retrieval corpus. If F1-Macro on CTI-200 drops by more than a few absolute points, the original results were substantially driven by corpus leakage rather than by the multi-step retrieval and NLI reasoning.

Watch

Extended reading notes

Core claim

The central claim is that a retrieval-augmented LLM pipeline built specifically for the cyber threat domain can match or exceed the credibility-verification performance of dedicated evidence-based classifiers while producing human-readable justifications. Concretely, LRCTI first summarizes a verbose CTI report into a single verifiable claim, then runs an adaptive multi-step retrieval loop over an in-house CTI corpus—retrieving candidate paragraphs, filtering them with LLM feedback, selecting sentence-level evidence, and re-retrieving when the evidence is judged irrelevant, insufficient, or low-confidence—and finally classifies the claim as credible, incredible, or not-enough-information with a confidence score. The authors report that this design outperforms six evidence-based baselines (DeClarE, HAN, EHIAN, MAC, GET, MUSER) and four LLM-based baselines (GPT-3.5-turbo, ChatGLM2-6B, WEBGLM-2B, ProgramFC) by over 5% F1 on CTI-200 and by roughly 10% on PolitiFact. The paper further argues that the multi-step retrieval loop and the summarization module are both essential, since ablating either one degrades performance, and that the retrieved evidence is interpretable enough for human analysts to agree with the model 81.6% of the time in a user study.

Load-bearing premise

The load-bearing premise is that the in-house CTI retrieval corpus does not contain the CTI-200 test reports or near-duplicates of them; if it does, the reported F1 gains could come from retrieving the answer itself rather than from evidence-based credibility reasoning.

Editorial extensions

If this is right

  • If LRCTI is correct, CTI credibility verification can be fully automated and made explainable, reducing reliance on slow, inconsistent manual analyst review.
  • The same summarization–retrieval–NLI loop transfers to general fake-news detection, as the PolitiFact results suggest, and likely to other evidence-grounded verification tasks with domain-specific corpora.
  • The paper's ablation and retrieval-step experiments indicate that two to three retrieval rounds are optimal, giving a practical cost–accuracy tradeoff for deployment.
  • Because the final output includes retrieved evidence fragments and a structured justification alongside the label, LRCTI could be integrated into analyst workflows as an assistive triage tool.
  • The confidence score with an overconfidence correction factor provides a natural threshold for flagging claims that need human review rather than automatic acceptance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported gains on CTI-200 may be inflated if the in-house retrieval corpus contains the very reports being verified or near-duplicates produced by the same authors; a held-out corpus experiment is the only way to rule out this leakage channel.
  • The paper compares corpus domains only in terms of retrieval speed and similarity (Table IV), not downstream F1, so the contribution of corpus quality versus the multi-step loop itself is not separately quantified.
  • An implicit extension is that the overconfidence factor $\theta$ and the confidence threshold $\lambda < 0.5$ are tunable levers that will shift the precision–recall operating point; real deployments would need to calibrate them per domain.
  • The design suggests a general recipe for low-resource verification: summarize the artifact into a claim, build or scrape a focused corpus, and iterate retrieval until the LLM's NLI confidence crosses a threshold—an approach that should be tested on non-cyber domains like medical or financial claims.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes LRCTI, an LLM-based framework for cyber threat intelligence (CTI) credibility verification. The pipeline consists of an LLM-based text summarization module that converts a report into a claim, an adaptive multi-step evidence retrieval module that iteratively collects paragraphs and sentences from a purpose-built CTI corpus, and a prompt-based NLI module that outputs a credibility label (credible/incredible) with a confidence score and an explanation. The authors evaluate LRCTI on CTI-200 and PolitiFact, reporting F1-Macro and F1-Micro scores of 90.9% and 93.6% on CTI-200, outperforming ten baselines, and they include ablations, a user study, and a comparison of retrieval efficiency between their CTI corpus and Wikipedia.

Significance. If the reported gains were trustworthy, LRCTI would be a meaningful advance in CTI credibility verification, combining retrieval-augmented LLM reasoning with interpretable justifications, and the paper would be among the first to apply multi-step evidence retrieval in this domain. The paper also attempts a user study to show the practical utility of the generated evidence, which is a positive feature. However, the experimental evidence as presented is compromised by a likely corpus contamination issue, an incorrect metric equivalence claim, and a lack of statistical rigor; the credibility of the central performance claim therefore cannot be assessed from the manuscript. No code or dataset is released, which further limits verification of the results.

major comments (4)
  1. [Section III-D and Section IV-A.1] The in-house CTI corpus described in Section III-D is built by scraping cybersecurity sources, while the main benchmark CTI-200 (Section IV-A.1, reference [41]) was introduced by the same research group. The paper never states that the CTI-200 test documents or their near-duplicates were excluded from the corpus; the deduplication described in Section III-D is applied only to the corpus itself. If a test report (or a very similar report) is in the retrieval corpus, the retriever can surface the original text as evidence, and the NLI module can classify from that text, making the F1 gains in Table II reflect answer memorization rather than credibility reasoning. This is a load-bearing confound that invalidates the attribution of the reported improvements to the multi-step retrieval mechanism.
  2. [Section IV-B and Tables II-III] The statement that 'precision is numerically equivalent to F1-Macro in our experimental setting' is incorrect for standard binary classification: F1-Macro is the average of the per-class F1 scores, which is not generally equal to either global or per-class precision. This raises the possibility that the reported headline metrics were miscalculated, and it is not a mere presentation issue because the paper's central claim depends on these numbers. Additionally, the abstract claims 'over 5%' improvement in both F1-Micro and F1-Macro, but Table II shows F1-Micro improves from 0.894 (MUSER) to 0.936 (LRCTI), a relative gain of only 4.7%.
  3. [Section IV-A.3 and Eq. (5)] The experimental section gives no error bars, no significance tests, and no description of train/test splits or repeated runs, so the reported single-point gains over baselines cannot be statistically assessed. Moreover, the re-retrieval threshold is internally inconsistent: Section IV-A.3 sets the overconfidence factor to θ=0.7, but the text in Section III-B and the caption of Figure 2 state that re-retrieval triggers when λ<0.5, where λ=θ×Conf (Eq. 5); with θ=0.7, λ<0.5 corresponds to Conf<0.714, not Conf<0.5. This inconsistency affects the operational meaning of the multi-step mechanism and should be clarified before the reported behavior can be reproduced.
  4. [Section IV-E and Table IV] The RQ5 comparison between the Wikipedia corpus and the CTI corpus uses the same potentially confounded CTI corpus, so it does not address the leakage concern raised in Major Comment 1. In addition, Table IV only reports retrieval time and a similarity score r(c,e); it does not report downstream verification accuracy, so it cannot support the claim that the CTI corpus improves credibility verification performance. The relevance of this experiment to the paper's central claim is therefore unclear.
minor comments (5)
  1. [Section III-D] The text about corpus construction is duplicated verbatim: the paragraph beginning 'The overall corpus construction process' and the subsequent paragraphs through '...related IOCs' appear twice in Section III-D.
  2. [Section IV-B] The statement of 'relative improvements of 5% and 10%' on CTI-200 and PolitiFact is inconsistent with the values in Tables II and III; for example, the F1-Macro improvement on PolitiFact from MUSER (0.732) to LRCTI (0.861) is about 17.6%, not 10%.
  3. [Section IV-C and Figure 7] The claim that 'beyond three retrieval steps, performance gains plateau and may even degrade slightly' appears to be based on a single curve with no error bars or numerical values; please provide the underlying data or a statistical analysis.
  4. [Section IV-D] The user study reports 'significantly outperforms' based on 8 participants and 40 claims, but no significance test, inter-annotator agreement, or confidence intervals are provided; the word 'significantly' is not supported.
  5. [Author list] The author affiliations contain typos: '4rd' and '5rd' should be '4th' and '5th'.

Circularity Check

2 steps flagged · score 6.0 of 10

CTI-200 gains and the RQ5 corpus-efficiency comparison may reduce to self-retrieval because the in-house retrieval corpus is scraped from the same cybersecurity sources as the authors' own CTI-200 benchmark, with no stated exclusion of test documents.

  1. self definitional [Section III-D (Corpus construction) + Section IV-A.1 (Datasets) + Eqs. (3)-(4)]
    "We developed a custom web scraping system specifically targeting multiple authoritative data sources in the cybersecurity domain, such as the ATT&CK framework, Wikipedia, and prominent security blogs. ... The CTI-200 dataset [41] contains cyber threat intelligence reports collected from multiple cybersecurity websites."

    The retrieval corpus is scraped from the same class of sources ('cybersecurity websites', 'prominent security blogs') as the CTI-200 benchmark, which is cited to the authors' own prior work [41]. No passage states that CTI-200 test reports or near-duplicates are excluded from the corpus. If a test report is present, the dense retriever in Eq. (3) ranks that report's own paragraphs as top evidence, so the evidence set Es in Eq. (4) is the input document itself; the NLI then classifies a claim against its own source text. The reported CTI-200 F1 gains (Table II) would then reflect self-retrieval of the answer, not multi-step credibility reasoning.

  2. self definitional [Section IV-E (RQ 5, Table IV)]
    "In this experiment, we compared the retrieval efficiency of the Wikipedia corpus and the Cyber Threat Intelligence (CTI) corpus using a common set of claims derived from CTI reports."

    The queries are 'derived from CTI reports', and the CTI corpus was built by scraping cybersecurity sources such as security blogs (Section III-D). With no stated exclusion of the source reports, the corpus can contain the very reports from which the queries were generated; retrieving from that corpus then returns the query's own source text, producing the high r(c,e)=0.895 in Table IV. The comparison to Wikipedia is therefore not a test of domain relevance but of whether the corpus contains the answer documents, so the RQ5 result is guaranteed by construction rather than measuring retrieval quality.

full rationale

The paper's core method (summarization, iterative retrieval, and NLI) is not definitionally circular: the claim C is generated from the input report, evidence Es is retrieved from a corpus, and the NLI output is a separate classification. However, the empirical validation on the CTI-specific side is confounded by corpus/benchmark overlap. The retrieval corpus is constructed in-house by scraping cybersecurity sources, while CTI-200 is the authors' own benchmark collected from the same class of sources; the paper never states that test documents were excluded from the corpus. Under that construction, the 'evidence' retrieved for a CTI-200 claim can be the very report being verified, so the F1 improvements on CTI-200 (Table II) and the higher relevance score for the CTI corpus in RQ5 (Table IV) can reduce to self-retrieval rather than independent evidence reasoning. The external PolitiFact benchmark provides some independent support for the framework, which prevents a score of 8 or above, but the CTI-specific headline result and the RQ5 comparison are partially circular by construction.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several unreported hyperparameters and on an in-house corpus whose relationship to the test set is not disclosed. None of these are standard constants; all are chosen by the authors for this pipeline. The 'established evidence pool' and 'updated claims' are procedural artifacts, not invented entities.

free parameters (6)
  • alpha = not reported
    Weight for positional score p(si) in sentence selection, Eq. (1); the value is never given or tuned in the paper.
  • beta = not reported
    Weight for semantic similarity q(si) in Eq. (1); the value is never given or tuned in the paper.
  • theta = 0.7
    Overconfidence correction factor in Eq. (5), set to 0.7 in the experiments with no sensitivity analysis.
  • top-k paragraph count = 50 candidates per claim; final k unspecified
    The system retrieves 50 candidate paragraphs per claim, but the number of paragraphs finally selected for evidence is not specified.
  • top-j evidence sentence count = not reported
    The number of sentences selected into the evidence set Es is never stated.
  • lambda threshold = 0.5
    Confidence threshold for triggering re-retrieval, fixed without sensitivity analysis.
assumptions (4)
  • ad hoc to paper The in-house CTI corpus is representative of CTI-200 test claims and does not contain the test documents or their near-duplicates.
    Corpus construction (Section III-D) and CTI-200 (Section IV-A.1, reference [41]) are described separately, but no overlap exclusion is stated.
  • domain assumption ROUGE1-F1 of a sentence against the whole document is a valid importance signal for selecting key sentences.
    Used in Eq. (1) without any validation or comparison to alternative sentence-scoring methods.
  • domain assumption LLM confidence scores, after multiplying by theta, are calibrated well enough to gate re-retrieval.
    Eq. (5) assumes a fixed discount factor makes the confidence signal reliable, but no calibration study is reported.
  • domain assumption Labels in CTI-200 and PolitiFact are ground truth.
    No annotation quality assurance, inter-annotator agreement, or label audit is reported for either dataset.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LRCTI: A Large Language Model-Based Framework for Multi-Step Evidence Retrieval and Reasoning in Cyber Threat Intelligence Credibility Verification." pith.science (2026). https://pith.science/paper/WFCTK5YV

@misc{pith2026250711310,
  author       = {Pith},
  title        = {Pith review of: LRCTI: A Large Language Model-Based Framework for Multi-Step Evidence Retrieval and Reasoning in Cyber Threat Intelligence Credibility Verification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WFCTK5YV}},
  note         = {Machine review of arXiv:2507.11310}
}
read the original abstract

Verifying the credibility of Cyber Threat Intelligence (CTI) is essential for reliable cybersecurity defense. However, traditional approaches typically treat this task as a static classification problem, relying on handcrafted features or isolated deep learning models. These methods often lack the robustness needed to handle incomplete, heterogeneous, or noisy intelligence, and they provide limited transparency in decision-making-factors that reduce their effectiveness in real-world threat environments. To address these limitations, we propose LRCTI, a Large Language Model (LLM)-based framework designed for multi-step CTI credibility verification. The framework first employs a text summarization module to distill complex intelligence reports into concise and actionable threat claims. It then uses an adaptive multi-step evidence retrieval mechanism that iteratively identifies and refines supporting information from a CTI-specific corpus, guided by LLM feedback. Finally, a prompt-based Natural Language Inference (NLI) module is applied to evaluate the credibility of each claim while generating interpretable justifications for the classification outcome. Experiments conducted on two benchmark datasets, CTI-200 and PolitiFact show that LRCTI improves F1-Macro and F1-Micro scores by over 5%, reaching 90.9% and 93.6%, respectively, compared to state-of-the-art baselines. These results demonstrate that LRCTI effectively addresses the core limitations of prior methods, offering a scalable, accurate, and explainable solution for automated CTI credibility verification

Figures

Figures reproduced from arXiv: 2507.11310 by the authors.

Figure 1
Figure 1. Comparison of Different CTI Credibility Verifica [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Our framework consists of three key steps: (a) [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Illustration of progressive paragraph selection during [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Re-retrieval trigger conditions. Irrelevant evidence refers to content unrelated to the claim. Insufficient evidence indicates a lack of support for definitive conclusions. Low confidence suggests uncertainty in the current judgment, requiring further evidence retrieva…
Figure 5
Figure 5. Figure 5: The construction process based on the CTI corpus. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Ablation Study Results: LRCTI represents the perfor [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Ablation experiments with different numbers of re [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Explanation of the Case Study: Text with a colored [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Text summary(ts) Prompt Instructions: You will be provided with the following response format: Explanation: [Explanation of why the judgment was made.] Answer: [0 / 1 / 2] - If Answer is 0 or 1: simply return the number as is. - If Answer is 2, also include the followi…
Figure 10
Figure 10. Figure 10: Response correction(rc) Prompt [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: Step check(sc) Prompt [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 28 canonical work pages

  1. [41]

    Kgv: Integrating large language models with knowledge graphs for cyber threat intelligence credibility assessment,

    Z. Wu, F. Tang, M. Zhao, and Y . Li, “Kgv: Integrating large language models with knowledge graphs for cyber threat intelligence credibility assessment,” 2024. [Online]. Available: https://arxiv.org/abs/2408.08088

    arXiv 2024

  2. [1]

    Towards improved cyber security information sharing,

    L. Dandurand and O. S. Serrano, “Towards improved cyber security information sharing,” in 2013 5th International Conference on Cyber Conflict (CYCON 2013) . IEEE, 2013, pp. 1–16

    work page 2013

  3. [2]

    Stiocs: Active learning-based semi-supervised training framework for ioc extraction,

    B. Tang, X. Li, J. Wang, W. Ge, Z. Yu, and T. Lin, “Stiocs: Active learning-based semi-supervised training framework for ioc extraction,” Computers and Electrical Engineering , vol. 112, p. 108981, 2023

    work page 2023

  4. [3]

    Tweetcred: Real- time credibility assessment of content on twitter,

    A. Gupta, P. Kumaraguru, C. Castillo, and P. Meier, “Tweetcred: Real- time credibility assessment of content on twitter,” in Social Informatics: 6th International Conference, SocInfo 2014, Barcelona, Spain, November 11-13, 2014. Proceedings 6 . Springer, 2014, pp. 228–243

    work page 2014

  5. [4]

    Sentence-level evidence embedding for claim verification with hierarchical attention networks

    J. Ma, W. Gao, S. Joty, and K.-F. Wong, “Sentence-level evidence embedding for claim verification with hierarchical attention networks.” Association for Computational Linguistics, 2019

    work page 2019

  6. [5]

    Declare: Debunking fake news and false claims using evidence-aware deep learning,

    K. Popat, S. Mukherjee, A. Yates, and G. Weikum, “Declare: Debunking fake news and false claims using evidence-aware deep learning,” arXiv preprint arXiv:1809.06416, 2018

    arXiv 2018

  7. [6]

    Hierarchical Multi-head Attentive Network for Evidence-aware Fake News Detection

    N. V o and K. Lee, “Hierarchical multi-head attentive network for evidence-aware fake news detection,” arXiv preprint arXiv:2102.02680 , 2021

    work page Pith review arXiv 2021

  8. [7]

    Evidence-aware fake news detection with graph neural networks,

    W. Xu, J. Wu, Q. Liu, S. Wu, and L. Wang, “Evidence-aware fake news detection with graph neural networks,” in Proceedings of the ACM web conference 2022, 2022, pp. 2501–2510

    work page 2022

Show all 47 references
  1. [8]

    Muser: A multi-step evidence retrieval enhancement framework for fake news detection,

    H. Liao, J. Peng, Z. Huang, W. Zhang, G. Li, K. Shu, and X. Xie, “Muser: A multi-step evidence retrieval enhancement framework for fake news detection,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2023, pp. 4461–4472

    2023

  2. [9]

    A survey on large language model (llm) security and privacy: The good, the bad, and the ugly,

    Y . Yao, J. Duan, K. Xu, Y . Cai, Z. Sun, and Y . Zhang, “A survey on large language model (llm) security and privacy: The good, the bad, and the ugly,” High-Confidence Computing, p. 100211, 2024

    2024

  3. [10]

    Wikicoref: An english coreference- annotated corpus of wikipedia articles,

    A. Ghaddar and P. Langlais, “Wikicoref: An english coreference- annotated corpus of wikipedia articles,” in Proceedings of the Tenth Inter- national Conference on Language Resources and Evaluation (LREC’16) , 2016, pp. 136–142

    2016

  4. [11]

    A survey on technical threat intelligence in the age of sophisticated cyber attacks,

    W. Tounsi and H. Rais, “A survey on technical threat intelligence in the age of sophisticated cyber attacks,” Computers & security , vol. 72, pp. 212–233, 2018

    2018

  5. [12]

    ‘towards a methodology for evaluating threat intelligence feeds,

    P. Pawli´nski, P. Kijewski, and A. D. Kompanek, “‘towards a methodology for evaluating threat intelligence feeds,” 2016

    2016

  6. [13]

    Pure: Generating quality threat intelligence by clustering and correlating osint,

    R. Azevedo, I. Medeiros, and A. Bessani, “Pure: Generating quality threat intelligence by clustering and correlating osint,” in 2019 18th IEEE International Conference On Trust, Security And Privacy In Computing And Communications/13th IEEE International Conference On Big Data...

    2019

  7. [14]

    A multi-source threat intelligence confidence value evaluation method based on machine learning,

    H. T. Hansheng LIU, T. L. Mingxia BO, Jianfeng NIU, and L. LI, “A multi-source threat intelligence confidence value evaluation method based on machine learning,” Telecommunications Science, vol. 36, no. 1, pp. 119–126, 2020

    2020

  8. [15]

    Sharing machine learning models as indicators of compromise for cyber threat intelligence,

    D. Preuveneers and W. Joosen, “Sharing machine learning models as indicators of compromise for cyber threat intelligence,” Journal of Cybersecurity and Privacy , vol. 1, no. 1, pp. 140–163, 2021

    2021

  9. [16]

    Security threat model under internet of things using deep learning and edge analysis of cyberspace governance,

    Z. Li, Y . Ge, J. Guo, M. Chen, and J. Wang, “Security threat model under internet of things using deep learning and edge analysis of cyberspace governance,” International Journal of System Assurance Engineering and Management, vol. 13, no. Suppl 3, pp. 1164–1176, 2022

    2022

  10. [17]

    Scoring model for iocs by combining open intelligence feeds to reduce false positives,

    J. Ermerins, N. van Noort, J. de Novais Marques, and L. Velasco, “Scoring model for iocs by combining open intelligence feeds to reduce false positives,” in Proc. of Security and Network Engineering , 2020

    2020

  11. [18]

    Large language models struggle to learn long-tail knowledge,

    N. Kandpal, H. Deng, A. Roberts, E. Wallace, and C. Raffel, “Large language models struggle to learn long-tail knowledge,” in International Conference on Machine Learning . PMLR, 2023, pp. 15 696–15 707

    2023

  12. [19]

    Prompt-and-align: prompt-based social alignment for few-shot fake news detection,

    J. Wu, S. Li, A. Deng, M. Xiong, and B. Hooi, “Prompt-and-align: prompt-based social alignment for few-shot fake news detection,” in Proceedings of the 32nd ACM International Conference on Information and Knowledge Management , 2023, pp. 2726–2736

    2023

  13. [20]

    Explainable claim verification via knowledge- grounded reasoning with large language models,

    H. Wang and K. Shu, “Explainable claim verification via knowledge- grounded reasoning with large language models,” arXiv preprint arXiv:2310.05253, 2023

    2023 arXiv

  14. [21]

    Active retrieval augmented generation,

    Z. Jiang, F. F. Xu, L. Gao, Z. Sun, Q. Liu, J. Dwivedi-Yu, Y . Yang, J. Callan, and G. Neubig, “Active retrieval augmented generation,” arXiv preprint arXiv:2305.06983, 2023

    2023 arXiv

  15. [22]

    Replug: Retrieval-augmented black-box language models,

    W. Shi, S. Min, M. Yasunaga, M. Seo, R. James, M. Lewis, L. Zettle- moyer, and W.-t. Yih, “Replug: Retrieval-augmented black-box language models,” arXiv preprint arXiv:2301.12652 , 2023

    2023 arXiv

  16. [23]

    Factcheck- gpt: End-to-end fine-grained document-level fact-checking and correction of llm output,

    Y . Wang, R. G. Reddy, Z. M. Mujahid, A. Arora, A. Rubashevskii, J. Geng, O. M. Afzal, L. Pan, N. Borenstein, A. Pillai et al., “Factcheck- gpt: End-to-end fine-grained document-level fact-checking and correction of llm output,” arXiv preprint arXiv:2311.09000 , 2023

    2023 arXiv

  17. [24]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24 824–24 837, 2022

    2022

  18. [25]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,” arXiv preprint arXiv:2210.03629, 2022

    2022 arXiv

  19. [26]

    Tree of thoughts: Deliberate problem solving with large language models,

    S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y . Cao, and K. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  20. [27]

    In-context retrieval-augmented language models,

    O. Ram, Y . Levine, I. Dalmedigos, D. Muhlgay, A. Shashua, K. Leyton- Brown, and Y . Shoham, “In-context retrieval-augmented language models,” Transactions of the Association for Computational Linguistics , vol. 11, pp. 1316–1331, 2023

    2023

  21. [28]

    Improving language models by retrieving from trillions of tokens,

    S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driessche, J.-B. Lespiau, B. Damoc, A. Clark et al. , “Improving language models by retrieving from trillions of tokens,” in International conference on machine learning . PMLR, 2022, pp. 2206– 2240

    2022

  22. [29]

    Webglm: Towards an efficient web-enhanced question answering system with human preferences,

    X. Liu, H. Lai, H. Yu, Y . Xu, A. Zeng, Z. Du, P. Zhang, Y . Dong, and J. Tang, “Webglm: Towards an efficient web-enhanced question answering system with human preferences,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2023, pp. 4549–4560

    2023

  23. [30]

    Pegasus: Pre-training with extracted gap-sentences for abstractive summarization,

    J. Zhang, Y . Zhao, M. Saleh, and P. Liu, “Pegasus: Pre-training with extracted gap-sentences for abstractive summarization,” in International conference on machine learning . PMLR, 2020, pp. 11 328–11 339

    2020

  24. [31]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  25. [32]

    Improving language understanding by generative pre-training,

    A. Radford, K. Narasimhan, T. Salimans, I. Sutskever et al., “Improving language understanding by generative pre-training,” 2018

    2018

  26. [33]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of machine learning research , vol. 21, no. 140, pp. 1–67, 2020

    2020

  27. [34]

    Dense passage retrieval for open-domain question answering,

    V . Karpukhin, B. O˘guz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih, “Dense passage retrieval for open-domain question answering,” arXiv preprint arXiv:2004.04906 , 2020

    2004 arXiv

  28. [35]

    One embedder, any task: Instruction-finetuned text embeddings,

    H. Su, W. Shi, J. Kasai, Y . Wang, Y . Hu, M. Ostendorf, W.-t. Yih, N. A. Smith, L. Zettlemoyer, and T. Yu, “One embedder, any task: Instruction-finetuned text embeddings,” arXiv preprint arXiv:2212.09741, 2022

    2022 arXiv

  29. [36]

    Palm: Scaling language modeling with pathways,

    A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann et al. , “Palm: Scaling language modeling with pathways,” Journal of Machine Learning Research, vol. 24, no. 240, pp. 1–113, 2023

    2023

  30. [37]

    Adversarial retriever-ranker for dense text retrieval,

    H. Zhang, Y . Gong, Y . Shen, J. Lv, N. Duan, and W. Chen, “Adversarial retriever-ranker for dense text retrieval,” arXiv preprint arXiv:2110.03611, 2021

    2021 arXiv

  31. [38]

    Self-consistency improves chain of thought reasoning in language models,

    X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” arXiv preprint arXiv:2203.11171 , 2022

    2022 arXiv

  32. [39]

    Unleashing the emergent cognitive synergy in large language models: A task- solving agent through multi-persona self-collaboration,

    Z. Wang, S. Mao, W. Wu, T. Ge, F. Wei, and H. Ji, “Unleashing the emergent cognitive synergy in large language models: A task- solving agent through multi-persona self-collaboration,” arXiv preprint arXiv:2307.05300, 2023

    2023 arXiv

  33. [40]

    Fakenewsnet: A data repository with news content, social context, and spatiotemporal information for studying fake news on social media,

    K. Shu, D. Mahudeswaran, S. Wang, D. Lee, and H. Liu, “Fakenewsnet: A data repository with news content, social context, and spatiotemporal information for studying fake news on social media,” Big data, vol. 8, no. 3, pp. 171–188, 2020

    2020

  34. [42]

    Evidence-aware hierarchical interactive attention networks for explainable claim verifi- cation,

    L. Wu, Y . Rao, X. Yang, W. Wang, and A. Nazir, “Evidence-aware hierarchical interactive attention networks for explainable claim verifi- cation,” in Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence , 2021, ...

    2021

  35. [43]

    Chatgpt-3.5 turbo,

    OpenAI, “Chatgpt-3.5 turbo,” https://www.openai.com/chatgpt, 2024, version 3.5 Turbo

    2024

  36. [44]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools,

    T. GLM, A. Zeng, B. Xu, B. Wang, C. Zhang, D. Yin, D. Zhang, D. Rojas, G. Feng, H. Zhao et al., “Chatglm: A family of large language models from glm-130b to glm-4 all tools,” arXiv preprint arXiv:2406.12793 , 2024

    2024 arXiv

  37. [45]

    Fact-checking complex claims with program-guided reasoning,

    L. Pan, X. Wu, X. Lu, A. T. Luu, W. Y . Wang, M.-Y . Kan, and P. Nakov, “Fact-checking complex claims with program-guided reasoning,” arXiv preprint arXiv:2305.12744, 2023. APPENDIX A. Prompts’ Description Type Description • Text summary: This prompt is used to convert long ne...

    2023 arXiv

  38. [46]

    These sentences should form a set \( R \)

    From the provided CTI content, identify the \( M \) most critical sentences that capture the core intelligence. These sentences should form a set \( R \)

  39. [47]

    {Insert CTI content here}

    Based on the set \( R \), generate a concise, fluent, and informative **abstractive summary** that accurately reflects the main idea of the original CTI content. CTI Content: “{Insert CTI content here}” **Constraints:** -The summary must begin with the prefix: **"Abstractive S...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.