{"id":"ec2ea858-5a63-490d-9ffc-4e4dd71f6dc8","arxiv_id":"2507.11310","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"LRCTI uses an LLM to summarize threat reports, retrieve evidence in several rounds, and judge each claim credible or incredible, reporting strong F1 gains on CTI-200 and PolitiFact.","lead":"This paper introduces LRCTI, a three-stage LLM pipeline that summarizes cyber threat reports into claims, retrieves evidence over multiple rounds from a cybersecurity corpus, and uses natural language inference to label each claim credible or incredible. The authors report large F1 gains over ten baselines on two datasets, but the main CTI benchmark and retrieval corpus are built in-house, and no code or data is released.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The CTI-200 test documents may be present in the retrieval corpus, so the reported F1 gains could reflect answer retrieval rather than credibility reasoning.","rationale":"The reader's weakest_assumption correctly identifies the absence of any stated or demonstrated separation between the CTI-200 test set and the in-house retrieval corpus. This is the single most load-bearing concern because every downstream claim—multi-step retrieval superiority, NLI effectiveness, module ablatability—rests on the assumption that retrieved evidence is independent of the test answer. The paper provides internal deduplication of the corpus but no cross-corpus exclusion, no data release, and no overlap analysis. This is a mechanistic, checkable gap, not an accusation of misconduct. The concern is amplified by the fact that CTI-200 was created by the same authors (reference [41]) and by the inconsistent dataset statistics in Table I (a benchmark named CTI-200 with 1000 total entries), but the core issue remains the potential train/test leakage through the retrieval store. A concrete overlap test would settle whether the performance claims survive an uncorrupted evaluation. Because this concern directly invalidates the headline result unless resolved, the REJECT verdict is warranted. Other issues, such as missing error bars and unreported hyperparameters, are secondary but reinforce the need for an artifact release.","tokens_in":14114,"tokens_out":2686,"duration_ms":32089,"concrete_test":"Run a near-duplicate detection pass between every CTI-200 test report and every document in the constructed CTI corpus (e.g., embedding cosine similarity or MinHash with a conservative threshold, followed by manual inspection of the top pairs). If any test document has a near-duplicate in the corpus, remove those corpus documents and rerun the full LRCTI pipeline; if F1-Macro or F1-Micro drops substantially, the reported numbers are contaminated by retrieval of the answer rather than by credibility reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that LRCTI improves F1-Macro and F1-Micro to 90.9% and 93.6% on CTI-200, attributed to multi-step evidence retrieval and reasoning. This attribution requires that the retrieved evidence actually supports the judgment rather than merely repeating the answer. Section III-D describes building an in-house corpus by scraping cybersecurity sources, with deduplication applied only within the corpus. The CTI-200 benchmark (Section IV-A.1, reference [41]) was introduced by the same authors, and the paper never states that test documents or their near-duplicates are excluded from the corpus. If a test report (or a near-duplicate) is in the corpus, the retriever can surface the original text, and the NLI module can trivially classify it as credible or incredible from that text. The gains then reflect retrieval of the label, not the proposed reasoning. The corpus-vs-Wikipedia comparison in RQ5 (Table IV) does not test this, since both settings use the same confounded corpus. The ablation results (Figure 6) also cannot distinguish retrieval-of-answer from retrieval-of-evidence. No artifact, split, or overlap analysis is provided to rule out this confound, making the headline performance claim uninterpretable as evidence for the framework's reasoning ability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LRCTI, an LLM-based framework for cyber threat intelligence (CTI) credibility verification. The pipeline consists of an LLM-based text summarization module that converts a report into a claim, an adaptive multi-step evidence retrieval module that iteratively collects paragraphs and sentences from a purpose-built CTI corpus, and a prompt-based NLI module that outputs a credibility label (credible/incredible) with a confidence score and an explanation. The authors evaluate LRCTI on CTI-200 and PolitiFact, reporting F1-Macro and F1-Micro scores of 90.9% and 93.6% on CTI-200, outperforming ten baselines, and they include ablations, a user study, and a comparison of retrieval efficiency between their CTI corpus and Wikipedia.","tokens_in":14324,"tokens_out":3719,"duration_ms":45128,"significance":"If the reported gains were trustworthy, LRCTI would be a meaningful advance in CTI credibility verification, combining retrieval-augmented LLM reasoning with interpretable justifications, and the paper would be among the first to apply multi-step evidence retrieval in this domain. The paper also attempts a user study to show the practical utility of the generated evidence, which is a positive feature. However, the experimental evidence as presented is compromised by a likely corpus contamination issue, an incorrect metric equivalence claim, and a lack of statistical rigor; the credibility of the central performance claim therefore cannot be assessed from the manuscript. No code or dataset is released, which further limits verification of the results.","major_comments":[{"comment":"The in-house CTI corpus described in Section III-D is built by scraping cybersecurity sources, while the main benchmark CTI-200 (Section IV-A.1, reference [41]) was introduced by the same research group. The paper never states that the CTI-200 test documents or their near-duplicates were excluded from the corpus; the deduplication described in Section III-D is applied only to the corpus itself. If a test report (or a very similar report) is in the retrieval corpus, the retriever can surface the original text as evidence, and the NLI module can classify from that text, making the F1 gains in Table II reflect answer memorization rather than credibility reasoning. This is a load-bearing confound that invalidates the attribution of the reported improvements to the multi-step retrieval mechanism.","section":"Section III-D and Section IV-A.1"},{"comment":"The statement that 'precision is numerically equivalent to F1-Macro in our experimental setting' is incorrect for standard binary classification: F1-Macro is the average of the per-class F1 scores, which is not generally equal to either global or per-class precision. This raises the possibility that the reported headline metrics were miscalculated, and it is not a mere presentation issue because the paper's central claim depends on these numbers. Additionally, the abstract claims 'over 5%' improvement in both F1-Micro and F1-Macro, but Table II shows F1-Micro improves from 0.894 (MUSER) to 0.936 (LRCTI), a relative gain of only 4.7%.","section":"Section IV-B and Tables II-III"},{"comment":"The experimental section gives no error bars, no significance tests, and no description of train/test splits or repeated runs, so the reported single-point gains over baselines cannot be statistically assessed. Moreover, the re-retrieval threshold is internally inconsistent: Section IV-A.3 sets the overconfidence factor to θ=0.7, but the text in Section III-B and the caption of Figure 2 state that re-retrieval triggers when λ<0.5, where λ=θ×Conf (Eq. 5); with θ=0.7, λ<0.5 corresponds to Conf<0.714, not Conf<0.5. This inconsistency affects the operational meaning of the multi-step mechanism and should be clarified before the reported behavior can be reproduced.","section":"Section IV-A.3 and Eq. (5)"},{"comment":"The RQ5 comparison between the Wikipedia corpus and the CTI corpus uses the same potentially confounded CTI corpus, so it does not address the leakage concern raised in Major Comment 1. In addition, Table IV only reports retrieval time and a similarity score r(c,e); it does not report downstream verification accuracy, so it cannot support the claim that the CTI corpus improves credibility verification performance. The relevance of this experiment to the paper's central claim is therefore unclear.","section":"Section IV-E and Table IV"}],"minor_comments":[{"comment":"The text about corpus construction is duplicated verbatim: the paragraph beginning 'The overall corpus construction process' and the subsequent paragraphs through '...related IOCs' appear twice in Section III-D.","section":"Section III-D"},{"comment":"The statement of 'relative improvements of 5% and 10%' on CTI-200 and PolitiFact is inconsistent with the values in Tables II and III; for example, the F1-Macro improvement on PolitiFact from MUSER (0.732) to LRCTI (0.861) is about 17.6%, not 10%.","section":"Section IV-B"},{"comment":"The claim that 'beyond three retrieval steps, performance gains plateau and may even degrade slightly' appears to be based on a single curve with no error bars or numerical values; please provide the underlying data or a statistical analysis.","section":"Section IV-C and Figure 7"},{"comment":"The user study reports 'significantly outperforms' based on 8 participants and 40 claims, but no significance test, inter-annotator agreement, or confidence intervals are provided; the word 'significantly' is not supported.","section":"Section IV-D"},{"comment":"The author affiliations contain typos: '4rd' and '5rd' should be '4th' and '5th'.","section":"Author list"}],"recommendation":"reject","confidential_remarks":"The most serious concern is the possible overlap between the CTI-200 test set and the retrieval corpus, both produced by the same group; the paper provides no overlap analysis or exclusion statement, which alone would justify rejection in a journal that requires reproducible evidence. The incorrect claim about precision and F1-Macro equivalence further suggests a fundamental problem with the reported metrics. I would encourage the authors to re-evaluate the evaluation protocol, run a clean-room experiment with an independent corpus, and provide statistical rigor; a substantially revised paper could then be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The framework is a sensible extension of MUSER: summarize the report into claims, iteratively retrieve evidence from a domain-specific corpus, then run prompt-based NLI. That combination is new in the CTI space, and the paper describes it clearly. The PolitiFact results (F1-Macro 0.861 vs. MUSER's 0.732) are impressive and harder to explain away by corpus leakage, since that dataset comes from FakeNewsNet and the corpus is CTI-specific. Credit is also due for the user study, which at least attempts to measure interpretability rather than just asserting it.\n\nThe soft spot is the one the stress-test note flags: the CTI-200 benchmark and the retrieval corpus were both built by the same group, and the paper never states that the test documents (or near-duplicates) are excluded from the corpus. Deduplication is described only within the corpus. If a test report ends up in the retrieval index, the retriever can surface the very text being verified, and the NLI module can trivially read off the label. The F1 gains on CTI-200 then measure memory, not reasoning. The ablation and RQ5 do not rule this out because both settings use the same confounded corpus. This is a load-bearing omission, not a cosmetic one.\n\nOther weaknesses are more mechanical but still matter: no error bars or significance tests, no stated train/test split for the baselines or the LLM prompts, and no code or data release. There is also a duplicated paragraph block in Section III-D that suggests rushed editing, and at least one typo in the index terms. These are fixable.\n\nThe paper is not a rejection because the idea is bad; it is a rejection because the central claim is currently unsupported. If the authors release the corpus and test set, show that no test document overlaps the retrieval index, and report variance, the work could become a solid contribution to a practical subfield. As it stands, the CTI-200 numbers cannot be taken at face value.\n\nFor your purposes: worth a glance as an example of how benchmark construction can silently undermine a RAG system's evaluation, but I would not cite the headline numbers yet. Send it to a serious referee if you expect the authors to fix the data hygiene; otherwise the paper is not ready for archival publication.","headline":"A plausible LLM pipeline for CTI credibility checks, but the headline F1 gains on the in-house benchmark are uninterpretable until the authors rule out corpus contamination.","tokens_in":14902,"tokens_out":1573,"would_cite":false,"duration_ms":20639,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a three-stage LLM pipeline—summarization, adaptive multi-step evidence retrieval, and prompt-based natural language inference—can verify cyber threat intelligence credibility with F1-Macro 90.9% and F1-Micro 93.6%…","keywords":["cyber threat intelligence","credibility verification","large language models","multi-step evidence retrieval","natural language inference","fake news detection","retrieval-augmented generation","explainability"],"falsifier":"Re-run LRCTI after deleting every CTI-200 document (and all documents with high lexical or semantic overlap with the test reports) from the retrieval corpus. If F1-Macro on CTI-200 drops by more than a few absolute points, the original results were substantially driven by corpus leakage rather than by the multi-step retrieval and NLI reasoning.","tokens_in":13857,"feed_emoji":"🛡️","tokens_out":4448,"duration_ms":52261,"temperature":0.7,"pith_summary":"The paper proposes LRCTI, a framework that automates credibility verification of cyber threat intelligence (CTI) reports by turning each report into a short claim, iteratively retrieving supporting evidence from a CTI-specific corpus, and using an LLM to judge whether the evidence entails, contradicts, or leaves uncertain the claim. The authors argue that this three-step design overcomes the limits of older methods that rely on handcrafted features or single-pass classification, which are brittle on noisy, heterogeneous intelligence and opaque in their reasoning. On the CTI-200 dataset, LRCTI reaches 90.9% F1-Macro and 93.6% F1-Micro, and on the PolitiFact fake-news benchmark it reaches 86.1% on both scores, beating ten baselines by a wide margin. If the results hold, the framework offers a scalable, explainable way to vet threat reports before they are trusted in defense operations.","feed_headline":"LLM retrieval framework verifies cyber threat reports at 90.9% F1","feed_subtitle":"Multi-step evidence search plus reasoning beats feature-based and one-shot LLM baselines on two benchmarks.","key_machinery":"The load-bearing mechanism is the iterative retrieve-and-reason loop. The summarization module scores each sentence by $w(s_i) = \\text{ROUGE1-F1}(s_i, S) + \\alpha \\cdot p(s_i) + \\beta \\cdot q(s_i)$, where $p(s_i)$ is positional weight and $q(s_i)$ is semantic similarity to other sentences, then feeds the top-$m$ sentences to an LLM prompt that produces a one-sentence claim. The retrieval module uses a dense retriever to get the top-$k$ paragraphs and a progressive LLM-guided selection that refines the paragraph set, then selects the top-$j$ evidence sentences by similarity to the claim. A re-retrieval trigger fires when the reasoner returns NEI or when the adjusted confidence $\\lambda = \\theta \\times \\text{Conf}$ falls below $\\theta = 0.7$; the loop then incorporates previously established evidence and generates updated sub-claims to drive another retrieval round. The reasoning module is a prompt-based NLI step that labels the claim and writes an explanation. The whole pipeline runs on GPT-3.5-turbo with temperature 0.","core_discovery":"The central claim is that a retrieval-augmented LLM pipeline built specifically for the cyber threat domain can match or exceed the credibility-verification performance of dedicated evidence-based classifiers while producing human-readable justifications. Concretely, LRCTI first summarizes a verbose CTI report into a single verifiable claim, then runs an adaptive multi-step retrieval loop over an in-house CTI corpus—retrieving candidate paragraphs, filtering them with LLM feedback, selecting sentence-level evidence, and re-retrieving when the evidence is judged irrelevant, insufficient, or low-confidence—and finally classifies the claim as credible, incredible, or not-enough-information with a confidence score. The authors report that this design outperforms six evidence-based baselines (DeClarE, HAN, EHIAN, MAC, GET, MUSER) and four LLM-based baselines (GPT-3.5-turbo, ChatGLM2-6B, WEBGLM-2B, ProgramFC) by over 5% F1 on CTI-200 and by roughly 10% on PolitiFact. The paper further argues that the multi-step retrieval loop and the summarization module are both essential, since ablating either one degrades performance, and that the retrieved evidence is interpretable enough for human analysts to agree with the model 81.6% of the time in a user study.","pith_inferences":["The reported gains on CTI-200 may be inflated if the in-house retrieval corpus contains the very reports being verified or near-duplicates produced by the same authors; a held-out corpus experiment is the only way to rule out this leakage channel.","The paper compares corpus domains only in terms of retrieval speed and similarity (Table IV), not downstream F1, so the contribution of corpus quality versus the multi-step loop itself is not separately quantified.","An implicit extension is that the overconfidence factor $\\theta$ and the confidence threshold $\\lambda < 0.5$ are tunable levers that will shift the precision–recall operating point; real deployments would need to calibrate them per domain.","The design suggests a general recipe for low-resource verification: summarize the artifact into a claim, build or scrape a focused corpus, and iterate retrieval until the LLM's NLI confidence crosses a threshold—an approach that should be tested on non-cyber domains like medical or financial claims."],"forward_implications":["If LRCTI is correct, CTI credibility verification can be fully automated and made explainable, reducing reliance on slow, inconsistent manual analyst review.","The same summarization–retrieval–NLI loop transfers to general fake-news detection, as the PolitiFact results suggest, and likely to other evidence-grounded verification tasks with domain-specific corpora.","The paper's ablation and retrieval-step experiments indicate that two to three retrieval rounds are optimal, giving a practical cost–accuracy tradeoff for deployment.","Because the final output includes retrieved evidence fragments and a structured justification alongside the label, LRCTI could be integrated into analyst workflows as an assistive triage tool.","The confidence score with an overconfidence correction factor provides a natural threshold for flagging claims that need human review rather than automatic acceptance."],"supporting_citations":[{"why":"MUSER is the closest multi-step evidence retrieval baseline and the strongest evidence-based comparison, providing the method that LRCTI extends and the user-study baseline it outperforms.","marker":"[8]"},{"why":"FakeNewsNet supplies the PolitiFact dataset used to evaluate LRCTI's performance on general fake-news detection.","marker":"[40]"},{"why":"The KGV paper introduced the CTI-200 dataset used as the primary benchmark, so it defines the test set and its provenance.","marker":"[41]"},{"why":"GPT-3.5-turbo is the LLM backbone used for summarization, retrieval feedback, and NLI in every module of LRCTI.","marker":"[43]"},{"why":"ProgramFC is a strong LLM-based fact-checking baseline whose public configuration is used as the benchmark-setting, providing a direct point of comparison for retrieval-guided reasoning.","marker":"[45]"},{"why":"DeClarE is an evidence-aware deep learning baseline that anchors the G1 group, representing the static feature-and-attention approach that LRCTI claims to surpass.","marker":"[5]"},{"why":"The Dense Passage Retriever is the underlying dense retriever used for paragraph retrieval in the multi-step loop.","marker":"[34]"},{"why":"FLARE's active retrieval-augmented generation approach motivates the adaptive, feedback-driven retrieval strategy that LRCTI adapts to the CTI domain.","marker":"[21]"}],"fun_headline_variants":["LLM pipeline retrieves, reasons, and verifies threat intel at 90.9% F1","Threat intel credibility: LLM with iterative evidence search hits 93.6% F1","Multi-step LLM retrieval verifies CTI claims with explainable output","LLM-based evidence loop improves CTI credibility verification by 5%+"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the in-house CTI retrieval corpus does not contain the CTI-200 test reports or near-duplicates of them; if it does, the reported F1 gains could come from retrieving the answer itself rather than from evidence-based credibility reasoning.","fun_headline_variants_meta":{"raw":{"variants":["LLM pipeline retrieves, reasons, and verifies threat intel at 90.9% F1","Threat intel credibility: LLM with iterative evidence search hits 93.6% F1","Multi-step LLM retrieval verifies CTI claims with explainable output","LLM-based evidence loop improves CTI credibility verification by 5%+"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000718,"raw_usage":{"total_tokens":3303,"prompt_tokens":1102,"completion_tokens":2201,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":718,"completion_tokens_details":{"reasoning_tokens":2107}},"tokens_in":718,"tokens_out":2201,"duration_ms":19734,"temperature":1.0,"reasoning_tokens":2107,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:10:50.644316+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run LRCTI after deleting every CTI-200 document (and all documents with high lexical or semantic overlap with the test reports) from the retrieval corpus. If F1-Macro on CTI-200 drops by more than a few absolute points, the original results were substantially driven by corpus leakage rather than by the multi-step retrieval and NLI reasoning.","supporting_citations":[{"cited_title":"Muser: A multi-step evidence retrieval enhancement framework for fake news detection,","cited_arxiv_id":null,"evidence_quote":"MUSER is the closest multi-step evidence retrieval baseline and the strongest evidence-based comparison, providing the method that LRCTI extends and the user-study baseline it outperforms."},{"cited_title":"Fakenewsnet: A data repository with news content, social context, and spatiotemporal information for studying fake news on social media,","cited_arxiv_id":null,"evidence_quote":"FakeNewsNet supplies the PolitiFact dataset used to evaluate LRCTI's performance on general fake-news detection."},{"cited_title":"Chatgpt-3.5 turbo,","cited_arxiv_id":null,"evidence_quote":"GPT-3.5-turbo is the LLM backbone used for summarization, retrieval feedback, and NLI in every module of LRCTI."}],"review_version":1}