{"id":"620521da-c00f-4803-ac40-d9d738a9db20","arxiv_id":"2608.06778","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"TTP-R1, a retrieval-then-select system fine-tuned with reinforcement learning, reports state-of-the-art average F1 on four multi-label MITRE ATT&CK extraction benchmarks, with sub-technique F1 7.4 points above Claude Sonnet 4.5 with RAG and 28x lower latency.","lead":"The paper presents TTP-R1, a system that reads cyber attack reports and automatically labels which known attack techniques they contain, using a small 8B language model that chooses from a short retrieved candidate list. Across four CTI benchmarks it reports better average accuracy than large commercial models including Claude Sonnet 4.5 with retrieval, while running about 28 times faster on a single GPU.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The retriever upper bound is acknowledged, but the more load-bearing risk is benchmark validity: on templated CTI datasets with random splits, the training-corpus retriever may return near-duplicate training sentences, so the 7.4 pp gain over Claude RAG could reflect pattern memorization rather…","rationale":"The paper is internally consistent and the arithmetic in Tables 2 and 3 supports the average F1 claims as stated. The prompt/candidate-list comparison with Claude RAG is fair at the level of the inference-time interface. I also credit the paper for acknowledging the retriever recall cap in its limitations section. However, I do not think that cap is the most load-bearing concern for the central claim. Since the cap applies to all RAG baselines and TTP-R1's achieved recall on Expert is far below the cap, the cap does not explain the reported relative advantage. The more serious risk is that the benchmark splits permit retrieval-corpus leakage through near-duplicate training and test sentences, especially on the highly templated TRAM and Procedures datasets. If a test sentence has a near-duplicate training sentence for the same technique, the hybrid retriever will trivially place the correct label in the candidate list, and a model fine-tuned on prompts generated by that same retriever can exploit the distribution more effectively than a zero-shot frontier model. The reported 96.9–97.3% retrieval recall on those datasets is exactly what one would expect under such overlap. This concern does not accuse the authors of anything improper; it asks whether the reported 7.4 pp gain measures generalizable extraction or memorization of training-side retrieval patterns. The absence of released code, checkpoints, or retrieval outputs makes this impossible to check from the paper alone. The concrete near-duplicate-filtered evaluation and source-level split test would settle the question. Because the verdict was already CONDITIONAL and this concern strengthens, not weakens, the need for conditions, I keep the reader's verdict unchanged rather than moving it.","tokens_in":15177,"tokens_out":12948,"duration_ms":91677,"concrete_test":"Compute for each test sample the maximum token n-gram overlap (e.g., Jaccard similarity, or fraction of shared 8-grams) between the test text and the top-25 retrieved training corpus entries that produced its candidate list. Restrict the evaluation to test samples whose best overlap is below a conservative threshold such as Jaccard < 0.4 or zero shared 8-grams, and recompute Tables 2 and 3 on this subset. If TTP-R1's +7.4 pp average sub-technique F1 over Claude Sonnet 4.5 w/ RAG shrinks or reverses, the headline gain is driven by train/test template overlap rather than by the retrieval-plus-RL pipeline's generalizable extraction. A complementary check is to rebuild the retrieval corpus under a source-level split (all sentences from the same report, procedure example, or URL placed in the same split) and rerun the full comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The comparison with Claude RAG is controlled in the sense that baselines receive the same candidate list, so the retriever recall cap reported in Section 4.3 does not by itself undermine the relative F1 claim: the same cap applies to all RAG baselines, and TTP-R1's sub-technique recall on Expert (34.7%) is far below the 78.6% cap, so the cap is not even binding on its current performance. The more load-bearing risk is that the benchmark may not support the generalization claim. All four datasets adopt the splits of Lekssays et al. (Section 4.1). Procedures is composed of one-sentence expert-written summaries from ATT&CK procedure examples, and TRAM is short templated fragments; both are highly formulaic. The candidate list for every test sample is built by retrieving from the training corpus C (Section 3.1). If the split is random at the sentence or paragraph level, many test sentences will have near-duplicate training sentences carrying the same gold label. The reported retrieval recall of 97.3% on TRAM and 96.9% on Procedures is consistent with this possibility. TTP-R1 is fine-tuned on prompts produced by exactly this retriever, so it can internalize the training-side retrieval distribution; Claude RAG, given the same candidates at inference, has no opportunity to learn those patterns. The 7.4 pp average sub-technique F1 gain could therefore be a benchmark artifact caused by train/test template overlap rather than a generally better extraction policy. This requires no misreporting; it is a question of whether the chosen splits prevent retrieval-corpus leakage. The paper provides no overlap statistics, no source-level split, and no analysis of performance restricted to non-near-duplicate test items.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TTP-R1, a two-stage framework for multi-label MITRE ATT&CK technique extraction from CTI text. Stage 1 retrieves a small candidate set from a training corpus using a hybrid BM25 + ATT&CK-BERT retriever and fine-tunes an 8B LLM (Ministral-8B-Instruct, with an additional Qwen3-8B ablation) to select technique indices from that set. Stage 2 applies GRPO with a decomposed verifiable reward that separately scores precision, recall, and output format, using decoupled normalization and a selective threshold that restricts RL updates to examples that remain difficult after SFT. On four CTI benchmarks (TRAM, Procedures, Derived Procedures, Expert), the paper reports the best average macro-F1 at both technique and sub-technique granularity, a +7.4 pp sub-technique F1 gain over Claude Sonnet 4.5 with the same retrieval-augmented prompt, and a 28x latency reduction when served as an 8B model on a single GPU. The paper also includes ablations across two base models and an honest limitations section acknowledging the retriever upper bound and the modest size of RL gains.","tokens_in":15482,"tokens_out":6399,"duration_ms":60163,"significance":"If the empirical results hold under overlap-controlled evaluation, the contribution is significant: it shows that a retrieval-constrained 8B model trained with RLVR can match or beat frontier-LLM RAG pipelines on multi-label ATT&CK extraction while being far cheaper to serve. The retrieve-then-select formulation, the decomposed precision/recall/format rewards with decoupled normalization, and the selective RL strategy are coherent and clearly described. The paper also gives credit where due: the RAG baselines receive the same candidate list, so the retriever recall cap does not by itself invalidate the relative comparison; the ablation across two base model families is a useful robustness check; and the limitations section correctly identifies the retriever upper bound. However, the headline benchmark claim rests on single-run numbers on splits whose train/test overlap is not analyzed, and the most directly related RLVR baseline (Minerva) is cited but not compared. No code is released, which limits reproducibility. These issues are load-bearing for the central claim and require revision.","major_comments":[{"comment":"The main benchmarking claim may be inflated by train/test near-duplication. The candidate list for every test sample is built from the training corpus C (Section 3.1), and the paper adopts Lekssays et al.'s splits without stating whether they are random at the sentence or paragraph level. On TRAM and Procedures, sub-technique retrieval recall is 97.3% and 96.9%, respectively; if the splits are random, many test sentences will have near-identical training sentences carrying the same gold labels. TTP-R1 is fine-tuned on prompts produced by exactly this retriever, so it can internalize the training-side retrieval distribution, whereas inference-only baselines such as Claude RAG cannot. The retriever recall cap itself does not undermine the relative comparison, because all RAG baselines receive the same candidate list; what needs to be addressed is the overlap. Please report train/test near-duplicate statistics, and where possible report results on overlap-controlled splits (for example, grouping near-duplicates into the same split or deduplicating the corpus) to show that the 7.4 pp sub-technique gain over Claude RAG is not a template-memorization artifact.","section":"Section 4.1 / 4.3"},{"comment":"The results are reported for a single run with no error bars, no multiple seeds, and no significance tests. The average F1 advantage over TechniqueRAG is 3.9 points at technique level and 4.2 points at sub-technique level, and the advantage over TTP-R1 w/o RL is 2.3 and 2.2 points; these margins could easily lie within run-to-run variance, particularly on Expert, which has only 157 test samples and an average of 3.32 labels per sample. Please report variance across at least three seeds and, if feasible, paired significance tests (for instance, bootstrap resampling or McNemar-type tests over per-sample F1) for the key comparisons in Tables 2 and 3.","section":"Tables 2 and 3"},{"comment":"Minerva [2] is discussed in Related Work as applying RLVR to a broad suite of CTI subtasks including technique extraction, but it does not appear in the Compared Methods of Section 4.2. Because the paper's central contribution is an RLVR-based technique extraction method, the absence of any empirical comparison to Minerva leaves the state-of-the-art claim open with respect to the most closely related RLVR approach. Please add Minerva as a baseline on the four benchmarks, or if its published results do not cover these benchmarks, state that explicitly and justify why a direct comparison is not possible.","section":"Section 2.3 / 4.2"}],"minor_comments":[{"comment":"The paragraph beginning 'Aro.dat is, in fact, an encrypted and compressed PlugX payload' appears between the author affiliations and Figure 1, is unrelated to the surrounding introduction, and looks like a leftover from another document. It should be removed or rewritten as a properly introduced motivating example.","section":"Section 1"},{"comment":"Equations (3) and (4) do not define precision and recall when the predicted set is empty. Since the format reward only requires non-empty <answer> content and the answer could still be empty or malformed, please specify the convention used for zero-size predicted sets in the reward computation.","section":"Section 3.2"},{"comment":"The paper does not mention a code or model release, and several implementation details are underspecified (for example, the deduplication criterion for retrieved entries, tie-breaking in the rank(.) merge, and the number of RL optimization steps). Releasing the training and evaluation code, or at least the exact retrieval and decoding configurations, would substantially strengthen the reproducibility of the benchmark claims.","section":"Reproducibility"},{"comment":"The latency comparison in Figure 5 measures TTP-R1 locally with unoptimized HuggingFace inference while measuring the frontier baselines through Bedrock API calls. API latency includes network overhead and serving infrastructure effects, so the 28x speedup is not an apples-to-apples comparison; please report local latency for all models or state explicitly which portions of the measured latency are attributable to serving versus generation.","section":"Section 4.6 / Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The unexplained PlugX paragraph in Section 1 suggests possible copy-paste contamination from another manuscript; I would verify its provenance before publication. The decisive technical issue is the train/test overlap of the adopted splits: if the splits are random at sentence level, the headline comparison may not transfer to real-world CTI, and the paper should be required to provide overlap statistics and, ideally, overlap-controlled results. The missing Minerva baseline is also a scope and completeness concern for a paper whose contribution is explicitly RLVR-based."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick read on arXiv:2608.06778. The paper proposes TTP-R1: retrieval-augmented SFT followed by GRPO with decomposed precision/recall rewards, for multi-label ATT&CK technique extraction. The new bit is the combination — prior work did retrieve-then-generate with token-level training, and the closest RLVR baseline (Minerva) only handles single-label extraction. The decomposed reward with decoupled normalization is a sensible adaptation of GDPO. The empirical work is honest: four benchmarks, ablations over two base models, and the RL gains are small but consistent. The method description is clear enough to reimplement, and the tables are internally consistent.\n\nWhat I like: the paper acknowledges the retriever upper bound and modest RL gains in its own limitations. The comparison against TechniqueRAG is meaningful because the same splits and base model are used. The efficiency claim is directionally plausible, though the latency comparison mixes API serving and local GPU inference, so I'd treat the 28x factor as approximate.\n\nThe soft spots are real but not fatal. No code or checkpoints, no error bars or significance tests. No comparison with Minerva, which is cited but never evaluated. The most important concern is split overlap: candidate lists are retrieved from the training corpus, and TRAM and Procedures are short, templated sentences with retrieval recall above 96%. If the splits are random at sentence level, many test items are near-duplicates of training items, and the SFT stage could internalize the retriever's output distribution. The paper gives no overlap statistics and no source-level split to rule this out. That doesn't kill the relative comparison against Claude RAG on these splits, but it undercuts the generalization claim.\n\nThere's also an unexplained Aro.dat/PlugX paragraph sitting in Section 1 that has nothing to do with the paper. That is a manuscript hygiene issue a referee should flag.\n\nBottom line: the paper deserves a serious referee. The method is coherent, the contribution is a useful incremental step, and the benchmark-validity question is exactly what peer review should push on. Recommend conditional acceptance after the authors add overlap statistics and ideally a source-level split robustness check.","headline":"Solid, checkable empirical work on RLVR for multi-label ATT&CK extraction; the benchmark claim needs a leakage sanity check and the PlugX artifact must be cleaned up.","tokens_in":16087,"tokens_out":4807,"would_cite":true,"duration_ms":41396,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-stage retrieval-then-select pipeline with a decomposed reinforcement-learning reward achieves the best average F1 yet reported for multi-label ATT&CK technique extraction on four benchmarks, using an 8B model that runs 28x faster…","keywords":["cyber threat intelligence","ATT&CK taxonomy","attack technique extraction","retrieval-augmented generation","reinforcement learning with verifiable rewards","GRPO","multi-label classification","LLM fine-tuning"],"falsifier":"An experiment that trains TTP-R1 with an oracle retriever guaranteed to include every true technique, keeping all other details identical, would settle whether the selection stage is actually the bottleneck: if Expert sub-technique F1 stays near the reported 44.2 percent instead of tracking the higher recall bound, the reward design or the SFT initialization, not retrieval, limits performance.","tokens_in":14944,"feed_emoji":"🛡️","tokens_out":11712,"duration_ms":101601,"temperature":0.7,"pith_summary":"The paper argues that extracting ATT&CK attack techniques from cyber threat intelligence text should be modeled as retrieval-then-select, not as open-ended generation. It proposes TTP-R1, which first shrinks the taxonomy to a short candidate list with a hybrid lexical and semantic retriever and then fine-tunes an LLM to pick the correct subset from that list. The central move is a second training stage with reinforcement learning using verifiable rewards, where precision, recall, and output format are scored as separate channels normalized independently before combination. On four benchmarks, TTP-R1 achieves the highest average F1 at both parent and sub-technique granularity, including a 7.4-point gain in sub-technique F1 over Claude Sonnet 4.5 with the same candidates, while running 28 times faster as an 8B model on a single GPU. If the paper is right, high-quality multi-label TTP mapping can run locally at small-model cost instead of depending on expensive frontier APIs.","feed_headline":"8B model tops frontier LLMs on ATT&CK mapping by 7.4 F1","feed_subtitle":"A decomposed precision/recall reward lifts sub-technique F1 by 7.4 points and cuts latency 28x.","key_machinery":"The mechanism that carries the argument is the decomposed verifiable reward with decoupled normalization. Precision and recall are each computed as the average of a parent-technique-level overlap and a sub-technique-level overlap against the ground-truth set, the format channel verifies that the answer block is well-formed, and each channel is z-normalized inside its GRPO group before the weighted sum, so no single reward can dominate the policy gradient. This gives the optimizer separate signals for over-prediction and under-prediction, the two failure modes the paper attributes to prompted frontier models and to SFT-only models respectively. The supporting constraint is the hybrid retriever—BM25 plus ATT&CK-BERT, merged by minimum rank with a top-25 candidate cap—which turns a several-hundred-label space into a selection problem and, the paper notes, sets an upper bound on attainable recall.","core_discovery":"The central claim is that direct, set-level supervision can fix the multi-label recall gap that token-level objectives leave behind. After supervised fine-tuning, the model is trained with Group Relative Policy Optimization using a decomposed verifiable reward: one channel scores precision, one scores recall, and one checks output format; precision and recall are each averaged over parent and sub-technique granularity, and every channel is normalized within its group before weighted aggregation. TTP-R1 reaches the highest average F1 on TRAM, Procedures, Derived Procedures, and Expert at both granularities. At the sub-technique level it beats Claude Sonnet 4.5 with retrieval augmentation by 7.4 F1 points, and because it emits short index lists rather than long justifications it answers in about 0.34 seconds per query, 28 times faster. The paper also reports that the RL stage selectively raises recall on multi-label benchmarks while keeping the precision gains of SFT, and that the pipeline transfers to a second 8B base model family.","pith_inferences":["Inference: since retrieval recall caps what any trained policy can output (78.6 percent at the sub-technique level on Expert), the next substantial accuracy gain on that benchmark will come from better retrieval, not from further policy optimization.","Inference: the same retrieve-then-select-plus-RLVR recipe should transfer to other hierarchical low-resource label spaces, such as ATLAS for AI-system threats or vulnerability taxonomies, whenever a retriever can be built with high recall; the paper names ATLAS as future work but does not test it.","Inference: the 28x latency comparison is conservative in a favorable direction, because TTP-R1 was timed with unoptimized local inference while the baselines were timed through API serving; production serving of the small model would likely widen the gap.","Inference: a testable extension is to replace the fixed top-25 hybrid retrieval with iterative or query-expanded retrieval and check whether Expert sub-technique F1 rises toward the 78.6 percent recall ceiling."],"forward_implications":["At an average sub-technique F1 of 63.9% across the four splits, TTP-R1 is the strongest published method on these benchmarks, with its largest edge on the paragraph-level multi-label Expert set.","An 8B model with retrieval-augmented SFT plus RLVR can beat frontier APIs at this task, so accurate TTP mapping is feasible on a single GPU for teams that cannot call expensive hosted models.","Decoupling precision and recall into separately normalized reward channels changes the direction of the RL gradient; the paper reports technique-level recall gains of 18 percent on Derived Procedures and 24 percent on Expert with only a modest precision cost.","Because retrieval plus SFT supplies most of the performance, the candidate-list quality is the primary determinant of end-to-end accuracy; the RL stage is a targeted correction for multi-label completeness rather than a large accuracy jump."],"supporting_citations":[{"why":"Supplies the retrieve-then-select paradigm, the train/test splits, and the baseline results that TTP-R1 extends and compares against.","marker":"[15]"},{"why":"Contributed the Procedures, Derived Procedures, and Expert benchmarks and the noise-contrastive baseline that carry the multi-label evaluation.","marker":"[20]"},{"why":"Introduced Group Relative Policy Optimization, the policy-update algorithm used in Stage 2.","marker":"[25]"},{"why":"Provided reward-decoupled normalization, which TTP-R1 adapts to keep one reward channel from dominating the gradient.","marker":"[17]"},{"why":"Provides ATT&CK-BERT, the domain-pretrained dense retriever used in the hybrid candidate search.","marker":"[1]"},{"why":"Provides BM25, the lexical half of the hybrid retriever that supplies candidate techniques.","marker":"[23]"},{"why":"Provides the Sentence-BERT architecture underlying the dense retrieval component.","marker":"[22]"},{"why":"IntelEX is a training-free retrieval-plus-judge baseline compared against for multi-label extraction.","marker":"[29]"},{"why":"Claude Sonnet 4.5 is the frontier baseline that TTP-R1 outperforms by 7.4 F1 points at sub-technique level.","marker":"[4]"}],"fun_headline_variants":["8B TTP-R1 beats Claude by 7.4 F1 on ATT&CK extraction, 28x faster","Small model, big win: 8B tops Claude on ATT&CK by 7.4 F1 with RL","8B model uses RL to outscore Claude Sonnet 4.5 by 7.4 F1 on ATT&CK","8B model runs 28x faster and beats Claude on ATT&CK by 7.4 F1","RL with verifiable rewards lets 8B model top ATT&CK mapping, +7.4 F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The retrieval step must place every true technique in the candidate list, because the model can only select from that list; on the Expert benchmark roughly 21 percent of true sub-techniques are absent after retrieval, so no policy trained inside this framework can recover them.","fun_headline_variants_meta":{"raw":{"variants":["8B TTP-R1 beats Claude by 7.4 F1 on ATT&CK extraction, 28x faster","Small model, big win: 8B tops Claude on ATT&CK by 7.4 F1 with RL","8B model uses RL to outscore Claude Sonnet 4.5 by 7.4 F1 on ATT&CK","8B model runs 28x faster and beats Claude on ATT&CK by 7.4 F1","RL with verifiable rewards lets 8B model top ATT&CK mapping, +7.4 F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001026,"raw_usage":{"total_tokens":4365,"prompt_tokens":1024,"completion_tokens":3341,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":640,"completion_tokens_details":{"reasoning_tokens":3196}},"tokens_in":640,"tokens_out":3341,"duration_ms":23295,"temperature":1.0,"reasoning_tokens":3196,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:57:04.931415+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An experiment that trains TTP-R1 with an oracle retriever guaranteed to include every true technique, keeping all other details identical, would settle whether the selection stage is actually the bottleneck: if Expert sub-technique F1 stays near the reported 44.2 percent instead of tracking the higher recall bound, the reward design or the SFT initialization, not retrieval, limits performance.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the retrieve-then-select paradigm, the train/test splits, and the baseline results that TTP-R1 extends and compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributed the Procedures, Derived Procedures, and Expert benchmarks and the noise-contrastive baseline that carry the multi-label evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides ATT&CK-BERT, the domain-pretrained dense retriever used in the hybrid candidate search."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"IntelEX is a training-free retrieval-plus-judge baseline compared against for multi-label extraction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Claude Sonnet 4.5 is the frontier baseline that TTP-R1 outperforms by 7.4 F1 points at sub-technique level."}],"review_version":1}