{"id":"bccd3444-d040-4cb2-ad5e-f05aef5fc76c","arxiv_id":"2604.27601","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"SecGoal is a new expert-annotated benchmark dataset covering 15 protocols for extracting formalizable security goals from natural-language documents, paired with the AIFG framework; fine-tuned Gemma2-9B reaches 66.6% precision and 97.6% recall on held-out protocols.","lead":"The paper creates SecGoal, an expert-annotated dataset from 15 protocols, and AIFG, a framework to turn extracted goals into formal properties. A smart generalist might read it to see how fine-tuned smaller language models can make security specification extraction more precise and usable for formal verification.","discovery_kind":"unclear","skeptic_critique":{"model":"grok-4.3","headline":"SecGoal annotations lack reported inter-annotator agreement or guideline details, leaving ground-truth reliability unverified.","rationale":"The reader's weakest_assumption directly isolates the same annotation-reliability risk that is load-bearing for any new benchmark whose central claim is an improvement in extraction precision. Because the supplied review was abstract-only, the absence of IAA numbers is the clearest open question; confirming or refuting it via the concrete test above would settle whether the benchmark numbers are robust.","tokens_in":1723,"tokens_out":330,"duration_ms":22092,"concrete_test":"Randomly sample 150 sentences from the held-out test protocols; have two additional security-expert annotators independently label each sentence using only the guidelines published in the paper; compute Cohen's kappa between each new annotator and the original label set. Kappa < 0.65 would confirm substantial label noise and weaken the reported precision gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result (Gemma2-9B-FT lifting extraction precision from 24.0% to 66.6% while reaching 97.6% property recall on held-out protocols) treats the expert labels as fixed, objective ground truth for the binary distinction \"formalizable security goal vs. non-goal.\" No inter-annotator agreement statistic, annotation protocol, or adjudication procedure is referenced in the supplied abstract; if the boundary between security goals and background/implementation text is even moderately subjective, both the precision metric and the fine-tuning improvement become sensitive to annotator-specific idiosyncrasies rather than to model capability.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces SecGoal, an expert-annotated dataset covering 15 widely deployed protocols, as a benchmark for extracting formalizable security goal statements from natural-language protocol documents. It also presents AIFG, a schema- and flow-conditioned framework for structured formal security property generation. The central empirical result is that fine-tuning smaller open-source LLMs on SecGoal substantially improves extraction precision (Gemma2-9B-FT reaches 66.6% from a 24.0% baseline) while attaining 97.6% property recall on held-out test protocols, outperforming larger prompted LLMs and encoder baselines; AIFG further shows that concise goal inputs enable high-recall property generation.","tokens_in":1865,"tokens_out":533,"duration_ms":38240,"significance":"If the ground-truth annotations are shown to be reliable, SecGoal and AIFG would constitute a useful, reproducible resource for reducing the manual effort in formal verification pipelines by automating the identification of security goals from protocol specifications. The fine-tuning results provide concrete evidence that domain-specific adaptation can address the precision bottleneck observed in zero-shot LLM extraction, which is a practically relevant finding for the security community.","major_comments":[{"comment":"Dataset construction section: the manuscript provides no inter-annotator agreement statistic, no annotation guidelines, and no adjudication procedure for labeling sentences as formalizable security goals versus background or implementation content. Because all reported precision and recall figures (including the 24.0% \to 66.6% improvement and 97.6% recall) are computed against these labels as fixed ground truth, the absence of IAA directly undermines the interpretability of the benchmark and the fine-tuning gains.","section":"Dataset construction section"},{"comment":"Evaluation on held-out protocols: the paper does not report statistical significance tests or confidence intervals on the precision/recall differences between fine-tuned and prompted models. Given that the headline claim rests on these specific numeric improvements, the lack of statistical grounding makes it difficult to assess whether the observed gains are robust.","section":"Evaluation section"}],"minor_comments":[{"comment":"The abstract and §1 could more clearly state the train/test split sizes and the exact definition of 'formalizable' used by annotators.","section":"Abstract and Introduction"}],"recommendation":"major_revision","confidential_remarks":"The work is a reasonable fit for a security venue; the primary concern is the missing validation of the new annotated dataset, which is a standard expectation and should be straightforward to address."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback on dataset reliability and evaluation robustness. We respond to each major comment below.","responses":[{"response":"We agree that explicit documentation of the annotation process is necessary for benchmark interpretability. The SecGoal annotations were produced by domain experts using a written protocol that defines formalizable security goals versus background/implementation content; this protocol and the adjudication steps (initial labeling followed by expert review for disagreements) were omitted from the initial submission for brevity. In revision we will add a dedicated subsection (and appendix) reproducing the annotation guidelines verbatim, describing the annotator pool, and reporting any available inter-annotator agreement on a sampled subset. If the primary annotations were performed by a single lead expert with secondary review, we will state this limitation explicitly rather than claim multi-annotator IAA.","revision_made":"yes","referee_comment":"[Dataset construction section] Dataset construction section: the manuscript provides no inter-annotator agreement statistic, no annotation guidelines, and no adjudication procedure for labeling sentences as formalizable security goals versus background or implementation content. Because all reported precision and recall figures (including the 24.0% \to 66.6% improvement and 97.6% recall) are computed against these labels as fixed ground truth, the absence of IAA directly undermines the interpretability of the benchmark and the fine-tuning gains."},{"response":"We accept that the current manuscript lacks statistical grounding for the reported deltas. Although the absolute improvements are large, we will recompute the held-out metrics with bootstrap confidence intervals and paired significance tests (McNemar’s test for precision/recall at the sentence level) and include these results in the revised evaluation section and tables.","revision_made":"yes","referee_comment":"[Evaluation section] Evaluation on held-out protocols: the paper does not report statistical significance tests or confidence intervals on the precision/recall differences between fine-tuned and prompted models. Given that the headline claim rests on these specific numeric improvements, the lack of statistical grounding makes it difficult to assess whether the observed gains are robust."}],"tokens_in":1464,"tokens_out":454,"duration_ms":42082,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's core contribution is SecGoal, an expert-annotated dataset spanning 15 protocols, paired with the AIFG framework for turning extracted goals into structured properties. On held-out protocols, fine-tuning Gemma2-9B raises extraction precision from 24% to 66.6% while hitting 97.6% property recall, beating larger prompted models and encoder baselines.\n\nThe work does a clean job of separating training and test protocols and reporting concrete metrics that address the precision problem in LLM-based extraction. The held-out design avoids obvious circularity, and the results give a practical signal that targeted data can make smaller models more selective.\n\nThe main weakness is the absence of any inter-annotator agreement numbers or annotation guidelines. The stress-test concern holds: without those, it is hard to know how consistently experts drew the line between formalizable security goals and other protocol text. That boundary looks potentially subjective, so the reported gains rest on unverified labels.\n\nThe paper engages honestly with the manual-extraction bottleneck in formal methods and cites relevant prior work without overclaiming. No load-bearing fitting or internal contradictions appear in the abstract or reported results.\n\nThis is for researchers building LLM-assisted formal verification pipelines or security specification tools. A reader who needs a reusable benchmark for goal extraction would find the dataset and numbers useful, provided the annotation reliability can be checked.\n\nSend it to peer review so referees can examine the annotation protocol and ask for agreement statistics or adjudication details.","headline":"SecGoal supplies a new annotated dataset for security goal extraction and shows fine-tuning lifts precision on held-out protocols, but missing annotation details leave the ground truth shaky.","tokens_in":2350,"tokens_out":379,"would_cite":false,"duration_ms":20319,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Fine-tuning a 9B LLM on expert annotations raises precision of extracting formalizable security goals from 24% to 66.6% with 97.6% recall.","keywords":["security goal extraction","protocol documents","formal verification","LLM fine-tuning","benchmark dataset","cryptographic protocols","structured property generation"],"falsifier":"Independent experts re-annotate the held-out test protocols; if the fine-tuned model then scores below 40% precision on the new labels, the reported selectivity gains would be falsified.","tokens_in":2633,"feed_emoji":"🔒","tokens_out":728,"duration_ms":28743,"temperature":0.7,"pith_summary":"The paper presents SecGoal, an expert-annotated dataset spanning 15 deployed protocols, to measure how well models can pull formalizable security goal statements out of natural-language protocol documents. Frontier LLMs reach high recall of the underlying properties but low precision because they mix in non-goal material such as background or implementation details. Fine-tuning Gemma2-9B on the dataset lifts precision to 66.6% while keeping property recall at 97.6%, beating both larger prompted models and encoder baselines. The companion AIFG framework shows that concise extracted goals support high-recall structured property generation, leaving over-generation as the chief remaining limit. The work therefore supplies a dataset, benchmark, and generation method aimed at reducing the manual step that currently blocks formal verification of cryptographic protocols.","feed_headline":"Fine-tuned 9B model extracts security goals at 66.6% precision","feed_subtitle":"Expert-annotated benchmark lets smaller LLMs beat larger prompted models on identifying formalizable goals in 15 real protocols.","key_machinery":"SecGoal expert-annotated dataset of 15 protocols paired with AIFG, a schema- and flow-conditioned generator that turns extracted goal statements into structured formal security properties.","core_discovery":"Expert-annotated data in SecGoal lets smaller open-source LLMs become far more selective extractors of formalizable security goals than larger prompted models; on held-out protocols Gemma2-9B-FT reaches 66.6% extraction precision and 97.6% property recall while AIFG demonstrates that concise goal statements enable high-recall structured property generation, with expert-vetted inputs exposing over-generation as the dominant remaining bottleneck.","pith_inferences":["The same annotation-plus-fine-tuning pattern could be applied to other specification documents such as API standards or hardware manuals.","If the precision gain holds, formal-verification teams could process many more protocols per year without increasing expert review time.","A follow-up experiment could test whether the same fine-tuned model maintains its precision on protocols released after the training cut-off.","The bottleneck of over-generation might be addressed by adding a second filtering stage that scores generated properties against the original document text."],"forward_implications":["Smaller open-source models fine-tuned on SecGoal outperform larger prompted LLMs on selective extraction.","Property recall near 98% is achievable alongside the precision lift.","Concise extracted goals fed to AIFG produce high-recall structured properties.","Over-generation appears only when inputs come from automatic extraction rather than expert vetting."],"fun_headline_variants":["Gemma2-9B-FT reaches 66.6% precision on SecGoal extraction task","Fine-tuned 9B model hits 97.6% recall for formal security goals","SecGoal dataset raises LLM selectivity on protocol goal extraction","AIFG enables high-recall structured property generation from goals"],"cache_read_input_tokens":64,"weakest_assumption_plain":"Expert annotations correctly and consistently mark which sentences in the protocol documents are formalizable security goals rather than background or implementation text.","fun_headline_variants_meta":{"raw":{"variants":["Gemma2-9B-FT reaches 66.6% precision on SecGoal extraction task","Fine-tuned 9B model hits 97.6% recall for formal security goals","SecGoal dataset raises LLM selectivity on protocol goal extraction","AIFG enables high-recall structured property generation from goals"]},"model":"grok-4.3","cost_usd":0.006236,"raw_usage":{"total_tokens":2944,"prompt_tokens":684,"num_sources_used":0,"completion_tokens":80,"cost_in_usd_ticks":62362000,"prompt_tokens_details":{"text_tokens":684,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2180,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":684,"tokens_out":80,"duration_ms":23822,"temperature":1.0,"reasoning_tokens":2180,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-01T08:16:38.379679+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Independent experts re-annotate the held-out test protocols; if the fine-tuned model then scores below 40% precision on the new labels, the reported selectivity gains would be falsified.","supporting_citations":[],"review_version":2}