{"id":"361d39b0-1d1e-4360-8164-a223c5a50cca","arxiv_id":"2505.20658","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"KGST, a fine-tune-then-retrieve-then-refine pipeline, improves natural language to Signal Temporal Logic translation accuracy on the new STL-DivEn dataset and on DeepSTL.","lead":"Researchers built a 16,000-pair dataset that pairs everyday English sentences with precise Signal Temporal Logic formulas, then used it to teach an LLM to translate between the two. Tests show their translate-and-refine system beats several strong baselines, but the evaluation has unresolved data and measurement caveats.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No train/test deduplication in STL-DivEn means KGST's retrieved references may contain near-identical answers; the reported gains over GPT-4 could be inflated by formula copying during refinement.","rationale":"The reader's weakest assumption identifies the same core risk: the retrieval corpus is drawn from the training split with no documented train/test deduplication, and the evaluation metrics are string-based. This is the most load-bearing concern because KGST's headline advantage over GPT-4 is concentrated in the refinement step, which is exactly where the retrieved pairs enter the context. If near-duplicate training pairs supply the reference formula, GPT-4 can copy it, making the central claim of 'substantially improves accuracy' misleading. The DeepSTL result and the human evaluation provide independent support, but both are exposed to the same leakage: human evaluators also judge correctness of formulas that may have been copied from a retrieved near-duplicate. The lack of error bars is a secondary concern because the reported gaps are large; the ablation and scaling results are informative but do not rule out retrieval leakage. A concrete deduplication and retrieval-control experiment can settle whether the measured gains are genuine. The numerical inconsistencies in Table 1 (e.g., GPT-4 BLEU reported as 0.1931 in text vs 0.0831 in the table) are concerning but not load-bearing for the central claim. Overall, a CONDITIONAL verdict is appropriate: the paper should be accepted only if the artifact check confirms the results are robust to train/test overlap.","tokens_in":14625,"tokens_out":4593,"duration_ms":49073,"concrete_test":"Use the released STL-DivEn and DeepSTL data to compute the maximum ROUGE-L F1 (or Sentence-BERT cosine similarity) between each test natural language and every training natural language, and report the fraction of test sentences whose nearest training sentence exceeds a chosen threshold (e.g., ROUGE-L >= 0.6). Then rerun KGST on the full test set with the refinement step constrained to exclude any retrieved pair whose NL is above that threshold; as a control, also rerun with top-5 replaced by random training pairs. If STL Formula Accuracy drops substantially (e.g., the gap to GPT-4 is halved) or if the predicted STL exactly equals the retrieved reference STL in a high fraction of the near-duplicate subset, the reported KGST advantage is at least partly retrieval leakage rather than translation skill.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that KGST substantially improves NL-to-STL transformation accuracy, and the load-bearing condition is that the measured gain comes from the generate-then-refine method rather than from test-time retrieval of the ground-truth formula. Section 4.1 filters each new pair against the current seed set with a ROUGE < 0.5 threshold, but Section 5.1 then randomly splits the resulting 16k pairs into 14k training / 2k test with no documented deduplication across the split. KGST retrieves the top-5 NL-STL pairs from the training split (Section 4.2, K=5) and feeds them to GPT-4 during refinement (Figure 8 prompt). If a test sentence has a near-duplicate in the training split, the reference STL formula is placed directly in the refinement context. The exact-token STL Formula Accuracy metric (Appendix B) then rewards the model for copying that formula. This could inflate KGST's reported 0.5587 vs GPT-4's 0.4733 on STL-DivEn, and it may also affect the DeepSTL result (0.4538 vs 0.2537), since DeepSTL is template-generated and may contain repeated or near-repeated sentences. The empirical question is whether the measured gap survives when retrieval cannot expose near-identical answers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces STL-DivEn, a 16,000-pair natural-language-to-STL dataset constructed through a seed set, clustering-based exemplar selection, GPT-4-driven augmentation, ROUGE-based diversity filtering, and human validation. It also proposes KGST, a generate-then-refine framework that fine-tunes LLaMA-3-8B to produce a preliminary STL formula and then uses GPT-4, guided by the top-5 most similar NL-STL pairs retrieved from the training set, to refine it. The authors report that KGST achieves the best STL Formula Accuracy, Template Accuracy, and BLEU on both STL-DivEn (0.5587, 0.5627, 0.2142) and DeepSTL (0.4538, 0.4939, 0.5686) compared to baselines, and that human evaluation marks KGST's outputs correct on 62.4% of STL-DivEn and 54.6% of DeepSTL test cases. The central claim is that KGST substantially improves NL-to-STL transformation accuracy and that STL-DivEn is a more diverse and useful dataset than existing benchmarks.","tokens_in":14852,"tokens_out":6390,"duration_ms":63773,"significance":"If the results hold, the paper would provide a valuable public dataset and a transferable pipeline for a practically relevant but data-scarce task. The strength is that the method is evaluated on both the self-created dataset and an established external benchmark (DeepSTL), and the repository is publicly available. However, the quantitative evidence as presented does not yet fully establish the central claim: the evaluation lacks train/test deduplication safeguards, the primary metric is a strict token-level match that penalizes semantically equivalent formulas, no variance or significance testing is reported, and the self-created test set shares its generation model (GPT-4) with KGST's refinement stage. These gaps leave open the possibility that the reported gains are inflated by retrieval of near-duplicate references or by GPT-4 self-consistency rather than by genuine translation skill.","major_comments":[{"comment":"The STL-DivEn construction pipeline filters new pairs against the growing seed set using a ROUGE threshold of 0.5 (Section 4.1), but the later random split of 16,000 pairs into 14,000 training and 2,000 test instances (Section 5.1) is not accompanied by any documented cross-split deduplication. Because KGST retrieves the top-5 NL-STL pairs from the training set and places them in the GPT-4 refinement prompt (Section 4.2, Figure 8), a test sentence that has a near-duplicate in the training set would put the reference STL formula directly into the refinement context. The reported STL Formula Accuracy of 0.5587 on STL-DivEn and 0.4538 on DeepSTL could therefore overstate the method's translation ability. Please report the maximum ROUGE-L or embedding similarity between each test sentence and its retrieved training references, and run an ablation that removes retrieved pairs with high similarity to the input (or retrieves from a disjoint pool) to quantify the effect of near-duplicate exposure.","section":"Section 5.1 / Section 4.1"},{"comment":"The STL Formula Accuracy metric defined in Appendix B is a token-level alignment score: it treats formulas as token sequences and counts the number of tokens that appear in the same positions in the reference and prediction. This penalizes semantically equivalent formulas that differ in variable names, operator ordering, or syntactically different but logically equivalent forms (e.g., the disjunctive form of |z2|>0.5 as z2<-0.5 or z2>0.5). Table 5's Case 2 is a concrete example where the KGST output and the ground truth are semantically equivalent yet would not receive a perfect token-alignment score. Since Table 1 relies on this metric for the main quantitative comparison, the paper should supplement it with a semantic-equivalence evaluation, such as a monitor-based equivalence check on sampled signals or a human-labeled semantic correctness sample, and report the correlation between token accuracy and semantic correctness.","section":"Appendix B / Section 5.1"},{"comment":"Tables 1 and 2 report single point estimates without standard deviations, confidence intervals, or significance tests. The differences between KGST and the strongest baselines on STL-DivEn are moderate (0.5587 vs. 0.4790 for formula accuracy; 62.4% vs. 55.0% in human evaluation), and the human evaluation is based on 100 pairs with five annotators but does not report inter-annotator agreement or how individual labels are aggregated. Please report means and variances over multiple fine-tuning seeds and refinement runs, and provide a paired significance test (e.g., McNemar's test on the same test instances) for the headline comparisons with KGST.","section":"Sections 5.2.1 and 5.2.2"},{"comment":"The STL-DivEn test set was generated by GPT-4 with human validation, and KGST's refinement stage also uses GPT-4 to rewrite the preliminary formula. On STL-DivEn, some of KGST's advantage may therefore reflect GPT-4's preference for formulas that resemble its own generation style rather than a generalizable improvement in NL-to-STL translation. The large gain on DeepSTL mitigates this concern, but it is based on a single external benchmark whose 16,000-sample subset was not described in enough detail. Please validate on an additional external or human-authored held-out test set, or replace the GPT-4 refinement with a different model in a controlled experiment, to separate model-consistency effects from genuine translation improvement.","section":"Sections 4.2 and 5.2.1"}],"minor_comments":[{"comment":"The metric example describes the formula as 'eventually (a < 5)' and 'eventually (b < 5)', but the formula and template are written with 'G' instead of 'F' and the subsequent token-alignment explanation refers to 'G'. Please correct this inconsistency so the example matches the intended temporal operator.","section":"Appendix B"},{"comment":"The 'N-gram diversity' statistic used to support the dataset diversity claim is not defined in the main text. Please state its definition (e.g., ratio of distinct n-grams to total n-grams), the value of n used, and the averaging procedure so that Table 3 is reproducible.","section":"Section 5.3.1 / Table 3"},{"comment":"The human evaluation section does not explain how the five annotators' labels are combined into the reported correctness percentage. Please clarify whether the result is a majority vote, full agreement, or an average over annotators, and report inter-annotator agreement (e.g., Fleiss' kappa).","section":"Section 5.2.2"},{"comment":"For DeepSTL, the paper says 14,000 training and 2,000 test samples are randomly selected, but DeepSTL contains 120,000 sentences (Table 3b). Please clarify how the 16,000-sample subset was sampled from the full DeepSTL dataset and whether the split was stratified by template or operator distribution.","section":"Section 5.1"},{"comment":"The choices of k=5 for k-means clustering and the ROUGE threshold of 0.5 are presented without justification or sensitivity analysis. Please report how these hyperparameters were chosen and whether the dataset statistics change materially for nearby values.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a worthwhile problem and the released repository is a positive step. The main obstacle to publication is the lack of train/test deduplication evidence and the reliance on token-level metrics; these are fixable with additional experiments and analysis. If the authors can show that KGST's retrieval does not expose near-duplicate references and that the metric-based advantage survives semantic-equivalence checks, the paper could become acceptable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper gives you two things worth having: a new 16k-pair NL-STL dataset called STL-DivEn, built with a sensible generate-filter-validate loop, and a generate-then-refine pipeline (KGST) that fine-tunes LLaMA-3-8B and then has GPT-4 refine the draft using retrieved NL-STL pairs. The reported gains over GPT-4 and DeepSeek on STL-DivEn are plausible, and the ablations show both fine-tuning and refinement contribute. The diversity statistics make a real case that STL-DivEn is richer than DeepSTL, and the human evaluation supports the direction. The authors also acknowledge the dataset's GPT-4 origin and that it may not cover real-world requirement patterns, which is honest.\n\nThe soft spots are mostly in the evaluation. There are no error bars or significance tests anywhere, and the STL Formula Accuracy metric in Appendix B is confusing as written—the example doesn't line up with its own template. There are also numerical mismatches between the text and Table 1 (e.g., GPT-4 BLEU on STL-DivEn is 0.0831 in the table but 0.1931 in the text; DeepSeek template accuracy differs too). These should be fixed before publication.\n\nThe bigger concern is the DeepSTL result. DeepSTL is template-generated, and the paper's own statistics show 120k sentences from a vocabulary of only 265 words—so exact or near-exact duplicates are likely across the random 14k/2k split. Since KGST retrieves top-5 similar pairs from the training split and feeds them to GPT-4, a test sentence with a near-identical training match puts the ground-truth STL directly in the refinement context. That could explain the large jump from 0.25 to 0.45 on DeepSTL. The stress-test note about STL-DivEn itself does not land the same way: the ROUGE < 0.5 filter is applied against the full growing seed set, so within STL-DivEn no two pairs should be near-duplicates of each other. The same-model concern (GPT-4 generates the data and does the refinement) is real but partly mitigated by the external DeepSTL benchmark and the human evaluation.\n\nThis is a useful engineering contribution for the NL-to-STL subfield, and the dataset is probably worth having regardless of the pipeline. It deserves serious peer review, but the reviewers should ask for a deduplication analysis on DeepSTL, significance testing or error bars, and a cleaned-up metric definition. I would engage with it, and if the authors release the code and data, I'd likely cite it.","headline":"Solid dataset and pipeline for NL-to-STL, but the DeepSTL gain may be inflated by retrieval from a template-generated benchmark with likely train/test overlap.","tokens_in":15455,"tokens_out":3743,"would_cite":true,"duration_ms":38495,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Natural-language requirements can be translated into Signal Temporal Logic most accurately by generating a draft with a fine-tuned LLM and refining it with GPT-4 guided by similar NL-STL examples, the paper claims.","keywords":["natural language to Signal Temporal Logic","STL translation","dataset construction with LLMs","generate-then-refine","retrieval-augmented refinement","formal specification generation","LLM fine-tuning"],"falsifier":"Take the 2,000 test sentences from STL-DivEn, compute their ROUGE similarity to the top-5 retrieved training pairs, and rerun KGST with the retrieval pool restricted to pairs whose similarity to the test sentence is below 0.5, or replaced by random pairs; if formula accuracy drops toward the no-refinement level, the measured gains come from retrieval rather than from general translation ability.","tokens_in":14356,"feed_emoji":"📐","tokens_out":9726,"duration_ms":89814,"temperature":0.7,"pith_summary":"Signal Temporal Logic (STL) is a formal language for specifying the timing and value constraints of cyber-physical systems, but writing STL formulas by hand is slow and error-prone. This paper attacks the bottleneck from two sides: it builds STL-DivEn, a 16,000-pair dataset of natural-language requirements paired with STL formulas, and it introduces KGST, a generate-then-refine translation method. In KGST, a fine-tuned large language model first drafts an STL formula, then GPT-4 rewrites that draft using the five most similar NL-STL pairs retrieved from the training set as reference knowledge. On STL-DivEn and on the existing DeepSTL test set, KGST reports the highest formula accuracy, template accuracy, BLEU, and human-judged correctness among the tested methods. The intended payoff is that engineers could convert everyday requirements into machine-checkable specifications without being logicians.","feed_headline":"Retrieval-guided LLM rewrite lifts English-to-STL translation","feed_subtitle":"New 16k-pair dataset plus GPT-4 refinement guided by five examples beats GPT-4, DeepSeek, and self-refinement","key_machinery":"The load-bearing mechanism is a generate-then-refine loop with an external knowledge base. The first stage is a fine-tuned LLaMA-3-8B generator; the second is GPT-4 prompting with the top-5 nearest NL-STL pairs retrieved from the training set by similarity, plus the original sentence and the draft formula. The external pairs supply concrete operator-scope patterns, timing intervals, and identifier conventions that the draft may have gotten wrong. The dataset itself is the second piece of machinery: STL-DivEn's construction, with hand-crafted seeds, k-means exemplar selection, GPT-4 augmentation, ROUGE-based diversity filtering, and human validation, is designed so that retrieved references cover varied formula shapes and vocabulary. The ablation results, which show that removing either fine-tuning or refinement lowers accuracy and that GPT-4 self-refinement without external references performs worse, are what carry the claim that both stages matter.","core_discovery":"The paper's central claim is that retrieval-guided refinement, not a stronger single model, is what pushes NL-to-STL translation past previous limits. The authors construct STL-DivEn by hand-writing 120 seed pairs, clustering them to choose representative exemplars, asking GPT-4 to generate new pairs from those exemplars, filtering by syntax checks and ROUGE-diversity thresholds, and validating by human annotators. They then fine-tune LLaMA-3-8B on the dataset to produce a preliminary formula for a new natural-language sentence, retrieve the top-5 most similar NL-STL pairs from the training set, and instruct GPT-4 to correct the preliminary formula against those references. The reported result is that this pipeline achieves STL formula accuracy of 0.5587 on STL-DivEn and 0.4538 on DeepSTL, ahead of GPT-4, DeepSeek, and self-refinement, with human correctness ratings of 62.4% and 54.6%. The authors interpret the gap between KGST and GPT-4-based self-refinement as evidence that external formula examples carry information the model's internal knowledge lacks.","pith_inferences":["Beyond the paper's own claims, the retrieval design suggests that a test sentence with a near-duplicate training sentence could be answered largely by copying; measuring KGST on a deduplicated split would separate retrieval memory from genuine translation skill.","The same retrieve-and-refine pattern should transfer to other formal-specification tasks, such as LTL, timed automata, or executable assertions, since the only ingredient it needs is a corpus of reference pairs and a judge LLM.","If exact token-aligned accuracy were replaced by semantic equivalence checks, for example by monitoring the predicted and reference formulas on random signals, absolute accuracies would probably shift, though the ranking of a retrieval-guided pipeline over self-refinement might persist."],"forward_implications":["A 16,000-pair NL-STL corpus now exists that is more diverse in vocabulary, formula structure, and identifier usage than DeepSTL, giving the field a common training and evaluation resource.","A fine-tuned draft followed by external-knowledge refinement is a practical recipe: on both STL-DivEn and DeepSTL it outperforms prompting strong LLMs directly and outperforms self-refinement.","Because KGST's gains hold on DeepSTL, a dataset generated from grammar templates rather than LLMs, the method is not tied to the style of its own training data.","Scaling experiments reported in the paper show formula accuracy rising with dataset size, implying that further dataset expansion is a plausible route to better translation.","The refinement step specifically reduces operator misuse, value errors, syntax violations, and semantic mismatches, per the paper's error-tracking analysis."],"supporting_citations":[{"why":"Defines Signal Temporal Logic semantics, the target formalism the paper translates into.","marker":"Maler and Ničković (2004)"},{"why":"Introduces the DeepSTL dataset and translation baseline; KGST is evaluated against it and its Template Accuracy metric.","marker":"He et al. (2022)"},{"why":"Proposes NL2TL, the prior LLM-based NL-to-TL dataset and T5 fine-tuning approach that this work extends.","marker":"Chen et al. (2023)"},{"why":"Documents GPT-4, the model used for dataset augmentation and for KGST's refinement stage.","marker":"Achiam et al. (2023)"},{"why":"Describes DeepSeek-V3, a strong LLM baseline that KGST outperforms on both datasets.","marker":"Liu et al. (2024)"},{"why":"Provides the Self-Refine baseline whose worse results support the claim that external knowledge, not self-feedback, drives refinement gains.","marker":"Madaan et al. (2024)"},{"why":"Defines ROUGE, used as the diversity filter in STL-DivEn construction to reject generated pairs too similar to existing seeds.","marker":"Lin (2004)"}],"fun_headline_variants":["Exemplar-guided LLM refinement outperforms self-correction in STL translation","New 16k dataset and retrieval-augmented LLM lift STL generation","Formula retrieval beats model size in English-to-STL translation","Exemplar-guided correction, not a bigger model, improves STL translation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the random training/test split keeps test sentences from being near-duplicates of the retrieved training references, and that exact token-aligned formula matching measures correctness; if either assumption fails, KGST's reported edge could reflect copying from memory rather than translation skill.","fun_headline_variants_meta":{"raw":{"variants":["Exemplar-guided LLM refinement outperforms self-correction in STL translation","New 16k dataset and retrieval-augmented LLM lift STL generation","Formula retrieval beats model size in English-to-STL translation","Exemplar-guided correction, not a bigger model, improves STL translation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000936,"raw_usage":{"total_tokens":4052,"prompt_tokens":1044,"completion_tokens":3008,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":660,"completion_tokens_details":{"reasoning_tokens":2927}},"tokens_in":660,"tokens_out":3008,"duration_ms":21987,"temperature":1.0,"reasoning_tokens":2927,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:50:02.632108+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 2,000 test sentences from STL-DivEn, compute their ROUGE similarity to the top-5 retrieved training pairs, and rerun KGST with the retrieval pool restricted to pairs whose similarity to the test sentence is below 0.5, or replaced by random pairs; if formula accuracy drops toward the no-refinement level, the measured gains come from retrieval rather than from general translation ability.","supporting_citations":[],"review_version":1}