{"id":"e7498f6f-b7f7-4d4b-a6d9-7ea2b47c6f18","arxiv_id":"2501.16453","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A GPT-2 transformer trained on synthetic IEC-61850 traffic plus weak pseudo-labels detects four held-out zero-day attack types with 85-95% accuracy without retraining.","lead":"This paper tests whether a transformer can catch never-before-seen cyberattacks on digital substations by reading recent network packets and weak labels as context. On a public IEC-61850 attack dataset, the method reports above 85% accuracy on held-out attack types, which matters because substation trip messages are largely unauthenticated and zero-day attacks can trigger outages.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hyperparameters (Ksyn, mixing ratios) are selected using the held-out OOD attack accuracies, so the reported >85% zero-day accuracy may be an in-sample estimate rather than an unbiased measure of zero-day generalization.","rationale":"The reader's stated weakest_assumption is that linear mixtures of known attack features form a manifold broad enough to cover held-out zero-day attacks. My primary concern is complementary but more directly threatening to the empirical claim: the hyperparameters controlling those mixtures (Ksyn) and the training-label mixture ratios were selected using the same OOD attacks that are later reported as the zero-day test results. This selection leak, if present, would make the Table 1 numbers optimistic regardless of whether the manifold assumption holds. The reader's rationale does mention this as a fixable issue, but their formal weakest_assumption does not. I therefore mark partial agreement. The correct verdict remains CONDITIONAL: the paper's core idea is plausible and the experiments are extensive, but the central empirical claim of >85% zero-day accuracy should not be accepted at face value until a clean validation split—or leave-one-attack-out evaluation—confirms the numbers. The overstatement about baselines failing is secondary; even if baselines perform well on some OOD attacks, MDTF's strong performance on Masquerade Fake Normal and Random Replay is still notable, but the selection leak undermines even that.","tokens_in":18423,"tokens_out":6181,"duration_ms":54475,"concrete_test":"Re-run the MDTF/MTF experiments with a held-out validation protocol: split the ERENO attack types into three disjoint sets—ID training attacks (e.g., 3 GOOSE + 2 SV), validation-OOD attacks used only for selecting Ksyn and mixing ratios, and a final test-OOD attack set never touched during hyperparameter choice. Report 0-shot and max-shot accuracy on the final test-OOD set. A stronger version: leave-one-attack-out across all 9 attack types, selecting hyperparameters on internal validation folds, and report the average held-out accuracy. If any final held-out attack falls below 85% 0-shot accuracy, or if the rank vs. baselines changes, the headline claim is not supported as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim, that MDTF detects zero-day attacks at >85% accuracy, rests on Table 1. But the experimental protocol does not separate hyperparameter selection from OOD evaluation. Section 2.1 reports that Ksyn=700 'yielded the best performance' and Section 2.2 that mixing ratios 60/40 (TF) and 95/5 (DTF) 'yielded the highest attack detection accuracy' without describing any validation set. If these values were chosen by inspecting the same OOD test attacks (Poisoned High Rate, Masquerade Fake Fault, Masquerade Fake Normal, Random Replay) used in Table 1, the reported accuracies are optimistically biased. The abstract's 'state-of-the-art baselines fail' is also an overstatement: Table 1 shows Logistic Regression, Decision Tree, Random Forest, SVM, RNN, and LSTM reach 1.000 on three of the four OOD attacks; only Masquerade Fake Normal is a clear failure. The load-bearing premise for zero-day generalization is thus unverified: we cannot distinguish genuine ICL generalization from test-set-tuned hyperparameters.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an in-context learning (ICL) approach to zero-day attack detection in IEC-61850 digital substations. The authors train GPT-2-based transformer variants on synthetic attack classes generated by a multi-mixing procedure, condition the model on weak-classifier hard labels (simple transformer, TF) or full class distributions (distributional transformer, DTF), and evaluate on held-out attack types from the ERENO dataset. They report 85.8–100% zero-shot and 94.9–100% max-shot OOD detection accuracy, 100% accuracy on normal traffic, and a deployment latency analysis showing compatibility with IEC-61850 timing constraints. The central claim is that MDTF detects zero-day attacks above 85% accuracy while existing baselines fail.","tokens_in":18639,"tokens_out":6607,"duration_ms":71186,"significance":"If the results are unbiased, the paper provides a practically relevant demonstration of transformers' in-context learning for intrusion detection, with a useful deployment-oriented timing model and a sensitivity analysis relating weak-classifier quality to final detection accuracy. The use of a real IEC-61850 dataset, comparison with several standard classifiers, and the explicit treatment of weak labels are strengths. However, the headline numerical claims are weakened by the absence of a validation split for hyperparameter selection, and the abstract's statement that 'state-of-the-art baselines fail' is an overstatement of what Table 1 shows. The work is therefore an interesting empirical contribution whose main quantitative claims require re-evaluation under a cleaner protocol.","major_comments":[{"comment":"The zero-day detection numbers in Table 1 are likely optimistically biased because the key hyperparameters were selected using the same OOD attacks that later constitute the test set. Section 2.1 reports that Ksyn=700 'yielded the best performance' based on Figure 3's curves, which are computed on exactly the four OOD attack types (Poisoned High Rate, Masquerade Fake Fault, Masquerade Fake Normal, Random Replay). Section 2.2 similarly states that mixing ratios 60/40 for TF and 95/5 for DTF 'yielded the highest attack detection accuracy during testing,' with no validation split described. Since these same OOD attacks are used in Table 1, the reported accuracies are in-sample model-selection estimates rather than unbiased zero-day generalization estimates. The authors should either introduce a validation split for hyperparameter selection or transparently report Table 1 as development-set results and provide a separate validation protocol for the final claim.","section":"Sections 2.1, 2.2, Table 1"},{"comment":"The abstract's claim that 'the existing state-of-the-art baselines fail' is not supported by Table 1. Logistic Regression, Decision Tree, Random Forest, SVM, RNN, and LSTM achieve 1.000 accuracy on Poisoned High Rate, Masquerade Fake Fault, and Random Replay; their failure is concentrated on Masquerade Fake Normal (and Random Replay for Naive Bayes, DNN, CNN, and hard voting). The contribution is therefore better described as improving detection on the hardest OOD attack types, not as a blanket failure of all baselines. The abstract and Section 2.4 should be revised to state this more precisely.","section":"Abstract and Table 1"},{"comment":"The external validity of the 'zero-day' claim rests on an unverified assumption that held-out attacks lie near the synthetic manifold generated by multi-mixing. Algorithm 1 constructs new classes as equal-weight arithmetic means of subsets of the five known attack classes, so the four ERENO OOD attacks used in Table 1 are tested under the assumption that they are plausible points on or near this synthetic manifold. The paper provides no analysis quantifying this proximity (e.g., distances in feature space or a visualization) and no evaluation on attacks generated by a different process. The zero-day generalization claim should be scoped accordingly, or such an analysis should be added.","section":"Section 4.1 and Table 1"}],"minor_comments":[{"comment":"The text states that GPU TensorRT yields Taverage = 2.44 ms for BS=1, but Table 2 reports 0.00227 ± 0.00030 s, which is 2.27 ms. Please correct this inconsistency.","section":"Section 2.5 and Table 2"},{"comment":"Equation (1) defines Anew as a weighted sum with coefficients α_k, but Algorithm 1 uses equal weights of 1/len(attack_mixture). The relationship between α_k and the algorithm's uniform averaging should be clarified.","section":"Section 4.1"},{"comment":"There is a typo in the Discussion: 'time senstive' should be 'time-sensitive'.","section":"Section 3"},{"comment":"No code or data availability statement is provided. Since the ERENO dataset is public, a link to the code would significantly improve reproducibility.","section":"General"},{"comment":"The DNN row contains the entry '0' for Poisoned High Rate; format it as 0.000 for consistency with the other entries.","section":"Table 1"},{"comment":"The sensitivity analysis trains synthetic weak classifiers with some OOD attacks included as in-distribution to vary their accuracy. This is a useful simulation, but it should be explicitly labeled as a favorable-condition ablation, because deployed weak classifiers in a true zero-day scenario would not have seen the OOD attack classes.","section":"Section 2.3"}],"recommendation":"major_revision","confidential_remarks":"The core idea is promising and the deployment analysis is useful, but the selection-on-test protocol is a serious flaw that must be addressed before the quantitative claims can be taken at face value. The paper would benefit from a clear validation split for hyperparameter choice, and the abstract should be aligned with the actual baseline results in Table 1. If the authors rerun the experiments with a proper validation protocol and revise the claims accordingly, the manuscript could become a solid empirical contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know before reading: the extension is real but the headline result is not yet trustworthy. The new piece is the distributional transformer (DTF), which conditions on weak-classifier probability distributions rather than hard labels. On the ERENO public dataset it reports zero-shot accuracy 85.8–100% on four held-out attack types, with no failure cases. That claim is exactly where the paper is soft.\n\nThe paper does some things well. It evaluates on a public, realistic IEC-61850 testbed. It includes a sensitivity analysis showing DTF holds up when weak classifiers are only 40–50% accurate, which is a useful engineering result. The deployment latency study—TensorRT on a GPU, batch sizing, downsampling—is a genuine attempt to address real-time requirements of 3 ms GOOSE messaging, which most IDS papers ignore. The writing is clear and the limitations section acknowledges the reliance on weak classifier quality.\n\nNow the soft spots. The selection leak is real. Ksyn=700 and the mixing ratios (60/40 for TF, 95/5 for DTF) are described as 'yielded the best performance' after looking at OOD test accuracy, with no validation split. That makes the reported >85% zero-day accuracy an in-sample estimate, not an unbiased measure of generalization. It is fixable, but as it stands the central claim is under-supported. The abstract also overstates the baselines: Table 1 shows logistic regression, decision trees, RNN and LSTM reach 1.000 on three of the four OOD attacks; only Masquerade Fake Normal is a clear failure. Calling them 'state-of-the-art' is generous; they are generic ML baselines, not specialized IEC-61850 IDS. The zero-day framing is narrow: the OOD attacks are from the same testbed and feature space, and the synthetic training classes are linear mixtures of the known attacks. That may be a reasonable approximation, but it is an assumption, and the paper doesn't test it against attacks from a different distribution. There is also a minor internal inconsistency in the deployment math: Section 2.5 defines Tpro as Tbatch + TTF while the nomenclature defines it as max(Tbatch, TTF), and the text quotes 2.44 ms for TensorRT BS=1 while Table 2 says 0.00227 s (2.27 ms).\n\nNet: the DTF idea is worth taking seriously and the empirical study is substantial, but the evaluation protocol prevents me from trusting the headline numbers. Send it to peer review, with a referee asked to demand a proper validation split and a corrected abstract. A colleague working on ICL for intrusion detection should read it for the DTF design and the deployment analysis; cite it for those pieces once the numbers are re-run cleanly.","headline":"A real extension (distributional transformer) buried under an evaluation protocol that tunes on the test attacks; the zero-day numbers are likely optimistic but the approach deserves a serious referee.","tokens_in":19248,"tokens_out":2778,"would_cite":false,"duration_ms":25272,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that in-context learning lets a transformer detect zero-day attacks on IEC-61850 digital substation traffic with 85.8–100% accuracy at first encounter, without retraining.","keywords":["in-context learning","IEC-61850","intrusion detection systems","zero-day attacks","GPT-2 transformer","distributional transformer","multi-mixing"],"falsifier":"Run the same MDTF training on the five ID classes, then evaluate on attacks purposely engineered to lie outside the convex hull of the ID feature distributions (for example, adversarial perturbations of normal packets that cross the decision boundary while preserving protocol validity); if zero-shot accuracy falls below 80% while the weak classifiers remain above 50% accuracy, the multi-mixing manifold premise is falsified.","tokens_in":18178,"feed_emoji":"⚡","tokens_out":6146,"duration_ms":49789,"temperature":0.7,"pith_summary":"This paper argues that the in-context learning ability of transformers—the same mechanism that lets large language models adapt to new tasks from a few prompt examples—can be turned into an intrusion detection system for IEC-61850 digital substations. The authors train a GPT-2-style transformer on packets from five known attack classes plus normal traffic, augmented by synthetic classes, and at deployment feed it a window of recent packets labeled only by weak classifiers. The transformer predicts whether the newest packet is normal or an attack. On the ERENO dataset, the best variant (a distributional transformer consuming label probabilities rather than hard labels) achieves between 85.8% and 100% accuracy on four held-out zero-day attack types even before seeing any instance of them, and reaches 94.9–100% after a few in-context examples, while standard machine-learning baselines fail on at least one of these attacks. If correct, this gives substation operators a detector that can recognize novel attacks without retraining and with zero false positives on normal traffic.","feed_headline":"Transformer spots zero-day substation attacks at 85%+ accuracy","feed_subtitle":"A distributional transformer generalizes to never-seen attack types within the 3 ms IEC-61850 timing budget.","key_machinery":"The framework has three moving parts. Multi-mixing generates synthetic attack classes by averaging randomly selected subsets of the five known attack-class feature sets, multiplying the class diversity available for training. Weak classifiers, trained on this augmented set, provide pseudo-labels for in-context packets; the distributional transformer (DTF) consumes the full probability distribution each weak classifier assigns across the synthetic classes, which the authors find gives better zero-shot accuracy and robustness than the hard-label variant. The transformer itself is a decoder-only GPT-2 model with causal attention, so its prediction for the query packet depends only on the preceding labeled packets in the window, which is exactly the temporal structure of a substation packet stream.","core_discovery":"The central discovery is that in-context learning transfers from language to network traffic: a transformer conditioned on a short history of packet-label pairs can flag an unseen attack class as anomalous, and can improve its detection of that attack as more of its packets appear in the context window, all without parameter updates. The mixed-trained distributional transformer (MDTF) was the strongest instantiation, with no failure cases across the four out-of-distribution attack types in Table 1 and perfect normal-traffic accuracy; it also respects the IEC-61850 3 ms message transmission time when run on a GPU with TensorRT. The paper frames this as evidence that transformer-based IDS can generalize to zero-day attacks where fixed-rule and fixed-classifier baselines do not.","pith_inferences":["If real zero-day attacks use packet structures far from linear mixtures of the five known classes, the reported zero-shot accuracy is not guaranteed to transfer; the paper does not test that distribution-shift regime.","A straightforward extension would be to apply the same weak-label plus distributional-transformer pipeline to other industrial control protocols, such as Modbus or DNP3, to see whether the in-context generalization is specific to IEC-61850 traffic or a broader property of transformer sequence models.","The sensitivity analysis implies a practical rule of thumb: maintain weak-classifier accuracy above roughly 40–50% on out-of-distribution traffic, or the transformer's detection advantage shrinks and may disappear.","An adversarial implication the authors do not discuss: an attacker who sends a burst of similar attack packets actually trains the detector's in-context signal, so a stealthier adversary would vary features across packets to avoid giving the transformer coherent in-context evidence."],"forward_implications":["A zero-day attack can be flagged on its first packet with 85–100% accuracy, giving operators a chance to act before the attack propagates.","Because normal traffic is classified with 100% accuracy, the approach can be deployed without flooding operators with false alarms.","The detector adapts to an ongoing attack within a few in-context examples (max-shot accuracy 94.9–100%), so it improves as the attack continues.","With a mix of 5% ground-truth and 95% weak-classifier label distributions during training, the distributional transformer avoids the hyperparameter sensitivity of the hard-label variant.","The reported per-sample inference time of 2.27 ms on a GPU with TensorRT fits inside the 3 ms IEC-61850 transmission budget, suggesting real-time deployment is plausible."],"supporting_citations":[{"why":"Attention Is All You Need supplies the transformer architecture whose self-attention is the basis for in-context learning.","marker":"[18]"},{"why":"Language Models Are Few-Shot Learners establishes the few-shot in-context learning capability that this paper transfers to packet streams.","marker":"[19]"},{"why":"The authors' prior zero-day ICL paper is the approach this work extends with distributional labels.","marker":"[21]"},{"why":"The ERENO IEC-61850 dataset provides the nine attack scenarios used for training and zero-day evaluation.","marker":"[29]"},{"why":"GPT-2 is the specific decoder-only transformer instantiated for the intrusion detection task.","marker":"[30]"},{"why":"mixup is the data-augmentation idea that multi-mixing generalizes to create synthetic attack classes.","marker":"[35]"},{"why":"Shows that data distributional properties drive emergent in-context learning in transformers, motivating the multi-mixing diversity strategy.","marker":"[36]"}],"fun_headline_variants":["Zero-day grid attacks caught by transformer in-context learning","Transformer IDs unseen substation attacks without retraining","In-context learning spots novel cyberattacks in power grids","85% accuracy on zero-day IEC-61850 attacks via ICL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on the premise that the synthetic attack classes produced by averaging features of the five known attacks form a region broad enough that the four held-out attack types fall inside it, and that the weak classifiers' out-of-distribution accuracy stays above roughly 40–50%; if either fails, the reported zero-shot accuracy would not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Zero-day grid attacks caught by transformer in-context learning","Transformer IDs unseen substation attacks without retraining","In-context learning spots novel cyberattacks in power grids","85% accuracy on zero-day IEC-61850 attacks via ICL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1403,"prompt_tokens":870,"completion_tokens":533,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":464}},"tokens_in":486,"tokens_out":533,"duration_ms":5097,"temperature":1.0,"reasoning_tokens":464,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:11:18.608126+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same MDTF training on the five ID classes, then evaluate on attacks purposely engineered to lie outside the convex hull of the ID feature distributions (for example, adversarial perturbations of normal packets that cross the decision boundary while preserving protocol validity); if zero-shot accuracy falls below 80% while the weak classifiers remain above 50% accuracy, the multi-mixing manifold premise is falsified.","supporting_citations":[{"cited_title":"Advances in neural information processing systems 30 (2017)","cited_arxiv_id":null,"evidence_quote":"Attention Is All You Need supplies the transformer architecture whose self-attention is the basis for in-context learning."},{"cited_title":"In: 2024 IEEE International Confer- ence on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm), pp","cited_arxiv_id":null,"evidence_quote":"The authors' prior zero-day ICL paper is the approach this work extends with distributional labels."},{"cited_title":"IEEE Transactions on Dependable and Secure Computing (2023)","cited_arxiv_id":null,"evidence_quote":"The ERENO IEC-61850 dataset provides the nine attack scenarios used for training and zero-day evaluation."},{"cited_title":"OpenAI blog 1(8), 9 (2019)","cited_arxiv_id":null,"evidence_quote":"GPT-2 is the specific decoder-only transformer instantiated for the intrusion detection task."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"mixup is the data-augmentation idea that multi-mixing generalizes to create synthetic attack classes."},{"cited_title":"Advances in Neural Information Processing Systems 35, 18878– 18891 (2022)","cited_arxiv_id":null,"evidence_quote":"Shows that data distributional properties drive emergent in-context learning in transformers, motivating the multi-mixing diversity strategy."}],"review_version":1}