{"id":"d1bd98de-f792-47ff-bfa9-07e7947f1a25","arxiv_id":"2510.00831","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On a synthetic 90 kV double-line grid, a multilayer perceptron reaches ~0.98–0.99 F1 for fault classification at 10–50 ms windows and R²≈0.8 for fault localization, outperforming all other tested classical models.","lead":"This paper benchmarks a dozen classical machine-learning models on two power-grid protection tasks—classifying the type of a grid fault and estimating where along a line it occurred—using the same synthetic electromagnetic transient data and short time windows. It reports that a multilayer perceptron is the most accurate for both tasks, and that fault localization is much harder than fault classification.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-validation split not grouped by simulation episode; overlapping sliding windows cause train/test leakage that likely inflates the reported F1 and R², undermining model ranking.","rationale":"The paper is a useful controlled benchmark, but its central claims rest entirely on the performance numbers. The reader correctly identifies the most load-bearing weakness: the cross-validation split is not reported as episode-grouped, and the sliding-window construction creates heavy overlap between windows from the same episode. This is not a minor implementation detail; if the split is random at the window level, the model sees near-duplicates of training windows at test time, which directly inflates both F1 and R² and can change the ordering of models. The proposed concrete test—re-running with GroupKFold by episode—would settle the question. The internal inconsistency between abstract and conclusion about FL window-length dependence is secondary but reinforces the need for a careful re-check. I agree with the reader's conditional assessment: the paper should be accepted only after the split is clarified or corrected, and the numbers re-evaluated.","tokens_in":6402,"tokens_out":4635,"duration_ms":34834,"concrete_test":"Run the identical pipeline with GroupKFold using simulation episode IDs as groups (or leave-one-episode-out) for at least the 10 ms and 50 ms windows and the MLP, GB, and stacking models. Compare the grouped-split F1 and R² to the paper's reported values; if the 50 ms FC F1 drops from 0.99 by more than 0.05 or the FL R² drops from 0.806 by more than 0.1, the leakage is confirmed and the rankings/task-difficulty conclusions must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.2 states only 'A 5-fold cross-validation was used' without specifying the split unit. Section 2.1 describes 9,023 simulation episodes, each cropped to ±80 ms around the fault and segmented with a 5 ms sliding window step at window lengths 10–50 ms. Consecutive windows overlap by 50–90% and share identical FC labels (no-fault vs. fault type) or identical FL targets (fault location is constant per episode). If the 5-fold split is at the window level rather than grouped by episode, windows from the same episode appear in both training and validation. Because every episode has a unique combination of domain-randomized parameters (line lengths, load, fault location, external grid settings), the model can memorize episode-specific signatures instead of learning general fault patterns. This would inflate macro-F1 (reported up to 0.992) and R² (reported up to 0.806), directly affecting the central conclusions about which models are competitive and about the relative difficulty of FC vs. FL. A secondary inconsistency supports the need for scrutiny: the abstract claims FL localization error is stable across horizons, while the conclusion states the best FL models 'required longer temporal context.' Locating the exact CV procedure is thus the most load-bearing issue.","agreement_with_reader":"agree"},"referee_report":null,"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: this is a genuinely useful benchmark for the power-system-protection ML crowd — the first side-by-side comparison of fault classification and fault localization across 14 model families on a common EMT dataset, with realistic time windows and runtime costs. But the evaluation protocol has a hole that needs patching before the numbers can be taken at face value.\n\nWhat I liked: the design is sensible. They extend their earlier fault-detection benchmark to two harder tasks, use domain-randomized EMT simulations with 9,023 episodes, and test window lengths from 10 to 50 ms. The finding that MLP and gradient boosting handle classification with F1 around 0.99 while localization is much harder, with the best R² around 0.8, is plausible and roughly consistent with the literature. The runtime analysis is a nice addition.\n\nThe soft spots, in order of size. First, Section 2.2 just says \"A 5-fold cross-validation was used\" without specifying the split unit. The windows are generated with a 5 ms sliding step and overlap heavily; windows from the same episode share the same fault type and location. If the split is at the window level, every fold will have training examples that are near-copies of validation examples, and the reported F1 and R² are inflated. This is load-bearing for all the model rankings and task-difficulty claims. The authors need to state that the split is grouped by episode, or re-run with that grouping. Second, there are internal inconsistencies: the abstract says F1 > 0.98 at 10 ms, but the body reports MLP at 0.97 and GB at 0.42 at 10 ms. Similarly, the abstract says localization error is stable across horizons while the conclusion says the best models needed longer temporal context. These need reconciling. Third, minor: for a protection audience, adding a conventional distance-relay baseline would make the comparison more meaningful. And there's no data/code release mentioned, which limits reproducibility.\n\nBottom line: it's a solid empirical contribution to a narrow subfield, but the CV ambiguity is not cosmetic — it affects every headline number. Send it to review with a demand that the split be clarified or fixed. I wouldn't cite the numbers until that's resolved.","headline":"Useful controlled benchmark for ML in power-system protection, but the cross-validation split is under-specified and likely leaks, so the headline numbers need fixing before they can be trusted.","tokens_in":7154,"tokens_out":3660,"would_cite":false,"duration_ms":26293,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Under identical sensing and timing conditions, this paper finds that fault classification in power transmission is nearly solved by nonlinear ML models reading raw voltage/current windows, while fault localization plateaus at about 80 perce","keywords":["fault classification","fault localization","power system protection","machine learning benchmark","electromagnetic transient simulation","transmission line","sliding window","domain randomization"],"falsifier":"Re-run the evaluation with folds grouped by simulation episode IDs (or any non-overlapping partition of the 9023 episodes). If the best F1 drops well below 0.99 or the best R2 drops below 0.8, the claim that FC is 'solvable' and that FL plateaus at ~10% error is an artifact of temporal leakage rather than a property of the data.","tokens_in":6379,"feed_emoji":"⚡","tokens_out":4545,"duration_ms":38446,"temperature":0.7,"pith_summary":"The paper reports a controlled benchmark of machine-learning models for two protection tasks, fault classification (FC) and fault localization (FL), using a single electromagnetic transient dataset with the same preprocessing for both. It claims that FC reaches a macro F1 of 0.992 with a multilayer perceptron and about 0.98 with gradient boosting, already at 10 ms windows, indicating that fault-type information is present in the earliest transients. It claims that FL is fundamentally harder: only MLP, stacking, and voting ensembles approach R2 = 0.806, corresponding to a localization error of about 10% of line length, and longer windows do not close the gap. Simpler and linear models fail on both tasks. The sympathetic reader would take this as evidence that raw V/I windows contain enough information to classify faults but not to localize them precisely without additional grid knowledge.","feed_headline":"Fault type hits 0.99 F1; fault location tops at 0.81 R2","feed_subtitle":"Same raw voltage/current windows: classification is nearly solved, localization stalls at 10% line-length error.","key_machinery":"The load-bearing setup is a domain-randomized electromagnetic transient simulation of a two-line transmission grid, with three-phase voltage and current recorded at eight protection relays. Sliding windows of 10–50 ms (5 ms step, 6400 Hz sampling) form the input features, and the same standardized pipeline feeds both a multi-class classifier (no fault plus ten fault types) and a regressor for normalized fault distance. The comparison across thirteen classifiers and eleven regressors under identical conditions is what turns the raw F1/R2 numbers into claims about task difficulty rather than model preferences.","core_discovery":"The central claim is a side-by-side comparison under identical conditions: the same dataset, windowing, and validation protocol produce a near-perfect classifier but a mediocre regressor. The best classifier reaches F1 = 0.992 ± 0.001; gradient boosting matches at longer windows. The best regressors (MLP, stacking, voting) reach only R2 = 0.806 ± 0.008, i.e., about 10% of normalized line length error, with no model improving substantially with longer context. The paper concludes that the two protection tasks have different information requirements: classification can be solved from short transient windows, while localization saturates without impedance, topology, or other grid parameters.","pith_inferences":["The paper's own protocol permits a dataset-leakage check: because windows are generated with a 5 ms step from 1 s episodes, a random 5-fold split likely places overlapping windows from the same episode in both training and validation; a grouped split could materially lower the reported F1/R2, and the authors should report whether folds are episode-independent.","If the localization ceiling is truly input-limited rather than model-limited, a direct test is to add per-relay impedance or pre-fault voltage/current ratios and see whether R2 moves; the paper's framework is ready for that extension.","The same dataset could support a fault-detection benchmark by relabeling windows with fault-onset distance, completing the detection/classification/localization trio under one protocol.","The sharp tier separation (e.g., KNN dropping from 0.86 in FC to about 0.3 in FL) suggests that model families suited to classification should not be assumed transferable to regression in protection tasks."],"forward_implications":["Fault type can be recognized from raw V/I in 10 ms windows with F1 above 0.98, indicating protection decisions on fault class are feasible at relay timescales.","Longer context windows improve weak models but add little to the best classifiers, so additional history does not compensate for model capacity in FC.","For localization, no tested model breaches ~0.81 R2 even with 50 ms windows; the error floor around 10% of line length points to missing inputs such as impedance or topology rather than missing time.","The runtime ordering (linear models under 0.05 ms, tree ensembles 1–2 ms, top regressors about 0.56 ms) means the most accurate localizers are still feasible for near-real-time use but not for ultra-low-latency protection."],"fun_headline_variants":["Same data, split results: fault type F1 0.99, location R2 0.81","Fault classifier nearly perfect; localizer capped at 10% error","Identical windows: classification solved, localization stalls","ML fault type: 0.99 F1; fault location: 0.81 R2 on same data","Controlled test: classification near-solved, localization ~10% off"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing assumption is that the 5-fold cross-validation is unbiased; since the paper does not state that folds are grouped by simulation episode (Section 2.2), overlapping windows from the same episode can appear in both training and validation, which would inflate the reported F1 and R2 and shift the model ranking.","fun_headline_variants_meta":{"raw":{"variants":["Same data, split results: fault type F1 0.99, location R2 0.81","Fault classifier nearly perfect; localizer capped at 10% error","Identical windows: classification solved, localization stalls","ML fault type: 0.99 F1; fault location: 0.81 R2 on same data","Controlled test: classification near-solved, localization ~10% off"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000821,"raw_usage":{"total_tokens":3441,"prompt_tokens":764,"completion_tokens":2677,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":2569}},"tokens_in":508,"tokens_out":2677,"duration_ms":16869,"temperature":1.0,"reasoning_tokens":2569,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T12:59:29.440388+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the evaluation with folds grouped by simulation episode IDs (or any non-overlapping partition of the 9023 episodes). If the best F1 drops well below 0.99 or the best R2 drops below 0.8, the claim that FC is 'solvable' and that FL plateaus at ~10% error is an artifact of temporal leakage rather than a property of the data.","supporting_citations":[],"review_version":1}