Controlled Comparison of Machine Learning Models for Fault Classification and Localization in Power System Protection

Andreas Maier; Changhun Kim; Christian Bergler; Georg Kordowich; Johann J\"ager; Julian Oelhaf; Paula Andrea P\'erez-Toro; Siming Bayer

Controlled Comparison of Machine Learning Models for Fault Classification and Localization in Power System Protection

T0 review · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Under identical sensing and timing conditions, this paper finds that fault classification in power transmission is nearly solved by nonlinear ML models reading raw voltage/current windows, while fault localization plateaus at about 80 perce

desk verdict Useful controlled benchmark for ML in power-system protection, but the cross-validation split is under-specified and likely leaks, so the headline numbers need fixing before they can be trusted. read the letter →

arxiv 2510.00831 v2 pith:JWHN6FXW submitted 2025-10-01 cs.AI cs.LGeess.SP

Julian Oelhaf , Georg Kordowich , Changhun Kim , Paula Andrea Pérez-Toro , Christian Bergler , Andreas Maier , Johann Jäger , Siming Bayer This is my paper

classification cs.AIcs.LGeess.SP

keywords faultclassificationlocalizationpowersystemprotectionmachinelearningbenchmarkelectromagnetictransientsimulationtransmissionlineslidingwindowdomainrandomization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports a controlled benchmark of machine-learning models for two protection tasks, fault classification (FC) and fault localization (FL), using a single electromagnetic transient dataset with the same preprocessing for both. It claims that FC reaches a macro F1 of 0.992 with a multilayer perceptron and about 0.98 with gradient boosting, already at 10 ms windows, indicating that fault-type information is present in the earliest transients. It claims that FL is fundamentally harder: only MLP, stacking, and voting ensembles approach R2 = 0.806, corresponding to a localization error of about 10% of line length, and longer windows do not close the gap. Simpler and linear models fail on both tasks. The sympathetic reader would take this as evidence that raw V/I windows contain enough information to classify faults but not to localize them precisely without additional grid knowledge.

What carries the argument

The load-bearing setup is a domain-randomized electromagnetic transient simulation of a two-line transmission grid, with three-phase voltage and current recorded at eight protection relays. Sliding windows of 10–50 ms (5 ms step, 6400 Hz sampling) form the input features, and the same standardized pipeline feeds both a multi-class classifier (no fault plus ten fault types) and a regressor for normalized fault distance. The comparison across thirteen classifiers and eleven regressors under identical conditions is what turns the raw F1/R2 numbers into claims about task difficulty rather than model preferences.

What would settle it

Re-run the evaluation with folds grouped by simulation episode IDs (or any non-overlapping partition of the 9023 episodes). If the best F1 drops well below 0.99 or the best R2 drops below 0.8, the claim that FC is 'solvable' and that FL plateaus at ~10% error is an artifact of temporal leakage rather than a property of the data.

Watch

Extended reading notes

Core claim

The central claim is a side-by-side comparison under identical conditions: the same dataset, windowing, and validation protocol produce a near-perfect classifier but a mediocre regressor. The best classifier reaches F1 = 0.992 ± 0.001; gradient boosting matches at longer windows. The best regressors (MLP, stacking, voting) reach only R2 = 0.806 ± 0.008, i.e., about 10% of normalized line length error, with no model improving substantially with longer context. The paper concludes that the two protection tasks have different information requirements: classification can be solved from short transient windows, while localization saturates without impedance, topology, or other grid parameters.

Load-bearing premise

The load-bearing assumption is that the 5-fold cross-validation is unbiased; since the paper does not state that folds are grouped by simulation episode (Section 2.2), overlapping windows from the same episode can appear in both training and validation, which would inflate the reported F1 and R2 and shift the model ranking.

Editorial extensions

If this is right

Fault type can be recognized from raw V/I in 10 ms windows with F1 above 0.98, indicating protection decisions on fault class are feasible at relay timescales.
Longer context windows improve weak models but add little to the best classifiers, so additional history does not compensate for model capacity in FC.
For localization, no tested model breaches ~0.81 R2 even with 50 ms windows; the error floor around 10% of line length points to missing inputs such as impedance or topology rather than missing time.
The runtime ordering (linear models under 0.05 ms, tree ensembles 1–2 ms, top regressors about 0.56 ms) means the most accurate localizers are still feasible for near-real-time use but not for ultra-low-latency protection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's own protocol permits a dataset-leakage check: because windows are generated with a 5 ms step from 1 s episodes, a random 5-fold split likely places overlapping windows from the same episode in both training and validation; a grouped split could materially lower the reported F1/R2, and the authors should report whether folds are episode-independent.
If the localization ceiling is truly input-limited rather than model-limited, a direct test is to add per-relay impedance or pre-fault voltage/current ratios and see whether R2 moves; the paper's framework is ready for that extension.
The same dataset could support a fault-detection benchmark by relabeling windows with fault-onset distance, completing the detection/classification/localization trio under one protocol.
The sharp tier separation (e.g., KNN dropping from 0.86 in FC to about 0.3 in FL) suggests that model families suited to classification should not be assumed transferable to regression in protection tasks.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

The short version: this is a genuinely useful benchmark for the power-system-protection ML crowd — the first side-by-side comparison of fault classification and fault localization across 14 model families on a common EMT dataset, with realistic time windows and runtime costs. But the evaluation protocol has a hole that needs patching before the numbers can be taken at face value.

What I liked: the design is sensible. They extend their earlier fault-detection benchmark to two harder tasks, use domain-randomized EMT simulations with 9,023 episodes, and test window lengths from 10 to 50 ms. The finding that MLP and gradient boosting handle classification with F1 around 0.99 while localization is much harder, with the best R² around 0.8, is plausible and roughly consistent with the literature. The runtime analysis is a nice addition.

The soft spots, in order of size. First, Section 2.2 just says "A 5-fold cross-validation was used" without specifying the split unit. The windows are generated with a 5 ms sliding step and overlap heavily; windows from the same episode share the same fault type and location. If the split is at the window level, every fold will have training examples that are near-copies of validation examples, and the reported F1 and R² are inflated. This is load-bearing for all the model rankings and task-difficulty claims. The authors need to state that the split is grouped by episode, or re-run with that grouping. Second, there are internal inconsistencies: the abstract says F1 > 0.98 at 10 ms, but the body reports MLP at 0.97 and GB at 0.42 at 10 ms. Similarly, the abstract says localization error is stable across horizons while the conclusion says the best models needed longer temporal context. These need reconciling. Third, minor: for a protection audience, adding a conventional distance-relay baseline would make the comparison more meaningful. And there's no data/code release mentioned, which limits reproducibility.

Bottom line: it's a solid empirical contribution to a narrow subfield, but the CV ambiguity is not cosmetic — it affects every headline number. Send it to review with a demand that the split be clarified or fixed. I wouldn't cite the numbers until that's resolved.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: empirical benchmark with self-citations for context, not for the results.

full rationale

This paper is an empirical benchmark, not a derivation. The central results (F1 scores, R^2 values, runtimes) are obtained by training and evaluating specified ML models on a generated EMT dataset, with metrics reported from a 5-fold cross-validation. No parameter is fitted to a subset of data and then renamed as a prediction, and no variable is defined in terms of the quantity it is claimed to explain. The author-overlapping citations ([18], [20], [12]) supply dataset generation details, preprocessing conventions, and literature framing; they are not used as evidence for the numerical outcomes or to force a particular conclusion. The possible cross-validation leakage from overlapping sliding windows is a methodological validity concern, not a circularity step, and therefore does not change the circularity score. No specific reduction of a claimed result to its own inputs can be quoted from the paper.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new entities. Its claims rest on the representativeness of the synthetic data and on the validity of the cross-validation scheme, both of which are domain assumptions rather than derived results.

assumptions (3)

domain assumption EMT simulations in DIgSILENT PowerFactory faithfully represent fault transients in a 90 kV double-line grid.
All data are synthetic; no validation against field records or a public benchmark is provided. Section 2.1.
domain assumption The parameter ranges (line lengths, loads, fault locations, external grid settings) from reference [18] are typical operating conditions.
If the ranges are unrepresentative, the measured performance will not transfer to real grids. Section 2.1.
domain assumption Sliding windows from the same simulation episode can be treated as independent samples in 5-fold cross-validation.
Windows overlap by 5 ms steps and share the same fault type/location; the paper does not state an episode-level split. If violated, error estimates are optimistic. Section 2.2.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Controlled Comparison of Machine Learning Models for Fault Classification and Localization in Power System Protection." pith.science (2026). https://pith.science/paper/JWHN6FXW

@misc{pith2026251000831,
  author       = {Pith},
  title        = {Pith review of: Controlled Comparison of Machine Learning Models for Fault Classification and Localization in Power System Protection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JWHN6FXW}},
  note         = {Machine review of arXiv:2510.00831}
}

read the original abstract

The increasing complexity of modern power systems, driven by the integration of inverter-based and distributed energy resources, challenges the reliability of conventional protection schemes and motivates the use of machine learning for protection tasks. However, published results are often difficult to compare because datasets, sensing assumptions, and decision horizons vary across studies. This paper presents a controlled comparison of machine learning models for fault classification (FC) and fault localization (FL) under identical sensing, timing, and validation conditions on a common electromagnetic transient dataset, using decision windows of 10-50 ms to reflect protection-relevant time scales. For FC, the best-performing nonlinear models achieve F1 scores above 0.98 already at 10 ms, while lower-capacity models degrade at shorter horizons but improve with longer windows, indicating that relevant fault-type information is already present in the earliest transient. For FL, the top-performing models reach a stable localization error of about 10 % of normalized line length across all evaluated horizons, while weaker models form a clearly separated second performance tier. Line-resolved analysis shows that localization accuracy varies across grid segments, indicating topology-dependent difficulty rather than insufficient temporal context alone. These findings provide a controlled reference for comparing machine learning models across two protection tasks with fundamentally different information requirements.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

PROTECT-90: A Fault Dataset for Power System Protection
eess.SP 2026-06 unverdicted novelty 6.0 of 10

PROTECT-90 is a new open dataset of 9,022 EMT-simulated short-circuit episodes on a standardized 90 kV double-line topology with machine-readable metadata for benchmarking power system protection methods.

Reference graph

Works this paper leans on

25 extracted references · 1 linked inside Pith · cited by 1 Pith paper

[1]

Increasing shares of inverter-based generation and the adoption of hybrid AC-DC architectures [1] expand the spectrum of operating and fault scenarios in modern grids [2]

INTRODUCTION The transition towards decentralized power systems, driven by the integration of renewable energy sources (RES) and dis- tributed energy resources (DER), fundamentally reshapes the grid dynamics. Increasing shares of inverter-based generation and the adoption of hybrid AC-DC architectures [1] expand the spectrum of operating and fault scenari...

arXiv 2025
[2]

Double Line

METHODOLOGY This section outlines the experimental framework used to evaluate FC and FL in power system protection. We describe the dataset generation and preprocessing pipeline, the task formulations, and the set of models and evaluation protocols applied in this study. 2.1. Dataset and Grid Topology To systematically evaluate ML models for FC and FL, we...

2025
[3]

2 summarizes the FC results

EXPERIMENTS AND RESULTS Fig. 2 summarizes the FC results. The MLP achieved the best performance with F1 scores up to 0.99 and remained stable across all windows, aside from a slight drop at 10 ms (0.97). The GB matched this level at 50-30 ms (0.98), but declined sharply at shorter windows, reaching 0.75 and 0.42. In con- trast, KNN and RF reached lower ma...
[4]

CONCLUSION This paper presents a benchmarking study of machine learn- ing models for FC and FL in power system protection. The re- sults show that FC can be solved with high accuracy from raw V /Isignals using models such as MLP and GB, with longer windows providing only marginal improvements, while sim- pler methods fail to generalize. In contrast, FL pr...

2025
[5]

Pro- tection of Distribution Systems with Distributed Energy Resources,

Protection and Automation (B5) and Active Distribution Systems and Distributed Energy Resources (C6), “Pro- tection of Distribution Systems with Distributed Energy Resources,” Tech. Rep., CIGRE, 2015

2015
[6]

Der Zellulare Ansatz – VDE Studie,

VDE, “Der Zellulare Ansatz – VDE Studie,” Tech. Rep., VDE Verband der Elektrotechnik Elektronik In- formationstechnik e.V ., 2015

2015
[7]

Secure and de- pendable protection relay behaviour in extremely high loaded transmission systems,

J. Schindler, J. Prommetta, and J. J¨ager, “Secure and de- pendable protection relay behaviour in extremely high loaded transmission systems,” in15th International Conference on Developments in Power System Protec- tion (DPSP 2020), Liverpool, UK, 2020, pp. 6 pp.–6 pp., Institution of Engineering and Technology

2020
[8]

Lewis Blackburn,Protective Relaying: Principles and Applications, F ourth Edition, Taylor & Francis Group, Baton Rouge, 4th ed edition, 2014

J. Lewis Blackburn,Protective Relaying: Principles and Applications, F ourth Edition, Taylor & Francis Group, Baton Rouge, 4th ed edition, 2014

2014

Show all 25 references

[9]

Wai-Kai Chen,The Electrical Engineering Hand- book, Elsevier Academic Press, Boston, 2005, OCLC: 57371415

2005
[10]

System separation in the Continental Eu- rope Synchronous Area on 8 January 2021 – 2nd up- date,

ENTSO-E, “System separation in the Continental Eu- rope Synchronous Area on 8 January 2021 – 2nd up- date,” Jan. 2021

2021
[11]

A hybrid Protection Scheme based on Deep Reinforcement Learning,

G. Kordowich, M. Jaworski, T. Lorz, C. Scheibe, and J. Jaeger, “A hybrid Protection Scheme based on Deep Reinforcement Learning,” in2022 IEEE PES Innovative Smart Grid Technologies Conference Europe (ISGT- Europe), 2022, pp. 1–6

2022
[12]

A novel PRP based deterministic, redundant and re- silient IEC 61850 substation communication architec- ture,

S.M. Suhail Hussain, Mohd Asim Aftab, and Ikbal Ali, “A novel PRP based deterministic, redundant and re- silient IEC 61850 substation communication architec- ture,”Perspectives in Science, vol. 8, pp. 747–750, Sept. 2016

2016
[13]

Ultrafast Transmission Line Fault De- tection Using a DWT-Based ANN,

A. Abdullah, “Ultrafast Transmission Line Fault De- tection Using a DWT-Based ANN,”IEEE Transactions on Industry Applications, vol. 54, no. 2, pp. 1182–1193, Mar. 2018

2018
[14]

Impact of Data Spar- sity on Machine Learning for Fault Detection in Power System Protection,

J. Oelhaf, G. Kordowich, C. Kim, P. A. Perez-Toro, A. Maier, J. Jager, and S. Bayer, “Impact of Data Spar- sity on Machine Learning for Fault Detection in Power System Protection,” 2025, Preprint, accepted at EU- SIPCO 2025

2025
[15]

A review of machine learning applications in power system protection and emergency control: opportuni- ties, challenges, and future directions,

Gayashan Porawagamage, Kalana Dharmapala, J. Se- bastian Chaves, Daniel Villegas, and Athula Rajapakse, “A review of machine learning applications in power system protection and emergency control: opportuni- ties, challenges, and future directions,”Frontiers in Smart Grids, vo...

2024
[16]

A Scoping Review of Machine Learning Applications in Power System Pro- tection and Disturbance Management,

J. Oelhaf, G. Kordowich, M. Pashaei, C. Bergler, A. Maier, J. J¨ager, and S. Bayer, “A Scoping Review of Machine Learning Applications in Power System Pro- tection and Disturbance Management,” 2025, Unpub- lished manuscript (under review)

2025
[17]

An Intelligent Time-Domain ANN-Based Method for Fault Identification in CSC-HVDC Systems,

A. S. Da Silva, R. C. Dos Santos, and G. T. De Alen- car, “An Intelligent Time-Domain ANN-Based Method for Fault Identification in CSC-HVDC Systems,”Smart Grids and Sustainable Energy, vol. 10, no. 2, pp. 50, 2025

2025
[18]

Enhancing Fault Detection and Classification in Wind Farm Power Gen- eration Using Convolutional Neural Networks (CNN) by Leveraging LVRT Embedded in Numerical Relays,

T. Kandil, A. Harris, and R. Das, “Enhancing Fault Detection and Classification in Wind Farm Power Gen- eration Using Convolutional Neural Networks (CNN) by Leveraging LVRT Embedded in Numerical Relays,” IEEE Access, vol. 13, pp. 104828–104843, 2025

2025
[19]

A Deep Learning Approach for Fault Detection and Localization in MT-VSC-HVDC System Utilizing Wavelet Scattering Transform,

M. Mishra, D. A. Gadanayak, A. Pragati, and J. G. Singh, “A Deep Learning Approach for Fault Detection and Localization in MT-VSC-HVDC System Utilizing Wavelet Scattering Transform,”IEEE Access, vol. 13, pp. 95647–95664, 2025

2025
[20]

Fault Location in Three Terminal Transmission Lines Using Artificial Neural Networks,

D. Vaidya and M. N. Alam, “Fault Location in Three Terminal Transmission Lines Using Artificial Neural Networks,” in2025 13th International Conference on Smart Grid (icSmartGrid), 2025, pp. 583–586

2025
[21]

Integrating ANN and ANFIS for effective Fault Detec- tion and Location in Modern Power Grid,

G. K. Yadav, M. K. Kirar, S. C. Gupta, and J. Rajender, “Integrating ANN and ANFIS for effective Fault Detec- tion and Location in Modern Power Grid,”Science and Technology for Energy Transition, vol. 80, pp. 34, 2025

2025
[22]

A System- atic Evaluation of Machine Learning Methods for Fault Detection and Line Identification in Electrical Power Grids,

J. Oelhaf, G. Kordowich, P. A. P ´erez-Toro, T. Arias- Vergara, A. Maier, J. J ¨ager, and S. Bayer, “A System- atic Evaluation of Machine Learning Methods for Fault Detection and Line Identification in Electrical Power Grids,” inICASSP 2025 - IEEE International Con- ference on...

2025
[23]

Hybrid fuzzy evaluation algorithm for power system protection security assessment,

G. J. Meyer, T. Lorz, R. Wehner, J. Jaeger, M. Dauer, and R. Krebs, “Hybrid fuzzy evaluation algorithm for power system protection security assessment,”Electric Power Systems Research, vol. 189, pp. 106555, 2020

2020
[24]

A generic data generation framework for short circuit detection training of neural networks,

M. Wang, G. Kordowich, and J. J ¨ager, “A generic data generation framework for short circuit detection training of neural networks,” inPESS+PELSS 2022; Power and Energy Student Summit, 2022, pp. 49–54

2022
[25]

Dis- tanzschutzalgorithmen,

F. Mahr, S. Henninger, M. Biller, and J. J ¨ager, “Dis- tanzschutzalgorithmen,” inElektrische Energiesysteme, pp. 487–551. Springer Fachmedien Wiesbaden, Wies- baden, 2021

2021

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.

[1] [1]

Increasing shares of inverter-based generation and the adoption of hybrid AC-DC architectures [1] expand the spectrum of operating and fault scenarios in modern grids [2]

INTRODUCTION The transition towards decentralized power systems, driven by the integration of renewable energy sources (RES) and dis- tributed energy resources (DER), fundamentally reshapes the grid dynamics. Increasing shares of inverter-based generation and the adoption of hybrid AC-DC architectures [1] expand the spectrum of operating and fault scenari...

arXiv 2025

[2] [2]

Double Line

METHODOLOGY This section outlines the experimental framework used to evaluate FC and FL in power system protection. We describe the dataset generation and preprocessing pipeline, the task formulations, and the set of models and evaluation protocols applied in this study. 2.1. Dataset and Grid Topology To systematically evaluate ML models for FC and FL, we...

2025

[3] [3]

2 summarizes the FC results

EXPERIMENTS AND RESULTS Fig. 2 summarizes the FC results. The MLP achieved the best performance with F1 scores up to 0.99 and remained stable across all windows, aside from a slight drop at 10 ms (0.97). The GB matched this level at 50-30 ms (0.98), but declined sharply at shorter windows, reaching 0.75 and 0.42. In con- trast, KNN and RF reached lower ma...

[4] [4]

CONCLUSION This paper presents a benchmarking study of machine learn- ing models for FC and FL in power system protection. The re- sults show that FC can be solved with high accuracy from raw V /Isignals using models such as MLP and GB, with longer windows providing only marginal improvements, while sim- pler methods fail to generalize. In contrast, FL pr...

2025

[5] [5]

Pro- tection of Distribution Systems with Distributed Energy Resources,

Protection and Automation (B5) and Active Distribution Systems and Distributed Energy Resources (C6), “Pro- tection of Distribution Systems with Distributed Energy Resources,” Tech. Rep., CIGRE, 2015

2015

[6] [6]

Der Zellulare Ansatz – VDE Studie,

VDE, “Der Zellulare Ansatz – VDE Studie,” Tech. Rep., VDE Verband der Elektrotechnik Elektronik In- formationstechnik e.V ., 2015

2015

[7] [7]

Secure and de- pendable protection relay behaviour in extremely high loaded transmission systems,

J. Schindler, J. Prommetta, and J. J¨ager, “Secure and de- pendable protection relay behaviour in extremely high loaded transmission systems,” in15th International Conference on Developments in Power System Protec- tion (DPSP 2020), Liverpool, UK, 2020, pp. 6 pp.–6 pp., Institution of Engineering and Technology

2020

[8] [8]

Lewis Blackburn,Protective Relaying: Principles and Applications, F ourth Edition, Taylor & Francis Group, Baton Rouge, 4th ed edition, 2014

J. Lewis Blackburn,Protective Relaying: Principles and Applications, F ourth Edition, Taylor & Francis Group, Baton Rouge, 4th ed edition, 2014

2014