REVIEW 3 major objections 4 minor 23 references
Pre-, In-, and Post-Processing Class Imbalance Mitigation Techniques for Failure Detection in Optical Networks
T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Adjusting the decision threshold of a trained random forest raises failure-detection F1 by 15.3 percent over the imbalanced baseline, outperforming all the pre-, in-, and post-processing alternatives tested in this comparison.
desk verdict Useful benchmark of 20 imbalance fixes for optical failure detection, but the headline F1 ranking is unverified because the paper never describes a held-out split. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the comparison protocol built on the standard taxonomy of pre-, in-, and post-processing bias mitigation, together with a random-forest baseline, F1 as the primary metric, 100-run averaging, and inference-time measurement. Threshold Adjustment is the post-processing step that scans decision thresholds on the trained model's probability outputs and selects the threshold that maximizes F1, thereby rebalancing the operating point without modifying data or model. This mechanism is what lets the paper attribute the F1 gain to the decision rule rather than to data transformation or algorithm changes, and it underpins the claimed performance-complexity trade-off.
What would settle it
Re-run the twenty-technique comparison on a second, independently collected optical-network failure dataset with a different imbalance ratio; if Threshold Adjustment no longer produces the largest F1 gain over the baseline, or the margin over the best in-processing method disappears, the ranking is dataset-specific rather than general.
Extended reading notes
Core claim
The paper claims that for failure detection in optical networks with a strongly imbalanced class distribution, the most effective way to improve F1 is not to resample or augment the training data, and not to change the learning procedure, but to adjust the decision threshold applied to the trained model's predictions. On the experimental dataset, reduced to 7859 normal and 194 failure samples, a random-forest baseline reaches an F1 of 0.7659; Threshold Adjustment lifts that by 15.3 percent, the largest relative gain among all tested methods. Random Under-sampling gives the best pre-processing result at 12 percent, and ensemble learning gives the best in-processing result at 13.6 percent. The paper also reports that post-processing methods reduce the variance-to-mean ratio of F1 relative to the baseline, indicating more stable performance, and that Random Under-sampling provides the lowest inference time of all techniques.
Load-bearing premise
The ranking of the twenty techniques is assumed to carry over from one experimental dataset and one random-forest setup to real optical-network failure data.
Editorial extensions
If this is right
- Operators can improve an already-trained failure detector by about 15.3 percent in F1 by adding threshold adjustment, with no retraining and no new data collection.
- In latency-sensitive monitoring, Random Under-sampling is the preferred mitigation, delivering a 12 percent F1 gain with the lowest inference time.
- Post-processing methods also stabilize results: threshold adjustment lowers the variance-to-mean ratio of F1 from 4.83 for the baseline to 3.21.
- Because post-processing acts on predictions rather than on data, its benefit should be less sensitive to data quality than the generative and sampling methods tested.
Reading between the lines
- If the ranking holds, threshold adjustment can likely be stacked on top of any classifier, including the neural-network monitors used in soft-failure identification, giving a model-agnostic class-imbalance fix.
- The 15.3 percent versus 13.6 percent gap between post-processing and in-processing is small enough that on other imbalance ratios the ordering could flip; testing on a public telemetry dataset would show where the break-even lies.
- Because threshold adjustment trades false alarms against missed failures, it could be calibrated to an operator's cost function rather than to F1 alone, an option the paper does not explore.
- Combining Random Under-sampling for training speed with threshold adjustment at inference might capture both benefits, a pipeline the paper treats as separate alternatives.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper compares 20 class-imbalance mitigation techniques (pre-processing, in-processing, and post-processing) for failure detection in optical networks using an experimental testbed dataset. The authors report that the post-processing technique Threshold Adjustment achieves the largest relative F1-score improvement (15.3%) over an unbalanced random-forest baseline (F1 = 0.7659), while Random Under-sampling (RUS) provides the fastest inference. The paper also reports variance-to-mean ratios for the baseline and the best technique in each category, and concludes that post-processing methods are both effective and stable.
Significance. If the empirical ranking is statistically reliable, the paper makes a useful practical contribution by systematically comparing a broad set of imbalance-mitigation methods on real optical-network failure data, and by highlighting post-processing as an under-explored but effective family. The use of an experimental testbed dataset, the breadth of 20 techniques, and the explicit attention to inference-time trade-offs are strengths. However, the evaluation protocol is underspecified, no error bars or confidence intervals are provided for the central comparison, and the reported variance-to-mean ratios are internally inconsistent with the bounded F1 metric. The central claim that Threshold Adjustment is best is therefore currently unverified.
major comments (3)
- [Experimental Dataset and Baseline (Tab. 1)] The manuscript never describes how the available samples are split into training and test sets. Tab. 1 states that the baseline is 'obtained using the original (imbalanced) dataset' and that 'all subsequent experiments using mitigation techniques are evaluated relative to this baseline,' but if F1 scores are computed on the training data, the 15.3% gain of Threshold Adjustment could be an in-sample artifact, especially if its decision threshold is tuned on the same data. Please specify the exact train/validation/test split or cross-validation scheme, and confirm that Threshold Adjustment's threshold is selected on a validation set while F1 is reported on a held-out test set.
- [Results and Discussion (Fig. 4)] The variance-to-mean ratios in Fig. 4 are impossible for a bounded metric like F1. For a baseline mean of 0.7659, the maximum possible variance-to-mean ratio is at most 0.2341 if the variance is computed on F1 values in [0, 1]; the reported values of 4.83, 3.34, 3.47, and 3.21 are an order of magnitude larger. Either the ratio is computed on a different scale (e.g., percentages) or the variance estimate is erroneous. This inconsistency directly undermines the claimed stability benefit and must be corrected and recomputed.
- [Results and Discussion (Figs. 2 and 3)] All F1 results are presented as point estimates without error bars, confidence intervals, or significance tests. With only 194 failure samples after preprocessing, the headline gap between Threshold Adjustment (15.3%) and the best in-processing method (13.6%) could be within run-to-run variability. The manuscript also does not state the hyperparameters used for the 20 techniques (e.g., SMOTE sampling ratios, number of estimators for Bagging/Boosting, threshold selection procedure). Please report the per-run distributions or error bars, apply a paired statistical test across the 100 runs, and provide a full table of hyperparameters so the comparison is reproducible.
minor comments (4)
- [Author line and captions] There are formatting artifacts in the author line ('Y ousuf Moiz Ali') and garbled citation-like text in the Fig. 2 caption; please proofread the final PDF.
- [Tab. 1] Only the F1 score is reported for the baseline; the text states that accuracy, precision, and recall also improved, but these numbers are not shown. For a failure-detection task these metrics are informative and should be reported if space permits.
- [Fig. 3] The inference-time comparison lacks experimental details such as the hardware platform, software versions, and whether timings are averaged over multiple runs; please specify these to make the trade-off claim meaningful.
- [Class Imbalance Mitigation Techniques] Several named techniques (e.g., Cluster-based Massaging, Cost-sensitive Threshold, Reweighting Predictions) are not explicitly cited; if they are variants of methods in [14] or [17], this should be stated.
Circularity Check
No circularity: the F1-gain ranking is an empirical comparison, not a derivation from fitted inputs.
full rationale
The paper's central claim is an empirical ranking of F1-score improvements across pre-, in-, and post-processing class-imbalance mitigation techniques. There is no derivation chain in which a predicted quantity is constructed from the same fitted quantity. The baseline is a measured random-forest F1 score of 0.7659 on the original imbalanced dataset, and each technique's F1 score is measured relative to that baseline. Threshold Adjustment is a standard post-hoc decision-threshold procedure; its reported 15.3% gain is a measured outcome on the evaluation protocol, not a quantity that is defined in terms of the claim itself. The techniques are adopted from external literature (SMOTE, RUS, CTGAN, Bagging, etc.), and the paper's own contributions are experimental rather than derived from its own fitted parameters. The only self-citations appear as related-work references for data augmentation and generative approaches in optical-network failure management; they are not load-bearing for the central ranking. Concerns about the unspecified train/test split or the implausible variance-to-mean ratios in Figure 4 are validity and reproducibility risks, not circularity: they do not show that any conclusion is equivalent to its input by construction. Under the required standard of quoting a specific equation or construction that reduces the result to its inputs, no circular step can be identified.
Assumptions & free parameters
free parameters (3)
- Decision threshold for Threshold Adjustment =
not reported (optimized for F1)
- Sampling ratios for RUS and SMOTE =
not reported
- Number of estimators for Bagging/Boosting =
not reported
assumptions (3)
- domain assumption The experimental testbed dataset from [21] is representative of real optical network failure data.
- domain assumption Random Forest is an appropriate base model for comparing mitigation techniques.
- domain assumption F1 score is the correct objective for this failure detection task.
Cite this review
Pith. "Pith review of Pre-, In-, and Post-Processing Class Imbalance Mitigation Techniques for Failure Detection in Optical Networks." pith.science (2026). https://pith.science/paper/HRPP52NJ
@misc{pith2026250721119,
author = {Pith},
title = {Pith review of: Pre-, In-, and Post-Processing Class Imbalance Mitigation Techniques for Failure Detection in Optical Networks},
year = {2026},
howpublished = {\url{https://pith.science/paper/HRPP52NJ}},
note = {Machine review of arXiv:2507.21119}
}
read the original abstract
We compare pre-, in-, and post-processing techniques for class imbalance mitigation in optical network failure detection. Threshold Adjustment achieves the highest F1 gain (15.3%), while Random Under-sampling (RUS) offers the fastest inference, highlighting a key performance-complexity trade-off.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
A tutorial on machine learning for failure management in optical networks
F . Musumeci, C. Rottondi, G. Corani, S. Shahkarami, F . Cugini, and M. Tornatore, “A tutorial on machine learning for failure management in optical networks”, Journal of Lightwave Technology, vol. 37, no. 16, pp. 4125–4139,
-
[2]
M. Healy, A. Baum, and F . Musumeci, “Addressing data scarcity in ml-based failure-cause identification in opti- cal networks through generative models”, Optical Fiber Technology, vol. 90, p. 104 137, 2025. DOI: 10.1016/j. yofte.2025.104137
arXiv 2025
-
[3]
L. Z. Khan, J. Pedro, N. Costa, L. De Marinis, A. Napoli, and N. Sambo, “Data augmentation to improve perfor- mance of neural networks for failure management in optical networks”, Journal of Optical Communications and Networking, vol. 15, no. 1, pp. 57–67, 2022. DOI: 10.1364/JOCN.472605
-
[4]
H. Lun, M. Fu, Y . Zhang,et al., “A gan based soft fail- ure detection and identification framework for long-haul coherent optical communication systems”, Journal of Lightwave Technology, vol. 41, no. 8, pp. 2312–2322,
-
[5]
L. Z. Khan, P . J. Freire, J. Pedro, N. Costa, A. Napoli, and N. Sambo, “Data augmentation to reduce compu- tational complexity of neural-network-based soft-failure cause identifier”, in 2023 Optical Fiber Communications Conference and Exhibition (OFC), IEEE, 2023, pp. 1–3. DOI: 10.1364/OFC.2023.M3G.3
-
[6]
C. Xing, C. Zhang, B. Y e,et al., “Failure data augmenta- tion for optical network equipment using time-series gen- erative adversarial networks”, in2023 Optical Fiber Com- munications Conference and Exhibition (OFC) , IEEE, 2023, pp. 1–3. DOI: 10.1364/OFC.2023.M3G.4
-
[7]
The potential of data augmentation for failure management in optical networks
N. Sambo, L. Z. Khan, J. Pedro, N. Costa, L. De Marinis, and A. Napoli, “The potential of data augmentation for failure management in optical networks”, in Photonic Networks and Devices, Optica Publishing Group, 2023, NeM3B–1. DOI: 10.1364/NETWORKS.2023.NeM3B.1
-
[8]
L. Z. Khan, J. Pedro, N. Costa, A. Sgambelluri, A. Napoli, and N. Sambo, “Model and data-centric machine learn- ing algorithms to address data scarcity for failure iden- tification”, Journal of Optical Communications and Net- working, vol. 16, no. 3, pp. 369–381, 2024. DOI: 10 . 1364/JOCN.511863
work page 2024
Show all 23 references
-
[9]
Monitoring data augmentation of spectral information using vae and gan for soft-failure identification
L. E. Kruse, S. Kühl, A. Dochhan, and S. Pachnicke, “Monitoring data augmentation of spectral information using vae and gan for soft-failure identification”, in Opti- cal Fiber Communication Conference, Optica Publishing Group, 2024, pp. M3I–4. DOI: 10.1364/OFC.2024.M3I. 4
2024 doi
-
[10]
Shap-assisted ee-lightgbm model for explainable fault diagnosis in practical optical networks
C. Zhang, Y . Chen, M. Zhang, Z. Liu, and D. Wang, “Shap-assisted ee-lightgbm model for explainable fault diagnosis in practical optical networks”, Journal of Op- tical Communications and Networking , vol. 17, no. 2, pp. 81–94, 2025. DOI: 10.1364/JOCN.527872
2025 doi
-
[11]
Application of ml algorithms for prediction of the qot in optical net- works with imbalanced and incomplete data
P . Cichosz, S. Kozdrowski, and S. Sujecki, “Application of ml algorithms for prediction of the qot in optical net- works with imbalanced and incomplete data”, in 2021 International Conference on Software, Telecommunica- tions and Computer Networks (SoftCOM), IEEE, 2021, pp. 1...
2021
-
[12]
A stacking ensemble ml-based failure prediction model for optical networks with imbalanced data
Z. Sun, C. Zhang, M. Zhang, F . Y ang, and D. Wang, “A stacking ensemble ml-based failure prediction model for optical networks with imbalanced data”, in 2023 Asia Communications and Photonics Conference/2023 International Photonics and Optoelectronics Meetings (ACP/POEM), IEE...
2023
-
[13]
Survey on machine learning biases and mitigation techniques
S. Siddique, M. A. Haque, R. George, K. D. Gupta, D. Gupta, and M. J. H. Faruk, “Survey on machine learning biases and mitigation techniques”, Digital, vol. 4, no. 1, pp. 1–68, 2023. DOI: 10.3390/digital4010001
2023 doi
-
[14]
Bias mitigation for machine learning classifiers: A com- prehensive survey
M. Hort, Z. Chen, J. M. Zhang, M. Harman, and F . Sarro, “Bias mitigation for machine learning classifiers: A com- prehensive survey”, ACM Journal on Responsible Com- puting, vol. 1, no. 2, pp. 1–52, 2024. DOI: 10 . 1145 / 3631326
2024
-
[15]
Smote: Synthetic minority over-sampling technique
N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P . Kegelmeyer, “Smote: Synthetic minority over-sampling technique”, Journal of artificial intelligence research , vol. 16, pp. 321–357, 2002. DOI: 10.1613/jair.953
2002 doi
-
[16]
Adasyn: Adaptive synthetic sampling approach for imbalanced learning
H. He, Y . Bai, E. A. Garcia, and S. Li, “Adasyn: Adaptive synthetic sampling approach for imbalanced learning”, in 2008 IEEE international joint conference on neural networks (IEEE world congress on computational intel- ligence), Ieee, 2008, pp. 1322–1328. DOI: 10 . 1109 / IJ...
2008
-
[17]
A study of the behavior of several methods for balancing ma- chine learning training data
G. E. Batista, R. C. Prati, and M. C. Monard, “A study of the behavior of several methods for balancing ma- chine learning training data”,ACM SIGKDD Explorations Newsletter, vol. 6, no. 1, pp. 20–29, 2004.DOI: 10.1145/ 1007730.1007735
2004
- [18]
-
[19]
Exploratory undersam- pling for class-imbalance learning
X.-Y . Liu, J. Wu, and Z.-H. Zhou, “Exploratory undersam- pling for class-imbalance learning”, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernet- ics), vol. 39, no. 2, pp. 539–550, 2008. DOI: 10.1109/ TSMCB.2008.2007853
2008
-
[20]
C. Chen, A. Liaw, L. Breiman,et al., Using random forest to learn imbalanced data. Technical report. University of California, Berkeley, 2004. [Online]. Available: https:// statistics.berkeley.edu/sites/default/files/ tech-reports/666.pdf
2004
-
[21]
Learning long-and short-term temporal patterns for ml- driven fault management in optical communication net- works
M. F . Silva, A. Pacini, A. Sgambelluri, and L. Valcarenghi, “Learning long-and short-term temporal patterns for ml- driven fault management in optical communication net- works”, IEEE Transactions on Network and Service Man- agement, vol. 19, no. 3, pp. 2195–2206, 2022. DOI: 1...
2022
-
[2019]
DOI: 10.1109/JLT.2019.2922586
2019
-
[2023]
DOI: 10.1109/JLT.2022.3227719
2022
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.