{"id":"c1f0b5a5-16cd-4969-a652-6c426776d87e","arxiv_id":"2509.00057","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"On experimental optical-network data, threshold adjustment improves failure-detection F1 by up to 15.3%, while CTGAN data augmentation improves failure-identification F1 by up to 24.2%.","lead":"A benchmark study compares three families of techniques for handling rare failure events in optical networks. For detecting failures, adjusting the decision threshold gives the biggest F1 gain, while generated-data methods help most for identifying which failure occurred.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation protocol lacks a described held-out split for threshold/cost tuning, so the headline F1 gains and cross-method rankings may be optimistically biased; a validation/test split re-run is needed.","rationale":"The paper is a broad benchmark with no code or formal verification. Its central claim is the ranking of imbalance-mitigation methods, which depends entirely on the reported F1 numbers. The most load-bearing condition is that those numbers are unbiased estimates of generalization performance. The text never specifies that post-processing tunables are selected on a held-out validation set; Algorithm 1's pseudocode literally takes true labels as input, and Section C.3 describes tuning the scaling factor to maximize the metric. This makes the reader's weakest assumption correct. A strict three-way split re-run would settle whether the headline gains survive. I do not see an internal inconsistency; the issue is an under-specified and potentially optimistic evaluation protocol, so the reader's CONDITIONAL verdict remains appropriate without change.","tokens_in":28597,"tokens_out":3307,"duration_ms":40778,"concrete_test":"Re-run dataset 1 and dataset 3 with a fixed three-way split (e.g., 60/20/20) stratified by class; tune Algorithm 1's τ, Eq. (5)'s costs, and the Reweighting scaling factor on the validation fold only, then compute F1 on the held-out test fold. Repeat over 100 independent splits and compare the mean and 95% CI with Figs. 13 and 17. If the Threshold Adjustment advantage over EL (13.6%) and RUS (12%) shrinks below the reported margin, or if CTGAN's 24.2% gain is not statistically separable from SMOTE/ROS, the central ranking is not established. Also report paired significance tests across methods.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparative claim rests on F1 improvements reported in Figs. 13, 16, and 17. Algorithm 1 (Threshold Adjustment) requires \"Predicted probabilities ŷ, True labels y\" and searches τ to maximize F1 on those labels; Section C.3 says the Reweighting scaling factor \"can be tuned to maximize the optimization metric\"; Section C.2's cost values in Eq. (5) also need to be set. The experimental section (Section 3) describes averaging over 100 runs but never states that these parameters are tuned on a validation set distinct from the test set whose F1 is reported. If the same labels are used both to select τ (and costs/scaling) and to compute the reported F1, the 15.3% detection gain and the ranking of post-processing over pre/in-processing methods are optimistically biased; post-processing methods are the most affected because their entire mechanism is a test-time decision-rule fit. The paper's own conclusion that Threshold Adjustment is best \"with reduced dependence on the quality of the dataset\" therefore rests on an unverified evaluation protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a comparative study of pre-, in-, and post-processing class-imbalance mitigation techniques for failure detection and failure identification in optical networks, using three experimental datasets with varying class overlap. The central claims are that post-processing Threshold Adjustment gives the largest F1 improvement (up to 15.3%) for binary failure detection, Generative AI methods (especially CTGAN) give the largest gains (up to 24.2%) for multi-class failure identification, and that method choice should depend on class overlap and latency constraints. The evaluation averages results over 100 runs and uses FDR and PCA to characterize dataset separability.","tokens_in":28900,"tokens_out":3361,"duration_ms":42189,"significance":"If the reported improvements and rankings are unbiased, the paper would be a useful practical benchmark for an under-explored direction, with broad method coverage and validation on experimental testbed data. The use of 100-run averaging and VMR for selected methods is a positive feature. However, the evaluation protocol as described permits post-processing parameters to be tuned on the same labels whose F1 is then reported, which directly affects the headline improvements and the decision blueprint. The contribution is therefore conditional on a corrected evaluation protocol.","major_comments":[{"comment":"The headline F1 improvements may be optimistically biased because the paper does not describe a held-out validation split. Algorithm 1 takes 'Predicted probabilities ŷ, True labels y' and searches τ to maximize F1 on those labels; Section C.3 says the Reweighting scaling factor 'can be tuned to maximize the optimization metric'; and Eq. (5) requires misclassification costs CFP/CFN that must be set. If these parameters are selected using the same test labels whose F1 is then reported, the 15.3% Threshold Adjustment gain (Fig. 13) and the 16.4% Cost-sensitive Threshold gain (Fig. 17) are inflated. This is load-bearing because the rankings and the Fig. 19 blueprint rest on these numbers. Please add an explicit train/validation/test split, select all tuning parameters on the validation set, and report test F1 with confidence intervals.","section":"Section 3 / 4.A, Algorithm 1, Eq. (5), Section C.3"},{"comment":"The paper makes comparative claims about methods whose F1 scores differ by only a few percentage points, but it reports only averages over 100 runs. No standard deviations, error bars, or significance tests are provided; VMR is reported for only three methods in Fig. 15. Without variance estimates, the cross-method ranking and statements such as 'post-processing has limited impact' in multi-class settings are not statistically supported. Please report per-method variability and use paired tests or confidence intervals for the key comparisons.","section":"Section 4.A/4.B, Figs. 13, 16, 17"},{"comment":"The inference-time comparisons are central to the latency-critical recommendations (RUS fastest, Threshold Adjustment slower, CTGAN minimal inference time), but the measurement protocol is absent. No hardware details, number of timing repetitions, units, or axis labels are given in Figs. 14 and 18, and no variance or confidence intervals are reported for timings. Add a concrete timing protocol so the latency claims can be assessed.","section":"Section 4.A, Figs. 14, 18"}],"minor_comments":[{"comment":"The statement at the end of Section 2.C that post-processing methods 'do not change the learned decision function' is contradicted by Algorithm 2 (Sample Weighting), which retrains the model on reweighted training data. Please either reclassify this method or revise the statement.","section":"Section 2.C, Algorithm 2"},{"comment":"Several figures contain garbled placeholder tokens (e.g., '/uni00000025/...') and missing axis labels/units, making the quantitative results difficult to verify. Please regenerate the figures with readable labels and value annotations.","section":"Figures 13, 14, 17, 18"},{"comment":"The Perturbation formula uses rand() without specifying the random-number generation policy or seed. Since results are averaged over 100 runs, please state whether the same seed/process was used across methods.","section":"Section 2.A.5, Eq. (1)"},{"comment":"The choice of model differs between failure detection (Random Forest) and failure identification (Neural Network), and the RF uses default scikit-learn parameters while the NN uses parameters from [13]. Please justify this asymmetry explicitly, since it may affect cross-task comparisons.","section":"Section 3.A, Section 3.B"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's main contribution is a broad empirical comparison, and the experimental datasets are valuable. The evaluation protocol, however, has a load-bearing weakness: post-processing parameters appear to be tuned on the same data used to report F1. This needs a re-run with a proper validation split before the central rankings can be accepted. The paper also includes unusual author-name spacing and garbled figure text, which should be cleaned up editorially."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a genuinely useful benchmark paper, not a breakthrough. It compares a wide range of pre-, in-, and post-processing imbalance methods on three experimental optical-network datasets, and it is the first to bring post-processing methods into this domain. The FDR-based analysis of class overlap is a nice touch, and the two-dataset validation (hard vs soft failures) gives the qualitative story more weight. The paper also reports 100-run averages and checks variance-to-mean ratio for the top methods; that is more than many comparable papers do.\n\nThe soft spot is the one the stress-test note points to, and I don't think it's manufactured. Algorithm 1 is threshold adjustment that searches tau to maximize F1 on true labels; Section C.3 says the reweighting scaling factor can be tuned to maximize the optimization metric; the cost values in Eq. (5) have to be set somehow. The experimental section says everything is averaged over 100 runs, but never says a validation split was used to tune these parameters before computing the reported F1. If the same test labels are used both for tuning and reporting, the 15.3% detection gain and the ranking of post-processing over pre/in-processing are optimistically biased, and post-processing is exactly the family that is most sensitive to this because its whole mechanism is fitting a decision rule on the evaluation labels. I agree this is load-bearing: the rankings rest on those F1 numbers.\n\nA few qualifications. The paper's qualitative conclusions—post-processing helps most in binary failure detection, GenAI helps most when classes are separable, and the choice depends on latency—are plausible and partly supported by the consistency between datasets 1 and 2. The methods themselves are all standard, so the novelty is the systematic comparison, not any single algorithm. The figures are dense but readable enough. There is also no explicit statement about significance testing; only VMR for three selected methods. These are fixable in a revision.\n\nWho is this for? Someone working on failure management in optical networks who wants practical method-selection guidance. It is not a theoretical contribution. In my view it deserves a serious referee process, but the referees should condition acceptance on a re-run with a held-out validation set for threshold/cost/scaling parameters and error bars across all compared methods.","headline":"Useful practical benchmark with a real evaluation-protocol hole: threshold/cost/scaling parameters appear to be tuned on the same test labels used to report F1, so the headline gains and rankings need a held-out validation re-run before they can be trusted.","tokens_in":29281,"tokens_out":2765,"would_cite":true,"duration_ms":32609,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Threshold tuning lifts optical fault detection F1 by 15.3 percent, the paper reports.","keywords":["class imbalance","optical networks","failure detection","failure identification","F1 score","threshold adjustment","generative AI","post-processing"],"falsifier":"Run the same twenty methods with a strict three-way split: tune the threshold, costs, and reweighting scale on a validation fold, then report F1 on a held-out test fold. If Threshold Adjustment's margin over Random Under-Sampling and the baseline shrinks below 15.3% or reverses, the claim that post-processing is best for detection fails.","tokens_in":28575,"feed_emoji":"📡","tokens_out":3723,"duration_ms":46880,"temperature":0.7,"pith_summary":"This paper asks which class-imbalance remedy actually works best when machine learning is used to detect and identify failures in optical networks. It directly compares pre-processing (resampling and generative synthetic data), in-processing (weighted learning, ensembles, meta-learning), and post-processing (threshold and cost adjustments) on experimental network datasets. The central finding is that the best method depends on the task: for binary failure detection, adjusting the decision threshold after training gives the largest F1-score gain, up to 15.3%, while for multi-class failure identification, generative AI data augmentation, especially CTGAN, gives gains up to 24.2%. Class overlap and latency constraints determine which family of methods should be chosen, and the paper condenses this into a practical selection flowchart. The reader should care because these are cheap, deployable choices that can improve network failure management without redesigning the model.","feed_headline":"Threshold tuning lifts optical fault detection F1 by 15.3 percent","feed_subtitle":"A head-to-head test of imbalance fixes says post-processing wins for detection, generative data for identification.","key_machinery":"The central organizing device is the pre/in/post-processing taxonomy, with the Fisher Discriminant Ratio (FDR) used as a scalar measure of class overlap that predicts when generative methods will succeed. The load-bearing algorithmic object for the detection result is Algorithm 1, a threshold search that scans candidate probability thresholds and selects the one maximizing F1 on labeled predictions. For the identification result, the load-bearing objects are CTGAN and CVAE, conditional generative models that synthesize minority-class tabular samples. The paper's practical output is a decision flowchart combining problem type, class overlap, and latency.","core_discovery":"Across experimental optical-network datasets with different degrees of class overlap, the paper claims that no single imbalance-mitigation technique dominates. For failure detection, post-processing methods—particularly Threshold Adjustment, which searches the probability threshold that maximizes F1—outperform all pre- and in-processing alternatives, raising F1 by up to 15.3% over baseline with the lowest variance across runs. For failure identification, generative AI methods, especially CTGAN, provide the largest improvement, up to 24.2%, when classes are well separated. The paper attributes this to the Fisher Discriminant Ratio: when classes overlap heavily, generative models produce poor","pith_inferences":["An implication not drawn in the paper: for networks that already have deployed failure detectors, threshold adjustment may deliver most of the benefit of data rebalancing at near-zero operational cost, since it touches only the prediction layer.","The paper's FDR-based explanation suggests a cheap, testable diagnostic: compute the average Fisher Discriminant Ratio on monitoring features first, then select a method family; this could be validated on other tasks such as failure localization.","Because the reported F1 numbers are computed after tuning threshold, cost values, and reweighting scale on the same labeled data used for evaluation, a strict train/validation/test split could change the size of the 15.3% and 24.2% gains. This is an open question, not a claim the paper makes.","The overlap-dependent ranking likely extends beyond optical networks to other imbalanced tabular classification problems with continuous features, though the paper only argues for the optical-network setting."],"forward_implications":["For binary failure detection, operators with an already-trained model can get the largest F1 gain by re-tuning the decision threshold on the model's output probabilities, without retraining or resampling.","For multi-class failure identification with well-separated classes, synthetic minority-class data from CTGAN provides the biggest gain, but post-processing methods require careful per-class tuning and give smaller improvements.","When inference latency is critical, Random Under-Sampling is the recommended choice for detection despite not being the top performer in F1.","Class overlap, measured by FDR, is a useful predictor of method family: generative AI fails in low-FDR (overlapping) datasets and excels in high-FDR (separable) datasets.","The choice of mitigation strategy reduces to a small set of questions—detection vs identification, overlap vs separation, latency-critical vs not—rather than a single universally best method."],"supporting_citations":[{"why":"Supplies the experimental hard- and soft-failure datasets and testbed used for failure detection, including the baseline comparison.","marker":"[52]"},{"why":"Supplies dataset 3 and the baseline neural-network architecture used for failure identification.","marker":"[13]"},{"why":"Defines SMOTE, the canonical over-sampling method compared across all datasets.","marker":"[26]"},{"why":"Defines CTGAN, the generative method that achieves the largest failure-identification gain.","marker":"[36]"},{"why":"Provides the pre/in/post-processing taxonomy and the massaging and perturbation label-flipping methods.","marker":"[24]"},{"why":"Justifies threshold adjustment for imbalanced classification and underpins Algorithm 1.","marker":"[46]"},{"why":"Prior study testing model- and data-centric imbalance methods on experimental optical data, the work this paper extends.","marker":"[21]"},{"why":"Supplies the hybrid ensemble plus undersampling approach and SHAP analysis used as in-processing context.","marker":"[22]"}],"fun_headline_variants":["Threshold tuning lifts optical fault detection F1 15.3%, GenAI for ID","No single imbalance fix: threshold for detection, CTGAN for ID","Post-processing beats SMOTE for detection; GenAI wins identification","15.3% F1 gain via threshold tuning for fault detection; GenAI for ID","Class overlap determines best fix: threshold for detection, GenAI for ID"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The headline F1 gains assume the optimal threshold, cost values, and reweighting scale are chosen on data separate from the data used to report the score; if the same labeled examples were used for both tuning and scoring, the gains would look better than they truly are.","fun_headline_variants_meta":{"raw":{"variants":["Threshold tuning lifts optical fault detection F1 15.3%, GenAI for ID","No single imbalance fix: threshold for detection, CTGAN for ID","Post-processing beats SMOTE for detection; GenAI wins identification","15.3% F1 gain via threshold tuning for fault detection; GenAI for ID","Class overlap determines best fix: threshold for detection, GenAI for ID"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000742,"raw_usage":{"total_tokens":3128,"prompt_tokens":707,"completion_tokens":2421,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":451,"completion_tokens_details":{"reasoning_tokens":2321}},"tokens_in":451,"tokens_out":2421,"duration_ms":18390,"temperature":1.0,"reasoning_tokens":2321,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:43:47.153956+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same twenty methods with a strict three-way split: tune the threshold, costs, and reweighting scale on a validation fold, then report F1 on a held-out test fold. If Threshold Adjustment's margin over Random Under-Sampling and the baseline shrinks below 15.3% or reverses, the claim that post-processing is best for detection fails.","supporting_citations":[{"cited_title":"Learning long-and short-term temporal patterns for ML-driven fault management in optical communication networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the experimental hard- and soft-failure datasets and testbed used for failure detection, including the baseline comparison."},{"cited_title":"Data augmentation to improve performance of neural networks for failure management in optical networks,","cited_arxiv_id":null,"evidence_quote":"Supplies dataset 3 and the baseline neural-network architecture used for failure identification."},{"cited_title":"Smote: synthetic minority over-sampling technique,","cited_arxiv_id":null,"evidence_quote":"Defines SMOTE, the canonical over-sampling method compared across all datasets."},{"cited_title":"Bias mitiga- tion for machine learning classifiers: A comprehensive survey,","cited_arxiv_id":null,"evidence_quote":"Provides the pre/in/post-processing taxonomy and the massaging and perturbation label-flipping methods."},{"cited_title":"Finding the best classification threshold in imbalanced classification,","cited_arxiv_id":null,"evidence_quote":"Justifies threshold adjustment for imbalanced classification and underpins Algorithm 1."},{"cited_title":"Model and data-centric machine learning algorithms to address data scarcity for failure identification,","cited_arxiv_id":null,"evidence_quote":"Prior study testing model- and data-centric imbalance methods on experimental optical data, the work this paper extends."},{"cited_title":"Shap-assisted ee-lightgbm model for explainable fault diagnosis in practical optical networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the hybrid ensemble plus undersampling approach and SHAP analysis used as in-processing context."}],"review_version":1}