{"id":"f94f004e-6ec6-4142-b510-74365d2281ee","arxiv_id":"2507.01571","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Ruleset tuning that reduces label imbalance improves DeepCASE's alert classification and the correctness of its explanations in a real SOC dataset.","lead":"This paper measures how the imbalance between true incidents and false alarms in a security operations center affects an alert-classification method called DeepCASE. It finds that reducing this imbalance, by tuning detection rules, improves both classification accuracy and the quality of the machine's explanations.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-dataset hyperparameter optimization is a confound in Table 1: low-IR gains largely vanish under default hyperparameters, so the -1.03 label-IR coefficient likely absorbs hyperparameter effects.","rationale":"The reader's CONDITIONAL verdict is reasonable, but the most load-bearing threat is not the expert ground truth (which the paper itself disclaims in Section 5) nor the acknowledged label/event-IR collinearity (which the authors handle with a caveat). It is the unacknowledged confound between label IR and per-dataset hyperparameter optimization. Table 4 shows the optima differ greatly across datasets; Section 3.2 states every run uses the dataset-specific optima. The regression therefore varies two things at once. The paper's own control experiment (Appendix C.1) provides direct evidence that hyperparameters matter more than IR for the low-IR dataset: optimized macro F1 is 0.854 vs 0.447 with default hyperparameters, so the low-IR advantage in Table 1 is substantially manufactured by optimization. The hyperparameter control experiment is not included in the regression, so Table 1 cannot rule this out. If the authors rerun with fixed hyperparameters and the coefficient survives, the conclusion is solid; if not, the abstract overstates the role of imbalance. This is a concrete, checkable threat. I therefore disagree with the reader's identification of the weakest assumption, while agreeing with the CONDITIONAL outcome.","tokens_in":19565,"tokens_out":8724,"duration_ms":96680,"concrete_test":"Rerun every tuned and control dataset with a single fixed hyperparameter configuration (e.g., the original DeepCASE defaults from Appendix C.1) in addition to the optimized runs, then refit the Table 1 robust linear model with a hyperparameter-set indicator or on the fixed-configuration runs only. If the label-IR coefficient drops substantially (e.g., below -0.5) or loses significance, the -1.03 effect is largely a hyperparameter artifact rather than evidence that imbalance per se drives performance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim is that label IR has a far larger effect on relaxed F1 than other dataset characteristics (Table 1, coefficient -1.03). But every dataset in Table 1 is run with its own optimized hyperparameters (Section 3.2: 'Each run uses the hyperparameters obtained from the hyperparameter optimization described in Appendix A'; Table 4 shows materially different optima, e.g., min cluster size 50 for unfiltered vs 5 for all tuned sets, epsilon 0.8 vs 1.0). Hyperparameters therefore co-vary with IR across the entire regression, so the label-IR coefficient captures the joint effect of IR and hyperparameter choice, not IR alone. The paper's own hyperparameter control experiment (Appendix C.1, Figure 3) shows the low-IR dataset's macro F1 collapses from 0.854 (optimized) to 0.447 (original-work defaults), while other IR levels change far less (unfiltered 0.502 to 0.499, high 0.486 to 0.468, medium 0.802 to 0.754). The text confirms the pattern for relaxed F1 too: 'the performance ... for the tuned datasets ... is higher than that in the hyperparameter control experiment, especially for the low IR dataset.' Since the control datasets in Table 1 also use the corresponding tuned hyperparameters, the regression cannot separate imbalance from hyperparameter adjustment. This threatens both the performance conclusion and the explanation comparison (Section 4.2), which likewise uses per-dataset optimized models.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper uses a real SOC alert dataset with replayed attacks to study whether ruleset tuning, by reducing label imbalance, improves the classification performance and explanation quality of DeepCASE. The authors construct unfiltered, high-IR, medium-IR, and low-IR datasets, plus control datasets varying dataset size, event imbalance, heterogeneity, and dimensionality, and run DeepCASE on each with per-dataset hyperparameter optimization. A regression over 80 runs reports a label-IR coefficient of -1.03 on a relaxed F1-score (Table 1), and a comparison of attention vectors against one expert's labels suggests that tuned datasets produce more similar explanations (Figure 2). The paper concludes that ruleset tuning can benefit both classification performance and explainability of alert post-processing methods.","tokens_in":19884,"tokens_out":7805,"duration_ms":88350,"significance":"The practical question addressed here is important: if simple ruleset tuning improves the input quality for ML-based alert post-processing, then SOCs have a low-cost lever for improving automation. The paper makes a useful contribution by working with real SOC data, by proposing and using a relaxed F1 metric that reflects analyst workload, and by attempting to control for dataset size, heterogeneity, and dimensionality. It also explicitly discusses several known pitfalls in ML-for-security evaluations. However, the central attribution of the performance gain to label imbalance is currently confounded by per-dataset hyperparameter optimization and by collinearity with event IR, and the explanation-correctness claim is weakened by the paper's own admission that expert agreement is no better than DeepCASE-to-expert agreement. These are load-bearing issues that require additional experimental and statistical work.","major_comments":[{"comment":"The regression that supports the paper's main quantitative claim conflates label imbalance with per-dataset hyperparameter optimization. Section 3.2 states that each run uses the hyperparameters obtained from the optimization in Appendix A, and Table 4 shows materially different optima across datasets (e.g., minimum cluster size 50 for unfiltered vs. 5 for all tuned sets; epsilon 0.8 vs. 1.0). The hyperparameter control experiment in Appendix C.1 shows that under the original default hyperparameters the low-IR dataset's macro F1 falls from 0.854 to 0.447, whereas the other datasets change far less. The coefficient of -1.03 on label IR in Table 1 therefore absorbs the joint effect of imbalance and hyperparameter choice, not the effect of imbalance alone. Because Section 4.2's explanation comparison also uses the per-dataset optimized models, the same confound applies to the explanation results. A fixed-hyperparameter comparison, or a design that treats hyperparameters as a factor and estimates their interaction, is required to support the attribution.","section":"§3.2, Table 1, Appendix C.1"},{"comment":"The standard errors and confidence intervals in Table 1 are computed on 80 runs that are not independent: the runs consist of five repetitions of each of 16 dataset configurations (four IR-experiment datasets plus twelve control datasets). A robust linear model applied to the run-level data treats the five runs from one configuration as independent observations, which is pseudoreplication and will produce overconfident intervals. The reported interval [-1.04, -1.02] for the label-IR coefficient is implausibly narrow given the dispersion visible in Figure 1. The regression should be estimated with standard errors clustered by dataset configuration, or using configuration-level means.","section":"Table 1"},{"comment":"The paper cannot separate label IR from event IR. The event-IR variable was dropped from the regression because its VIF exceeded 10, and Section 5 acknowledges that part of the label-IR effect may be due to event IR. Because ruleset tuning changes both quantities together, the evidence supports a claim about imbalance generally, not specifically about label imbalance as the title and research questions assert. The authors should either obtain data or experiments that vary label IR while holding event IR fixed, or reframe the central claim accordingly.","section":"Table 1, §5"},{"comment":"The explanation-correctness conclusion is stronger than the paper's own validity caveat allows. Section 5 states that expert 1 and expert 2 agreed with each other about as much as DeepCASE agreed with expert 1, and that 'we cannot draw strong conclusions from the correctness of DeepCASE's explanations.' Given that statement, Figure 2's comparison of DeepCASE to expert 1's vectors cannot support the Section 4.2 conclusion that tuning 'effectively assists DeepCASE in providing correct explanations.' At most it shows that tuning moves DeepCASE closer to one expert's judgments. The conclusion should be reworded, or additional validation of the ground truth should be provided.","section":"§4.2, §5, Appendix D"},{"comment":"The acknowledged temporal data snooping is not shown to be neutral across treatments. The procedure copies test-set Incident sequences into the training set until the train and test Incident percentages are similar, so the amount of copying is a function of the imbalance manipulation itself. The statement in Section 5 that the pitfall is 'present equally in all performed experiments' needs quantitative support: the authors should report, per dataset, how many test sequences were copied and how the resulting training-set composition differs, and ideally run a sensitivity analysis without copying.","section":"§3.2, §5"}],"minor_comments":[{"comment":"The text uses 'Variance Influence Factor'; the standard term is 'Variance Inflation Factor'.","section":"§3.2"},{"comment":"There is a typo in 'context hetetogeneity'; it should read 'heterogeneity'.","section":"§3.2"},{"comment":"The KS test p-value of 0.15 only indicates a failure to reject a distributional hypothesis; it is not a measure of variance explained. Reporting R-squared or adjusted R-squared would make the fit assessment more informative.","section":"§4.1, Table 1"},{"comment":"The conclusion states that 'label imbalance' is the key driver, without the hedge that Section 5 applies ('be it event or label'). Align the conclusion with the acknowledged ambiguity.","section":"§6"},{"comment":"The data and code are described as archived at the institution and shared upon request; a public artifact would materially strengthen reproducibility of the regression and control experiments.","section":"Data Availability"}],"recommendation":"major_revision","confidential_remarks":"I see value in this paper and would not reject it. The practical message that ruleset tuning can help DeepCASE may survive a more careful analysis, but the current manuscript's attribution to label IR specifically is not supported because hyperparameters, event IR, and label IR are confounded, and because the expert ground truth is admitted to be weak. The required fixes are experimental and statistical rather than cosmetic, hence major revision rather than minor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The headline result — that label imbalance, rather than dataset size, heterogeneity, or dimensionality, drives DeepCASE's performance — is not established, because the regression's label-IR coefficient is entangled with per-dataset hyperparameter optimization. The paper's own control experiment (Appendix C.1, Figure 3) shows the low-IR dataset's macro F1 collapses from 0.854 under optimized hyperparameters to 0.447 under the original defaults, while the unfiltered, high-IR, and medium-IR datasets barely move. Since every dataset in Table 1 runs with its own tuned hyperparameters, the -1.03 coefficient plausibly absorbs a good chunk of hyperparameter effect. That is the central weakness, and it is load-bearing.\n\nWhat is genuinely new: this is the first study I know of that tries to measure imbalance effects on alert post-processing using real SOC data, and the control-dataset design is a serious attempt to separate imbalance from dataset size, heterogeneity, and dimensionality. The relaxed F1 metric that treats rejected incidents as TPs and rejected non-incidents as FPs is a reasonable operational approximation. The authors are also transparent about their limitations — the test-to-train copying of incident sequences, the collinearity between event IR and label IR, and the weak expert agreement on explanations. That honesty is appreciated.\n\nWhere else does it strain? The explanation-correctness claim (RQ2) rests on a ground truth where expert 1 and expert 2 agree only about as much as DeepCASE agrees with expert 1. The paper itself concedes no strong conclusions can be drawn. And the explanation analysis skips the control datasets, so the apparent improvement under tuning could just be the hyperparameter effect again.\n\nWho benefits from this paper: researchers working on SOC alert triage, especially those applying DeepCASE or similar semi-supervised methods. It is also a useful methodological example of how to build control datasets, even if the controls don't fully save the main claim.\n\nMy recommendation: a serious referee should see this. The empirical work, the real data, and the control design deserve review, and a competent reviewer will push the authors to either fix the hyperparameter confound — e.g., compare per-IR fixed hyperparameters, or move the default-hyperparameter results into the main regression — or soften the attribution from 'label imbalance' to 'dataset tuning.' As it stands, I would not accept the main claim, but the paper is a solid major-revision candidate.","headline":"An ambitious case study with real SOC data whose central imbalance claim is entangled with per-dataset hyperparameter tuning; worth review, but the main regression doesn't isolate label imbalance.","tokens_in":20351,"tokens_out":3518,"would_cite":true,"duration_ms":36440,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Imbalanced alert streams degrade both the performance and the explanation quality of DeepCASE, and tuning detection rules to reduce imbalance restores both.","keywords":["Security Operations Center","network intrusion detection","label imbalance","Imbalance Ratio","ruleset tuning","DeepCASE","alert explainability","relaxed F1-score"],"falsifier":"Feed DeepCASE a version of the Low-IR dataset in which the same imbalance ratio is reached by randomly under-sampling Non-Incident sequences instead of by retroactive rule filtering, while keeping the unfiltered event types; if the relaxed F1 gain disappears, the effect is caused by rule-based complexity reduction, not by label imbalance itself.","tokens_in":19385,"feed_emoji":"🛡️","tokens_out":7946,"duration_ms":93028,"temperature":0.7,"pith_summary":"This paper is trying to establish that label imbalance is what hurts automated alert post-processing in security operations centers, and that the ordinary practice of tuning detection rules can fix it. The evidence is a case study of DeepCASE, a semi-supervised alert-context classifier, run on 24.5 million alerts from a real SOC with 616 replayed successful attacks. The paper builds datasets that vary imbalance by simulating the SOC's rule-filtering process while controlling for dataset size, heterogeneity, and dimensionality. Its main quantitative result is a regression on a workload-aware relaxed F1-score in which the imbalance ratio has a coefficient around $-1.03$, roughly an order of magnitude larger than any other tested factor. A second analysis finds that DeepCASE's attention-based explanations become more similar to an expert's judgment as imbalance falls, though the paper explicitly qualifies this result because the two experts did not agree strongly with each other.","feed_headline":"Ruleset tuning lifts DeepCASE's accuracy and its explanations","feed_subtitle":"Real SOC data show label imbalance dominates dataset size and heterogeneity as a driver of performance.","key_machinery":"The central object is the Imbalance Ratio, defined in the paper as $\\mathrm{IR}=1-\\frac{1}{n_c-1}\\sum_{i=1}^{n_c}\\frac{n_{c_i}}{n-n_{c_i}}$, which approaches 1 for highly imbalanced class distributions. The argument is carried by a robust linear regression on the relaxed F1-score, a workload-aware metric that treats alerts rejected by DeepCASE as analyst workload, over datasets that separately vary label imbalance, dataset size, context heterogeneity, and dimensionality. For explanations, the machinery is the cosine similarity between DeepCASE's vectors of total attention per event and a manually constructed expert ground truth. The proposed mechanism for improvement is that disabling noisy detection rules removes many majority-class false-positive alerts, lowering the imbalance without introducing new data.","core_discovery":"The paper's central claim, stated on its own terms, is that label imbalance is the dominant determinant of DeepCASE's operational classification performance and that reducing imbalance through ruleset tuning improves both classification and explanation quality. The regression on the relaxed F1-score gives the label imbalance ratio a coefficient of $-1.03$ ($p < 0.01$), while dataset size, heterogeneity, and unique events have coefficients between $-0.04$ and $0.01$, leading the authors to conclude that imbalance, not these other data characteristics, drives the loss. On the explanation side, a substantial share of DeepCASE's vectors of total attention per event are far from the expert-labeled relevance vectors on the unfiltered data, and the whole similarity distribution shifts upward as the imbalance ratio falls through the tuned datasets. The paper concludes that ruleset tuning, by suppressing noisy majority-class alerts, is an effective way to improve both the accuracy and the explainability of alert post-processing.","pith_inferences":["If the causal story is right, data-level imbalance remedies such as under-sampling the majority class or oversampling incidents should reproduce most of the tuning gain; this is a testable hypothesis the paper does not run.","The explanation-correctness result is only as strong as expert consensus; with expert agreement comparable to DeepCASE-to-expert agreement, a panel-based ground truth would be needed to certify the improvement.","Applying the same tuning simulation to NoDoze or AlertPro would test whether the imbalance effect generalizes beyond DeepCASE to other context-based alert triage methods.","The relaxed F1-score could be adopted more widely as an operational performance metric for alert triage, but its usefulness depends on SOCs making the analyst workload cost of rejected alerts explicit."],"forward_implications":["SOCs that already tune their rulesets can expect DeepCASE-style post-processing to retrieve more incidents and reject fewer non-incident alerts.","Label imbalance, rather than dataset size or context heterogeneity, should be reported and controlled in any evaluation of alert post-processing methods.","Micro-averaged F1 masks minority-class failures in imbalanced SOC data, so workload-aware metrics such as the relaxed F1-score should be preferred.","Reducing imbalance through input-data quality offers a complementary path to improving automation alongside algorithmic fixes such as focal loss or cost-sensitive learning.","Ruleset tuning alone reduced irrelevant alerts by 99.6 percent in the study, meaning traditional detection hygiene and ML-based post-processing are not competing approaches."],"supporting_citations":[{"why":"Supplies DeepCASE, the post-processing method under study, along with the baseline hyperparameters and the attention-vector explanation mechanism that the experiments vary.","marker":"[8]"},{"why":"Defines the multi-class Imbalance Ratio used to quantify label imbalance and to set up the dataset comparisons.","marker":"[27]"},{"why":"Documents ruleset tuning as a standard SOC workflow that suppresses noisy rules and thereby reduces imbalance, motivating the tuning simulation.","marker":"[7]"},{"why":"Provides the real SOC dataset with replayed successful attacks that supplies the Incident and Non-Incident ground truth.","marker":"[31]"},{"why":"Supplies the catalogue of machine-learning pitfalls such as spurious correlations and data snooping, which shape the design of the control datasets.","marker":"[21]"},{"why":"Grounds the need for human evaluation of explanations and supports the expert-labeling setup used in the explanation-correctness analysis.","marker":"[24]"},{"why":"Characterizes the extreme imbalance of real SOC alert data, establishing the real-world relevance of the research question.","marker":"[5]"}],"fun_headline_variants":["Ruleset tuning curbs imbalance, improving DeepCASE's accuracy and explanations","Less class imbalance via ruleset tuning lifts DeepCASE's performance and explainability","DeepCASE gains accuracy and explainability when ruleset tuning reduces label imbalance","Label imbalance dominates DeepCASE's failures; ruleset tuning fixes both accuracy and expl"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quality of an explanation is measured by how well it matches a single security expert's opinion, yet the paper's own validation shows that two experts agree with each other about as much as DeepCASE agrees with one expert, so the measured improvement in explanation correctness has no settled meaning without stronger expert consensus.","fun_headline_variants_meta":{"raw":{"variants":["Ruleset tuning curbs imbalance, improving DeepCASE's accuracy and explanations","Less class imbalance via ruleset tuning lifts DeepCASE's performance and explainability","DeepCASE gains accuracy and explainability when ruleset tuning reduces label imbalance","Label imbalance dominates DeepCASE's failures; ruleset tuning fixes both accuracy and explanations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000175,"raw_usage":{"total_tokens":1255,"prompt_tokens":887,"completion_tokens":368,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":282}},"tokens_in":503,"tokens_out":368,"duration_ms":4844,"temperature":1.0,"reasoning_tokens":282,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:47:30.216980+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Feed DeepCASE a version of the Low-IR dataset in which the same imbalance ratio is reached by randomly under-sampling Non-Incident sequences instead of by retroactive rule filtering, while keeping the unfiltered event types; if the relaxed F1 gain disappears, the effect is caused by rule-based complexity reduction, not by label imbalance itself.","supporting_citations":[{"cited_title":"DeepCASE: Semi-Supervised Contextual Analysis of Security Events,","cited_arxiv_id":null,"evidence_quote":"Supplies DeepCASE, the post-processing method under study, along with the baseline hyperparameters and the attention-vector explanation mechanism that the experiments vary."},{"cited_title":"’give me structure’: Synthesis and evaluation of a (network) threat analysis process supporting tier 1 investigations in a security operation center,","cited_arxiv_id":null,"evidence_quote":"Provides the real SOC dataset with replayed successful attacks that supplies the Incident and Non-Incident ground truth."},{"cited_title":"Dos and don’ts of machine learning in computer security,","cited_arxiv_id":null,"evidence_quote":"Supplies the catalogue of machine-learning pitfalls such as spurious correlations and data snooping, which shape the design of the control datasets."},{"cited_title":"True attacks, attack attempts, or benign triggers? an empirical measurement of network alerts in a security operations center,","cited_arxiv_id":null,"evidence_quote":"Characterizes the extreme imbalance of real SOC alert data, establishing the real-world relevance of the research question."}],"review_version":1}