{"id":"947640cc-b3e4-4bd9-a3d9-fb6f5f67bb9c","arxiv_id":"2501.14694","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"AutoGAD selects hyperparameters for self-supervised graph anomaly detection without labels by maximizing the contrast score margin of the anomaly scores.","lead":"This paper shows that many graph anomaly detection methods tune their settings using ground-truth labels, leaking information and inflating reported performance. It proposes AutoGAD, a label-free way to choose settings by measuring how separated the anomaly scores are, and tests it on ten methods and ten datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1 justifies Eq. 2 on true labels, but AutoGAD selects on Eq. 3 with pseudo-labels and a supplied k; the theoretical guarantee does not cover the actual selection procedure.","rationale":"After reading the paper, the strongest claim is that AutoGAD can perform label-free hyperparameter selection and approximately match the best AUC. The load-bearing assumption is that the internal metric T(f) of Eq. 3 is a reliable proxy for detection performance. The paper's Theorem 1 does not establish this for Eq. 3: it is proved for Eq. 2 using true anomaly and normal sets, while Eq. 3 uses top-k pseudo-labels and the remaining n−k objects. The authors explicitly assume the anomaly ratio is approximately known, which is a non-trivial prior and weakens the 'truly unsupervised' framing. The empirical evidence is suggestive—gains over median are positive for 8/10 algorithms—but Table 5 contains several large negative gains, so the 'close to max' claim is not uniform. A direct rank-correlation test between T and AUC across configurations would resolve whether the selection criterion actually tracks performance. The missing DSV baseline is a secondary issue; it does not affect the internal validity of the criterion. My recommendation is to keep the reader's CONDITIONAL verdict: the paper makes a valuable point about label leakage and proposes a plausible heuristic, but the theoretical justification and the robustness of the metric need strengthening before the central claim can be fully accepted.","tokens_in":33062,"tokens_out":7099,"duration_ms":61012,"concrete_test":"For each algorithm and dataset in Tables 3–5, compute the Spearman rank correlation between T(f) from Eq. 3 and true AUC over the full grid of hyperparameter configurations used in the paper. If the median correlation across algorithm–dataset pairs is below ~0.3, or if more than 20% of pairs show ρ<0.2, then AutoGAD's selection criterion is not reliably aligned with detection performance and the 'approximately best possible' claim fails. This test directly checks whether the pseudo-label metric is a valid proxy, independent of the theorem's assumptions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1.3 replaces Eq. 2 with Eq. 3, where the 'normal' set is the remaining n−k objects rather than the top-k true normals, and the 'anomaly' set is the top-k predicted anomalies. Theorem 1 in §5.1.2 is proved for true O and I sets, so it does not justify Eq. 3. When k is misspecified or the score ranking is poor, the top-k pseudo-anomalies can be mostly normal, and maximizing the margin between this polluted set and the rest can select configurations that separate an arbitrary high-scoring cluster from the bulk—without improving true detection. Section 6.5.3 acknowledges that k assumes approximate knowledge of the anomaly ratio, which is itself a form of prior information that weakens the 'truly unsupervised' claim. The theorem's derivation is also heuristic: it argues for large µO−µI and small aδO+bδI, then equates that with maximizing (µO−µI)/√(δ²O+δ²I), which is not a formal equivalence. Empirically, Table 5 shows double-digit AUC losses relative to max for several algorithm–dataset pairs (e.g., ANEMONE on YelpChi −30.3%, SL-GAD on Facebook −39.4%, CoLA on YelpChi −30.2%), so 'close to best possible' is not uniformly supported. The central claim therefore rests on the unsupported assumption that T from Eq. 3 tracks AUC across configurations.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper identifies label leakage in SSL-based graph anomaly detection (GAD), where hyperparameters are often tuned using ground-truth labels, and proposes AutoGAD, a label-free internal evaluation strategy based on a modified Contrast Score Margin (CSM). The method selects SSL hyperparameters via grid search using Eq. (3), which measures the margin between the top-k predicted anomalous nodes and all remaining nodes, without using labels. The authors provide a theoretical analysis (Theorem 1) claiming that maximizing CSM is equivalent to minimizing false positive and false negative rates, and report extensive experiments across 10 SSL-based GAD algorithms and 10 datasets showing that CSM-selected hyperparameters beat the median and, in many cells, approach the maximum AUC. The paper also documents that many existing methods report overestimated performance due to label-based tuning.","tokens_in":33329,"tokens_out":5214,"duration_ms":44951,"significance":"The label-leakage documentation is a valuable and timely contribution: the paper gives concrete evidence (Table 1, Appendix B) that SSL-based GAD methods are highly sensitive to SSL hyperparameter choices and that reported performances often come from label-guided tuning. The proposed AutoGAD is a plausible and lightweight approach to label-free hyperparameter selection, and the experimental setup is extensive, including 10 algorithms, 10 datasets, multiple runs, sensitivity analyses, and released code. If the CSM criterion provably tracked AUC across hyperparameter configurations, this would be a practical advance for unsupervised graph anomaly detection. However, the theoretical support does not cover the actual pseudo-label-based metric, and the empirical claim of approximating the maximum AUC is not uniformly supported, so the central claim is not yet established.","major_comments":[{"comment":"Theorem 1 is proved for the true sets O and I of top-k anomalies and top-k normals, but the method selects hyperparameters with Eq. (3), in which the 'anomaly' set is the top-k pseudo-anomalies obtained from the detector's own scores and the 'normal' set is all remaining n−k objects. The Cantelli bounds in the theorem do not apply when O and I are replaced by these pseudo-labels; Section 5.1.3 itself notes that the pseudo-labels of top-k normals may be unreliable. Since the theorem is the stated theoretical justification for Eq. (3), this is a gap between the guarantee and the procedure actually used.","section":"§5.1.2–§5.1.3"},{"comment":"The proof argues that to keep both bounds small one wants µO−µI large and aδO+bδI small, and then asserts that this is 'equivalent' to maximizing T = (µO−µI)/sqrt(δO^2+δI^2). This is not a formal equivalence: a ratio can increase because the denominator shrinks, without the separate bounds improving, and the proof gives no argument that optimizing the single ratio is equivalent to simultaneously controlling the two inequalities. The theorem as stated is therefore not established, even for true O and I.","section":"§5.1.2, proof of Theorem 1"},{"comment":"The claim that CSM-selected hyperparameters 'approximately achieve the best possible performance' is not uniformly supported by the reported gains over maximal AUC. Several cells have double-digit losses: CoLA on YelpChi −30.2%, ANEMONE on YelpChi −30.3%, SL-GAD on Facebook −39.4%, GAAN on Facebook −53.7%, and CONAD on YelpChi −26.3%; the averages in the rightmost column are −3.8% to −14.1%. The paper should either qualify the claim, analyze the failure cases, or restrict it to cells where the loss is small.","section":"Table 5 and §6.5.2, observation 3"},{"comment":"The method requires the number of pseudo-anomalies k, and the paper states that it 'operated under the assumption that the anomaly ratio within a dataset is approximately known.' This is external prior information that weakens the 'truly unsupervised' claim, and it is load-bearing because with a wrong k or a poor score ranking the top-k set is polluted and maximizing Eq. (3) can separate an arbitrary high-scoring cluster from the bulk without improving detection. The sensitivity analysis in Figure 3 covers only CiteSeer, and it does not test the realistic case where the supplied k is far from the true ratio for the algorithms and datasets on which Table 5 shows large losses.","section":"§6.5.3, Sensitivity to k"},{"comment":"Because Eq. (3) is defined from the detector's own anomaly scores, maximizing it could reward score-separation artifacts rather than true detection. The paper provides cross-method evidence in Appendix F that CSM does not track AUC across heterogeneous detectors (Pearson correlations 0.070 on Cora and −0.488 on Amazon), but it does not report the analogous within-algorithm correlation between CSM and AUC over the HP grids used in Tables 3–5. This within-method criterion validity is the direct empirical support needed for the HPO claim; without it, the benchmark results in Tables 3–5 remain the only partial evidence.","section":"Appendix F and §5.1.3"}],"minor_comments":[{"comment":"The caption says 'PubWeb' but the dataset is PubMed; also, 'BlogCataLog' is spelled inconsistently throughout the text and tables.","section":"Table 2 caption"},{"comment":"Table 6 attributes CONAD to 'Zhang et al. (2022)', but the CONAD method is cited and described in the text as Xu et al. (2022b); this citation should be corrected.","section":"Table 6"},{"comment":"Equation (3) drops the 1/k factor inside the square root of Eq. (2) without comment; since k is a free parameter, the revised metric's scale and k-dependence change and this change should be discussed explicitly.","section":"§5.1.3, Eq. (3)"},{"comment":"The text says '8 out 10 algorithms' where '8 out of 10' is intended; also, the example citing SL-GAD as highly effective is odd because its average gain over max AUC is −7.5%, worse than several other methods.","section":"§6.5.2"},{"comment":"The sensitivity analysis for k is shown only on CiteSeer, while the surrounding text and the 'truly unsupervised' framing suggest broader support; the figure caption or text should clarify that Figure 3 alone does not establish stability across datasets.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be of interest to DMKD readers, and the label-leakage documentation is the strongest part. My main concern is that the headline claim is stronger than the evidence: Table 5 contains several large negative cells, and Theorem 1 does not justify Eq. (3). I would not reject, because the core idea is testable and the empirical setup is extensive; a revision that narrows the claim, strengthens the within-method criterion-validity analysis, and either removes or formalizes the k-assumption could make it publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. First, it documents a real problem: many SSL-based graph anomaly detection papers tune hyperparameters using labels and report the best results, which is label leakage. The sensitivity analysis (Table 1) quantifies how much this matters, and the authors deserve credit for putting that on the table with code and data. Second, their proposed fix—selecting hyperparameters with Contrast Score Margin—works better than a random pick in most cells but is not as reliable as the title suggests. The claim that AutoGAD gets close to best possible is undercut by double-digit AUC gaps on several algorithm–dataset pairs (e.g., ANEMONE on YelpChi −30.3%, SL-GAD on Facebook −39.4%).\n\nWhat is actually new is the application of CSM to SSL-based GAD and the empirical leakage study. CSM itself is not new (Xu et al. 2019), and internal evaluation strategies are surveyed in Ma et al. 2023. The modified Eq. 3 is a sensible stability tweak, and the experiments are extensive: 10 algorithms, 10 datasets, five runs.\n\nThe main weakness is the theory. Theorem 1 proves that maximizing the margin between true anomaly scores and true normal scores controls false positive and false negative rates. But AutoGAD maximizes Eq. 3 on pseudo-labels—the top-k predicted anomalies versus the rest. The theorem does not cover that setting, so the central claim that T(f) tracks AUC across configurations is not formally grounded. The proof also has a heuristic step: using Cantelli's inequality to show you want a large μO−μI and small aδO+bδI, then equating that with maximizing (μO−μI)/√(δ²O+δ²I) is not a rigorous equivalence. Additionally, Section 6.5.3 admits that k assumes approximate knowledge of the anomaly ratio, which is itself a mild prior and weakens the truly unsupervised framing.\n\nThat said, the paper is upfront about these limitations. Appendix F shows CSM does not work for selecting between heterogeneous detectors, which is honest. The missing comparison to DSV (Yoo et al. 2023) is a gap but not a fatal one. Overall, this is a useful empirical contribution with an over-claimed theoretical wrapper. It deserves serious peer review and, after revision, could be a solid reference for anyone doing unsupervised HP tuning in GAD.","headline":"A useful empirical study of label leakage in SSL-based graph anomaly detection, with a pragmatic internal evaluation strategy that mostly beats random hyperparameter choice—but the theory does not cover the actual pseudo-label procedure.","tokens_in":33872,"tokens_out":2528,"would_cite":true,"duration_ms":21632,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AutoGAD claims that a label-free score based on the Contrast Score Margin can select SSL hyperparameters for graph anomaly detection at nearly the performance of label-guided tuning.","keywords":["graph anomaly detection","self-supervised learning","hyperparameter tuning","label leakage","internal evaluation","contrast score margin","unsupervised model selection","graph neural networks"],"falsifier":"Take a dataset whose true anomalies are known, hide the labels, and compute the Spearman correlation between the modified Contrast Score Margin in Eq. 3 and true AUC over all grid configurations while varying $k$ from 0.25x to 4x the true anomaly count; if the correlation is near zero or negative for a reasonable value of $k$, the claim that maximizing $T$ approximates maximizing AUC fails on that setting.","tokens_in":32839,"feed_emoji":"🕸️","tokens_out":5557,"duration_ms":50616,"temperature":0.7,"pith_summary":"The paper argues that many recent self-supervised graph anomaly detection (GAD) methods are not truly unsupervised: their hyperparameters are chosen either arbitrarily or by looking at ground-truth labels, and the label-guided choices inflate reported AUC. It proposes AutoGAD, a plug-and-play procedure that selects SSL strategies' combination weights and hyperparameters using the Contrast Score Margin, an internal metric computed only from anomaly scores, with no labels. The central claim is that maximizing this margin is equivalent to simultaneously minimizing false positive and false negative rates, and that grid search over this score picks configurations at or near the best achievable AUC on most of the ten algorithms and ten datasets tested. If correct, practitioners can tune SSL-based GAD methods without label leakage, and several published performance numbers need to be re-read as optimistic upper bounds.","feed_headline":"Unsupervised score picks near-best anomaly detector settings","feed_subtitle":"AutoGAD's Contrast Score Margin selects SSL hyperparameters within a few points of the best possible AUC.","key_machinery":"The central object is the modified Contrast Score Margin, $T(f) = \\frac{\\hat{\\mu}_O - \\tilde{\\mu}_I}{\\sqrt{\\hat{\\delta}^2_O + \\tilde{\\delta}^2_I}}$, where $\\hat{\\mu}_O, \\hat{\\delta}^2_O$ are the mean and variance of the anomaly scores of the $k$ predicted anomalous nodes and $\\tilde{\\mu}_I, \\tilde{\\delta}^2_I$ are the mean and variance of the remaining $n-k$ nodes. The statistic measures how far the top-$k$ pseudo-anomalies stand out from the rest, normalized by score spread, and runs in linear time on the score vector. Grid search over a discretized hyperparameter space maximizes $T$ to choose the configuration. The theoretical support uses Cantelli's inequality to bound the probability that a true anomaly falls below its group mean minus a multiple of its standard deviation, and the probability that a normal node rises above its group mean plus a multiple; making the gap between the two means large relative to the standard deviations is therefore the same as shrinking both false-negative and false-positive bounds. The theorem is stated for true anomaly sets, whereas in practice the sets are replaced by pseudo-labels derived from top-$k$ scores.","core_discovery":"For a fixed SSL-based graph anomaly detection algorithm, the modified Contrast Score Margin $T(f) = \\frac{\\hat{\\mu}_O - \\tilde{\\mu}_I}{\\sqrt{\\hat{\\delta}^2_O + \\tilde{\\delta}^2_I}}$ ranks hyperparameter configurations almost as well as true AUC when the top-$k$ pseudo-anomalies are the $k$ nodes with the highest anomaly scores and the remaining $n-k$ nodes form the pseudo-normal group. The paper claims in Theorem 1, via Cantelli's inequality, that maximizing this margin simultaneously minimizes false positives and false negatives, and it shows empirically that selecting the configuration with the largest $T$ yields AUC close to the maximum over a large grid for most algorithm-dataset pairs. A correctness boundary stated by the paper is that this score should select among hyperparameter settings of the same algorithm with comparable loss scales, not among heterogeneous detectors; the appendix shows it fails at the latter.","pith_inferences":["If the anomaly ratio is not approximately known, the effectiveness of AutoGAD could degrade sharply, because $k$ sets the pseudo-label boundary; a testable extension would feed a label-free estimate of $k$ (e.g., from score gaps or a silhouette-style criterion) into the margin and check whether selection quality is preserved.","The same margin-based reasoning might transfer to unsupervised model selection beyond graphs, such as tabular or time-series anomaly detectors, whenever scores from one detector family are comparable; the paper does not claim this extension.","The reported sensitivity results imply that even moderate hyperparameter variation can change AUC by 15 to 30 percent for several methods, so an ensemble over several high-margin configurations may be a more robust practical recipe than picking a single best configuration, though the paper only reports single-configuration selection."],"forward_implications":["AutoGAD gives a label-free procedure for choosing augmentation functions, hyperparameters, and combination weights for any SSL-based GAD algorithm that outputs anomaly scores, removing the need for a labeled validation set.","Because the selection criterion is computed entirely from scores, the same detector can be adapted per dataset through grid search instead of relying on fixed heuristics that were tuned with labels.","Published results of SSL-based GAD baselines that selected hyperparameters using ground-truth labels should be regarded as upper bounds; a truly unsupervised deployment is likely to land lower, sometimes substantially.","The Contrast Score Margin is not a universal model selector: the authors show it fails to rank heterogeneous detection algorithms, so it should be used only within one algorithm's hyperparameter space with comparable loss scales."],"supporting_citations":[{"why":"Introduces the Contrast Score Margin that AutoGAD modifies and adopts for graph anomaly detection.","marker":"(Xu et al, 2019)"},{"why":"Supplies ANEMONE, a representative contrastive SSL-GAD baseline whose label-guided hyperparameter tuning the paper revisits.","marker":"(Jin et al, 2021a)"},{"why":"Provides a benchmark showing similar gaps between average and maximum AUC across hyperparameter settings, supporting the sensitivity findings.","marker":"(Liu et al, 2022b)"},{"why":"Documents hyperparameter sensitivity in unsupervised outlier detection and motivates internal evaluation strategies for model selection.","marker":"(Zhao et al, 2021b)"},{"why":"Reviews and evaluates internal evaluation strategies, positioning the Contrast Score Margin among candidate unsupervised model-selection criteria.","marker":"(Ma et al, 2023)"},{"why":"Supplies AnomalyDAE, a generative baseline whose label-selected hyperparameters are tested in the AutoGAD experiments.","marker":"(Fan et al, 2020)"},{"why":"Supplies SL-GAD, a combined contrastive and generative method with label-fine-tuned hyperparameters, used to demonstrate large performance variation.","marker":"(Zheng et al, 2021)"}],"fun_headline_variants":["No-label score chooses top hyperparameters for graph anomaly detectors","Self-supervised score ranks settings near true AUC without label leakage","Internal margin picks near-best model config in unsupervised graph AD","Auto-tune SSL detectors without peeking at labels","Contrast score selects hyperparameters close to optimal AUC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes the anomaly ratio is roughly known so that $k$, the number of nodes treated as pseudo-anomalies in the score, is near the true number of anomalies; if $k$ is wrong, the top-$k$ set contains normal nodes and the margin being maximized no longer tracks true detection quality.","fun_headline_variants_meta":{"raw":{"variants":["No-label score chooses top hyperparameters for graph anomaly detectors","Self-supervised score ranks settings near true AUC without label leakage","Internal margin picks near-best model config in unsupervised graph AD","Auto-tune SSL detectors without peeking at labels","Contrast score selects hyperparameters close to optimal AUC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000201,"raw_usage":{"total_tokens":1388,"prompt_tokens":965,"completion_tokens":423,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":343}},"tokens_in":581,"tokens_out":423,"duration_ms":4234,"temperature":1.0,"reasoning_tokens":343,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:53:39.501720+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a dataset whose true anomalies are known, hide the labels, and compute the Spearman correlation between the modified Contrast Score Margin in Eq. 3 and true AUC over all grid configurations while varying $k$ from 0.25x to 4x the true anomaly count; if the correlation is near zero or negative for a reasonable value of $k$, the claim that maximizing $T$ approximates maximizing AUC fails on that setting.","supporting_citations":[],"review_version":1}