{"id":"0f4a64f6-6a9f-454b-ae6c-3c7d9dcaf9cc","arxiv_id":"2412.07326","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Class-conditional anomaly detection and SHAP-based consistency checks provide a new way to score the quality of tabular adversarial samples, and a seven-attack comparison shows transferability-based attacks are the hardest to detect.","lead":"This paper builds a framework for measuring the quality of adversarial attacks on table-based machine learning models, adding class-specific anomaly detection and checks on how the model's reasoning changes, on top of standard success-rate metrics. The framework's comparison of seven attack strategies shows query-based attacks are easier to spot, while transferability-based attacks often slip past standard detectors.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SHAP-based anomaly metrics are never calibrated on benign samples; with up to 115 features the min/max 'normal range' rule can flag ordinary samples, making CSAD detection rates uninterpretable. Section 5.5's 'consistently outperforms' claim also contradicts Table 6.","rationale":"The paper's central claim is that CSAD and the dual SHAP/feature-space criteria reliably measure adversarial sample quality. The load-bearing condition is that the SHAP-based anomaly flag has a low and measured false-positive rate on benign samples. That condition is absent and likely violated: the rule is a per-feature training min/max interval with no correction for multiple testing. This is an internal calibration gap, not a disagreement with the field's consensus. The Section 5.5 universal wording is too strong given the negative rows in Table 6. The concern is fixable: report benign FPR, calibrate thresholds, and correct the summary claim. The paper has genuine independent support: feature-space FPRs are reported, Appendix C includes extensive Holm-corrected pairwise tests, and code is provided. The reader's weakest assumption is the same underlying issue, so I agree with the Reader. The appropriate verdict remains CONDITIONAL; a revision should add the benign-sample SHAP calibration and an honest per-row summary before the central quality-measurement claims are accepted.","tokens_in":48646,"tokens_out":5925,"duration_ms":58866,"concrete_test":"Using the released GitHub code, compute the Section 3.3 SHAP-based anomaly flag on benign test samples (or a held-out benign split) for every dataset, target model, and predicted class, using exactly the training min/max ranges and the 'at least one feature outside' rule; report per-class FPR. If the FPR is materially above the feature-space FPRs in Section 5.4 (e.g., more than 10-20% on the 115-feature Hate dataset), the detection rates in Tables 5-6 are not valid evidence for CSAD or for the attack-quality conclusions, and the thresholds would need recalibration or multiple-testing correction. A secondary check is to tabulate the sign of CSAD-minus-standard for every row in Table 6 to test the 'consistently outperforms' wording.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 defines the normal SHAP range for class c, feature f as [min(SHAP_f,c), max(SHAP_f,c)] over benign training samples, and flags any sample with at least one feature outside that range. This is an out-of-sample min/max envelope, not a calibrated quantile or density rule; no false-positive rate for the SHAP-based metric is reported anywhere, while Section 5.4 reports FPR only for the AE/IF feature-space detectors. For a dataset with 115 features (Hate), a benign test sample can easily fall outside a training min/max interval for at least one feature purely by sampling variation, especially with class-conditional SHAP distributions. If that FPR is not tiny, the anomaly rates in Tables 5 and 6 mostly measure an uncontrolled flag rather than adversarial incoherence. The Section 5.5 statement that CSAD 'consistently and significantly outperforms the standard approach across all datasets' is also internally contradicted by Table 6: VideoTQ/RF random 13.8% vs 0.0%, VideoTQ/RF GB imp 17.0% vs 0.0%, VideoTQ/LGB GB imp 10.6% vs 3.4%, VideoTQ/XGB boundary 47.0% vs 38.9%, and Hate/GB LGB imp 100.0% vs 81.2%. This does not disprove the method, but it means the reported 'superiority' and attack-quality rankings rest on uncalibrated thresholds and an overstated consistency claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a methodological framework for adversarial attacks and defenses on tabular data, with three main contributions: (i) a regression-based technique for perturbing dependent features while preserving sample coherence; (ii) a class-specific anomaly detection approach (CSAD) applied to both feature-space anomaly detectors (autoencoder and isolation forest) and SHAP-based model-interpretation metrics; and (iii) a large comparative empirical study of two query-based attacks and five transferability-based attacks across four tree-based models on three datasets. The evaluation assesses attacker risk, effort, and adversarial sample quality via anomaly detection rates and SHAP-based stability metrics. The main reported findings are that query-based attacks achieve near-perfect success but produce large, detectable distortions and require many queries, whereas transferability-based attacks are less successful but produce more coherent, harder-to-detect samples, and that CSAD substantially outperforms a global threshold approach for detecting adversarial samples.","tokens_in":48794,"tokens_out":8951,"duration_ms":83132,"significance":"If the proposed framework is valid, it would fill a real gap in the tabular adversarial robustness literature by providing objective, scalable metrics for adversarial sample coherence and by systematically comparing attack strategies. The paper's methodological ambition is substantial, and the authors make their artifacts publicly available, which supports reproducibility. The feature-space anomaly detection evaluation is carefully calibrated (FPR reported for AE/IF), and the comparative attack study is broad. However, the central claim that CSAD and the SHAP-based dual criteria reliably measure adversarial sample quality is weakened by an uncalibrated SHAP threshold rule and by an internally inconsistent claim of uniform superiority. The significance of the work thus depends on the extent to which these issues can be resolved in revision.","major_comments":[{"comment":"The definition of the normal SHAP range as [min(SHAP_f,c), max(SHAP_f,c)] over benign training samples makes the Importance-Based Anomaly Detection Rate an uncalibrated out-of-sample min/max envelope rule. For the Hate dataset with 115 features, a benign test sample will very likely fall outside at least one training min/max interval, yet the paper reports no false-positive rate for any of the SHAP-based metrics; Section 5.4 reports FPR only for the AE/IF feature-space detectors. As a result, the absolute anomaly rates in Table 5 and the CSAD-vs-standard comparison in Table 6 are uninterpretable: the higher CSAD rates may simply reflect a higher uncalibrated FPR of the per-class reference rather than a genuine detection advantage. The manuscript should either calibrate the SHAP criterion (e.g., per-feature quantiles, with FPR measured on a benign validation split) or explicitly report and adjust for the FPR.","section":"Section 3.3"},{"comment":"Section 5.5 states that 'CSAD consistently and significantly outperforms the standard approach across all datasets (p<0.001)' with 'uniformly large Cohen's g=1.0.' This is contradicted by Table 6, where the standard approach yields higher detection rates than CSAD in multiple cells: VideoTQ RF random 13.8% vs 0.0%, VideoTQ RF GB imp. 17.0% vs 0.0%, VideoTQ LGB GB imp. 10.6% vs 3.4%, VideoTQ XGB boundary 47.0% vs 38.9%, and Hate GB LGB imp. 100.0% vs 81.2%. Moreover, a Cohen's g of 1.0 for a paired McNemar test is impossible when the marginal rates differ in both directions; Appendix C itself lists non-significant random-transferability cases (e.g., Table 16, Hate random: p=1.00). The claim of uniform superiority must be removed or replaced with an analysis that accounts for significance per attack–model combination and for the direction of each difference.","section":"Section 5.5 / Table 6"},{"comment":"The reported CSAD advantage over the standard approach is partly constructed by the experimental setup. In a binary problem, a successful adversarial sample sits near the decision boundary while retaining traits of its true class; comparing it only with benign samples of the predicted class will almost mechanically produce a higher anomaly rate than comparing it with the pooled benign distribution. The paper does not control for this by reporting the SHAP-based false-positive rates for benign samples under the class-specific versus global references. Without such a control, the conclusion that class-specific evaluation is 'essential' (Section 5.5) is not supported by the data; the observed gap may be an artifact of differing reference-set sizes and spreads. The authors should test whether CSAD improves the ROC or precision-recall trade-off at matched FPR values, rather than comparing raw flag rates.","section":"Section 3.3 / Section 5.5"},{"comment":"The regression-based technique for dependent features (Section 3.2) is a central claimed contribution, but its effectiveness is never directly evaluated. It is applied only to the ICU dataset (Section 4.5), and the paper does not compare attack outcomes, anomaly detection rates, or coherence metrics with and without the regression correction. The claim that this technique 'ensures' coherence and 'maintains feature consistency' (Section 3.2) is therefore unsupported by the experiments. An ablation study, at least on the ICU dataset, is needed to substantiate this component of the framework.","section":"Section 3.2 / Section 4.5"}],"minor_comments":[{"comment":"There are typos in the text: 'extened' should be 'extended', and 'to to' should be 'to'.","section":"Section 3.3"},{"comment":"The threshold in Eq. (1) uses '2 * std' but the text later refers to 'k=2' without defining a general k; the notation should be made consistent.","section":"Equation (1) / Section 4.2"},{"comment":"The Hate preprocessing description states that 'top-40 most important features' were selected 'based on the Pearson correlation coefficient' but does not specify the target variable or whether the correlation is with the class label; please clarify.","section":"Section 4.2"},{"comment":"The definitions of alpha, beta, and t in the time equations are confusing, and the numeric result '~243.5673 seconds' contains a stray tilde and ambiguous units; the presentation should be cleaned up.","section":"Section 5.3 / Equations (4)-(5)"},{"comment":"Some effect-size entries are reported as 'L' even when the accompanying p-value is not significant (e.g., Table 16, Hate random: p=1.00, effect size L); the reporting convention for effect size on non-significant tests should be explained or revised.","section":"Appendix C"},{"comment":"The caption states that the better detection rate is highlighted in bold, but the table as displayed does not show bold formatting; please ensure the formatting is visible in the final version.","section":"Table 6"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concerns land on two load-bearing issues: the SHAP-based metric is uncalibrated, and the Section 5.5 claim of uniform CSAD superiority is internally contradicted by Table 6. These are not mere presentation problems; they affect the validity of the central contribution. That said, the paper is not beyond repair: a careful revision that calibrates the SHAP criterion, reports FPRs for the SHAP metrics, corrects or reanalyzes the CSAD comparison, and adds an ablation for the regression-based perturbation technique would substantially improve the manuscript. The scope is appropriate for the journal, and the authors' decision to release code and data is a positive sign."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it is a serious, well-organized empirical contribution to tabular adversarial ML: CSAD (class-conditional anomaly scoring) and the SHAP-based importance-stability metrics are genuinely new relative to the cited prior work in this niche, and the seven-attack comparison across four tree models is the most thorough I have seen in this subfield. The main qualitative result—query-based attacks achieve near-perfect success but produce large detectable distortions and up to ~120k queries, while transferability attacks are subtler but less successful—is supported by the tables and the statistical tests. I would trust that descriptive finding.\n\nThe soft spots are real but not fatal. The biggest one: the SHAP-based anomaly metric is never calibrated on benign samples. Section 3.3 defines the normal range as [min, max] over benign training SHAP values per class and flags any sample with at least one feature outside that range. For the Hate dataset with 115 features, the chance of a benign sample tripping at least one feature is not negligible, and the paper never reports a benign false-positive rate for this metric. The FPR numbers in Section 5.4 are for the AE/IF feature-space detectors, not for the SHAP envelope. Until that calibration is done, the SHAP-based detection rates in Tables 5 and 6 are uninterpretable.\n\nThe second issue is the Section 5.5 claim that CSAD 'consistently and significantly outperforms the standard approach across all datasets' with Cohen's g = 1.0. The paper's own Table 6 contains multiple rows where CSAD is worse, e.g., VideoTQ/RF random 0.0% vs 13.8%, VideoTQ/RF GB imp 0.0% vs 17.0%, and several others. The McNemar tests may show an overall advantage, but the sentence as written is false and should be qualified to say 'in the majority of attack–model pairs' or similar.\n\nA minor point: the regression-based perturbation technique, listed as a primary contribution, is applied to only four ICU features, with no with/without ablation and no report of regression fit quality. It is a reasonable idea, but the evidence for it is thin. I also agree with the reader's circularity concern: in a binary problem, successful adversarial samples sit near the boundary while retaining true-class traits, so evaluating them against the predicted class distribution partly guarantees higher anomaly rates. That does not invalidate CSAD as an evaluation tool, but the paper should acknowledge the construction more directly and compare against a calibrated baseline.\n\nWho should read this: researchers working on tabular adversarial attacks and defenses, especially those who want a reusable evaluation recipe. It deserves a serious referee, not a desk reject, but it needs major revision: calibrate the SHAP metric on benign data, fix the overclaim about consistency, and either ablate the regression perturbation or stop calling it a headline contribution.","headline":"A solid empirical framework for tabular adversarial evaluation, but the SHAP anomaly metric is uncalibrated and the headline CSAD claim overstates what Table 6 actually shows.","tokens_in":49552,"tokens_out":1840,"would_cite":false,"duration_ms":20416,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that adversarial sample quality in tabular data should be judged by class-specific anomaly detection and SHAP-based interpretation stability, and that this framework exposes transferability-based attacks as far harder to…","keywords":["tabular data","adversarial attacks","adversarial defenses","class-specific anomaly detection","SHAP","black-box attacks","transferability attacks","feature coherence"],"falsifier":"Run the paper's CSAD SHAP anomaly test on benign test samples and record the fraction with at least one out-of-range feature-importance value; if that benign false-positive rate is comparable to the adversarial detection rates reported in Table 6, the claimed CSAD advantage would be an artifact of threshold choice rather than a detection improvement.","tokens_in":48229,"feed_emoji":"🛡️","tokens_out":5773,"duration_ms":59851,"temperature":0.7,"pith_summary":"Tabular machine-learning models face a distinct adversarial problem: a perturbed record must not only fool the model but also keep its feature relationships coherent, and standard success and perturbation metrics do not measure that. This paper proposes a framework with three parts: a regression-based technique for adjusting dependent features during an attack, a class-specific anomaly detection approach called CSAD that scores samples against benign data of their predicted class, and SHAP-based evaluation of whether a sample changes the model's internal feature-importance reasoning. The paper evaluates two query-based and five transferability-based attacks on three tabular datasets with four tree models. Its central empirical claim is that query-based attacks succeed near-perfectly but produce large, detectable distortions, whereas transferability-based attacks have lower success but are often nearly undetectable by both feature-space and SHAP-based criteria. If this is right, defenders should monitor interpretation stability as well as input coherency, and should not treat query-based attacks alone as representative of real-world risk.","feed_headline":"Class-aware detector catches subtle tabular attacks","feed_subtitle":"New framework scores adversarial samples by predicted class, showing transfer attacks stay near-invisible.","key_machinery":"The load-bearing object is Class-Specific Anomaly Detection (CSAD), defined as training one anomaly detector per class and scoring an adversarial sample against the detector of its predicted class, rather than against a global benign distribution. CSAD is applied to two evaluation criteria: Feature Space Coherency, measured by isolation forest and autoencoder anomaly rates, and Model Interpretation Stability, measured by the class-specific range of each feature's SHAP values over benign training samples, flagging any sample whose SHAP value for at least one feature falls outside that range. A second mechanism is the regression-based dependent-feature perturbation, which trains a regression model per dependent feature and uses it to correct the feature after each attack step so the crafted sample stays coherent with the rest of the record. These mechanisms carry the argument because they turn 'coherence' into an operational quantity.","core_discovery":"On the paper's own terms, the discovery is that adversarial sample quality in the tabular domain is better measured by class-conditional anomaly assessment than by global anomaly thresholds or by perturbation magnitude. The central instrument is CSAD, which trains a separate detector for each predicted class and evaluates each adversarial sample only against the benign distribution of the class the model assigns to it; combined with SHAP importance ranges, this consistently identifies adversarial samples that a global detector misses, with the paper reporting p < 0.001 and effect sizes near Cohen's g = 1.0 for the CSAD advantage. The paper further reports that small L2 distance is not a reliable sign of invisibility, since some low-distortion transferability-based samples are still flagged as anomalous, and that feature-space and SHAP-based anomaly rates respond independently to different attacks. The authors would summarize their result as: class-specific evaluation plus interpretation stability gives a concrete, scalable way to quantify whether an adversarial tabular sample would actually evade detection.","pith_inferences":["Beyond the paper: the class-conditional principle of CSAD could extend to non-tabular modalities, but its benefit will depend on whether per-class benign subsamples are large enough to estimate reliable reference distributions.","Beyond the paper: before CSAD is used as a defense, its benign false-positive rate should be measured on the exact task; the per-feature min/max range on a 115-feature dataset would likely need percentile-based calibration.","Beyond the paper: the reported invisibility of successful transfer attacks suggests a defense that monitors interpretation stability rather than raw feature values alone, since input-space detectors miss these samples.","Beyond the paper: the regression-based dependent-feature correction was tested only on the ICU dataset; a direct test on datasets with known nonlinear or mutually dependent features would show how far the coherence guarantee generalizes."],"forward_implications":["Query-based attacks (boundary and HopSkipJump) achieve near-perfect success but at the cost of many changed features, large L2 distortions, and query counts up to roughly 120,000, making them expensive and detectable.","Transferability-based attacks need only one query but succeed less often; the samples that do transfer are mostly invisible to both autoencoder and isolation-forest detectors and usually leave SHAP importance stable.","Feature-importance-based feature selection in transfer attacks raises transfer success and keeps perturbations smaller than random selection does.","Under the paper's CSAD metric, class-specific anomaly scoring detects adversarial samples at rates several times higher than global scoring across datasets, including cases where global scoring finds none.","Small L2 distance does not by itself mean a tabular sample is undetectable, so perturbation norms should not be the sole quality criterion."],"supporting_citations":[{"why":"Supplies the tabular-validity adaptation of decision-based attacks that the query-based attacks build on.","marker":"[7]"},{"why":"Provides the surrogate model architecture, adversarial objective, and preprocessing pipeline used for transferability-based attacks.","marker":"[8]"},{"why":"Provides the feature-importance-based feature selection strategy and the Hate dataset preprocessing used in the evaluation.","marker":"[9]"},{"why":"Defines the SHAP framework that the Model Interpretation Stability metrics are built on.","marker":"[21]"},{"why":"Supplies the Hateful Users on Twitter dataset used as one of the three empirical benchmarks.","marker":"[23]"},{"why":"Supplies the Intensive Care Unit dataset used as the second empirical benchmark and the setting for regression-dependent feature perturbation.","marker":"[24]"},{"why":"Defines the boundary attack that the paper adapts as one of the query-based attacks.","marker":"[25]"},{"why":"Defines the HopSkipJump attack that the paper adapts as the second query-based attack.","marker":"[26]"},{"why":"Provides the autoencoder-based anomaly detection approach used as one of the feature-space coherency detectors.","marker":"[42]"},{"why":"Provides the isolation forest algorithm used as the other feature-space coherency detector.","marker":"[45]"}],"fun_headline_variants":["Class-specific detector catches tabular attacks others miss","CSAD scores adversarial samples per class, not globally","Transfer attacks stay near-invisible under class-aware metric","Small perturbation doesn't mean safe: class-aware detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that benign samples rarely produce a feature-importance value outside the per-class range of benign training samples, so an out-of-range value can be treated as a sign of attack.","fun_headline_variants_meta":{"raw":{"variants":["Class-specific detector catches tabular attacks others miss","CSAD scores adversarial samples per class, not globally","Transfer attacks stay near-invisible under class-aware metric","Small perturbation doesn't mean safe: class-aware detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000196,"raw_usage":{"total_tokens":1405,"prompt_tokens":1031,"completion_tokens":374,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":647,"completion_tokens_details":{"reasoning_tokens":312}},"tokens_in":647,"tokens_out":374,"duration_ms":4563,"temperature":1.0,"reasoning_tokens":312,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:57:52.880988+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's CSAD SHAP anomaly test on benign test samples and record the fraction with at least one out-of-range feature-importance value; if that benign false-positive rate is comparable to the adversarial detection rates reported in Table 6, the claimed CSAD advantage would be an artifact of threshold choice rather than a detection improvement.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the surrogate model architecture, adversarial objective, and preprocessing pipeline used for transferability-based attacks."},{"cited_title":"Grolman, H","cited_arxiv_id":null,"evidence_quote":"Provides the feature-importance-based feature selection strategy and the Hate dataset preprocessing used in the evaluation."},{"cited_title":"Ribeiro, P","cited_arxiv_id":null,"evidence_quote":"Supplies the Hateful Users on Twitter dataset used as one of the three empirical benchmarks."},{"cited_title":"Hanberger, D","cited_arxiv_id":null,"evidence_quote":"Supplies the Intensive Care Unit dataset used as the second empirical benchmark and the setting for regression-dependent feature perturbation."},{"cited_title":"1277–1294","cited_arxiv_id":null,"evidence_quote":"Defines the HopSkipJump attack that the paper adapts as the second query-based attack."},{"cited_title":"Sakurada, T","cited_arxiv_id":null,"evidence_quote":"Provides the autoencoder-based anomaly detection approach used as one of the feature-space coherency detectors."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the isolation forest algorithm used as the other feature-space coherency detector."}],"review_version":1}