{"id":"a6c822b0-3666-431f-ac38-c74119495050","arxiv_id":"2501.05588","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A random-shuffle attack that preserves one-dimensional feature distributions while destroying correlations can fool classifiers and, used as data augmentation, occasionally beats standard tabular generators on AUROC.","lead":"This paper introduces RDSA, an adversarial attack that shuffles feature values so that each variable keeps its own distribution but the correlations between variables are broken. The authors use these shuffled examples to retrain classifiers on six tasks, reporting improved accuracy on some and mixed or negative results on others.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Augmentation gains are not shown to arise from correlation focusing; a random-shuffle control is needed before the central claim can be accepted.","rationale":"The reader's weakest assumption identifies the same underlying premise: that retraining on correlation-broken adversaries forces the network to focus on correlations and generalize better. My stress-test sharpens this into a concrete confound: the RDSA augmentation pipeline combines two operations, marginal-preserving shuffling and adversarial selection, and the paper does not isolate which operation produces the reported AUROC improvements. The reader also noted the empirical contradiction on VBF and Rain in Australia; in my view that contradiction is evidence for the confound rather than a separate limitation. Because the paper's central attack claim (high fooling ratios with preserved one-dimensional distributions) is plausible and supported by the reported metrics, and because the concern is about the interpretation of the augmentation results rather than the attack's validity, the conditional verdict remains appropriate. I recommend no change to the reader's verdict, but the authors should add the proposed control and soften the mechanistic language in Sections 6 and 8 if the control does not favor RDSA.","tokens_in":20434,"tokens_out":2887,"duration_ms":32615,"concrete_test":"Run the Section 6 augmentation pipeline with a Random Shuffle Augmentation (RSA) control: shuffle the same number of features per sample using the same histograms and the same maximum number of attempts, but accept the shuffled sample regardless of the model's prediction. Retrain with correct labels and compare AUROC against RDSA, CTGAN, TVAE, and the data-starved baseline on TopoDNN, MNIST784, HAR, VBF, and Rain in Australia. If RSA matches or exceeds RDSA, the adversarial selection is unnecessary and the correlation-focusing mechanism is unsupported; if RDSA clearly outperforms RSA, the mechanism is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the hypothesis, stated in Sections 1 and 6, that retraining on RDSA-generated adversaries forces the network to focus on correlations and thereby improves generalization. The reported AUROC gains on four of six datasets are consistent with this story, but they are also consistent with a much weaker mechanism: RDSA resamples each selected feature independently from the global one-dimensional histogram (Section 3, Appendix D), so it injects label-preserving noise that destroys individual-level joint structure. Such noise can act as a generic regularizer regardless of whether the network 'focuses on correlations.' The paper provides no control that breaks correlations in the same way but without conditioning on model misclassification, so the adversarial-selection step is confounded with the marginal-preserving shuffling step. The fact that VBF and Rain in Australia degrade under RDSA augmentation (Figure 8) further suggests the benefit is dataset-dependent and not tied to the proposed mechanism. This is an overclaim rather than an internal inconsistency, but it is load-bearing because the abstract and conclusions attribute the empirical improvements to the correlation-focusing effect.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces Random Distribution Shuffle Attack (RDSA), an adversarial attack that resamples input features from their one-dimensional histograms to break inter-feature correlations while preserving marginal distributions. The authors evaluate RDSA on six classification tasks (VBF, TopoDNN, Rain in Australia, MIMIC-IV, MNIST784, HAR), reporting high fooling ratios with small Jensen-Shannon distances and substantial correlation-matrix changes. They further apply RDSA-generated adversaries as a data-augmentation strategy under data-starved training regimes, comparing against CTGAN, TVAE, and LowProFool (LPF). The paper claims that adversarial training with RDSA improves classifier performance and robustness, with the proposed mechanism being that retraining on correlation-broken adversaries forces networks to focus on correlations. The reported experiments show AUROC gains over the data-starved baseline on TopoDNN, MIMIC-IV, MNIST784, and HAR, while VBF and Rain in Australia show mostly negative effects.","tokens_in":20640,"tokens_out":4430,"duration_ms":44010,"significance":"The attack itself is a simple and interesting contribution: it directly targets the correlation structure of tabular data while preserving marginals, and the paper demonstrates this behavior quantitatively across diverse domains. The evaluation is broad, including six datasets and comparisons with three established augmentation baselines, with uncertainty estimated over repeated runs. The pseudo-code and the detailed appendices facilitate replication, provided the issues noted below are fixed. If the central mechanism claim were established, the work would be a useful addition to adversarial robustness and tabular data augmentation. However, the load-bearing hypothesis that augmentation gains arise from 'focusing on correlations' is not directly tested, and the negative results on two datasets limit the generality of the conclusions as currently stated.","major_comments":[{"comment":"The central claim that retraining on RDSA adversaries forces the model to 'focus on correlations' and thereby improves clean-data generalization is not supported by the experimental design. Section 1 states this hypothesis, and Section 6 invokes it to interpret the MIMIC-IV results, but the experiments do not isolate the correlation-focusing mechanism. RDSA combines two operations: marginal-preserving independent resampling of features and selection of samples that fool the model. A control that breaks correlations in the same way but without conditioning on model misclassification (e.g., random shuffles of the same variables, or shuffles drawn without the adversarial query) is missing. Without such a control, the observed AUROC improvements on four datasets are equally consistent with generic label-preserving noise acting as a regularizer. Moreover, the hypothesis predicts benefits whenever correlations are informative, yet for VBF and Rain in Australia all augmentation methods including RDSA decrease AUROC below the data-starved baseline (Figure 8c,d). The paper should either provide a direct test of the correlation-focusing mechanism or substantially weaken the causal language in the abstract and conclusions.","section":"Sections 1 and 6; Figure 8"},{"comment":"The provided pseudo-code for RDSA does not match the textual description and is not runnable as written. The function call `random.sample(range(0, len(input), nVars))` is missing the required second argument `k`; moreover, using `nVars` as the step of `range` selects indices 0, nVars, 2*nVars, ..., rather than a random subset of `nVars` variables as described in Section 3 (the intended call is presumably `random.sample(range(len(input)), nVars)`). In addition, the pseudo-code's inner loop resamples every selected variable at every one of the `max_steps` attempts, while the text in Section 3 suggests that after each full shuffling attempt the model is queried; the code as written would re-shuffle all variables repeatedly before a single prediction, which is inconsistent with the description and with the reported 'up to 100 shuffle attempts'. The handling of failed examples (when the function returns `None`) is also unspecified, which affects the fooling-ratio calculation. The method definition should be corrected and clarified.","section":"Appendix D, pseudo-code; Section 3"},{"comment":"The conclusions overstate the generality of the augmentation results. The abstract claims that 'correct application of the proposed novel attack can result in a significant improvement in classification performance', and Section 8 says retraining 'showed the capability to significantly improve their performance and robustness, often outperforming standard data augmentation methods'. This is contradicted by the Rain in Australia and VBF results, where Figure 8 shows that all RDSA configurations (and indeed all augmentation methods tested) perform below the base model within uncertainties. The paper should revise these statements to acknowledge that the benefit is dataset-dependent, or provide an analysis of why the mechanism fails on these datasets. As written, the conclusion is not supported by the full set of reported results.","section":"Abstract and Section 8 (Conclusions) versus Figure 8"}],"minor_comments":[{"comment":"The sentence 'Given that correlations between input features are also crucial in many other disciplines.' is an incomplete sentence fragment and should be integrated into the surrounding text.","section":"Abstract"},{"comment":"The correlation-difference metric is defined as ⟨cf⟩? Actually, the average correlation difference is defined with the equation after '⟨cc⟩ :=' in Section 5, but the text and Figure 5 caption refer to it as ⟨s⟩. This notation inconsistency should be fixed.","section":"Section 5, equations for metrics"},{"comment":"Section 6 states the process is repeated 100 times, but the pipeline description in the same section says 'using a fixed random seed of 42 to ensure consistent results across 50 iterations.' The number of iterations should be unified.","section":"Section 6 and 7 (data augmentation pipeline)"},{"comment":"The data and code availability statement is a placeholder ('Currently not in here, as it is not anonymous. We will put the links back in here once the anonymous phase is over.'). For reproducibility, the final version should include the actual links or a statement of availability.","section":"Appendix A"},{"comment":"There is a typo: 'continouous' should be 'continuous'. Similar minor typos appear elsewhere (e.g., 'actuall' in Section 1).","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The reader's conditional assessment aligns with my own. The strongest issue is the missing control for the mechanism, not the attack itself. The pseudo-code bug is concrete and should be fixed before publication. The paper might benefit from positioning RDSA primarily as a correlation-targeting attack with potential augmentation benefits on select benchmarks, rather than as a generally superior augmentation method."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core attack result is real, and the paper deserves a serious look. RDSA shuffles feature values according to their marginal histograms, and the reported metrics support the claim that it fools networks while keeping one-dimensional distributions nearly intact. The JSD and correlation-difference plots make that case cleanly, and running it across six datasets with comparisons to CTGAN, TVAE, and LPF is solid empirical work.\n\nWhat is actually new is the framing of marginal-preserving shuffling as an adversarial attack and as a data-augmentation method. The operation itself is textbook permutation, but the application and evaluation are legitimate extensions, and the paper does not oversell the novelty beyond that.\n\nThe soft spots are real but manageable. The load-bearing hypothesis that retraining on RDSA adversaries forces the network to focus on correlations is not tested. A control that shuffles without conditioning on model misclassification would separate the adversarial-selection mechanism from generic label-preserving noise. Without it, the AUROC gains on four of six datasets are just as consistent with ordinary regularization, and the two negative results (VBF, Rain in Australia) weaken the universal claim. The abstract and conclusions overstate the case, even though the limitations section does acknowledge the negative results.\n\nThere are also smaller issues: no code or data is released, and the appendix has an explicit placeholder saying the links will be added after the anonymous phase. The RDSA pseudo-code contains a bug—the get_vars_shuffle function calls random.sample with a range stepped by nVars instead of sampling nVars indices, so it would not do what the text describes. The augmentation pipeline says the process is repeated 100 times but the text mentions 50 iterations with a fixed seed; that inconsistency needs fixing.\n\nWho is this for? High-energy physicists doing classification with tabular inputs will find the attack useful as a correlation-based robustness check, and the augmentation method is cheap enough to try even where its benefits are uncertain. The paper is worth a serious referee, but the referee should insist on the control experiment and on code/data availability.\n\nRecommendation: send to peer review. The attack claim is sound and worth publishing; the augmentation claim needs tempering and a mechanism check before it is accepted at face value.","headline":"RDSA is a simple but genuinely effective correlation-breaking attack; the augmentation story needs a control before the mechanism claim holds.","tokens_in":21149,"tokens_out":2307,"would_cite":true,"duration_ms":24622,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that adversarial examples made by shuffling feature values along their own distributions can serve as data augmentation, improving classification performance on several benchmarks by forcing models to attend to…","keywords":["adversarial attacks","data augmentation","input feature correlations","Random Distribution Shuffle Attack","tabular data","particle physics classification","classifier robustness","feature distributions"],"falsifier":"On the VBF and Rain in Australia panels of Figure 8, every tested augmentation method, RDSA included, has mean AUROC below the data-starved baseline, so reproducing that systematic degradation is the concrete check against any general claim that correlation-breaking augmentation improves classification.","tokens_in":20232,"feed_emoji":"🎲","tokens_out":10918,"duration_ms":97517,"temperature":0.7,"pith_summary":"The paper introduces a new adversarial attack, the Random Distribution Shuffle Attack (RDSA), which perturbs an input by randomly redrawing a chosen subset of its features from finely binned histograms of the full dataset. Because each feature is sampled from the same marginal distribution, the one-dimensional statistics of the data remain nearly unchanged, while the correlations between features are destroyed. The authors argue that retraining a classifier on these correctly labeled, correlation-broken adversaries works as data augmentation, and they report AUROC gains over data-starved baselines on four of six tasks spanning particle physics, medicine, handwritten digits, and human activity recognition. The motivating idea is that in physics and other sciences, correlations among observables encode the underlying laws, so a network forced to rely on them should generalize better.","feed_headline":"Correlation-breaking attacks boost retrained models on 4/6 tasks","feed_subtitle":"Preserving single-feature statistics while breaking correlations makes classifiers relearn what matters.","key_machinery":"The central object is the Random Distribution Shuffle Attack (RDSA), which resamples selected input features from their empirical one-dimensional histograms, so that marginals are preserved while inter-feature correlations are broken by the shuffling. The load-bearing mechanism is the contrast between these two effects, measured by the Jensen-Shannon distance (small) and the mean absolute difference of correlation matrices (large); after retraining on such adversaries, a classifier that relied on marginal statistics can no longer do so and must encode correlations to classify correctly.","core_discovery":"RDSA is a non-gradient attack: for each correctly predicted input it selects nVars features at random, redraws their values from empirical histograms built over the whole dataset, and repeats the shuffle until the model returns a different class or a step limit is reached. The resulting adversarial examples achieve high fooling ratios on all six benchmarks while keeping the Jensen-Shannon distance between original and adversarial feature distributions small, often around $10^{-2}$, and while increasing the mean absolute difference between the clean and adversarial correlation matrices. Used as augmentation, RDSA retrains the data-starved model on the reduced clean set plus correctly labeled adversaries, and the paper reports improved AUROC on TopoDNN, MIMIC-IV, MNIST784, and HAR, with all augmentation methods including RDSA degrading AUROC on the VBF and Rain in Australia tasks.","pith_inferences":["RDSA could be used as a diagnostic probe: the fooling ratio as a function of how many features are shuffled measures how strongly a model depends on pairwise correlations, useful for comparing models trained on real versus simulated data.","Because RDSA stays on the empirical marginal manifold, combining it with gradient-based attacks would probe a complementary class of vulnerabilities, giving a fuller robustness profile than either attack alone.","A testable extension is to shuffle only selected feature pairs and track the AUROC drop; the pairs whose shuffling hurts most are the correlation structure the model actually relies on, turning the attack into an interpretability tool.","The negative VBF and Rain in Australia results suggest RDSA augmentation should not be applied blindly; a confidence-weighted or per-class version might recover gains where the current recipe fails."],"forward_implications":["RDSA produces high-fooling-ratio adversaries while keeping one-dimensional feature distributions almost unchanged, a property gradient-based attacks like LowProFool do not share.","Adversarial training with RDSA improves AUROC over a data-starved baseline on TopoDNN, MIMIC-IV, MNIST784, and HAR, and is competitive with or better than CTGAN, TVAE, and LPF on several of these tasks.","RDSA applies to tabular data across physics, weather, medicine, and activity recognition, so the approach is not domain-specific.","Shuffling all continuous features drives inter-feature correlations toward zero while preserving marginals, providing a controlled way to test how much a classifier depends on correlations."],"supporting_citations":[{"why":"Establishes the adversarial-example concept that motivates generating and retraining on adversarial inputs.","marker":"[4]"},{"why":"Provides the FGSM gradient-attack baseline that RDSA is contrasted with.","marker":"[5]"},{"why":"Provides the PGD gradient-attack baseline that RDSA is contrasted with.","marker":"[7]"},{"why":"Provides the LowProFool tabular attack used as the main comparison baseline.","marker":"[12]"},{"why":"Provides the CTGAN and TVAE tabular data augmentation baselines that RDSA is measured against.","marker":"[13]"},{"why":"Defines the VBF Higgs-boson classification task and network used as a physics benchmark.","marker":"[2]"},{"why":"Defines the TopoDNN jet-tagging task and network used as a physics benchmark.","marker":"[22]"},{"why":"Supports the adversarial-training recipe of augmenting with adversaries and retraining.","marker":"[8]"}],"fun_headline_variants":["Shuffle attack breaks correlations, aids 4 of 6 tasks","Correlation-breaking shuffle boosts 4 benchmarks in retrain","RDSA adversarial augmentation lifts AUROC on 4 of 6 tasks","Non-gradient attack on correlations improves 4 classifiers","Input-correlation attack as augmentation: wins on 4 tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that training a network on correctly labeled examples whose inter-feature correlations have been destroyed, while keeping each feature's own distribution intact, makes it rely on the true correlations and generalize better on clean data.","fun_headline_variants_meta":{"raw":{"variants":["Shuffle attack breaks correlations, aids 4 of 6 tasks","Correlation-breaking shuffle boosts 4 benchmarks in retrain","RDSA adversarial augmentation lifts AUROC on 4 of 6 tasks","Non-gradient attack on correlations improves 4 classifiers","Input-correlation attack as augmentation: wins on 4 tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000183,"raw_usage":{"total_tokens":1303,"prompt_tokens":921,"completion_tokens":382,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":297}},"tokens_in":537,"tokens_out":382,"duration_ms":4384,"temperature":1.0,"reasoning_tokens":297,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:12:21.450188+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the VBF and Rain in Australia panels of Figure 8, every tested augmentation method, RDSA included, has mean AUROC below the data-starved baseline, so reproducing that systematic degradation is the concrete check against any general claim that correlation-breaking augmentation improves classification.","supporting_citations":[],"review_version":1}