Pith. sign in

REVIEW 3 major objections 45 references

Mixing local adversarial perturbations into hierarchical sparse selection yields more reproducible features and better out-of-distribution accuracy without denser supports.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 23:35 UTC pith:Z6BUSC7H

load-bearing objection Solid LassoNet + first-order adversarial mix with code and honest diagnostics; ColoredMNIST gains are real and sparsity-matched, but the SERS abstract numbers rest on unmatched feature counts and Table 7 shows the method can hurt under tight budgets. the 3 major comments →

arxiv 2607.03839 v1 pith:Z6BUSC7H submitted 2026-07-04 cs.LG

Adversarial LassoNet: Robust Feature Selection via Stability-Driven Sparse Learning

Zhen Huang , Peicheng Xu , Junbiao Pang , Yulong Zheng This is my paper
classification cs.LG
keywords sparse feature selectionadversarial trainingLassoNethierarchical sparsityout-of-distribution robustnesssupport reproducibilityNeural Tangent KernelSERS
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard sparse feature selection optimizes for clean training accuracy and can lock onto shortcut variables that fail under noise or distribution shift. This paper argues that the same hierarchical sparsity mechanism used by LassoNet can be steered toward stable features by adding a first-order worst-case input perturbation to the training objective. The mixed loss keeps the original proximal sparsity pipeline intact while the perturbation signal preferentially weakens features whose predictive contribution collapses under small local changes. On a high-dimensional medical spectroscopy task and a controlled color-spurious digit benchmark, the method keeps competitive sparse accuracy, lifts out-of-distribution accuracy, and makes the selected feature sets more consistent across random seeds. A spectral diagnostic further links the gain to a less concentrated empirical Neural Tangent Kernel and milder Hessian curvature.

Core claim

Integrating a tractable first-order adversarial stability term into LassoNet’s hierarchical sparsity objective produces feature supports that remain competitive on clean sparse selection while improving out-of-distribution robustness and cross-run support reproducibility under nearly matched sparsity, with corresponding gains on a high-dimensional lung-cancer SERS screening task.

What carries the argument

The mixed objective L_mix = (1−α)L(clean) + α L(x + r*), where r* is the closed-form first-order ℓ2 adversarial direction ρ ∇_x ℓ / (‖∇_x ℓ‖ + ε), optimized by alternating perturbation construction with LassoNet’s hierarchical proximal updates; an NTK effective-rank expansion supplies a local spectral-growth condition that explains reduced gradient concentration.

Load-bearing premise

The method assumes that features most sensitive to small worst-case input changes are the ones that should be pruned as unstable shortcuts, rather than useful but fragile signal.

What would settle it

On a controlled spurious-correlation benchmark with matched feature budgets, if AdLNet’s selected supports show no higher OOD accuracy or Jaccard reproducibility than vanilla LassoNet (or systematically drop known stable features while retaining color shortcuts), the stability-driven selection claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. The paper proposes Adversarial LassoNet (AdLNet), which augments LassoNet’s hierarchical sparsity with a mixed objective that balances clean predictive loss and first-order input-space adversarial stability (Eqs. 3–7, Algorithm 1). Under a local Lipschitz-gradient assumption, the authors derive a residual-bounded first-order approximation of local adversarial sensitivity and give an NTK-inspired spectral analysis (Lemma 2, Proposition 3) linking perturbation-driven training to increased effective rank when a spectral growth score is positive. Empirically, AdLNet is competitive with sparse baselines on six public datasets at a matched budget k=50, improves ColoredMNIST OOD accuracy and Jaccard support reproducibility under nearly matched sparsity, and reports gains on a high-dimensional SERS lung-cancer dataset versus vanilla LassoNet, with code released.

Significance. If the stability-driven selection story holds, the work is a useful and practical bridge between hierarchical sparse feature selection and adversarial robustness, especially for high-dimensional scientific/medical settings where support reproducibility matters. Strengths include a clean residual bound (Appendix A), preservation of LassoNet’s hierarchical proximal pipeline rather than a new sparsifier, public code/dataset, and sparsity-matched ColoredMNIST evidence for modest OOD and reproducibility gains. The NTK spectral diagnostics are a thoughtful geometric interpretation even if only diagnostic. The main applied significance hinges on whether SERS gains can be isolated from feature-count effects; that isolation is currently incomplete.

major comments (3)
  1. Abstract and Table 1 claim a 5.3% test-accuracy and 6.0% AUC gain on SERS over vanilla LassoNet, but the comparison is not sparsity-matched (119 vs 256 selected features under n_selected<300). Appendix Table 7 shows that under a high-sparsity budget (k≈120) the adversarial model collapses (test acc 72.14→52.86, AUC 0.775→0.482), while under moderate sparsity (k≈250) it wins while selecting slightly fewer features. The medical headline is therefore not yet isolated from feature-count confounds. Please either (i) report the main SERS comparison under a fixed matched budget with the same validation protocol, or (ii) substantially qualify the abstract/Table 1 claim and lead with the controlled-budget analysis.
  2. The load-bearing premise (Sections 3.1–3.3) is that local worst-case ℓ2 sensitivity (Eqs. 3–6) preferentially identifies spuriously correlated features so that hierarchical proximal updates prune shortcuts rather than useful but perturbation-sensitive signal. ColoredMNIST supports a modest OOD/reproducibility effect under matched sparsity, but the SERS high-sparsity failure in Table 7 shows the proxy can remove predictive signal when the budget is tight. The paper needs a clearer discussion of when the proxy is expected to help versus hurt, and ideally a diagnostic that separates “stable predictive support” from “smoother denser predictor.”
  3. Section 4.4 and Proposition 3: the NTK analysis is local and first-order (Θ̃(ρ)=Θ+ρE+R(ρ)), while Ĝ_macro compares two fully trained checkpoints on clean inputs. The manuscript already labels this diagnostic, but the abstract and conclusion still present “NTK-inspired spectral analysis” as characterizing how training reduces gradient concentration. Please keep the claim strictly diagnostic, or provide a more direct empirical check of the local growth condition G(E;Θ) rather than only the macroscopic post-hoc score.

Circularity Check

0 steps flagged

No significant circularity: AdLNet’s mixed objective, first-order perturbation, and NTK growth condition are independent of the reported accuracy/OOD gains; experiments are external benchmarks, not tautologies of the definitions.

full rationale

This is a standard empirical method paper. The training objective L_mix = (1−α)L_clean + α L_adv (Eq. 7) and the hierarchical proximal pipeline (Algorithm 1) are not defined in terms of test accuracy, OOD gap, Jaccard, or SERS AUC; those metrics are measured on held-out data after training. The first-order adversarial approximation (Eqs. 5–6 under Assumption 1) is a routine Cauchy–Schwarz linearization with an O(ρ²) residual bound (Appendix A), not a fit renamed as a prediction. The NTK analysis (Lemma 2 / Proposition 3) states a sufficient local condition G(E;Θ)>0 for effective-rank increase; the post-training ˆG_macro diagnostic is explicitly labeled “empirical diagnostic rather than a direct proof,” so it does not force the OOD claim by construction. There is no load-bearing self-citation uniqueness theorem, no ansatz smuggled from the authors’ prior work, and no renaming of a known empirical law as a first-principles derivation. Validation selection of α/λ/M is ordinary hyperparameter practice and does not make the derivation circular. Experimental confounds (e.g., unmatched SERS feature counts in Table 1 vs. Table 7) are correctness/isolation issues, not circularity of the claimed derivation chain.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The method rests on standard adversarial-training and LassoNet machinery plus a few tunable knobs and one local smoothness assumption. The NTK spectral story adds a conditional growth criterion rather than a free physical entity. No new particles or forces; the invented objects are the method name and the spectral growth score used as interpretation.

free parameters (4)
  • adversarial mixing coefficient α
    Balances clean vs adversarial loss; chosen from {0,0.2,...,1.0} by validation accuracy and is central to the stability–fidelity trade-off.
  • perturbation radius ρ
    Sets the ℓ2 ball for r*; controls how strong the stability signal is. Treated as a hyperparameter in Algorithm 1.
  • hierarchy parameter M and sparsity path λ
    Inherited from LassoNet; jointly determine which features survive HierProx and thus the reported sparse supports.
  • numerical stabilizer ε in r*
    Ad hoc constant in the normalized gradient direction to avoid division by zero.
axioms (4)
  • domain assumption Assumption 1: per-example loss has locally Lipschitz input gradient on the ρ-ball, justifying S_ρ = ρ‖∇xℓ‖2 + O(ρ²).
    Invoked in §3.2 and proved in Appendix A; standard for ReLU-type nets but not verified on the SERS spectra.
  • ad hoc to paper Empirical NTK admits a first-order expansion Θ̃(ρ)=Θ+ρE+R(ρ) with ‖R‖F=o(ρ) and Θ̃(ρ)⪰0 for small ρ.
    §3.4 / Appendix B; needed for the effective-rank Taylor expansion and Proposition 3.
  • domain assumption Local worst-case input sensitivity is a useful training signal for pruning spuriously correlated features under hierarchical sparsity.
    Core modeling premise of §3.1–3.3; not derived, only motivated and tested empirically.
  • standard math LassoNet hierarchical proximal constraint correctly implements input-level feature selection (‖W_j^(1)‖∞ ≤ M‖θ_j‖2).
    Taken from Lemhadri et al. 2021 and reused unchanged.
invented entities (2)
  • Adversarial LassoNet (AdLNet) mixed objective independent evidence
    purpose: Name the stability-driven sparse learner that couples L_mix with HierProx.
    Methodological construct; independent evidence is the empirical tables and public code, not an external physical measurement.
  • Spectral growth score G(E;Θ) / Ĝ_macro no independent evidence
    purpose: Interpret when adversarial training raises NTK effective rank and diagnose gradient concentration.
    Defined in Lemma 2 / Proposition 3 and estimated post hoc in Table 4; not independently measured outside this analysis.

pith-pipeline@v1.1.0-grok45 · 19905 in / 3475 out tokens · 33147 ms · 2026-07-11T23:35:21.610605+00:00 · methodology

0 comments
read the original abstract

Sparse feature selection is critical for high-dimensional machine learning, yet traditional $\ell_1$-regularized methods are often brittle under observational noise and spurious correlations, leading to unstable feature supports and degraded generalization. Although adversarial training has been widely used to improve model robustness, its interaction with hierarchical sparse feature selection remains underexplored. In this work, we propose Adversarial LassoNet (AdLNet), a stability-driven sparse feature selection framework that integrates input-space adversarial perturbations with the hierarchical sparsity mechanism of LassoNet. We derive a tractable first-order adversarial approximation under local smoothness assumptions and provide an NTK-inspired spectral analysis to characterize how perturbation-driven training can reduce gradient concentration. Experiments on high-dimensional SERS data, six public benchmark datasets, and ColoredMNIST show that AdLNet maintains competitive sparse-selection performance while improving out-of-distribution robustness by 4.4\% and feature support reproducibility by 6.3\% under nearly matched support sparsity on ColoredMNIST. On the high-dimensional lung cancer screening dataset, AdLNet achieves a 5.3\% test accuracy gain and a 6.0\% AUC improvement over vanilla LassoNet. Code and dataset are available at https://github.com/719573/Adversarial-LassoNet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

45 extracted references · 3 linked inside Pith

  1. [1]

    arXiv preprint arXiv:1907.02893 , year=

    Invariant Risk Minimization , author=. arXiv preprint arXiv:1907.02893 , year=

    Pith/arXiv arXiv 1907

  2. [2]

    Proceedings of the IEEE , volume=

    Gradient-based learning applied to document recognition , author=. Proceedings of the IEEE , volume=

  3. [3]

    arXiv preprint arXiv:1708.07747 , year=

    Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms , author=. arXiv preprint arXiv:1708.07747 , year=

    Pith/arXiv arXiv

  4. [4]

    2015 , howpublished=

    Expression data from the brain of control and trisomic mice , author=. 2015 , howpublished=

    2015

  5. [5]

    Advances in Neural Information Processing Systems , volume=

    Spoken letter recognition , author=. Advances in Neural Information Processing Systems , volume=

  6. [6]

    Columbia Object Image Library (COIL-20) , author=

  7. [7]

    European Symposium on Artificial Neural Networks , pages=

    A public domain dataset for human activity recognition using smartphones , author=. European Symposium on Artificial Neural Networks , pages=

  8. [8]

    Journal of the Royal Statistical Society: Series B , volume=

    Regression Shrinkage and Selection via the Lasso , author=. Journal of the Royal Statistical Society: Series B , volume=

  9. [9]

    Journal of the Royal Statistical Society: Series B , volume=

    Model Selection and Estimation in Regression with Grouped Variables , author=. Journal of the Royal Statistical Society: Series B , volume=

  10. [10]

    Journal of the Royal Statistical Society: Series B , volume=

    The Group Lasso for Logistic Regression , author=. Journal of the Royal Statistical Society: Series B , volume=

  11. [11]

    Journal of Machine Learning Research , volume=

    An Introduction to Variable and Feature Selection , author=. Journal of Machine Learning Research , volume=

  12. [12]

    Journal of Machine Learning Research , volume=

    LassoNet: A Neural Network with Feature Sparsity , author=. Journal of Machine Learning Research , volume=

  13. [13]

    International Conference on Learning Representations , year=

    Explaining and Harnessing Adversarial Examples , author=. International Conference on Learning Representations , year=

  14. [14]

    International Conference on Learning Representations , year=

    Towards Deep Learning Models Resistant to Adversarial Attacks , author=. International Conference on Learning Representations , year=

  15. [15]

    AAAI Conference on Artificial Intelligence , year=

    Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing Their Input Gradients , author=. AAAI Conference on Artificial Intelligence , year=

  16. [16]

    International Conference on Learning Representations , year=

    Sharpness-Aware Minimization for Efficiently Improving Generalization , author=. International Conference on Learning Representations , year=

  17. [17]

    International Conference on Learning Representations , year=

    Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization , author=. International Conference on Learning Representations , year=

  18. [18]

    Advances in Neural Information Processing Systems , volume=

    Neural Tangent Kernel: Convergence and Generalization in Neural Networks , author=. Advances in Neural Information Processing Systems , volume=

  19. [19]

    Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages=

    ``Why Should I Trust You?'': Explaining the Predictions of Any Classifier , author=. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages=

  20. [20]

    Advances in Neural Information Processing Systems , volume=

    A Unified Approach to Interpreting Model Predictions , author=. Advances in Neural Information Processing Systems , volume=

  21. [21]

    Harvard Journal of Law & Technology , volume=

    Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR , author=. Harvard Journal of Law & Technology , volume=

  22. [22]

    IEEE Transactions on Medical Imaging , volume=

    FISTA-Net: Learning a Fast Iterative Shrinkage Thresholding Network for Inverse Problems in Imaging , author=. IEEE Transactions on Medical Imaging , volume=. 2021 , publisher=

    2021

  23. [23]

    Advances in Neural Information Processing Systems , volume=

    A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learning , author=. Advances in Neural Information Processing Systems , volume=

  24. [24]

    IEEE Transactions on Signal Processing , volume=

    On the Adversarial Robustness of LASSO Based Feature Selection , author=. IEEE Transactions on Signal Processing , volume=. 2021 , doi=

    2021

  25. [25]

    Electronics , volume =

    Yang, Libo and Zhu, Dawei and Liu, Xuemei and Cui, Pei , title =. Electronics , volume =. 2023 , publisher =

    2023

  26. [26]

    Scientific Reports , volume=

    Application of high-dimensional feature selection: evaluation for genomic prediction in man , author=. Scientific Reports , volume=. 2015 , doi=

    2015

  27. [27]

    Scientific Reports , volume=

    Bird's Eye View feature selection for high-dimensional data , author=. Scientific Reports , volume=. 2023 , doi=

    2023

  28. [28]

    Scientific Reports , volume=

    Raman spectroscopy and topological machine learning for cancer grading , author=. Scientific Reports , volume=. 2023 , doi=

    2023

  29. [29]

    Scientific Reports , volume=

    Raman spectroscopy and artificial intelligence to predict the Bayesian probability of breast cancer , author=. Scientific Reports , volume=. 2021 , doi=

    2021

  30. [30]

    Nature Machine Intelligence , volume=

    Shortcut learning in deep neural networks , author=. Nature Machine Intelligence , volume=. 2020 , doi=

    2020

  31. [31]

    Nature Communications , volume=

    Detecting shortcut learning for fair medical AI using shortcut testing , author=. Nature Communications , volume=. 2023 , doi=

    2023

  32. [32]

    Nature Communications , volume=

    Causality in digital medicine , author=. Nature Communications , volume=. 2021 , doi=

    2021

  33. [33]

    Science , volume=

    Adversarial attacks on medical machine learning , author=. Science , volume=. 2019 , doi=

    2019

  34. [34]

    Nature Communications , volume=

    Advancing diagnostic performance and clinical usability of neural networks via adversarial training and dual batch normalization , author=. Nature Communications , volume=. 2021 , doi=

    2021

  35. [35]

    Scientific Reports , volume=

    Training calibration-based counterfactual explainers for deep learning models in medical image analysis , author=. Scientific Reports , volume=. 2022 , doi=

    2022

  36. [36]

    Nature Communications , volume=

    Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks , author=. Nature Communications , volume=. 2021 , doi=

    2021

  37. [37]

    Proceedings of the National Academy of Sciences , volume=

    Benign overfitting in linear regression , author=. Proceedings of the National Academy of Sciences , volume=. 2020 , publisher=

    2020

  38. [38]

    International Conference on Machine Learning , pages=

    Feature selection using stochastic gates , author=. International Conference on Machine Learning , pages=. 2020 , organization=

    2020

  39. [39]

    International Conference on Machine Learning , pages=

    Concrete autoencoders for differentiable feature selection and reconstruction , author=. International Conference on Machine Learning , pages=. 2019 , organization=

    2019

  40. [40]

    Advances in Neural Information Processing Systems , volume=

    Learning both weights and connections for efficient neural network , author=. Advances in Neural Information Processing Systems , volume=

  41. [41]

    International Conference on Learning Representations , year=

    The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks , author=. International Conference on Learning Representations , year=

  42. [42]

    arXiv preprint arXiv:1904.10509 , year=

    Generating long sequences with sparse transformers , author=. arXiv preprint arXiv:1904.10509 , year=

    Pith/arXiv arXiv 1904

  43. [43]

    Advances in Neural Information Processing Systems , volume=

    Big Bird: Transformers for longer sequences , author=. Advances in Neural Information Processing Systems , volume=

  44. [44]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    A Single-Step, Sharpness-Aware Minimization is All You Need to Achieve Efficient and Accurate Sparse Training , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  45. [45]

    International Conference on Learning Representations (ICLR) , year=

    Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization , author=. International Conference on Learning Representations (ICLR) , year=