{"id":"8fc65ac3-4837-4a38-a5ac-d8f5fabb532d","arxiv_id":"2502.02820","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"QLEACE removes all quadratically available class information from a representation, reliably slows feedforward networks, but can inject higher-order information that lets stronger architectures learn faster.","lead":"The authors derive QLEACE, a new method that erases all information a quadratic classifier could use from a dataset's class means and covariances, and test what happens when neural networks train on such erased images. They find that erasing simple linear statistics reliably slows learning, but erasing quadratic statistics can backfire, sometimes speeding learning up.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"QLEACE's exact second-moment guarantee is not verified empirically; ill-conditioned covariances can break the full-rank assumption, so backfiring may be residual quadratic signal rather than injected higher-order information.","rationale":"The reader's weakest_assumption is correct and is the most load-bearing point. The theoretical derivation in §2 is coherent, but the leap from theorem to experiment is unverified. Theorem 2.6's conclusion is conditional on full-rank Σ_i and on the fixed-point solver returning the true barycenter; neither is established for raw image data. Without a post-hoc check of second-moment guardedness, the experimental claims do not distinguish between 'erased quadratic info, model uses higher-order Shannon info' and 'erasure failed, model uses leftover quadratic info.' This is a concrete, checkable gap. I therefore keep the reader's CONDITIONAL verdict: the paper's theoretical contribution is promising, but the empirical support for the central causal claim is conditional on QLEACE being implemented as advertised. The reader's additional objections (no code/data, ambiguous ALF-QLEACE rank, untested explanation) are secondary but reinforce the need for a diagnostic. A single test—measuring residual class-conditional covariance after QLEACE—would settle whether the concern lands.","tokens_in":14270,"tokens_out":6194,"duration_ms":55638,"concrete_test":"Using the released code (or the described algorithm), apply QLEACE to CIFAR-10 and compute the spectral/Frobenius norm of Σ_k − Σ̄ for each class on the transformed training set and a held-out set. If the maximum residual is not negligible relative to ||Σ̄||, the method is not achieving second-moment guardedness. Additionally, retrain the ConvNeXt backfiring configuration with QLEACE built on a ridge-regularized covariance Σ_i + εI; if the backfiring disappears, the original result was an artifact of inexact erasure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.2's Theorem 2.6 and Lemma 2.5 assume each class-conditional covariance Σ_i is full rank (the theorem states it) and that a deterministic Monge map exists (footnote 2 admits this is assumed). On CIFAR-10/SVHN, the raw pixel space has d=3072 and n≈50k–73k; the sample covariance is formally full-rank but extremely ill-conditioned. The paper never states whether QLEACE uses shrinkage, a pseudoinverse, or runs the fixed-point algorithm on raw empirical covariances. If the solver returns an approximate barycenter or unstable square roots, the transformed class-conditional means/covariances will not be equal, violating the very precondition Theorem 2.2 needs for 'no quadratic information'. The paper reports no diagnostic (e.g., max ||Σ_k − Σ̄|| after erasure), so the reader cannot tell whether the guarantee holds in the experiments. Consequently, the headline finding—QLEACE backfires in ConvNeXts because models use 'injected higher order Shannon information'—rests on an unverified assumption about residual second-order statistics. If residual quadratic information remains, the backfiring could be caused by simple second-order features, and the DSB interpretation collapses. This is load-bearing because the paper's central comparison is between exact quadratic erasure and approximate variants.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper derives QLEACE, a closed-form optimal-transport-based erasure method that equalizes class-conditional means and covariances, and compares it with LEACE, an approximate label-free variant (ALF-QLEACE), and a gradient-based erasure method on CIFAR-10, CIFARNet, and SVHN. The main empirical findings are that LEACE consistently slows learning across architectures, while QLEACE slows learning in feedforward networks but backfires in ConvNeXt and Swin models, and approximate variants can act as data augmentation. The authors interpret backfiring as evidence that expressive architectures exploit injected higher-order Shannon information about class labels.","tokens_in":14507,"tokens_out":8161,"duration_ms":69050,"significance":"If the QLEACE guarantee holds in practice, the paper provides a clean minimal-distortion method for removing all second-order class information and a novel intervention-based demonstration of the distributional simplicity bias. The theoretical core (Theorems 2.2, 2.3, and 2.6) is clearly stated with proofs in the appendix; the use of maximal update parametrization and prequential MDL is appropriate, and the width-scaling comparison is informative. The backfiring phenomenon is a striking and potentially influential result. However, the empirical verification of the second-moment guarantee is absent, which leaves the main interpretation underdetermined.","major_comments":[{"comment":"The exact second-moment guarantee is not verified empirically. The theorem assumes full-rank class-conditional covariance matrices and an exact solution to the barycenter equation (4), but on raw-pixel CIFAR-10/SVHN (d=3072) the empirical covariance is severely ill-conditioned; the paper does not state whether QLEACE uses shrinkage, a pseudoinverse, or truncated SVD, and it reports no post-erasure diagnostic of residual class-conditional covariance differences. Since the central backfiring interpretation attributes the late ConvNeXt/Swin performance to injected higher-order Shannon information, the missing quadratic-residual check leaves the plausible alternative that residual second-order signal drives the effect. Please report a diagnostic such as max_k ||Σ_k − Σ̄||_F (and the mean/covariance deviations for all erasers) and specify the numerical solver, regularization, and convergence tolerance.","section":"Section 2.2 (Theorem 2.6) and Section 4"},{"comment":"The rank of the ALF-QLEACE projection is specified inconsistently. Eq (11) says 'rank d−k', the next paragraph says 'rank d − 1', Figure 1 says 'rank d − 15', and the iterative-erasure paragraph says 'up to d rank 1 erasures' can be added. In addition, the claimed min-max optimality of truncating the singular-value decomposition of the worst-class difference matrix is asserted but not proved; the objective (11) is over a projection matrix P and is not obviously solved by that heuristic. Please give the exact algorithm (rank parameter, iteration count, class selection rule) and a proof or citation for the projection optimality.","section":"Section 2.3, Eq (11), and Figure 1"},{"comment":"The gradient-based erasure method is not reproducible. The three loss-term weights are said to be 'tuned' to ensure near-zero quadratic information while staying close to the original images, but their final values, the L-BFGS hyperparameters, and the optimization budget are not reported, and no code or data are released. Because the comparison across erasers is the empirical backbone of the paper, please provide code/data or a complete hyperparameter table (including per-dataset values) for all erasure methods.","section":"Section 2.4"},{"comment":"The causal claim that backfiring reflects learned use of 'injected higher order Shannon information' is not directly supported by any quantitative measurement of higher-order statistics. The paper shows sample images and learning curves, but does not report third- or fourth-order conditional moment differences, mutual-information estimates, or the accuracy of a nonlinear probe on QLEACE'd data. Without such a probe, the alternative explanation that the backfiring is due to residual quadratic information (or some other artifact) is not excluded. Please add a higher-order statistic probe or soften the causal interpretation.","section":"Section 4, QLEACE paragraph"}],"minor_comments":[{"comment":"'Backfiring occurs in all two- and three-layer MLPs on the CIFARNet dataset' conflicts with the Figure 8 caption, which says 'for some model dimensions'; please reconcile the text with the figure.","section":"Section 4, CIFARNet"},{"comment":"The notation 'k∼K' is nonstandard and should be 'k ∈ K'; also, the sentence invoking the min-max theorem should state explicitly which matrix norm is being used.","section":"Section 2.3, Eq (11)"},{"comment":"The z-score normalization analysis is described as 'ambiguous' but no sample sizes, standard errors, or test details are given; please clarify how the ambiguity was assessed.","section":"Section 4 and Table 16"},{"comment":"The footnote acknowledges that the Monge problem may not have a solution; since real image covariances may be low-rank, a brief discussion of a Kantorovich relaxation or a regularized solver would help readers judge when QLEACE's guarantee applies.","section":"Section 2.2, footnote 2"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of the journal and the theory is sound, but the empirical claims need substantial strengthening before publication. The absence of code/data and the untested numerical realization of QLEACE are the main risks. The self-citation pattern (LEACE, DSB) is appropriate given the direct reliance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful thing here is QLEACE—the closed-form method for erasing all class-conditional mean and covariance differences via a Gaussian Wasserstein barycenter—and the backfiring observation in convolutional architectures. The math is the strong part: Theorems 2.2, 2.3, and 2.6 are coherent, the proof in the appendix checks out, and the extension from LEACE's linear erasure to quadratic moments is a genuine contribution. The experimental finding that QLEACE consistently slows learning in MLPs but later triggers a sharp loss collapse in ConvNeXts and Swins is real and worth understanding.\n\nThat said, the paper overreaches in its causal story. The claim that sophisticated architectures 'learn to use injected higher order Shannon information' is not directly measured. The guarantee of exact quadratic guardedness rests on full-rank covariance assumptions and a deterministic Monge map; on CIFAR-10 pixel space, sample covariances are extremely ill-conditioned. The paper never says whether the barycenter solve is regularized, and it reports no diagnostic of residual class-conditional covariance mismatch. If residual second-order signal remains, the backfiring could be simple quadratic features, not higher-order Shannon information. That is a load-bearing gap, not a nit.\n\nOther soft spots are smaller. No code or data is released, so the numbers are not checkable. The ALF-QLEACE rank is specified inconsistently—rank d−k in the text, rank d−1 a few paragraphs later, rank d−15 in the figure caption. The gradient-based erasure loss weights are tuned but not reported. All of these are fixable in revision.\n\nI don't think the paper is fatally flawed. The mathematical contribution is solid, and the backfiring phenomenon deserves publication. But I'd want a serious referee to push on the empirical support: require code, a description of the covariance solver, a residual-moment diagnostic, and either measurement of the leaked Shannon information or softer causal language. A competent editor should send this to review, not desk-reject.","headline":"Solid closed-form quadratic erasure with a real backfiring finding, but the empirical support is thinner than the causal claims.","tokens_in":15064,"tokens_out":2767,"would_cite":true,"duration_ms":24839,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper derives QLEACE, a closed-form method that removes all class information available to quadratic classifiers, and shows it reliably slows feedforward learning while expressive vision models recover the labels from higher-order…","keywords":["concept erasure","quadratic moments","LEACE","QLEACE","distributional simplicity bias","optimal transport","prequential MDL","learning dynamics"],"falsifier":"Run QLEACE on a dataset whose class-conditional covariance is rank-deficient, then fit a quadratic classifier to the erased data; if it beats chance, the full-rank assumption is violated and the closed-form guarantee is not met. Equivalently, on any real dataset, compare the empirical mean and covariance of each erased class to the barycenter target; any significant mismatch is direct evidence the method did not equalize second moments.","tokens_in":14024,"feed_emoji":"🧠","tokens_out":9946,"duration_ms":83278,"temperature":0.7,"pith_summary":"The paper tries to establish that erasing simple, low-order statistical features makes learning harder, and that how much harder reveals how networks learn. It derives QLEACE, a closed-form map that equalizes each class's mean and covariance with the dataset-wide mean and covariance, thereby removing all information available to quadratic classifiers. Training on QLEACE'd data reliably slows feedforward networks, but expressive vision models eventually exploit higher-order statistics that QLEACE inadvertently injects, sometimes learning faster than on the original data. The paper also studies approximate label-free erasure, which avoids injection but acts like data augmentation on some datasets. These results matter because they test the distributional simplicity bias and warn practitioners that moment erasure is not the same as information erasure.","feed_headline":"Erasing quadratic features slows small nets, backfires in large ones","feed_subtitle":"New eraser removes all low-order clues; expressive vision models later exploit hidden higher-order info.","key_machinery":"The load-bearing machine is QLEACE's per-class affine transport map $T_i(x) = \\Sigma_i^{-1/2}(\\Sigma_i^{1/2}\\bar{\\Sigma}\\Sigma_i^{1/2})^{1/2}\\Sigma_i^{-1/2}(x-m_i)+\\bar{m}$, with target mean $\\bar{m}$ and covariance $\\bar{\\Sigma}$ defined as the Gaussian Wasserstein barycenter of the class means and covariances. This map is proven to minimize the total expected squared edit among all transports that equalize class means and covariances, and moment equalization is proven sufficient and, under conditions on the loss, necessary for a degree-$N$ polynomial classifier to be no better than the constant predictor. The other machinery includes LEACE linear erasure, ALF-QLEACE's rank-$d-r$ projection that removes the directions of largest class variance differences, L-BFGS gradient-based moment erasure, and prequential MDL as the measure of learning difficulty.","core_discovery":"The paper's central discovery is that there is a closed-form, provably minimal way to erase every piece of class information that a quadratic, degree-2 polynomial classifier could use, and that this erasure behaves very differently from linear erasure. QLEACE applies, per class, the optimal transport map from the class-conditional distribution to a common target distribution whose mean and covariance are the Gaussian Wasserstein barycenter of the class means and covariances; Theorems 2.2 and 2.3 show matching first- and second-order moments makes the constant predictor optimal among polynomial predictors, and Theorem 2.6 proves this particular target minimizes the expected squared edit. On CIFAR-10, CIFARNet, and SVHN, QLEACE consistently slows feedforward MLPs and LeNets more than LEACE does, but in ConvNeXt and Swin models it backfires after about 16 epochs: the models exploit higher-order Shannon information QLEACE injected about class labels and can reach lower final loss than on unerased data. The approximate label-free variant ALF-QLEACE avoids injection because it applies the same projection to every point, yet it sometimes makes learning easier, acting as data augmentation. The paper concludes that quadratic concept erasure is unreliable in practice, while LEACE is a reliable way to make features less salient.","pith_inferences":["A testable extension the authors leave implicit: if backfiring is caused by class-dependent geometry, then a deep network trained on QLEACE'd data should show the same late-epoch loss collapse on any dataset where the barycenter maps are non-isometric, while ALF-QLEACE should never show it.","We infer that moment erasure is insufficient as a fairness or safety intervention in deployment, since a trained adversarial network, not a polynomial classifier, is the relevant threat model; a label-independent or Shannon-information erasure would be safer.","The sharp transition around epoch 16 suggests that probing intermediate layers of ConvNeXt or Swin before and after that epoch could reveal exactly where higher-order statistics are decoded, making the mechanism observable rather than inferred from loss curves.","An open design question follows: can the data-augmentation effect of ALF-QLEACE be harnessed deliberately? Since it removes high-variance class-specific directions, it may act as a regularizer; comparing it to standard augmentation on the same architectures would separate the two effects."],"forward_implications":["Because QLEACE transforms each class with a different map, it can inject label-dependent geometry into higher-order moments; the observed backfiring means moment-matching is not information-erasure when the downstream network is expressive.","LEACE remains the reliable erasure: it raises prequential MDL and final loss across every architecture, dataset, width, and depth tested, with roughly constant effect as networks widen.","The backfiring transition, which appears around 16 epochs in ConvNeXt and Swin on CIFAR-10, is direct evidence that distributional simplicity bias breaks down once networks have had enough time to exploit higher-order statistics.","ALF-QLEACE and gradient-based erasure do not backfire because they do not depend on the class label, but their effects are smaller and sometimes negative, acting as data augmentation, so quadratic erasure is dataset- and architecture-dependent.","Any practitioner wanting to make a feature harder to learn should prefer LEACE; quadratic erasers should be used with caution and validated for backfiring on the target architecture."],"supporting_citations":[{"why":"Supplies LEACE, the linear concept erasure used throughout, and the theorem template this paper extends to polynomial predictors.","marker":"Belrose et al. (2023)"},{"why":"Provides the distributional simplicity bias evidence and the CIFARNet dataset that motivate the inverted experiment.","marker":"Belrose et al. (2024)"},{"why":"Gives the Gaussian Wasserstein barycenter result used to define QLEACE's target mean and covariance.","marker":"Rüschendorf & Uckelmann (2002)"},{"why":"Provides the 2-Wasserstein lower bound and the optimal map form that make QLEACE's per-class transport cost-minimal.","marker":"Cuesta-Albertos et al. (1996)"},{"why":"Supplies the fixed-point algorithm used to solve the barycenter equation for the target covariance.","marker":"Álvarez-Esteban et al. (2016)"},{"why":"Defines the prequential MDL framework used to quantify learning difficulty across erasure methods.","marker":"Voita & Titov (2020)"},{"why":"Supplies the maximal update parametrization that lets the paper transfer hyperparameters across widths, supporting the width-independence conclusions.","marker":"Yang et al. (2021)"},{"why":"Provides the L-BFGS optimizer used in the gradient-based moment erasure experiments.","marker":"Liu & Nocedal (1989)"}],"fun_headline_variants":["Quadratic erasure slows small nets, backfires in large via hidden info","QLEACE removes all quadratic clues, but large models exploit injected randomness","Erasing simple features: small models stumble, large models adapt to higher-order cues","Minimal quadratic erasure Slows small nets, but big nets leverage class-label info"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guarantee that QLEACE removes all quadratically available information assumes every class has a full-rank covariance matrix and an exact deterministic transport map exists; on real image data those conditions often fail, so the computed erasure may fall short of the guarantee.","fun_headline_variants_meta":{"raw":{"variants":["Quadratic erasure slows small nets, backfires in large via hidden info","QLEACE removes all quadratic clues, but large models exploit injected randomness","Erasing simple features: small models stumble, large models adapt to higher-order cues","Minimal quadratic erasure Slows small nets, but big nets leverage class-label info"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000294,"raw_usage":{"total_tokens":1725,"prompt_tokens":974,"completion_tokens":751,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":666}},"tokens_in":590,"tokens_out":751,"duration_ms":7641,"temperature":1.0,"reasoning_tokens":666,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T10:59:38.421866+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run QLEACE on a dataset whose class-conditional covariance is rank-deficient, then fit a quadratic classifier to the erased data; if it beats chance, the full-rank assumption is violated and the closed-form guarantee is not met. Equivalently, on any real dataset, compare the empirical mean and covariance of each erased class to the barycenter target; any significant mismatch is direct evidence the method did not equalize second moments.","supporting_citations":[],"review_version":1}