Pith. sign in

REVIEW 3 major objections 5 minor 4 references

Approximate Equivariance via Projection-based Regularisation

T0 review · 3 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A projection-based regulariser steers networks toward any desired degree of equivariance.

desk verdict A clean projection-based regularizer with a useful Fourier-domain implementation, but the CT-MAR experiment doesn't explain how the C4 projection applies to standard CNNs. read the letter →

arxiv 2601.05028 v2 pith:3GTO2PNJ submitted 2026-01-08 cs.LG

Torben Berndt , Jan Stühmer This is my paper
classification cs.LG MSC 68T0722D1043A30
keywords approximateequivarianceprojectionregularisationequivariantReynoldsoperatorPeter–WeyldecompositionFourierdomaincontinuoussymmetrygroups
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that approximate equivariance can be induced by penalising the distance of each linear layer's weight matrix from its projection onto the equivariant subspace, rather than by penalising sample-wise equivariance violations. This projection distance provably brackets the worst-case equivariance defect up to a factor of two, and for continuous groups it can be computed exactly and efficiently in the Fourier/Peter–Weyl domain by zeroing off-diagonal frequency blocks. The authors show that a single hyperparameter tunes the degree of enforced symmetry, and report consistent gains in accuracy and large runtime savings over sample-based regularisers. The central payoff would be a parameter-free, sampling-free, architecture-level control of symmetry as an inductive bias.

What carries the argument

The orthogonal projection P (the Reynolds or G-smoothing operator) onto the subspace of G-equivariant linear maps, combined with the Peter–Weyl decomposition. In the spectral domain, equivariant maps are block-diagonal over irreducible representations, so P(T) is obtained by masking and averaging frequency blocks; this makes the penalty exact and efficient for continuous groups. The identity ∥T−P(T)∥ ≤ E(T) ≤ 2∥T−P(T)∥ is the quantitative heart of the method.

What would settle it

Train the same network twice on a symmetry-breaking task, once with the projection regulariser and once with a sample-based penalty that tracks the true equivariance defect, and measure the empirical worst-case defect E(W) per layer over a dense grid of group elements. If some layer simultaneously has large Frobenius projection distance yet small spectral defect (or vice versa), the bound ∥T−P(T)∥ ≤ E(T) ≤ 2∥T−P(T)∥ fails numerically and the claimed control of the defect by the implemented regulariser is not what is being optimised.

Watch

Extended reading notes

Core claim

The central claim is that for any linear map T between unitary G-modules, the distance ∥T−P(T)∥ to the equivariant subspace (where P is the Haar-averaging projection) is equivalent up to a factor of two to the worst-case equivariance defect E(T) = sup_{g∈G} ∥π_out(g)T − Tπ_in(g)∥. Hence regularising layer weights by ∥W−P(W)∥ controls the network's true equivariance error through a composition bound. The paper further proves that in the Fourier domain the projection acts simply: it zeroes all off-diagonal irreducible-representation blocks and averages within each diagonal block, giving a closed-form, efficiently computable penalty for groups such as SO(n).

Load-bearing premise

The theory bounds the worst-case equivariance defect using the operator/Lipschitz norm of the layer weights, but the implementation penalises the Frobenius norm and measures defects as data averages; the factor-of-two guarantee need not transfer unless these norms are comparable in practice.

Editorial extensions

If this is right

  • Networks can be tuned from fully equivariant to fully flexible by adjusting λ⊥ (the penalty on the non-equivariant component), while λ_G controls the norm of the equivariant component.
  • The penalty has zero estimator variance and needs no data sampling or extra forward passes, so training throughput roughly doubles compared with sample-based regularisers in the reported CT experiments.
  • For steerable or spectrally parameterised layers, projection reduces to masking and averaging, costing O(N) per layer.
  • The factor-of-two bound extends to whole networks: the global equivariance error is bounded by a Lipschitz-weighted sum of per-layer projection distances.
  • A train-then-project variant (projecting a trained non-equivariant model into the equivariant subspace at test time) is feasible but yields weaker reconstruction quality than during-training regularisation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the norm gap between the implemented Frobenius penalty and the theoretical operator norm is benign in practice, the method could serve as a drop-in regulariser for any differentiable architecture with linear layers, provided an efficient projection for the relevant symmetry group can be derived.
  • The block-diagonal view suggests a natural diagnostic: the per-irrep projection norms reveal which symmetry channels the network actually uses, potentially guiding architecture design (e.g., which multiplicities to keep or discard).
  • A testable extension would combine the projection penalty with a learned or sampled measure over the group, letting the data choose between exact and broken symmetry separately for each frequency block.
  • The runtime advantage over sample-based penalties should grow with group size and input dimension, because sampling coverage degrades in high dimensions while the Fourier projection cost is dimension-controlled.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces projection-based regularisation for approximate equivariance. For linear layers, the regulariser penalises ||W - P(W)||, where P is the Reynolds operator projecting onto the G-equivariant subspace. The authors present a Fourier-domain method for computing P efficiently for compact groups, give a theoretical bound relating the penalty to the worst-case equivariance defect (Lemma 3.2), decompose the defect through layer compositions (Lemma 3.3, Corollary 3.4), and report experiments on an SO(2)-invariance toy problem, imperfectly symmetric smoke dynamics, and CT metal artefact reduction. The central claim is that this weight-level, group-orbit-wide penalty outperforms sample-based regularisers in both task performance and training runtime.

Significance. The core idea—using the orthogonal projection onto the equivariant subspace as a regulariser—is attractive, theoretically clean for linear layers, and potentially efficient, especially for continuous groups where sample-based penalties are costly. The Fourier-domain projection (Section 3.2-3.5) is a useful technical contribution, and the composition bound in Corollary 3.4 connects per-layer penalties to global equivariance error. If the empirical validation were airtight, this would be a solid contribution to the approximate-equivariance literature. However, the main real-world experiment (CT-MAR) currently has an unresolved gap between the assumed steerable C4 action and the actual CNN architectures, which undermines the headline empirical claim. The theoretical statements also contain an overgeneralisation and a norm mismatch that need correction before the paper can be accepted.

major comments (3)
  1. [§4.3, Eq. (21), Table 2] The projection formula in Eq. (21) is derived for steerable CNN layers whose channels are organised into orientation groups of four (regular representation of C4). The paper applies this projection-based regulariser to ACDNet, DICDNet, and OSCNet, which are standard convolutional architectures (their parameter counts are unchanged by adding the regulariser). No mapping is given from an ordinary convolution kernel K∈R^{C_out×C_in×s×s} to the four-orientation tensor in Eq. (21), nor any justification that the orientation-channel permutation S^r corresponds to rotating a CT image. If the group action is not the true rotation action on these networks' feature spaces, the regulariser does not penalise violation of rotational equivariance, and the reported improvements in Table 2 do not support the paper's claims. This is a load-bearing gap in the main empirical validation; it must be fixed by
  2. [Lemma 3.2, §3.1, Appendix A.1] Lemma 3.2 is stated for 'every (Lipschitz) function T between G-modules with unitary actions', but the proof uses the operator/Lipschitz seminorm and the Reynolds projection P in Eq. (1). The orthogonal decomposition in Lemma 2.1 (from Elesedy & Zaidi) holds for a function space H that is a Hilbert space with respect to the Hilbert-Schmidt inner product; the space of Lipschitz functions is not such a space, and P need not be an orthogonal projection there. The proof in A.1 effectively treats P as a norm-1 averaging operator, which may not yield the claimed distance-to-equivariant-subspace property for nonlinear maps. The statement should be restricted to linear maps (which is all the paper actually needs for Corollary 3.4 and the regulariser), with the proof adapted accordingly. As stated, the lemma overgeneralises and misleads.
  3. [§3.1, Eq. (20), Appendix E] The theory bounds the worst-case equivariance defect E(T) using the operator (sup) norm and the same norm for ∥T−P(T)∥. The implemented regulariser, however, uses the Frobenius norm on layer weights, and the measured defect in Eq. (20) is a sample average over data points and group elements. The factor-of-2 bound in Lemma 3.2 does not directly transfer to these quantities without additional norm-equivalence or sampling assumptions. The ablation in Appendix E shows empirically that the Frobenius norm behaves reasonably, but it does not provide the missing theoretical link. Since the abstract claims 'penalises non-equivariance at an operator level', the paper should either implement the penalty in the operator norm, or state and prove an appropriate norm-equivalence result (including dimension dependence) that justifies the practical choice.
minor comments (5)
  1. [§1, 'decompostion' typo] Typo in the introduction: 'decompostion' should be 'decomposition'.
  2. [Figures 3, 4, 9] The empirical equivariance defect E_emp in Eq. (20) is essentially the quantity that the regulariser minimises (up to sampling and norm), so the decreasing defect in these figures largely reflects the training objective. It would be helpful to state this explicitly and to report held-out task performance alongside the defect, so that the figures are not read as independent evidence of generalisation.
  3. [Appendix A.4, Eq. (46)] The line 'dπλ= trAπ = =⇒λ= δπ,1 / dπ' contains a typo and is missing justification; please re-derive cleanly. Also, in Eq. (42) the substitution k=ghg^{-1} and the subsequent equality need a consistency check.
  4. [§3.5, Theorem 3.10] The notation 'V_π^* ⊗ V' is used for the multiplicity space; this is a bit nonstandard and could confuse readers. A short explanation of the decomposition of the fiber space under π^*⊗ρ would improve readability.
  5. [Table 2] The table's throughput figures and the text's statement about '42–47% lower' and '54–61%' gaps are not fully traceable to the displayed numbers, because the table does not show the max-feasible batch-12 throughput for the sample-based method. Please add the batch-size annotation and make the comparison explicit.

Circularity Check

1 steps flagged · score 2.0 of 10

No load-bearing circularity; one mild tautological evaluation metric in the toy SO(2) experiments.

  1. other [Section 4.1, Eq. (20), Figures 3/4 (also Appendix D, Figure 9)]
    "As the regularisation intensifies, the decision boundary becomes increasingly SO(2)-invariant, confirming that the proposed projection-based regulariser effectively pushes the model toward invariance. Consistently, the empirical equivariance defect E_emp(T) = Σ_{k,l} ||ρ_out(g_l)T(x_k)−T(ρ_in(g_l)x_k)|| (20) ... decreases from left to right."

    The objective minimized in this same section is L = L_task + λ_G Σ_i ||W_i|| + λ_⊥ Σ_i ||W_i − M⊙W_i||, while Lemma 3.2 shows E(T) ≤ 2||T−P(T)||. The reported E_emp is a sampled, data-averaged proxy for E(T), i.e. essentially the quantity the projection penalty controls. Reporting that E_emp decreases as λ_⊥ increases is therefore close to restating that the penalized term is being minimized; it is a sanity-check tautology rather than independent evidence. The central empirical claims, however, rest on PSNR/SSIM benchmarks that are independent of the penalty, so this is a minor evaluative circularity, not a load-bearing one.

full rationale

The core derivation is self-contained and non-circular. The projection operator P(T) is defined as the Haar-averaged Reynolds operator, and Lemma 3.2 is proved directly from that definition using unitarity and the triangle inequality; it does not assume the result it is used to justify. The Fourier/Peter–Weyl results (Theorems 3.8, 3.10) are classical representation-theoretic facts with proofs supplied in the appendix. No parameters are fitted to the quantities being predicted, and the method is benchmarked against external PSNR/SSIM targets. There are no load-bearing self-citations: the cited Elesedy & Zaidi lemma is standard, and the Kim et al. corollary is re-proved in the appendix. The CT-MAR concern raised by the skeptic is a correctness/implementation gap—the projection in Eq. (21) is derived for steerable C4 orientation-channel layers, but the paper does not specify how ordinary ACDNet/DICDNet/OSCNet kernels are mapped into that 4×4 orientation structure—but that is an external-validity issue, not circularity. The only mild circular element is the toy-experiment reporting of E_emp, which largely mirrors the regularizer being minimized; this does not infect the independent task metrics. Overall circularity is therefore low.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim relies on classical compact-group representation theory plus the orthogonal-projection decomposition of the Reynolds operator. The only new free parameters are the two regularizer weights; no new physical or mathematical entities are introduced.

free parameters (2)
  • λ_G (equivariant penalty weight) = 1.0 (CT-MAR; tuned over {1.0, 10^-1, ..., 10^-6})
    Balances preserving the equivariant component; chosen by hyperparameter sweep.
  • λ_⊥ (non-equivariant penalty weight) = varies per experiment (1.0 for toy; swept over {0, 10^-2, 10^-4, 10^-6} for smoke)
    Controls the strength of the equivariance bias; tuned per task, so it is a free parameter of the method as evaluated.
assumptions (3)
  • standard math Peter–Weyl theorem and Schur's lemma for compact groups
    Used in Theorems 3.8/3.10 and Appendix B to express L^2(G) as ⊕_π Vπ ⊗ Vπ* and to characterize intertwiners as identity on the representation factor.
  • standard math The Reynolds operator P is an orthogonal projection onto the equivariant subspace (Lemma 2.1, Elesedy & Zaidi 2021)
    Basis for the decomposition T = P(T) + (T − P(T)) and for using ∥T − P(T)∥ as the equivariance penalty.
  • ad hoc to paper Lemma 3.2 extends the projection/defect equivalence to general Lipschitz functions with unitary actions
    The proof uses operator-norm-style inequalities on Lipschitz seminorms and assumes H = S ⊕ A holds for nonlinear function classes; the paper applies the penalty only to linear layers, so this is an unverified generalization as stated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Approximate Equivariance via Projection-based Regularisation." pith.science (2026). https://pith.science/paper/3GTO2PNJ

@misc{pith2026260105028,
  author       = {Pith},
  title        = {Pith review of: Approximate Equivariance via Projection-based Regularisation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3GTO2PNJ}},
  note         = {Machine review of arXiv:2601.05028}
}
abstract

Equivariance is a powerful inductive bias in neural networks, improving generalisation and physical consistency. Recently, however, non-equivariant models have regained attention, due to their better runtime performance and imperfect symmetries that might arise in real-world applications. This has motivated the development of approximately equivariant models that strike a middle ground between respecting symmetries and fitting the data distribution. Existing approaches in this field usually apply sample-based regularisers which depend on data augmentation at training time, incurring a high sample complexity, in particular for continuous groups such as $SO(3)$. This work instead approaches approximate equivariance via a projection-based regulariser which leverages the orthogonal decomposition of linear layers into equivariant and non-equivariant components. In contrast to existing methods, this penalises non-equivariance at an operator level across the full group orbit, rather than point-wise. We present a mathematical framework for computing the non-equivariance penalty exactly and efficiently in both the spatial and spectral domain. In our experiments, our method consistently outperforms prior approximate equivariance approaches in both model performance and efficiency, achieving substantial runtime gains over sample-based regularisers.

Figures

Figures reproduced from arXiv: 2601.05028 by the authors.

Figure 2
Figure 2. Commutative diagrams showing how to apply the projection operator in Fourier space. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Controlling the degree of learned SO(2) invariance by tuning the parameters λG and λ⊥, which penalise the projections of the equivariant and non-equivariant components, respectively. 4.1 EXAMPLE: LEARNED SO(2) INVARIANCE We first want to illustrate the approach in Section 3 on a simple toy problem ( [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Effect of increasing angular perturbation at fixed projection strength. Each panel shows the decision bound [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Effect of angular perturbations and projection strength. Columns vary the angular wave amplitude [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: Qualitative comparison of the baseline methods (left column) with each of the sample-based (middle column) [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]
Figure 7
Figure 7. Figure 7: Controlling the degree of learned SO(2) invariance by varying the values of λG and λ⊥ over the grid {0, 0.001, 0.01, 0.1}. Index-wise, for any (p, q, α, β, i, j), this reads [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]
Figure 8
Figure 8. Figure 8: Classification accuracy on the CIFAR and MNIST datasets for models trained with varying values of [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: Equivariance defect for models trained on the CIFAR and MNIST classification tasks for varying values of [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

4 extracted references · 2 linked inside Pith

  1. [2022]

    10 Preprint

    URLhttps://openreview.net/forum?id=_xwr8gOBeV1. 10 Preprint. Under review. Johann Brehmer, Sönke Behrends, Pim De Haan, and Taco Cohen. Does equivariance matter at scale? InNeurIPS 2024 Workshop on Symmetry and Geometry in Neural Representations, 2024. URLhttps://openreview.net/ forum?id=L4gb2wvVhM. Johann Brehmer, Víctor Bresó, Pim de Haan, Tilman Plehn,...

    arXiv 2024

  2. [2023]

    Hyunsu Kim, Hyungi Lee, Hongseok Yang, and Juho Lee

    Curran Associates Inc. Hyunsu Kim, Hyungi Lee, Hongseok Yang, and Juho Lee. Regularizing Towards Soft Equivariance Under Mixed Symmetries, 2023. URLhttps://arxiv.org/abs/2306.00356. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014. Theodoros Kouzelis, Ioannis Kakogeorgiou, Spyros Gidaris, and ...

    arXiv 2023

  3. [2024]

    as radially symmetric as possible

    URLhttps://openreview.net/forum?id=mCOBKZmrzD. Daniel McNeela. Almost equivariance via lie algebra convolutions, 2024. URLhttps://arxiv.org/abs/2310. 13164. Samuel E Otto, Nicholas Zolman, J Nathan Kutz, and Steven L Brunton. A unified framework to enforce, discover, and promote symmetry in machine learning.arXiv preprint arXiv:2311.00212, 2023. Stefanos ...

    arXiv 2024

  4. [2025]

    Elyssa Hofgard, Rui Wang, Robin Walters, and Tess Smidt

    URLhttps://openreview.net/forum?id=7BLXhmWvwF. Elyssa Hofgard, Rui Wang, Robin Walters, and Tess Smidt. Relaxed equivariant graph neural networks.ELLIS Workshop on Geometry-grounded Representation Learning and Generative Modeling, ICML, 2024. Philipp Holl and Nils Thuerey.Φ flow (PhiFlow): Differentiable simulations for pytorch, tensorflow and jax. InInte...

    2024

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.