Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architectures

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A kernel-composition scheme builds orthogonal convolutions that natively support stride, transposition, groups, and dilation.

desk verdict AOC is a genuinely useful orthogonal-convolution construction with native stride and other modern features, proven under circular padding; the zero-padding accuracy setting is only quasi-orthogonal, and the overhead numbers are slightly oversold. read the letter →

arxiv 2501.07930 v3 pith:DYKFZ5EJ submitted 2025-01-14 cs.AI cs.NE

Thibaut Boissin (IRIT) , Franck Mamalet , Thomas Fel , Agustin Martin Picard , Thomas Massena (IRIT) , Mathieu Serrurier (IRIT) This is my paper · ORCID
classification cs.AIcs.NE MSC 68T07
keywords orthogonalconvolution1-Lipschitznetworkscertifiablerobustnessstridedtransposedgroupeddilatedblock-convolution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Orthogonal convolutions — layers whose underlying linear map preserves norms — are valuable for certified adversarial robustness, normalizing flows, GANs, and stable gradient propagation, but existing constructions do not support the features modern networks actually use: strides, transposed upsampling, grouping, and dilation. This paper introduces AOC, which composes two existing kernel constructions — BCOP, which builds orthogonal kernels of any size at stride one, and RKO, which the paper proves is orthogonal when kernel size equals stride — using the block-convolution operator $\circledast$. The paper claims that with the internal channel count set to $c = \max(c_i, \lfloor c_o/s^2 \rfloor)$, the fused kernel is a strict orthogonal convolution for any valid stride, and that the same recipe extends to transposed, grouped, and dilated convolutions. If this holds, orthogonal layers can be inserted into U-Nets, GANs, and ImageNet-scale training with roughly 13% overhead, rather than being confined to small-scale demonstrations.

What carries the argument

The load-bearing mechanism is the block-convolution operator $\circledast$, which computes the equivalent kernel of two composed convolutions without executing either one: $(B \circledast A) \star x = B \star (A \star x)$, producing a kernel of size $k_A + k_B - 1$. AOC pairs a BCOP factor — orthogonal at stride one for any kernel size, built from stacked symmetric projectors $N$ and $I - N$ — with an RKO factor, orthogonal precisely when its kernel size equals its stride, a fact the paper proves by reordering the strided convolution into a permutation matrix followed by a matrix product. The internal channel dimension $c = \max(c_i, \lfloor c_o/s^2 \rfloor)$ is what aligns the two factors' orthogonality types, and an efficient implementation (batched block-convolution and a parallel associative scan) is what makes the construction cheap enough for large images.

What would settle it

Build the Toeplitz matrix of a trained AOC layer on a small image ($8\times8$) using zero padding and compute its singular values: any singular value departing from 1 by more than $10^{-4}$ would falsify exact orthogonality in deployed settings. A complementary check is to verify the reported 'AOC accurate' result directly — the paper itself reports 0% certified accuracy at radius $36/255$, which a padding-aware bound should reproduce.

Watch

Extended reading notes

Core claim

The paper's central claim is that the kernel $K_{\mathrm{AOC}} = K_{\mathrm{RKO}} \circledast K_{\mathrm{BCOP}}$, with internal channel count $c = \max(c_i, \lfloor c_o / s^2 \rfloor)$, is an orthogonal convolution with stride $s$ and kernel size $k$. The construction fuses a BCOP kernel — built by stacking symmetric projectors into $1\times2$ and $2\times1$ orthogonal convolutions and composing them — with an RKO kernel, a kernel reshaped from an orthogonal matrix, which this paper proves to be orthogonal exactly when its kernel size equals its stride. The choice of $c$ ensures that both factors are row-orthogonal or both column-orthogonal, so the composition inherits orthogonality from the corresponding property of orthogonal matrices. From this core result the paper derives native transposed orthogonal convolutions (the transpose of an orthogonal convolution is again orthogonal), grouped orthogonal convolutions (the Toeplitz matrix is block diagonal), and dilated orthogonal convolutions, and it notes that the covered parameter range $s \le k$ is exhaustive because no orthogonal kernel exists when the stride exceeds the kernel size.

Load-bearing premise

The proofs assume circular padding at the image boundary, so with the standard zero padding used in most architectures the layer is only quasi-orthogonal along the borders, and the exact norm preservation that tight robustness certificates rely on is not guaranteed.

Editorial extensions

If this is right

  • Native strided orthogonal convolutions replace stride emulation through input reshaping, cutting the parameter count of strided layers by a factor of $s^2$ and removing the cubic-cost blowup of emulated methods.
  • Transposed orthogonal convolutions become available for upsampling, so U-Nets, VAEs, and invertible normalizing-flow layers can be built from norm-preserving components.
  • Grouped and dilated orthogonal convolutions let depthwise and large-receptive-field architectures stay inside the 1-Lipschitz framework where robustness certificates are tight.
  • The reported scaling behaviour — the overhead falls as batch size grows, reaching about 13% on ImageNet-scale ResNet-34 — puts certifiably robust training on large datasets within a practical compute budget.
  • Because the covered configurations include every case where an orthogonal kernel can exist (stride no larger than kernel size), the method is a full coverage of the feasible parameter space rather than a partial patch.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One consequence the authors leave implicit: with the circular-padding assumption in the proofs, any zero-padding deployment inherits a border effect, so certificates should be computed with a padding-aware Lipschitz bound rather than trusting exact orthogonality; the paper's own high-accuracy run reports 0% provable accuracy at radius $36/255$.
  • If the scalability numbers hold, the practical blocker for certified robustness at ImageNet scale shifts from layer cost to data and training budget, which are the constraints already identified by the robust-training literature.
  • A natural follow-up would be to swap the BCOP factor for a complete parametrization of orthogonal convolutions (BCOP provably misses some of them); the same $\circledast$ machinery would then add striding to a strictly more expressive base class.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes AOC (Adaptive Orthogonal Convolution), a construction that composes a BCOP kernel with an RKO kernel through the block-convolution operator to obtain convolutional layers that are orthogonal while supporting native stride, transposition, grouping, and dilation. The main theoretical claims are Propositions 2.4, 2.5, 2.7, and 2.8, with proofs in Appendix G. The paper also contributes an efficient implementation (the Orthogonium package) and evaluates AOC on CIFAR-10 and ImageNet-1K for certified robustness and on a ResNet-34 scalability benchmark. The authors report that AOC achieves competitive robustness certificates in the circular-padding setting and much lower training overhead than prior orthogonal convolution methods at scale.

Significance. If the orthogonality claims hold, AOC is a practically valuable contribution: it is, to my knowledge, the first explicit construction that natively supports strided, transposed, grouped, and dilated orthogonal convolutions, and it comes with an open-source implementation and an unusually extensive unit-test suite (1442 tests reported in Appendix D). The proofs in Appendix G are mostly clear and are stated under explicit assumptions. The main caveat is that the orthogonality proofs assume circular padding, whereas one of the two experimental configurations uses zero padding and is only quasi-orthogonal; this boundary issue is acknowledged in Appendix H but is not reflected in the abstract and introduction, where the method is presented as delivering strict orthogonality for standard CNN use. The scalability results are encouraging but are reported without statistical variation and with a small numerical inconsistency against the abstract.

major comments (4)
  1. [Section 2.1, Appendix H, Table 2] The orthogonality proofs are explicitly conditioned on circular padding (Section 2.1: 'we assume circular padding in all proofs'), yet the 'AOC accurate' configuration in Appendix H uses zero padding and is described as '1-Lipschitz and quasi-orthogonal (orthogonal everywhere except for the images border)'. Under zero padding the strict orthogonality used in Proposition 2.3 does not hold, and Table 2 shows the consequence: 'AOC accurate' attains 91.5% clean accuracy but 0% provable accuracy at radius 36/255. The abstract and introduction nevertheless present AOC as providing strictly orthogonal layers for standard CNN architectures. This is a load-bearing scope mismatch that must be fixed by either restricting the orthogonality and certification claims to the circular-padding setting, adding a complete treatment of the zero-padding boundary, or clearly repositioning the zero-padding experiments as quasi-orthogonal rather than orthogonal.
  2. [Abstract and Section 3.2 / Table 3] The abstract states that AOC incurs 'only a 10% slowdown compared to unconstrained models in realistic Imagnet 1k training setup', but Section 3.2 reports 'a training time only 13% slower than its unconstrained counterpart' and Table 3 lists 1.75x at batch 128 and 1.13x at batch 512. The numbers should be made consistent, and the exact configuration (batch size, architecture, hardware) behind the advertised percentage should be stated in the abstract or immediately in Section 3.2.
  3. [Table 2 and Appendix H] The robustness comparison in Table 2 mixes numbers quoted from other papers that use different architectures, training recipes, loss parameters, and data augmentation (see Appendix H, Table 4), and the 'AOC robust*' row uses extra data while other rows are marked with an asterisk only in the footnote. The comparison is therefore indicative rather than controlled. I recommend reporting per-configuration details in the table caption, adding error bars or multiple seeds for the AOC rows, and marking which rows use extra data directly in the table.
  4. [Appendix G.5] The proof of Proposition 2.5 chooses the internal channel c = max(ci, floor(co/s^2)), but the inequalities in the proof are stated in terms of co/s^2 rather than floor(co/s^2). The argument is valid in the integer cases shown, but the proof should explicitly handle the non-integer case to justify that floor(co/s^2) always satisfies the required row- or column-orthogonality condition. This is a small but load-bearing gap in the central construction.
minor comments (4)
  1. [Throughout] There are several typos and infelicities, including 'Adaptative' in the title, 'Imagnet 1k' in the abstract, 'smae setup' in Appendix C.2, and 'we evaluate' beginning a sentence with a lowercase letter in the caption of Table 2. These should be corrected in a final pass.
  2. [Table 3] The scalability table reports single timing measurements without error bars or repeated trials. Since the central scalability claim depends on small overhead differences, at least a few repeated runs with standard deviations should be reported for the headline batch sizes.
  3. [Section 3.1 / Observation 1] The sentence 'This comes at the cost of 0% provable accuracy (certificates smaller than the radius ϵ = 36/255)' is accurate, but the text should explicitly connect this to the quasi-orthogonality under zero padding rather than presenting it only as an accuracy-robustness trade-off.
  4. [Appendix G.4] The RKO proof assumes that the input height and width are multiples of the stride and that the input is partitioned into non-overlapping blocks. This assumption should be stated in the proposition itself, not only in the proof, so readers know the exact domain of validity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the AOC orthogonality result is proved from the Toeplitz/block-convolution algebra rather than assumed or fitted, and the self-citations are not load-bearing.

full rationale

The central claim (Proposition 2.5, Appendix G.5) derives orthogonality of KAOC = KRKO ? KBCOP by direct matrix algebra: after fixing c = max(ci, floor(co/s^2)), the two factors are shown to be both row- or both column-orthogonal, so Proposition 2.3 applies. RKO's orthogonality for k = s is proved in the paper via a permutation argument (Appendix G.4), and BCOP orthogonality is reproduced from Li et al. with proof in Appendix G.3; neither is assumed as a black box. Transposed, grouped, and dilated cases follow from the same Toeplitz/block-diagonal structure (G.6-G.8). The evaluation uses external benchmarks and comparisons, so no fitted parameter is renamed as a prediction. Self-citations exist (RKO originates in Serrurier et al. 2021, with overlapping authors, and the s > k impossibility is attributed to Achour et al. 2022, co-authored by Mamalet), but these are used as provenance or as external theorems, not as evidence for the key k = s orthogonality step, which is proved in this paper. Appendix H does acknowledge that the zero-padding 'AOC accurate' setting is only quasi-orthogonal and achieves 0% provable accuracy at radius 36/255; this is a clearly stated limitation of the experimental setup, not a circular derivation. The orthogonality proofs are conditional on circular padding, and that condition is explicit, so the derivation is self-contained rather than circular.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The method relies on standard linear algebra of convolutions as Toeplitz operators and on the block-convolution operator; the main domain assumption is circular padding in the proofs, and the BCOP parametrization is known to be incomplete, so AOC inherits a restricted parametrization of the orthogonal-convolution manifold.

assumptions (4)
  • standard math The block-convolution operator ⊛ correctly represents composition of convolutions and is associative.
    Used throughout Section 2 and Appendix G.1-G.2; it is a standard result for Toeplitz/convolution composition.
  • domain assumption Convolution is represented as a Toeplitz matrix and orthogonality is defined via that matrix, assuming circular padding.
    Explicitly stated in Section 2.1: 'we assume circular padding in all proofs'. The experimental zero-padding case is only quasi-orthogonal.
  • domain assumption The BCOP parametrization is incomplete, and AOC inherits this limitation.
    Appendix F discusses the known counterexample for 2x2 orthogonal convolutions and states that BCOP does not span the full orthogonal-convolution manifold.
  • domain assumption No orthogonal convolution exists when stride s > kernel size k.
    Cited from Achour et al. 2022 in Section 2.2 to justify the covered range of configurations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architectures." pith.science (2026). https://pith.science/paper/DYKFZ5EJ

@misc{pith2026250107930,
  author       = {Pith},
  title        = {Pith review of: An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architectures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DYKFZ5EJ}},
  note         = {Machine review of arXiv:2501.07930}
}
read the original abstract

Orthogonal convolutional layers are valuable components in multiple areas of machine learning, such as adversarial robustness, normalizing flows, GANs, and Lipschitz-constrained models. Their ability to preserve norms and ensure stable gradient propagation makes them valuable for a large range of problems. Despite their promise, the deployment of orthogonal convolution in large-scale applications is a significant challenge due to computational overhead and limited support for modern features like strides, dilations, group convolutions, and transposed convolutions. In this paper, we introduce AOC (Adaptative Orthogonal Convolution), a scalable method that extends a previous method (BCOP), effectively overcoming existing limitations in the construction of orthogonal convolutions. This advancement unlocks the construction of architectures that were previously considered impractical. We demonstrate through our experiments that our method produces expressive models that become increasingly efficient as they scale. To foster further advancement, we provide an open-source python package implementing this method, called Orthogonium ( https://github.com/deel-ai/orthogonium ) .

Figures

Figures reproduced from arXiv: 2501.07930 by the authors.

Figure 1
Figure 1. AOC enables the construction of orthogonal kernels with customizable sizes and strides. By leveraging the ⊛ operator, we can fuse kernels obtained from two existing methods, namely BCOP and RKO. With this approach, we can build orthogonal kernels that support native striding, effectively mitigating the drawbacks of the two base methods. convolution. Unlike native striding, emulation increases the number of parameter… view at source ↗
Figure 2
Figure 2. Fast implementation of AOC. We achieve a highly scalable parametrization thanks to optimizations at every level of our method: starting from the ⊛ operator 2a, to BCOP 2b, to our complete method 2c. It results in a method with a lower overhead as scale increases. For a given convolution with stride defined by Eq. (2), the transposed convolution corresponds to the application of the transposed matrix (SsK) T , invert… view at source ↗
Figure 3
Figure 3. Impact of the orthogonalization method on AOC: in the smae setup as (Li et al., 2019) we observe that the [PITH_FULL_IMAGE:figures/full_fig_p020_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The ⊛ can be used to enable s ̸= 1 and ci ̸= co configurations on SLL. The flexibility of the ⊛ allows for operations resulting in a block with a similar structure as the original ResNet block. SLL layer for convolutions, proposed in (Araujo et al., 2023), is a 1-Lipsc…
Figure 5
Figure 5. Figure 5: Architecture used for CIFAR-10 accurate setting. This architecture makes use of its flexibility to allow performance with a limited parameter count. Left, table of the global network architecture. Right, detail of he residual depthwise block. All hyperparameters of the…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Distributed Retraction-Free and Communication-Efficient Optimization on the Stiefel Manifold

    math.OC 2025-06 conditional novelty 6.0 of 10

    EF-Landing provably converges at O(1/sqrt(N K)) for distributed stochastic problems on the Stiefel manifold while using compressed communication and no retraction.

  2. HOFT: Householder Orthogonal Fine-tuning

    cs.LG 2025-05 conditional novelty 5.0 of 10

    HOFT and SHOFT fine-tune foundation models with two Householder-built orthogonal matrices, matching or beating LoRA, DoRA, OFT, BOFT and HRA on reasoning, translation, image generation and math.

Reference graph

Works this paper leans on

13 extracted references · 11 canonical work pages · cited by 2 Pith papers

  1. [1]

    Checking of numerical Instabilities: Issues arising from floating-point precision, such as those introduced by small epsilon values added to avoid division by zero

    work page

  2. [2]

    Checking the orthogonality of a layer under stride, group, transposition, and dilation conditions

    Checking for implementation discrepancies: Differences between mathematical formalism and its translation to popular frameworks (e.g., SOC proofs assume circular padding, while its implementation uses zero padding). Checking the orthogonality of a layer under stride, group, transposition, and dilation conditions. The numerical stability and the convergenc...

    work page

  3. [3]

    increase the margin, increase the model size and number of epochs until it reaches 100% training accuracy

    work page

  4. [4]

    repeat steps 2. and 3. until the desired robustness is achieved. The target margin is m = 3 2 √ 2 to match the parameters used by (Araujo et al., 2023). In practical contexts, m should be set to m = ϵ √ 2 αL where ϵ is the targeted robustness radius, L the model’s Lipschitz constant and α the scaling factor applied on the data (for instance when standard ...

    work page 2023

  5. [7]

    This method, though accurate, is computationally expensive for large input images

    Explicit SVD on Toeplitz Matrices: Using the impulse response approach, we construct the Toeplitz matrix for any padding and stride, allowing direct computation of singular values. This method, though accurate, is computationally expensive for large input images

    work page

  6. [8]

    Acc.” refers to the high-accuracy setting, and “Rob

    Product Bound for BCOP and RKO Kernels: The upper bound for the BCOP kernel is computed using standard methods, while the SVD of the reshaped RKO kernel is used for direct evaluation. Unit testing of the implementation. We used both of these two approaches in our unit tests. This enables us to ensure that the second method (which is faster and more scalab...

    work page 2022

  7. [9]

    start with a given architecture and a low margin m = 0

    work page

  8. [10]

    The model reached its maximum generalization for this margin

    increase data augmentation until train accuracy goes below 100%. The model reached its maximum generalization for this margin

    work page

Show all 13 references
  1. [456]

    Kiani, B., Balestriero, R., LeCun, Y ., and Lloyd, S

    pmlr, 2015. Kiani, B., Balestriero, R., LeCun, Y ., and Lloyd, S. projunn: efficient method for training deep networks with unitary matrices. Advances in Neural Information Processing Systems, 35:14448–14463, 2022. Kingma, D. P. Auto-encoding variational bayes. arXiv preprint ...

    2015 arXiv

  2. [582]

    B´ethune, L

    PMLR, 2019. B´ethune, L. Deep learning with Lipschitz constraints. PhD thesis, Universit´e de Toulouse, 2024. 9 An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architectures B´ethune, L., Boissin, T., Serrurier, M., Mamalet, F., Friedrich, C., and Gonz...

    2019 arXiv

  3. [1820]

    Li, Q., Haque, S., Anil, C., Lucas, J., Grosse, R

    URL http://eudml.org/doc/203444. Li, Q., Haque, S., Anil, C., Lucas, J., Grosse, R. B., and Ja- cobsen, J.-H. Preventing gradient attenuation in lipschitz constrained convolutional networks. Advances in neural information processing systems, 32, 2019. Li, Y ., Zhang, X., and C...

    2019 arXiv

  4. [2022]

    Although already proven by previous work, proof of this can be found in Appendix G.3

    noted that additional constraints are needed, proposing a half-rank symmetric projector to construct a1 × 2 orthogonal convolution: from a column-orthogonal matrix M ∈ Rc× c 2 5, the matrix N = M MT ∈ Rc×c is a symmetric projector that satisfies: N = N 2 = N T and (I − N ) = (...

    2022

  5. [2023]

    The residual connections were removed, and a constant value of 1.15 replaced the scalar factors

    with the original convolutions replaced by AOC convolutions (which include downsampling convolutions). The residual connections were removed, and a constant value of 1.15 replaced the scalar factors. This resulted in a network with a tighter Lipschitz constant evaluation, whic...

    2009

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.