{"id":"577d037e-1895-4e22-84da-5e9589c61e48","arxiv_id":"2501.07930","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Combining BCOP and RKO kernels with a channel-sizing rule yields orthogonal convolutions with native stride, transpose, group, and dilation support.","lead":"This paper introduces AOC, a method for building orthogonal convolutional layers that natively support striding, transposed convolutions, grouping, and dilation, features earlier orthogonal-convolution methods only emulated or lacked. Training overhead at large batch sizes is about 13 percent versus a standard convolution, making orthogonal layers practical for ImageNet-scale models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The AOC orthogonality proof assumes circular padding (Sec. 2.1); under the zero padding used in the 'AOC accurate' experiments, the layer is only quasi-orthogonal and yields 0% provable accuracy (Table 2, App. H), so the headline guarantee does not transfer to default CNN use.","rationale":"The reader's verdict is CONDITIONAL and identifies the circular-padding assumption as the weakest point; I agree. In good faith, the core composition proof is clean: Proposition 2.4 (RKO with k=s) uses a non-overlapping patch argument, Proposition 2.5 chooses the internal channel count so that KBCOP and SsKRKO are both row- or both column-orthogonal, and the block-convolution algebra is standard. I do not see an internal inconsistency in the circular-padding version of the claim, and the open-source implementation plus the 1442 unit tests are real supporting evidence. The load-bearing weakness is external: the guarantee is stated for circular padding, while standard CNN practice and the paper's own high-accuracy experiments use zero padding. The paper's Table 2 demonstrates the practical stakes (0% provable accuracy for the zero-padding 'accurate' model), and Appendix H explicitly concedes quasi-orthogonality at borders. This does not refute the central mathematical claim, but it narrows its applicability substantially, so the paper should either state the circular-padding requirement prominently or provide zero-padding certificates. The limitation is present in the manuscript itself, so this is not an invented objection; it is a matter of emphasis and scope. I therefore recommend leaving the reader's CONDITIONAL verdict unchanged.","tokens_in":33374,"tokens_out":21247,"duration_ms":226418,"concrete_test":"Construct the exact Toeplitz matrix by impulse responses for an AOC layer with, say, ci=64, co=128, k=3, s=2 on a 32x32 input, and compute the orthogonality deviation (max singular value deviation from 1, or the appropriate residual norm such as ||(SsK)(SsK)^T - I|| for the row case) under zero padding and under circular padding. If the zero-padding deviation is materially above machine precision (e.g., >1e-3) while circular padding is near machine precision, the orthogonality guarantee is padding-dependent. Then retrain the 'AOC accurate' CIFAR-10 architecture from Appendix H with circular padding and the same hyperparameters; if provable accuracy at 36/255 moves from 0% to a positive value, the zero-padding quasi-orthogonality is confirmed as the cause of the certificate collapse.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that KAOC = KRKO ⊛ KBCOP is an orthogonal convolution with stride and that the construction extends to transposed, grouped, and dilated convolutions. The load-bearing condition is that both factors are orthogonal in the padding actually used. Section 2.1 states 'we assume circular padding in all proofs,' and Appendix H confirms that the 'AOC accurate' configuration uses zero padding, where the layer is '1-Lipschitz and quasi-orthogonal (orthogonal everywhere except for the images border).' Under zero padding, the BCOP factor does not satisfy the strict orthogonality needed for Proposition 2.3, so norm preservation, tightness of the 1-Lipschitz product bound, and the certified-robustness guarantee all degrade. Table 2 shows the practical consequence: 'AOC accurate' reaches 91.5% clean accuracy but 0% provable accuracy at radius 36/255. The limitation is acknowledged in the appendix but underemphasized: the abstract and introduction frame AOC as delivering orthogonal layers for adversarial robustness, while only the circular-padding 'robust' setting is fully orthogonal. The mathematical construction may be sound under circular padding, but as a drop-in replacement for standard CNNs, which use zero padding, the advertised property does not hold.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AOC (Adaptive Orthogonal Convolution), a construction that composes a BCOP kernel with an RKO kernel through the block-convolution operator to obtain convolutional layers that are orthogonal while supporting native stride, transposition, grouping, and dilation. The main theoretical claims are Propositions 2.4, 2.5, 2.7, and 2.8, with proofs in Appendix G. The paper also contributes an efficient implementation (the Orthogonium package) and evaluates AOC on CIFAR-10 and ImageNet-1K for certified robustness and on a ResNet-34 scalability benchmark. The authors report that AOC achieves competitive robustness certificates in the circular-padding setting and much lower training overhead than prior orthogonal convolution methods at scale.","tokens_in":33644,"tokens_out":3656,"duration_ms":39734,"significance":"If the orthogonality claims hold, AOC is a practically valuable contribution: it is, to my knowledge, the first explicit construction that natively supports strided, transposed, grouped, and dilated orthogonal convolutions, and it comes with an open-source implementation and an unusually extensive unit-test suite (1442 tests reported in Appendix D). The proofs in Appendix G are mostly clear and are stated under explicit assumptions. The main caveat is that the orthogonality proofs assume circular padding, whereas one of the two experimental configurations uses zero padding and is only quasi-orthogonal; this boundary issue is acknowledged in Appendix H but is not reflected in the abstract and introduction, where the method is presented as delivering strict orthogonality for standard CNN use. The scalability results are encouraging but are reported without statistical variation and with a small numerical inconsistency against the abstract.","major_comments":[{"comment":"The orthogonality proofs are explicitly conditioned on circular padding (Section 2.1: 'we assume circular padding in all proofs'), yet the 'AOC accurate' configuration in Appendix H uses zero padding and is described as '1-Lipschitz and quasi-orthogonal (orthogonal everywhere except for the images border)'. Under zero padding the strict orthogonality used in Proposition 2.3 does not hold, and Table 2 shows the consequence: 'AOC accurate' attains 91.5% clean accuracy but 0% provable accuracy at radius 36/255. The abstract and introduction nevertheless present AOC as providing strictly orthogonal layers for standard CNN architectures. This is a load-bearing scope mismatch that must be fixed by either restricting the orthogonality and certification claims to the circular-padding setting, adding a complete treatment of the zero-padding boundary, or clearly repositioning the zero-padding experiments as quasi-orthogonal rather than orthogonal.","section":"Section 2.1, Appendix H, Table 2"},{"comment":"The abstract states that AOC incurs 'only a 10% slowdown compared to unconstrained models in realistic Imagnet 1k training setup', but Section 3.2 reports 'a training time only 13% slower than its unconstrained counterpart' and Table 3 lists 1.75x at batch 128 and 1.13x at batch 512. The numbers should be made consistent, and the exact configuration (batch size, architecture, hardware) behind the advertised percentage should be stated in the abstract or immediately in Section 3.2.","section":"Abstract and Section 3.2 / Table 3"},{"comment":"The robustness comparison in Table 2 mixes numbers quoted from other papers that use different architectures, training recipes, loss parameters, and data augmentation (see Appendix H, Table 4), and the 'AOC robust*' row uses extra data while other rows are marked with an asterisk only in the footnote. The comparison is therefore indicative rather than controlled. I recommend reporting per-configuration details in the table caption, adding error bars or multiple seeds for the AOC rows, and marking which rows use extra data directly in the table.","section":"Table 2 and Appendix H"},{"comment":"The proof of Proposition 2.5 chooses the internal channel c = max(ci, floor(co/s^2)), but the inequalities in the proof are stated in terms of co/s^2 rather than floor(co/s^2). The argument is valid in the integer cases shown, but the proof should explicitly handle the non-integer case to justify that floor(co/s^2) always satisfies the required row- or column-orthogonality condition. This is a small but load-bearing gap in the central construction.","section":"Appendix G.5"}],"minor_comments":[{"comment":"There are several typos and infelicities, including 'Adaptative' in the title, 'Imagnet 1k' in the abstract, 'smae setup' in Appendix C.2, and 'we evaluate' beginning a sentence with a lowercase letter in the caption of Table 2. These should be corrected in a final pass.","section":"Throughout"},{"comment":"The scalability table reports single timing measurements without error bars or repeated trials. Since the central scalability claim depends on small overhead differences, at least a few repeated runs with standard deviations should be reported for the headline batch sizes.","section":"Table 3"},{"comment":"The sentence 'This comes at the cost of 0% provable accuracy (certificates smaller than the radius ϵ = 36/255)' is accurate, but the text should explicitly connect this to the quasi-orthogonality under zero padding rather than presenting it only as an accuracy-robustness trade-off.","section":"Section 3.1 / Observation 1"},{"comment":"The RKO proof assumes that the input height and width are multiples of the stride and that the input is partitioned into non-overlapping blocks. This assumption should be stated in the proposition itself, not only in the proof, so readers know the exact domain of validity.","section":"Appendix G.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid engineering contribution with a mostly clear theoretical core, and the open-source package and unit-testing effort are commendable. My main concern for the editor is the gap between the advertised 'strict orthogonality' and the actual experimental setting: the zero-padding 'accurate' models are only quasi-orthogonal, and the paper's own Table 2 shows 0% provable accuracy for them. This is fixable by reframing the claims and making the padding assumption prominent, but it is not merely a presentation issue because it affects the headline robustness guarantee. The comparison-table issues are secondary but should be tightened before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a good paper and you should know about it. It shows you can get orthogonal convolutions with native stride, transposed, grouped, and dilated variants by composing BCOP kernels with RKO kernels via block-convolution. The new bits are Prop 2.4 (RKO is orthogonal when kernel size equals stride) and the channel-sizing rule Prop 2.5. Both are proved cleanly in the appendix, and the implementation is serious: fast block-convolution via grouped conv, parallel scan for BCOP iteration, and an open-source library with 1442 unit tests. The ImageNet scaling experiment is a real step beyond what existing methods show, and the overhead drops with batch size as claimed, though the exact number depends on batch size (13% at 512, 75% at 128).\n\nThe main soft spot is the padding story. Orthogonality is proven under circular padding; Section 2.1 says so plainly. Under zero padding, the layer is only quasi-orthogonal. The 'AOC accurate' model in Table 2 uses zero padding and gets 0% provable accuracy at 36/255, which is consistent but easily misread as a failure of the method. The paper does flag this in Appendix H, but the abstract and intro lead with adversarial robustness benefits, so the caveat could be more prominent. The construction itself is sound under circular padding, and the 'robust' models use it.\n\nOther small issues: Table 2 compares numbers from other papers under different training recipes, so treat it as indicative. Table 3 has no error bars; timings can be noisy. Also, the abstract says '10% slowdown' while Section 3.2 says '13% slower' for batch 512—minor inconsistency but worth fixing.\n\nI'd send this to review. The central construction is correct, the code is public, and the limitations, while underemphasized, are stated rather than hidden. A revision that moves the padding caveat into the abstract and aligns the overhead claims would make it stronger. This is a paper worth a serious referee, and I'd expect it to be cited by anyone working on Lipschitz or orthogonal layers.","headline":"AOC is a genuinely useful orthogonal-convolution construction with native stride and other modern features, proven under circular padding; the zero-padding accuracy setting is only quasi-orthogonal, and the overhead numbers are slightly oversold.","tokens_in":34210,"tokens_out":2822,"would_cite":true,"duration_ms":27266,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"A kernel-composition scheme builds orthogonal convolutions that natively support stride, transposition, groups, and dilation.","keywords":["orthogonal convolution","1-Lipschitz networks","certifiable robustness","strided convolution","transposed convolution","grouped convolution","dilated convolution","block-convolution"],"falsifier":"Build the Toeplitz matrix of a trained AOC layer on a small image ($8\\times8$) using zero padding and compute its singular values: any singular value departing from 1 by more than $10^{-4}$ would falsify exact orthogonality in deployed settings. A complementary check is to verify the reported 'AOC accurate' result directly — the paper itself reports 0% certified accuracy at radius $36/255$, which a padding-aware bound should reproduce.","tokens_in":33203,"feed_emoji":"🧩","tokens_out":10558,"duration_ms":86678,"temperature":0.7,"pith_summary":"Orthogonal convolutions — layers whose underlying linear map preserves norms — are valuable for certified adversarial robustness, normalizing flows, GANs, and stable gradient propagation, but existing constructions do not support the features modern networks actually use: strides, transposed upsampling, grouping, and dilation. This paper introduces AOC, which composes two existing kernel constructions — BCOP, which builds orthogonal kernels of any size at stride one, and RKO, which the paper proves is orthogonal when kernel size equals stride — using the block-convolution operator $\\circledast$. The paper claims that with the internal channel count set to $c = \\max(c_i, \\lfloor c_o/s^2 \\rfloor)$, the fused kernel is a strict orthogonal convolution for any valid stride, and that the same recipe extends to transposed, grouped, and dilated convolutions. If this holds, orthogonal layers can be inserted into U-Nets, GANs, and ImageNet-scale training with roughly 13% overhead, rather than being confined to small-scale demonstrations.","feed_headline":"Orthogonal convolutions gain native stride, groups, dilation","feed_subtitle":"Two kernel families compose to preserve norms while supporting stride, groups, dilation, and upsampling at scale.","key_machinery":"The load-bearing mechanism is the block-convolution operator $\\circledast$, which computes the equivalent kernel of two composed convolutions without executing either one: $(B \\circledast A) \\star x = B \\star (A \\star x)$, producing a kernel of size $k_A + k_B - 1$. AOC pairs a BCOP factor — orthogonal at stride one for any kernel size, built from stacked symmetric projectors $N$ and $I - N$ — with an RKO factor, orthogonal precisely when its kernel size equals its stride, a fact the paper proves by reordering the strided convolution into a permutation matrix followed by a matrix product. The internal channel dimension $c = \\max(c_i, \\lfloor c_o/s^2 \\rfloor)$ is what aligns the two factors' orthogonality types, and an efficient implementation (batched block-convolution and a parallel associative scan) is what makes the construction cheap enough for large images.","core_discovery":"The paper's central claim is that the kernel $K_{\\mathrm{AOC}} = K_{\\mathrm{RKO}} \\circledast K_{\\mathrm{BCOP}}$, with internal channel count $c = \\max(c_i, \\lfloor c_o / s^2 \\rfloor)$, is an orthogonal convolution with stride $s$ and kernel size $k$. The construction fuses a BCOP kernel — built by stacking symmetric projectors into $1\\times2$ and $2\\times1$ orthogonal convolutions and composing them — with an RKO kernel, a kernel reshaped from an orthogonal matrix, which this paper proves to be orthogonal exactly when its kernel size equals its stride. The choice of $c$ ensures that both factors are row-orthogonal or both column-orthogonal, so the composition inherits orthogonality from the corresponding property of orthogonal matrices. From this core result the paper derives native transposed orthogonal convolutions (the transpose of an orthogonal convolution is again orthogonal), grouped orthogonal convolutions (the Toeplitz matrix is block diagonal), and dilated orthogonal convolutions, and it notes that the covered parameter range $s \\le k$ is exhaustive because no orthogonal kernel exists when the stride exceeds the kernel size.","pith_inferences":["One consequence the authors leave implicit: with the circular-padding assumption in the proofs, any zero-padding deployment inherits a border effect, so certificates should be computed with a padding-aware Lipschitz bound rather than trusting exact orthogonality; the paper's own high-accuracy run reports 0% provable accuracy at radius $36/255$.","If the scalability numbers hold, the practical blocker for certified robustness at ImageNet scale shifts from layer cost to data and training budget, which are the constraints already identified by the robust-training literature.","A natural follow-up would be to swap the BCOP factor for a complete parametrization of orthogonal convolutions (BCOP provably misses some of them); the same $\\circledast$ machinery would then add striding to a strictly more expressive base class."],"forward_implications":["Native strided orthogonal convolutions replace stride emulation through input reshaping, cutting the parameter count of strided layers by a factor of $s^2$ and removing the cubic-cost blowup of emulated methods.","Transposed orthogonal convolutions become available for upsampling, so U-Nets, VAEs, and invertible normalizing-flow layers can be built from norm-preserving components.","Grouped and dilated orthogonal convolutions let depthwise and large-receptive-field architectures stay inside the 1-Lipschitz framework where robustness certificates are tight.","The reported scaling behaviour — the overhead falls as batch size grows, reaching about 13% on ImageNet-scale ResNet-34 — puts certifiably robust training on large datasets within a practical compute budget.","Because the covered configurations include every case where an orthogonal kernel can exist (stride no larger than kernel size), the method is a full coverage of the feasible parameter space rather than a partial patch."],"supporting_citations":[{"why":"Supplies the BCOP kernel family, the stride-one orthogonal construction that AOC composes with RKO.","marker":"(Li et al., 2019)"},{"why":"Supplies RKO, the reshaped-kernel orthogonalization whose orthogonality at stride equals kernel size is proven here.","marker":"(Serrurier et al., 2021)"},{"why":"Foundational block-convolution construction and orthogonal $1\\times2$ / $2\\times1$ kernels that BCOP builds on.","marker":"(Xiao et al., 2018)"},{"why":"Provides the dilation-orthogonality equivalence that AOC adopts for dilated convolutions.","marker":"(Su et al., 2022)"},{"why":"Establishes existence limits for orthogonal convolutions, including that none exist when stride exceeds kernel size.","marker":"(Achour et al., 2022)"},{"why":"Shows that orthogonal layers with MaxMin activations make the product Lipschitz bound tight, the basis of the certificate evaluation.","marker":"(Anil et al., 2019)"},{"why":"Frequency-domain Cayley baseline that AOC compares against for speed and feature support.","marker":"(Trockman & Kolter, 2021)"},{"why":"Provides the training recipe and data augmentation for the robust* results.","marker":"(Hu et al., 2023)"}],"fun_headline_variants":["AOC: orthogonal convolutions with stride, groups, dilation","Efficient orthogonal convolutions now support modern ops","Scalable orthogonal convolutions for flexible CNN designs","Orthogonal convs get stride, dilation, groups, and upsampling","AOC: adaptive orthogonal convolution, now with modern ops"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proofs assume circular padding at the image boundary, so with the standard zero padding used in most architectures the layer is only quasi-orthogonal along the borders, and the exact norm preservation that tight robustness certificates rely on is not guaranteed.","fun_headline_variants_meta":{"raw":{"variants":["AOC: orthogonal convolutions with stride, groups, dilation","Efficient orthogonal convolutions now support modern ops","Scalable orthogonal convolutions for flexible CNN designs","Orthogonal convs get stride, dilation, groups, and upsampling","AOC: adaptive orthogonal convolution, now with modern ops"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000644,"raw_usage":{"total_tokens":2973,"prompt_tokens":967,"completion_tokens":2006,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":1924}},"tokens_in":583,"tokens_out":2006,"duration_ms":13797,"temperature":1.0,"reasoning_tokens":1924,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:29:03.385652+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build the Toeplitz matrix of a trained AOC layer on a small image ($8\\times8$) using zero padding and compute its singular values: any singular value departing from 1 by more than $10^{-4}$ would falsify exact orthogonality in deployed settings. A complementary check is to verify the reported 'AOC accurate' result directly — the paper itself reports 0% certified accuracy at radius $36/255$, which a padding-aware bound should reproduce.","supporting_citations":[],"review_version":1}