Pith. sign in

REVIEW 4 major objections 5 minor 15 references

On the Depth Scalability of Logic Gate Networks

T0 review · 4 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read To make logic gate networks scale with depth, each gate must keep a private hidden spine and a direct input anchor; with this wiring, accuracy keeps improving to 150 layers.

desk verdict Plausible input-anchored topology for LGNs, but the τ hyperparameter and missing code leave the depth-scaling claim conditional. read the letter →

arxiv 2607.21633 v2 pith:7KXQEIHR submitted 2026-07-19 cs.LG cs.AIcs.LO

Taegun An , Dohun kim , Haebeom Lee , Changhee Joo This is my paper · ORCID
classification cs.LGcs.AIcs.LO
keywords LogicGateNetworksdepthscalingcreditassignmentinputanchoringBooleancircuitsdifferentiablegatesstraight-throughestimationrepresentationlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that logic gate networks (LGNs) fail to scale with depth for two distinct reasons: gradients vanish, and random wiring merges the output-specific computational paths, so hidden layers lose usable credit. It shows that even when optimization is stabilized with skip-biased initialization and straight-through estimation, randomly wired LGNs still saturate or degrade with depth. The proposed topology, Input-Anchored Logic Gate Networks (IALGNs), gives each gate one private hidden input (a spine) and one direct input-bit anchor, preserving a separate lineage to each output while keeping input access at every layer. On MNIST, CIFAR-10, and CIFAR-100, IALGN shows consistent fixed-width accuracy gains up to 150 layers, whereas alternative topologies do not. If correct, this separates credit-preserving topology from mere trainability and yields a constructive recipe for deep Boolean circuits.

What carries the argument

The carrying mechanism is the IALGN wiring rule: h_i^ℓ = g_i^ℓ(h_i^{ℓ−1}, x_{b_{ℓ,i}}), where one input pin (the spine) passes the same-index hidden feature from the previous layer and the other pin (the anchor) is a directly selected bit of the original binarized input. This two-pin structure prevents output-path merging while keeping input access at every layer. The paper's diagnostics—path sharing, descendant-conditioned gradient coverage, and gradient purity—show that random wiring merges output paths and dilutes credit, whereas IALGN maintains one private lineage per output with coherent gradients. Random-k_x anchor relaxation adds a learned soft selection among k_x candidate input anch

What would settle it

Run RWLGN and IALGN on CIFAR-10 with identical width and depth and a shared grid of τ values (including each method's best τ), then compare the depth–accuracy curves. If some τ setting gives RWLGN the same upward scaling as IALGN, the central claim fails.

Watch

Extended reading notes

Core claim

The central discovery is that depth scalability in logic gate networks requires both stable optimization and a topology that preserves output-specific credit. Randomly wired LGNs that are successfully stabilized with skip-biased initialization and straight-through estimation still fail to improve with depth because early-layer gates become ancestors of nearly every output, diluting or canceling the gradient contributions. IALGN prevents this by giving each gate a private one-to-one hidden spine (so each output keeps its own lineage) and a direct anchor to the original input (so each layer has task-relevant information without routing through another hidden unit). The resulting networks exhib

Load-bearing premise

The comparison assumes the logit scale τ (and the shared optimization settings) is equally fair to every topology; if τ was tuned toward IALGN rather than swept for each baseline, the depth-scaling gap could be a hyperparameter artifact.

Editorial extensions

If this is right

  • Fixed-width IALGNs improve classification accuracy with depth up to 150 layers on MNIST, CIFAR-10, and CIFAR-100, while randomly wired LGNs and the reimplemented baseline saturate or degrade.
  • The depth gains come from input anchoring, not generic wiring flexibility: relaxing the hidden spine (k_h > 1) helps at depth 4 but hurts at depth 50.
  • Trained IALGN circuits mostly preserve a private hidden state (about 88% spine-preserving operations) with sparse anchor-conditioned updates, so nominal depth is not the same as functional path length.
  • Optimization fixes alone do not confer depth scaling: under the strongest shared protocol, the random-wired network still declines with depth.
  • At inference each gate keeps a single selected anchor, so the trained model is a discrete two-input Boolean circuit with no train-versus-inference gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The IALGN wiring is a discrete analogue of residual or dense connectivity: the spine acts as a learned state, the anchor as a fixed input skip. It could be transplanted into other discrete or quantized architectures (lookup-table networks, binarized networks) to test whether depth scaling reappears there.
  • Because each gate reads only one input bit, trained circuits are extremely input-sparse; a natural test is whether the accuracy scaling survives when anchors are limited to a task-relevant input region, and whether the learned anchor selection consistently picks interpretable bits.
  • The paper stops short of budget-matched comparisons; a width-versus-depth study under a fixed total-gate budget would clarify how much of the gain is topology versus extra parameters.
  • The observed identity-heavy circuits suggest a cheap optimization trick: regularize gate logits toward pass-through operations during training, which might accelerate convergence or improve scaling in other LGN variants.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that deep Logic Gate Networks (LGNs) fail to benefit from depth for two distinct reasons: optimization collapse, which can be mitigated by skip-biased initialization and straight-through estimation, and topology-induced degradation of output-specific credit, which persists even when training is stabilized. The authors propose Input-Anchored Logic Gate Networks (IALGNs), where each gate takes one input from a private hidden spine (usually the same-index previous-layer gate) and one input directly from the original input. They report credit-assignment diagnostics (path sharing, gradient coverage, gradient purity) showing that random wiring dilutes or conflicts output-specific gradients, whereas IALGN preserves coherent private credit. Empirically, on MNIST, CIFAR-10, and CIFAR-100, IALGN shows consistent fixed-width accuracy improvements with depth up to 150 layers, while RWLGN and a reimplemented LDLGN baseline saturate or degrade. Additional analyses (linear probes, kh ablations, operation-aware effective depth) are used to support the claim that IALGN's depth scaling arises from state preservation and sparse input-conditioned refinement rather than raw path length or generic wiring flexibility.

Significance. If the main claim holds, the paper offers a simple, credit-preserving topology that makes depth useful in differentiable Boolean circuits, a problem that has limited LGN research. The paper's strengths include multi-seed results with standard deviations, controlled comparisons under shared optimization settings, several ablations (kh, kx, nb, initialization/gradient-estimator variants), and explicit diagnostics that go beyond raw accuracy. The main result is not circular: accuracy is measured against independent baselines. However, the supporting evidence has gaps that need to be addressed before the claim is fully convincing: the logit-scaling hyperparameter τ is not defined and not sensitivity-checked, the D=150 result rests on a single configuration, the LDLGN baseline is a self-described reimplementation without code, and the credit-diagnostic sampling protocol may favor the proposed architecture. These issues are fixable, so the paper is a promising candidate for major revision rather than rejection.

major comments (4)
  1. [§6, Table 4, Appendix A] The logit-scaling hyperparameter τ is listed per dataset/width but is never defined in the main text, and no sensitivity analysis over τ is reported for RWLGN or LDLGN. Since the central claim is that IALGN's depth-scaling gap arises from topology under identical optimization settings, an unexamined τ that varies per experiment leaves open the possibility that the baselines are suppressed by a hyperparameter choice. Please define τ, state how the values were selected, and provide a sweep of τ for RWLGN (and LDLGN) at representative depths to show that the qualitative depth-scaling gap is robust.
  2. [Abstract, Table 1] The abstract claims 'consistent fixed-width depth–accuracy scaling up to 150 layers,' but D=150 appears for only one configuration (CIFAR-10, W=12k). Other sweeps stop at D=80 (CIFAR-10 W=128k, CIFAR-100) or D=100 (MNIST). At the single D=150 point, the improvement over D=100 is 0.09±0.20 (56.50±0.11 vs 56.41±0.17), which is not significant. Either extend the depth sweeps to at least one more width/dataset, or qualify the 'up to 150 layers' claim as preliminary.
  3. [Table 1, LDLGN reimpl.] The LDLGN baseline is explicitly described as a reimplementation 'not a reproduction of the original reported results,' and no code is provided. Since this baseline is used in the headline comparison (and in Figure 8), a reader cannot verify that the reimplementation faithfully represents the original method. Please release code (or a detailed implementation specification) and, if possible, validate the reimplementation against published numbers under the same no-augmentation protocol, or remove LDLGN from the central comparison and rely on RWLGN as the main baseline.
  4. [§3, Fig. 2, Appendix B] The paired hidden-output sampling procedure uses identical sampled indices for hidden gates and output gates. The text argues this avoids bias against private-chain architectures, but it may instead favor IALGN: if an IALGN hidden gate's chain output index is in the output sample, that gate is guaranteed to have a reachable sampled output, whereas a randomly wired hidden gate's sampled output descendants are not guaranteed. This could inflate IALGN's coverage or purity relative to RWLGN. The authors should report what happens under independent uniform sampling of hidden and output gates, or justify why the paired scheme does not introduce a systematic bias. The credit diagnostics are the main mechanistic evidence, so this point is important.
minor comments (5)
  1. [Figure 1 caption] The caption should explicitly state which curves use which initialization/STE combination, especially the RWLGN curve, to avoid confusion with the four IALGN settings.
  2. [§5, Figure 4] The probe depth labels (0,1,10,...,100) are not specified in the caption; please state that depth 0 refers to the binarized input before any logic layer, as mentioned in the body text.
  3. [Table 2 caption] There is a stray period in the caption ('... benefit deep models. Topology...'). Also, the table caption should clarify that 'Relaxed source' refers to the pin that is relaxed, not the output source.
  4. [Appendix C] The STE equations use exsoft and exhard; it would be helpful to define the softmax temperature or scaling used to compute exsoft from the anchor-selection logits. This is related to the τ question and should be made explicit.
  5. [References] The reference to the AAAI copyright line in the header is unusual; please ensure the venue and year are correct.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: depth-scaling claim is an independent benchmark result; diagnostic metrics are explanatory and not fitted inputs.

full rationale

The paper's central claim—fixed-width depth–accuracy scaling for IALGN up to 150 layers with RWLGN and LDLGN baselines saturating or degrading—is an externally measured empirical result (Table 1), not a quantity derived from the model's definitions. Accuracy is reported on standard test sets under shared initialization, optimizer, STE, and objective; the only architectural change is the wiring source (Eq. 1 vs. Eq. 2), and the kx=1 fixed-anchor ablation, the kh relaxation (Table 2), and initialization/STE variants (Fig. 6) dissociate the depth trend from optimization hyperparameters. The credit diagnostics (Fig. 2) are explanatory post-hoc metrics; the paired-sampling rule makes IALGN coverage/purity partly definitional, but the depth-scaling conclusion is not fitted to or inferred from these diagnostics, so this is at most a minor self-referentiality, not a circular derivation. The paper explicitly limits its claim ('does not establish superiority at every fixed gate budget') and labels LDLGN reimpl. as a within-paper reference rather than an official reproduction. The only fairness gap—the undefined per-dataset logit scaling τ in Table 4—is a hyperparameter-sensitivity concern, not a demonstrated equivalence between input and output; no text shows τ was fit to favor IALGN. No load-bearing self-citation or imported uniqueness theorem appears.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central new content is a wiring topology, not a new mathematical or physical object. Accuracy scaling is empirical and needs no fitted constants, but the explanatory story and the architecture comparisons lean on hand-set hyperparameters, inherited optimization heuristics, and diagnostic thresholds.

free parameters (5)
  • logit scaling τ = MNIST: 10; CIFAR-10 W=12k: 30; CIFAR-10 W=128k: 100; CIFAR-100: 30
    Set per dataset/width in Table 4 with no published justification or sensitivity check. This is load-bearing because all compared architectures use it, and an unfavorable τ could handicap RWLGN.
  • anchor candidate count kx = 32 (default)
    Chosen for all main IALGN runs. Figure 9 shows it affects absolute accuracy, though Figure 7 shows fixed anchors still give depth scaling.
  • spine candidate count kh = 1 (fixed identity spine)
    The core design choice. Ablations show relaxing kh to 8/16/32 hurts performance at D=50, supporting the fixed-spine claim.
  • input binarization resolution nb = 4 (MNIST), 16 (CIFAR)
    Thermometer-encoding resolution. Interacts with anchor selection (Figure 9) and changes the input dimension available to anchor pins.
  • active-credit threshold t_{ℓ,i} = 10^-5 × max contribution, floor 10^-12
    Defines 'active' output-credit in the coverage/purity diagnostics. Changing this threshold could change the headline credit conclusions.
assumptions (6)
  • standard math Boolean circuit depth strictly enlarges representational power (Hastad 1986).
    Invoked in the Introduction to motivate why depth should help in LGNs; not proven inside this paper.
  • domain assumption Skip-biased initialization and straight-through estimators stabilize deep LGN optimization.
    Taken from prior LGN work and used to separate optimization collapse from topology-induced credit degradation.
  • standard math The stopgrad-based output-specific gradient decomposition exactly recovers the ordinary loss gradient.
    Appendix B defines g_{o→ℓ,i} via a stopgrad decomposition; it is exact for first-order gradient terms and is used to justify coverage/purity.
  • domain assumption Paired hidden–output sampling yields comparable coverage statistics across topologies.
    Appendix B asserts this avoids architecture-dependent sampling bias, but the equal-index sampling is a choice that makes IALGN's one-to-one lineage structure visible.
  • domain assumption Linear probes and activation entropy measure representational usefulness of hidden layers.
    Used to claim deeper IALGN layers are progressively more informative; probe accuracy is a standard but indirect proxy for representation quality.
  • domain assumption The operation-aware effective-depth recursion captures how nominal depth is used.
    Appendix F defines the recursion; the authors themselves note it 'does not measure output-specificity or credit quality', so it supports only a partial interpretation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Depth Scalability of Logic Gate Networks." pith.science (2026). https://pith.science/paper/7KXQEIHR

@misc{pith2026260721633,
  author       = {Pith},
  title        = {Pith review of: On the Depth Scalability of Logic Gate Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7KXQEIHR}},
  note         = {Machine review of arXiv:2607.21633}
}
abstract

Logic Gate Networks (LGNs) compute through compositions of Boolean operations, yet existing LGNs do not reliably benefit from increased depth. We identify two causes: optimization collapse and topology-induced degradation of output-specific credit that persists even after skip-biased initialization and straight-through estimation stabilize training. We introduce Input-Anchored Logic Gate Networks (IALGNs), in which each gate combines a private hidden spine with a direct input anchor. This topology prevents output-path merging while retaining input access at every layer. Credit diagnostics show that random wiring dilutes or conflicts output-specific gradients, whereas IALGN maintains usable and coherent credit. Random-$k_x$ relaxation improves anchor selection without relaxing the spine. Across MNIST, CIFAR-10, and CIFAR-100, IALGN exhibits consistent fixed-width depth--accuracy scaling up to 150 layers, while alternative topologies saturate or degrade. Linear probes, topology ablations, and operation-aware analysis show that trained IALGNs preserve private states and apply sparse anchor-conditioned updates. These results indicate that scalable LGN depth requires both stable optimization and credit-preserving information access.

Figures

Figures reproduced from arXiv: 2607.21633 by the authors.

Figure 1
Figure 1. Optimization stability alone is insufficient for depth scalability. (a,b) Layer-wise update magnitudes under different [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Output-path sharing and output-specific credit assignment in 100-layer CIFAR-10 LGNs. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Input-Anchored Logic Gate Networks and random- [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Progressive representation learning enabled by input anchoring. (a) Layer-wise linear-probe accuracy on frozen [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Selective depth utilization in trained 100-layer net [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Depth scaling under initialization and gradient [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Depth scaling of fixed-anchor IALGN on CIFAR [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 9
Figure 9. Figure 9: Effect of input encoding and anchor candidate size [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 8
Figure 8. Figure 8: Test accuracy versus the number of deployed [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 1 canonical work pages

  1. [1]

    Proceedings of the Eighteenth Annual ACM Symposium on Theory of Computing , pages =

    Hastad, J , title =. Proceedings of the Eighteenth Annual ACM Symposium on Theory of Computing , pages =. 1986 , doi =

    1986

  2. [2]

    Weight Agnostic Neural Networks , volume =

    Gaier, Adam and Ha, David , booktitle =. Weight Agnostic Neural Networks , volume =

  3. [3]

    2025 , eprint=

    From MNIST to ImageNet: Understanding the Scalability Boundaries of Differentiable Logic Gate Networks , author=. 2025 , eprint=

    2025

  4. [4]

    Deep Differentiable Logic Gate Networks , volume =

    Petersen, Felix and Borgelt, Christian and Kuehne, Hilde and Deussen, Oliver , booktitle =. Deep Differentiable Logic Gate Networks , volume =

  5. [5]

    Convolutional Differentiable Logic Gate Networks , volume =

    Petersen, Felix and Kuehne, Hilde and Borgelt, Christian and Welzel, Julian and Ermon, Stefano , booktitle =. Convolutional Differentiable Logic Gate Networks , volume =. doi:10.52202/079017-3851 , pages =

    work page doi:10.52202/079017-3851

  6. [6]

    Advances in Neural Information Processing Systems , doi =

    Shakir Yousefi and Andreas Plesner and Till Aczel and Roger Wattenhofer , title=. Advances in Neural Information Processing Systems , doi =

  7. [7]

    arXiv , primaryClass=

    Light Differentiable Logic Gate Networks , author=. arXiv , primaryClass=. 2025 , eprint=

    2025

  8. [8]

    arXiv , primaryClass=

    Recurrent Deep Differentiable Logic Gate Networks , author=. arXiv , primaryClass=. 2025 , eprint=

    2025

Show all 15 references
  1. [9]

    Proceedings of the International Conference on Neuro-symbolic Systems , pages=

    Logic Gate Neural Networks are Good for Verification , author=. Proceedings of the International Conference on Neuro-symbolic Systems , pages=. 2025 , volume=

    2025

  2. [10]

    arXiv , primaryClass=

    Rapid Inference of Logic Gate Neural Networks for Anomaly Detection in High Energy Physics , author=. arXiv , primaryClass=. 2025 , eprint=

    2025

  3. [11]

    Deep Stochastic Logic Gate Networks , year=

    Kim, Youngsung , journal=. Deep Stochastic Logic Gate Networks , year=

  4. [12]

    1910.00780 , journal=

    How does topology influence gradient propagation and model performance of deep networks with DenseNet-type skip connections? , author=. 1910.00780 , journal=

    1910 arXiv

  5. [13]

    arXiv , primaryClass=

    Residual Networks Behave Like Ensembles of Relatively Shallow Networks , author=. arXiv , primaryClass=. 2016 , eprint=

    2016

  6. [14]

    2026 , eprint=

    Fully Trainable Deep Differentiable Logic Gate Networks and Lookup Table Networks , author=. 2026 , eprint=

    2026

  7. [15]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , month =

    Fojcik, Katarzyna and Zioma, Renaldas and Armaitis, Jogundas , title =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , month =. 2026 , pages =

    2026

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.