Pith. sign in

REVIEW 2 major objections 2 minor 58 references

Recursive Binding on a Budget: Subspace Carving in Order-p Tensor Memories

T0 review · 2 major / 2 minor · reviewed 2026-06-27 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Orthogonal Subspace Carving decouples tensor order from recursion depth for constant-memory deep binding.

desk verdict OSC uses null-space projections to hold recursion depth fixed while keeping the tensor order constant, but the capacity after repeated projections is the part that still needs checking. read the letter →

arxiv 2606.11391 v1 pith:IWYG364P submitted 2026-06-09 cs.LG

Travis Pence , Daisuke Yamada , Vikas Singh This is my paper
classification cs.LG
keywords orthogonalsubspacecarvingtensorproductrepresentationsvectorsymbolicarchitecturesrecursivebindingnullspaceprojectionsmemoryefficiencyCliffordalgebrasuperposition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Tensor product representations deliver structural fidelity for symbolic reasoning but grow exponentially with recursion depth, while vector symbolic architectures keep fixed size at the cost of noisy superposition. The paper introduces Orthogonal Subspace Carving to bind fillers to roles by projecting each new role basis into the null space of prior roles before summing into a fixed order-p tensor. This geometric separation keeps bound structures orthogonal inside the same memory trace, so tensor order no longer scales with depth. Retrieval switches to recognition of the bound components rather than direct extraction, which in turn permits the component vectors themselves to be far smaller than the overall memory. The construction also recovers ordinary tensor products as a special case inside a Clifford-algebra formulation.

What carries the argument

Orthogonal Subspace Carving (OSC): successive null-space projections of each new role vector before it is bound and added into the fixed-order memory tensor.

What would settle it

Measure the number of distinct bound structures that can be reliably recognized after k recursive levels; if recognition accuracy collapses after only a few levels even with exact arithmetic, the constant-footprint claim fails.

Watch

Extended reading notes

Core claim

By projecting role vectors onto successive null spaces before aggregation, OSC maintains a fixed-size order-p tensor while supporting recursive binding to arbitrary depth; retrieval occurs through recognition of the carved subspaces rather than noisy unbinding, and traditional TPR binding appears as the special case in which no projections are applied.

Load-bearing premise

Exact null-space projections remain feasible and the carved subspaces stay sufficiently populated to sustain high-capacity superposition as recursion depth grows.

Editorial extensions

If this is right

  • Recursive structures of arbitrary depth fit inside a memory tensor whose size is independent of depth.
  • Component vectors can be orders of magnitude smaller than the memory tensor while still supporting reliable recognition.
  • High levels of superposition become feasible without the usual capacity penalty of vector-symbolic superposition.
  • Tensor-product binding is recovered exactly when the projection step is omitted, placing TPR inside the same Clifford-algebra framework.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same carving technique might be applied to other binding operations that currently rely on explicit concatenation or concatenation-like growth.
  • Approximate rather than exact projections could still suffice for moderate depths if the orthogonality requirement is relaxed to near-orthogonality.
  • Recognition-based retrieval opens a route to hybrid symbolic-neural systems that store structure in fixed hardware memory budgets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces Orthogonal Subspace Carving (OSC), a binding mechanism for order-p tensor memories in which fillers are projected onto the null space of the current role basis before superposition into a fixed-size tensor. This is claimed to decouple tensor order from recursion depth, permitting arbitrarily deep recursive structures within constant memory while supporting retrieval by recognition rather than reconstruction; component vectors can therefore be orders of magnitude smaller than the memory tensor. The work also presents TPR as a special case of binding in Clifford algebra and supplies a Clifford-algebraic formulation of OSC.

Significance. If the capacity bounds hold, the construction would constitute a genuine advance over both classical Tensor Product Representations (exponential growth with depth) and Vector Symbolic Architectures (capacity loss from noisy superposition), enabling high-fidelity symbolic recursion at fixed dimensionality. The Clifford-algebraic unification is a conceptual strength that may open further algebraic generalizations.

major comments (2)
  1. [Abstract, §3] Abstract and §3 (method): the central claim that successive null-space projections preserve sufficient subspace volume for high-capacity superposition at arbitrary depth is load-bearing, yet no explicit lower bound on the dimension of the orthogonal complement after d projections, nor a capacity scaling law relating remaining dimension to the number of fillers, is supplied. Without such a bound the recognition-retrieval guarantee cannot be verified.
  2. [§4] §4 (capacity analysis): the statement that recognition remains reliable under high superposition is asserted without a derivation showing that the effective dimension after k projections declines slower than the number of bound items; the skeptic's concern that the noise floor rises for d>3 is therefore unaddressed by any quantitative argument.
minor comments (2)
  1. [§2.2] Notation for the projection operator P_⊥ is introduced without an explicit definition in terms of the role basis matrix; a one-line equation would remove ambiguity.
  2. [Figure 2] Figure 2 caption refers to 'component vectors' but the axis labels use a different symbol; consistency would improve readability.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful reading and constructive critique. The two major comments correctly identify places where the capacity analysis can be made more rigorous; we address each point below and will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: [Abstract, §3] Abstract and §3 (method): the central claim that successive null-space projections preserve sufficient subspace volume for high-capacity superposition at arbitrary depth is load-bearing, yet no explicit lower bound on the dimension of the orthogonal complement after d projections, nor a capacity scaling law relating remaining dimension to the number of fillers, is supplied. Without such a bound the recognition-retrieval guarantee cannot be verified.

    Authors: We agree that an explicit lower bound on the dimension of the remaining orthogonal complement after successive projections is needed to make the capacity claim fully rigorous. In the revised manuscript we will add a short derivation in §3 showing that, when each role basis has rank at most r and the ambient space has dimension n, the carved subspace after d steps has dimension at least n − d r (under linear independence). We will then state a simple volume-based capacity scaling that relates the number of admissible fillers to the dimension of this carved subspace, thereby supplying the missing recognition-retrieval guarantee. revision: yes

  2. Referee: [§4] §4 (capacity analysis): the statement that recognition remains reliable under high superposition is asserted without a derivation showing that the effective dimension after k projections declines slower than the number of bound items; the skeptic's concern that the noise floor rises for d>3 is therefore unaddressed by any quantitative argument.

    Authors: We will expand §4 with a quantitative argument that compares the linear decline in effective dimension (n − k r) against the exponential growth in the number of distinguishable fillers permitted by the tensor-product structure inside the carved subspace. The added derivation will show that the inner-product noise floor remains bounded for d > 3 provided the total carved dimension stays below n, directly addressing the concern about reliability under high superposition. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity; construction presented as independent projection mechanism

full rationale

The abstract introduces OSC as a projection-based binding method that enforces orthogonality within a fixed-order tensor, decoupling depth from dimensionality. No equations or steps are shown that define a quantity in terms of itself, rename a fitted parameter as a prediction, or rely on self-citation chains for load-bearing uniqueness claims. The Clifford-algebra reformulation of TPR is stated as a derived result rather than an input assumption. The central capacity and retrieval claims rest on geometric properties of successive null-space projections, which are external to the definition of OSC itself and therefore constitute an independent (if unproven here) argument rather than a circular reduction.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

Abstract-only review supplies no concrete free parameters, axioms, or invented entities; the central claim rests on unstated geometric and algebraic assumptions whose details are unavailable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Recursive Binding on a Budget: Subspace Carving in Order-p Tensor Memories." pith.science (2026). https://pith.science/paper/IWYG364P

@misc{pith2026260611391,
  author       = {Pith},
  title        = {Pith review of: Recursive Binding on a Budget: Subspace Carving in Order-p Tensor Memories},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IWYG364P}},
  note         = {Machine review of arXiv:2606.11391}
}
read the original abstract

Tensor Product Representations provide the structural fidelity required for symbolic reasoning in models but suffer from exponential dimensionality growth when encoding deep recursive structures. Conversely, Vector Symbolic Architectures maintain constant dimensionality but sacrifice capacity and fidelity due to noisy compression via superposition. In this work, we propose Orthogonal Subspace Carving (OSC), a memory architecture that binds fillers to roles by projecting onto the null space of the role basis before aggregating into a fixed order-p tensor. OSC uses projections to enforce geometric orthogonality between bound structures within a static memory trace. We show that this mechanism decouples the tensor order from the structural depth, enabling deep recursive binding within a constant memory footprint. By performing retrieval via recognition, this construction allows for component vectors that are orders of magnitude smaller than the memory tensor, giving superior memory efficiency in settings involving high superposition. We also show that TPR is a special case of binding in Clifford algebra, and give a Clifford formulation of OSC.

Figures

Figures reproduced from arXiv: 2606.11391 by the authors.

Figure 1
Figure 1. Raw signal attributes are identified (Step 1) and mapped to vectors using a codebook (Step 2). These are then bound to their corresponding roles and superposed to form the memory (Step 3). So, when we associate 1 role to a filler, the memory space is an order-2 tensor. When binding |C| roles to a filler, the memory space is a order-(|C|+ 1) tensor with d |C|+1 entries. This grows exponentially with context size. Def… view at source ↗
Figure 2
Figure 2. Visualization of the rejection process. The vector v is projected onto orthogonal complements of C1 and C2, resulting in nearly orthogonal rejection vectors. The more vectors in a filler, the less likelihood that all rejection vectors will align. High level Principle. The reversal described above is key. Standard approaches mark storage locations explicitly: “role 1 goes in dimension 7”. OSC instead says: “this cont… view at source ↗
Figure 3
Figure 3. Performance comparison of 14 VSAs at d = 4096 and OSC across d = 25, 50, 75, 100, 125, 150, 175, and 200 for p = 2 at increasing bundle counts with unique fillers and role depth 1. OSC outperforms all VSAs on both retrieval and recognition tasks. Note the difference in Y axes between graphs. Standard deviation is denoted by the shaded regions. We provide comparisons for other role depths and VSA dimensions in Append… view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: Performance of HLB at varying dimensions against OSC for p = 3. Identically colored runs share the same superposition memory size (not total memory). VSAs are slightly superior than OSC in memory capacity, but OSC’s sub-linear scaling requires the VSA to have 1178× OSC…
Figure 4
Figure 4. Figure 4: Minimum filler dimension required to achieve 99% accuracy. A dimension was selected only if mean accuracy across 10 trials exceeded 99% at that dimension but fell below 99% at the next lowest dimension. The dotted line tracks the actual dimension points, while the soli…
Figure 6
Figure 6. Figure 6: Performance comparison of 14 VSAs at d = 8100 and OSC across d = 25, 50, 75, 100, 125, 150, 175, and 200 for p = 2 at increasing bundle counts with unique fillers and role depth 1. OSC outperforms all VSAs on both retrieval and recognition tasks. Note the difference in…
Figure 7
Figure 7. Figure 7: Performance comparison of 14 VSAs at d = 8100 and OSC across d = 25, 50, 75, 100, 125, 150, 175, and 200 for p = 2 at increasing bundle counts with unique fillers and role depth 48. OSC outperforms all VSAs on both retrieval and recognition tasks. Note the difference i…
Figure 8
Figure 8. Figure 8: Performance of HLB at varying dimensions against OSC for p = 2. Identically colored runs share the same superposition memory size (not total memory). VSAs are slightly superior than OSC in memory capacity, but OSC’s sub-linear scaling requires the VSA to have 2002 ∗ 10…
Figure 9
Figure 9. Figure 9: Performance of HLB at varying dimensions against OSC for p = 3. Identically colored runs share the same superposition memory size (not total memory). VSAs are slightly superior than OSC in memory capacity, but OSC’s sub-linear scaling requires the VSA to have 703 ∗ 100…
Figure 10
Figure 10. Figure 10: Adapted from [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: Retrieval accuracy when using the recognition-based retrieval approach for FHRR, HLB, MAP I, and VTB. The curves for d = 4096 and d = 8100 match those of [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: Heatmap of retrieval accuracy for filler dimensions ranging from 50 to 200 in increments of 10. Accuracy is normalized per dimension, with the maximum value scaled to 1. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

58 extracted references · 10 canonical work pages

  1. [1]

    Cognitive Science , volume=

    Physical symbol systems , author=. Cognitive Science , volume=. 1980 , publisher=

    1980

  2. [2]

    Artificial Intelligence , volume=

    The knowledge level , author=. Artificial Intelligence , volume=. 1982 , publisher=

    1982

  3. [3]

    Cognitive Computation , volume=

    Hyperdimensional computing: An introduction to computing in distributed representation with high-dimensional random vectors , author=. Cognitive Computation , volume=. 2009 , publisher=

    2009

  4. [4]

    2001 , publisher=

    The algebraic mind: Integrating connectionism and cognitive science , author=. 2001 , publisher=

    2001

  5. [5]

    International Conference on Learning Representations , year=

    Measuring compositional generalization: A comprehensive method on realistic data , author=. International Conference on Learning Representations , year=

  6. [6]

    2020 , publisher=

    Kim, Najoung and Linzen, Tal , booktitle=. 2020 , publisher=

    2020

  7. [7]

    SLOG : A Structural Generalization Benchmark for Semantic Parsing

    Li, Bingzhi and Donatelli, Lucia and Koller, Alexander and Linzen, Tal and Yao, Yuekun and Kim, Najoung. SLOG : A Structural Generalization Benchmark for Semantic Parsing. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. doi:10.18653/v1/2023.emnlp-main.194

    work page doi:10.18653/v1/2023.emnlp-main.194 2023

  8. [8]

    Advances in Neural Information Processing Systems , volume=

    Faith and Fate: Limits of Transformers on Compositionality , author=. Advances in Neural Information Processing Systems , volume=

Show all 58 references
  1. [9]

    Tensor product variable binding and the representation of symbolic structures in connectionist systems , journal =

    Paul Smolensky , abstract =. Tensor product variable binding and the representation of symbolic structures in connectionist systems , journal =. 1990 , issn =. doi:https://doi.org/10.1016/0004-3702(90)90007-M , url =

    1990 doi

  2. [10]

    2023 , eprint=

    Differentiable Tree Operations Promote Compositional Generalization , author=. 2023 , eprint=

    2023

  3. [11]

    2024 , eprint=

    Compositional Generalization Across Distributional Shifts with Sparse Tree Operations , author=. 2024 , eprint=

    2024

  4. [12]

    Neural Computation , volume =

    Gosmann, Jan and Eliasmith, Chris , title =. Neural Computation , volume =. 2019 , month =. doi:10.1162/neco_a_01179 , url =

    2019 doi

  5. [13]

    Plate, T. A. , journal=. Holographic reduced representations , year=

  6. [14]

    A comparison of vector symbolic architectures , volume=

    Schlegel, Kenny and Neubert, Peer and Protzel, Peter , year=. A comparison of vector symbolic architectures , volume=. Artificial Intelligence Review , publisher=. doi:10.1007/s10462-021-10110-3 , number=

    doi

  7. [15]

    and Osipov, Evgeny and Rahimi, Abbas , year=

    Kleyko, Denis and Rachkovskij, Dmitri A. and Osipov, Evgeny and Rahimi, Abbas , year=. A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part I: Models and Data Transformations , volume=. ACM Computing Surveys , publisher=. doi:10.1145/3538531 , number=

    doi

  8. [16]

    International Conference on Artificial Neural Networks , year=

    Binary Spatter-Coding of Ordered K-Tuples , author=. International Conference on Artificial Neural Networks , year=

  9. [17]

    2003 , publisher=

    Holographic Reduced Representation: Distributed Representation for Cognitive Structures , author=. 2003 , publisher=

    2003

  10. [18]

    2015 , eprint=

    Representing Objects, Relations, and Sequences , author=. 2015 , eprint=

    2015

  11. [19]

    1998 , url=

    Multiplicative Binding, Representation Operators & Analogy , author=. 1998 , url=

    1998

  12. [20]

    A Walsh Hadamard Derived Linear Vector Symbolic Architecture , url =

    Alam, Mohammad Mahmudul and Oberle, Alexander and Raff, Edward and Biderman, Stella and Oates, Tim and Holt, James , booktitle =. A Walsh Hadamard Derived Linear Vector Symbolic Architecture , url =. doi:10.52202/079017-0089 , editor =

    doi

  13. [21]

    Advances in neural information processing systems , volume=

    Understanding hyperdimensional computing for parallel single-pass learning , author=. Advances in neural information processing systems , volume=

  14. [22]

    2024 , eprint=

    Generalized Holographic Reduced Representations , author=. 2024 , eprint=

    2024

  15. [23]

    IEEE transactions on Knowledge and Data Engineering , volume=

    Representation and processing of structures with binary sparse distributed codes , author=. IEEE transactions on Knowledge and Data Engineering , volume=. 2002 , publisher=

    2002

  16. [24]

    2015 IEEE Biomedical Circuits and Systems Conference (BioCAS) , pages=

    High-dimensional computing with sparse vectors , author=. 2015 IEEE Biomedical Circuits and Systems Conference (BioCAS) , pages=. 2015 , organization=

    2015

  17. [25]

    Advances in neural information processing systems , volume=

    Learning with holographic reduced representations , author=. Advances in neural information processing systems , volume=

  18. [26]

    Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining , pages=

    Extreme multi-label loss functions for recommendation, tagging, ranking & other missing label applications , author=. Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining , pages=

  19. [27]

    IEEE transactions on computers , volume=

    Correlation matrix memories , author=. IEEE transactions on computers , volume=. 2009 , publisher=

    2009

  20. [28]

    1972 , issn =

    A simple neural network generating an interactive memory , journal =. 1972 , issn =. doi:https://doi.org/10.1016/0025-5564(72)90075-2 , url =

    1972 doi

  21. [29]

    IEEE Transactions on Systems, Man, and Cybernetics , number=

    Associatron-a model of associative memory , author=. IEEE Transactions on Systems, Man, and Cybernetics , number=. 2007 , publisher=

    2007

  22. [30]

    IEEE Transactions on Computers , volume=

    Representation of associated data by matrix operators , author=. IEEE Transactions on Computers , volume=. 1973 , publisher=

    1973

  23. [31]

    , author=

    Neural networks and physical systems with emergent collective computational abilities. , author=. Proceedings of the national academy of sciences , volume=

  24. [32]

    IEEE transactions on Information Theory , volume=

    The capacity of the Hopfield associative memory , author=. IEEE transactions on Information Theory , volume=. 2003 , publisher=

    2003

  25. [33]

    Physical Review Letters , volume=

    Number of stable points for spin-glasses and neural networks of higher orders , author=. Physical Review Letters , volume=. 1987 , publisher=

    1987

  26. [34]

    Advances in Neural Information Processing Systems , volume=

    Dense associative memory for pattern recognition , author=. Advances in Neural Information Processing Systems , volume=

  27. [35]

    Journal of Statistical Physics , volume=

    On a model of associative memory with huge storage capacity , author=. Journal of Statistical Physics , volume=. 2017 , publisher=

    2017

  28. [36]

    Neural computation , volume=

    Dense associative memory is robust to adversarial inputs , author=. Neural computation , volume=. 2018 , publisher=

    2018

  29. [37]

    Physical Review Letters , volume=

    Exponential capacity of dense associative memories , author=. Physical Review Letters , volume=. 2024 , publisher=

    2024

  30. [38]

    Advances in Neural Information Processing Systems , volume=

    Provably optimal memory capacity for modern hopfield models: Transformer-compatible dense associative memories as spherical codes , author=. Advances in Neural Information Processing Systems , volume=

  31. [39]

    arXiv preprint arXiv:2404.03827 , year=

    Uniform memory retrieval with larger capacity for modern hopfield models , author=. arXiv preprint arXiv:2404.03827 , year=

  32. [40]

    arXiv preprint arXiv:1403.3369 , year=

    Controlling recurrent neural networks by conceptors , author=. arXiv preprint arXiv:1403.3369 , year=

  33. [41]

    Journal of Machine Learning Research , volume=

    Using conceptors to manage neural long-term memories for temporal patterns , author=. Journal of Machine Learning Research , volume=

  34. [42]

    International Conference on Learning Representations , year=

    Overcoming catastrophic interference using conceptor-aided backpropagation , author=. International Conference on Learning Representations , year=

  35. [43]

    Nature Machine Intelligence , volume=

    Continual learning of context-dependent processing in neural networks , author=. Nature Machine Intelligence , volume=. 2019 , publisher=

    2019

  36. [44]

    International conference on artificial intelligence and statistics , pages=

    Orthogonal gradient descent for continual learning , author=. International conference on artificial intelligence and statistics , pages=. 2020 , organization=

    2020

  37. [45]

    arXiv preprint arXiv:2103.09762 , year=

    Gradient projection memory for continual learning , author=. arXiv preprint arXiv:2103.09762 , year=

  38. [46]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Continual learning with scaled gradient projection , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  39. [47]

    Proceedings of the 30th ACM International Conference on Multimedia , pages=

    Class gradient projection for continual learning , author=. Proceedings of the 30th ACM International Conference on Multimedia , pages=

  40. [48]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Adaptive orthogonal projection for batch and online continual learning , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  41. [49]

    Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    Orthogonal subspace learning for language model continual learning , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    2023

  42. [50]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Put the Space of LoRA Initialization to the Extreme to Preserve Pre-trained Knowledge , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  43. [51]

    and Dahiya, K

    Bhatia, K. and Dahiya, K. and Jain, H. and Kar, P. and Mittal, A. and Prabhu, Y. and Varma, M. , title =

  44. [52]

    2003 , publisher=

    Geometric algebra for physicists , author=. 2003 , publisher=

    2003

  45. [53]

    2012 , eprint=

    Geometric Algebra , author=. 2012 , eprint=

    2012

  46. [54]

    , journal=

    Welch, L. , journal=. Lower bounds on the maximum cross correlation of signals (Corresp.) , year=

  47. [55]

    Advances in Neural Information Processing Systems , year=

    Composing Linear Layers from Irreducibles , author=. Advances in Neural Information Processing Systems , year=

  48. [56]

    Transactions on machine learning research , year=

    From complexity to clarity: Analytical expressions of deep neural network weights via clifford algebra and convexity , author=. Transactions on machine learning research , year=

  49. [57]

    and de Keninck, Steven and Welling, Max and Brandstetter, Johannes , title =

    Ruhe, David and Gupta, Jayesh K. and de Keninck, Steven and Welling, Max and Brandstetter, Johannes , title =. International Conference on Machine Learning , year =

  50. [58]

    International Conference on Machine Learning , year=

    Clifford-Steerable Convolutional Neural Networks , author=. International Conference on Machine Learning , year=

Pith tools

Reviewed June 27, 2026 · model on record in the stance chip above.