Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices

T0 review · 3 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read KromHC parametrizes residual mixing as a Kronecker product of small doubly stochastic matrices, guaranteeing exact doubly stochasticity with O(n^2C) parameters.

desk verdict A genuinely useful parametrization of residual matrices for Hyper-Connections, with a clean proof and real parameter savings; the empirical evidence, however, is too thin to back the scalability claims. read the letter →

arxiv 2601.21579 v2 pith:6ZC5MP3X submitted 2026-01-29 cs.CL cs.LG

Wuyang Zhou , Yuxuan Gu , Giorgos Iacovides , Danilo Mandic This is my paper
classification cs.CLcs.LG MSC 15B5115A6968T07
keywords KroneckerproductdoublystochasticmatricesBirkhoffpolytopemanifold-constrainedhyper-connectionsresidualconnectionstransformerpretrainingtensornetworksparameterefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

KromHC settles a trade-off in manifold-constrained hyper-connections: either the residual mixing matrix is only approximately doubly stochastic (mHC's Sinkhorn-Knopp projection) or exactly doubly stochastic but with factorial parameter growth (mHC-lite's Birkhoff-von-Neumann parametrization). The paper builds the n×n residual matrix as a Kronecker product of smaller doubly stochastic factor matrices, one per mode of a tensorized residual stream. Because the Kronecker product of doubly stochastic matrices is again doubly stochastic, row and column sums are exact by construction while parameter complexity drops to O(n^2C). In LLM pretraining at 60M and 186M scale with four residual streams, KromHC matches or beats existing mHC variants on training loss, validation bits-per-byte, CORE score, commonsense reasoning, and language-modeling averages, with fewer added parameters; widening the stream to n=8 or n=16 further lowers training loss and validation BPB. If the restriction holds at larger widths, this gives a stable, parameter-lean way to scale residual-stream mixing in transformers.

What carries the argument

The central object is the Kronecker-product residual matrix H_res^l = U^K_l ⊗ ... ⊗ U^1_l, with each U^k_l a learnable point in the Birkhoff polytope B_{i_k}, the set of nonnegative matrices whose rows and columns sum to one. This is a Tucker-structured tensor network applied to the tensorized residual stream: mode-n products with the factors along each residual-stream mode, and an identity along the feature mode. The two load-bearing identities are Birkhoff-von-Neumann's theorem, which lets each small factor be a convex combination of permutation matrices, and Theorem 4.2, Kronecker closure of doubly stochastic matrices, which guarantees that the assembled H_res^l is exactly doubly stochast

What would settle it

At n=8 or n=16, train the same 12-block model with an exact but unrestricted doubly stochastic residual matrix — for example, a full Sinkhorn-Knopp projection run long enough to converge — and compare training loss, BPB, and downstream scores with KromHC. If the unrestricted model's advantage grows with n, the Kronecker restriction is the bottleneck. A cheaper check: take a trained mHC residual matrix and compute its best Kronecker-product approximation; if the approximation error is large at the widths KromHC targets, the restriction is likely costly.

Watch

Extended reading notes

Core claim

At the heart of KromHC is a Kronecker-product parametrization of the residual matrix. Given residual-stream width n=∏_{k=1}^K i_k, the input is tensorized into an order-(K+1) tensor, and mixing is applied along each residual-stream mode by a small doubly stochastic factor matrix U^k_l ∈ B_{i_k}; the feature mode is left untouched. Unfolding the result gives H_res^l X_l = (U^K_l ⊗ ... ⊗ U^1_l)X_l. Each factor is itself learned as a convex combination of i_k! permutation matrices (Birkhoff-von-Neumann theorem), and Theorem 4.2 proves that the Kronecker product of doubly stochastic matrices is doubly stochastic, so H_res^l is always exactly doubly stochastic, not approximately. This preserves n

Load-bearing premise

The load-bearing premise is that the Kronecker-product sub-polytope is expressive enough that replacing the full set of doubly stochastic matrices with small-factor Kronecker products does not hurt residual mixing; the paper's strongest benchmark evidence is at n=4, while the n=8 and n=16 evidence covers only training loss and validation BPB, and a large prime n would collapse the construction back to factorial cost.

Editorial extensions

If this is right

  • Every residual matrix, and every product of residual matrices across layers, is exactly doubly stochastic, so the column-sum drift that accumulates in mHC is zero by construction.
  • The method is PyTorch-native: no specialized Sinkhorn-Knopp kernels are needed, only Kronecker products of small convex-combination factors.
  • At n=4, the 60M and 186M models trained with KromHC reached the best CORE score and best average commonsense and language-modeling accuracies among the compared mHC variants, while adding fewer parameters.
  • Increasing residual-stream width from 4 to 16 lowered training loss and validation bits-per-byte in the reported 12-block runs, suggesting the parametrization remains useful as n grows.
  • Because H_res^l is exactly doubly stochastic, the identity-mapping and norm-preservation guarantees of mHC carry over unchanged.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural test beyond the paper's tables: run the n=4 downstream benchmarks at n=8 and n=16. If the loss and BPB gains shown there translate to CORE and reasoning scores, the Kronecker restriction is not bottlenecking expressivity; if they do not, the scalability claim would be limited to training metrics.
  • The factorization choice is a tunable expressiveness-versus-efficiency knob: powers of two give the smallest factors (i_k=2, only 2 permutation matrices each), while larger or mixed factors give a richer subset of the Birkhoff polytope at higher cost. That trade-off could be explored adaptively per layer.
  • For widths with a large prime factor, the parametrization degrades toward mHC-lite's factorial cost; padding to a nearby power of two or three, as the paper suggests but does not test, would be a practical workaround worth validating.
  • More broadly, this gives a blueprint for any module that needs a cheap, exactly doubly stochastic linear map: tensorize the domain, apply small convex-combination permutation factors per mode, and unfold; the Kronecker closure theorem guarantees the constraint.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper proposes KromHC, a variant of manifold-constrained Hyper-Connections in which the n×n residual mixing matrix H_res^l is parametrized as a Kronecker product of smaller doubly stochastic factor matrices, each built as a convex combination of permutation matrices. The construction guarantees exact double stochasticity (Theorem 4.2, proved in Appendix B) and reduces parameter complexity to O(n^2 C). Experiments on LLM pretraining (D=6 and 12, n=4) compare KromHC against standard residual connections, mHC, and mHC-lite on training loss, validation BPB, CORE score, commonsense reasoning, and language modeling; KromHC is reported to match or outperform baselines with fewer added parameters. Scaling experiments for n=8 and n=16 are reported only as training-loss and validation-BPB gaps relative to KromHC's own n=4 baseline.

Significance. If the empirical claims hold, KromHC offers a simple, PyTorch-native way to obtain exactly doubly stochastic residual matrices without the O(n^3C) cost of iterative Sinkhorn-Knopp projection or the factorial cost of mHC-lite. The mathematical core is sound: Theorem 4.2 is correctly proved in Appendix B, and the parameter-count analysis is transparent. The release of code and the use of a standard pretraining recipe improve reproducibility. However, the central scalability claim—that KromHC remains competitive as the residual-stream width n grows—is not directly supported by the experiments, because the n=8 and n=16 results are self-relative only and lack comparisons against mHC/mHC-lite.

major comments (3)
  1. [§5.4, Figure 4, Table 6] The scaling experiments report only KromHC's training-loss and validation-BPB gaps relative to its own n=4 run. There is no comparison against mHC or mHC-lite at n=8 or n=16, and no downstream evaluation (CORE, commonsense, language modeling) at these widths. The abstract and introduction claim KromHC 'matches or even outperforms' SOTA mHC variants while scaling; for n>4 this is not demonstrated. Given that the Kronecker restriction leaves only K=log2(n) free parameters in H_res^l (for i_k=2) versus (n-1)^2 for the full Birkhoff polytope, the expressive-capacity risk is real. This gap is load-bearing for the scalability contribution.
  2. [Tables 3–5] No seeds, error bars, or significance tests are reported. Several numerical differences are tiny (e.g., Table 3, D=6: training loss 3.488 for KromHC vs 3.493 for mHC; validation BPB for KromHC and Residual are both 1.047), while individual downstream tasks fluctuate widely (e.g., Table 4, D=6: COPA 54.0 for KromHC vs 60.0 for mHC; Table 5, D=12: BBH-CS 13.6 for KromHC vs 14.7 for mHC-lite). The claim of 'matches or outperforms' rests on averages over a small number of runs; without variance estimates, the improvements are not established as robust.
  3. [§4.3, Remark 4.5] The paper emphasizes the O(n^2C) parameter complexity, but this count is dominated by H_pre and H_post projections. The residual matrix H_res^l itself has only K = log2(n) free parameters when i_k=2, whereas the unconstrained Birkhoff polytope has dimension (n-1)^2. The authors acknowledge in the Limitations that large prime n causes a parameter issue, but they do not address whether the logarithmic-dimensional Kronecker subset remains expressive enough as n grows. This is a conceptual point that should be discussed explicitly, especially since the n=4 experiments are the only place where KromHC is compared with mHC variants.
minor comments (5)
  1. [§5.4] The text states the experiments use n∈{4,8,12}, but Figure 4 and Table 6 report n=16. Please correct the inconsistency.
  2. [Table 5] In the D=12 mHC-lite row, '14.76' appears to be a formatting error; it should likely read '14.7'.
  3. [Notation, §4.2] The expression igotimes_{k=K}^{1} U^k_l is nonstandard; define it explicitly (e.g., as U^K_l ⊗ … ⊗ U^1_l) to avoid confusion with the index order.
  4. [Appendix G] There is a typo: 'calcualted' should be 'calculated'.
  5. [Figure 3] The exact parameter expressions shown in the legend are useful; consider also including the per-layer parameter formula from §4.3 in the caption or text for readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the double-stochasticity guarantee is proved from standard Kronecker-product and Birkhoff facts, not fitted or self-referential.

full rationale

The claimed derivation chain is self-contained. Theorem 4.2 (Kronecker closure of doubly stochastic matrices) is proven in Appendix B directly from non-negativity, row/column sums, and the Kronecker mixed-product identity; it does not import the paper's own empirical claims. The factor matrices are parametrized as convex combinations of permutation matrices via the Birkhoff–von-Neumann theorem, an external standard result, and their convex coefficients are learned. The guaranteed double stochasticity of H^res_l is therefore a mathematical consequence of the construction, not a fitted prediction. The zero-MAE statement in Figure 2 is a mathematical consequence for exactly doubly stochastic matrices, and is presented as a stability property, not as an empirical discovery obtained by fitting. Parameter counts are direct enumerations of the parametrization in Eq. (14). The paper's self-citations (tensor-network surveys, TensorLLM/TeRA) are contextual related work and are not load-bearing for Theorem 4.2 or the architecture's guarantee. The scaling experiments in Section 5.4 report only self-relative gaps and do not compare against mHC at n=8/16; this is an evidential limitation rather than a circular reduction. No fitted parameter is renamed as a prediction, and no load-bearing argument reduces to a self-citation.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The theoretical guarantee rests on standard linear algebra (Birkhoff-von Neumann, Kronecker product properties) and a domain assumption from prior mHC work. The only hand-chosen constants are initialization values, which do not affect the mathematical guarantee. The empirical claim additionally assumes the Kronecker restriction is expressive enough; this is weakly validated.

free parameters (1)
  • Initialization constants (b_res,k, α_res, α_pre, α_post, b_pre, b_post) = b_res=[0,-8]; α=0.01; b_pre/b_post=-1 except one index set to 1
    Hand-chosen in §5.1 to initialize near identity. They affect the experimental results but not the doubly-stochastic guarantee, which is structural.
assumptions (5)
  • standard math Birkhoff-von Neumann theorem: every doubly stochastic matrix is a convex combination of permutation matrices.
    Invoked in Theorem 4.1 and Eq. (14) to parametrize each factor matrix U_k^l. Standard and true.
  • standard math Kronecker product identity (A⊗B)(C⊗D) = (AC)⊗(BD), and the closure of doubly stochastic matrices under Kronecker products.
    Used in Appendix B to prove Theorem 4.2. The proof is correct.
  • standard math Tensorizing the residual stream and applying mode-n products with factor matrices is equivalent to applying a Kronecker-product residual matrix to the flattened stream.
    Section 4.1, Eq. (9)-(10). Standard Tucker/Kronecker algebra.
  • domain assumption Residual mixing stability is governed by the residual matrices being doubly stochastic (norm ≤ 1, compositional closure).
    Adopted from mHC (Xie et al., 2025) and used to justify KromHC's stability claims; not re-derived in this paper.
  • domain assumption The Kronecker-product restriction of the Birkhoff polytope is expressive enough to match the performance of full doubly stochastic parametrizations.
    Implicit in Section 4.2 and tested only partially: full benchmarks at n=4, loss-only at n=8,16. If false, the matching/outperforming claim fails.

how reviews work

0 comments
Cite this review

Pith. "Pith review of KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices." pith.science (2026). https://pith.science/paper/6ZC5MP3X

@misc{pith2026260121579,
  author       = {Pith},
  title        = {Pith review of: KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6ZC5MP3X}},
  note         = {Machine review of arXiv:2601.21579}
}
abstract

The success of Hyper-Connections (HC) in neural networks (NN) has also highlighted issues related to training instability and restricted scalability. The Manifold-Constrained Hyper-Connections (mHC) mitigate these challenges by projecting the residual connection space onto a Birkhoff polytope, however, it faces two issues: 1) its iterative Sinkhorn-Knopp (SK) algorithm does not always yield exactly doubly stochastic residual matrices; 2) mHC incurs a prohibitive $O(n^3C)$ parameter complexity with $n$ as the width of the residual stream and $C$ as the feature dimension. The recently proposed mHC-lite reparametrizes the residual matrix via the Birkhoff-von-Neumann theorem to guarantee double stochasticity, but also faces a factorial explosion in its parameter complexity, $O \left( nC \cdot n! \right)$. To address both challenges, we propose KromHC, which uses the Kronecker products of smaller doubly stochastic matrices to parametrize the residual matrix in mHC. By enforcing manifold constraints across the factor residual matrices along each mode of the tensorized residual stream, KromHC guarantees exact double stochasticity of the residual matrices while reducing parameter complexity to only $O(n^2C)$. Experiments show that KromHC matches or even outperforms other state-of-the-art (SOTA) mHC variants, while requiring significantly fewer trainable parameters. The code is at https://github.com/wz1119/KromHC.

Figures

Figures reproduced from arXiv: 2601.21579 by the authors.

Figure 1
Figure 1. Illustration of variants of manifold-constrained hyper-connections with a residual stream width n = 8. (a) mHC: utilizes iterative Sinkhorn-Knopp (SK) algorithm to approximate a doubly stochastic residual matrix; (b) mHC-lite: builds the residual matrix as convex combinations of n! permutation matrices, but becomes infeasible for a large n; (c) KromHC (Ours): constructs the residual matrix as the Kronecker products … view at source ↗
Figure 2
Figure 2. Numerical stability analysis of the products of residual matrices. The plot compares the Mean Absolute Error (MAE) between the column sum of QL−1 i=0 Hres L−i and 1 in an LLM with D = 12 transformer blocks and L = 24 layers of HC. The stan￾dard mHC architecture exhibits a MAE of around 0.05, indicating potential training instabilities. The mHC-lite and KromHC have exact doubly stochastic matrices, thus yielding zero… view at source ↗
Figure 3
Figure 3. The number of learnable parameters against the number of residual streams, n, per hyper-connection in mHC, mHC-lite, and KromHC. We assume the feature dimension, C, to be 512. Also, n is factored into Qlog2(n) m=1 2, i.e., i1 = i2 = · · · = iK = 2. To answer this question, we propose KromHC, which uses the Kronecker products (Van Loan, 2000) of smaller doubly stochastic matrices to parametrize the residual matrix in… view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: presents the gradient norm trajectories across the last 2000 training steps. Identical model configurations (12 transformer blocks and n = 4 residual streams) were used for mHC, mHC-lite and our KromHC. It is worth noting that our KromHC consistently achieved the lowes…
Figure 4
Figure 4. Figure 4: Training loss and validation BPB gaps of KromHC at different widths of the residual stream, n, compared to n = 4. Exponential Moving Average (EMA) is applied to the raw loss before the calculation of the loss gap. 5.5. Gradient Norm [PITH_FULL_IMAGE:figures/full_fig_p…
Figure 7
Figure 7. Figure 7: Tensor network diagram of the proposed KromHC method. circle, with each line emanating from the circle correspond￾ing to a tensor mode index. Also, connecting two index lines implies a tensor contraction over the connected mode indices. E. Parametrization of HC In this…
Figure 8
Figure 8. Figure 8: shows the raw gradient norm across 7000 training steps of mHC, mHC-lite and KromHC at D = 12. 1000 2000 3000 4000 5000 6000 7000 Steps 0.0 0.5 1.0 1.5 2.0 Gradient Norm mHC mHC-lite KromHC [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopes

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    TBP-mHC proposes parameterizations of the Birkhoff polytope via transportation polytopes that achieve exact double stochasticity for hyper-connections using only (n-1)^2 degrees of freedom.

Reference graph

Works this paper leans on

24 extracted references · 12 linked inside Pith · cited by 1 Pith paper

  1. [1]

    LoTR: Low Tensor Rank Weight Adap- tation.arXiv preprint arXiv:2402.01376,

    Bershatsky, D., Cherniuk, D., Daulbaev, T., Mikhalev, A., and Oseledets, I. LoTR: Low Tensor Rank Weight Adap- tation.arXiv preprint arXiv:2402.01376,

    arXiv

  2. [8]

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hen- dricks, L. A., Welbl, J., Clark, A., et al. Train- ing Compute-Optimal Large Language Models.arXiv preprint arXiv:2203.15556,

    arXiv

  3. [11]

    net/forum?id=Bkg6RiCqY7

    URL https://openreview. net/forum?id=Bkg6RiCqY7. Mak, B. and Flanigan, J. Residual Matrix Transformers: Scaling the Size of the Residual Stream.arXiv preprint arXiv:2506.22696,

    arXiv

  4. [12]

    Common- senseQA: A Question Answering Challenge Targeting Commonsense Knowledge

    Talmor, A., Herzig, J., Lourie, N., and Berant, J. Common- senseQA: A Question Answering Challenge Targeting Commonsense Knowledge. InProceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4149–4158,

    2019

  5. [13]

    Tensor Networks Meet Neural Net- works: A Survey and Future Perspectives.arXiv preprint arXiv:2302.09019,

    Wang, M., Pan, Y ., Xu, Z., Li, G., Yang, X., Mandic, D., and Cichocki, A. Tensor Networks Meet Neural Net- works: A Survey and Future Perspectives.arXiv preprint arXiv:2302.09019,

    arXiv

  6. [14]

    Muddformer: Breaking residual bottlenecks in transformers via mul- tiway dynamic dense connections.arXiv preprint arXiv:2502.12170,

    Xiao, D., Meng, Q., Li, S., and Yuan, X. Muddformer: Breaking residual bottlenecks in transformers via mul- tiway dynamic dense connections.arXiv preprint arXiv:2502.12170,

    arXiv

  7. [15]

    H., Menezes, A., Qin, T., and Yan, R

    Xie, S., Zhang, H., Guo, J., Tan, X., Bian, J., Awadalla, H. H., Menezes, A., Qin, T., and Yan, R. ResiDual: Trans- former with Dual Residual Connections.arXiv preprint arXiv:2304.14802,

    arXiv

  8. [16]

    mHC: Manifold-Constrained Hyper-Connections.arXiv preprint arXiv:2512.24880,

    Xie, Z., Wei, Y ., Cao, H., Zhao, C., Deng, C., Li, J., Dai, D., Gao, H., Chang, J., Zhao, L., et al. mHC: Manifold-Constrained Hyper-Connections.arXiv preprint arXiv:2512.24880,

    arXiv

Show all 24 references
  1. [17]

    L., and Mandic, D

    Xu, M., Xu, Y . L., and Mandic, D. P. TensorGPT: Efficient Compression of Large Language Models based on Tensor- Train Decomposition.arXiv preprint arXiv:2307.00526,

    arXiv

  2. [18]

    and Gao, J

    Yang, Y . and Gao, J. mHC-lite: You Don’t Need 20 Sinkhorn-Knopp Iterations.arXiv preprint arXiv:2601.05732,

  3. [20]

    URL https://aclanthology.org/P19-1472/

    doi: 10.18653/v1/P19-1472. URL https://aclanthology.org/P19-1472/. Zhong, W., Cui, R., Guo, Y ., Liang, Y ., Lu, S., Wang, Y ., Saied, A., Chen, W., and Duan, N. AGIEval: A Human- Centric Benchmark for Evaluating Foundation Models. 10 KromHC: Manifold-Constrained Hyper-Connect...

    2024 doi

  4. [21]

    There are 22 selected tasks, where in each task, accuracy is linearly scaled so that 0 indicates random- guess performance and 1 implies perfect accuracy

    proposed the CORE metric to provide a robust low-variance, centered accuracy score for LLM evaluation. There are 22 selected tasks, where in each task, accuracy is linearly scaled so that 0 indicates random- guess performance and 1 implies perfect accuracy. The final CORE scor...

    2024

  5. [22]

    introduces learnable per-layer scalars which improves the model performance. More specifically, each layer’s input is calcualted as ˜xl =λ resid l · 13 KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices xl +λ x0 l ·x 0, where x0 is the init...

    2024

  6. [23]

    Additionally, following the best practice in Karpathy (2025), different learning rates (LR) are used for embedding layer, LM layer, main branch and hyper-connections branch

    is used for the hyper-connections streams, embedding layer, and lan- guage modeling (LM) head layer. Additionally, following the best practice in Karpathy (2025), different learning rates (LR) are used for embedding layer, LM layer, main branch and hyper-connections branch. Ta...

    2025

  7. [24]

    Raw gradient norm across 7000 training steps

    1000 2000 3000 4000 5000 6000 7000 Steps 0.0 0.5 1.0 1.5 2.0Gradient Norm mHC mHC-lite KromHC Figure 8.Gradient norm dynamics across training. Raw gradient norm across 7000 training steps. 14

    2000

  8. [1946]

    Highway Transformer: Self- Gating Enhanced Self-Attentive Networks.arXiv preprint arXiv:2004.08178,

    Chai, Y ., Jin, S., and Hou, X. Highway Transformer: Self- Gating Enhanced Self-Attentive Networks.arXiv preprint arXiv:2004.08178,

    2004 arXiv

  9. [2000]

    Cross-Layer Retrospective Retrieving via Layer Atten- tion.arXiv preprint arXiv:2302.03985,

    Fang, Y ., Cai, Y ., Chen, J., Zhao, J., Tian, G., and Li, G. Cross-Layer Retrospective Retrieving via Layer Atten- tion.arXiv preprint arXiv:2302.03985,

    arXiv

  10. [2015]

    Cichocki, A., Lee, N., Oseledets, I., Phan, A.-H., Zhao, Q., Mandic, D

    doi: 10.1109/MSP.2013.2297439. Cichocki, A., Lee, N., Oseledets, I., Phan, A.-H., Zhao, Q., Mandic, D. P., et al. Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 1 Low- rank Tensor Decompositions.Foundations and Trends® in Machine Learning, 9(4-...

    2013

  11. [2016]

    BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

    Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human ...

    2019

  12. [2019]

    URL https://aclanthology.org/N19-1300/

    doi: 10.18653/v1/N19-1300. URL https://aclanthology.org/N19-1300/. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Chal- lenge.arXiv preprint arXiv:1803.05457,

    arXiv

  13. [2023]

    Tensor- LLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMs

    Gu, Y ., Zhou, W., Iacovides, G., and Mandic, D. Tensor- LLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMs. InProceedings of 2025 International Joint Conference on Neural Networks (IJCNN), pp. 1–8, 2025a. doi: 10.1109/IJCNN64981. 2025.1122858...

    2025

  14. [2024]

    Muon is Scalable for LLM Training.arXiv preprint arXiv:2502.16982,

    Liu, J., Su, J., Yao, X., Jiang, Z., Lai, G., Du, Y ., Qin, Y ., Xu, W., Lu, E., Yan, J., et al. Muon is Scalable for LLM Training.arXiv preprint arXiv:2502.16982,

    arXiv

  15. [2025]

    Accessed: 2026-01-13

    URL https://github.com/ karpathy/nanochat. Accessed: 2026-01-13. Kocijan, V ., Davis, E., Lukasiewicz, T., Marcus, G., and Morgenstern, L. The Defeat of the Winograd Schema Challenge.Artificial Intelligence, 325:103971,

    2026

  16. [2026]

    LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra- Low-Parameter Fine-Tuning of Large Language Mod- els

    Yang, Y ., Zhou, J., Wong, N., and Zhang, Z. LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra- Low-Parameter Fine-Tuning of Large Language Mod- els. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics:...

    2024

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.