Language as a Wave Phenomenon: Semantic Phase Locking and Interference in Neural Networks

Alper Y{\i}ld{\i}r{\i}m; \.Ibrahim Y\"uceda\u{g}

REVIEW 5 major objections 5 minor 23 references

Language as a Wave Phenomenon: Semantic Phase Locking and Interference in Neural Networks

T0 review · 5 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that the phase angle of a complex-valued representation can carry semantic reasoning, and that subtractive interference is a sufficient primitive for language modeling at the evaluated scale.

desk verdict A clearly written complex-valued architecture paper with a promising idea, but the central claim that phase interference drives the reported gains is not supported by the current experiments. read the letter →

arxiv 2512.01208 v5 pith:TPABN6PY submitted 2025-12-01 cs.LG cs.AIcs.CL

Alper Y{i}ld{i}r{i}m , İbrahim Yücedağ This is my paper

classification cs.LGcs.AIcs.CL

keywords phasecodingsubtractiveinterferencecomplex-valuedneuralnetworkssemanticlockingspectralreasoninghybridarchitecturelanguagemodelingmechanisticinterpretability

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The authors aim to separate two ways a neural network can encode meaning: by signal strength (magnitude, as in standard Transformers) or by signal direction (phase, as in waves). They build a complex-valued encoder, PRISM, that forces meaning into phase by requiring unit magnitude and using harmonic convolutions instead of attention. Their hybrid Wave-Particle Transformer mixes a small attention stream with a phase-based spectral stream and reaches 4.94 perplexity on a masked-language benchmark versus 5.28 for a standard Transformer, using 18% fewer core reasoning parameters. Mechanistic probes show that synonyms and antonyms become phase-aligned while random words do not, and that ambiguity resolution happens through a sharp phase rotation with near-constant magnitude. If the claim holds, it would establish that reasoning can be done by erasing unwanted signal through interference rather than only by amplifying wanted signal.

What carries the argument

The key machinery is the Phase-Rotating Interference Spectral Model (PRISM), a complex-valued sequence encoder that treats tokens as phasors z = r·e^(iθ) with semantic identity in the angle θ. A strict unit-norm constraint (|z| = 1) removes magnitude as a carrier of importance. Attention is replaced by a Gated Harmonic Convolution: the sequence is Fourier-transformed, multiplied by a learnable global filter, transformed back, and gated separately on real and imaginary parts so the phase angle can be steered. The Wave-Particle Transformer extends this by running a small attention-based 'sensory' stream in parallel with a phase-based 'relational' stream and fusing them with a one-layer attenti

What would settle it

Train a real-valued control that matches every non-phase ingredient of the Wave-Particle Transformer (skip connection, same normalization, dual-stream layout, attention refiner, weight decay 0.0) with magnitudes only and compare perplexity; or ablate the trained model by randomizing phase angles while preserving magnitudes. If the real-valued control matches 4.94 PPL, or if phase randomization leaves performance largely intact, the phase-mechanism claim is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that semantic relationships can be encoded geometrically as phase alignment in a fixed-magnitude complex space: synonyms lock phase, antonyms align too, and random pairs sit at a noise floor. The model resolves polysemy by applying a heavy-tailed phase rotation at a specific layer while keeping gain near 1.0, which the authors interpret as ruling out magnitude-based 'shouting'. The headline result is that the Wave-Particle Transformer, which combines a lightweight attention stream with a phase-based spectral stream, achieves 4.94 perplexity on WikiText-103 masked reconstruction versus 5.28 for a standard Transformer, with 18% fewer core reasoning parameters. The auth

Load-bearing premise

The central claim rests on the assumption that the perplexity advantage of the hybrid model comes from phase interference itself, rather than from the extra architectural and regularization differences—the skip connection, the different normalization, the dual-stream topology, the added attention refiner, and the use of weight decay 0.0 for the phase model versus 0.01 for the baselines.

Editorial extensions

If this is right

If phase can carry semantic reasoning, a model can suppress noise by destructive interference instead of amplifying signal, suggesting that reasoning requires the capacity to erase, not just accumulate.
Hybrid architectures that decouple intensity (magnitude) from direction (phase) can match or beat a standard Transformer with fewer active reasoning parameters, implying that not every reasoning operation needs quadratic attention.
Sequence length matters not only as context but as spectral resolution: longer sequences provide denser interference fields, so the model's performance should continue to benefit from long inputs in ways that rate-only models may not.
A single lightweight attention layer at the output can serve as a readout for a phase-based encoder, suggesting that deep attention stacks can be partially replaced by passive spectral mixing.
Purely additive spectral mixing is insufficient for contextual reasoning at long sequence lengths; a phase-based subtractive component appears necessary to prevent collapse.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Our inference: if phase coherence is genuinely the carrier of semantic relatedness, interpretability tools could track phase alignment rather than attention weights or activation magnitudes, and phase coherence could become a diagnostic for semantic closeness in complex-valued models.
Our inference: the 'spectral starvation' account of short-input collapse is testable by running a rate-based recurrent baseline on isolated tokens; if that baseline also collapses, the phenomenon may be about insufficient context rather than specifically about phase.
Our inference: the architecture points toward a concrete hardware roadmap—a passive optical unit for O(N log N) interference and a small digital unit for attention—but the paper itself does not demonstrate such a device, only the algorithmic blueprint.
Our inference: the phase-locking of antonyms suggests that the model groups concepts by shared topic rather than by polarity, which, if general, would imply that phase encodes topical relatedness and that logical opposition is represented by an offset within a common phase band.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is a well-organized exploration of complex-valued phase coding, and I want to give it credit where it's due. The specific architecture family (PRISM, HSSM, WPT) and the WikiText-103 numbers at 33M scale are new. They release code, they use FNet as a spectral control, they try to match parameter counts, and they are upfront that this is a mechanistic probe rather than a foundation-model race. The phase-locking measurements, synonyms > antonyms > random, are a reasonable sanity check even if small and possibly driven by embedding similarity. The rate-versus-phase distinction is worth taking seriously.

But the load-bearing claim does not hold up. The main result in Table 4 compares WPT against a Transformer baseline while changing weight decay (0.0 for PRISM variants, 0.01 for baselines), normalization (RMSNorm vs LayerNorm), topology (dual-stream, skip connections), and adding a 1-layer attention refiner. Those are not phase-specific changes, so the perplexity gap cannot be attributed to phase coding. The abstract promises an interventional phase-disruption ablation, but I could not find it in the body. That is a real omission, not just an overstatement. The L=1/L=2 "spectral starvation" experiment also lacks a rate-coding baseline at those lengths; repetition collapse could simply be insufficient context.

Some smaller issues reinforce the pattern. The iso-energetic gain near 1.0 is partly a consequence of the Phase-Preserving Layer Normalization, which standardizes magnitudes, so it is not independent evidence that the model encodes meaning in phase. The R values in Table 2 are small (0.07–0.20) and come from about 120 pairs, without controlling for magnitude or frequency. And setting weight decay to zero for PRISM is itself a regularization difference that could affect generalization independently of phase.

These are not minor quibbles; the mechanistic conclusion rests on the confounded comparison. Still, the paper is coherent on its own terms and engages honestly with the literature. It deserves a serious referee, because the question matters and the authors have done real experimental work. The referee should reject the current version but ask for an actual phase-disruption ablation, matched baselines, and control experiments at L=1/L=2. If those come through, the story could be compelling. As it stands, I would not cite it as evidence for phase-based reasoning.

Referee Report

5 major / 5 minor

Summary. The paper introduces PRISM, a complex-valued architecture that imposes unit-norm phasors and gated harmonic convolutions, and claims that phase-based spectral interference can serve as a sufficient computational primitive for semantic reasoning. It reports phase-locking statistics, a qualitative spectral-density threshold, and a hybrid Wave-Particle Transformer (WPT) that achieves lower perplexity than a Transformer baseline. The paper concludes that subtractive interference is an algorithmic existence proof for semantic reasoning and proposes a blueprint for optical-digital co-processors.

Significance. If substantiated, the proposal would be a novel contribution to mechanistic interpretability and efficient sequence modeling. The paper has useful ingredients: it explicitly constrains magnitude to isolate phase, compares against an FNet spectral baseline, defines physical metrics, and releases code. However, the current experiments do not isolate the phase mechanism. The key WPT comparison is confounded by architectural and regularization differences, the abstract-promised interventional phase-disruption ablation is absent, and the iso-energetic gain evidence is partly circular. The significance therefore remains speculative until these controls are provided.

major comments (5)

[§6.1–6.2, Table 4] The central quantitative claim is the WPT improvement over the Transformer (4.94 vs 5.28 PPL). This comparison is not attributable to phase coding: §6.1 sets weight decay to 0.0 for PRISM Hybrid but 0.01 for baselines; §6.2 gives WPT RMSNorm instead of LayerNorm, a skip connection, a dual-stream topology, and a 1-layer attentive refiner, all absent from the Transformer baseline. In addition, the pure PRISM Hybrid is worse than the Transformer (6.06 vs 5.28 PPL), which undercuts the 'sufficient primitive' language in §8.2. A matched control with identical architecture and regularization except for the phase mechanism is required.
[Abstract; §5–§7] The abstract promises 'interventional ablations' showing that preserving phase largely maintains performance while disrupting phase causes severe degradation. No such experiment appears in the manuscript. §5.2 compares ambiguous and unambiguous tokens but does not perturb phase; §7 is correlational. Without an explicit phase-disruption intervention (e.g., randomizing angles while keeping magnitudes), the causal role of phase is not tested.
[§3.5, Eq. (8); §5.2] The reported iso-energetic gain g≈1.0 is partly a consequence of the Phase-Preserving Layer Normalization in Eq. (8), which standardizes magnitudes, together with the enforced unit-norm constraint. Using g≈1.0 as empirical validation of phase coding is therefore circular. The metric should be computed on pre-normalization activations, or the claim should be removed.
[§5.4] The claim that L=1/L=2 repetition collapse is due to 'spectral starvation' is not tested against a rate-coding baseline. A Transformer or FNet model at the same short lengths would also be expected to fail or repeat; without this control, the result is consistent with a trivial lack of context. No error bars or seeds are reported for this experiment, and the description is qualitative.
[§5.1, Eq. (13); Tables 2, 5] The phase-locking differences are small (e.g., mean R 0.198 vs 0.072), computed on N=122–133 pairs, with no error bars, significance tests, or multiple seeds. Because Eq. (13) weights by |z_a,k||z_b,k|, the metric can be driven by ordinary magnitude/similarity correlations rather than phase. A control that randomizes phase while preserving magnitude is needed to establish that the coherence is phase-specific.

minor comments (5)

[§5.1 vs Table 2] The text reports antonym mean R=0.117, while Table 2 reports 0.133. These values should be reconciled.
[§7 vs Table 5] The text says FNet exhibits 'R≈0.61, Table 5', but Table 5 lists FNet synonym R=0.49, antonym R=0.61, random R=0.12. Please clarify which category is being quoted.
[§3.1, Eq. (1)] The notation ω_k=1/10000^{k/d} is described as a spectrum of geometric frequencies; please define the domain of k and the resulting units to avoid confusion with the positional phase ωt.
[Appendix G] The heading says 'Appendix H' while the section letter is G. The numbering should be corrected.
[§4.2, Table 1] PRISM (U) has 128.4M parameters while the others are 69–73M; please clarify whether this is intentional and why it is included in the fairness discussion.

Circularity Check

1 steps flagged · score 6.0 of 10

Iso-energetic validation is imposed by Phase-Preserving Layer Normalization; the main benchmark comparison is confounded but not definitionally circular.

self definitional [§3.5 Eq. (8); §4.4 Eq. (10); §5.2 'Validation of Phase Coding']
"PPLN(z) = z/(|z|+ε) · ((|z|−µ)/√(σ²+ε) · γ + β) ... This decouples the signal’s energy from its phase, reinforcing the Iso-Energetic reasoning regime ... [§5.2] Crucially, Figure 3(b) confirms that this steering is Iso-Energetic. The gain for ambiguous tokens does not diverge from unambiguous tokens (g_amb ≈ g_unamb ≈ 1.0). This empirically rules out amplitude modulation (”shouting”) as a mechanism."

The gain metric g(l)=∥y(l)∥/∥x(l)∥ is measured on a network whose own normalization (Eq. 8) standardizes magnitudes before each layer. The paper explicitly states that PPLN 'reinforces the Iso-Energetic reasoning regime', so g≈1 is a built-in property of the normalization rather than an empirical discovery. Using this forced gain to 'rule out amplitude modulation' validates the model against its own design constraint, not the phase-coding hypothesis.

full rationale

The strongest quantitative claim (WPT 4.94 vs Transformer 5.28 PPL, Table 4) is not circular by construction: the comparison is empirical and the architectures differ in several ways (weight decay 0.0 vs 0.01, RMSNorm vs LayerNorm, skip connection, dual-stream topology, 1-layer refiner). Those are confounds that undermine attribution to phase, but they are not a definitional reduction. The abstract's promised 'interventional ablations' (preserving/disrupting phase) do not appear in the manuscript as described, which is missing support rather than circularity. The one clear circular step is the iso-energetic validation: PPLN (Eq. 8) explicitly normalizes magnitudes and is described as reinforcing the Iso-Energetic regime, so the subsequent g≈1 result in §5.2 is a consequence of the normalization, not independent evidence that the model avoids amplitude modulation. Because this circular validation supports the mechanistic 'phase coding' interpretation but the central benchmark comparison retains independent (if confounded) content, a score of 6 is appropriate.

Assumptions & free parameters 5 free parameters · 5 assumptions · 3 invented entities

The ledger captures the hand-chosen constants and assumptions that carry the paper's mechanistic interpretation. Most learned weights are standard and omitted; the load-bearing choices are listed.

free parameters (5)

RoSE positional frequency base b=10000 = 10000
Eq. (1) sets ω_k = 1/10000^{k/d}; the base is chosen by hand (borrowed from RoPE) and determines the positional phase schedule, a load-bearing part of the phase-encoding claim.
Attenuation factor α≈0.74 = 0.74
§5.3 hand-picks α to simulate passive optical insertion loss; the '97% performance retention' result depends on this value, and no sensitivity analysis is given.
Weight decay (PRISM Hybrid) = 0.0 = 0.0
§6.1 sets weight decay to 0.0 for PRISM Hybrid and 0.01 for all baselines, breaking hyperparameter parity and potentially explaining some of the WPT PPL improvement.
Peak learning rate = 6e-4 = 6e-4
§4.3 restricts all models to 6e-4 because the Transformer baseline was unstable at 8e-4, handicapping PRISM; not a fitted value but an ad hoc constraint.
Dynamic RoSE steering phase ϕ_steer (hypernetwork)
§6.2 introduces a content-dependent phase shift produced by an unspecified hypernetwork; the mechanism is not defined, and it is a learned component of the high-performance WPT that is not present in the static probe.

assumptions (5)

domain assumption FNet's implicit phase is restricted to the discrete set {0, π} because its activations are real.
Section 7 uses this to argue FNet is a 'Binary Limit' while PRISM has continuous phase. This treats the sign of a real activation as a phase angle, which is an interpretation rather than a property of the model.
domain assumption A layer gain g≈1.0 implies the network does not use amplitude modulation.
Section 5.2 concludes 'no cheating via amplitude' from g≈1.0, but the PPLN (Eq. 8) standardizes magnitudes, so near-unity gain is partly a consequence of the chosen normalization, not an independent measurement.
domain assumption Repetition collapse at L=1/L=2 is caused by spectral starvation rather than lack of context.
Section 5.4 and Appendix F interpret the failure as 'insufficient spectral density,' but no rate-based baseline is evaluated at L=1/L=2, so the cause is untested.
domain assumption The FNet control isolates phase coding from all other architectural differences.
Appendix E.1 claims FNet vs PRISM differ only in real vs complex coding, but PRISM also has learnable filters, Cartesian gating, ModReLU, PPLN, and skip connections; the isolation assumption is false.
domain assumption Phase coherence R computed from raw embeddings is a valid measure of semantic locking.
Section 5.1 weights phase coherence by embedding magnitudes and interprets the result as neural phase locking; this assumes learned complex embeddings' angular alignment reflects semantic categories, but it could simply reflect distributional similarity.

invented entities (3)

Global 'carrier wave anisotropy'
purpose: Explains why pure PRISM has high phase coherence (R≈0.49) for random pairs in §7 (Table 5).
The paper postulates a global anisotropy to account for unwanted high baseline coherence; no independent measurement or prediction is provided.
Semantic Phase Compass
purpose: Metaphorical construct for observed phase alignment of synonyms/antonyms in §5.1.
This is a descriptive label for the R values, not a falsifiable entity; no out-of-sample prediction.
Topic Binder
purpose: Explains why antonyms show the highest phase locking in HSSM (§7), reinterpreting negation as shared topic binding.
Ad hoc explanation introduced after the observation; no independent test.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Language as a Wave Phenomenon: Semantic Phase Locking and Interference in Neural Networks." pith.science (2026). https://pith.science/paper/TPABN6PY

@misc{pith2026251201208,
  author       = {Pith},
  title        = {Pith review of: Language as a Wave Phenomenon: Semantic Phase Locking and Interference in Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TPABN6PY}},
  note         = {Machine review of arXiv:2512.01208}
}

abstract

In standard Transformer architectures, semantic importance is often conflated with activation magnitude, obscuring the geometric structure of latent representations. To disentangle these factors, we introduce PRISM, a complex-valued architecture designed to isolate the computational role of phase. By enforcing a strict unit-norm constraint ($|z| = 1$) and replacing attention with gated harmonic convolutions, the model is encouraged to utilize subtractive interference in the frequency domain to suppress noise, rather than relying on magnitude-based gating. We utilize this constrained regime to study a hybrid architecture -- fusing phase-based routing with standard attention -- which achieves improved parameter efficiency and representation quality compared to baselines in our evaluated settings. Mechanistically, interventional ablations indicate that the model carries substantial task-relevant information in phase: preserving phase largely maintains performance, whereas disrupting phase causes severe degradation. Together, these results suggest that phase-based spectral interference is a usable computational mechanism for neural sequence modeling at the evaluated scale.

Figures

Figures reproduced from arXiv: 2512.01208 by the authors.

**Figure 1.** The PRISM Architecture. (Left) Input tokens are encoded as complex phasors via RoSE. (Center) Gated Harmonic Convolutions replace O(N 2 ) attention with O(N log N) spectral filters H. (Right) Iso-Energetic Norm enforces unit gain (g ≈ 1.0), forcing semantic encoding into phase shifts. a complex latent space C d by learning separate real and imaginary components, forming a content vector zt ∈ C d . To encode positio… view at source ↗

**Figure 2.** The Semantic Phase Compass. Distribution of phase differences ∆θ for three representative pairs. (A) Synonyms exhibit strong phase locking, indicating constructive interference. (B) Antonyms also show high coherence, confirming a shared spectral basis. (C) Unrelated words display a uniform, random phase distribution. regime of Neural Phase Locking. Synonymous pairs align their spectral components to maximize interf… view at source ↗

**Figure 4.** The Spectral Density Threshold. Constructive reasoning requires a dense interference field (red/blue fringes). Single tokens (L = 1) and minimal pairs (L = 2) lack sufficient spectral resolution, causing generation collapse, whereas deeper sequences provide the bandwidth for semantic filtering. This behavior directly parallels N-slit diffraction in classical optics, where fringe resolution scales with the number o… view at source ↗

Figures from the paper (3 more)

**Figure 5.** Figure 5: Topological Divergence in Semantic Phase-Locking. While both architectures achieve high Phase Coherence (R), the mechanisms differ fundamentally. (Left) FNet operates in a Binary Phase Regime, restricted to sign flips (0 ◦ vs 180◦ ), forcing a discrete “Digital” logic…

**Figure 6.** Figure 6: The Spectral Efficiency Crossover. Analysis of computational effort (Gate Openness, Red) versus semantic impact (Phase Rotation, Blue). While early layers expend high energy for low impact, a crossover at Layer 2 leads to a high-leverage regime (η > 100) in deep layer…

**Figure 7.** Figure 7: The HSSM Architecture. A dual-stream hourglass topology processing Rate (FNet) and Phase (PRISM) in parallel before fusion. This design decouples additive magnitude accumulation from subtractive phase interference. D.1. Validity Before Extrapolation Furthermore, we ex…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 3 linked inside Pith

[1]

and Lafferty, J

Altabaa, A. and Lafferty, J. Disentangling and integrating relational and sensory information in transformer architectures. In Forty-second International Conference on Machine Learning, 2025

2025
[2]

W., Cohen, J

Altabaa, A., Webb, T. W., Cohen, J. D., and Lafferty, J. Abstractors and relational cross-attention: An inductive bias for explicit relational reasoning in transformers. In Twelfth International Conference on Learning Representations, 2024

2024
[3]

Unitary evolution recurrent neural networks

Arjovsky, M., Shah, A., and Bengio, Y. Unitary evolution recurrent neural networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML'16, pp.\ 1120–1128. JMLR.org, 2016

2016
[4]

Human brain networks function in connectome-specific harmonic waves

Atasoy, S., Donnelly, I., and Pearson, J. Human brain networks function in connectome-specific harmonic waves. Nature Communications, 7 0 (1): 0 10340, 2016

2016
[5]

Principled weight initialization for hypernetworks

Chang, O., Flokas, L., and Lipson, H. Principled weight initialization for hypernetworks. In International Conference on Learning Representations, 2020

2020
[6]

and Gu, A

Dao, T. and Gu, A. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning (ICML), 2024

2024
[7]

Deco, G., Sanz Perl, Y., and Kringelbach, M. L. Complex harmonics reveal low-dimensional manifolds of critical brain dynamics. Physical Review E, 111 0 (1): 0 014410, 2025

2025
[8]

Multi30k: Multilingual english-german image descriptions

Elliott, D., Frank, S., Sima'an, K., and Specia, L. Multi30k: Multilingual english-german image descriptions. In Proceedings of the 5th Workshop on Vision and Language, pp.\ 70--74, 2016

2016

Show all 23 references

[9]

and Dao, T

Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

2023 arXiv
[10]

Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp.\ 1026--1034, 2015

2015
[11]

and Komachi, M

Hirasawa, T. and Komachi, M. Multilingual neural machine translation with nested embedding prediction. In Proceedings of the 33rd Pacific Asia Conference on Language, Information and Computation, 2019

2019
[12]

Y., Kang, B., and Balasubramanian, N

Kim, H. Y., Kang, B., and Balasubramanian, N. On initializing transformers with pre-trained embeddings. arXiv preprint arXiv:2407.12514, 2024

2024 arXiv
[13]

and Tsvetkov, Y

Kumar, S. and Tsvetkov, Y. Von mises-fisher loss for training sequence to sequence models with continuous outputs. In International Conference on Learning Representations, 2019

2019
[14]

Variational autoencoder with disentanglement priors for low-resource task-specific natural language generation

Li, Z., Qu, L., Xu, Q., Wu, T., Zhan, T., and Haffari, G. Variational autoencoder with disentanglement priors for low-resource task-specific natural language generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.\ 10335--10356, 2022

2022
[15]

J., Kay, K

Naselaris, T., Prenger, R. J., Kay, K. N., Oliver, M., and Gallant, J. L. Bayesian reconstruction of natural images from human brain activity. Neuron, 63 0 (6): 0 902--915, 2009

2009
[16]

Nguyen, T. Q. and Salazar, J. Transformers without tears: Improving the normalization of self-attention. arXiv preprint arXiv:1910.05895, 2019

1910 arXiv
[17]

Qi, Y., Sachan, D., Felix, M., Padmanabhan, S., and Neubig, G. When and why are pre-trained word embeddings useful for neural machine translation? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua...

2018
[18]

Roformer: Enhanced transformer with rotary position embedding

Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

2024
[19]

Bert rediscovers the classical nlp pipeline

Tenney, I., Das, D., and Pavlick, E. Bert rediscovers the classical nlp pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.\ 4593--4601, 2019

2019
[20]

and Thottingal, S

Tiedemann, J. and Thottingal, S. Opus-mt-building open translation services for the world. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pp.\ 479--480, 2020

2020
[21]

N., Kaiser, ., and Polosukhin, I

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017

2017
[22]

W., Frankland, S

Webb, T. W., Frankland, S. M., Altabaa, A., Segert, S., Krishnamurthy, K., Campbell, D., Russin, J., Giallanza, T., O'Reilly, R., Lafferty, J., and Cohen, J. D. The relational bottleneck as an inductive bias for efficient abstraction. Trends in Cognitive Sciences, 28 0 (9): 0 ...

2024
[23]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [1]

and Lafferty, J

Altabaa, A. and Lafferty, J. Disentangling and integrating relational and sensory information in transformer architectures. In Forty-second International Conference on Machine Learning, 2025

2025

[2] [2]

W., Cohen, J

Altabaa, A., Webb, T. W., Cohen, J. D., and Lafferty, J. Abstractors and relational cross-attention: An inductive bias for explicit relational reasoning in transformers. In Twelfth International Conference on Learning Representations, 2024

2024

[3] [3]

Unitary evolution recurrent neural networks

Arjovsky, M., Shah, A., and Bengio, Y. Unitary evolution recurrent neural networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML'16, pp.\ 1120–1128. JMLR.org, 2016

2016

[4] [4]

Human brain networks function in connectome-specific harmonic waves

Atasoy, S., Donnelly, I., and Pearson, J. Human brain networks function in connectome-specific harmonic waves. Nature Communications, 7 0 (1): 0 10340, 2016

2016

[5] [5]

Principled weight initialization for hypernetworks

Chang, O., Flokas, L., and Lipson, H. Principled weight initialization for hypernetworks. In International Conference on Learning Representations, 2020

2020

[6] [6]

and Gu, A

Dao, T. and Gu, A. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning (ICML), 2024

2024

[7] [7]

Deco, G., Sanz Perl, Y., and Kringelbach, M. L. Complex harmonics reveal low-dimensional manifolds of critical brain dynamics. Physical Review E, 111 0 (1): 0 014410, 2025

2025

[8] [8]

Multi30k: Multilingual english-german image descriptions

Elliott, D., Frank, S., Sima'an, K., and Specia, L. Multi30k: Multilingual english-german image descriptions. In Proceedings of the 5th Workshop on Vision and Language, pp.\ 70--74, 2016

2016