Geometric Hyena Networks for Large-scale Equivariant Learning

Artem Moskalev; Junjie Xu; Mangal Prakash; Rui Liao; Tianyu Cui; Tommaso Mansi

REVIEW 2 major objections 5 minor 1 cited by

Geometric Hyena Networks for Large-scale Equivariant Learning

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that global geometric context, the long-range spatial relations that matter for RNA stability, riboswitch regulation, and protein dynamics, can be captured equivariantly without the quadratic cost of self-attention.

desk verdict A genuinely new equivariant long-convolution operator, but the appendix code doesn't match the core equation, so the empirical results need a check before I'd trust them. read the letter →

arxiv 2505.22560 v1 pith:6TEZ6UST submitted 2025-05-28 cs.LG

Artem Moskalev , Mangal Prakash , Junjie Xu , Tianyu Cui , Rui Liao , Tommaso Mansi This is my paper

classification cs.LG

keywords SE(3)equivariancelongconvolutionequivariantlearninggeometricgraphsRNApropertypredictionproteinmoleculardynamicsFFTassociativerecall

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that global geometric context, the long-range spatial relations that matter for RNA stability, riboswitch regulation, and protein dynamics, can be captured equivariantly without the quadratic cost of self-attention. It introduces Geometric Hyena, an architecture that applies long-convolutional operators to SE(3)-equivariant learning by replacing dot-product attention with a vector long convolution built on cross products. The model processes sequences of geometric tokens, treating atoms in canonical order and alternating local message passing with global convolution. If the claims hold, equivariant models can scale to tens of thousands of atoms and far longer contexts while matching or beating existing equivariant transformers on molecular benchmarks.

What carries the argument

The load-bearing device is the vector long convolution, defined as $(q \otimes_\times k)_i = \sum_j q_i \times k_{j-i}$. Writing the cross product componentwise with the Levi-Civita symbol reduces each output component to a signed sum of two scalar convolutions, so the whole vector operation becomes six scalar FFT convolutions, preserving rotation equivariance because the cross product commutes with rotations. Around this, the architecture stacks an equivariant projection (a one-layer EGNN augmented with learned global context tokens) that provides local context and forms queries, keys, and values; selective gating that masks convolution outputs; and key-value normalization that removes the cubic growth of output magnitudes and stabilizes training. The paper also derives how the same construction extends to higher-order steerable representations via Clebsch-Gordan coefficients.

What would settle it

Take a molecular benchmark with fixed coordinates and randomly permute the node order while keeping the same structure; if Geometric Hyena's error rises sharply compared with a permutation-equivariant baseline like EGNN, then the sequence ordering, not the geometry, is carrying the reported advantage.

Watch

Extended reading notes

Core claim

Geometric Hyena is presented as the first equivariant long-convolutional network for geometric systems. Its central move is to define a vector long convolution over 3D token positions whose global context aggregation is equivariant under rotations and translations: the cross product of query and key vectors is expanded by the Levi-Civita identity into six scalar convolutions, each computed by FFT, so the whole operation runs in $O(N \log N)$. Scalar and vector convolutions are then combined into a geometric long convolution that also mixes invariant and equivariant subspaces via scalar-vector products and dot products. On all-atom RNA stability and degradation, riboswitch switching-factor, and protein molecular-dynamics benchmarks, the model reports the lowest error among equivariant baselines, and on 30,000-token sequences it processes context roughly 20 times faster than equivariant self-attention while allowing about 72 times longer context on the same GPU budget.

Load-bearing premise

The load-bearing premise is that a canonical, biologically meaningful ordering can be imposed on geometric graph nodes, such as IUPAC order for biomolecules, because the FFT-based convolution is not permutation equivariant except under cyclic shifts.

Editorial extensions

If this is right

If correct, equivariant modeling of biomolecules no longer needs quadratic self-attention matrices, so the same GPU budget can cover roughly 72 times longer sequences, the regime of full-length RNA and large proteins.
The reported gains on all-atom RNA and protein molecular dynamics suggest that long-range geometric context, not just local bonding neighborhoods, carries much of the signal in these prediction tasks.
Because the model is differentiable and trained end-to-end, the $O(N \log N)$ context cost transfers to training as well, letting larger equivariant models be trained on the same hardware.
The new geometric associative recall benchmark provides a simple task on which future equivariant models can be probed for in-context geometric retrieval behavior before expensive molecular runs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Going beyond the paper: the canonical-order assumption suggests a direct test: if a task's geometry is order-arbitrary, Geometric Hyena's global context may degrade, and learned node orderings could extend the approach to point clouds and meshes.
Because long convolutions act like learned global filters, Geometric Hyena may also suit physical fields such as force fields or electrostatic potentials, where smooth long-range interactions are naturally convolutional rather than discrete graph edges.
The Levi-Civita decomposition suggests a family of FFT-implementable equivariant operators for higher-order tensors; the paper sketches the Clebsch-Gordan extension, and testing it on type-2 features such as force or polarizability tensors would show whether the efficiency survives beyond vectors.
On the interpretability side, the associative-recall formulation ties induction-head-style behavior to geometry, so a natural follow-up is to measure whether recall accuracy on geometric bigrams predicts molecular-task performance the way associative recall does for language models.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

This paper introduces the first equivariant long-convolutional model I know of for geometric data, and it solves a real bottleneck: global context in equivariant models is usually done via self-attention, which is quadratic. The trick is a vector cross-product convolution, decomposed into six FFT scalar convolutions. That's clean and clearly correct. The paper also shows concrete scaling advantages on RNA and protein tasks, and the geometric associative recall task is a useful addition.

The main flaw is a mismatch between the math and the code. Equation 3 defines the vector long convolution without any normalization. Appendix Code 1, however, applies `irfft` with `norm='backward'` (which already divides by the sequence length) and then divides by N again. If the experiments used that code, the actual operator is a 1/N-scaled version of the one defined and proved equivariant in the paper. Since N varies from hundreds to over 11k across samples, this is not a constant that a learned weight can absorb. Equivariance still holds (scaling by a sample-wise constant is harmless), and the cross-product decomposition is still valid, so the core contribution stands. But the paper is internally inconsistent, and either the equation or the code must be corrected before the empirical claims can be taken at face value.

Other weaknesses are less severe. Equiformer and VNT go OOM on the all-atom tasks, which makes the comparison incomplete. The gains over G-Transformer are small on some datasets. The canonical-ordering assumption is real but explicitly scoped: the paper restricts itself to ordered geometric graphs like biomolecules, where the ordering is physically meaningful. The ablation is single-seed, which is common but not ideal.

I'd send this to peer review. The idea is solid, and the normalization issue is fixable with a unit test and a corrected appendix. The authors also need to release the code so the discrepancy can be checked. For someone working on large equivariant models, this is worth reading.

Referee Report

2 major / 5 minor

Summary. The paper introduces Geometric Hyena, an equivariant long-convolutional architecture for ordered geometric graphs. The core operator is a vector long convolution defined by cross products (Eq. 3), decomposed into six scalar FFT convolutions (Eq. 4), and optionally combined with scalar convolutions to form a geometric long convolution (Eqs. 9-10). The architecture uses equivariant EGNN-style projections with local and global context, gating, and key-value normalization. Experiments cover a new geometric associative recall task, RNA property prediction (Open Vaccine, Ribonanza-2k, Tc-Ribo), and all-atom protein MD, reporting competitive results and sub-quadratic scaling. The paper claims to be the first equivariant long-convolutional model and to preserve SE(3) equivariance.

Significance. If the implementation matches the mathematical claims, this is a valuable contribution: it provides a clean, parameter-free decomposition of cross-product convolution into FFT-based scalar convolutions, a sub-quadratic alternative to equivariant self-attention, and a new mechanistic-interpretability task. The paper is also explicit about its scope limitation to ordered geometric graphs, which is appropriately disclosed. However, the reproducibility of the core operator is compromised by an apparent mismatch between the appendix code and Eq. 3, which must be resolved before the empirical claims can be accepted.

major comments (2)

[Appendix A.6.1, Code 1, line 53] The final division by N in the provided PyTorch implementation is inconsistent with the definition in Eq. 3. Under PyTorch's default 'backward' normalization, torch.fft.irfft(..., norm='backward') already returns the circular convolution sum, so the extra '/N' scales the output by 1/N relative to u_eqv_i = sum_{j=1}^N q_i x k_{j-i}. Because N varies across inputs (up to 7800 and 11300 atoms in Table 1), this is not a constant factor that a single learnable weight can absorb. I recommend adding a unit test that compares Code 1 to a direct loop over Eq. 3 for small random inputs, and either removing the extra normalization or revising Eq. 3 and the equivariance proofs to include a 1/N factor.
[Appendix A.4.2] The proof of SE(3) equivariance for the vector long convolution relies on centering input tokens relative to their center of mass, applying the convolution, and then uncentering. However, Code 1 is explicitly described as 'without centering', and the paper does not state whether the full model in the experiments actually includes this centering step. If centering is omitted, the model is only SO(3) equivariant, not SE(3) equivariant as claimed in Eq. (1). Please clarify where and how centering is applied in the forward pass, and confirm that the experimental results reflect an SE(3)-equivariant model.

minor comments (5)

[Tables 1-3] The model name is misspelled as 'G-Heyna' in Tables 1, 2, and 3; this should be corrected to 'G-Hyena'.
[Section 4.1] The text says 'our model supports up to 2.7M million tokens'; this should be '2.7M tokens' or '2.7 million tokens'.
[Abstract and Introduction] The phrase 'requiring significantly less memory and compute that equivariant self-attention' (Abstract) and similar wording in the Introduction should read 'than' instead of 'that'.
[Table 5] The ablation table is difficult to read because the column headers and checkmarks do not align clearly; the row labeled 'QK' is confusing. Please reformat the table or its caption to indicate which variant each row represents.
[Tables 1 and 3] Equiformer is reported as out-of-memory on all-atom tasks, leaving the strongest global-context baseline absent from those comparisons. A sentence discussing this limitation would help calibrate the claim that Geometric Hyena outperforms all existing equivariant models.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the geometric long-convolution derivation is mathematically general and the empirical claims are evaluated on held-out benchmarks.

full rationale

The central derivation chain is self-contained: Equation 3 defines the vector long convolution as a cross-correlation-style sum, Equation 4 expands it via the Levi-Civita identity into six scalar FFT convolutions, and the equivariance proofs in Appendix A.4.2 operate on those definitions rather than on fitted quantities. The benchmark results compare Geometric Hyena against external and internally designed baselines on standard train/test splits; no target constant is fitted and renamed as a prediction. The canonical-ordering assumption is an explicit stated scope condition, not a circular reliance on the model's own outputs. The self-citations in the related-work and reference sections are contextual and do not carry the load-bearing argument. The appendix code in A.6.1 may contain an extra 1/N normalization relative to Equation 3, which would be an implementation/reproducibility concern rather than a circularity: the implemented operator would differ from the defined one, but it would not make the claimed results equivalent to their inputs by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The paper's central claim rests on the availability of canonical order for molecules, the correctness of the FFT decomposition of the vector cross-product convolution, and standard equivariance-composition arguments. It introduces two architecture components (the SE(3)-Hyena operator and vector long convolution) that are new but are validated internally and on held-out tasks. The trainable weights in the geometric long convolution are learned parameters, not fitted constants, so they do not raise circularity concerns.

free parameters (3)

lambda_i scalar-vector interaction weights = trained
In the geometric long convolution, the contributions of scalar-scalar, dot product, scalar-vector, and cross product terms are weighted by trainable weights lambda_i. These are learned from data and the central operation depends on them, though they are learned rather than hand-fitted.
top-k neighbors radius r and k = not explicitly reported in main text
The EGNN local context projection uses top-k neighbors within a radius r, but the specific values are not given in the main text and may be in the appendix details or omitted. This is an architectural hyperparameter affecting local context.
number of global context tokens G = not reported in main text
The equivariant projection layer uses G global context tokens, but the value of G is not specified in the available text. It affects the global summary and is a design choice.

assumptions (4)

domain assumption Biomolecules such as RNA and proteins have a canonical ordering via IUPAC rules
The entire method depends on ordered geometric graphs, and the paper asserts that IUPAC rules provide a canonical order for biomolecules. This is a domain assumption about the applicability of the method.
domain assumption Circular FFT convolution approximates the desired long convolution for sequence modeling
The scalar and vector long convolutions use circular FFT-based convolution, which is standard in Hyena-style models but involves boundary assumptions. The paper does not discuss padding or boundary effects, but the claim is that it works for the tasks.
standard math Equivariance of individual modules composes to equivariance of the whole network
The paper relies on the standard result that composing equivariant layers yields an equivariant network (Weiler & Cesa, 2019). This is a standard theorem.
standard math The projection function, a modified EGNN with global messages, remains E(n)-equivariant
The proof references EGNN's equivariance and the invariance of distances to global context tokens. The extension is plausible but the proof is sketched rather than exhaustive.

invented entities (2)

SE(3)-Hyena operator independent evidence
purpose: Global context aggregation for invariant and equivariant tokens through scalar and vector long convolutions.
It is a new architecture component with falsifiable performance implications: it is tested on multiple held-out molecular tasks where it must predict unseen labels. Its equivariance is proved and its implementation is provided.
Equivariant vector long convolution independent evidence
purpose: Aggregates global context of vector tokens via cross products, implemented with six FFT scalar convolutions.
This is a new mathematical operator with a concrete implementation and clear testable equivariance properties, evaluated across several tasks.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Geometric Hyena Networks for Large-scale Equivariant Learning." pith.science (2026). https://pith.science/paper/6TEZ6UST

@misc{pith2026250522560,
  author       = {Pith},
  title        = {Pith review of: Geometric Hyena Networks for Large-scale Equivariant Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6TEZ6UST}},
  note         = {Machine review of arXiv:2505.22560}
}

read the original abstract

Processing global geometric context while preserving equivariance is crucial when modeling biological, chemical, and physical systems. Yet, this is challenging due to the computational demands of equivariance and global context at scale. Standard methods such as equivariant self-attention suffer from quadratic complexity, while local methods such as distance-based message passing sacrifice global information. Inspired by the recent success of state-space and long-convolutional models, we introduce Geometric Hyena, the first equivariant long-convolutional model for geometric systems. Geometric Hyena captures global geometric context at sub-quadratic complexity while maintaining equivariance to rotations and translations. Evaluated on all-atom property prediction of large RNA molecules and full protein molecular dynamics, Geometric Hyena outperforms existing equivariant models while requiring significantly less memory and compute that equivariant self-attention. Notably, our model processes the geometric context of 30k tokens 20x faster than the equivariant transformer and allows 72x longer context within the same budget.

Figures

Figures reproduced from arXiv: 2505.22560 by the authors.

**Figure 1.** Left: GPU forward runtime comparison. Geometric Hyena scales sub-quadratically and achieves a considerable speedup compared to other equivariant models with global context. Right: Peak GPU memory consumption for G-Hyena is the most efficient for long sequences. Processing global geometric context with equivariance is challenging due to the computational demands of processing high-dimensional data at scale. Existing… view at source ↗

**Figure 2.** Geometric Hyena block. (a) Geometric Hyena block includes the SE(3)-Hyena operator and equivariant projections. (b) The SE(3)-Hyena operator includes query, key, value projection, geometric long convolution for global context aggregation, and gating. mapping embedded tokens into query, key, and value triplets and serving as an output projection layer. Because we have scalar and vector features, the projection layer … view at source ↗

**Figure 3.** Top: The MSE (↓) between retrieved and target vectors for the geometric associative recall task over various sequence lengths. Bottom: The study of geometric associative recall performance of different models across varying hidden dimensions and vocabulary size. mensions and vocabulary sizes affect the performance. We vary the hidden dimension by multiplying it by factors of [0.25, 0.5, 0.75, 1.], and we vary the v… view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Geometric associative recall task. A geometric sequence consists of key and value vector tokens, where consecutive key-value pairs form bigrams. Geometric associative recall requires retrieving the value vector corresponding to a query, where the query matches one of t…

**Figure 5.** Figure 5: Scalar-vector interactions in geometric long convolution. Blue lines represent interactions leading to a scalar output α3, and red lines are interactions leading to a vector output r3. Scalar output computation The scalar output α3 involves two terms: (i) scalarsc…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Platonic Transformers: A Solid Choice For Equivariance
cs.CV 2025-10 conditional novelty 6.0 of 10

Platonic Transformers achieve exact equivariance to translations plus discrete Platonic-solid rotations by lifting features into multiple reference frames and sharing one RoPE attention across them, with a linear-time...

Reference graph

Works this paper leans on

80 extracted references · 44 canonical work pages · cited by 1 Pith paper

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

Vn-transformer: Rotation-equivariant attention for vector neurons

Assaad, S., Downey, C., Al-Rfou, R., Nayakanti, N., and Sapp, B. Vn-transformer: Rotation-equivariant attention for vector neurons. 2022

work page 2022
[3]

and Sali, A

Baker, D. and Sali, A. Protein structure prediction and structural genomics. Science, 294: 0 93 -- 96, 2001. URL https://api.semanticscholar.org/CorpusID:7193705

work page 2001
[4]

J., Vadgama, S., Hesselink, R

Bekkers, E. J., Vadgama, S., Hesselink, R. D., van der Linden, P. A., and Romero, D. W. Fast, expressive se (n) equivariant networks through weight-sharing in position-orientation space. arXiv preprint arXiv:2310.02970, 2023

arXiv 2023
[5]

J., and Welling, M

Brandstetter, J., Hesselink, R., van der Pol, E., Bekkers, E. J., and Welling, M. Geometric and physical quantities improve e (3) equivariant message passing. In International Conference on Learning Representations, 2021

work page 2021
[6]

D., Behrends, S., and Cohen, T

Brehmer, J., Haan, P. D., Behrends, S., and Cohen, T. Geometric algebra transformer. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=M7r2CO4tJC

work page 2023
[7]

Are high-degree representations really unnecessary in equivariant graph neural networks? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

Cen, J., Li, A., Lin, N., Ren, Y., Wang, Z., and Huang, W. Are high-degree representations really unnecessary in equivariant graph neural networks? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=M0ncNVuGYN

work page 2024
[8]

Nomenclature of inorganic chemistry: Iupac recommendations 2005

Damhus, T., Hartshorn, R., and Hutton, A. Nomenclature of inorganic chemistry: Iupac recommendations 2005. Chem. Int, 27: 0 25--26, 2005

work page 2005

Show all 80 references

[9]

Flashattention: Fast and memory-efficient exact attention with io-awareness

Dao, T., Fu, D., Ermon, S., Rudra, A., and R \'e , C. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35: 0 16344--16359, 2022

2022
[10]

Openvaccine: Covid-19 mrna vaccine degradation prediction, 2020

Das, R., Wayment-Steele, H., Soon Kim, D., Choe, C., Tunguz, B., Reade, W., and Demkin, M. Openvaccine: Covid-19 mrna vaccine degradation prediction, 2020. URL https://kaggle.com/competitions/stanford-covid-vaccine

2020
[11]

L., Fernando, A., Botev, A., Cristian-Muraru, G., Gu, A., Haroun, R., Berrada, L., Chen, Y., Srinivasan, S., et al

De, S., Smith, S. L., Fernando, A., Botev, A., Cristian-Muraru, G., Gu, A., Haroun, R., Berrada, L., Chen, Y., Srinivasan, S., et al. Griffin: Mixing gated linear recurrences with local attention for efficient language models. arXiv preprint arXiv:2402.19427, 2024

2024 arXiv
[12]

Gauge equivariant mesh cnns: Anisotropic convolutions on geometric graphs

De Haan, P., Weiler, M., Cohen, T., and Welling, M. Gauge equivariant mesh cnns: Anisotropic convolutions on geometric graphs. arXiv preprint arXiv:2003.05425, 2020

2003 arXiv
[13]

Euclidean, projective, conformal: Choosing a geometric algebra for equivariant transformers

de Haan, P., Cohen, T., and Brehmer, J. Euclidean, projective, conformal: Choosing a geometric algebra for equivariant transformers. In Proceedings of the 27th International Conference on Artificial Intelligence and Statistics, volume 27, 2024. URL https://arxiv.org/abs/2311.04744

2024 arXiv
[14]

Vector neurons: a general framework for so(3)-equivariant networks

Deng, C., Litany, O., Duan, Y., Poulenard, A., Tagliasacchi, A., and Guibas, L. Vector neurons: a general framework for so(3)-equivariant networks. arXiv preprint arXiv:2104.12229, 2021

2021 arXiv
[15]

Geometric algebra for computer science (revised edition): An object-oriented approach to geometry

Dorst, L., Fontijne, D., and Mann, S. Geometric algebra for computer science (revised edition): An object-oriented approach to geometry. Morgan Kaufmann, 2009

2009
[16]

P., Ma, Z.-M., et al

Du, Y., Wang, L., Feng, D., Wang, G., Ji, S., Gomes, C. P., Ma, Z.-M., et al. A new perspective on building efficient and expressive 3d equivariant graph neural networks. Advances in Neural Information Processing Systems, 36, 2023

2023
[17]

Y., Dao, T., Saab, K

Fu, D. Y., Dao, T., Saab, K. K., Thomas, A. W., Rudra, A., and R \'e , C. Hungry hungry hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212.14052, 2022

2022 arXiv
[18]

Se (3)-transformers: 3d roto-translation equivariant attention networks

Fuchs, F., Worrall, D., Fischer, V., and Welling, M. Se (3)-transformers: 3d roto-translation equivariant attention networks. Advances in neural information processing systems, 33: 0 1970--1981, 2020

1970
[19]

S., Riley, P

Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E. Neural message passing for quantum chemistry. In International conference on machine learning, pp.\ 1263--1272. PMLR, 2017

2017
[20]

Tuning the performance of synthetic riboswitches using machine learning

Groher, A.-C., Jager, S., Schneider, C., Groher, F., Hamacher, K., and Suess, B. Tuning the performance of synthetic riboswitches using machine learning. ACS synthetic biology, 8 0 (1): 0 34--44, 2018

2018
[21]

and Dao, T

Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

2023 arXiv
[22]

Efficiently modeling long sequences with structured state spaces

Gu, A., Goel, K., and R \'e , C. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021 a

2021 arXiv
[23]

Combining recurrent, convolutional, and continuous-time models with linear state space layers

Gu, A., Johnson, I., Goel, K., Saab, K., Dao, T., Rudra, A., and R \'e , C. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems, 34: 0 572--585, 2021 b

2021
[24]

Equivariant graph hierarchy-based neural networks

Han, J., Huang, W., Xu, T., and Rong, Y. Equivariant graph hierarchy-based neural networks. Advances in Neural Information Processing Systems, 35: 0 9176--9187, 2022

2022
[25]

C., Karagianes, T

He, S., Huang, R., Townley, J., Kretsch, R. C., Karagianes, T. G., Cox, D. B., Blair, H., Penzar, D., Vyaltsev, V., Aristova, E., et al. Ribonanza: deep learning of rna structure through dual crowdsourcing. bioRxiv, 2024

2024
[26]

S., and Ma, T

Hwang, E., Thost, V., Dasgupta, S. S., and Ma, T. Revisiting virtual nodes in graph neural networks for link prediction, 2022. URL https://openreview.net/forum?id=ETiaOyNwJW

2022
[27]

N., and Dror, R

Jing, B., Eismann, S., Soni, P. N., and Dror, R. O. Equivariant graph neural networks for 3d macromolecular structure. arXiv preprint arXiv:2106.03843, 2021 a

2021 arXiv
[28]

Jing, B., Eismann, S., Suriana, P., Townshend, R. J. L., and Dror, R. Learning from protein structure with geometric vector perceptrons. In International Conference on Learning Representations, 2021 b . URL https://openreview.net/forum?id=1YLJDvSx6J4

2021
[29]

and Gasteiger, J

Jochum, C. and Gasteiger, J. Canonical numbering and constitutional symmetry. Journal of Chemical Information and Computer Sciences, 17 0 (2): 0 113--117, 1977

1977
[30]

Pure transformers are powerful graph learners

Kim, J., Nguyen, D., Min, S., Cho, S., Lee, M., Lee, H., and Hong, S. Pure transformers are powerful graph learners. Advances in Neural Information Processing Systems, 35: 0 14582--14595, 2022

2022
[31]

Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

2014 arXiv
[32]

Kipf, T. N. and Welling, M. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016

2016 arXiv
[33]

Equivariant flows: exact likelihood generative learning for symmetric densities

K \"o hler, J., Klein, L., and No \'e , F. Equivariant flows: exact likelihood generative learning for symmetric densities. In International conference on machine learning, pp.\ 5361--5370. PMLR, 2020

2020
[34]

Rethinking graph transformers with spectral attention

Kreuzer, D., Beaini, D., Hamilton, W., L \'e tourneau, V., and Tossou, P. Rethinking graph transformers with spectral attention. Advances in Neural Information Processing Systems, 34: 0 21618--21629, 2021

2021
[35]

Mamba-nd: Selective state space modeling for multi-dimensional data

Li, S., Singh, H., and Grover, A. Mamba-nd: Selective state space modeling for multi-dimensional data. arXiv preprint arXiv:2402.05892, 2024

2024 arXiv
[36]

and Smidt, T

Liao, Y.-L. and Smidt, T. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=KwmPfARgOTD

2023
[37]

and Hutter, F

Loshchilov, I. and Hutter, F. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016

2016 arXiv
[38]

Low, J. T. and Weeks, K. M. Shape-directed rna secondary structure prediction. Methods, 52 0 (2): 0 150--158, 2010

2010
[39]

J., and Smeulders, A

Moskalev, A., Sepliarskaia, A., Bekkers, E. J., and Smeulders, A. W. On genuine invariance learning without weight-tying. In Topological, Algebraic and Geometric Learning Workshops 2023, pp.\ 218--227. PMLR, 2023

2023
[40]

Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution

Nguyen, E., Poli, M., Faizi, M., Thomas, A., Wornow, M., Birch-Sykes, C., Massaroli, S., Patel, A., Rabideau, C., Bengio, Y., et al. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. Advances in neural information processing systems, 36, 2024

2024
[41]

In-context learning and induction heads

Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022

2022 arXiv
[42]

L., Gu, A., Fernando, A., Gulcehre, C., Pascanu, R., and De, S

Orvieto, A., Smith, S. L., Gu, A., Fernando, A., Gulcehre, C., Pascanu, R., and De, S. Resurrecting recurrent neural networks for long sequences. In International Conference on Machine Learning, pp.\ 26670--26698. PMLR, 2023

2023
[43]

P., Simeon, G., Galvelis, R., Mirarchi, A., Eastman, P., Doerr, S., Thölke, P., Markland, T

Pelaez, R. P., Simeon, G., Galvelis, R., Mirarchi, A., Eastman, P., Doerr, S., Thölke, P., Markland, T. E., and Fabritiis, G. D. Torchmd-net 2.0: Fast neural network potentials for molecular simulations, 2024

2024
[44]

Y., Dao, T., Baccus, S., Bengio, Y., Ermon, S., and R \'e , C

Poli, M., Massaroli, S., Nguyen, E., Fu, D. Y., Dao, T., Baccus, S., Bengio, Y., Ermon, S., and R \'e , C. Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning, pp.\ 28043--28078. PMLR, 2023

2023
[45]

W., Nguyen, E., Ponnusamy, P., Deiseroth, B., Kersting, K., Suzuki, T., Hie, B., Ermon, S., Ré, C., Zhang, C., and Massaroli, S

Poli, M., Thomas, A. W., Nguyen, E., Ponnusamy, P., Deiseroth, B., Kersting, K., Suzuki, T., Hie, B., Ermon, S., Ré, C., Zhang, C., and Massaroli, S. Mechanistic design and scaling of hybrid architectures. In ICML, 2024. URL https://openreview.net/forum?id=GDp7Gyd9nf

2024
[46]

P., Luu, A

Ramp \'a s ek, L., Galkin, M., Dwivedi, V. P., Luu, A. T., Wolf, G., and Beaini, D. Recipe for a general, powerful, scalable graph transformer. Advances in Neural Information Processing Systems, 35: 0 14501--14515, 2022

2022
[47]

W., Kuzina, A., Bekkers, E

Romero, D. W., Kuzina, A., Bekkers, E. J., Tomczak, J. M., and Hoogendoorn, M. Ckconv: Continuous kernel convolution for sequential data. arXiv preprint arXiv:2102.02611, 2021

2021 arXiv
[48]

and Bahri, C

Rowe, D. and Bahri, C. Clebsch--gordan coefficients of su (3) in su (2) and so (3) bases. Journal of Mathematical Physics, 41 0 (9): 0 6544--6565, 2000

2000
[49]

Clifford group equivariant neural networks

Ruhe, D., Brandstetter, J., and Forr \'e , P. Clifford group equivariant neural networks. In Thirty-seventh Conference on Neural Information Processing Systems, 2023 a . URL https://openreview.net/forum?id=n84bzMrGUD

2023
[50]

K., De Keninck, S., Welling, M., and Brandstetter, J

Ruhe, D., Gupta, J. K., De Keninck, S., Welling, M., and Brandstetter, J. Geometric clifford algebra networks. In International Conference on Machine Learning, pp.\ 29306--29337. PMLR, 2023 b

2023
[51]

K., Bronstein, M

Rusch, T. K., Bronstein, M. M., and Mishra, S. A survey on oversmoothing in graph neural networks. arXiv preprint arXiv:2303.10993, 2023

2023 arXiv
[52]

Rna secondary structure prediction using deep learning with thermodynamic integration

Sato, K., Akiyama, M., and Sakakibara, Y. Rna secondary structure prediction using deep learning with thermodynamic integration. Nature communications, 12 0 (1): 0 941, 2021

2021
[53]

G., Hoogeboom, E., and Welling, M

Satorras, V. G., Hoogeboom, E., and Welling, M. E (n) equivariant graph neural networks. In International conference on machine learning, pp.\ 9323--9332. PMLR, 2021

2021
[54]

Caduceus: Bi-directional equivariant long-range dna sequence modeling

Schiff, Y., Kao, C.-H., Gokaslan, A., Dao, T., Gu, A., and Kuleshov, V. Caduceus: Bi-directional equivariant long-range dna sequence modeling. arXiv preprint arXiv:2403.03234, 2024

2024 arXiv
[55]

u tt, K., Kindermans, P.-J., Sauceda Felix, H. E., Chmiela, S., Tkatchenko, A., and M \

Sch \"u tt, K., Kindermans, P.-J., Sauceda Felix, H. E., Chmiela, S., Tkatchenko, A., and M \"u ller, K.-R. Schnet: A continuous-filter convolutional neural network for modeling quantum interactions. Advances in neural information processing systems, 30, 2017

2017
[56]

and Beckstein, O

Seyler, S. and Beckstein, O. Molecular dynamics trajectory for benchmarking mdanalysis, 6 2017. URL: https://figshare. com/articles/Molecular\_dynamics\_ trajectory\_for\_benchmarking\_MDAnalysis/5108170, doi, 10: 0 m9, 2017

2017
[57]

E2efold-3d: End-to-end deep learning method for accurate de novo rna 3d structure prediction

Shen, T., Hu, Z., Peng, Z., Chen, J., Xiong, P., Hong, L., Zheng, L., Wang, Y., King, I., Wang, S., et al. E2efold-3d: End-to-end deep learning method for accurate de novo rna 3d structure prediction. arXiv preprint arXiv:2207.01586, 2022

2022 arXiv
[58]

Implicit neural representations with periodic activation functions

Sitzmann, V., Martel, J., Bergman, A., Lindell, D., and Wetzstein, G. Implicit neural representations with periodic activation functions. Advances in neural information processing systems, 33: 0 7462--7473, 2020

2020
[59]

Southern, J., Di Giovanni, F., Bronstein, M., and Lutzeyer, J. F. Understanding virtual nodes: Oversmoothing, oversquashing, and node heterogeneity. arXiv preprint arXiv:2405.13526, 2024

2024 arXiv
[60]

A., Gil-Ley, A., Pinamonti, G., Poblete, S., Jurecka, P., et al

Sponer, J., Bussi, G., Krepl, M., Banas, P., Bottaro, S., Cunha, R. A., Gil-Ley, A., Pinamonti, G., Poblete, S., Jurecka, P., et al. Rna structural dynamics as captured by molecular simulations: a comprehensive overview. Chemical reviews, 118 0 (8): 0 4177--4338, 2018

2018
[61]

Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds

Thomas, N., Smidt, T., Kearnes, S., Yang, L., Li, L., Kohlhoff, K., and Riley, P. Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds. arXiv preprint arXiv:1802.08219, 2018

2018 arXiv
[62]

N., Kaiser, ., and Polosukhin, I

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017

2017
[63]

M., Keating, K

Wadley, L. M., Keating, K. S., Duarte, C. M., and Pyle, A. M. Evaluating and learning from rna pseudotorsional space: quantitative validation of a reduced representation for rna structure. Journal of molecular biology, 372 0 (4): 0 942--957, 2007

2007
[64]

State space model for new-generation network alternative to transformers: A survey

Wang, X., Wang, S., Ding, Y., Li, Y., Wu, W., Rong, Y., Kong, W., Huang, J., Li, S., Yang, H., et al. State space model for new-generation network alternative to transformers: A survey. arXiv preprint arXiv:2404.09516, 2024 a

2024 arXiv
[65]

Neural p\ 3\ m: A long-range interaction modeling enhancer for geometric GNN s

Wang, Y., Cheng, C., Li, S., Ren, Y., Shao, B., Liu, G., Heng, P.-A., and Zheng, N. Neural p\ 3\ m: A long-range interaction modeling enhancer for geometric GNN s. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 b . URL https://openreview....

2024
[66]

K., Kladwang, W., Watkins, A

Wayment-Steele, H. K., Kladwang, W., Watkins, A. M., Kim, D. S., Tunguz, B., Reade, W., Demkin, M., Romano, J., Wellington-Oguri, R., Nicol, J. J., et al. Deep learning models for predicting rna degradation via dual crowdsourcing. Nature Machine Intelligence, 4 0 (12): 0 1174-...

2022
[67]

and Cesa, G

Weiler, M. and Cesa, G. General e (2)-equivariant steerable cnns. Advances in neural information processing systems, 32, 2019

2019
[68]

Pointconv: Deep convolutional networks on 3d point clouds

Wu, W., Qi, Z., and Fuxin, L. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp.\ 9621--9630, 2019

2019
[69]

Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., and Philip, S. Y. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32 0 (1): 0 4--24, 2020

2020
[70]

B., and Schlick, T

Xin, Y., Laing, C., Leontis, N. B., and Schlick, T. Annotation of tertiary interactions in rna structures reveals variations and correlations. Rna, 14 0 (12): 0 2465--2477, 2008

2008
[71]

Beyond sequence: Impact of geometric context for rna property prediction

Xu, J., Moskalev, A., Mansi, T., Prakash, M., and Liao, R. Beyond sequence: Impact of geometric context for rna property prediction. arXiv preprint arXiv:2410.11933, 2024

2024 arXiv
[72]

HARMONY : A multi-representation framework for RNA property prediction

Xu, J., Moskalev, A., Mansi, T., Prakash, M., and Liao, R. HARMONY : A multi-representation framework for RNA property prediction. In ICLR 2025 Workshop on AI for Nucleic Acids, 2025. URL https://openreview.net/forum?id=nzUsRhtnBa

2025
[73]

Graphformers: Gnn-nested transformers for representation learning on textual graph

Yang, J., Liu, Z., Xiao, S., Li, C., Lian, D., Agrawal, S., Singh, A., Sun, G., and Xie, X. Graphformers: Gnn-nested transformers for representation learning on textual graph. Advances in Neural Information Processing Systems, 34: 0 28798--28810, 2021

2021
[74]

HELM : Hierarchical encoding for m RNA language modeling

Yazdani-Jahromi, M., Prakash, M., Mansi, T., Moskalev, A., and Liao, R. HELM : Hierarchical encoding for m RNA language modeling. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=MMHqnUOnl0

2025
[75]

Artificial intelligence for science in quantum, atomistic, and continuum systems

Zhang, X., Wang, L., Helwig, J., Luo, Y., Fu, C., Xie, Y., Liu, M., Lin, Y., Xu, Z., Yan, K., et al. Artificial intelligence for science in quantum, atomistic, and continuum systems. arXiv preprint arXiv:2307.08423, 2023

2023 arXiv
[76]

Improving equivariant graph neural networks on large geometric graphs via virtual nodes learning

Zhang, Y., Cen, J., Han, J., Zhang, Z., ZHOU, J., and Huang, W. Improving equivariant graph neural networks on large geometric graphs via virtual nodes learning. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=wWdkNkUY8k

2024
[77]

Implicit convolutional kernels for steerable cnns

Zhdanov, M., Hoffmann, N., and Cesa, G. Implicit convolutional kernels for steerable cnns. Advances in Neural Information Processing Systems, 36, 2024 a

2024
[78]

Clifford-steerable convolutional neural networks, 2024 b

Zhdanov, M., Ruhe, D., Weiler, M., Lucic, A., Brandstetter, J., and Forré, P. Clifford-steerable convolutional neural networks, 2024 b

2024
[79]

Erwin: A tree-based hierarchical transformer for large-scale physical systems

Zhdanov, M., Welling, M., and van de Meent, J.-W. Erwin: A tree-based hierarchical transformer for large-scale physical systems. In International Conference on Machine Learning ( ICML ) , 2025

2025
[80]

Vision mamba: Efficient visual representation learning with bidirectional state space model

Zhu, L., Liao, B., Zhang, Q., Wang, X., Liu, W., and Wang, X. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417, 2024

2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

Vn-transformer: Rotation-equivariant attention for vector neurons

Assaad, S., Downey, C., Al-Rfou, R., Nayakanti, N., and Sapp, B. Vn-transformer: Rotation-equivariant attention for vector neurons. 2022

work page 2022

[3] [3]

and Sali, A

Baker, D. and Sali, A. Protein structure prediction and structural genomics. Science, 294: 0 93 -- 96, 2001. URL https://api.semanticscholar.org/CorpusID:7193705

work page 2001

[4] [4]

J., Vadgama, S., Hesselink, R

Bekkers, E. J., Vadgama, S., Hesselink, R. D., van der Linden, P. A., and Romero, D. W. Fast, expressive se (n) equivariant networks through weight-sharing in position-orientation space. arXiv preprint arXiv:2310.02970, 2023

arXiv 2023

[5] [5]

J., and Welling, M

Brandstetter, J., Hesselink, R., van der Pol, E., Bekkers, E. J., and Welling, M. Geometric and physical quantities improve e (3) equivariant message passing. In International Conference on Learning Representations, 2021

work page 2021

[6] [6]

D., Behrends, S., and Cohen, T

Brehmer, J., Haan, P. D., Behrends, S., and Cohen, T. Geometric algebra transformer. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=M7r2CO4tJC

work page 2023

[7] [7]

Are high-degree representations really unnecessary in equivariant graph neural networks? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

Cen, J., Li, A., Lin, N., Ren, Y., Wang, Z., and Huang, W. Are high-degree representations really unnecessary in equivariant graph neural networks? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=M0ncNVuGYN

work page 2024

[8] [8]

Nomenclature of inorganic chemistry: Iupac recommendations 2005

Damhus, T., Hartshorn, R., and Hutton, A. Nomenclature of inorganic chemistry: Iupac recommendations 2005. Chem. Int, 27: 0 25--26, 2005

work page 2005