{"id":"cbd4d9e6-eb83-43d5-b5ed-409d57bd303a","arxiv_id":"2505.22560","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":8.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Geometric Hyena is an equivariant long-convolutional architecture that captures global geometric context with sub-quadratic complexity and outperforms equivariant transformer baselines on several RNA and protein prediction tasks.","lead":"This paper introduces Geometric Hyena, a neural network that processes global 3D geometric context in molecules at sub-quadratic cost while preserving rotation and translation equivariance. It reports faster and more memory-efficient scaling than equivariant transformers on RNA and protein tasks, potentially enabling much longer biomolecular sequences to be modeled.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Appendix code for the core vector long convolution appears to apply an extra 1/N normalization, so the implemented operator may not match Eq. 3; a unit test should settle this before the empirical claims are accepted.","rationale":"The paper's theoretical construction is coherent: Eq. 4 correctly decomposes the cross-product convolution into six scalar FFT convolutions, and the equivariance proof for rotations follows from the equivariance of the cross product, with translation handled by centering. The reader's stated weakest assumption, the reliance on canonical ordering, is a legitimate scope limitation, but the paper explicitly restricts itself to geometrically ordered graphs and discloses the permutation-sensitivity limitation in the Conclusions. Moreover, the G-Transformer baseline uses the same sequence ordering, so the empirical comparisons are not invalidated by the ordering assumption alone. The more load-bearing issue is that the only published implementation of the core vector long convolution appears not to implement Eq. 3: it applies an extra 1/N scaling after the inverse FFT's built-in 1/N normalization. This affects the relative magnitude of the global vector context across sequences of different lengths, so it cannot be dismissed as a fixed constant. The proposed unit test can settle this in minutes. This does not change the reader's CONDITIONAL verdict; on the contrary, it makes the requested code release and verification concrete and necessary. If the test shows the implementation actually matches Eq. 3 (or if corrected results preserve the tables), the central claim stands; if not, the empirical claims must be re-evaluated.","tokens_in":24872,"tokens_out":22172,"duration_ms":282382,"concrete_test":"Write a unit test comparing Code 1's VectorLongConv output against a direct O(N^2) implementation of Eq. 3 for small random inputs, e.g., N=8, q,k shaped (1,1,N,3), with circular indices. Compute the elementwise ratio; if the ratio is exactly N, the published implementation is off by 1/N. If the mismatch is confirmed, rerun the all-atom RNA experiments (Table 1) and protein-MD experiments (Table 3) with the normalization corrected (remove the final /N, or use irfft with norm='forward') and check whether the reported margins persist. If they persist, the bug is non-load-bearing; if they change materially, the empirical claims depend on an implementation error.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central operator is defined in Section 2.2, Eq. 3, as u_eqv_i = sum_{j=1}^N q_i x k_{j-i}, with no 1/N factor. The only implementation of this operator given in the paper, Code 1 in Appendix A.6.1, computes convfft with torch.fft.irfft(..., norm='backward', n=N, dim=-2) and then, in the final line, divides the reduced result by N. In PyTorch's documented 'backward' normalization convention, the inverse FFT already applies a 1/N factor; the extra division therefore scales the output by an additional 1/N relative to Eq. 3. Because this scale depends on N, and N varies across and within the benchmark datasets (up to 7800 and 11300 atoms in Table 1), the mismatch is not a constant that a single learned weight can absorb. If the experimental implementation matches the appendix code, the reported results are for an operator different from the one defined and proved equivariant; if the experiments use a corrected implementation, the appendix code is misleading and the paper is not reproducible as written. The reader's canonical-ordering concern is real but scoped by the paper's explicit restriction to ordered geometric graphs, and the G-Transformer baseline shares the same ordering, so I do not consider it the most load-bearing issue.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Geometric Hyena, an equivariant long-convolutional architecture for ordered geometric graphs. The core operator is a vector long convolution defined by cross products (Eq. 3), decomposed into six scalar FFT convolutions (Eq. 4), and optionally combined with scalar convolutions to form a geometric long convolution (Eqs. 9-10). The architecture uses equivariant EGNN-style projections with local and global context, gating, and key-value normalization. Experiments cover a new geometric associative recall task, RNA property prediction (Open Vaccine, Ribonanza-2k, Tc-Ribo), and all-atom protein MD, reporting competitive results and sub-quadratic scaling. The paper claims to be the first equivariant long-convolutional model and to preserve SE(3) equivariance.","tokens_in":25155,"tokens_out":9956,"duration_ms":97956,"significance":"If the implementation matches the mathematical claims, this is a valuable contribution: it provides a clean, parameter-free decomposition of cross-product convolution into FFT-based scalar convolutions, a sub-quadratic alternative to equivariant self-attention, and a new mechanistic-interpretability task. The paper is also explicit about its scope limitation to ordered geometric graphs, which is appropriately disclosed. However, the reproducibility of the core operator is compromised by an apparent mismatch between the appendix code and Eq. 3, which must be resolved before the empirical claims can be accepted.","major_comments":[{"comment":"The final division by N in the provided PyTorch implementation is inconsistent with the definition in Eq. 3. Under PyTorch's default 'backward' normalization, torch.fft.irfft(..., norm='backward') already returns the circular convolution sum, so the extra '/N' scales the output by 1/N relative to u_eqv_i = sum_{j=1}^N q_i x k_{j-i}. Because N varies across inputs (up to 7800 and 11300 atoms in Table 1), this is not a constant factor that a single learnable weight can absorb. I recommend adding a unit test that compares Code 1 to a direct loop over Eq. 3 for small random inputs, and either removing the extra normalization or revising Eq. 3 and the equivariance proofs to include a 1/N factor.","section":"Appendix A.6.1, Code 1, line 53"},{"comment":"The proof of SE(3) equivariance for the vector long convolution relies on centering input tokens relative to their center of mass, applying the convolution, and then uncentering. However, Code 1 is explicitly described as 'without centering', and the paper does not state whether the full model in the experiments actually includes this centering step. If centering is omitted, the model is only SO(3) equivariant, not SE(3) equivariant as claimed in Eq. (1). Please clarify where and how centering is applied in the forward pass, and confirm that the experimental results reflect an SE(3)-equivariant model.","section":"Appendix A.4.2"}],"minor_comments":[{"comment":"The model name is misspelled as 'G-Heyna' in Tables 1, 2, and 3; this should be corrected to 'G-Hyena'.","section":"Tables 1-3"},{"comment":"The text says 'our model supports up to 2.7M million tokens'; this should be '2.7M tokens' or '2.7 million tokens'.","section":"Section 4.1"},{"comment":"The phrase 'requiring significantly less memory and compute that equivariant self-attention' (Abstract) and similar wording in the Introduction should read 'than' instead of 'that'.","section":"Abstract and Introduction"},{"comment":"The ablation table is difficult to read because the column headers and checkmarks do not align clearly; the row labeled 'QK' is confusing. Please reformat the table or its caption to indicate which variant each row represents.","section":"Table 5"},{"comment":"Equiformer is reported as out-of-memory on all-atom tasks, leaving the strongest global-context baseline absent from those comparisons. A sentence discussing this limitation would help calibrate the claim that Geometric Hyena outperforms all existing equivariant models.","section":"Tables 1 and 3"}],"recommendation":"major_revision","confidential_remarks":"The FFT normalization issue in Appendix A.6.1 is the primary concern. If the authors can show via a unit test that the reported experiments use a corrected implementation, the paper should be acceptable after a thorough revision. The 'first equivariant long-convolutional model' claim may also warrant a more careful comparison with concurrent works such as Neural P3M and Erwin, though this is not a blocking issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper introduces the first equivariant long-convolutional model I know of for geometric data, and it solves a real bottleneck: global context in equivariant models is usually done via self-attention, which is quadratic. The trick is a vector cross-product convolution, decomposed into six FFT scalar convolutions. That's clean and clearly correct. The paper also shows concrete scaling advantages on RNA and protein tasks, and the geometric associative recall task is a useful addition.\n\nThe main flaw is a mismatch between the math and the code. Equation 3 defines the vector long convolution without any normalization. Appendix Code 1, however, applies `irfft` with `norm='backward'` (which already divides by the sequence length) and then divides by N again. If the experiments used that code, the actual operator is a 1/N-scaled version of the one defined and proved equivariant in the paper. Since N varies from hundreds to over 11k across samples, this is not a constant that a learned weight can absorb. Equivariance still holds (scaling by a sample-wise constant is harmless), and the cross-product decomposition is still valid, so the core contribution stands. But the paper is internally inconsistent, and either the equation or the code must be corrected before the empirical claims can be taken at face value.\n\nOther weaknesses are less severe. Equiformer and VNT go OOM on the all-atom tasks, which makes the comparison incomplete. The gains over G-Transformer are small on some datasets. The canonical-ordering assumption is real but explicitly scoped: the paper restricts itself to ordered geometric graphs like biomolecules, where the ordering is physically meaningful. The ablation is single-seed, which is common but not ideal.\n\nI'd send this to peer review. The idea is solid, and the normalization issue is fixable with a unit test and a corrected appendix. The authors also need to release the code so the discrepancy can be checked. For someone working on large equivariant models, this is worth reading.","headline":"A genuinely new equivariant long-convolution operator, but the appendix code doesn't match the core equation, so the empirical results need a check before I'd trust them.","tokens_in":25688,"tokens_out":3841,"would_cite":true,"duration_ms":42628,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that global geometric context, the long-range spatial relations that matter for RNA stability, riboswitch regulation, and protein dynamics, can be captured equivariantly without the quadratic cost of self-attention.","keywords":["SE(3) equivariance","long convolution","equivariant learning","geometric graphs","RNA property prediction","protein molecular dynamics","FFT convolution","associative recall"],"falsifier":"Take a molecular benchmark with fixed coordinates and randomly permute the node order while keeping the same structure; if Geometric Hyena's error rises sharply compared with a permutation-equivariant baseline like EGNN, then the sequence ordering, not the geometry, is carrying the reported advantage.","tokens_in":24682,"feed_emoji":"🧬","tokens_out":6295,"duration_ms":64324,"temperature":0.7,"pith_summary":"The paper claims that global geometric context, the long-range spatial relations that matter for RNA stability, riboswitch regulation, and protein dynamics, can be captured equivariantly without the quadratic cost of self-attention. It introduces Geometric Hyena, an architecture that applies long-convolutional operators to SE(3)-equivariant learning by replacing dot-product attention with a vector long convolution built on cross products. The model processes sequences of geometric tokens, treating atoms in canonical order and alternating local message passing with global convolution. If the claims hold, equivariant models can scale to tens of thousands of atoms and far longer contexts while matching or beating existing equivariant transformers on molecular benchmarks.","feed_headline":"Rotation-equivariant context for molecules now costs O(N log N), not quadratic","feed_subtitle":"The Geometric Hyena model beats equivariant transformers on RNA and protein tasks while using a fraction of the memory.","key_machinery":"The load-bearing device is the vector long convolution, defined as $(q \\otimes_\\times k)_i = \\sum_j q_i \\times k_{j-i}$. Writing the cross product componentwise with the Levi-Civita symbol reduces each output component to a signed sum of two scalar convolutions, so the whole vector operation becomes six scalar FFT convolutions, preserving rotation equivariance because the cross product commutes with rotations. Around this, the architecture stacks an equivariant projection (a one-layer EGNN augmented with learned global context tokens) that provides local context and forms queries, keys, and values; selective gating that masks convolution outputs; and key-value normalization that removes the cubic growth of output magnitudes and stabilizes training. The paper also derives how the same construction extends to higher-order steerable representations via Clebsch-Gordan coefficients.","core_discovery":"Geometric Hyena is presented as the first equivariant long-convolutional network for geometric systems. Its central move is to define a vector long convolution over 3D token positions whose global context aggregation is equivariant under rotations and translations: the cross product of query and key vectors is expanded by the Levi-Civita identity into six scalar convolutions, each computed by FFT, so the whole operation runs in $O(N \\log N)$. Scalar and vector convolutions are then combined into a geometric long convolution that also mixes invariant and equivariant subspaces via scalar-vector products and dot products. On all-atom RNA stability and degradation, riboswitch switching-factor, and protein molecular-dynamics benchmarks, the model reports the lowest error among equivariant baselines, and on 30,000-token sequences it processes context roughly 20 times faster than equivariant self-attention while allowing about 72 times longer context on the same GPU budget.","pith_inferences":["Going beyond the paper: the canonical-order assumption suggests a direct test: if a task's geometry is order-arbitrary, Geometric Hyena's global context may degrade, and learned node orderings could extend the approach to point clouds and meshes.","Because long convolutions act like learned global filters, Geometric Hyena may also suit physical fields such as force fields or electrostatic potentials, where smooth long-range interactions are naturally convolutional rather than discrete graph edges.","The Levi-Civita decomposition suggests a family of FFT-implementable equivariant operators for higher-order tensors; the paper sketches the Clebsch-Gordan extension, and testing it on type-2 features such as force or polarizability tensors would show whether the efficiency survives beyond vectors.","On the interpretability side, the associative-recall formulation ties induction-head-style behavior to geometry, so a natural follow-up is to measure whether recall accuracy on geometric bigrams predicts molecular-task performance the way associative recall does for language models."],"forward_implications":["If correct, equivariant modeling of biomolecules no longer needs quadratic self-attention matrices, so the same GPU budget can cover roughly 72 times longer sequences, the regime of full-length RNA and large proteins.","The reported gains on all-atom RNA and protein molecular dynamics suggest that long-range geometric context, not just local bonding neighborhoods, carries much of the signal in these prediction tasks.","Because the model is differentiable and trained end-to-end, the $O(N \\log N)$ context cost transfers to training as well, letting larger equivariant models be trained on the same hardware.","The new geometric associative recall benchmark provides a simple task on which future equivariant models can be probed for in-context geometric retrieval behavior before expensive molecular runs."],"supporting_citations":[{"why":"Supplies the long-convolution operator and gating design that Geometric Hyena makes equivariant.","marker":"Poli et al., 2023"},{"why":"Provides the EGNN local message passing and equivariance proof reused as the projection layer.","marker":"Satorras et al., 2021"},{"why":"Equiformer is the main equivariant self-attention baseline benchmarked for accuracy and cost.","marker":"Liao & Smidt, 2023"},{"why":"Vector neurons underlie the Vector Neuron Transformer baseline compared in runtime and associative recall.","marker":"Deng et al., 2021"},{"why":"OpenVaccine supplies the RNA stability prediction benchmark.","marker":"Das et al., 2020"},{"why":"Ribonanza supplies the RNA degradation prediction benchmark.","marker":"He et al., 2024"},{"why":"Tc-Ribo supplies the riboswitch switching-factor prediction benchmark.","marker":"Groher et al., 2018"},{"why":"ProteinMD supplies the molecular-dynamics trajectory prediction benchmark.","marker":"Han et al., 2022"}],"fun_headline_variants":["Equivariant long-conv model: global context in O(N log N)","Sub-quadratic equivariance: Geometric Hyena beats transformers","Global equivariant context for molecules at O(N log N) cost","Geometric Hyena: equivariant global context with less compute"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a canonical, biologically meaningful ordering can be imposed on geometric graph nodes, such as IUPAC order for biomolecules, because the FFT-based convolution is not permutation equivariant except under cyclic shifts.","fun_headline_variants_meta":{"raw":{"variants":["Equivariant long-conv model: global context in O(N log N)","Sub-quadratic equivariance: Geometric Hyena beats transformers","Global equivariant context for molecules at O(N log N) cost","Geometric Hyena: equivariant global context with less compute"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1400,"prompt_tokens":903,"completion_tokens":497,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":422}},"tokens_in":519,"tokens_out":497,"duration_ms":5680,"temperature":1.0,"reasoning_tokens":422,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:02:39.757901+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a molecular benchmark with fixed coordinates and randomly permute the node order while keeping the same structure; if Geometric Hyena's error rises sharply compared with a permutation-equivariant baseline like EGNN, then the sequence ordering, not the geometry, is carrying the reported advantage.","supporting_citations":[{"cited_title":"Y., Dao, T., Baccus, S., Bengio, Y., Ermon, S., and R \\'e , C","cited_arxiv_id":null,"evidence_quote":"Supplies the long-convolution operator and gating design that Geometric Hyena makes equivariant."},{"cited_title":"and Smidt, T","cited_arxiv_id":null,"evidence_quote":"Equiformer is the main equivariant self-attention baseline benchmarked for accuracy and cost."},{"cited_title":"Openvaccine: Covid-19 mrna vaccine degradation prediction, 2020","cited_arxiv_id":null,"evidence_quote":"OpenVaccine supplies the RNA stability prediction benchmark."},{"cited_title":"Tuning the performance of synthetic riboswitches using machine learning","cited_arxiv_id":null,"evidence_quote":"Tc-Ribo supplies the riboswitch switching-factor prediction benchmark."},{"cited_title":"Equivariant graph hierarchy-based neural networks","cited_arxiv_id":null,"evidence_quote":"ProteinMD supplies the molecular-dynamics trajectory prediction benchmark."}],"review_version":1}