Pith. sign in

REVIEW 3 major objections 4 minor 21 references

Universal Transforming Geometric Network

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Replacing the recurrent core of an end-to-end protein structure predictor with a Universal Transformer encoder lowers average dRMSD by 1.7 Å on free-modeling CASP12 targets and 0.7 Å on template-based targets.

desk verdict A clean but statistically thin swap of RNNs for Universal Transformers in protein structure prediction; the idea is worth refereeing, but the headline margins could easily be initialization noise. read the letter →

arxiv 1908.00723 v1 pith:HR472Y6J submitted 2019-08-02 q-bio.BM cs.LG

Jin Li This is my paper · ORCID
classification q-bio.BMcs.LG
keywords proteinstructurepredictionend-to-enddifferentiablelearningUniversalTransformerrecurrentgeometricnetworkCASP12dRMSDtorsionalanglesself-attention
topics Protein Folding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Protein structure prediction from an amino-acid sequence is a central problem in computational biology, and end-to-end differentiable models offer a direct sequence-to-structure mapping. This paper proposes UTGN, a modification of the recurrent geometric network (RGN) that replaces the recurrent-neural-network core with the encoder of a Universal Transformer, a self-attention architecture that refines all residue representations in parallel. On the CASP12 benchmark, the best UTGN variant reports a 1.7 Å lower dRMSD than RGN on free-modeling targets and 0.7 Å lower on template-based targets, with better TM-scores, faster convergence, and no exploding-gradient instability. The result is offered as evidence that the internal representation, not just the geometric output layer, is a key design choice for end-to-end protein structure prediction.

What carries the argument

The load-bearing machinery is the Universal Transformer encoder used as the internal representation. Unlike an RNN, which passes information along the residue order, the UT encoder recurs over revisions of the whole sequence: at each revision, multi-head scaled dot-product self-attention lets every residue attend to every other residue, then a position-wise transition function refines the representations, with layer normalization and residual connections repeated over six layers. The best-performing transition is a one-dimensional separable convolution (kernel size 3), which mixes positions and channels in separate steps; a fully-connected transition performs worse. Adaptive Computation Time lets each residue stop being revised once a halting probability crosses a threshold. The final representations are decoded through a learned alphabet of 60 angle triples into backbone torsional angles, and a recurrent geometric unit converts those angles into 3D Cartesian coordinates; the training loss is dRMSD between predicted and experimental distance maps.

What would settle it

Run UTGN-SepConv and RGN from several random initializations with matched parameter counts and matched training budgets, then plot per-target dRMSD distributions for the FM and TBM splits; the central claim fails if the best RGN run overlaps UTGN's median or if the gap closes when total training time or hyperparameters are equalized.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that replacing the RNN internal representation in RGN with the encoder portion of the Universal Transformer produces a better end-to-end differentiable protein structure predictor at roughly matched model size (about two million parameters each). The best configuration, UTGN-SepConv, lowers average dRMSD from 19.8 Å to 18.1 Å on free-modeling CASP12 targets and from 17.8 Å to 17.1 Å on template-based targets, with corresponding TM-score gains of 0.013 and 0.008. UTGN with a fully-connected transition improves dRMSD on both categories but does not improve TM-score on template-based targets, so the separable-convolution transition is part of the claimed advantage. The paper also reports that UTGN converges about twice as fast, runs about six times faster per epoch, never shows RGN's exploding-gradient behavior, and is less sensitive to initialization.

Load-bearing premise

The central comparison rests on the single RGN run being a fair, typical baseline: the paper reports no variance estimates, and if that run was poorly initialized or undertuned, the 1.7 Å and 0.7 Å margins could shrink or disappear.

Editorial extensions

If this is right

  • If the reported margins hold, an end-to-end differentiable model without biophysical priors can outperform a recurrent baseline on CASP12, making the choice of internal representation a primary factor in structure prediction accuracy.
  • UTGN's faster, more stable training lowers the practical cost of end-to-end protein structure prediction and reduces the need for multiple random restarts.
  • The superiority of UTGN-SepConv over UTGN-FF indicates that the transition function inside the transformer is not incidental; separable convolution is a design choice worth preserving or improving.
  • Because the torsional-angle-to-Cartesian decoder is shared with RGN, the encoder swap can be evaluated in isolation: any accuracy gain is attributed to the representation, not the geometric output layer.
  • A scale-up to more parameters and longer training, which the paper identifies as a requirement for state-of-the-art performance, is a direct next step if the reported training stability persists.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The comparison does not control for every architectural difference—head count, layer count, and transition type differ between UTGN and RGN—so the reported gains should not be read as proof that self-attention alone is responsible; an ablation with matched components would settle that.
  • The paper notes that RGN initialization can strongly affect results but reports a single RGN run; re-running both models across random seeds would show whether the 1.7 Å and 0.7 Å margins are robust or partly an artifact of the chosen baseline.
  • A length-stratified breakdown of dRMSD on CASP12 targets would directly test the paper's 'global dependencies' explanation: the transformer advantage should grow on longer proteins if limited long-range information flow is the real bottleneck in RNNs.
  • The same encoder-swap recipe could transfer to other sequence-to-geometry problems, such as RNA or polymer structure prediction, or to versions that replace PSSM features with learned sequence embeddings; these are extensions the paper suggests but does not test.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes the Universal Transforming Geometric Network (UTGN), an end-to-end differentiable protein structure prediction architecture that replaces the recurrent neural network (RNN) internal representation of the earlier Recurrent Geometric Network (RGN) with the encoder portion of the Universal Transformer. The model uses multi-head self-attention with either a fully-connected or a separable-convolution transition function, adaptive computation time, and an angularization layer that converts final hidden states into torsional angles and then Cartesian coordinates. Training uses a dRMSD loss on the CASP12 ProteinNet dataset. The authors report that UTGN with a separable-convolution transition improves over RGN by 1.7 Å dRMSD on free-modeling targets and 0.7 Å on template-based modeling targets, while also claiming faster convergence and more stable training. The architecture is described in moderate detail, and a public code link is provided.

Significance. If the reported accuracy improvements are statistically reliable, this is a meaningful contribution: it demonstrates that a transformer-based, recurrent-refinement encoder can replace RNNs in end-to-end protein structure prediction, with potential benefits for global dependency modeling and training stability. The paper also provides a useful architectural description and makes code available, which supports reproducibility. However, the central comparative claim currently rests on a single baseline run, and the paper itself notes that RGN performance is highly initialization-sensitive. The absence of variance estimates, repeated runs, or per-target paired comparisons means the headline margins are not yet established as beyond baseline noise. The contribution is therefore promising but requires additional empirical support before the central claim can be accepted.

major comments (3)
  1. [Section 3.5, Tables 1 and 2] The headline improvements of 1.7 Å on FM and 0.7 Å on TBM are computed against a single RGN run, while the authors state in Section 3.5 that "different initializations in RGNs can produce very different evaluation results." Without a distribution over RGN initializations, confidence intervals, or per-target paired differences, the reported margins cannot be distinguished from initialization noise. Please report results over multiple seeds for both RGN and UTGN, along with per-target paired comparisons and an estimate of variance.
  2. [Section 3.2, Tables 1 and 2] The comparison conflates the change of internal representation with other architectural differences: UTGN uses 8 heads, 6 layers, a transition function that is either fully connected or a separable convolution, and an input dimension of 256, whereas the RGN baseline uses a recurrent size of 240. The claim that replacing the RNN with a Universal Transformer is the cause of the improvement is therefore not established. Please include a matched-capacity baseline or an ablation that isolates the representation change, such as an RGN with a comparable number of parameters or a UTGN variant with an RNN-style transition.
  3. [Section 3.5, Tables 1 and 2] The generic "UTGN" claim is weakened by the observation that the UTGN-FF variant has worse TM-score than RGN on both FM (0.174 vs 0.181) and TBM (0.198 vs 0.200). Since the positive conclusion rests entirely on the UTGN-SepConv configuration, the paper should either restrict the claim to that specific variant or provide an explanation for why the TM-score regression under the FF transition does not undercut the architecture-level conclusion.
minor comments (4)
  1. [Section 3.5, Eq. (14)] The text and tables use both "RMSD" and "dRMSD" inconsistently: Eq. (14) defines a global RMSD after alignment, while the reported metric in Tables 1 and 2 is labeled dRMSD. Please clarify which metric is actually reported and ensure the notation is consistent throughout.
  2. [Section 3.5, training claims] The claims that RGN takes about 6 times longer per epoch and that UTGN converges about 2 times faster are not supported by any learning curves, wall-clock measurements, or epoch counts. Please provide quantitative evidence or temper these claims.
  3. [Section 3.3 and Eq. (13)] The dRMSD loss expression in Eq. (13) appears to lack a square root in the denominator; the typical definition is the Frobenius norm divided by sqrt(L(L-1)). If the implemented loss is a constant multiple of the correct dRMSD, this does not affect the relative comparison, but the formula should be corrected or clarified.
  4. [Throughout] The manuscript contains several typographical and grammatical errors, such as "UTGN achieve" in the abstract and Section 5, "the protein prediction competition was dominated" in the discussion, and "recur over representations" in Section 2.2. A careful proofreading pass would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the UTGN comparison is an empirical held-out evaluation whose predictions are not defined by the inputs.

full rationale

The paper's central claim is an empirical architecture comparison on CASP12. The UTGN model is constructed from standard components (one-hot and PSSM input features, positional encodings, universal-transformer encoder, dynamic halting, angular prediction, recurrent geometric units) and is trained with a supervised dRMSD loss against experimental structures. The reported improvements in dRMSD and TM-score come from held-out CASP12 test targets from ProteinNet, not from fitting a parameter to those targets and then renaming the fit as a prediction. Hyperparameter choices such as head count, layer count, and transition type are made before the reported evaluation and are standard model selection, not circular reasoning. The RGN baseline is an externally defined architecture from AlQuraishi, and the paper does not rely on a self-citation chain or an imported uniqueness theorem to justify its choices. The paper's admission that different RGN initializations produce very different results is a robustness concern about the comparison, not evidence that the claimed 1.7 Angstrom and 0.7 Angstrom margins are definitionally forced by the model's construction. No equation in the paper defines the predicted structure or the evaluation metrics in terms of the target quantity itself, and no fitted parameter is renamed as a prediction. Therefore, no specific circular step can be exhibited.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard supervised training and a set of hand-chosen hyperparameters. No new physical entity is introduced. The fixed-bond approximation and the benchmark choice are assumptions inherited from RGN and ProteinNet. Learned weights are fit to training data, which is normal for machine learning, but it means the paper's claim is empirical rather than a first-principles derivation.

free parameters (7)
  • UT encoder hidden dimension = 256
    Hand-chosen in Section 3.2; affects capacity and is not derived from data.
  • Number of attention heads = 8
    Hand-chosen in Section 3.2.
  • Number of UT layers = 6
    Hand-chosen in Section 3.2.
  • ACT halting threshold = 0.5
    Hand-chosen in Section 3.2.
  • Maximum ACT recurrence = 10
    Hand-chosen in Section 3.2.
  • Angularization alphabet size = 60
    Hand-chosen in Section 3.2; sets the discretization of torsional angles.
  • Learned network weights = approximately 2 million parameters
    Fitted to the training split through the dRMSD loss; standard supervised learning, but the central claim depends on this training.
assumptions (4)
  • domain assumption Backbone bond lengths and bond angles can be treated as fixed while torsional angles vary.
    Invoked in Section 2.4 for structure construction; inherited from RGN and not justified in this paper.
  • domain assumption The ProteinNet CASP12 thinning-90% split, with missing residues excluded from dRMSD, is a valid benchmark for comparing models.
    Used in Section 3.1 without analysis of how missing-residue exclusion affects the comparison.
  • standard math Universal Transformer self-attention, layer normalization, and adaptive computation time behave as described in the cited works.
    Background components from Dehghani et al., Vaswani et al., Ba et al., and Graves, not re-derived here.
  • domain assumption dRMSD as a training loss and RMSD/TM-score as evaluation metrics faithfully reward structure quality.
    The paper adopts these metrics as standard without analyzing their agreement or potential biases.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Universal Transforming Geometric Network." pith.science (2026). https://pith.science/paper/HR472Y6J

@misc{pith2026190800723,
  author       = {Pith},
  title        = {Pith review of: Universal Transforming Geometric Network},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HR472Y6J}},
  note         = {Machine review of arXiv:1908.00723}
}
abstract

The recurrent geometric network (RGN), the first end-to-end differentiable neural architecture for protein structure prediction, is a competitive alternative to existing models. However, the RGN's use of recurrent neural networks (RNNs) as internal representations results in long training time and unstable gradients. And because of its sequential nature, it is less effective at learning global dependencies among amino acids than existing transformer architectures. We propose the Universal Transforming Geometric Network (UTGN), an end-to-end differentiable model that uses the encoder portion of the Universal Transformer architecture as an alternative for internal representations. Our experiments show that compared to RGN, UTGN achieve a $1.7$ \si{\angstrom} improvement on the free modeling portion and a $0.7$ \si{\angstrom} improvement on the template based modeling of the CASP12 competition.

Figures

Figures reproduced from arXiv: 1908.00723 by the authors.

Figure 1
Figure 1. Vector representation of the amino acids are fed into the encoder of the UT. During each [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Complete architecture of the encoder portion of the universal transformer. [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Scatter-plot comparing individual FM and TBM predictions of RGN and UTGN feed [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Scatter-plot comparing individual FM and TBM predictions of RGN and UTGN with [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Comparison between predicted and actual structure of proteins T0865 (RMSD of [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 14 canonical work pages

  1. [1]

    Ethan C Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M. Church. Unified rational protein engineering with sequence-only deep representation learning. 2019

    work page 2019

  2. [2]

    End-to-end differentiable learning of protein structure

    Mohammed AlQuraishi. End-to-end differentiable learning of protein structure. Cell systems, 8 4: 0 292--301.e3, 2019 a

    work page 2019

  3. [3]

    Proteinnet: a standardized data set for machine learning of protein structure

    Mohammed AlQuraishi. Proteinnet: a standardized data set for machine learning of protein structure. In BMC Bioinformatics, 2019 b

    work page 2019

  4. [4]

    Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. ArXiv, abs/1607.06450, 2016

    arXiv 2016

  5. [5]

    Xception: Deep learning with depthwise separable convolutions

    François Chollet. Xception: Deep learning with depthwise separable convolutions. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 1800--1807, 2016

    work page 2017

  6. [6]

    Universal transformers

    Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. ArXiv, abs/1807.03819, 2019

    arXiv 2019

  7. [7]

    Adaptive computation time for recurrent neural networks

    Alex Graves. Adaptive computation time for recurrent neural networks. ArXiv, abs/1603.08983, 2016

    arXiv 2016

  8. [8]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014

    arXiv 2014

Show all 21 references
  1. [9]

    Irwin D. Kuntz. Structure-based strategies for drug design and discovery. Science, 257 5073: 0 1078--82, 1992

    1992

  2. [10]

    Ab initio molecular dynamics: basic theory and advanced methods

    Dominik Marx and Jurg Hutter. Ab initio molecular dynamics: basic theory and advanced methods. Cambridge University Press, 2010

    2010

  3. [11]

    Judson, and Krzysztof Fidelis

    John Moult, Jesper Tejlgaard Pedersen, Richard S. Judson, and Krzysztof Fidelis. A large-scale experiment to assess protein structure prediction methods. Proteins, 23 3: 0 ii--v, 1995

    1995

  4. [12]

    Understanding the exploding gradient problem

    Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. Understanding the exploding gradient problem. ArXiv, abs/1211.5063, 2012

    2012 arXiv

  5. [13]

    Potter, Aurelien Luciani, Sean R

    Simon C. Potter, Aurelien Luciani, Sean R. Eddy, Youngmi Park, Rodrigo Lopez, and Robert D. Finn. Hmmer web server: 2018 update. In Nucleic Acids Research, 2018

    2018

  6. [14]

    Self-attention with relative position representations

    Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In NAACL-HLT, 2018

    2018

  7. [15]

    Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov

    Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15: 0 1929--1958, 2014

    1929

  8. [16]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017

    2017

  9. [17]

    Accurate de novo prediction of protein contact map by ultra-deep learning model

    Sheng Wang, Siqi Sun, Zhen Li, Renyu Zhang, and Jinbo Xu. Accurate de novo prediction of protein contact map by ultra-deep learning model. bioRxiv, pp.\ 073239, 2016

    2016

  10. [18]

    How significant is a protein structure similarity with tm-score = 0.5? Bioinformatics, 26 7: 0 889--95, 2010

    Jinrui Xu and Yang Zhang. How significant is a protein structure similarity with tm-score = 0.5? Bioinformatics, 26 7: 0 889--95, 2010

    2010

  11. [19]

    The i-tasser suite: protein structure and function prediction

    Jianyi Yang, Renxiang Yan, Ambrish Roy, Dong Lai Xu, Jonathan Poisson, and Yang Arthur Zhang. The i-tasser suite: protein structure and function prediction. Nature Methods, 12: 0 7--8, 2014

    2014

  12. [20]

    Scoring function for automated assessment of protein structure template quality

    Yang Zhang and Jeffrey Skolnick. Scoring function for automated assessment of protein structure template quality. Proteins, 57 4: 0 702--10, 2004

    2004

  13. [21]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.