Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Training Transformers for Mesh-Based Simulations

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A Transformer that uses the mesh adjacency matrix as a sparse attention mask beats message-passing networks on 3D CFD, cutting all-rollout error by 52% versus MeshGraphNet at similar training speed.

desk verdict Solid empirical work on adjacency-masked graph transformers for CFD, but the headline 38.8% SOTA gain doesn't hold up against the paper's own Table 1 once you count the PLATE dataset. read the letter →

arxiv 2508.18051 v1 pith:AF222KOB submitted 2025-08-25 cs.LG

Paul Garnier , Vincent Lannelongue , Jonathan Viquerat , Elie Hachem This is my paper · ORCID
classification cs.LG
keywords graphtransformersmesh-basedsimulationcomputationalfluiddynamicsadjacencymatrixattentionmaskingscalinglawsmessagepassingmeshgraphs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a Transformer whose attention is masked by the mesh adjacency matrix can replace message-passing graph networks for learned physics simulation, as long as the mask is augmented with three cheap tricks: dilated hops, random shortcut edges, and a few globally connected boundary nodes. On six 3D CFD benchmarks, the largest such model improves on the previous best published results by 38.8% on average and reduces full-rollout error by 52% relative to MeshGraphNet at similar training speed, while the smallest model matches MeshGraphNet at roughly one seventh the compute and one sixth the size. The paper also claims a scaling law, optimal parameter count growing like the 0.75 power of the training FLOPs budget, that lets a practitioner pick model size for a fixed budget. A sympathetic reader would care because this points to attention as a more scalable default than message passing for large, complex meshes, without needing multigrid or mesh-coarsening machinery.

What carries the argument

The load-bearing mechanism is the masked multi-head self-attention block, which computes $(A \odot \operatorname{softmax}(QK^\top/\sqrt{d}))V$ with the undirected adjacency matrix $A$ as a hard sparsity mask, so each node attends only to mesh neighbors while the mask is applied inside the softmax via the Hadamard product. Three augmentations of $A$ carry the receptive-field expansion: Dilation uses $A^2$ or $A^3$ on half the heads of later layers; Random Connections adds 20% random symmetric edges regenerated at each step; Global Attention connects 1% of dataset-specific boundary nodes to all other nodes. The augmented matrix keeps attention cost near $O(n^{1+\lambda})$ rather than $O(n \cdot \deg_{\max}^K)$, and the Gated MLP with GeLU keeps per-layer FLOPs close to the $2P$ approximation used for scaling-law analysis.

What would settle it

On a fresh mesh dataset with no boundary-condition node labels (no inlet, wall, obstacle, or airfoil types), train the architecture with only dilation and random edges and compare all-rollout RMSE to MeshGraphNet under identical noise, epochs, and FLOPs; if the gap is far below the 38.8% average improvement reported here, the global-attention component is doing the work. A second, cheaper check: rerun the PLATE contact benchmark with edge-type features restored, since the paper's own result there shows the edge-free design's advantage largely disappears.

Watch

Extended reading notes

Core claim

The central discovery, stated on the paper's own terms, is that the graph's adjacency matrix is a sufficient and efficient attention mask for learned simulation, and that three augmentations of that matrix—dilated sliding windows (using powers of the adjacency matrix on selected heads), random connections regenerated at every training and inference step, and global attention to a small sample of physically important boundary nodes—widen the effective receptive field at near-linear cost. With more than 60 trained models the paper shows the architecture obeys a compute-optimal scaling law, $N \propto C^{0.75}$, and that the resulting 51M-parameter XL model outperforms the previous state of the art by 38.8% on average and MeshGraphNet by 52% on all-rollout RMSE, while the smallest S model reaches MeshGraphNet parity with 7x faster training. The paper argues the gain comes from replacing message passing's fixed aggregation with multi-head softmax weighting that can turn off irrelevant neighbors, which mitigates over-squashing and preserves feature diversity across layers.

Load-bearing premise

The load-bearing premise is that the per-dataset choice of globally attended nodes (inlet, wall, airfoil, and obstacle nodes) is a legitimate, fixed part of the method and that the baselines were tuned and run under comparable compute budgets; if either fails on a new mesh, the reported edge over message passing would shrink.

Editorial extensions

If this is right

  • Attention masked by the graph adjacency matrix can substitute for message passing entirely, giving a better accuracy-per-FLOP trade-off: matched accuracy at one-sixth the parameter count, or a 38.8% average error reduction at matched training speed.
  • The measured scaling law $N \propto C^{0.75}$ gives a concrete recipe for sizing a mesh Transformer from a FLOPs budget, with an optimal model around 50M parameters for 30 epochs on the coarse aneurysm dataset.
  • The three augmentations generalize to new meshes without multigrid or mesh coarsening, supporting training on meshes up to 300k nodes and 3 million edges on a single GPU.
  • Masked pretraining, applied by stacking two transformers and reconstructing masked node features, is compatible with the architecture and lifts the augmented XL model from 25% to 38.8% average improvement over the previous state of the art.
  • On out-of-distribution meshes the coarse-trained model is only 80% worse on all-rollout RMSE than a model trained on the fine meshes, and remains better than MeshGraphNet trained directly on the fine meshes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If global attention is treated as a tunable component, the honest test on a new geometry is to drop the per-dataset boundary-node choice and use a fixed global token or random node subset; the paper's ablation shows the method still helps with dilation and random edges alone, so the drop should be measurable rather than fatal.
  • The PLATE result suggests contact- and boundary-dominated problems are where the edge-free design gives up the most; adding edge-type or signed-distance features back into the node tokens is a natural extension that could recover that margin while keeping the transformer structure.
  • Because the paper trains its scaling law on one 3D dataset, the exponent 0.75 may be dataset-dependent; a testable extension is to repeat the isoFLOP sweep on a non-CFD mesh task and check whether the exponent moves toward the 0.5 found in much larger language-model studies.
  • The architecture's use of raw coordinates as positional features makes it non-equivariant under rotations and translations; a follow-up that replaces coordinates with a relative or equivariant embedding would test whether the 38.8% gain survives geometric augmentations.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript introduces a Graph Transformer for mesh-based physical simulation in which the adjacency matrix is used directly as a sparse attention mask, augmented by Dilated Sliding Windows, Random Connections, and Global Attention. The authors report an extensive ablation study, a scaling-law analysis based on over 60 trained models, and final results claiming that the largest model surpasses the previous state-of-the-art by 38.8% on average and beats MeshGraphNet by 52% on all-rollout RMSE, while the smallest model matches MeshGraphNet at 7x faster and 6x smaller. The paper also openly discusses the PLATE dataset limitation, the removal of edge features, and the lack of invariance to geometric transformations.

Significance. If the central claim were reproducible, this would be a significant contribution: it would show that a simple sparse-attention Transformer can outperform message-passing baselines on several CFD surrogate benchmarks, while providing a practical scaling rule for model sizing. Strengths of the paper include public code and datasets, the breadth of the adjacency-matrix ablation, the use of standard benchmarks, and the explicit disclosure of the PLATE weakness. However, the headline numerical claim is not reproducible from the paper's own Table 1, and several secondary claims depend on choices fitted to the same data used for final evaluation, so the empirical contribution currently requires revision.

major comments (4)
  1. [Abstract and Table 1] The headline claim that the largest model surpasses SOTA by 38.8% on average is not reproducible from Table 1. Using the per-dataset best prior model in the all-rollout rows and including Masked Multigrid as a prior SOTA, as the table itself lists it, the Masked XL/1 model is worse than the prior best on PLATE (13.8 vs. 4.5, both times 10^-3) and improves on the other five datasets by roughly 53%, 27%, 49%, 62%, and 45%, giving a simple per-dataset average of about 5%, not 38.8%. If the claimed average excludes Masked Multigrid, or excludes PLATE, or is computed against a different baseline set, that choice is not disclosed in the abstract or Section 6.3. Please state explicitly which baselines define SOTA for each dataset, report the per-dataset relative improvements, and recompute the headline average; as written, the claim is misleading.
  2. [Table 1 and Figure 1] The central empirical comparison is presented without error bars in Table 1. Figure 1 reports standard deviations over five runs for some configurations, but the table does not, and it is not stated whether the Masked XL/1 numbers are the mean of multiple seeds or a single run. Because the 38.8% claim depends on ratios between numbers that can vary across seeds, please report mean and standard deviation (or confidence intervals) for the six datasets and for the baselines, or state clearly where the per-run results are available.
  3. [Section 6.1, Figure 3] The scaling-law exponent N proportional to C^0.75 and the 'optimal' 50M model size are fitted to training loss and validation RMSE on the 3D-COARSE ANEURYSM dataset and then used to select the final model sizes that are evaluated on that same dataset and its near relatives. As presented, this is a descriptive fit rather than a predictive scaling law; it does not validate the exponent on held-out data. Please either validate the exponent on a different dataset or a different mesh resolution, or explicitly reframe the section as a fit to the chosen dataset and state that the exponent was not used as an independent prediction.
  4. [Section 3 and Table A.1] The final configuration depends on dataset-specific salience knowledge: the Global Attention nodes in Table A.1 are chosen per dataset as wall, inlet, obstacle, or airfoil nodes, and the Table A.1 note says they were selected based on the influence of their boundary conditions on the FEM solver. Furthermore, the noise scales in Section A.1 were set partly by looking at the model's average one-step error. This means the reported gains over message passing may partially reflect dataset-specific tuning rather than the architecture alone. Please quantify the sensitivity to these choices, for example by comparing the chosen global nodes against random node subsets of the same size, and state in the main text that applying the method to a new mesh requires identifying salient boundary nodes.
minor comments (6)
  1. [Section 2.3] The phrase 'Laplacien Eigen Vectors' contains a typo and should be 'Laplacian Eigenvectors'.
  2. [Table 2] The parameter counts in Table 2 use decimal commas (0,012, 0,019, 0,072) while the rest of the paper uses decimal points; please standardize the notation.
  3. [Section 6.3] The claims 'outperforms the current SOTA by 25%' and 'achieves an average improvement of 38.8%' are stated without the formula or per-dataset table behind them; Section 6.3 should reference the calculation explicitly.
  4. [Equation (1)] The masked attention formula applies softmax to the full QK^T matrix before the Hadamard product with A; please clarify whether the attention weights are renormalized after masking, since this affects the interpretation of the attention distribution.
  5. [Section 1 and Section 3] The statement that Random Connections and Global Attention are 'the first time' these augmentations are adapted to Graph Neural Networks is too strong, given the existence of sparse global-token and random-connection patterns in long-sequence transformers; please qualify the novelty claim.
  6. [Appendix G] The non-physics results in Table 4 are reported without standard deviations, and the MNIST accuracy of 98.2 is below the cited SOTA of 98.76; the sentence 'our architecture is robust even without any Positional Encoding' should be supported by error bars or by a statement that the differences are within noise.

Circularity Check

1 steps flagged · score 3.0 of 10

Scaling-law 'confirmation' is in-sample (the fitted curve is read back to select XL), but the headline SOTA benchmark comparison is external and not circular.

  1. fitted input called prediction [Section 6.1 (Scaling Laws), with model list in Appendix B Table 2]
    "For six given FLOPs budgets ..., we train an extensive range of models for different number of training steps on the 3D-C OARSE ANEURYSM dataset and select the final training loss ... For each FLOPs budget, we plot the parameters of the local minima model and fit a power law ... Based on said power law, the optimal size for a model trained on the 3D-C OARSE ANEURYSM for 30 epochs is around 50M parameters. We confirm this hypothesis by training a large base model (XL) of this size."

    The power law N∝C^0.75 is fitted to models trained on 3D-COARSE-ANEURYSM, and Appendix B (Table 2) lists the XL/51M model among the models trained for the isoFLOPs curves. The 'optimal size around 50M' is read off that same fitted curve and then 'confirmed' by training XL, so the confirmation is an in-sample consistency check rather than an independent prediction. The size is not forced by definition (any exponent could fit), so this is a mild circularity, not a complete reduction.

full rationale

The central architecture claim — adjacency-matrix-masked transformer with augmented receptive fields — is evaluated against external baselines (MeshGraphNet, BSMS-GNN, Multigrid, Masked Multigrid, etc.) on public or published CFD datasets, so the SOTA comparison does not reduce to fitted values. The masking pretraining is cited from the authors' prior MeshMask work, but that is an external prior publication and the current paper's architecture results are not derived solely from that citation. The only identifiable circularity is in the scaling-law section: the 'prediction' of the optimal ~50M model size is obtained from a power law fitted to models trained on the same dataset, and the XL model used to 'confirm' it is listed among the models trained for that same isoFLOPs study. This makes the confirmation in-sample rather than a true out-of-sample prediction. Separately, the abstract's '38.8% average SOTA improvement' is not straightforwardly reproducible from Table 1 when Masked Multigrid is taken as the per-dataset best baseline (notably on PLATE), but that is a benchmarking/reporting inconsistency, not a circularity. Overall, the main benchmark results have independent content, so the circularity score is low.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central performance claim is empirical and rests on several fitted design choices, dataset-specific node selection, and the isoFLOP fitting procedure. No new physical entities are introduced.

free parameters (5)
  • Global Attention node fraction = 1% (ablation tested 1%, 5%, 10%)
    Selected by ablation on 3D-COARSE ANEURYSM; final models use 1% (Section 6.2).
  • Random Edge fraction = 20% (ablation tested 10%, 20%, 50%)
    Selected by ablation; 20% gives the best improvement per added training time (Section 6.2).
  • Dilation schedule = A^2 on half of the heads in the last 5 layers
    Selected by ablation among 2-Dilation, 3-Dilation, and 2-3-Dilation (Sections 3 and 6.2).
  • Scaling-law exponent = 0.75
    Power-law fit to isoFLOP curves on 3D-COARSE ANEURYSM using final training loss (Section 6.1).
  • Input noise scales = Per dataset, listed in Table A.1
    Chosen from prior papers or from the average one-step error of an un-noised model (Appendix A.1).
assumptions (5)
  • ad hoc to paper Edge attributes can be omitted without losing information essential to the target physics.
    The architecture drops edge features entirely; the authors note this removes edge-type indicators and is one reason PLATE gains are marginal (Sections 2 and 7).
  • domain assumption Node coordinates are a sufficient positional encoding for CFD graphs.
    Sections 2.3 and 6.2 conclude Laplacian and random-walk encodings are worse than raw 3D coordinates; this underlies the non-invariance of the model.
  • ad hoc to paper Sparse local attention with random and global connections is at least as expressive as message passing for these PDEs.
    Appendix C gives informal arguments based on softmax weighting and multiple heads, not a theorem; the method's motivation assumes this premise.
  • domain assumption FLOPs is well approximated by 2P for the proposed transformer and is not for message passing.
    Appendix D uses this approximation for all isoFLOP curves and scaling-law fits; if the ratio is wrong, the optimal model size shifts.
  • domain assumption Training and evaluation on FEM-generated datasets with 100 training trajectories is representative of mesh-based simulation performance.
    All headline results are on these benchmarks, with one coarse-to-fine transfer test (Sections 4 and 6.3); cross-solver generalization is not established.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Training Transformers for Mesh-Based Simulations." pith.science (2026). https://pith.science/paper/AF222KOB

@misc{pith2026250818051,
  author       = {Pith},
  title        = {Pith review of: Training Transformers for Mesh-Based Simulations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AF222KOB}},
  note         = {Machine review of arXiv:2508.18051}
}
abstract

Simulating physics using Graph Neural Networks (GNNs) is predominantly driven by message-passing architectures, which face challenges in scaling and efficiency, particularly in handling large, complex meshes. These architectures have inspired numerous enhancements, including multigrid approaches and $K$-hop aggregation (using neighbours of distance $K$), yet they often introduce significant complexity and suffer from limited in-depth investigations. In response to these challenges, we propose a novel Graph Transformer architecture that leverages the adjacency matrix as an attention mask. The proposed approach incorporates innovative augmentations, including Dilated Sliding Windows and Global Attention, to extend receptive fields without sacrificing computational efficiency. Through extensive experimentation, we evaluate model size, adjacency matrix augmentations, positional encoding and $K$-hop configurations using challenging 3D computational fluid dynamics (CFD) datasets. We also train over 60 models to find a scaling law between training FLOPs and parameters. The introduced models demonstrate remarkable scalability, performing on meshes with up to 300k nodes and 3 million edges. Notably, the smallest model achieves parity with MeshGraphNet while being $7\times$ faster and $6\times$ smaller. The largest model surpasses the previous state-of-the-art by $38.8$\% on average and outperforms MeshGraphNet by $52$\% on the all-rollout RMSE, while having a similar training speed. Code and datasets are available at https://github.com/DonsetPG/graph-physics.

Figures

Figures reproduced from arXiv: 2508.18051 by the authors.

Figure 1
Figure 1. RMSE over autoregressive trajectories on the COARSE ANEURYSM Dataset. We showcase a Message Passing approach (MPS with 15 layers of width 128), a bi-stride (BSMS), 2 attention based models and our architecture. (left) Performances are shown without any masking pre-training and with a standard Adjacency Matrix without augmentations. Our new architecture constantly beats the previous SOTA based on message-passing arch… view at source ↗
Figure 2
Figure 2. (left) Details of the Augmented Adjacency matrices (1) Default Adjacency Matrix A. (2) Adjacency Matrix with a Dilation of size 2. (3) Adjacency Matrix with Global Attention based on node 1. (4) Adjacency matrix with Random Jumpers between nodes (1, 5) and (3, 7). (right) Dataset Overview From left to right: CYLINDER, PLATE, AIRFOIL, 2D-ANEURYSM, 3D-COARSEANEURYSM and 3D-ANEURYSM. also studied variations of this pos… view at source ↗
Figure 3
Figure 3. (left) IsoFLOPs curves. For a fixed FLOPs budget, we train different models for the corresponding number of iterations with a matched cosine cycle length. For each isoFLOPs curve, we find a local minimum. (middle) We plot the model at the local minimum of each isoFLOPs curve and show the power law estimation. (right) We display our prediction with the ones from Hoffman and Kaplan. Our trained models are sampled in b… view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: (left) All-Rollout RMSE for every A augmentations. (right) Final training loss with varying number of layers L and embedding size d. Random Edges (RE) Adjacency matrix with random connections offers the best improvements for the smallest cost (15% improvements for a 15…
Figure 5
Figure 5. Figure 5: We display details of 2 datasets: 3D-ANEURYSM and CYLINDER. (top) We display the Systolic velocity streamlines colour-coded with the vertical component on the left, and a detail overview of the mesh of the right. (bottom) We display streamlines of the velocity around a…
Figure 6
Figure 6. Figure 6: Time spent in days for the training of a [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: The Masked Transformer architecture. (left) Our model takes the graph nodes’ features and the adjacency matrix as inputs. The adjacency matrix is improved with Dilation, Global Attention, and Random Jumpers. (middle) Each transformer is made of masked Multi-Head Self-A…
Figure 8
Figure 8. Figure 8: (left) IsoFLOPs curves. For a fixed FLOPs budget, we train different models for the corresponding number of iterations with a matched cosine cycle length. For each isoFLOPs curve, we find a local minimum. (right) We plot the model at the local minimum of each isoFLOPs …
Figure 9
Figure 9. Figure 9: (left) We plot the FLOPs versus the All-Rollout RMSE on the testing set and find a strong correlation. (right) We plot the training loss versus the All-Rollout RMSE on the testing set and find a strong correlation. E.5 Comparison with Message Passing In [PITH_FULL_IMA…
Figure 10
Figure 10. Figure 10: Optimizer and Learning Rate. We plot training loss for models trained with Adam, AdamW, an exponential decay schedule, and a cosine schedule. The cosine schedule and AdamW make All-Rollout RMSE more stable across training and provide better results given sufficient tr…
Figure 11
Figure 11. Figure 11: (left) We plot the training loss on 20 epochs of our models versus a message-passing model with 15 layers. We find the loss to be higher than even our smallest model. (right) We find that the message-passing model makes less efficient training for the same FLOPs as ou…
Figure 14
Figure 14. Figure 14: 21 [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]
Figure 12
Figure 12. Figure 12: Fields are captured from t = 0 and every 50 time steps. Row 1: Velocity field with mesh. Row 2, 3, 4 Magnitude of the attention for each node, in the order A, A2 and Augmented A. Row 5, 6, 7 Magnitude of the attention from a single node, in the order A, A2 and Augment…
Figure 13
Figure 13. Figure 13: Attention from a single node with the Augmented adjacency matrix. Each row is a [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: Uncurated random shapes from the validation cylinder, 2D-Aneurysm, and 3D-Aneurysm [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]
Figure 15
Figure 15. Figure 15: Prediction on one shape from the validation cylinder dataset. [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]
Figure 16
Figure 16. Figure 16: Prediction on 2 shapes from the validation 2D-Aneurysm dataset. [PITH_FULL_IMAGE:figures/full_fig_p024_16.png]
Figure 17
Figure 17. Figure 17: Prediction on 3 shapes from the validation 3D-Aneurysm dataset. [PITH_FULL_IMAGE:figures/full_fig_p025_17.png]
Figure 18
Figure 18. Figure 18: Flow inside the aneurysm, 2D plan comparison and velocity comparison on three selected points. a) Detailed flow analysis of the selected cases with Systolic flow lines inside the bulge. The two highlighted plans are at y = 8 and y = 10. The three selected points used …

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TRELLIS-Enhanced Surface Features for Comprehensive Intracranial Aneurysm Analysis

    cs.CV 2025-09 conditional novelty 6.0 of 10

    TRELLIS-derived surface features improve aneurysm classification, segmentation, and hemodynamic simulation, including a 15% lower blood-flow prediction error.

Reference graph

Works this paper leans on

2 extracted references · 2 canonical work pages · cited by 1 Pith paper

  1. [1]

    in message passing, each edge is processed independently as a large batch of tokens

    work page

  2. [2]

    turn off

    in message passing, information is squashed using an aggregation function Instead of a single aggregator (e.g. sum/mean) or a small MLP that outputs attention coefficients, we have multiple heads plus the softmax QK T √ d . This allows richer, context-dependent weighting of neighbors since each head can highlight different features or patterns in the neig...

    work page 2022

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.