REVIEW 3 major objections 5 minor 34 references
Benchmarking machine learning models for predicting aerofoil performance
T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read This paper benchmarks four neural networks on the windAI_bench aerofoil dataset and finds that graph-based models overfit while MLP and PointNet generalize best to unseen aerofoils.
desk verdict Useful first benchmark of four surrogate models on NREL's windAI_bench with an unseen-foil split, but the specific claim that PointNet beats MLP on lift-coefficient accuracy rests on an unvalidated panel-method integration and does not hold at most training sizes. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The object that carries the comparison is the vorticity-to-lift evaluation pipeline, not any single network. Each model takes point-cloud inputs in a radius around the foil and predicts density, momentum, energy, and vorticity; the predicted vorticity at the aerofoil surface is then fed through the panel-method circulation integral $\Gamma=\int\int_A \omega\,dA$ and the lift relation $C_L=2\Gamma/(U_\infty c)$ to obtain a lift coefficient. The benchmark ranking is built from the RMSE of these predicted fields and of $C_L$ on 60 held-out aerofoils, repeated for training sets of 5, 20, 55, and 150 aerofoils.
What would settle it
Compute $C_L$ from the ground-truth vorticity fields in the windAI_bench test aerofoils using the paper's circulation integral $\Gamma=\int\int_A \omega\,dA$ and compare those $C_L$ values with the dataset's $C_L$ values; if the ground-truth $C_L$ error is comparable to or larger than the models' $C_L$ RMSE, the panel-method conversion, not the networks, is driving the reported lift errors.
Extended reading notes
Core claim
Working with the windAI_bench dataset, the paper trains four neural networks (MLP, PointNet, GraphSAGE, GUNet) on 5, 20, 55, and 150 aerofoils at 25 angles of attack and evaluates them on 60 unseen aerofoils, i.e. 1,500 test simulations. Its central finding is that the graph-based models fit the training data well but generalise poorly, with test RMSEs regularly more than double those of MLP and PointNet, whereas MLP gives the most consistently accurate predictions of the flow field and PointNet achieves the single best error for $C_L$ (0.748 after training on 55 aerofoils). The lift coefficients are not direct model outputs: each model's predicted vorticity is converted to circulation by $\Gamma=\int\int_A \omega\,dA$ and then to $C_L=2\Gamma/(U_\infty c)$ via the panel method, and the resulting $C_L$ values are compared with the dataset's values.
Load-bearing premise
The ranking of models by lift accuracy rests on the assumption that the panel-method formula $C_L=2\Gamma/(U_\infty c)$ converts predicted vorticity into a trustworthy lift coefficient; the paper does not check this conversion against ground-truth vorticity.
Editorial extensions
If this is right
- GraphSAGE and GUNet should be considered poorly suited to predicting flow over unseen aerofoils; their test errors are consistently around double or more those of MLP and PointNet at every training-set size.
- MLP is the most dependable architecture for predicting the flow field itself: it gives the lowest surface and fluid RMSE in seven of the eight test configurations.
- PointNet is competitive for lift prediction and achieved the single best $C_L$ RMSE (0.748) when trained on 55 aerofoils, despite not always having the best vorticity error.
- Five aerofoils of training data are insufficient for stable results; larger training sets help, but $C_L$ error does not smoothly decrease with more data, indicating the panel-method post-processing is part of the error budget.
Reading between the lines
- Beyond the paper: if the graph models' failure is overfitting rather than an architectural ceiling, early stopping, weight decay, or dropout could shrink the gap to MLP and PointNet; that would be a direct test of the overfitting explanation.
- Beyond the paper: because the windAI_bench data files do not record aerofoil names, the paper cannot check whether the train/test split is balanced across foil families; auditing the split could show that some geometries dominate the $C_L$ errors.
- Beyond the paper: the panel method cannot produce drag, so a full performance benchmark would need direct $C_D$ prediction or an alternative post-processing route; the same four-model comparison could be rerun with $C_D$ as the target.
- Beyond the paper: computing $C_L$ from the dataset's own ground-truth vorticity with the same panel-method integral would separate model error from post-processing error and would either confirm or overturn the reported lift-coefficient ranking.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper benchmarks four neural network models (MLP, PointNet, GraphSAGE, GUNet) on the NREL windAI_bench airfoil_2k dataset. The models are trained on subsets of 5, 20, 55, and 150 aerofoils and tested on 60 unseen aerofoils, predicting flow variables (density, momentum, energy, vorticity) and then computing the lift coefficient from predicted vorticity via a panel-method-style integration. The reported RMSEs show that GraphSAGE and GUNet have low training error but much higher test error than MLP and PointNet. The paper concludes that MLP and PointNet are the strongest models, with MLP more often accurate for flow-field prediction and PointNet more accurate for CL. The authors position the work as the first public benchmark of contemporary ML models on windAI_bench and compare their methodology with the AirfRANS benchmark.
Significance. If the broad ranking is substantiated, this is a useful reference for practitioners choosing ML architectures for aerofoil flow prediction in data-scarce settings, and it fills a real gap by testing on unseen foils, which the AirfRANS benchmark does not do. Strengths of the paper include the use of an external public dataset, a test set of unseen geometries, the release of code on GitHub, the comparison of four architectures, and the reporting of inference times. However, the more specific claim about PointNet's CL accuracy is not consistently supported by the tables and rests on an unvalidated post-processing step, so the significance of that particular result is currently limited.
major comments (3)
- [Section III-C, Eqs. (5)-(7)] The CL metric is computed by integrating predicted vorticity and comparing with the windAI_bench CL, but the integration is never validated by applying it to ground-truth vorticity fields. Without such validation, the reported CL RMSE conflates model prediction error with integration or discretization error, and the abstract's claim that PointNet gives more accurate CL is not established. The Discussion's own remark about the 'simplified nature of the panel method' (Section V) reinforces the need for this check. Please demonstrate that the integration reproduces the dataset CL when applied to the true vorticity, or restrict the conclusions to the flow-variable RMSEs.
- [Tables I-IV; Abstract] The claim that PointNet provides more accurate CL is only supported in the 55-foil experiment (Table III: 0.748 vs 0.795). In Tables I, II, and IV the MLP has the lower CL RMSE (0.873 vs 1.019, 0.997 vs 1.015, and 0.861 vs 0.954, respectively). No error bars or significance tests are reported, so the small observed differences cannot be distinguished from random variation. Please report multiple-seed statistics and revise the abstract and conclusion to state that PointNet achieved the lowest CL RMSE in one configuration, rather than that it generally provides more accurate CL.
- [Section VI Conclusion] The conclusion states that MLP's total RMSE at the surface, in the fluid, and for CL decreased with every increase in aerofoil number. This is contradicted by the tables for CL: MLP CL RMSE rises from 0.873 on 5 foils to 0.997 on 20 foils, and again from 0.795 on 55 foils to 0.861 on 150 foils. Please correct this statement and discuss the non-monotonic CL trend, which is directly relevant to the claim that more training data improves lift prediction.
minor comments (5)
- [Section III-A] The normalisation in Eq. (2) defines sigma* as rho* U / a_infinity, which appears to be dimensionless Mach-scaled density rather than a momentum variable; this inconsistency should be clarified or corrected.
- [Section IV / Table V] The comparison would be more interpretable with a statement of the training budget and hyperparameters (epochs, learning rate, batch size) and whether all models were given the same computational budget; otherwise the overfitting narrative for GraphSAGE and GUNet is plausible but not directly evidenced.
- [Section V] The paper acknowledges that foil names are not available, making it impossible to verify whether the training and test splits are balanced across foil families such as NACA and non-NACA foils; this limitation should be stated more prominently in the conclusions.
- [Abstract and Title] The title and abstract refer to 'aerofoil performance', but the paper only computes lift; drag is mentioned but not calculated. Consider narrowing the framing to lift prediction.
- [Section III-C] The term 'panel method' is used for the integration in Eqs. (5)-(7), but Eq. (5) integrates volumetric vorticity over an area; the relationship to the classical panel method should be clarified.
Circularity Check
No significant circularity: the benchmark compares independently trained models against external NREL ground truth using standard physics post-processing.
full rationale
The paper's central claim—that PointNet and MLP outperform GraphSAGE and GUNet on held-out aerofoils—is established by training each model on the external windAI_bench dataset and testing on 60 unseen aerofoils, with errors measured against ground-truth fluid variables provided by NREL. No parameter is fitted to the CL values that are later reported as predictions: CL is obtained from predicted vorticity through the standard panel-method relations (Eqs. 5–7), and then compared to the dataset's published CL values. The model architectures are inherited from the AirfRANS benchmark, but that is an external methodological choice, not a self-citation used to justify the conclusion. None of the paper's equations define a target in terms of the fitted parameters, and no uniqueness theorem or prior work by the same authors is invoked to force the ranking. The unvalidated panel-method integration is a legitimate concern about correctness or measurement error, but it is not circular: the integration is a fixed physical post-processing rule applied identically to all models, and the comparison target is external to the training. The paper is self-contained against external benchmarks, so the circularity score is 0.
Assumptions & free parameters
free parameters (2)
- Buffer zone distance for surface node extraction =
3.5e-6 m
- Radial cutoff for retaining mesh points =
0.7 m
assumptions (4)
- domain assumption windAI_bench HAM2D simulations provide accurate ground-truth flow fields and lift coefficients
- domain assumption Panel method with Gamma = integral of omega dA and CL = 2*Gamma/(U_infinity*c) accurately estimates lift from predicted vorticity
- domain assumption Shapely polygon with a 3.5e-6 m buffer correctly identifies aerofoil surface nodes
- domain assumption The 60-foil test set is representative and the split is unbiased
Cite this review
Pith. "Pith review of Benchmarking machine learning models for predicting aerofoil performance." pith.science (2026). https://pith.science/paper/PP36O3VX
@misc{pith2026250415993,
author = {Pith},
title = {Pith review of: Benchmarking machine learning models for predicting aerofoil performance},
year = {2026},
howpublished = {\url{https://pith.science/paper/PP36O3VX}},
note = {Machine review of arXiv:2504.15993}
}
abstract
This paper investigates the capability of Neural Networks (NNs) as alternatives to the traditional methods to analyse the performance of aerofoils used in the wind and tidal energy industry. The current methods used to assess the characteristic lift and drag coefficients include Computational Fluid Dynamics (CFD), thin aerofoil and panel methods, all face trade-offs between computational speed and the accuracy of the results and as such NNs have been investigated as an alternative with the aim that it would perform both quickly and accurately. As such, this paper provides a benchmark for the windAI_bench dataset published by the National Renewable Energy Laboratory (NREL) in the USA. In order to validate the methodology of the benchmarking, the AirfRANSdataset benchmark is used as both a starting point and a point of comparison. This study evaluates four neural networks (MLP, PointNet, GraphSAGE, GUNet) trained on a range of aerofoils at 25 angles of attack (4$^\circ$ to 20$^\circ$) to predict fluid flow and calculate lift coefficients ($C_L$) via the panel method. GraphSAGE and GUNet performed well during the training phase, but underperformed during testing. Accordingly, this paper has identified PointNet and MLP as the two strongest models tested, however whilst the results from MLP are more commonly correct for predicting the behaviour of the fluid, the results from PointNet provide the more accurate results for calculating $C_L$.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
A review of the uk and british channel islands practical tidal stream energy re- source,
D. Coles, A. Angeloudis, D. Greaves, G. Hastie, M. Lewis, L. MacKie, J. McNaughton, J. Miles, S. Neill, M. Piggott, D. Risch, B. Scott, C. Sparling, T. Stallard, P . Thies, S. Walker, D. White, R. Willden, and B. Williamson, “A review of the uk and british channel islands practical tidal stream energy re- source,” Proceedings of the Royal Society A: Mathe...
work page 2021
-
[2]
Contracts for difference (cfd) allocation round 6: results,
D. for Energy Security and N. Zero, “Contracts for difference (cfd) allocation round 6: results,” 2024. [Online]. Available: https://www.gov.uk/government/publications/ contracts-for-difference-cfd-allocation-round-6-results
work page 2024
-
[3]
B. Mannion, S. B. Leen, and S. Nash, “Development and assessment of a blade element momentum theory model for high solidity vertical axis tidal turbines,” Ocean Engineering , vol. 197, 2020. [Online]. Available: https://www.sciencedirect. com/science/article/pii/S0029801820300020
work page 2020
-
[4]
Rotor blade performance analysis with blade element momentum theory,
F. Mahmuddin, “Rotor blade performance analysis with blade element momentum theory,” Energy Procedia , vol. 105, pp. 1123–1129, 2017. [Online]. Available: https://www. sciencedirect.com/science/article/pii/S1876610217305180
work page 2017
-
[5]
Xfoil vs cfd performance predictions for high lift low reynolds number airfoils,
J. Morgado, R. Vizinho, M. Silvestre, and J. P ´ascoa, “Xfoil vs cfd performance predictions for high lift low reynolds number airfoils,” Aerospace Science and T echnology , vol. 52, pp. 207–214, 2016. [Online]. Available: https://www.sciencedirect. com/science/article/pii/S1270963816300839
work page 2016
-
[6]
Cfdbench: A large-scale benchmark for machine learning methods in fluid dynamics,
Y. Luo, Y. Chen, and Z. Zhang, “Cfdbench: A large-scale benchmark for machine learning methods in fluid dynamics,”
-
[7]
W. Xu, N. Grande Gutierrez, and C. McComb, “Megaflow2d: A parametric dataset for machine learning super-resolution in computational fluid dynamics simulations,” in Proceedings of Cyber-Physical Systems and Internet of Things Week 2023 , ser. CPS-IoT Week ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 100–104. [Online]. Available: ht...
arXiv 2023
-
[8]
Airfoil computational fluid dynamics - 2k shapes, 25 aoa’s, 3 re numbers,
D. Ramos, A. Glaws, R. King, B. Lee, O. Doronina, J. Baeder, G. Vijayakumar, and Z. Grey, “Airfoil computational fluid dynamics - 2k shapes, 25 aoa’s, 3 re numbers,” 02 2023. [Online]. Available: https://data.openei.org/submissions/5970
work page 2023
Show all 34 references
-
[9]
AirfRANS: High fidelity computational fluid dynamics dataset for approximating reynolds-averaged navier–stokes solutions,
F. Bonnet, J. A. Mazari, P . Cinnella, and P . Gallinari, “AirfRANS: High fidelity computational fluid dynamics dataset for approximating reynolds-averaged navier–stokes solutions,” Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks T rack...
2022 arXiv
-
[10]
Multilayer feedforward networks are universal approximators,
K. Hornik, M. Stinchcombe, and H. White, “Multilayer feedforward networks are universal approximators,” Neural Networks, vol. 2, no. 5, pp. 359–366, 1989. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ 0893608089900208
1989
-
[11]
Adam: A method for stochastic optimization,
D. P . Kingma and J. Ba, “Adam: A method for stochastic optimization,” 3rd International Conference for Learning Representations , 2017. [Online]. Available: https://arxiv.org/abs/1412.6980
2017 arXiv
-
[12]
Prediction of swirling flow field in combustor based on deep learning,
Z. Wang, K. Gong, W. Fan, C. Li, and W. Qian, “Prediction of swirling flow field in combustor based on deep learning,” Acta Astronautica, vol. 201, pp. 302–316, 12 2022
2022
-
[13]
Artificial neural networks in renewable energy systems applications: a review,
S. A. Kalogirou, “Artificial neural networks in renewable energy systems applications: a review,” Renewable and Sustainable Energy Reviews , vol. 5, pp. 373–401, 2001. [Online]. Available: www.elsevier.com/locate/rser
2001
-
[14]
A new model for learning in graph domains,
M. Gori, G. Monfardini, and F. Scarselli, “A new model for learning in graph domains,” in Proceedings. 2005 IEEE Interna- tional Joint Conference on Neural Networks, 2005. , vol. 2, 2005, pp. 729–734 vol. 2
2005
-
[15]
A comprehensive survey on graph neural networks,
Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and P . S. Yu, “A comprehensive survey on graph neural networks,” IEEE T ransactions on Neural Networks and Learning Systems , vol. 32, no. 1, pp. 4–24, 2021
2021
-
[16]
End-to-end wind turbine wake modelling with deep graph representation learning,
S. Li, M. Zhang, and M. D. Piggott, “End-to-end wind turbine wake modelling with deep graph representation learning,” Applied Energy , vol. 339, 6 2023
2023
-
[17]
Grid adaptive reduced-order model of fluid flow based on graph convolutional neural network,
J.-Z. Peng, Y.-Z. Wang, S. Chen, Z.-H. Chen, W.-T. Wu, and N. Aubry, “Grid adaptive reduced-order model of fluid flow based on graph convolutional neural network,” Physics of Fluids, vol. 34, no. 8, p. 087121, 08 2022. [Online]. Available: https://doi.org/10.1063/5.0100236
2022 doi
-
[18]
Graph convolutional networks applied to unstructured flow field data,
F. Ogoke, K. Meidani, A. Hashemi, and A. B. Farimani, “Graph convolutional networks applied to unstructured flow field data,” Machine Learning: Science and T echnology , vol. 2, 12 2021
2021
-
[19]
Semi-supervised classification with graph convolutional networks,
T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” 9 2016. [Online]. Available: http://arxiv.org/abs/1609.02907
2016 arXiv
-
[20]
Pointnet: Deep learning on point sets for 3d classification and segmentation,
C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,”
-
[21]
Inductive representation learning on large graphs,
W. L. Hamilton, R. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” NeurIPS, 2018. [Online]. Available: https://arxiv.org/abs/1706.02216
2018 arXiv
-
[22]
Graph u-nets,
H. Gao and S. Ji, “Graph u-nets,” in Proceedings of the 36th International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09– 15 Jun 2019, pp. 2083–2092. [Online]. Available: https://proce...
2019
-
[23]
U-net: Convolutional networks for biomedical image segmentation,
O. Ronneberger, P . Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” International Conference On Medical Image Computing And Computer Assisted Intervention, 5 2015. [Online]. Available: http://arxiv.org/abs/ 1505.04597
2015 arXiv
-
[24]
Fast graph representation learning with pytorch geometric,
M. Fey and J. E. Lenssen, “Fast graph representation learning with pytorch geometric,” ICLR, 3 2019. [Online]. Available: http://arxiv.org/abs/1903.02428
2019 arXiv
-
[25]
Scalable gradi- ent–enhanced artificial neural networks for airfoil shape design in the subsonic and transonic regimes,
M. A. Bouhlel, S. He, and J. R. Martins, “Scalable gradi- ent–enhanced artificial neural networks for airfoil shape design in the subsonic and transonic regimes,” Structural and Multidis- ciplinary Optimization , vol. 61, pp. 1363–1376, 4 2020
2020
-
[26]
Introduction to hdf5,
M. I. of Technology (MIT), “Introduction to hdf5,” last Accessed on: 2025-04-15. [Online]. Available: https://web.mit. edu/fwtools v3.1.0/www/H5.intro.html
2025
-
[27]
Shapely,
S. Gillies, C. van der Wel, J. Van den Bossche, M. W. Taves, J. Arnott, B. C. Ward, and others, “Shapely,” Apr. 2025. [Online]. Available: https://github.com/shapely/shapely
2025
-
[28]
Osher and R
S. Osher and R. Fedkiw, Constructing Signed Distance Functions . New York, NY: Springer New York, 2003, pp. 63–74. [Online]. Available: https://doi.org/10.1007/0-387-22746-6 7
2003 doi
-
[29]
Intermediate fluid mechanics,
J. Liburdy, “Intermediate fluid mechanics,” Tech. Rep., 2021
2021
-
[30]
Geometrical effects on the airfoil flow separation and transition,
W. Zhang, W. Cheng, W. Gao, A. Qamar, and R. Samtaney, “Geometrical effects on the airfoil flow separation and transition,” Computers & Fluids , vol. 116, pp. 60–73, 2015. [Online]. Available: https://www.sciencedirect.com/science/ article/pii/S0045793015001292
2015
-
[31]
Effects of relative thickness on aerodynamic characteristics of airfoil at a low reynolds number,
D. Ma, Y. Zhao, Y. Qiao, and G. Li, “Effects of relative thickness on aerodynamic characteristics of airfoil at a low reynolds number,” Chinese Journal of Aeronautics , vol. 28, no. 4, pp. 1003–1015, 2015. [Online]. Available: https://www. sciencedirect.com/science/article/pii...
2015
-
[32]
On the role and chal- lenges of cfd in the aerospace industry,
P . R. Spalart and V . Venkatakrishnan, “On the role and chal- lenges of cfd in the aerospace industry,”The Aeronautical Journal, vol. 120, no. 1223, p. 209–232, 2016
2016
-
[2017]
Available: https://arxiv.org/abs/1612.00593
[Online]. Available: https://arxiv.org/abs/1612.00593
-
[2024]
Available: https://arxiv.org/abs/2310.05963
[Online]. Available: https://arxiv.org/abs/2310.05963
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.