{"id":"ba299e30-253e-4a7f-b91e-dfbdc8ab4b28","arxiv_id":"2504.15993","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"On NREL's windAI_bench aerofoil dataset, MLP and PointNet outperform GraphSAGE and GUNet on unseen foils, with MLP best overall for flow prediction and PointNet best for lift coefficient only at one training size.","lead":"This paper tests four machine learning models on a public dataset of simulated airflow over wind and tidal turbine aerofoil shapes. It finds that the simpler MLP and PointNet models generalize best to unseen foils, while GraphSAGE and GUNet overfit.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The CL-based ranking of PointNet over MLP rests on an unvalidated panel-method integration; ground-truth vorticity may not reproduce the dataset's CL, so the reported CL RMSE may be dominated by integration error.","rationale":"I agree with the reader that the weakest point is the unvalidated CL post-processing. The central claim has two components: fluid-field prediction, where MLP is generally best, and CL accuracy, where PointNet wins. The second component is the one that differentiates the two recommended models, and it depends entirely on the panel method. The paper provides public code and a reproducible dataset, and the fluid-field RMSE tables are internally consistent; these give independent support for the general ranking of MLP and PointNet over graph models. But because the panel-method CL was not checked against ground-truth vorticity, the fine-grained CL ranking is not yet reliable. Other concerns, such as missing aerofoil names, lack of uncertainty estimates, and small training sizes, are real limitations but do not directly invalidate the relative ranking as much as an unvalidated metric. The proposed test is simple and can be done with the existing dataset and code. Therefore the reader's CONDITIONAL verdict is appropriate; no change in verdict is needed, but the condition should be made explicit.","tokens_in":12495,"tokens_out":3893,"duration_ms":37220,"concrete_test":"Recompute CL from the ground-truth vorticity fields for the 60 test aerofoils using the same panel-method integration (Eqs. 5-7) and surface-node selection as in Section III-C, and compare against the CL values in windAI_bench. If the ground-truth CL RMSE is comparable to or greater than the inter-model differences (roughly 0.1-0.2 in Tables III and IV), then the CL metric is dominated by integration or discretization error and the PointNet-vs-MLP CL ranking cannot be trusted. A secondary check is to vary the buffer-zone distance used to identify surface nodes; if CL RMSE changes materially, surface-node selection is another uncontrolled factor.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim distinguishes PointNet and MLP on the basis of CL accuracy computed in Section III-C: predicted vorticity is integrated via Eqs. 5-7 to get circulation and CL, then compared to windAI_bench CL values. This assumes the panel method, applied to predicted vorticity, is an unbiased estimator of the dataset's CL. No validation is reported: the same integration is never applied to ground-truth vorticity fields. If the integration itself has systematic error, the CL RMSE conflates model error with post-processing error. The model differences are small, e.g., Table III CL RMSE: MLP 0.795 vs PointNet 0.748; Table IV: MLP 0.861 vs PointNet 0.954, so a modest integration bias could change the ranking. The Discussion even notes the 'simplified nature of the panel method' and that surface-averaged vorticity RMSE does not track CL RMSE. Unless the integration is validated, the specific claim that PointNet is more accurate for CL is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks four neural network models (MLP, PointNet, GraphSAGE, GUNet) on the NREL windAI_bench airfoil_2k dataset. The models are trained on subsets of 5, 20, 55, and 150 aerofoils and tested on 60 unseen aerofoils, predicting flow variables (density, momentum, energy, vorticity) and then computing the lift coefficient from predicted vorticity via a panel-method-style integration. The reported RMSEs show that GraphSAGE and GUNet have low training error but much higher test error than MLP and PointNet. The paper concludes that MLP and PointNet are the strongest models, with MLP more often accurate for flow-field prediction and PointNet more accurate for CL. The authors position the work as the first public benchmark of contemporary ML models on windAI_bench and compare their methodology with the AirfRANS benchmark.","tokens_in":12697,"tokens_out":6003,"duration_ms":56022,"significance":"If the broad ranking is substantiated, this is a useful reference for practitioners choosing ML architectures for aerofoil flow prediction in data-scarce settings, and it fills a real gap by testing on unseen foils, which the AirfRANS benchmark does not do. Strengths of the paper include the use of an external public dataset, a test set of unseen geometries, the release of code on GitHub, the comparison of four architectures, and the reporting of inference times. However, the more specific claim about PointNet's CL accuracy is not consistently supported by the tables and rests on an unvalidated post-processing step, so the significance of that particular result is currently limited.","major_comments":[{"comment":"The CL metric is computed by integrating predicted vorticity and comparing with the windAI_bench CL, but the integration is never validated by applying it to ground-truth vorticity fields. Without such validation, the reported CL RMSE conflates model prediction error with integration or discretization error, and the abstract's claim that PointNet gives more accurate CL is not established. The Discussion's own remark about the 'simplified nature of the panel method' (Section V) reinforces the need for this check. Please demonstrate that the integration reproduces the dataset CL when applied to the true vorticity, or restrict the conclusions to the flow-variable RMSEs.","section":"Section III-C, Eqs. (5)-(7)"},{"comment":"The claim that PointNet provides more accurate CL is only supported in the 55-foil experiment (Table III: 0.748 vs 0.795). In Tables I, II, and IV the MLP has the lower CL RMSE (0.873 vs 1.019, 0.997 vs 1.015, and 0.861 vs 0.954, respectively). No error bars or significance tests are reported, so the small observed differences cannot be distinguished from random variation. Please report multiple-seed statistics and revise the abstract and conclusion to state that PointNet achieved the lowest CL RMSE in one configuration, rather than that it generally provides more accurate CL.","section":"Tables I-IV; Abstract"},{"comment":"The conclusion states that MLP's total RMSE at the surface, in the fluid, and for CL decreased with every increase in aerofoil number. This is contradicted by the tables for CL: MLP CL RMSE rises from 0.873 on 5 foils to 0.997 on 20 foils, and again from 0.795 on 55 foils to 0.861 on 150 foils. Please correct this statement and discuss the non-monotonic CL trend, which is directly relevant to the claim that more training data improves lift prediction.","section":"Section VI Conclusion"}],"minor_comments":[{"comment":"The normalisation in Eq. (2) defines sigma* as rho* U / a_infinity, which appears to be dimensionless Mach-scaled density rather than a momentum variable; this inconsistency should be clarified or corrected.","section":"Section III-A"},{"comment":"The comparison would be more interpretable with a statement of the training budget and hyperparameters (epochs, learning rate, batch size) and whether all models were given the same computational budget; otherwise the overfitting narrative for GraphSAGE and GUNet is plausible but not directly evidenced.","section":"Section IV / Table V"},{"comment":"The paper acknowledges that foil names are not available, making it impossible to verify whether the training and test splits are balanced across foil families such as NACA and non-NACA foils; this limitation should be stated more prominently in the conclusions.","section":"Section V"},{"comment":"The title and abstract refer to 'aerofoil performance', but the paper only computes lift; drag is mentioned but not calculated. Consider narrowing the framing to lift prediction.","section":"Abstract and Title"},{"comment":"The term 'panel method' is used for the integration in Eqs. (5)-(7), but Eq. (5) integrates volumetric vorticity over an area; the relationship to the classical panel method should be clarified.","section":"Section III-C"}],"recommendation":"major_revision","confidential_remarks":"The paper is a competent benchmark study with reproducible code and an external test set. The main work needed is to calibrate the claims to the evidence, specifically around the unvalidated CL post-processing and the inconsistent PointNet-CL ranking. I see no integrity or scope concerns beyond the overclaiming noted in the report."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is the first public benchmark of MLP, PointNet, GraphSAGE, and GUNet on NREL's windAI_bench with a proper unseen-foil test split, and the data-scaling behavior across 5 to 150 foils is genuinely useful. The broad result—GraphSAGE and GUNet overfit and generalize badly, while MLP and PointNet are the useful models—is credible and consistent across the tables. The authors also write honestly about limitations, including the missing foil names in the dataset and the simplified panel method.\n\nThe main soft spot is the CL post-processing. The paper computes CL by integrating predicted vorticity via the panel method (Eqs. 5–7) and compares to the dataset's CL, but it never validates that this integration reproduces the dataset CL from ground-truth vorticity. Given that integration error could easily be on the order of the reported differences (e.g., Table III: MLP CL 0.795 vs. PointNet 0.748; Table IV: MLP 0.861 vs. PointNet 0.954), the specific claim that PointNet is more accurate for CL is not established. At 5, 20, and 150 foils, MLP actually has lower CL RMSE; PointNet only wins at 55. So the abstract overreaches.\n\nRelatedly, there are no error bars or significance tests, so small differences between the two best models are not distinguishable. And the train/test split cannot be audited because the dataset does not expose foil names—a limitation the authors themselves flag.\n\nThe code is public and the experimental setup is standard, which is good. The comparison to AirfRANS is appropriate and carefully qualified, and the paper is clear about why a direct numerical comparison is not possible.\n\nWho is this for? Engineers or ML researchers wanting a sanity check on which surrogate models to try on wind or tidal foil data. It is a useful incremental contribution, not a breakthrough. The central ranking of graph models versus point/MLP methods is likely right, but the finer PointNet-versus-MLP CL claim needs validation of the integration and ideally repeated training runs.\n\nI would send this to peer review—it deserves referee time—with the clear expectation that the CL validation and uncertainty quantification be addressed. I would not yet cite it as a definitive benchmark result.","headline":"Useful first benchmark of four surrogate models on NREL's windAI_bench with an unseen-foil split, but the specific claim that PointNet beats MLP on lift-coefficient accuracy rests on an unvalidated panel-method integration and does not hold at most training sizes.","tokens_in":13223,"tokens_out":2479,"would_cite":false,"duration_ms":21591,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper benchmarks four neural networks on the windAI_bench aerofoil dataset and finds that graph-based models overfit while MLP and PointNet generalize best to unseen aerofoils.","keywords":["aerofoil performance prediction","neural networks","windAI_bench","lift coefficient","panel method","graph neural networks","PointNet","generalisation"],"falsifier":"Compute $C_L$ from the ground-truth vorticity fields in the windAI_bench test aerofoils using the paper's circulation integral $\\Gamma=\\int\\int_A \\omega\\,dA$ and compare those $C_L$ values with the dataset's $C_L$ values; if the ground-truth $C_L$ error is comparable to or larger than the models' $C_L$ RMSE, the panel-method conversion, not the networks, is driving the reported lift errors.","tokens_in":12281,"feed_emoji":"✈️","tokens_out":12471,"duration_ms":108124,"temperature":0.7,"pith_summary":"This paper asks which neural architecture can predict aerofoil performance accurately enough to serve as a fast complement to CFD in wind and tidal turbine design, and it supplies a first benchmark for the windAI_bench dataset with four contemporary models. It trains MLP, PointNet, GraphSAGE, and GUNet on 5, 20, 55, and 150 aerofoils and tests them on 60 aerofoils the models have never seen. The central finding is that the graph-based models fit training data well but fail on unseen aerofoils, while MLP and PointNet generalize best; MLP is more often right about the fluid field, and PointNet delivers the single best lift-coefficient error (a $C_L$ RMSE of 0.748 after training on 55 aerofoils). The practical upshot for a designer is that simpler point-cloud and MLP models are the safer choice, and graph architectures need a demonstration of generalisation before being trusted.","feed_headline":"Graph neural nets overfit aerofoils; MLP and PointNet generalize best","feed_subtitle":"Benchmark of four models on 2,000 foil shapes finds graph models train well but fail on unseen foils.","key_machinery":"The object that carries the comparison is the vorticity-to-lift evaluation pipeline, not any single network. Each model takes point-cloud inputs in a radius around the foil and predicts density, momentum, energy, and vorticity; the predicted vorticity at the aerofoil surface is then fed through the panel-method circulation integral $\\Gamma=\\int\\int_A \\omega\\,dA$ and the lift relation $C_L=2\\Gamma/(U_\\infty c)$ to obtain a lift coefficient. The benchmark ranking is built from the RMSE of these predicted fields and of $C_L$ on 60 held-out aerofoils, repeated for training sets of 5, 20, 55, and 150 aerofoils.","core_discovery":"Working with the windAI_bench dataset, the paper trains four neural networks (MLP, PointNet, GraphSAGE, GUNet) on 5, 20, 55, and 150 aerofoils at 25 angles of attack and evaluates them on 60 unseen aerofoils, i.e. 1,500 test simulations. Its central finding is that the graph-based models fit the training data well but generalise poorly, with test RMSEs regularly more than double those of MLP and PointNet, whereas MLP gives the most consistently accurate predictions of the flow field and PointNet achieves the single best error for $C_L$ (0.748 after training on 55 aerofoils). The lift coefficients are not direct model outputs: each model's predicted vorticity is converted to circulation by $\\Gamma=\\int\\int_A \\omega\\,dA$ and then to $C_L=2\\Gamma/(U_\\infty c)$ via the panel method, and the resulting $C_L$ values are compared with the dataset's values.","pith_inferences":["Beyond the paper: if the graph models' failure is overfitting rather than an architectural ceiling, early stopping, weight decay, or dropout could shrink the gap to MLP and PointNet; that would be a direct test of the overfitting explanation.","Beyond the paper: because the windAI_bench data files do not record aerofoil names, the paper cannot check whether the train/test split is balanced across foil families; auditing the split could show that some geometries dominate the $C_L$ errors.","Beyond the paper: the panel method cannot produce drag, so a full performance benchmark would need direct $C_D$ prediction or an alternative post-processing route; the same four-model comparison could be rerun with $C_D$ as the target.","Beyond the paper: computing $C_L$ from the dataset's own ground-truth vorticity with the same panel-method integral would separate model error from post-processing error and would either confirm or overturn the reported lift-coefficient ranking."],"forward_implications":["GraphSAGE and GUNet should be considered poorly suited to predicting flow over unseen aerofoils; their test errors are consistently around double or more those of MLP and PointNet at every training-set size.","MLP is the most dependable architecture for predicting the flow field itself: it gives the lowest surface and fluid RMSE in seven of the eight test configurations.","PointNet is competitive for lift prediction and achieved the single best $C_L$ RMSE (0.748) when trained on 55 aerofoils, despite not always having the best vorticity error.","Five aerofoils of training data are insufficient for stable results; larger training sets help, but $C_L$ error does not smoothly decrease with more data, indicating the panel-method post-processing is part of the error budget."],"supporting_citations":[{"why":"Supplies the windAI_bench dataset used for training and testing, including aerofoil geometries, flow fields, and ground-truth $C_L$ values.","marker":"[8]"},{"why":"Supplies the four model architectures (MLP, PointNet, GraphSAGE, GUNet) and the AirfRANS benchmark methodology this paper adapts.","marker":"[9]"},{"why":"Supplies the panel-method equations used to compute circulation and $C_L$ from predicted vorticity.","marker":"[29]"},{"why":"Defines PointNet, one of the two architectures identified as strongest.","marker":"[20]"},{"why":"Defines GraphSAGE, one of the graph architectures that trains well but fails on test data.","marker":"[21]"},{"why":"Defines GUNet, the other graph architecture that underperforms on unseen aerofoils.","marker":"[22]"}],"fun_headline_variants":["Aerofoil AI: graph nets overfit, MLP and PointNet lead","Graph models overfit aerofoils; MLP and PointNet excel","Neural nets for aerofoils: graphs overfit, PointNet wins CL","Benchmark: graph nets overfit, MLP predicts flow, PointNet lift","MLP and PointNet beat graph nets in aerofoil benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking of models by lift accuracy rests on the assumption that the panel-method formula $C_L=2\\Gamma/(U_\\infty c)$ converts predicted vorticity into a trustworthy lift coefficient; the paper does not check this conversion against ground-truth vorticity.","fun_headline_variants_meta":{"raw":{"variants":["Aerofoil AI: graph nets overfit, MLP and PointNet lead","Graph models overfit aerofoils; MLP and PointNet excel","Neural nets for aerofoils: graphs overfit, PointNet wins CL","Benchmark: graph nets overfit, MLP predicts flow, PointNet lift","MLP and PointNet beat graph nets in aerofoil benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000568,"raw_usage":{"total_tokens":2732,"prompt_tokens":1031,"completion_tokens":1701,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":647,"completion_tokens_details":{"reasoning_tokens":1599}},"tokens_in":647,"tokens_out":1701,"duration_ms":12231,"temperature":1.0,"reasoning_tokens":1599,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:12:36.299509+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute $C_L$ from the ground-truth vorticity fields in the windAI_bench test aerofoils using the paper's circulation integral $\\Gamma=\\int\\int_A \\omega\\,dA$ and compare those $C_L$ values with the dataset's $C_L$ values; if the ground-truth $C_L$ error is comparable to or larger than the models' $C_L$ RMSE, the panel-method conversion, not the networks, is driving the reported lift errors.","supporting_citations":[{"cited_title":"Airfoil computational fluid dynamics - 2k shapes, 25 aoa’s, 3 re numbers,","cited_arxiv_id":null,"evidence_quote":"Supplies the windAI_bench dataset used for training and testing, including aerofoil geometries, flow fields, and ground-truth $C_L$ values."},{"cited_title":"Intermediate fluid mechanics,","cited_arxiv_id":null,"evidence_quote":"Supplies the panel-method equations used to compute circulation and $C_L$ from predicted vorticity."},{"cited_title":"Pointnet: Deep learning on point sets for 3d classification and segmentation,","cited_arxiv_id":null,"evidence_quote":"Defines PointNet, one of the two architectures identified as strongest."},{"cited_title":"Graph u-nets,","cited_arxiv_id":null,"evidence_quote":"Defines GUNet, the other graph architecture that underperforms on unseen aerofoils."}],"review_version":1}