REVIEW 4 major objections 5 minor 1 cited by
Active Learning Enhanced Surrogate Modeling of Jet Engines in JuliaSim
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read An active-learning scheme brings a jet engine surrogate to 0.1 percent relative error.
desk verdict Headline claim contradicts own Table 4; the active learning trick is sensible and the error analysis is honest, but the paper needs a corrected claim and a matched comparison. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key machinery is the adaptive downsampling routine described in Algorithms 1 and 2. It densely samples the input flight envelope with quasi-Monte Carlo or Latin hypercube sampling, simulates about one million points with the physical engine model, computes the output extrema, then quasi-Monte Carlo samples the output space and uses a Manhattan-distance nearest-neighbor search to select a training subset of about 424,000 points whose output distribution is nearly uniform. This rebalanced set is then used to train the Digital Echo surrogate, a data-driven steady-state system that maps parameters to outputs through the equation f(p) = x. The active-learning step is what distinguishes this method from the brute-force baseline, and it is the mechanism the paper credits for the accuracy improvement.
What would settle it
Run the same active-learning surrogate on a dedicated test set of points lying on the Mach-number boundary listed in Table 2 (for example Mach 0.5 at -2000 feet, 0.58 at 10,000 feet, 0.68 at 20,000 feet, and 0.9 at 30,000 to 40,000 feet) and count the fraction with relative error above 0.1 percent; if that fraction is far higher than on the interior points, the claim of uniform flight-envelope accuracy fails.
Extended reading notes
Core claim
The central discovery is that an active-learning pipeline built from output-space quasi-Monte Carlo sampling and nearest-neighbor downsampling produces a Digital Echo surrogate of a turbofan engine with unusually high accuracy. On a test set of 100,000 points, the downsampled surrogate (trained on 424,000 points) places more than 98.4 percent of predictions for all output quantities within 0.1 percent relative error, and for turbine temperature and fuel flow the share reaches 99.98 percent or 100 percent. This improves on the same surrogate trained on a denser one-million-point set without output balancing, which leaves as much as 17.4 percent of net thrust predictions outside the 0.1 percent error band. Specific fuel consumption is not trained directly; it is computed as the ratio of fuel flow to net thrust, so its accuracy follows from the two trained outputs. The paper concludes that most of the remaining high error is confined to the Mach-number boundary of the flight envelope, and it demonstrates that cutting away 20 percent of the Mach range eliminates the majority of those outliers.
Load-bearing premise
The method relies on the premise that making the output distribution of the training set uniform is enough to keep relative error below 0.1 percent across the whole flight envelope, including the Mach-number edges where the paper's own data show errors concentrating and where a 20 percent trim of the input range is then applied.
Editorial extensions
If this is right
- If the surrogate really holds 0.1 percent relative error across the flight envelope, it can serve as a reliable low-cost evaluator for design optimization, uncertainty quantification, or many-query trade studies where the full NPSS engine model would be prohibitively expensive.
- Because specific fuel consumption is computed as the ratio of fuel flow to thrust, the claimed accuracy implies SFC predictions also fall within a similar error band, making the surrogate useful for cycle and mission analyses.
- The 20 percent Mach-number trim, which reduces high-error points by an order of magnitude, suggests a practical rule: when accuracy near the envelope edge is critical, either extend the sampling region or add a dedicated boundary-sampling pass.
- The success of output-space balancing over pure input-space density indicates that for data-hungry neural surrogate models, the distribution of training labels can matter more than the raw number of samples.
- A second adaptive downsampling pass, as the paper itself suggests, could push the already small boundary-error population even lower without changing the core method.
Reading between the lines
- Editorial inference: The procedure is essentially a data-pruning strategy that treats the output distribution as the main source of bias; the same principle could transfer to other physics-based surrogate problems where outputs are highly skewed, such as power-system contingencies or chemical-process simulations.
- Editorial inference: The paper tests the surrogate on 100,000 points after active learning but the pre-active-learning model was tested on one million points, so part of the accuracy gap may come from the different test-set sizes; a like-for-like comparison on the same test set would isolate the effect of rebalancing.
- Editorial inference: Since the boundary errors are tied to the Mach-altitude shape of the flight envelope, a testable extension is to replace the fixed 20 percent trim with adaptive boundary sampling near the Table 2 limits, which could recover the full envelope at equal accuracy.
- Editorial inference: A falsifiable prediction of the paper's logic is that any physics model whose outputs are highly skewed should see a similar accuracy jump from this downsampling scheme; checking that against a different full-order model would show whether the improvement is specific to this engine or generic.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript describes an active-learning workflow for building a surrogate model of a turbofan jet engine using JuliaSim's proprietary Digital Echo method. The workflow densely samples the flight envelope, simulates the full-order NPSS model, then downsamples the training set by quasi-Monte Carlo sampling of the output space and nearest-neighbor matching in the output space, with the goal of balancing the output distribution. The authors report that the resulting surrogate achieves 0.1% relative error for all quantities of interest, and compare this with a baseline surrogate trained on the full 1M-point quasi-Monte Carlo sample. Error analysis shows that the largest errors concentrate near the Mach-number boundary of the flight envelope, and the authors propose cutting 20% of the Mach range to reduce these errors.
Significance. If the headline claim were true, the paper would demonstrate a practically useful surrogate for jet-engine design optimization, with sub-0.1% relative error across the operational envelope. The paper's strengths are that it gives a concrete algorithmic description of the adaptive downsampling routine, reports quantitative accuracy numbers in tabular form, and localizes the residual errors in the input space. However, the central claim as stated in the abstract and introduction is not supported by the paper's own reported numbers, and the comparison against the brute-force baseline is confounded by different test-set sizes and a post hoc domain trim. The contribution is still potentially valuable as an empirical demonstration of output-space downsampling, but the claims need substantial revision and additional boundary-focused evaluation before the paper can be accepted.
major comments (4)
- [Abstract and Section IV, Table 4] The abstract and Section I claim that the surrogate 'demonstrates 0.1% relative error for all quantities of interest,' but Table 4 shows that only 98.44% of PerfInst_Fn and 99.49% of ShH_N test predictions fall within 0.1% relative error. Thus, for two of the four reported quantities, a nontrivial fraction of the test set exceeds the stated tolerance, contradicting the universal claim. The conclusion in Section V correctly softens this to 'greater than 98.4%,' but the abstract and introduction should be revised to match the data.
- [Section IV, Tables 3 and 4, and Figure 4] The pre/post active-learning comparison is confounded: the pre-AL surrogate is tested on 1M points while the post-AL surrogate is tested on 100k points, and the training set sizes differ (1M vs 424k). More importantly, the reported improvement is achieved only after the paper proposes cutting 20% off the Mach-number range (Section IV, Figure 4b). Since the original flight envelope in Table 2 includes the boundary regions, the claimed accuracy across the stated envelope is not demonstrated. The authors should report error statistics on the full envelope, and if the boundary trim is retained, should clearly present it as a restricted-domain result rather than as a success of the active-learning scheme alone.
- [Section III.B, Algorithms 1 and 2] The active-learning method balances the output distribution by nearest-neighbor selection in output space, but this does not address input-space coverage, particularly at the Mach-number boundaries where Figure 4 shows the largest errors. The post hoc 20% Mach trim is effectively an admission that the balancing scheme leaves boundary accuracy unresolved. The paper should either extend the method to explicitly handle boundary coverage or clearly scope the claim to the trimmed domain; as written, the method as described in Algorithms 1 and 2 does not by itself deliver the stated 0.1% accuracy across the full flight envelope.
- [Section III.A, Digital Echo] The core surrogate method, Digital Echo, is referenced to a non-public citation ([17] with no year or venue) and is not described beyond the equations x' = 0 and f(p) = x. Because the entire accuracy result depends on this proprietary method, the paper does not provide enough detail for a reader to reproduce the surrogate or to assess whether the observed errors are intrinsic to Digital Echo or to the active-learning sampling. At minimum, the authors should provide a public technical description or release the surrogate training code, or clearly state the algorithm's hyperparameters and architecture.
minor comments (5)
- [Section IV, Figure 4c] The caption states 'of the 300 or so points with relative error > 0.1%,' but Table 4 reports 1.56% of 100k test points for PerfInst_Fn, which is about 1560 points. This numerical inconsistency should be resolved, or the test-set size and counting procedure clarified.
- [Section V] There is a typo in 'jet enginees' which should be 'jet engines.'
- [Section IV] The word 'analyses' in 'an analyses of the remaining points' should be 'an analysis.'
- [References] Reference [17] is incomplete; it lacks a year, venue, and publication status, which makes it difficult for readers to access the Digital Echo method.
- [Section III.A, step 7] The pipeline description mentions evaluating models on a 'test set' after splitting into training and validation sets in step 4, but does not explain how the test set is generated or whether it is independent of the active-learning selection procedure. Clarifying this would strengthen the validity of the reported error statistics.
Circularity Check
No circularity: the accuracy claim is an empirical held-out evaluation rather than a derivation from fitted inputs.
full rationale
The paper's central claim is an empirical evaluation: a surrogate model trained on NPSS simulation data is assessed on a separately generated held-out test set, with results reported in Tables 3 and 4. The active learning procedure (Algorithms 1 and 2) selects training points by output-space quasi-Monte Carlo sampling and nearest-neighbor matching; this is a data-selection scheme, not a fitted parameter that is later re-labeled as a prediction. The relative-error statistics are measured on test points outside the training set, so the claim does not reduce to the training data or to the downsampling criterion by construction. The Digital Echo surrogate is cited to a non-public author reference, but the paper's accuracy result is supported by its own experiments rather than by that citation; the self-citation names the method rather than serving as the evidence for the empirical claim. Concerns that the abstract's universal '0.1% relative error for all quantities' overstates the tabulated percentages, or that the test set may differ between pre- and post-active-learning comparisons, are correctness and reporting issues, not circularity. No load-bearing derivation step equates a prediction with an input or with a self-cited theorem, so the appropriate circularity finding is none.
Assumptions & free parameters
free parameters (3)
- Digital Echo hyperparameters =
unspecified
- Downsampled training set size =
424,000
- Mach-number boundary cut =
20% (Mach 0.5 to 0.4)
assumptions (5)
- standard math Quasi-Monte Carlo sampling provides low-discrepancy, representative coverage of the input and output spaces.
- standard math Manhattan distance is an appropriate metric for matching output values in nearest-neighbor selection.
- domain assumption The NPSS turbofan model is a valid ground truth for the surrogate's target outputs.
- domain assumption A more uniform output distribution improves neural-network surrogate training.
- ad hoc to paper The test set is representative of the flight envelope and independent of the selection procedure.
Cite this review
Pith. "Pith review of Active Learning Enhanced Surrogate Modeling of Jet Engines in JuliaSim." pith.science (2026). https://pith.science/paper/O5X2E4OB
@misc{pith2026250107701,
author = {Pith},
title = {Pith review of: Active Learning Enhanced Surrogate Modeling of Jet Engines in JuliaSim},
year = {2026},
howpublished = {\url{https://pith.science/paper/O5X2E4OB}},
note = {Machine review of arXiv:2501.07701}
}
read the original abstract
Surrogate models are effective tools for accelerated design of complex systems. The result of a design optimization procedure using surrogate models can be used to initialize an optimization routine using the full order system. High accuracy of the surrogate model can be advantageous for fast convergence. In this work, we present an active learning approach to produce a very high accuracy surrogate model of a turbofan jet engine, that demonstrates 0.1\% relative error for all quantities of interest. We contrast this with a surrogate model produced using a more traditional brute-force data generation approach.
Forward citations
Cited by 1 Pith paper
-
Learning Gradient Flow: Using Equation Discovery to Accelerate Engineering Optimization
An optimizer that fits a SINDy polynomial model to recent optimization-variable trajectories and then integrates that surrogate flow instead of evaluating the true objective/gradient can cut gradient-evaluation counts...
Reference graph
Works this paper leans on
-
[17]
Scientific Machine Learning (SciML) Surrogates for Industry, Part 1: The Guiding Questions,
Rackauckas, C. V., and Abdelrehim, A., “Scientific Machine Learning (SciML) Surrogates for Industry, Part 1: The Guiding Questions,” ????
-
[1]
Yondo, R., Andrés, E., and Valero, E., “A review on design of experiments and surrogate models in aircraft real-time and many-query aerodynamic analyses,”Progress in aerospace sciences, Vol. 96, 2018, pp. 23–61
work page 2018
-
[2]
Tao, J., and Sun, G., “Application of deep learning based multi-fidelity surrogate model to robust aerodynamic design optimization,”Aerospace Science and Technology, Vol. 92, 2019, pp. 722–737
work page 2019
-
[3]
Approaches for quantifying uncertainties in computational modeling for aerospace applications,
Schaefer, J. A., Romero, V. J., Schafer, S. R., Leyde, B., and Denham, C. L., “Approaches for quantifying uncertainties in computational modeling for aerospace applications,”AIAA Scitech 2020 Forum, 2020, p. 1520
work page 2020
-
[4]
Uncertaintyquantificationinaeroelasticity,
Beran,P.,Stanford,B.,andSchrock,C.,“Uncertaintyquantificationinaeroelasticity,” Annualreviewoffluidmechanics ,Vol.49, No. 1, 2017, pp. 361–386
work page 2017
-
[5]
Jansson, T., Nilsson, L., and Redhe, M., “Using surrogate models and response surfaces in structural optimization–with application to crashworthiness design and sheet metal forming,”Structural and Multidisciplinary Optimization, Vol. 25, 2003, pp. 129–140
work page 2003
-
[6]
Building efficient response surfaces of aerodynamic functions with kriging and cokriging,
Laurenceau, J., and Sagaut, P., “Building efficient response surfaces of aerodynamic functions with kriging and cokriging,” AIAA journal, Vol. 46, No. 2, 2008, pp. 498–507. 6
work page 2008
-
[7]
Mesh deformation using radial basis functions for gradient-based aerodynamic shape optimization,
Jakobsson, S., and Amoignon, O., “Mesh deformation using radial basis functions for gradient-based aerodynamic shape optimization,”Computers & Fluids, Vol. 36, No. 6, 2007, pp. 1119–1136
work page 2007
Show all 32 references
-
[8]
Deep neural network for unsteady aerodynamic and aeroelastic modeling across multiple Mach numbers,
Li, K., Kou, J., and Zhang, W., “Deep neural network for unsteady aerodynamic and aeroelastic modeling across multiple Mach numbers,”Nonlinear Dynamics, Vol. 96, 2019, pp. 2157–2177
2019
-
[9]
Robust design optimization using surrogate models,
Keane, A. J., and Voutchkov, I. I., “Robust design optimization using surrogate models,”Journal of Computational Design and Engineering, Vol. 7, No. 1, 2020, pp. 44–55
2020
-
[10]
Surrogate model-based optimization framework: a case study in aerospace design,
Mack, Y., Goel, T., Shyy, W., and Haftka, R., “Surrogate model-based optimization framework: a case study in aerospace design,”Evolutionary computation in dynamic and uncertain environments, 2007, pp. 323–342
2007
-
[11]
Surrogate model selection for design space approximation and surrogatebased optimization,
Williams, B., and Cremaschi, S., “Surrogate model selection for design space approximation and surrogatebased optimization,” Computer aided chemical engineering, Vol. 47, Elsevier, 2019, pp. 353–358
2019
-
[12]
Advantages of surrogate models for architectural design optimization,
Wortmann, T., Costa, A., Nannicini, G., and Schroepfer, T., “Advantages of surrogate models for architectural design optimization,”AI EDAM, Vol. 29, No. 4, 2015, pp. 471–481
2015
-
[13]
Surrogate-based optimization,
Han, Z.-H., Zhang, K.-S., et al., “Surrogate-based optimization,”Real-world applications of genetic algorithms, Vol. 343, 2012, pp. 343–362
2012
-
[14]
Managing computational complexity using surrogate models: a critical review,
Alizadeh, R., Allen, J. K., and Mistree, F., “Managing computational complexity using surrogate models: a critical review,” Research in Engineering Design, Vol. 31, No. 3, 2020, pp. 275–298
2020
-
[15]
Surrogate model uncertainty quantification for reliability-based design optimization,
Li, M., and Wang, Z., “Surrogate model uncertainty quantification for reliability-based design optimization,”Reliability Engineering & System Safety, Vol. 192, 2019, p. 106432
2019
-
[16]
Numerical propulsion system simulation (NPSS) 1999 industry review,
Lytle, J., Follen, G., Naiman, C., and Evans, A., “Numerical propulsion system simulation (NPSS) 1999 industry review,” Tech. rep., 2000
1999
-
[18]
Composing modeling and simulation with machine learning in Julia,
Rackauckas, C., Gwozdz, M., Jain, A., Ma, Y., Martinuzzi, F., Rajput, U., Saba, E., Shah, V. B., Anantharaman, R., Edelman, A., et al., “Composing modeling and simulation with machine learning in Julia,”2022 Annual Modeling and Simulation Conference (ANNSIM), IEEE, 2022, pp. 1–17
2022
-
[19]
Julia: A fresh approach to numerical computing,
Bezanson, J., Edelman, A., Karpinski, S., and Shah, V. B., “Julia: A fresh approach to numerical computing,”SIAM review, Vol. 59, No. 1, 2017, pp. 65–98
2017
-
[20]
Accelerating simulation of stiff nonlinear systems using continuous-time echo state networks,
Anantharaman, R., Ma, Y., Gowda, S., Laughman, C., Shah, V., Edelman, A., and Rackauckas, C., “Accelerating simulation of stiff nonlinear systems using continuous-time echo state networks,”arXiv preprint arXiv:2010.04004, 2020
2010 arXiv
-
[21]
Stably accelerating stiff quantitative systems pharmacology models: Continuous-time echo state networks as implicit machine learning,
Anantharaman, R., Abdelrehim, A., Jain, A., Pal, A., Sharp, D., Edelman, A., Rackauckas, C., et al., “Stably accelerating stiff quantitative systems pharmacology models: Continuous-time echo state networks as implicit machine learning,”IFAC- PapersOnLine, Vol. 55, No. 23, 2022...
2022
-
[22]
Composable and reusable neural surrogates to predict system response of causal model components,
Anantharaman, R., Abdelrehim, A., Martinuzzi, F., Yalburgi, S., Saba, E., Fischer, K., Hertz, G., de Vos, P., Laughman, C., Ma, Y., et al., “Composable and reusable neural surrogates to predict system response of causal model components,”AAAI 2022 Workshop on AI for Design and...
2022
-
[23]
Continuous-time echo state networks for predicting power system dynamics,
Roberts, C., Lara, J. D., Henriquez-Auba, R., Bossart, M., Anantharaman, R., Rackauckas, C., Hodge, B.-M., and Callaway, D. S., “Continuous-time echo state networks for predicting power system dynamics,”Electric Power Systems Research, Vol. 212, 2022, p. 108562
2022
-
[24]
Anantharaman, R.,Approximation of Large Stiff Acausal Models, Massachusetts Institute of Technology, 2023
2023
-
[25]
Active Learning-CFD Integrated Surrogate-Based Framework for Shape Optimization of LTA Systems,
Tripathi, M., Kumar, S., Desale, Y. B., and Pant, R. S., “Active Learning-CFD Integrated Surrogate-Based Framework for Shape Optimization of LTA Systems,”AIAA AVIATION FORUM AND ASCEND 2024, 2024, p. 4120
2024
-
[26]
Active learning for efficient data-driven aerodynamic modeling in spaceplane design,
Zhang, H., Huang, W., Shen, Y., Xu, D.-y., and Niu, Y.-b., “Active learning for efficient data-driven aerodynamic modeling in spaceplane design,”Physics of Fluids, Vol. 36, No. 6, 2024
2024
-
[27]
UnbiasedNets: a dataset diversification framework for robustness bias alleviation in neural networks,
Naseer, M., Prabakaran, B. S., Hasan, O., and Shafique, M., “UnbiasedNets: a dataset diversification framework for robustness bias alleviation in neural networks,”Machine Learning, Vol. 113, No. 5, 2024, pp. 2499–2526. 7
2024
-
[28]
MetaBalance: high-performance neural networks for class-imbalanced data,
Bansal,A.,Goldblum,M.,Cherepanova,V.,Schwarzschild,A.,Bruss,C.B.,andGoldstein,T.,“MetaBalance: high-performance neural networks for class-imbalanced data,”arXiv preprint arXiv:2106.09643, 2021
2021 arXiv
-
[29]
Analysis of euclidean distance and manhattan distance measure in face recognition,
Malkauthekar, M., “Analysis of euclidean distance and manhattan distance measure in face recognition,”Third International Conference on Computational Intelligence and Information Technology (CIIT 2013), IET, 2013, pp. 503–507
2013
-
[30]
On Latin hypercube sampling,
Loh, W.-L., “On Latin hypercube sampling,”The annals of statistics, Vol. 24, No. 5, 1996, pp. 2058–2080
1996
-
[31]
Ensemble of surrogates,
Goel, T., Haftka, R. T., Shyy, W., and Queipo, N. V., “Ensemble of surrogates,”Structural and Multidisciplinary Optimization, Vol. 33, 2007, pp. 199–216
2007
-
[32]
Mixture of experts: a literature survey,
Masoudnia, S., and Ebrahimpour, R., “Mixture of experts: a literature survey,”Artificial Intelligence Review, Vol. 42, 2014, pp. 275–293. 8
2014
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.