Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Active Learning Enhanced Surrogate Modeling of Jet Engines in JuliaSim

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read An active-learning scheme brings a jet engine surrogate to 0.1 percent relative error.

desk verdict Headline claim contradicts own Table 4; the active learning trick is sensible and the error analysis is honest, but the paper needs a corrected claim and a matched comparison. read the letter →

arxiv 2501.07701 v1 pith:O5X2E4OB submitted 2025-01-13 cs.CE

classification cs.CE
keywords activelearningsurrogatemodelingjetenginedigitalechoquasi-MonteCarlosamplingnearest-neighbordownsamplingturbofanflightenvelope
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a jet-engine surrogate model can be trained to predict shaft speed, turbine temperature, net thrust, and fuel flow with 0.1 percent relative error across nearly the entire flight envelope. It argues that this accuracy comes from an active-learning step that rebalances the training data by output value, rather than simply densely sampling the input space. The authors contrast this with a brute-force approach using one million quasi-Monte Carlo input samples, which yields noticeably worse accuracy on the same test set. A high-accuracy surrogate matters because it can initialize design optimization on the full physics-based model, potentially making convergence faster and cheaper. The paper also shows that the remaining errors concentrate at the boundary of the Mach-number range and that trimming this edge by 20 percent reduces high-error points by an order of magnitude.

What carries the argument

The key machinery is the adaptive downsampling routine described in Algorithms 1 and 2. It densely samples the input flight envelope with quasi-Monte Carlo or Latin hypercube sampling, simulates about one million points with the physical engine model, computes the output extrema, then quasi-Monte Carlo samples the output space and uses a Manhattan-distance nearest-neighbor search to select a training subset of about 424,000 points whose output distribution is nearly uniform. This rebalanced set is then used to train the Digital Echo surrogate, a data-driven steady-state system that maps parameters to outputs through the equation f(p) = x. The active-learning step is what distinguishes this method from the brute-force baseline, and it is the mechanism the paper credits for the accuracy improvement.

What would settle it

Run the same active-learning surrogate on a dedicated test set of points lying on the Mach-number boundary listed in Table 2 (for example Mach 0.5 at -2000 feet, 0.58 at 10,000 feet, 0.68 at 20,000 feet, and 0.9 at 30,000 to 40,000 feet) and count the fraction with relative error above 0.1 percent; if that fraction is far higher than on the interior points, the claim of uniform flight-envelope accuracy fails.

Watch

Extended reading notes

Core claim

The central discovery is that an active-learning pipeline built from output-space quasi-Monte Carlo sampling and nearest-neighbor downsampling produces a Digital Echo surrogate of a turbofan engine with unusually high accuracy. On a test set of 100,000 points, the downsampled surrogate (trained on 424,000 points) places more than 98.4 percent of predictions for all output quantities within 0.1 percent relative error, and for turbine temperature and fuel flow the share reaches 99.98 percent or 100 percent. This improves on the same surrogate trained on a denser one-million-point set without output balancing, which leaves as much as 17.4 percent of net thrust predictions outside the 0.1 percent error band. Specific fuel consumption is not trained directly; it is computed as the ratio of fuel flow to net thrust, so its accuracy follows from the two trained outputs. The paper concludes that most of the remaining high error is confined to the Mach-number boundary of the flight envelope, and it demonstrates that cutting away 20 percent of the Mach range eliminates the majority of those outliers.

Load-bearing premise

The method relies on the premise that making the output distribution of the training set uniform is enough to keep relative error below 0.1 percent across the whole flight envelope, including the Mach-number edges where the paper's own data show errors concentrating and where a 20 percent trim of the input range is then applied.

Editorial extensions

If this is right

  • If the surrogate really holds 0.1 percent relative error across the flight envelope, it can serve as a reliable low-cost evaluator for design optimization, uncertainty quantification, or many-query trade studies where the full NPSS engine model would be prohibitively expensive.
  • Because specific fuel consumption is computed as the ratio of fuel flow to thrust, the claimed accuracy implies SFC predictions also fall within a similar error band, making the surrogate useful for cycle and mission analyses.
  • The 20 percent Mach-number trim, which reduces high-error points by an order of magnitude, suggests a practical rule: when accuracy near the envelope edge is critical, either extend the sampling region or add a dedicated boundary-sampling pass.
  • The success of output-space balancing over pure input-space density indicates that for data-hungry neural surrogate models, the distribution of training labels can matter more than the raw number of samples.
  • A second adaptive downsampling pass, as the paper itself suggests, could push the already small boundary-error population even lower without changing the core method.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The procedure is essentially a data-pruning strategy that treats the output distribution as the main source of bias; the same principle could transfer to other physics-based surrogate problems where outputs are highly skewed, such as power-system contingencies or chemical-process simulations.
  • Editorial inference: The paper tests the surrogate on 100,000 points after active learning but the pre-active-learning model was tested on one million points, so part of the accuracy gap may come from the different test-set sizes; a like-for-like comparison on the same test set would isolate the effect of rebalancing.
  • Editorial inference: Since the boundary errors are tied to the Mach-altitude shape of the flight envelope, a testable extension is to replace the fixed 20 percent trim with adaptive boundary sampling near the Table 2 limits, which could recover the full envelope at equal accuracy.
  • Editorial inference: A falsifiable prediction of the paper's logic is that any physics model whose outputs are highly skewed should see a similar accuracy jump from this downsampling scheme; checking that against a different full-order model would show whether the improvement is specific to this engine or generic.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript describes an active-learning workflow for building a surrogate model of a turbofan jet engine using JuliaSim's proprietary Digital Echo method. The workflow densely samples the flight envelope, simulates the full-order NPSS model, then downsamples the training set by quasi-Monte Carlo sampling of the output space and nearest-neighbor matching in the output space, with the goal of balancing the output distribution. The authors report that the resulting surrogate achieves 0.1% relative error for all quantities of interest, and compare this with a baseline surrogate trained on the full 1M-point quasi-Monte Carlo sample. Error analysis shows that the largest errors concentrate near the Mach-number boundary of the flight envelope, and the authors propose cutting 20% of the Mach range to reduce these errors.

Significance. If the headline claim were true, the paper would demonstrate a practically useful surrogate for jet-engine design optimization, with sub-0.1% relative error across the operational envelope. The paper's strengths are that it gives a concrete algorithmic description of the adaptive downsampling routine, reports quantitative accuracy numbers in tabular form, and localizes the residual errors in the input space. However, the central claim as stated in the abstract and introduction is not supported by the paper's own reported numbers, and the comparison against the brute-force baseline is confounded by different test-set sizes and a post hoc domain trim. The contribution is still potentially valuable as an empirical demonstration of output-space downsampling, but the claims need substantial revision and additional boundary-focused evaluation before the paper can be accepted.

major comments (4)
  1. [Abstract and Section IV, Table 4] The abstract and Section I claim that the surrogate 'demonstrates 0.1% relative error for all quantities of interest,' but Table 4 shows that only 98.44% of PerfInst_Fn and 99.49% of ShH_N test predictions fall within 0.1% relative error. Thus, for two of the four reported quantities, a nontrivial fraction of the test set exceeds the stated tolerance, contradicting the universal claim. The conclusion in Section V correctly softens this to 'greater than 98.4%,' but the abstract and introduction should be revised to match the data.
  2. [Section IV, Tables 3 and 4, and Figure 4] The pre/post active-learning comparison is confounded: the pre-AL surrogate is tested on 1M points while the post-AL surrogate is tested on 100k points, and the training set sizes differ (1M vs 424k). More importantly, the reported improvement is achieved only after the paper proposes cutting 20% off the Mach-number range (Section IV, Figure 4b). Since the original flight envelope in Table 2 includes the boundary regions, the claimed accuracy across the stated envelope is not demonstrated. The authors should report error statistics on the full envelope, and if the boundary trim is retained, should clearly present it as a restricted-domain result rather than as a success of the active-learning scheme alone.
  3. [Section III.B, Algorithms 1 and 2] The active-learning method balances the output distribution by nearest-neighbor selection in output space, but this does not address input-space coverage, particularly at the Mach-number boundaries where Figure 4 shows the largest errors. The post hoc 20% Mach trim is effectively an admission that the balancing scheme leaves boundary accuracy unresolved. The paper should either extend the method to explicitly handle boundary coverage or clearly scope the claim to the trimmed domain; as written, the method as described in Algorithms 1 and 2 does not by itself deliver the stated 0.1% accuracy across the full flight envelope.
  4. [Section III.A, Digital Echo] The core surrogate method, Digital Echo, is referenced to a non-public citation ([17] with no year or venue) and is not described beyond the equations x' = 0 and f(p) = x. Because the entire accuracy result depends on this proprietary method, the paper does not provide enough detail for a reader to reproduce the surrogate or to assess whether the observed errors are intrinsic to Digital Echo or to the active-learning sampling. At minimum, the authors should provide a public technical description or release the surrogate training code, or clearly state the algorithm's hyperparameters and architecture.
minor comments (5)
  1. [Section IV, Figure 4c] The caption states 'of the 300 or so points with relative error > 0.1%,' but Table 4 reports 1.56% of 100k test points for PerfInst_Fn, which is about 1560 points. This numerical inconsistency should be resolved, or the test-set size and counting procedure clarified.
  2. [Section V] There is a typo in 'jet enginees' which should be 'jet engines.'
  3. [Section IV] The word 'analyses' in 'an analyses of the remaining points' should be 'an analysis.'
  4. [References] Reference [17] is incomplete; it lacks a year, venue, and publication status, which makes it difficult for readers to access the Digital Echo method.
  5. [Section III.A, step 7] The pipeline description mentions evaluating models on a 'test set' after splitting into training and validation sets in step 4, but does not explain how the test set is generated or whether it is independent of the active-learning selection procedure. Clarifying this would strengthen the validity of the reported error statistics.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the accuracy claim is an empirical held-out evaluation rather than a derivation from fitted inputs.

full rationale

The paper's central claim is an empirical evaluation: a surrogate model trained on NPSS simulation data is assessed on a separately generated held-out test set, with results reported in Tables 3 and 4. The active learning procedure (Algorithms 1 and 2) selects training points by output-space quasi-Monte Carlo sampling and nearest-neighbor matching; this is a data-selection scheme, not a fitted parameter that is later re-labeled as a prediction. The relative-error statistics are measured on test points outside the training set, so the claim does not reduce to the training data or to the downsampling criterion by construction. The Digital Echo surrogate is cited to a non-public author reference, but the paper's accuracy result is supported by its own experiments rather than by that citation; the self-citation names the method rather than serving as the evidence for the empirical claim. Concerns that the abstract's universal '0.1% relative error for all quantities' overstates the tabulated percentages, or that the test set may differ between pre- and post-active-learning comparisons, are correctness and reporting issues, not circularity. No load-bearing derivation step equates a prediction with an input or with a self-cited theorem, so the appropriate circularity finding is none.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the Digital Echo surrogate (taken from an unpublished citation), the output-balancing assumption, the NPSS ground truth, and several unstated choices about the test set and hyperparameters. No new physical entities are introduced.

free parameters (3)
  • Digital Echo hyperparameters = unspecified
    Section III.A step 5 defines a hyperparameter space H and selects the best performing h_i, but the actual hyperparameters, neural architecture, and training settings are not reported.
  • Downsampled training set size = 424,000
    The nearest-neighbor routine in Algorithm 2 outputs a set whose size is determined by the number of unique matches; the target size 424k appears as a result, but no criterion for choosing it is given.
  • Mach-number boundary cut = 20% (Mach 0.5 to 0.4)
    In Section IV, the flight envelope is trimmed post hoc to reduce high-error points; this is a hand-chosen adjustment based on the observed error distribution.
assumptions (5)
  • standard math Quasi-Monte Carlo sampling provides low-discrepancy, representative coverage of the input and output spaces.
    Invoked in Section III.A and III.B to justify the dense 1M-sample sets.
  • standard math Manhattan distance is an appropriate metric for matching output values in nearest-neighbor selection.
    Used in Algorithm 2; no justification is given for choosing it over other metrics.
  • domain assumption The NPSS turbofan model is a valid ground truth for the surrogate's target outputs.
    The entire evaluation treats NPSS outputs as exact; no experimental validation is presented.
  • domain assumption A more uniform output distribution improves neural-network surrogate training.
    Stated in Section III.B with citations [27,28], but not tested independently within this paper.
  • ad hoc to paper The test set is representative of the flight envelope and independent of the selection procedure.
    The paper does not describe how the 100k test points were sampled or whether they overlap with the points used in the downsampling selection.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Active Learning Enhanced Surrogate Modeling of Jet Engines in JuliaSim." pith.science (2026). https://pith.science/paper/O5X2E4OB

@misc{pith2026250107701,
  author       = {Pith},
  title        = {Pith review of: Active Learning Enhanced Surrogate Modeling of Jet Engines in JuliaSim},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O5X2E4OB}},
  note         = {Machine review of arXiv:2501.07701}
}
read the original abstract

Surrogate models are effective tools for accelerated design of complex systems. The result of a design optimization procedure using surrogate models can be used to initialize an optimization routine using the full order system. High accuracy of the surrogate model can be advantageous for fast convergence. In this work, we present an active learning approach to produce a very high accuracy surrogate model of a turbofan jet engine, that demonstrates 0.1\% relative error for all quantities of interest. We contrast this with a surrogate model produced using a more traditional brute-force data generation approach.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Learning Gradient Flow: Using Equation Discovery to Accelerate Engineering Optimization

    math.OC 2026-02 conditional novelty 6.0 of 10

    An optimizer that fits a SINDy polynomial model to recent optimization-variable trajectories and then integrates that surrogate flow instead of evaluating the true objective/gradient can cut gradient-evaluation counts...

Reference graph

Works this paper leans on

32 extracted references · 31 canonical work pages · cited by 1 Pith paper

  1. [17]

    Scientific Machine Learning (SciML) Surrogates for Industry, Part 1: The Guiding Questions,

    Rackauckas, C. V., and Abdelrehim, A., “Scientific Machine Learning (SciML) Surrogates for Industry, Part 1: The Guiding Questions,” ????

  2. [1]

    A review on design of experiments and surrogate models in aircraft real-time and many-query aerodynamic analyses,

    Yondo, R., Andrés, E., and Valero, E., “A review on design of experiments and surrogate models in aircraft real-time and many-query aerodynamic analyses,”Progress in aerospace sciences, Vol. 96, 2018, pp. 23–61

  3. [2]

    Application of deep learning based multi-fidelity surrogate model to robust aerodynamic design optimization,

    Tao, J., and Sun, G., “Application of deep learning based multi-fidelity surrogate model to robust aerodynamic design optimization,”Aerospace Science and Technology, Vol. 92, 2019, pp. 722–737

  4. [3]

    Approaches for quantifying uncertainties in computational modeling for aerospace applications,

    Schaefer, J. A., Romero, V. J., Schafer, S. R., Leyde, B., and Denham, C. L., “Approaches for quantifying uncertainties in computational modeling for aerospace applications,”AIAA Scitech 2020 Forum, 2020, p. 1520

  5. [4]

    Uncertaintyquantificationinaeroelasticity,

    Beran,P.,Stanford,B.,andSchrock,C.,“Uncertaintyquantificationinaeroelasticity,” Annualreviewoffluidmechanics ,Vol.49, No. 1, 2017, pp. 361–386

  6. [5]

    Using surrogate models and response surfaces in structural optimization–with application to crashworthiness design and sheet metal forming,

    Jansson, T., Nilsson, L., and Redhe, M., “Using surrogate models and response surfaces in structural optimization–with application to crashworthiness design and sheet metal forming,”Structural and Multidisciplinary Optimization, Vol. 25, 2003, pp. 129–140

  7. [6]

    Building efficient response surfaces of aerodynamic functions with kriging and cokriging,

    Laurenceau, J., and Sagaut, P., “Building efficient response surfaces of aerodynamic functions with kriging and cokriging,” AIAA journal, Vol. 46, No. 2, 2008, pp. 498–507. 6

  8. [7]

    Mesh deformation using radial basis functions for gradient-based aerodynamic shape optimization,

    Jakobsson, S., and Amoignon, O., “Mesh deformation using radial basis functions for gradient-based aerodynamic shape optimization,”Computers & Fluids, Vol. 36, No. 6, 2007, pp. 1119–1136

Show all 32 references
  1. [8]

    Deep neural network for unsteady aerodynamic and aeroelastic modeling across multiple Mach numbers,

    Li, K., Kou, J., and Zhang, W., “Deep neural network for unsteady aerodynamic and aeroelastic modeling across multiple Mach numbers,”Nonlinear Dynamics, Vol. 96, 2019, pp. 2157–2177

  2. [9]

    Robust design optimization using surrogate models,

    Keane, A. J., and Voutchkov, I. I., “Robust design optimization using surrogate models,”Journal of Computational Design and Engineering, Vol. 7, No. 1, 2020, pp. 44–55

  3. [10]

    Surrogate model-based optimization framework: a case study in aerospace design,

    Mack, Y., Goel, T., Shyy, W., and Haftka, R., “Surrogate model-based optimization framework: a case study in aerospace design,”Evolutionary computation in dynamic and uncertain environments, 2007, pp. 323–342

  4. [11]

    Surrogate model selection for design space approximation and surrogatebased optimization,

    Williams, B., and Cremaschi, S., “Surrogate model selection for design space approximation and surrogatebased optimization,” Computer aided chemical engineering, Vol. 47, Elsevier, 2019, pp. 353–358

  5. [12]

    Advantages of surrogate models for architectural design optimization,

    Wortmann, T., Costa, A., Nannicini, G., and Schroepfer, T., “Advantages of surrogate models for architectural design optimization,”AI EDAM, Vol. 29, No. 4, 2015, pp. 471–481

  6. [13]

    Surrogate-based optimization,

    Han, Z.-H., Zhang, K.-S., et al., “Surrogate-based optimization,”Real-world applications of genetic algorithms, Vol. 343, 2012, pp. 343–362

  7. [14]

    Managing computational complexity using surrogate models: a critical review,

    Alizadeh, R., Allen, J. K., and Mistree, F., “Managing computational complexity using surrogate models: a critical review,” Research in Engineering Design, Vol. 31, No. 3, 2020, pp. 275–298

  8. [15]

    Surrogate model uncertainty quantification for reliability-based design optimization,

    Li, M., and Wang, Z., “Surrogate model uncertainty quantification for reliability-based design optimization,”Reliability Engineering & System Safety, Vol. 192, 2019, p. 106432

  9. [16]

    Numerical propulsion system simulation (NPSS) 1999 industry review,

    Lytle, J., Follen, G., Naiman, C., and Evans, A., “Numerical propulsion system simulation (NPSS) 1999 industry review,” Tech. rep., 2000

  10. [18]

    Composing modeling and simulation with machine learning in Julia,

    Rackauckas, C., Gwozdz, M., Jain, A., Ma, Y., Martinuzzi, F., Rajput, U., Saba, E., Shah, V. B., Anantharaman, R., Edelman, A., et al., “Composing modeling and simulation with machine learning in Julia,”2022 Annual Modeling and Simulation Conference (ANNSIM), IEEE, 2022, pp. 1–17

  11. [19]

    Julia: A fresh approach to numerical computing,

    Bezanson, J., Edelman, A., Karpinski, S., and Shah, V. B., “Julia: A fresh approach to numerical computing,”SIAM review, Vol. 59, No. 1, 2017, pp. 65–98

  12. [20]

    Accelerating simulation of stiff nonlinear systems using continuous-time echo state networks,

    Anantharaman, R., Ma, Y., Gowda, S., Laughman, C., Shah, V., Edelman, A., and Rackauckas, C., “Accelerating simulation of stiff nonlinear systems using continuous-time echo state networks,”arXiv preprint arXiv:2010.04004, 2020

  13. [21]

    Stably accelerating stiff quantitative systems pharmacology models: Continuous-time echo state networks as implicit machine learning,

    Anantharaman, R., Abdelrehim, A., Jain, A., Pal, A., Sharp, D., Edelman, A., Rackauckas, C., et al., “Stably accelerating stiff quantitative systems pharmacology models: Continuous-time echo state networks as implicit machine learning,”IFAC- PapersOnLine, Vol. 55, No. 23, 2022...

  14. [22]

    Composable and reusable neural surrogates to predict system response of causal model components,

    Anantharaman, R., Abdelrehim, A., Martinuzzi, F., Yalburgi, S., Saba, E., Fischer, K., Hertz, G., de Vos, P., Laughman, C., Ma, Y., et al., “Composable and reusable neural surrogates to predict system response of causal model components,”AAAI 2022 Workshop on AI for Design and...

  15. [23]

    Continuous-time echo state networks for predicting power system dynamics,

    Roberts, C., Lara, J. D., Henriquez-Auba, R., Bossart, M., Anantharaman, R., Rackauckas, C., Hodge, B.-M., and Callaway, D. S., “Continuous-time echo state networks for predicting power system dynamics,”Electric Power Systems Research, Vol. 212, 2022, p. 108562

  16. [24]

    Anantharaman, R.,Approximation of Large Stiff Acausal Models, Massachusetts Institute of Technology, 2023

  17. [25]

    Active Learning-CFD Integrated Surrogate-Based Framework for Shape Optimization of LTA Systems,

    Tripathi, M., Kumar, S., Desale, Y. B., and Pant, R. S., “Active Learning-CFD Integrated Surrogate-Based Framework for Shape Optimization of LTA Systems,”AIAA AVIATION FORUM AND ASCEND 2024, 2024, p. 4120

  18. [26]

    Active learning for efficient data-driven aerodynamic modeling in spaceplane design,

    Zhang, H., Huang, W., Shen, Y., Xu, D.-y., and Niu, Y.-b., “Active learning for efficient data-driven aerodynamic modeling in spaceplane design,”Physics of Fluids, Vol. 36, No. 6, 2024

  19. [27]

    UnbiasedNets: a dataset diversification framework for robustness bias alleviation in neural networks,

    Naseer, M., Prabakaran, B. S., Hasan, O., and Shafique, M., “UnbiasedNets: a dataset diversification framework for robustness bias alleviation in neural networks,”Machine Learning, Vol. 113, No. 5, 2024, pp. 2499–2526. 7

  20. [28]

    MetaBalance: high-performance neural networks for class-imbalanced data,

    Bansal,A.,Goldblum,M.,Cherepanova,V.,Schwarzschild,A.,Bruss,C.B.,andGoldstein,T.,“MetaBalance: high-performance neural networks for class-imbalanced data,”arXiv preprint arXiv:2106.09643, 2021

  21. [29]

    Analysis of euclidean distance and manhattan distance measure in face recognition,

    Malkauthekar, M., “Analysis of euclidean distance and manhattan distance measure in face recognition,”Third International Conference on Computational Intelligence and Information Technology (CIIT 2013), IET, 2013, pp. 503–507

  22. [30]

    On Latin hypercube sampling,

    Loh, W.-L., “On Latin hypercube sampling,”The annals of statistics, Vol. 24, No. 5, 1996, pp. 2058–2080

  23. [31]

    Ensemble of surrogates,

    Goel, T., Haftka, R. T., Shyy, W., and Queipo, N. V., “Ensemble of surrogates,”Structural and Multidisciplinary Optimization, Vol. 33, 2007, pp. 199–216

  24. [32]

    Mixture of experts: a literature survey,

    Masoudnia, S., and Ebrahimpour, R., “Mixture of experts: a literature survey,”Artificial Intelligence Review, Vol. 42, 2014, pp. 275–293. 8

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.