{"id":"f90fba61-2e6f-4b1d-b4a2-19b2b7e858a2","arxiv_id":"2501.07701","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Adaptive output-space downsampling improves a jet-engine surrogate model's accuracy, but the claimed 0.1% error for all quantities is not met by the paper's own data.","lead":"Engineers at JuliaHub trained a neural-network surrogate for a turbofan jet engine, using a data-selection trick that balances the training outputs, and report that most test points fall within 0.1% relative error. The paper claims this beats dense sampling, but the headline accuracy is overstated and the comparison is not apples-to-apples.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract claims 0.1% RE for all quantities, but Table 4 shows >1.5% of thrust points exceed 0.1%; boundary-stratified test needed.","rationale":"The paper's contribution is an engineering application of active learning to balance output distributions. The core method may improve average accuracy, but the advertised universal 0.1% accuracy is a quantitative claim that can be checked directly against Tables 3 and 4. Table 4 already shows it is false at the strict reading: 1.56% of thrust predictions and 0.51% of shaft-speed predictions exceed 0.1% relative error. The authors themselves soften the claim in the Conclusion ('near 0.1%', 'greater than 98.4%'), so the abstract is an overstatement. A boundary-stratified test would determine whether the residual error is a systematic boundary effect or scattered noise; either way, the central claim needs revision. The reader's conditional verdict is appropriate: the method may be useful, but the paper must either correct the accuracy claim or explicitly restrict the claimed envelope. The post-hoc 20% Mach trim in Section IV is not a validation of the method; it is a domain reduction that should be disclosed in the abstract and conclusion. Thus no change to the overall verdict is needed, but the conditions should include a corrected accuracy statement and a controlled comparison on identical test sets.","tokens_in":6145,"tokens_out":3486,"duration_ms":32502,"concrete_test":"Evaluate the post-active-learning surrogate on a fixed test set of 1M QMC points identical to the pre-AL test set, with additional stratified oversampling of the Mach-altitude boundary cells in Table 2 (e.g., at Mach upper bounds for each altitude and at altitude extremes). Compute per-QoI maximum RE and the fraction of boundary points with RE>0.1%. If any QoI has more than, say, 0.5% of boundary points above 0.1% RE, the '0.1% for all quantities' claim must be replaced by a quantile statement or explicitly restricted to the trimmed envelope.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim as stated in the abstract and Section I — 'demonstrates 0.1% relative error for all quantities of interest' — is not supported by the paper's own results. Table 4 reports the post-active-learning surrogate with only 98.44% of PerfInst_Fn test predictions within 0.1% RE (and 99.49% for ShH_N), meaning 1.56% and 0.51% of test points exceed the stated tolerance. The paper then shows in Figure 4 that the points exceeding 0.1% concentrate on the Mach-number boundary of the flight envelope, and proposes cutting 20% off the Mach range to reduce them. Because the flight envelope in Table 2 includes these boundary regions, the headline claim holds only after a post-hoc domain restriction. The output-space balancing in Algorithm 1 does not address input-space boundary coverage, so the proposed active learning scheme does not by itself deliver '0.1% for all QoIs' across the stated envelope. The confounded comparison (1M test points pre-AL vs 100k post-AL; training sizes 1M vs 424k) further weakens the quantitative contrast, but the decisive issue is that the reported data contradict the abstract's universal accuracy claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript describes an active-learning workflow for building a surrogate model of a turbofan jet engine using JuliaSim's proprietary Digital Echo method. The workflow densely samples the flight envelope, simulates the full-order NPSS model, then downsamples the training set by quasi-Monte Carlo sampling of the output space and nearest-neighbor matching in the output space, with the goal of balancing the output distribution. The authors report that the resulting surrogate achieves 0.1% relative error for all quantities of interest, and compare this with a baseline surrogate trained on the full 1M-point quasi-Monte Carlo sample. Error analysis shows that the largest errors concentrate near the Mach-number boundary of the flight envelope, and the authors propose cutting 20% of the Mach range to reduce these errors.","tokens_in":6437,"tokens_out":2260,"duration_ms":23589,"significance":"If the headline claim were true, the paper would demonstrate a practically useful surrogate for jet-engine design optimization, with sub-0.1% relative error across the operational envelope. The paper's strengths are that it gives a concrete algorithmic description of the adaptive downsampling routine, reports quantitative accuracy numbers in tabular form, and localizes the residual errors in the input space. However, the central claim as stated in the abstract and introduction is not supported by the paper's own reported numbers, and the comparison against the brute-force baseline is confounded by different test-set sizes and a post hoc domain trim. The contribution is still potentially valuable as an empirical demonstration of output-space downsampling, but the claims need substantial revision and additional boundary-focused evaluation before the paper can be accepted.","major_comments":[{"comment":"The abstract and Section I claim that the surrogate 'demonstrates 0.1% relative error for all quantities of interest,' but Table 4 shows that only 98.44% of PerfInst_Fn and 99.49% of ShH_N test predictions fall within 0.1% relative error. Thus, for two of the four reported quantities, a nontrivial fraction of the test set exceeds the stated tolerance, contradicting the universal claim. The conclusion in Section V correctly softens this to 'greater than 98.4%,' but the abstract and introduction should be revised to match the data.","section":"Abstract and Section IV, Table 4"},{"comment":"The pre/post active-learning comparison is confounded: the pre-AL surrogate is tested on 1M points while the post-AL surrogate is tested on 100k points, and the training set sizes differ (1M vs 424k). More importantly, the reported improvement is achieved only after the paper proposes cutting 20% off the Mach-number range (Section IV, Figure 4b). Since the original flight envelope in Table 2 includes the boundary regions, the claimed accuracy across the stated envelope is not demonstrated. The authors should report error statistics on the full envelope, and if the boundary trim is retained, should clearly present it as a restricted-domain result rather than as a success of the active-learning scheme alone.","section":"Section IV, Tables 3 and 4, and Figure 4"},{"comment":"The active-learning method balances the output distribution by nearest-neighbor selection in output space, but this does not address input-space coverage, particularly at the Mach-number boundaries where Figure 4 shows the largest errors. The post hoc 20% Mach trim is effectively an admission that the balancing scheme leaves boundary accuracy unresolved. The paper should either extend the method to explicitly handle boundary coverage or clearly scope the claim to the trimmed domain; as written, the method as described in Algorithms 1 and 2 does not by itself deliver the stated 0.1% accuracy across the full flight envelope.","section":"Section III.B, Algorithms 1 and 2"},{"comment":"The core surrogate method, Digital Echo, is referenced to a non-public citation ([17] with no year or venue) and is not described beyond the equations x' = 0 and f(p) = x. Because the entire accuracy result depends on this proprietary method, the paper does not provide enough detail for a reader to reproduce the surrogate or to assess whether the observed errors are intrinsic to Digital Echo or to the active-learning sampling. At minimum, the authors should provide a public technical description or release the surrogate training code, or clearly state the algorithm's hyperparameters and architecture.","section":"Section III.A, Digital Echo"}],"minor_comments":[{"comment":"The caption states 'of the 300 or so points with relative error > 0.1%,' but Table 4 reports 1.56% of 100k test points for PerfInst_Fn, which is about 1560 points. This numerical inconsistency should be resolved, or the test-set size and counting procedure clarified.","section":"Section IV, Figure 4c"},{"comment":"There is a typo in 'jet enginees' which should be 'jet engines.'","section":"Section V"},{"comment":"The word 'analyses' in 'an analyses of the remaining points' should be 'an analysis.'","section":"Section IV"},{"comment":"Reference [17] is incomplete; it lacks a year, venue, and publication status, which makes it difficult for readers to access the Digital Echo method.","section":"References"},{"comment":"The pipeline description mentions evaluating models on a 'test set' after splitting into training and validation sets in step 4, but does not explain how the test set is generated or whether it is independent of the active-learning selection procedure. Clarifying this would strengthen the validity of the reported error statistics.","section":"Section III.A, step 7"}],"recommendation":"major_revision","confidential_remarks":"The paper is an application note for a proprietary JuliaSim tool. The central contribution is an empirical demonstration of output-space downsampling, but the headline accuracy claim is contradicted by the paper's own Table 4. The authors should also consider whether the journal's audience would benefit from more detail on the Digital Echo method, since the current citation is non-public and the method is not described sufficiently for independent evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper's headline overclaims. It says the surrogate \"demonstrates 0.1% relative error for all quantities of interest,\" but Table 4 shows only 98.44% of net thrust predictions and 99.49% of shaft speed predictions land inside 0.1%. The claim holds only after the authors cut 20% off the Mach envelope, and that cut is post hoc. So the central quantitative claim is not supported by the paper's own data.\n\nWhat is actually worth your time: the active learning trick is simple and sensible. They generate a dense QMC sample of the input space, simulate, then QMC-sample the output space and nearest-neighbor match back to the training set to balance the output distribution. That is a standard balancing idea, but it is cleanly described and the paper shows it shifts the error favorably: the 0.1% bucket improves from 82.6% to 98.44% for net thrust, and the worst-hit quantities improve. The error analysis in Figure 4 is also honest and useful—they show the residual >0.1% errors concentrate on the Mach boundary, and that trimming the boundary reduces them by an order of magnitude. That is reproducible evidence of where the surrogate struggles.\n\nThe soft spots are real but not fatal to the whole approach. The pre/post comparison uses 1M test points before active learning and 100k after, with training sizes 1M vs 424k, so part of the improvement may be test-set size or luck. The Digital Echo surrogate is a black box referenced to a non-public citation, which limits independent evaluation. And the post-hoc Mach trim is exactly the kind of move that makes a \"global\" claim look local: the flight envelope in Table 2 includes the boundary, so the stated accuracy applies to a trimmed envelope, not the one declared at the start.\n\nWho gets value: practitioners setting up surrogate model workflows, especially in aerospace, and people using JuliaSim. It is not a methodological breakthrough—the balancing idea is known—but the application to a real NPSS turbofan with large-scale data is a useful data point.\n\nRecommendation: send to peer review, but the authors should be asked to correct the abstract, rerun the comparison with matched test sets and no domain trim, and either release the data or the Digital Echo details. As it stands, I would not trust the \"0.1% for all\" claim, but I would trust the trend.","headline":"Headline claim contradicts own Table 4; the active learning trick is sensible and the error analysis is honest, but the paper needs a corrected claim and a matched comparison.","tokens_in":6949,"tokens_out":3063,"would_cite":false,"duration_ms":29874,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An active-learning scheme brings a jet engine surrogate to 0.1 percent relative error.","keywords":["active learning","surrogate modeling","jet engine","digital echo","quasi-Monte Carlo sampling","nearest-neighbor downsampling","turbofan","flight envelope"],"falsifier":"Run the same active-learning surrogate on a dedicated test set of points lying on the Mach-number boundary listed in Table 2 (for example Mach 0.5 at -2000 feet, 0.58 at 10,000 feet, 0.68 at 20,000 feet, and 0.9 at 30,000 to 40,000 feet) and count the fraction with relative error above 0.1 percent; if that fraction is far higher than on the interior points, the claim of uniform flight-envelope accuracy fails.","tokens_in":5977,"feed_emoji":"✈️","tokens_out":3767,"duration_ms":37471,"temperature":0.7,"pith_summary":"The paper claims that a jet-engine surrogate model can be trained to predict shaft speed, turbine temperature, net thrust, and fuel flow with 0.1 percent relative error across nearly the entire flight envelope. It argues that this accuracy comes from an active-learning step that rebalances the training data by output value, rather than simply densely sampling the input space. The authors contrast this with a brute-force approach using one million quasi-Monte Carlo input samples, which yields noticeably worse accuracy on the same test set. A high-accuracy surrogate matters because it can initialize design optimization on the full physics-based model, potentially making convergence faster and cheaper. The paper also shows that the remaining errors concentrate at the boundary of the Mach-number range and that trimming this edge by 20 percent reduces high-error points by an order of magnitude.","feed_headline":"Active learning hits 0.1% error on a jet-engine surrogate","feed_subtitle":"Rebalancing training outputs pushes shaft speed, thrust, and fuel-flow predictions to near-perfect accuracy.","key_machinery":"The key machinery is the adaptive downsampling routine described in Algorithms 1 and 2. It densely samples the input flight envelope with quasi-Monte Carlo or Latin hypercube sampling, simulates about one million points with the physical engine model, computes the output extrema, then quasi-Monte Carlo samples the output space and uses a Manhattan-distance nearest-neighbor search to select a training subset of about 424,000 points whose output distribution is nearly uniform. This rebalanced set is then used to train the Digital Echo surrogate, a data-driven steady-state system that maps parameters to outputs through the equation f(p) = x. The active-learning step is what distinguishes this method from the brute-force baseline, and it is the mechanism the paper credits for the accuracy improvement.","core_discovery":"The central discovery is that an active-learning pipeline built from output-space quasi-Monte Carlo sampling and nearest-neighbor downsampling produces a Digital Echo surrogate of a turbofan engine with unusually high accuracy. On a test set of 100,000 points, the downsampled surrogate (trained on 424,000 points) places more than 98.4 percent of predictions for all output quantities within 0.1 percent relative error, and for turbine temperature and fuel flow the share reaches 99.98 percent or 100 percent. This improves on the same surrogate trained on a denser one-million-point set without output balancing, which leaves as much as 17.4 percent of net thrust predictions outside the 0.1 percent error band. Specific fuel consumption is not trained directly; it is computed as the ratio of fuel flow to net thrust, so its accuracy follows from the two trained outputs. The paper concludes that most of the remaining high error is confined to the Mach-number boundary of the flight envelope, and it demonstrates that cutting away 20 percent of the Mach range eliminates the majority of those outliers.","pith_inferences":["Editorial inference: The procedure is essentially a data-pruning strategy that treats the output distribution as the main source of bias; the same principle could transfer to other physics-based surrogate problems where outputs are highly skewed, such as power-system contingencies or chemical-process simulations.","Editorial inference: The paper tests the surrogate on 100,000 points after active learning but the pre-active-learning model was tested on one million points, so part of the accuracy gap may come from the different test-set sizes; a like-for-like comparison on the same test set would isolate the effect of rebalancing.","Editorial inference: Since the boundary errors are tied to the Mach-altitude shape of the flight envelope, a testable extension is to replace the fixed 20 percent trim with adaptive boundary sampling near the Table 2 limits, which could recover the full envelope at equal accuracy.","Editorial inference: A falsifiable prediction of the paper's logic is that any physics model whose outputs are highly skewed should see a similar accuracy jump from this downsampling scheme; checking that against a different full-order model would show whether the improvement is specific to this engine or generic."],"forward_implications":["If the surrogate really holds 0.1 percent relative error across the flight envelope, it can serve as a reliable low-cost evaluator for design optimization, uncertainty quantification, or many-query trade studies where the full NPSS engine model would be prohibitively expensive.","Because specific fuel consumption is computed as the ratio of fuel flow to thrust, the claimed accuracy implies SFC predictions also fall within a similar error band, making the surrogate useful for cycle and mission analyses.","The 20 percent Mach-number trim, which reduces high-error points by an order of magnitude, suggests a practical rule: when accuracy near the envelope edge is critical, either extend the sampling region or add a dedicated boundary-sampling pass.","The success of output-space balancing over pure input-space density indicates that for data-hungry neural surrogate models, the distribution of training labels can matter more than the raw number of samples.","A second adaptive downsampling pass, as the paper itself suggests, could push the already small boundary-error population even lower without changing the core method."],"supporting_citations":[{"why":"Supplies the full-order physics-based turbofan engine model (NPSS) that generates the training and test data.","marker":"[16]"},{"why":"Defines the Digital Echo surrogate method that is trained and evaluated throughout the paper.","marker":"[17]"},{"why":"Latin hypercube sampling generates the dense input-space samples used for the brute-force baseline and the active-learning pipeline.","marker":"[30]"},{"why":"Manhattan distance is the metric used in the nearest-neighbor downsampling routine that rebalances the output distribution.","marker":"[29]"},{"why":"Establishes the prior use of active learning in aerospace surrogate modeling that this work extends with an output-space balancing scheme.","marker":"[24–26]"}],"fun_headline_variants":["Active learning cuts jet engine surrogate error to 0.1%","Jet engine surrogate: active learning beats dense sampling","0.1% error on jet engine surrogate via active learning","Active learning outdoes brute force for jet engine surrogates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method relies on the premise that making the output distribution of the training set uniform is enough to keep relative error below 0.1 percent across the whole flight envelope, including the Mach-number edges where the paper's own data show errors concentrating and where a 20 percent trim of the input range is then applied.","fun_headline_variants_meta":{"raw":{"variants":["Active learning cuts jet engine surrogate error to 0.1%","Jet engine surrogate: active learning beats dense sampling","0.1% error on jet engine surrogate via active learning","Active learning outdoes brute force for jet engine surrogates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000336,"raw_usage":{"total_tokens":1810,"prompt_tokens":847,"completion_tokens":963,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":894}},"tokens_in":463,"tokens_out":963,"duration_ms":8586,"temperature":1.0,"reasoning_tokens":894,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:36:17.337547+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same active-learning surrogate on a dedicated test set of points lying on the Mach-number boundary listed in Table 2 (for example Mach 0.5 at -2000 feet, 0.58 at 10,000 feet, 0.68 at 20,000 feet, and 0.9 at 30,000 to 40,000 feet) and count the fraction with relative error above 0.1 percent; if that fraction is far higher than on the interior points, the claim of uniform flight-envelope accuracy fails.","supporting_citations":[{"cited_title":"Numerical propulsion system simulation (NPSS) 1999 industry review,","cited_arxiv_id":null,"evidence_quote":"Supplies the full-order physics-based turbofan engine model (NPSS) that generates the training and test data."},{"cited_title":"Scientific Machine Learning (SciML) Surrogates for Industry, Part 1: The Guiding Questions,","cited_arxiv_id":null,"evidence_quote":"Defines the Digital Echo surrogate method that is trained and evaluated throughout the paper."},{"cited_title":"On Latin hypercube sampling,","cited_arxiv_id":null,"evidence_quote":"Latin hypercube sampling generates the dense input-space samples used for the brute-force baseline and the active-learning pipeline."},{"cited_title":"Analysis of euclidean distance and manhattan distance measure in face recognition,","cited_arxiv_id":null,"evidence_quote":"Manhattan distance is the metric used in the nearest-neighbor downsampling routine that rebalances the output distribution."}],"review_version":1}