Pith. sign in

REVIEW 4 major objections 5 minor 19 references

A Data-Driven Approach to Estimate LEO Orbit Capacity Models

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SINDy and LSTM can forecast LEO population trends from Monte Carlo averages with errors mostly below one percent.

desk verdict The surrogate application is new and honest, but the forecasting claim is unproven because the train/test split is never specified. read the letter →

arxiv 2507.19365 v1 pith:IBDIDNQY submitted 2025-07-25 cs.LG

Braden Stock , Maddox McVarthy , Simone Servadio This is my paper · ORCID
classification cs.LG
keywords spacedebrisLEOorbitcapacitySINDyLSTMsparseregressionorbitalpopulationforecastingMOCATresidentobjects
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that two data-driven methods—sparse regression (SINDy) and a long short-term memory neural network (LSTM)—can forecast the number of active satellites, derelict satellites, and debris in low Earth orbit almost as accurately as a costly Monte Carlo simulator, but much faster. The motivation is practical: the fast analytical model currently used for such forecasts, the source-sink model, is less accurate, and space-traffic risk assessments need quick answers when launch rates or collision scenarios change. The authors train both methods on averaged output from the Monte Carlo model and report prediction errors mostly below one percent for total populations, with larger errors for debris in individual altitude shells. If the claim holds, these methods could replace the source-sink model as a lightweight surrogate while preserving much of the high-fidelity model's accuracy.

What carries the argument

Two machine-learning mechanisms carry the argument. SINDy forms a library of candidate functions (constant, polynomials up to order 3, optionally sines and cosines), then solves a sparsity-promoting least-squares problem to select the few terms that best explain the numerically computed time derivatives; the result is a set of polynomial ordinary differential equations. LSTM is a recurrent neural network with a memory cell and forget, input, and output gates, trained with a mean-squared-error loss and the Adam optimizer, that maps a 50-step look-back sequence to future states. The training signal for both is the mean trajectory of 4,000 Monte Carlo runs, averaged over time, object type, and shell; the paper's claims depend on that average being a faithful summary of the high-fidelity model.

What would settle it

A reader could settle this by repeating each experiment with an explicit temporal split: use only the first 80 years of the 100-year averaged Monte Carlo series for training and report errors only on the held-out final 20 years. If errors remain below 1% for total populations, the forecasting claim stands; if they jump sharply, the current numbers are in-sample. A second check is to run SINDy with periodic basis functions and see whether the 11-year oscillations in debris and derelict counts appear; the current polynomial library smooths them out, so reproducing them would strengthen the surrogate claim.

Watch

Extended reading notes

Core claim

The paper claims that two data-driven learners can stand in for the analytical source-sink model while staying close to the high-fidelity Monte Carlo simulation. SINDy takes the averaged 100-year trajectories of active satellites, derelicts, and debris, builds a library of polynomial terms, and uses sparse regression to select a small set of coefficients, producing polynomial differential equations; these reproduce short-term dynamics with errors below 3% but diverge near year 70 for the total population. The LSTM, trained on 100 averaged datasets (Case 1) and on the mean of 1,000 per-shell simulations (Case 2), forecasts the final 20 years with total-population errors below 1%, active-satellite per-shell errors around 0.5%, total per-shell errors about 3.75%, and derelict per-shell errors about 6%. The debris population in individual shells shows the largest errors and a noise band tied to the 11-year solar-cycle variation in atmospheric density. Taken together, the authors argue these results support replacing the source-sink evolutionary model with a lighter, more accurate surrogate.

Load-bearing premise

The load-bearing premise is that the reported errors are true forecast errors. The models are trained on averaged Monte Carlo simulation data and compared against 'training and validation' curves in the results, and the paper never states that the final 20 years used for prediction were held out; if those years were inside the training set, the sub-1% errors show fit, not forecasting.

Editorial extensions

If this is right

  • Total-population forecasts from the LSTM carry relative errors below 1% over a 20-year horizon, so the network can serve as a fast screening tool for orbit-capacity questions.
  • SINDy-derived polynomial ODEs reproduce short-term LEO population trends with errors below 3%, providing an analytical alternative to the source-sink ODEs, though they diverge beyond roughly 70 years.
  • Per-shell LSTM predictions are less accurate than totals, with average errors of about 6% for derelicts and higher for debris, yet still acceptable for quick assessment in the authors' judgment.
  • Because both methods are trained on the averaged output of the Monte Carlo model, they inherit its physics without requiring the per-object Monte Carlo computation during prediction.
  • The paper's planned extensions—higher-order libraries, trigonometric basis functions, and retraining on multiple datasets—would address the observed long-term divergence and per-shell debris error.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Our inference: the reported sub-1% errors should be treated as in-sample fit until the authors specify a temporal train/test split; the plots labeled 'training and validation' do not demonstrate forecasting on unseen future years.
  • Our inference: training the LSTM on the 4,000 individual simulations instead of only their mean would let the surrogate output an ensemble spread, giving uncertainty bands comparable to the Monte Carlo's 3-sigma range rather than a single average curve.
  • Our inference: if the surrogate works for LEO, the same pipeline—averaged high-fidelity runs feeding SINDy or LSTM—could be applied to other orbital regimes or to sensitivity studies of launch-rate and disposal-policy changes, since retraining is cheaper than full Monte Carlo reruns.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes two data-driven surrogates, SINDy and LSTM, trained on outputs of the MOCAT-MC high-fidelity Monte Carlo model, to predict the LEO population of active satellites, derelicts, and debris. SINDy is used to derive polynomial ODEs for the total population and for individual atmospheric shells (shells 9 through 35), while the LSTM is trained in two configurations: total population across all shells (Case 1) and per-shell populations (Case 2). The authors report short-term SINDy errors below 3% for the total population, per-shell SINDy performance with two exceptions, and LSTM prediction errors below 1% for total population and up to about 6% for per-shell debris. The stated goal is to provide a lighter, faster, and more accurate alternative to MOCAT-SSEM for forecasting future LEO population evolution.

Significance. If the reported accuracies were out-of-sample forecasting results, the paper would offer a useful contribution: lightweight surrogates for a computationally expensive Monte Carlo model would enable faster orbit-capacity assessment and policy studies. The dataset choice (4,000 MOCAT-MC simulations) is appropriate, the network architecture is reported in detail, and the problem is clearly motivated. However, the central claim of accurate forecasting is currently supported only by error numbers that appear to be computed on the same data used for fitting. The paper does not define a held-out prediction window, does not test on unseen simulations, and does not compare against MOCAT-SSEM or any baseline. These omissions are load-bearing because the abstract and conclusions explicitly claim that the surrogates 'provide accurate forecasting' and could 'replace the MOCAT-SSEM.' The work is at a preliminary stage, and the central claim is not yet established by the evidence presented.

major comments (4)
  1. [LSTM MODEL RESULTS (pp. 12-16, Figs. 8-15)] The paper never states that the last-20-year prediction window was excluded from training. Case 1 trains on averaged full 100-year trajectories from the first 1,000 MOCAT-MC simulations, and Case 2 trains on the time-step average of the same simulations; the figures compare predictions to 'corresponding training and validation values' without defining any validation split. As written, the sub-1% LSTM errors may be in-sample reconstruction rather than forecasting. The authors must specify an explicit time-based or simulation-based holdout (e.g., train on years 0-80, predict years 80-100, or train on a subset of simulations and test on the rest) and report errors on data not used in training.
  2. [SINDY MODEL RESULTS (pp. 10-12, Fig. 6)] The SINDy coefficients are fit to the same averaged MOCAT-MC time series used to compute the reported percent errors, so the '<3%' short-term error is a goodness-of-fit measure, not a forecast skill measure. The paper itself states that the derived ODEs diverge near year 70, which directly contradicts the notion that the model provides accurate long-term forecasting. A proper evaluation would fit the dynamics on an initial time segment and evaluate on a later segment, or otherwise demonstrate generalization to unseen data.
  3. [SINDY MODEL RESULTS (pp. 11-12, Fig. 7)] The per-shell SINDy evaluation excludes shells 1-8 and the top shell, and reports two additional failures (shells 16 and 24). The conclusion that the method is accurate 'for every shell but two exceptions' therefore overstates coverage: almost a quarter of the 36 shells are not validated, and the excluded low-altitude shells are precisely the ones where the manuscript notes the data are sparse and the dynamics are poor. The claims about SINDy as a full LEO surrogate should be restricted to the shells that were actually demonstrated, or the excluded shells should be evaluated.
  4. [Problem Statement and Conclusions (pp. 3, 17)] The claimed advantage over MOCAT-SSEM is not supported by any direct comparison. The paper reports no accuracy comparison between the proposed surrogates and MOCAT-SSEM on the same evaluation data, no runtime comparison, and no baseline such as a linear extrapolation or a simple mean predictor. Without such comparisons, phrases like 'more accurate than the SSEM counterpart' and 'valid substitute to the source-sink ODE approach' are not justified. The authors should add a quantitative comparison against MOCAT-SSEM (and at least one trivial baseline) on held-out data.
minor comments (5)
  1. [Title/Abstract and throughout] There are several typographical errors: 'Orbit Capacity Toll' should be 'Orbit Capacity Tool,' 'SiNDy' should be 'SINDy,' 'Idirium-Cosmos' should be 'Iridium-Cosmos,' and 'resample a second-order polynomial ODE' should probably be 'resemble a second-order polynomial ODE.'
  2. [NUMERICAL RESULTS, Fig. 3 caption] The caption states that the mean is blue, 3σ deviation is green, and Monte-Carlo simulations are red, but the text in Section 'The Dataset' says '3σ standard deviation in red.' Please make the color descriptions consistent.
  3. [LSTM Model Results, Case 2 (p. 15)] The sentence 'the maximum average percent error for the N object population increases drastically, as expected' is vague; please report the actual maximum value and state whether the 'drastically' increasing error still remains below an acceptable threshold.
  4. [References] Reference [9] is incomplete: it lacks a publication venue, year, and page or DOI information. This makes it difficult to verify the described baseline architecture.
  5. [SINDY Approach (p. 7)] The phrase 'the MOCAT-MC tool allows for the active satellite population to be kept around a target value' is relevant to the data-generation setup, but it should be stated in the Dataset section rather than in the SINDy parameter-tuning discussion, so that the reader understands the controlled simulation condition from the outset.

Circularity Check

2 steps flagged · score 6.0 of 10

SINDy and LSTM 'forecasting' errors are computed against the training data, with no explicit holdout for the last 20 years.

  1. fitted input called prediction [Section 'Numerical Results', 'SINDy Model Results' (Eq. (8), Fig. 6)]
    "The dynamics have been derived using a third-order, n = 3, polynomial library for the library functions. Thus, the plots in Figure 6 were generated using the sum of all satellites in LEO, divided by family. Figure 6a compares the MC data with the numerically integrated polynomial dynamics derived via the SINDy approach, while Fig. 6b shows the relative error, evaluated as a percentage between prediction and validation data."

    The SINDy coefficients are estimated by least-squares regression on the same averaged MOCAT-MC time series: Eq. (8), Xi = (Theta(X)^T Theta(X))^-1 Theta(X)^T Xdot, uses finite-difference derivatives of that series. The 'validation data' shown in Fig. 6 is therefore the training data, and the reported <3% short-term error is an in-sample reconstruction measure, not a forecast. No held-out period or separate validation set is described for the SINDy evaluation, so the claimed 'accurate overlapping for the short-term prediction' reduces to goodness-of-fit.

  2. fitted input called prediction [Section 'Numerical Results', 'LSTM Model Results', Cases 1 and 2 (Figs. 8-11, 12-15)]
    "For Case 1 ... the model was trained on total S, D, and N population data across all 36 atmospheric shells from the first 1,000 MOCAT-MC simulations. ... The model was then trained continuously on all 100 datasets ... Using the LSTM training and predicting methods as described earlier, prediction data were obtained for the last 20 years of the 100-year time period ... compared to the corresponding training and validation values."

    The paper never states that the last 20 years were excluded from the 100-year training trajectories. Because each of the averaged training datasets spans the full 100-year MOCAT-MC time series, a model trained on those datasets has seen the prediction window unless an explicit temporal train/test split is made. The figures are described as comparing predictions to 'training and validation values', but no validation split is defined, and Case 2 concludes that 'the LSTM model trained very well on these datasets' while reporting errors. Thus the sub-1% and 0.5-6% error values are, as written, in-sample fit rather than demonstrated out-of-sample forecasting, which is the load-bearing evidence for replacing MOCAT-SSEM.

full rationale

The paper's central claim is that SINDy and LSTM can provide accurate forecasting of LEO populations and potentially replace MOCAT-SSEM. The SINDy evaluation is explicitly performed on the same averaged MOCAT-MC data used to fit the polynomial coefficients via Eq. (8), so its reported short-term error is a training error by construction. The LSTM evaluation is ambiguous: the model is trained on 100 averaged datasets spanning the full 100-year period, and 'prediction data were obtained for the last 20 years', but the text never states that those last 20 years were held out from training or that the figures show a temporal validation split. Because every downstream claim about speed, accuracy, and SSEM replacement depends on the surrogate predicting data it has not seen, the paper as written reduces its headline accuracy numbers to goodness-of-fit. The self-citations to prior MOCAT work are background context and are not load-bearing for this circularity finding. The score of 6 reflects partial circularity: at least one reported 'prediction' error is in-sample, and the LSTM forecasting claim is not independently established without an explicit holdout statement.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No physical constants are fit; the load-bearing quantities are model hyperparameters and learned weights. The central assumptions are that MOCAT-MC is ground truth, the mean is a sufficient training target, low-order polynomials suffice, and shell coupling is nearest-neighbor. No new physical entities are introduced.

free parameters (4)
  • SINDy sparsification threshold = not reported
    Controls which polynomial coefficients are set to zero; the authors state it cannot be too low but do not give the chosen value, and it affects shell-specific failures.
  • SINDy polynomial order = order 2 per shell, order 3 total
    Selected to match the SSEM model and available compute; higher orders not explored.
  • LSTM architecture hyperparameters = Table 1 values
    Number of layers, hidden units, sequence length, batch size, and epochs were tuned on the data and final values reported.
  • LSTM network weights and biases = learned
    The predictive model parameters fitted to the averaged MOCAT-MC time series; no weights are released.
assumptions (5)
  • domain assumption MOCAT-MC is the high-fidelity ground truth for LEO population evolution.
    The dataset section states the training data comes from MOCAT-MC and is used to evaluate the surrogates; no independent observational validation is provided.
  • domain assumption Averaging 4,000 Monte Carlo simulations preserves the dynamics needed for accurate forecasting.
    Authors train on the mean across simulations, which smooths stochastic collision and explosion events; the paper does not test whether the mean is a sufficient statistic.
  • ad hoc to paper A low-order polynomial library (order 2-3) can represent the relevant LEO population dynamics.
    SINDy order is chosen to match the SSEM polynomial form and computational limits, not derived from the data.
  • domain assumption Each shell's dynamics are coupled only to the adjacent shells above and below.
    SINDy training uses batches of three shells (i-1, i, i+1) based on the SSEM structure; far-range couplings are not tested.
  • standard math LSTM trained with MSE loss on the averaged time series will generalize to new futures.
    This is the standard assumption behind supervised time-series learning; no formal generalization guarantee is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Data-Driven Approach to Estimate LEO Orbit Capacity Models." pith.science (2026). https://pith.science/paper/IBDIDNQY

@misc{pith2026250719365,
  author       = {Pith},
  title        = {Pith review of: A Data-Driven Approach to Estimate LEO Orbit Capacity Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IBDIDNQY}},
  note         = {Machine review of arXiv:2507.19365}
}
read the original abstract

Utilizing the Sparse Identification of Nonlinear Dynamics algorithm (SINDy) and Long Short-Term Memory Recurrent Neural Networks (LSTM), the population of resident space objects, divided into Active, Derelict, and Debris, in LEO can be accurately modeled to predict future satellite and debris propagation. This proposed approach makes use of a data set coming from a computational expensive high-fidelity model, the MOCAT-MC, to provide a light, low-fidelity counterpart that provides accurate forecasting in a shorter time frame.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 17 canonical work pages

  1. [1]

    Analysis of Debris from the Collision of the Cosmos 2251 and the Iridium 33 Satellites,

    T. Wang, “Analysis of Debris from the Collision of the Cosmos 2251 and the Iridium 33 Satellites,” Science & Global Security, V ol. 18, No. 2, 2010, pp. 87–118

    work page 2010

  2. [2]

    The characteristics and consequences of the break-up of the Fengyun-1C spacecraft,

    N. L. Johnson, E. Stansbery, J.-C. Liou, M. Horstman, C. Stokely, and D. Whitlock, “The characteristics and consequences of the break-up of the Fengyun-1C spacecraft,” Acta Astronautica, V ol. 63, No. 1-4, 2008, pp. 128–135

    work page 2008

  3. [3]

    New monte carlo model for the space environment,

    D. Jang, D. Gusmini, P. M. Siew, A. D’Ambrosio, S. Servadio, P. Machuca, and R. Linares, “New monte carlo model for the space environment,” Journal of Spacecraft and Rockets, 2025, pp. 1–22

    work page 2025

  4. [4]

    Novel source–sink model for space envi- ronment evolution with orbit capacity assessment,

    A. D’Ambrosio, S. Servadio, P. Mun Siew, and R. Linares, “Novel source–sink model for space envi- ronment evolution with orbit capacity assessment,” Journal of Spacecraft and Rockets, V ol. 60, No. 4, 2023, pp. 1112–1126

    work page 2023

  5. [5]

    Effects of orbit raising and deorbiting in source-sink evolutionary models,

    D. Gusmini, A. D’Ambrosio, S. Servadio, P. M. Siew, P. Di Lizia, and R. Linares, “Effects of orbit raising and deorbiting in source-sink evolutionary models,”Journal of Spacecraft and Rockets, V ol. 61, No. 3, 2024, pp. 784–797

    work page 2024

  6. [6]

    Sparse identification of nonlinear dynamics with control (SINDYc),

    S. L. Brunton, J. L. Proctor, and J. N. Kutz, “Sparse identification of nonlinear dynamics with control (SINDYc),” IFAC-PapersOnLine, V ol. 49, No. 18, 2016, pp. 710–715. 17

    work page 2016

  7. [7]

    A review on the long short-term memory model,

    G. Van Houdt, C. Mosquera, and G. N ´apoles, “A review on the long short-term memory model,”Artifi- cial intelligence review, V ol. 53, No. 8, 2020, pp. 5929–5955

    work page 2020

  8. [8]

    Recurrent neural networks,

    S. Grossberg, “Recurrent neural networks,” Scholarpedia, V ol. 8, No. 2, 2013, p. 1888

    work page 2013

Show all 19 references
  1. [9]

    Evaluating Current Machine Learning Techniques On Predicting Chaotic Systems,

    S. Klinkachorn and J. Parmar, “Evaluating Current Machine Learning Techniques On Predicting Chaotic Systems,”

  2. [10]

    Risk Index for the Optimal Ranking of Active Debris Removal Targets,

    S. Servadio, N. Simha, D. Gusmini, D. Jang, T. St. Francis, A. D’Ambrosio, G. Lavezzi, and R. Linares, “Risk Index for the Optimal Ranking of Active Debris Removal Targets,” Journal of Spacecraft and Rockets, V ol. 61, No. 2, 2024, pp. 407–420

    2024

  3. [11]

    Threat level estimation from possible break-up events in leo,

    S. Servadio, D. Jang, and R. Linares, “Threat level estimation from possible break-up events in leo,” AIAA SCITECH 2024 Forum, 2024, p. 1065

    2024

  4. [12]

    Parameters evolution in source-sink space popula- tion evolutionary models,

    E. Ashley, C. S. Sanz, S. Servadio, and G. Lavezzi, “Parameters evolution in source-sink space popula- tion evolutionary models,” arXiv preprint arXiv:2407.21000, 2024

    2024 arXiv

  5. [13]

    NASA’s New Breakup Model Of EVOLVE 4.0,

    N. L. Johnson, P. H. Krisko, J. Liou, and P. Anz-Meador, “NASA’s New Breakup Model Of EVOLVE 4.0,” Advances in Space Research, 2001, 10.1016/S0273-1177(01)00423-9

    2001 doi

  6. [14]

    JASON report on the impacts of large satellite constellations,

    J. Program, “JASON report on the impacts of large satellite constellations,” Published by the U.S. National Science Foundation (NSF), 7 2021. JASON Program Report No. JSR-20-2H

    2021

  7. [15]

    A new empirical thermo- spheric density model JB2008 using new solar and geomagnetic indices,

    B. Bowman, W. K. Tobiska, F. Marcos, C. Huang, C. Lin, and W. Burke, “A new empirical thermo- spheric density model JB2008 using new solar and geomagnetic indices,” AIAA/AAS astrodynamics specialist conference and exhibit, 2008, p. 6438

    2008

  8. [16]

    Optimal Active Debris Removal mission planning to inform policy decisions,

    N. Simha, S. Servadio, M. Lifson, G. Lavezzi, and R. Linares, “Optimal Active Debris Removal mission planning to inform policy decisions,” Acta Astronautica, V ol. 228, 2025, pp. 224–236

    2025

  9. [17]

    Nonlinear filtering with a polynomial series of Gaussian random variables,

    S. Servadio, R. Zanetti, and B. A. Jones, “Nonlinear filtering with a polynomial series of Gaussian random variables,” IEEE Transactions on Aerospace and Electronic Systems , V ol. 57, No. 1, 2020, pp. 647–658

    2020

  10. [18]

    Analytical Uncertainty Propagation and Maximum A Posteriori Filtering with the Koopman Operator,

    S. Servadio, G. Lavezzi, C. Hofmann, D. Wu, and R. Linares, “Analytical Uncertainty Propagation and Maximum A Posteriori Filtering with the Koopman Operator,” IEEE Transactions on Aerospace and Electronic Systems, 2025

    2025

  11. [19]

    Accurate discharge and water level forecasting using ensemble learning with genetic algorithm and singular spectrum analysis-based denoising,

    A. D. Nguyen, P. Le Nguyen, V . H. Vu, Q. V . Pham, V . H. Nguyen, M. H. Nguyen, T. H. Nguyen, and K. Nguyen, “Accurate discharge and water level forecasting using ensemble learning with genetic algorithm and singular spectrum analysis-based denoising,” Scientific Reports, V o...

    2022

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.