{"id":"205fc161-c164-42d3-a942-169caec98057","arxiv_id":"2412.04568","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A trained XGBoost classifier predicts stable and unstable orbits around a planet from nine initial orbital parameters with about 98% accuracy and at far lower cost than N-body simulation.","lead":"Researchers trained machine learning models on 100,000 computer simulations of a star, a planet, and a test particle to predict which orbits around the planet stay stable. The best model, XGBoost, classifies orbits with about 98% accuracy and runs roughly 100,000 times faster than the direct simulations.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 10^4-orbit survival label that defines ground truth in Section 3 may overstate true dynamical stability; all accuracy numbers and map comparisons inherit this finite-time proxy, and the paper itself concedes in §6.2 that labeled-stable particles might decay later.","rationale":"The reader's weakest_assumption is exactly the finite-time label, and I agree it is the most load-bearing point. I considered alternative concerns: the 98.48% test accuracy is higher than the per-map accuracies reported in Section 6 (e.g., 86.24% for the retrograde Domingos case), and no code or data artifacts are public. Those are real but secondary—the paper discloses the per-map accuracies, and missing artifacts are a reproducibility limitation rather than a flaw in the argument. The finite-time label is more fundamental because it biases the ground truth before training begins: the classifier, its accuracy, and every downstream map comparison all inherit the 10^4-orbit survival definition. The paper deserves credit for stating the criterion explicitly and for testing against Domingos et al. (2006), PDS110b, and Saturn's Inuit group; those benchmarks show the surrogate works within its operational definition. The proposed re-integration test would settle whether the finite-time label is a harmless proxy or a systematic overstatement of the stable regions. Because the reader already conditioned the verdict on this issue, the correct outcome is to keep the conditional verdict rather than change it.","tokens_in":14521,"tokens_out":6928,"duration_ms":73598,"concrete_test":"Draw a stratified sample of about 2,000 labeled-stable initial conditions, spanning prograde boundaries (0.3–0.5 R_Hill), retrograde boundaries (0.8–0.95 R_Hill), particle eccentricities up to 0.8, inclinations from 0° to 180°, and planet eccentricities up to 0.9. Reintegrate each case with the same REBOUND/IAS15 setup and collision/ejection criteria as Section 3, for 10^5 and 10^6 planetary orbital periods. Compute the fraction of labeled-stable orbits that decay and the inward shift of the empirical stable boundary. If the decay fraction exceeds about 5% or the boundary shifts inward by more than 0.05 R_Hill, the reported 98.48% accuracy and the map comparisons overstate true stability, and the paper should either relabel its product as '10^4-orbit survival maps' or retrain with longer integrations. If the decay fraction is negligible, the concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 defines a simulation as stable if the particle survives 10^4 planetary orbital periods without collision or ejection. This finite-time survival is the label for all 100,000 training examples, so the XGBoost classifier, the 98.48% test accuracy, and the reproduced maps in Sections 6.1–6.3 all measure agreement with a transient-survival proxy rather than long-term dynamical stability. The paper itself states in Section 6.2 that 'some of these stable particles might become unstable with longer integration times.' Near the Hill boundary in the elliptic three-body problem, 10^4-period survivors can be chaotic transients with heavy-tailed lifetimes; changing the stopping time moves the empirical boundary (for example, the retrograde boundary near 0.93 R_Hill would likely recede). Since the bias enters before training, no amount of hyperparameter optimization, resampling, or threshold tuning can fix it. The honest claim is that the model maps survival to 10^4 orbits; the title and abstract claim stable orbits and stable regions more generally. This is the most load-bearing concern because every reported metric and comparison inherits the same label.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents a supervised machine-learning surrogate for three-body stability maps in the star-planet-test-particle problem. The authors generate 100,000 dimensionless numerical integrations with rebound/IAS15, label each initial condition as stable or unstable according to survival for 10^4 planetary orbital periods, and train eight classifiers on nine orbital features. After hyperparameter and threshold tuning, an XGBoost model achieves 98.48% accuracy on a held-out 20% test set, with stable-class precision and recall of 94% and unstable-class values of about 99%. The model is then applied to reproduce published maps (Domingos et al. 2006), a PDS110b ring scenario, and Saturn's Inuit satellites, with reported accuracies of 86 to 98%, and to generate maps in about 0.5 seconds. The paper is clearly written and the workflow is easy to follow, but the finite-time definition of the ground-truth label is a central limitation that the manuscript itself acknowledges in Section 6.2.","tokens_in":14701,"tokens_out":6776,"duration_ms":74576,"significance":"Within the defined task of classifying finite-time survival for 10^4 planetary periods, the paper demonstrates a fast, internally consistent surrogate. The held-out evaluation, the threshold and hyperparameter tuning, and the comparisons against external systems are genuine strengths, and the feature-importance ranking is physically sensible. If the 10^4-period survival criterion is accepted, the inference-only speed-up of roughly five orders of magnitude is substantial and the comparison maps are informative. The significance for true long-term dynamical stability is, however, qualified by the ground-truth definition: every accuracy number and map inherits the finite-time proxy, so the title and abstract claims about 'stable orbits' and 'stable regions' overstate what the model has been validated to predict. The paper would be strengthened by a convergence test over integration time and by a clear statement that the maps are survival maps for the stated timescale.","major_comments":[{"comment":"The ground truth for all 100,000 training examples, and therefore for every reported metric and comparison map, is survival for 10^4 planetary orbital periods, not dynamical stability in a longer-term sense. The paper itself concedes in Section 6.2 that 'some of these stable particles might become unstable with longer integration times.' Because this finite-time proxy enters before training, the 98.48% test accuracy and the external comparisons in Sections 6.1 to 6.3 cannot be read as validating the model for long-term stability. Please either rescope the wording of the title, abstract, and conclusions to 'survival over 10^4 orbital periods' or add a convergence experiment, for example by relabeling with 10^3, 10^5, and 10^6-period integrations at boundary regions and checking whether the empirical stability boundary moves.","section":"Sec. 3 / Sec. 6.2"},{"comment":"The headline metrics, including the 98.48% accuracy and the 94% stable-class recall and precision, are point estimates from a single 20% holdout split. The model-selection comparison in Figure 5 relies on differences as small as about 1% between algorithms, and the claim that XGBoost is the best-performing model would be much more robust with bootstrap confidence intervals or repeated cross-validation. Please report uncertainty estimates for the test-set metrics, particularly for the minority stable class.","section":"Sec. 5.1"},{"comment":"The 'numerical simulation results' used as the reference in the Domingos et al., PDS110b, and Saturn comparisons are not described with the same level of detail as the training dataset. It is not stated whether these reference maps were produced with the same 10^4-period stopping time, the same collision radius prescription, and the same ejection criteria as the training labels. Without this specification, the comparison accuracies in Figures 9, 11, and 13 cannot be fully audited. Please state explicitly which integrator, stopping criterion, and stability criteria were used for each comparison map.","section":"Sec. 6.1 / Sec. 6.2"}],"minor_comments":[{"comment":"The sentence stating that XGBoost 'attains the same percentages for precision and recall' is ambiguous, because the immediately preceding numbers refer to LightGBM and Random Forest while the abstract quotes 94% for XGBoost; please give the exact XGBoost values in the text so that Figure 5 and the abstract agree.","section":"Sec. 5.1 / Fig. 5"},{"comment":"The collision and ejection percentages are reported as 53.56% and 46.44% of unstable particles in Section 3 but as 47.22% and 40.95% in Section 7; these are consistent only if the denominators differ, so please state the denominator explicitly in both places.","section":"Sec. 3 / Sec. 7"},{"comment":"The hyperparameter grids for the Grid and Random searches, the number of cross-validation repeats, and the random seeds are not specified; these details are important for interpreting the small performance differences between the tree-based ensembles in Figure 5.","section":"Sec. 5"},{"comment":"The paper does not currently provide code, data, or trained-model artifacts, and the web interface is described only as forthcoming, making exact reproduction of the headline results difficult; please include a reproducibility statement or a plan for releasing these artifacts.","section":"Sec. 7"},{"comment":"There is a small typographical error in the sentence 'We labeled each initial condition with the numerical results as either stable or un unstable system'; 'un unstable' should be 'unstable'.","section":"Sec. 3"}],"recommendation":"major_revision","confidential_remarks":"The central scientific question is well posed and the paper is honest about the finite-time nature of its labels, but the finite-time proxy is load-bearing for every claimed result. The revision path is feasible: rescope the language to finite-time survival or add a convergence experiment, report uncertainties on the metrics, and specify the reference-map generation. I do not see a reason to reject the manuscript, but the requested changes go beyond cosmetic revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something genuinely useful: it trains an XGBoost classifier on 100,000 three-body simulations and uses it to map stable regions around a single planet in seconds. The nine-dimensional feature space (mass ratio plus orbital elements of planet and test particle) is a natural extension of earlier ML work on multi-planet and circumbinary stability, and the paper is the first to apply that idea to Hill-radius satellite/ring configurations. The held-out test accuracy of 98.48% is credible, and the comparisons to Domingos et al., PDS 110b, and Saturn's Inuit group are a good idea because they test generalization to real cases. I appreciate that the authors report per-class precision/recall and don't hide the weaker retrograde performance (86.24%). The feature importance analysis also matches physical intuition, which is a good sanity check.\n\nThe soft spots are real but not fatal. The ground truth is finite-time survival: a particle is 'stable' if it survived 10^4 planetary periods. All accuracy numbers, including the 98.48%, measure agreement with that transient-survival proxy. The paper itself concedes in Section 6.2 that some labeled-stable particles may become unstable on longer timescales. So the title and abstract overstate slightly; the model is mapping survival over 10^4 orbits, not long-term dynamical stability. That's a common convention in the field, but it should be stated upfront. Second, the retrograde boundary is handled worse than the prograde one, and the model misses stable particles beyond 0.8 Hill radii at low eccentricity. The authors acknowledge this, but it limits the tool's reliability for retrograde ring systems. Third, no code, data, or model artifacts are provided (only a promised web interface), so the reported metrics can't be independently checked. The paper would be much stronger if the trained model and dataset were released.\n\nWho is this for? People working on exomoon or ring stability surveys who want a fast preliminary screening tool, and anyone teaching ML applications in orbital dynamics. It deserves a serious referee. I would send it to peer review, but request a revision that (a) rephrases the title/abstract to say 'survival over 10^4 orbits', (b) adds confidence intervals or at least error bars on the main metrics, and (c) makes the artifacts available. With those changes it would be a reliable reference.","headline":"A solid ML-surrogate paper for mapping short-term orbital survival around planets, but the 10^4-orbit stability label should be stated plainly in the title and abstract.","tokens_in":15301,"tokens_out":2120,"would_cite":true,"duration_ms":75918,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A gradient-boosted tree classifier trained on 100,000 three-body simulations predicts whether test-particle orbits around a planet remain stable for 10,000 planetary periods, reaching 98.48% accuracy and producing full stability maps in…","keywords":["machine learning","orbital stability","three-body problem","exoplanets","XGBoost","stability maps","N-body simulations","satellite and ring systems"],"falsifier":"Re-run the 20,000 test cases with the integration time extended from $10^{4}$ to $10^{6}$ planetary orbital periods and count how many orbits the model labels stable become unstable; if that fraction is more than a few percent, the 98.48% accuracy is an artifact of the finite integration horizon. A cheaper check is to compare the model's boundary predictions against fresh high-resolution integrations for a dense grid of retrograde inclinations near 180 degrees, the region the paper notes is most sensitive.","tokens_in":14263,"feed_emoji":"🪐","tokens_out":8106,"duration_ms":74055,"temperature":0.7,"pith_summary":"This paper claims that a machine-learning classifier can replace expensive numerical integration when mapping which orbits around a planet remain stable. The authors train a gradient-boosted tree model on 100,000 dimensionless three-body simulations, with stability defined as surviving 10,000 planetary orbital periods without collision or ejection. On held-out test data the model reaches 98.48% accuracy, 94% recall and precision for stable orbits, and 99% for unstable orbits; it also reproduces published stability maps for exomoon, ring, and Saturn-satellite systems with accuracies between 86% and 98%. A full stability map is produced in about 0.5 second, versus days for direct integration, so the method turns stability mapping into a fast screening tool.","feed_headline":"AI predicts stable orbits around planets in half a second","feed_subtitle":"XGBoost trained on 100,000 three-body runs hits 98.48% accuracy and turns days of N-body simulation into instant maps.","key_machinery":"The load-bearing object is a labeled dataset of $10^{5}$ dimensionless three-body simulations, each described by nine features: system mass ratio, particle semi-major axis, eccentricity, inclination, argument of pericenter, longitude of node, and true anomaly, plus planet eccentricity and true anomaly. Simulations are integrated with a high-accuracy numerical integrator until particle collision with the planet, ejection to a hyperbolic orbit, or the 10,000-period horizon, and each initial condition is labeled stable or unstable accordingly. The model that carries the argument is XGBoost, a gradient-boosted decision-tree ensemble, tuned with a decision threshold of 0.4 and selected hyperparameters; it maps the nine-dimensional feature space to a stability probability, and the paper uses its built-in feature-importance scores to show that particle semi-major axis, eccentricity, and inclination dominate the classification.","core_discovery":"The paper's central claim is that orbital stability in the star-planet-test-particle three-body problem is a learnable function of nine initial-condition features, and that a tuned gradient-boosted tree model extracts that function accurately enough to replace numerical integration for map generation. Training on 100,000 dimensionless simulations in which 'stable' means the particle neither collided with the planet nor was ejected within 10,000 planetary orbital periods, the best model reaches 98.48% test accuracy and an AUC of 0.9978, with recalls and precisions of 94% for stable particles and 99% for unstable ones. The same model reproduces published stability maps at accuracies of 95.86% for the prograde case and 86.24% for the retrograde case of Domingos et al. (2006), 93.68% for the PDS110b ring system, and 97.57% for Saturn's Inuit satellites, while producing a 10,000-point map in roughly 0.5 second. The authors present this as a method that makes stability mapping a fast screening tool; the finite-horizon definition of stability is the main caveat they state explicitly.","pith_inferences":["The advertised speedup applies to prediction after training; training itself required the 100,000 simulations, so the practical gain is for repeated map generation within the already-simulated parameter space.","Augmenting the training set with survival labels at 10^5 and 10^6 planetary periods would convert the binary classifier into a time-dependent survival predictor and directly test whether the finite-horizon definition drives the reported accuracy.","The paper's own remark about sensitivity near i=180 suggests that an active-learning scheme adding extra simulations near that boundary could reduce the misclassifications it reports there.","A natural hybrid use of the model is to pre-screen candidate exomoon orbits around the exoplanet catalogue and reserve full N-body integration for the uncertain boundary strips."],"forward_implications":["Stability maps around a star-planet system can be generated in seconds instead of days, making parameter surveys of many systems feasible.","Within the trained mass-ratio and orbital-element space, the same model applies to planet-satellite pairs, planetary ring systems, and binary minor planets without new integrations.","Feature-importance rankings identify particle semi-major axis, eccentricity, and inclination as the dominant stability parameters, with planet eccentricity secondary, which can guide future feature engineering.","The planned public web interface will let researchers obtain stability predictions for their own systems without running N-body simulations.","Discrepancies between ML maps and analytical boundaries, for example in the retrograde case, point to specific regions that deserve targeted numerical study."],"supporting_citations":[{"why":"Supplies the numerical integrator used to generate all 100,000 training labels.","marker":"Rein & Spiegel 2015"},{"why":"Provides the analytical stability-boundary expressions and the prograde/retrograde maps used as the primary comparison baseline.","marker":"Domingos et al. 2006"},{"why":"Used as a second comparison target; its 1.3 million simulation results for PDS110b test generalization to a real candidate ring system.","marker":"Pinheiro & Sfair 2021"},{"why":"Exoplanet catalogue that justifies the surveyed mass-ratio range and the collision-radius choice.","marker":"Schneider et al. 2011"},{"why":"Defines the extreme-gradient-boosting algorithm used for the best-performing model.","marker":"Chen & Guestrin 2016"},{"why":"Prior machine-learning stability classifier for compact multi-planet systems that establishes the speed-comparison context for the approach.","marker":"Tamayo et al. 2020"}],"fun_headline_variants":["ML maps stable orbits 100,000x faster than N-body","AI predicts planet orbit stability in under a second","XGBoost nails 98% accuracy on orbital stability maps","Stable orbit maps from AI: 100k simulations, 0.5s","AI maps stable orbits for exomoons and rings in 0.5s"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that 'stable' can be defined as surviving 10,000 planetary orbital periods; the paper itself notes in Section 6.2 that some particles labeled stable might become unstable with longer integration times, so every accuracy figure inherits this finite-horizon definition.","fun_headline_variants_meta":{"raw":{"variants":["ML maps stable orbits 100,000x faster than N-body","AI predicts planet orbit stability in under a second","XGBoost nails 98% accuracy on orbital stability maps","Stable orbit maps from AI: 100k simulations, 0.5s","AI maps stable orbits for exomoons and rings in 0.5s"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000818,"raw_usage":{"total_tokens":3634,"prompt_tokens":1050,"completion_tokens":2584,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":666,"completion_tokens_details":{"reasoning_tokens":2490}},"tokens_in":666,"tokens_out":2584,"duration_ms":16741,"temperature":1.0,"reasoning_tokens":2490,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:23:44.361762+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 20,000 test cases with the integration time extended from $10^{4}$ to $10^{6}$ planetary orbital periods and count how many orbits the model labels stable become unstable; if that fraction is more than a few percent, the 98.48% accuracy is an artifact of the finite integration horizon. A cheaper check is to compare the model's boundary predictions against fresh high-resolution integrations for a dense grid of retrograde inclinations near 180 degrees, the region the paper notes is most sensitive.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Used as a second comparison target; its 1.3 million simulation results for PDS110b test generalization to a real candidate ring system."},{"cited_title":"& Guestrin, C","cited_arxiv_id":null,"evidence_quote":"Defines the extreme-gradient-boosting algorithm used for the best-performing model."},{"cited_title":"2020, PNAS, 117, 18194","cited_arxiv_id":null,"evidence_quote":"Prior machine-learning stability classifier for compact multi-planet systems that establishes the speed-comparison context for the approach."}],"review_version":1}