{"id":"8aa3a681-2f8b-407a-9542-b63cf4eaff91","arxiv_id":"2508.04982","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"On a synthetic JWST/Ariel-style spectral database, XGBoost and SVM with per-spectrum normalization and log-abundance targets outperform random forests and other classical regressors for exoplanet atmospheric retrievals.","lead":"The authors test eight standard machine learning regression methods for reading planet atmospheres from transit spectra, using a large synthetic dataset, and also test six ways of preparing the data. They find that normalizing each spectrum and using log-scale chemical abundances makes the best models (XGBoost and support vector machines) the most accurate.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark may be inflated by instance-level train/test split: 105,887 spectra from 5,900 planets; if the same planet appears in both splits, the model can memorize planet-specific baselines and the reported XGB/SVM accuracy does not generalize to unseen planets.","rationale":"The reader's weakest assumption concerns applying a noiseless, isothermal, cloud-free synthetic training set to real noisy JWST data. That is a real limitation, but it is not the most load-bearing issue: the paper's quantitative evidence for the central claim is the synthetic benchmark itself, and that benchmark may be compromised before any real-data extrapolation is attempted. The text implies an instance-level random split of spectra, not a split by unique planet. With ~18 spectra per planet and the equilibrium temperature fixed per planet, test spectra drawn from planets already seen during training allow the model to recognize the planet-specific spectral baseline. This would make the reported test accuracy an artifact of repeated planet identities rather than a measure of generalization to new planets. A grouped-by-planet split is a simple, decisive check. If it degrades the results, the main claim 'ML regressors are capable of reliably reproducing the planetary parameters' is not established from the presented benchmark. If it does not degrade, the reader's original CONDITIONAL verdict remains appropriate for the remaining domain-shift and calibration concerns. I therefore keep the verdict CONDITIONAL, but the condition should be the grouped-split rerun, not merely caution about real-data application.","tokens_in":18475,"tokens_out":7984,"duration_ms":103610,"concrete_test":"Retrain the best-performing configurations (XGB and SVM on NL) with an 80/20 split grouped by unique planet: assign entire planets to training or test (e.g., ~4,720 vs ~1,180 planets), generating no more than one atmospheric realization per planet in the test set, and recompute the same MAE/bias figures. Report how many unique planets currently appear in both sides of the existing split. If the grouped-split test errors increase substantially (particularly for T), the reported instance-level benchmark was inflated by planet leakage and the central claim is not established; if errors are comparable, the leakage concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 2 generates 105,887 spectra from 5,900 unique planets (≈18 spectra per planet), each retaining the same stellar/planetary parameters and equilibrium temperature while only the trace abundances are re-sampled. Section 3.4 then splits 'a random subset of n=60,000 samples' 80/20 at the level of individual spectra; no grouping by unique planet is reported. Under such an instance-level split, spectra from the same planet appear in both training and test sets. Because the target T is the planet's fixed equilibrium temperature and the normalized spectral shape (Rayleigh/CIA slope, scale height) is determined by planet gravity/temperature, the model can memorize a planet's spectral baseline from training and exploit it on test spectra of the same planet. This artificially inflates test accuracy, especially for T, and may also help isolate trace-gas absorption features. The reported benchmark would then measure interpolation over repeated configurations of the same 5,900 planets rather than generalization to new planets, which is exactly what 'reliably reproducing the planetary parameters' requires. The WASP-39b case study cannot rescue this: it is one target, and the paper itself notes two predictions fall outside the training range. The grouped-split issue is thus the most load-bearing threat to the central claim.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks eight classical supervised regression algorithms (PLS, SVM, KNN, DT, RF, VOTE, STACK, XGB) for retrieving equilibrium temperature and five trace-gas abundances from synthetic exoplanet transmission spectra. It uses 105,887 noiseless spectra generated from 5,900 ABC/TauREx planets at 52 wavelengths and six preprocessing configurations (standardization vs per-spectrum normalization, linear vs log abundance targets, with/without spectral mean and standard deviation as extra features). Performance is assessed on 12,000 held-out spectra through scatter plots and quantile-binned error plots, along with training-time comparisons. The paper concludes that XGB and SVM trained on normalized spectra with log-abundance targets perform best, and it applies the XGB/NL model to the JWST WASP-39b spectrum, reporting abundances within ranges previously reported in the literature.","tokens_in":18784,"tokens_out":8829,"duration_ms":103030,"significance":"If the ranking survives a properly controlled validation, the paper is a useful, interpretable benchmark of classical ML methods against a realistic synthetic database, with concrete guidance on preprocessing and on accuracy/precision behavior across parameter space. Strengths include the large ground-truth test evaluation, explicit discussion of the low-abundance sensitivity 'knee' and of nonuniform sampling, realistic hedges on WASP-39b upper limits, and comparison with literature retrieval values. The main ranking, however, rests on an instance-level train/test split, unspecified hyperparameter selection, and visual rather than quantitative metrics, so the findings are not yet established at the level claimed. The work is potentially publishable after a corrected, more quantitative validation.","major_comments":[{"comment":"The train/test split is performed at the level of individual spectra, not grouped by unique planet. §2 says 105,887 spectra come from 5,900 planets with fixed stellar/planetary parameters and equilibrium temperature, only trace abundances being resampled (~18 spectra per planet). An 80/20 split of 60,000 spectra therefore places spectra from the same planet in both training and test sets. Since T is fixed per planet and the spectral baseline (scale height, Rayleigh/CIA slope) depends on planet parameters, the regressors can memorize planet-specific baselines and inflate test accuracy, particularly for T but possibly also for abundances through feature isolation. The reported benchmark then measures interpolation over repeated configurations of 5,900 planets, not generalization to new planets. The WASP-39b case is one unseen target and cannot compensate. Please redo the evaluation with a","section":"§2, §3.4"},{"comment":"Hyperparameters are listed (C=500, gamma=0.028, epsilon=0.1 for SVM; k=6 for KNN; 115 trees for RF; max_depth=6 and learning rate 0.1 for XGB, etc.) but no tuning procedure is described. The statement 'we optimized the respective hyperparameters to achieve best performance' does not say what data were used for selection. If the same test set was used to choose hyperparameters, the comparison is biased and the reported ranking could be circular. Provide the search space, selection criterion, and use a validation split or nested cross-validation independent of the test set.","section":"§4.8"},{"comment":"The central ranking (XGB/SVM best, RF underperforming) is supported by visual inspection of Figures 9–26 without a table of quantitative global metrics. The paper has no MAE, RMSE, R², or similar values per target and per configuration; the claim that XGB and SVM have 'low bias and variance' is not quantified. For a systematic benchmark, a summary table of metrics with uncertainties is essential to substantiate 'best performing' and to allow comparison with future work. Please add such a table, at least for the six preprocessing configurations and the top methods, ideally for all 48 experiments.","section":"§5.2–5.3"},{"comment":"The training database is noiseless, cloud-free, isothermal, H/He-dominated with fixed mean molecular weight, while the WASP-39b application uses a real, noisy, cloudy, non-isothermal spectrum. Test-set accuracy is therefore an upper bound for real data, and the single-planet agreement may be fortuitous; moreover two predicted abundances (CO, NH3) lie outside the sampled training range. The paper does hedge these points, but Section 7 states the models are 'capable of reliably reproducing the planetary parameters' and 'we were able to reproduce' literature values. Please add a noise-injected and/or cloudy test set, or explicitly limit the claim to the noiseless database and describe the WASP-39b results as illustrative rather than confirmatory.","section":"§2, §6"}],"minor_comments":[{"comment":"Typo: 'validated on a the case study' should be 'validated on the case study'; also 'W ASP-39b' has an anomalous space.","section":"Abstract"},{"comment":"Typo: 'as paert of a NeurIPS 2022 competition' should be 'as part of'. Also '105,887 planet instances' and later 's=105,887 planets' in §3.1 conflict with the 5,900 unique planets; use 'spectra' or 'samples' for clarity.","section":"§2"},{"comment":"Typo: 'esemble method' should be 'ensemble method'; also 'random subset of of the features' has a duplicated 'of'.","section":"§4.5"},{"comment":"Typo: 'preformance' should be 'performance'. The cross-reference 'Figure 5.3 provides yet another way' is not a valid figure number; it should refer to the correlation-matrix figure (currently labeled Figure 6).","section":"§5.3"},{"comment":"Figure cross-references are inconsistent: 'Figure 6 depicts the original (blue dots) and the rebinned (red diamonds) spectra' should be Figure 7, and 'we reproduce in Figure 6' referring to the last row of Figure 12 should be Figure 8. Also 'the same exact 52 spectral wavelength values' is redundant.","section":"§6"},{"comment":"The N M notation uses M_i and σ_i(M) for spectral mean and standard deviation, which can be confused with the spectrum M(λ). Consider renaming these features (e.g., μ_i, s_i). In addition, the manuscript provides no code or preprocessing scripts; only a link to the public ABC database is given, which is not sufficient to reproduce the exact 105,887-spectrum realization.","section":"§3.4 / Data Availability"}],"recommendation":"major_revision","confidential_remarks":"The paper is a useful systematic comparison, but I am not confident in the reported ranking until the split is grouped by planet and quantitative metrics are tabulated. The instance-level split concern is the most load-bearing and could change the conclusions; it is, however, fixable within the manuscript's scope. I would also encourage the editor to consider whether the journal expects code/data release for benchmark papers of this kind."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a well-scoped benchmark of eight classical regressors and six preprocessing schemes on 105,887 synthetic transmission spectra, and the finding that XGB and SVM beat random forests is a genuinely useful counterpoint to the RF-heavy literature. The parameter-resolved bias/variance plots are a nice diagnostic and the WASP-39b case study is honestly framed, including the fact that two predictions fall outside the training range. I think the paper deserves a serious referee, but it needs a grouped train/test split before the headline claim is taken at face value.\n\nThe soft spot is the split. Section 2 generates ~18 spectra per planet from 5,900 planets, with the equilibrium temperature and stellar/planetary parameters fixed per planet and only the trace-gas abundances re-sampled. Section 3.4 then takes a random 60,000 spectra and splits those 80/20 at the spectrum level, with no grouping by planet. So the same planet can appear in both train and test. The model can learn that a particular spectral baseline belongs to a particular planet and use that to predict T (and to some extent the gas signals) on a different spectrum of the same planet. That means the reported test error is part interpolation over repeated configurations of the same planets, not clean generalization to new planets. The stress-test note is right that this is the load-bearing threat. All models face the same leak, so the relative ranking may survive, but the absolute error numbers and the 'reliably reproducing the planetary parameters' claim are overstated until a grouped split shows otherwise.\n\nOther issues are minor by comparison. There is no code or generated data shipped; the paper points to the original ABC database, but not its own 105,887-spectrum realization. The hyperparameter tuning procedure is described only as 'optimized for best performance,' with no validation protocol, so test-set contamination cannot be ruled out. The 'uncertainty quantification' is descriptive—bias/variance as a function of the parameter—not calibrated prediction intervals. And the real-data validation is one planet, with two predictions outside the training distribution; the authors flag this, which I credit.\n\nWhat the paper does well: the 48-combination comparison is new as a systematic study, the normalization + log-target finding is consistent with prior ARIEL-related work and is argued clearly, and the knee analysis is a useful way to think about sensitivity limits. The writing is honest about the non-uniform ABC sampling and the limitations of the case study.\n\nNet: this is for the exoplanet ML-retrieval community and for people choosing a default regressor for fast retrievals. It deserves peer review, but the first revision should address the grouped split, ship the data, and pin down the tuning protocol. I'd cite it once the grouped-split results confirm the ranking.","headline":"A useful, systematic ML-retrieval benchmark whose headline ranking is plausible but whose test error may be inflated by an ungrouped train/test split; worth a referee, but needs a grouped split and a data release first.","tokens_in":19336,"tokens_out":3558,"would_cite":false,"duration_ms":38531,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that XGBoost and SVM regressors, trained on per-spectrum normalized spectra with logarithmic abundance targets, can reproduce exoplanet atmospheric parameters from transmission spectra and recover WASP-39b abundances consis","keywords":["exoplanet atmospheres","transmission spectroscopy","atmospheric retrieval","machine learning regression","XGBoost","support vector machines","uncertainty quantification","WASP-39b"],"falsifier":"Train the same XGB/NL regressor on the ABC-like database but generate test spectra with the forward model including clouds or hazes, non-isothermal temperature profiles, and realistic JWST noise; if retrieval errors grow sharply for the same abundances, or if the WASP-39b predictions leave the published ranges when noise realizations are varied, the central transfer claim fails. A cheaper check: feed the trained model a chemically self-consistent atmosphere (with CO/CH4 equilibrium) and see whether the predicted abundances remain accurate.","tokens_in":18343,"feed_emoji":"🪐","tokens_out":8902,"duration_ms":96168,"temperature":0.7,"pith_summary":"This paper claims that ordinary, interpretable regression algorithms—especially extreme gradient boosting (XGB) and support vector machines (SVM)—can retrieve an exoplanet's temperature and five trace-gas abundances from a transmission spectrum, as long as the spectra are normalized per sample and the abundance targets are logarithmic. The motivation is speed: standard Bayesian retrievals take hours to days per planet, while a trained regressor predicts almost instantly. In a benchmark on 105,887 synthetic spectra built from the Ariel Big Challenge database, XGB and SVM outperform partial least squares, k-nearest neighbors, decision trees, random forests, voting, and stacking. The paper also finds that the preprocessing of the spectra can affect performance as much as the choice of algorithm. The best model, applied to the JWST spectrum of WASP-39b, predicts abundances inside the ranges reported by earlier Bayesian retrievals.","feed_headline":"XGBoost matches slow Bayesian retrievals on exoplanet spectra","feed_subtitle":"On 105,887 synthetic spectra, sample-wise normalization plus log targets puts XGB and SVM on top.","key_machinery":"The key mechanism is per-spectrum normalization: each 52-bin transit spectrum is centered by subtracting its own mean and scaled by its own standard deviation, so the model learns spectral shape rather than overall flux level. This is paired with logarithmic targets for chemical abundances, which spreads out the low-abundance end of the training distribution. The third piece is the XGB regressor, a sequential ensemble of shallow decision trees that concentrates training weight on the hardest examples. The paper's information-content analysis, using feature correlations and a transverse decomposition, explains why normalization exposes feature-target correlations that standardization leaves h","core_discovery":"The central claim is that an XGB regressor trained on normalized transmission spectra with logarithmic abundance targets reliably reproduces the atmospheric parameters used to generate the spectra, and that this transfers to a real JWST spectrum. On the test set, the best models show low bias and variance across most of the parameter space, with accuracy degrading at very low abundances (a 'knee' near $10^{-6}$--$10^{-7}$) and at high temperatures, where the training set is sparse. XGB and SVM outperform PLS, KNN, DT, RF, VOTE, and STACK; per-spectrum normalization is consistently better than feature-wise standardization, while adding the spectral mean and standard deviation as extra feature","pith_inferences":["The benchmark uses noiseless synthetic spectra; injecting realistic JWST or Ariel noise into training or test spectra is a direct extension the paper does not perform, and it would test whether the ranking of methods survives.","Because the database samples abundances independently, it ignores chemical networks; testing on chemically self-consistent forward models could change which correlations the regressors exploit and where the low-abundance knee sits.","The paper's correlation-matrix diagnostic—predictions should stay uncorrelated when true targets are uncorrelated—could be turned into an automatic model-selection screen for future retrieval surrogates.","Applying the same normalization-plus-log-target recipe to emission spectra or to retrievals with cloud and haze parameters is untested; those settings have stronger degeneracies, and per-spectrum normalization may behave differently."],"forward_implications":["Trained regressors can turn a JWST/Ariel transmission spectrum into temperature and abundance estimates in milliseconds after a one-time training investment, making population-scale exoplanet surveys affordable.","Per-spectrum normalization plus log-abundance targets is a broadly useful preprocessing recipe for transmission-spectrum retrieval, not a quirk of one algorithm.","The abundance 'knee' near $10^{-6}$--$10^{-7}$ defines a practical detection floor for trace gases in this approach; predictions below it should be reported as upper limits.","Retrieval errors at high temperatures trace directly to sparse training coverage, so a uniformly sampled training database is needed before these models can be trusted for hot planets.","The binned bias-and-variance curves versus predicted values give a practical way to attach uncertainty estimates to individual real-spectrum predictions, where the true parameters are unknown."],"supporting_citations":[{"why":"Provides the TauREx3 forward radiative-transfer model used to generate all synthetic transmission spectra.","marker":"(Al-Refaie et al. 2021)"},{"why":"Describes the ABC dataset construction and the abundance sampling ranges the paper adopts.","marker":"(Changeat & Yip 2023)"},{"why":"Supplies the information-content argument that motivates per-spectrum normalization over standardization.","marker":"(Matchev et al. 2022b)"},{"why":"Provides the XGBoost algorithm used for the best-performing regressor.","marker":"(Chen & Guestrin 2016)"},{"why":"Supplies one of the two JWST WASP-39b spectra used in the real-data case study.","marker":"(Rustamkulov et al. 2023)"},{"why":"Supplies the additional JWST WASP-39b spectrum combined with the Rustamkulov data for the case study.","marker":"(Powell et al. 2024)"},{"why":"Provides the published retrieval ranges for CO2 and CO abundances compared in Table 2.","marker":"(Constantinou et al. 2023)"},{"why":"Provides the published upper-limit comparison for CH4 abundance in Table 2.","marker":"(Ahrer et al. 2023)"},{"why":"Provides the published upper-limit comparison for NH3 abundance in Table 2.","marker":"(Alderson et al. 2023)"},{"why":"Provides the published water-abundance value compared in Table 2.","marker":"(Tsiaras et al. 2018)"}],"fun_headline_variants":["XGBoost and SVM beat other ML for exoplanet spectral retrieval","ML matches Bayesian retrievals on exoplanet spectra, much faster","Normalization boosts XGBoost to top in exoplanet retrieval benchmark","XGBoost and SVM fast and accurate for exoplanet atmospheric parameters"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The paper assumes that a model trained on noiseless, isothermal, cloud-free, hydrogen-helium synthetic spectra with trace abundances sampled independently and uniformly can be applied to a real, noisy, cloudy, non-isothermal JWST spectrum after simple rebinning, and that point predictions outside the training range (e.g., CO and NH3 for WASP-39b) are still physically meaningful.","fun_headline_variants_meta":{"raw":{"variants":["XGBoost and SVM beat other ML for exoplanet spectral retrieval","ML matches Bayesian retrievals on exoplanet spectra, much faster","Normalization boosts XGBoost to top in exoplanet retrieval benchmark","XGBoost and SVM fast and accurate for exoplanet atmospheric parameters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000247,"raw_usage":{"total_tokens":1372,"prompt_tokens":726,"completion_tokens":646,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":470,"completion_tokens_details":{"reasoning_tokens":565}},"tokens_in":470,"tokens_out":646,"duration_ms":6919,"temperature":1.0,"reasoning_tokens":565,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T23:37:41.624531+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same XGB/NL regressor on the ABC-like database but generate test spectra with the forward model including clouds or hazes, non-isothermal temperature profiles, and realistic JWST noise; if retrieval errors grow sharply for the same abundances, or if the WASP-39b predictions leave the published ranges when noise realizations are varied, the central transfer claim fails. A cheaper check: feed the trained model a chemically self-consistent atmosphere (with CO/CH4 equilibrium) and see whether the predicted abundances remain accurate.","supporting_citations":[],"review_version":1}