{"id":"b119c1e1-7240-4279-ba38-e3267ba05056","arxiv_id":"2505.08410","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"In a grid of UCLCHEM dark-cloud models, SHAP and UMAP show temperature and density dominate nine molecular ratios, while CN/HCN and HNC/HCN are the most carbon-sensitive probes and CS/SO is the only oxygen-sensitive one.","lead":"This paper uses machine learning to rank which physical conditions control nine molecular line ratios in low-metallicity clouds at the Milky Way's edge. It finds that temperature, density, and the initial carbon abundance matter most, and suggests CN/HCN and HNC/HCN as carbon probes.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The carbon-sensitivity claim rests on XGBoost surrogates whose held-out accuracy is never reported; without surrogate error metrics, the SHAP-based ranking that makes CN/HCN and HNC/HCN 'excellent probes' is unverified.","rationale":"The reader's weakest assumption—unreported surrogate fit quality—is the same point I would put first. My reading of the full text confirms the supporting evidence is entirely SHAP-based, and no accuracy metric for the surrogate is given in Section 2.5 or Table B.1. I also checked whether the 'either molecule detected' filter makes the concern sharper: for HNC/HCN the maximum log-ratio is 0.00 in Table 2, so the filter mainly admits HCN-only models with very negative, unobservable ratios; those are exactly the samples whose SHAP contributions could drive the carbon ranking. This does not change the verdict, because the issues are addressable: reporting nested-CV errors, restricting the central claim to both-detected ratios, and validating on CHEMOUT would test the claim directly. The paper is careful to label conclusions as forward-model statements and to acknowledge the excluded regime, so I would keep the CONDITIONAL verdict rather than reject or accept.","tokens_in":21192,"tokens_out":4613,"duration_ms":47848,"concrete_test":"Retrain the nine XGBoost regressors under nested cross-validation and report held-out R² and RMSE per ratio. Then recompute the mean absolute SHAP importance of f_C for CN/HCN and HNC/HCN after removing every sample in which either numerator or denominator falls below 1e-12, using only the nested-CV-validated models. If held-out R² is below ~0.9 in the sparse high-density/low-temperature region, or if f_C no longer ranks second for both ratios in the both-detected subset, the central 'excellent probes' conclusion is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's conclusion that CN/HCN and HNC/HCN are 'excellent probes' for the initial carbon abundance is a statement about the SHAP importances of the nine XGBoost surrogates described in Section 2.5. The paper never reports the held-out error of any of these surrogates: Table B.1 lists the Optuna-selected hyperparameters, and the text says test error was used as the optimisation target, but no R², RMSE or MAE values appear anywhere. Because the same test split was used for hyperparameter selection, there is also no clean estimate of how the final retrained models generalize. The consequences matter precisely in the regimes the paper itself flags: non-convergent high-density/low-temperature models are excluded, and the 'either molecule detected' filter (Section 3.1) admits ratios in which one species is orders of magnitude below the 1e-12 threshold. If the surrogate predicts poorly in those sparse regions, TreeSHAP values inherit the error, so the f_C ranking for CN/HCN and HNC/HCN could be an artifact rather than a property of the UCLCHEM grid. The 'excellent probes' claim is therefore contingent on surrogate fidelity that is never demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a modeling study of nine molecular abundance ratios in dark-cloud conditions representative of the carbon- and oxygen-poor outer Milky Way. A six-dimensional parameter grid (n_H, T, ζ, F_UV, f_C, f_O) is sampled with 65,536 Sobol points, run with UCLCHEM to 10^7 yr, and ratio values are evaluated at 10^5 yr after applying a 10^-12 abundance detection threshold. For each ratio, an XGBoost regression forest is trained on the surviving grid points and TreeSHAP is used to rank the physical parameters by importance; SHAP vectors and ratio values are then embedded with UMAP to identify regimes of similar chemical behavior. The authors conclude that temperature and density are generally the most important parameters, that CN/HCN and HNC/HCN are sensitive to the initial carbon abundance and are therefore 'excellent probes' for it, that CS/SO is the only ratio with oxygen sensitivity, and that the cosmic-ray ionization rate cannot be constrained by these ratios.","tokens_in":21502,"tokens_out":8107,"duration_ms":78333,"significance":"If the carbon-sensitivity claim holds, the paper offers a practical observational route to test extrapolated metallicity gradients beyond ~16 kpc with CHEMOUT-type observations, which would be a valuable contribution. The pipeline is transparent in several respects: the UCLCHEM grid, Sobol sampling, detection threshold, Optuna hyperparameter search, and the TreeSHAP/UMAP choices are all documented, and the inclusion of nine ratios spanning different chemical families is a useful feature. The main limitation is that all sensitivity statements pass through nine XGBoost surrogates whose held-out accuracy is never reported; until that is fixed, the 'excellent probes' conclusion is not yet supported. With the requested fit metrics and a few clarifying corrections, the paper would be a solid methods-focused contribution to astrochemical interpretation.","major_comments":[{"comment":"The central conclusion that CN/HCN and HNC/HCN are 'excellent probes' for the initial carbon abundance is derived from TreeSHAP values of nine XGBoost surrogates, yet the manuscript never reports any generalization metric (e.g., R², RMSE, or MAE) for these surrogates. Table B.1 lists only the Optuna-selected hyperparameters, and Section 2.5 states that the test error was used as the optimisation target, so the same 30% split is not an independent test set for the final retrained models. Because TreeSHAP explanations inherit the errors of the fitted model, the sensitivity rankings in Figure 4 and the abstract's conclusion are unverified until held-out fit metrics are provided. Please report per-ratio fit statistics and, ideally, the fit quality in the sparse high-density/low-temperature and single-species-detection regimes.","section":"§2.5, Table B.1"},{"comment":"The dataset used for SHAP includes samples in which only one of the two molecules is above the 10^-12 threshold. As shown by the minima in Table 2 (e.g., log10(HCO+/HCN) = -23.14), many ratios are then extreme values that do not correspond to directly observable line ratios. These unobservable points can dominate the training distribution and may drive the SHAP attribution of f_C for CN/HCN and HNC/HCN. The paper should quantify how much of the carbon sensitivity is present in the both-detected subset alone and should report surrogate accuracy on that subset; otherwise the 'excellent probe' claim is not established for the regime in which the ratios can actually be observed.","section":"§3.1, Figure 2"},{"comment":"The abstract states that 'only CS/SO shows a sensitivity to the oxygen abundance,' but Figure 4 gives normalized f_O importances of 0.15 for HNC/HCN and CS/CN and 0.14 for CS/SO, and Section 4 item 7 states that 'CS/CN' has the largest oxygen dependence. These statements are mutually inconsistent. Please correct the summary and clarify whether the criterion is importance magnitude, monotonic direction, or something else, and reconcile the abstract with the quantitative importances.","section":"Abstract vs §4 item 7 and Figure 4"},{"comment":"The models vary only the initial carbon and oxygen abundances while keeping nitrogen, sulfur, and all other heavy elements at their solar values (Table A.1). The paper frames the study as representative of low-metallicity outer-Galaxy gas; if the real low-metallicity gas is depleted in N and S as well, the CN/HCN and CS/SO chemistries could behave differently. Please justify this assumption or test the sensitivity of the main conclusions to scaling the other heavy-element abundances down with metallicity.","section":"§2.1, Table A.1"}],"minor_comments":[{"comment":"The symbols and names in the last two rows appear interchanged: the row labeled 'Initial abundance of carbon' uses f_O/f_O,⊙, and the row labeled 'Initial abundance of oxygen' uses f_C/f_C,⊙. Please correct the symbols so that f_C is associated with the carbon abundance and f_O with oxygen.","section":"Table 1"},{"comment":"The text says 'our dataset is sampled on a regular grid,' but Section 2.1 describes Sobol sequence sampling, which is a low-discrepancy quasi-random sampling instead of a regular grid; the wording should be changed to avoid this inconsistency.","section":"§2.4"},{"comment":"The manuscript repeatedly refers to 'molecular line ratios,' but only abundance ratios are computed from UCLCHEM; no radiative transfer or excitation calculation is performed. The assumed proportionality between abundance ratios and observable line ratios should be stated explicitly as a caveat.","section":"§2.1, throughout"},{"comment":"The UMAP hyperparameters (k, d_min, w_ratio) were chosen by manual tuning with the stated goal of obtaining a 'smooth manifold'; a brief robustness check, such as varying k and d_min and describing how the grouping changes, would strengthen the interpretation of the UMAP-based claims.","section":"§2.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable application of interpretable ML to astrochemical grids, but the abstract's main claim about CN/HCN and HNC/HCN being 'excellent probes' rests entirely on SHAP values from surrogates whose held-out accuracy is never demonstrated. I recommend major revision rather than rejection because the missing metrics and the requested validation checks are within the scope of the paper and could be added without changing the modeling framework. The internal inconsistency about oxygen sensitivity should also be fixed before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on arXiv:2505.08410. It is a well-executed forward-model study: a 65k-model UCLCHEM grid over a relevant low-metallicity outer-Galaxy parameter space, interpreted with the authors' established XGBoost+SHAP+UMAP pipeline. The physical result—CN/HCN and HNC/HCN are sensitive to the initial carbon abundance, temperature, and density—is plausible and consistent with older literature (e.g., Milam et al. 2005). What the paper does well: the grid design is thoughtful (Sobol sampling, wide ranges, independent C and O depletion), the classical temperature-density maps (Fig. 3) are genuinely informative, and the paper is transparent about what it cannot constrain (cosmic rays, because T is fixed). It also honestly notes the non-convergent high-density/low-temperature region. As a mapping of ratio behavior in a previously under-explored parameter space, this is a useful contribution.\n\nThe soft spots are real but not fatal. The main one: the SHAP importances that drive every sensitivity ranking come from XGBoost surrogates, and the paper never reports the held-out accuracy of those surrogates. The text says test error was the Optuna target, but no R², RMSE, or MAE appears in Table B.1 or anywhere else. Given that the same test split was used for hyperparameter selection, there is no clean estimate of generalization, and the problem is most acute in sparse regions—precisely where the \"either molecule detected\" filter produces extreme ratios (one species orders of magnitude below the 1e-12 threshold). If the surrogate is inaccurate there, the SHAP rankings could be artifacts. I don't think they are—the rankings line up with physical expectations—but the claim \"excellent probes\" needs that metric to be credible.\n\nSecond, the \"either molecule\" filter is a deliberate choice, but the headline claim should be restricted to the both-detected subset (or at least checked there). Many of the extreme ratios in Table 2 are not directly observable. Third, the abstract's \"excellent probes\" is a forward-model statement; there is no test against observations (e.g., CHEMOUT sources). That's acceptable for a model characterization paper, but the language should not imply empirical validation.\n\nOverall: this is a solid, incremental paper for astrochemistry readers who want a ratio sensitivity map for low-metallicity clouds. It does not need a desk rejection. It needs a referee who asks for (1) surrogate fit metrics on a fresh holdout, (2) sensitivity of the SHAP ranking to the detection filter, and (3) a softened or qualified headline claim. With those changes, it would be publishable. I'd send it to review, and I'd cite it in my own work if I were mapping out carbon probes in the outer Galaxy.","headline":"A clean, useful parameter survey showing CN/HCN and HNC/HCN respond to carbon abundance in low-metallicity clouds, but the 'excellent probes' claim outruns what the unreported surrogate fit metrics can support.","tokens_in":22022,"tokens_out":3685,"would_cite":true,"duration_ms":36095,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The CN/HCN and HNC/HCN molecular ratios respond mainly to the initial carbon abundance, making them practical probes of metallicity in the carbon- and oxygen-poor outer Milky Way.","keywords":["astrochemical modeling","molecular line ratios","outer Milky Way","low metallicity","initial carbon abundance","SHAP","UMAP","gas-grain chemistry"],"falsifier":"A concrete check: report the held-out test error or $R^2$ of each of the nine regression forests and recompute the SHAP rankings after resampling or densifying the high-density, low-temperature part of the grid; the paper's carbon-abundance ranking for CN/HCN and HNC/HCN should survive. Observationally, measure CN/HCN and HNC/HCN in outer-Galaxy clouds beyond $16$ kpc where the metallicity gradient is only extrapolated: the models predict these ratios should shift with the initial carbon abundance in a specific direction.","tokens_in":1756,"feed_emoji":"🧪","tokens_out":3696,"duration_ms":117416,"temperature":0.7,"pith_summary":"This paper asks which physical conditions can be read off from molecular line ratios in the carbon- and oxygen-poor outer Milky Way. Using roughly 65,000 gas-grain models with varying density, temperature, cosmic-ray ionisation, UV field, and initial carbon and oxygen abundances, it trains a boosted regression forest for each of nine ratios and explains the forest with SHAP values. The paper claims that temperature and density dominate most ratios, but that CN/HCN and HNC/HCN respond strongly to the initial carbon abundance, making them useful observational probes of that parameter. If true, measuring these two ratios in outer-Galaxy clouds beyond about 16 kpc could test whether the extrapolated radial metallicity gradient is correct. The paper also introduces UMAP embeddings of the SHAP vectors as a way to see higher-order parameter dependencies that classical two-dimensional plots miss.","feed_headline":"Two molecular ratios reveal carbon scarcity at the Milky Way's edge","feed_subtitle":"Tracking CN/HCN and HNC/HCN in outer-Galaxy clouds could test the metallicity gradient beyond 16 kpc.","key_machinery":"The load-bearing mechanism is the pairing of a large gas-grain chemistry grid with an interpretable surrogate. UCLCHEM supplies time-dependent abundances for each parameter set sampled with Sobol sequences; for each of the nine ratios an XGBoost regression forest is trained on the physical parameters, and TreeSHAP decomposes every prediction into six additive per-feature contributions that sum to the predicted ratio. Those six-dimensional SHAP vectors, with the ratio itself attached, are then embedded with UMAP into two dimensions so that regions of parameter space with similar chemistry appear as clustered regions on a manifold. The SHAP decomposition is what lets the paper claim that carbon abundance, not just temperature or density, drives CN/HCN and HNC/HCN; UMAP is what lets it see where in parameter space that carbon sensitivity lives.","core_discovery":"On the paper's own terms, the discovery is a ranking: among the nine ratios, CN/HCN and HNC/HCN are the two whose behaviour across the model grid is most controlled by the initial carbon abundance after density, with carbon ahead of temperature as the second most important feature. Only CS/SO among the nine shows measurable sensitivity to the initial oxygen abundance. The models deplete carbon and oxygen independently down to one-twentieth of their solar values, in cold clouds of $10^3$ to $10^7$ cm$^{-3}$ and $10$ to $100$ K, take ratios at $10^5$ years, and discard any ratio whose molecule falls below an abundance of $10^{-12}$. From that grid the paper argues that temperature and density dominate most ratios, that the cosmic-ray ionisation rate cannot be constrained by these ratios in the studied range, and that the carbon-sensitive pair can therefore serve as an observational probe of the carbon abundance, and with it the metallicity gradient, in the outer Galaxy.","pith_inferences":["Beyond the paper: because CN/HCN and HNC/HCN are routinely observed in starbursts and active galaxies, the carbon-abundance sensitivity found here suggests those extragalactic systems could in principle be probed for their carbon-to-oxygen abundance with the same two ratios, not just outer-Galaxy clouds.","Beyond the paper: the low importance assigned to the cosmic-ray ionisation rate may be partly inherited from the model setup, in which the temperature is fixed and therefore decoupled from cosmic-ray heating; a coupled thermal-chemical grid could change the SHAP ranking.","Beyond the paper: a direct observational test would measure CN/HCN and HNC/HCN in clouds with independently known metallicities beyond $16$ kpc; the paper's models predict a systematic shift with the initial carbon abundance that an abundance-blind calibration could falsify."],"forward_implications":["If the carbon-abundance ranking is right, CN/HCN and HNC/HCN become practical metallicity probes: observers can test the extrapolated radial metallicity gradient beyond $16$ kpc by measuring these two ratios in outer-Galaxy clouds.","Because temperature and density dominate most of the nine ratios, observational use of the carbon-sensitive pair must first constrain the local temperature and density, or the abundance signature will be masked.","The cosmic-ray ionisation rate cannot be pinned down by these nine ratios over the studied range of $\\zeta = 10^{-17}$ to $10^{-14}$ s$^{-1}$, so constraining it will require other tracers or a grid with coupled thermal balance.","The SHAP-plus-UMAP workflow identifies which parameter subspace each ratio responds to, which can be used to set better priors in Bayesian backwards modelling of observed cores."],"supporting_citations":[{"why":"Supplies the UCLCHEM gas-grain model that generates the entire chemistry grid.","marker":"Holdship et al. 2017"},{"why":"Provides SHAP, the game-theoretic attribution method used to rank parameter importance.","marker":"Lundberg & Lee 2017"},{"why":"Supplies TreeSHAP, the tree-specific efficient SHAP algorithm applied to the boosted forests.","marker":"Lundberg et al. 2020"},{"why":"Provides UMAP, the manifold embedding used to group models by their SHAP vectors.","marker":"McInnes et al. 2020"},{"why":"Supplies XGBoost, the boosted regression forest whose predictions TreeSHAP explains.","marker":"Chen & Guestrin 2016"},{"why":"Provides the low-discrepancy sequence sampling that makes the six-dimensional parameter grid tractable.","marker":"Sobol' 1967"},{"why":"Sets the observed outer-Galaxy CHEMOUT context and the depletion ranges for carbon and oxygen abundances.","marker":"Fontani et al. 2024"},{"why":"Establishes CN/HCN as an abundance-sensitive ratio, which the paper's SHAP analysis reinforces for the carbon abundance.","marker":"Milam et al. 2005"},{"why":"Supplies the H + HNC to HCN + H isomerisation pathway that explains why HNC/HCN is not simply a temperature tracer in these models.","marker":"Hacar et al. 2020"}],"fun_headline_variants":["CN/HCN and HNC/HCN track carbon in outer Galaxy","Two ratios probe carbon in the outer Milky Way","Machine learning identifies carbon-sensitive ratios in outer Galaxy","CS/SO traces oxygen, CN/HCN traces carbon in outer Galaxy","Carbon abundance traced by CN/HCN and HNC/HCN ratios"],"cache_read_input_tokens":24064,"weakest_assumption_plain":"The entire sensitivity ranking depends on the boosted forest accurately reproducing the chemistry grid in every region of parameter space, including sparse high-density, low-temperature regions where some UCLCHEM models did not converge; if the surrogate is wrong there, the SHAP attributions could assign ratio changes to the wrong physical parameter.","fun_headline_variants_meta":{"raw":{"variants":["CN/HCN and HNC/HCN track carbon in outer Galaxy","Two ratios probe carbon in the outer Milky Way","Machine learning identifies carbon-sensitive ratios in outer Galaxy","CS/SO traces oxygen, CN/HCN traces carbon in outer Galaxy","Carbon abundance traced by CN/HCN and HNC/HCN ratios"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001111,"raw_usage":{"total_tokens":4690,"prompt_tokens":1065,"completion_tokens":3625,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":681,"completion_tokens_details":{"reasoning_tokens":3541}},"tokens_in":681,"tokens_out":3625,"duration_ms":27427,"temperature":1.0,"reasoning_tokens":3541,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:55:23.008264+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: report the held-out test error or $R^2$ of each of the nine regression forests and recompute the SHAP rankings after resampling or densifying the high-density, low-temperature part of the grid; the paper's carbon-abundance ranking for CN/HCN and HNC/HCN should survive. Observationally, measure CN/HCN and HNC/HCN in outer-Galaxy clouds beyond $16$ kpc where the metallicity gradient is only extrapolated: the models predict these ratios should shift with the initial carbon abundance in a specific direction.","supporting_citations":[{"cited_title":"2017, The Astronomical Journal, 154, 38","cited_arxiv_id":null,"evidence_quote":"Supplies the UCLCHEM gas-grain model that generates the entire chemistry grid."},{"cited_title":"& Lee, S.-I","cited_arxiv_id":null,"evidence_quote":"Provides SHAP, the game-theoretic attribution method used to rank parameter importance."},{"cited_title":"2020, UMAP : Uniform Manifold Approximation and Projection for Dimension Reduction","cited_arxiv_id":null,"evidence_quote":"Provides UMAP, the manifold embedding used to group models by their SHAP vectors."},{"cited_title":"& Guestrin, C","cited_arxiv_id":null,"evidence_quote":"Supplies XGBoost, the boosted regression forest whose predictions TreeSHAP explains."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the low-discrepancy sequence sampling that makes the six-dimensional parameter grid tractable."},{"cited_title":"2024, A&A, 691, A180","cited_arxiv_id":null,"evidence_quote":"Sets the observed outer-Galaxy CHEMOUT context and the depletion ranges for carbon and oxygen abundances."},{"cited_title":"D., & Van Dishoeck, E","cited_arxiv_id":null,"evidence_quote":"Supplies the H + HNC to HCN + H isomerisation pathway that explains why HNC/HCN is not simply a temperature tracer in these models."}],"review_version":1}