{"id":"157d8e2a-7400-455f-a63a-68b39ee7ec1a","arxiv_id":"1908.05833","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Weiss and Petigura show that resampling transit signal-to-noise instead of planet radii cannot generate independent radii, and that radius-based resampling rejects detection bias as the source of the Kepler peas-in-a-pod pattern.","lead":"This paper argues that a recent claim, that Kepler's detection bias creates the 'peas in a pod' pattern of similar planet sizes and spacings, is based on a flawed statistical test. The authors show that resampling signal-to-noise instead of planet radii preserves correlations and cannot test whether planet sizes are random.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The >10-sigma rejection rests on an untested choice of intrinsic radius scatter; a narrower log-normal prior could make detection-bias nulls produce strongly correlated radii.","rationale":"The reader's weakest assumption is the log-normal radius prior with sigma=1; this is also the most load-bearing concern about the paper's central claim. The paper's rejection of detection bias is quantitative: the null distribution of Pearson-R is compared to the observed R=0.62, yielding a z-score of about 13.6. That z-score is only meaningful if the null distribution is correct. The null's mean is near zero partly because the sigma=1 prior assigns large intrinsic scatter to planet radii, which dilutes any correlation induced by system-level factors. If the true intrinsic scatter is smaller, the shared stellar and photometric factors (Rstar, CDPP) that enter the detection threshold can make detected radii within a system strongly correlated even when the underlying radii are independent. The paper does not test this sensitivity, so the central conclusion is conditional on an arbitrary prior. Other possible concerns, such as the 1000-trial tail extrapolation behind '>10 sigma,' are secondary: they affect the reported significance level but would not overturn the direction of the result. The proposed concrete test, varying the prior and checking the resulting null distribution, directly targets the load-bearing assumption and would settle whether the claim survives.","tokens_in":8237,"tokens_out":24429,"duration_ms":267995,"concrete_test":"Rerun the §3.1 null resampling with the same redraw-until-SNR>=10 scheme but with log-normal priors sigma_lnR = 0.2, 0.3, 0.5, 1.0, and also with an empirical prior formed by resampling the radii of CKS single-planet systems. For each prior, record the mean and standard deviation of the null Pearson-R distribution and the implied significance of the observed R=0.62. In addition, perform a KS test comparing each synthetic detected radius distribution with the observed CKS multi-planet radius distribution. If the null mean rises above roughly 0.2, or the significance drops below 5 sigma, for any prior that passes the KS test, the headline >10-sigma rejection of detection bias is not robust.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central quantitative claim, stated in the abstract and §6, is that drawing planet radii directly from a log-normal prior and redrawing until SNR>=10 rules out detection bias with >10-sigma confidence. This claim depends entirely on the null distribution of the Pearson-R statistic in §3.1. The null uses ln(Rp/Rsun) ~ N(0,1), i.e., sigma=1, with no sensitivity test. Because the redraw-until-detected procedure conditions each planet's radius on SNR>=10, the detected radius inherits a multiplicative factor proportional to Rstar sqrt(CDPP) (Eqs. 1-3). Within a system this factor is common to all planets, so it can induce a positive correlation among detected radii in the pooled sample. The size of that correlation is controlled by the intrinsic radius scatter: a wide prior (sigma=1) lets true radius variation dominate and keeps the null correlation small, while a narrower prior (sigma~0.2-0.3) makes the common stellar/detection factor dominate, potentially producing strong within-system correlation. The observed R=0.62 may then be much less anomalous than the reported 13.6-sigma offset. The paper does not vary the prior, nor does it compare the synthetic detected radius distribution to the CKS observed distribution, so the 'rule out detection bias' conclusion is not yet robust to a plausible alternative underlying radius distribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript critiques the resampling method of Zhu (2019, Z20), which draws transit SNR at random and converts to planet radius, and argues that this procedure does not produce random, independent planet radii. The authors show algebraically (Eq. 4) that the resampled radius is proportional to the observed radius, so any correlation in the observed radii is partially preserved and the Z20 null test is unsuitable. They then present their own null test: drawing planet radii directly from a log-normal prior (mu=0, sigma=1) and redrawing until SNR>=10, claiming this rules out detection bias as the source of the peas-in-a-pod radius correlation with >10-sigma confidence. The paper also critiques Z20's period-ratio cut, arguing that the product cut induces a dependence between adjacent period ratios, and concludes that the period ratios of adjacent planets are indeed correlated when both are < 4.","tokens_in":8544,"tokens_out":7260,"duration_ms":73143,"significance":"If the central critique of Z20 is correct, the paper corrects a methodological error in a high-profile claim and strengthens the astrophysical interpretation of the peas-in-a-pod pattern. The derivation of Eq. 4 is elegant and the demonstration that Z20's resampling preserves the observed radius is convincing. The paper also makes a useful point about the period-ratio cut. However, the paper's own headline quantitative claim (>10-sigma rejection of detection bias) depends on untested assumptions about the underlying radius distribution, and the SNR-based test in Section 5 appears to use the same flawed logic the authors attribute to Z20. The significance of the paper is therefore real but would be substantially increased by robustness tests and a coherent SNR analysis.","major_comments":[{"comment":"The >10-sigma claim is not robust to the choice of the intrinsic radius distribution. The redraw-until-detected procedure conditions each planet's radius on SNR>=10, and because SNR contains a system-specific factor (Rstar, CDPP6hr, period), the detectable radius floor is common to all planets in a system. With a wide prior (sigma=1), intrinsic radius scatter dominates and keeps the null Pearson-R small, but with a narrower prior (e.g., sigma=0.2-0.3) the common detection floor can induce substantial within-system correlations in the synthetic detected radii. The paper does not vary sigma, does not fit mu and sigma to the observed CKS radius distribution after applying the detection model, and does not compare the synthetic detected radius distribution to the observed one (e.g., with a KS test). As a result, the statement that detection bias is ruled out with >10-sigma confidence is not yet supported; the null distribution of the test statistic is sensitive to the assumed marginal radius distribution.","section":"§3.1, Eq. (1) and null-test description"},{"comment":"The 8-sigma test of adjacent SNR correlation is circular with respect to the paper's own critique of Z20. The authors test whether drawing SNRs at random from the observed SNR distribution can reproduce the observed correlation between adjacent transit SNRs, and find it cannot. But this null destroys any correlation that would arise from the common system properties (Rstar, CDPP) entering the SNR expression (Eq. 1) even if the intrinsic radii are independent. A valid null for 'no astrophysical SNR correlation' should draw radii independently from a chosen prior, apply Eq. 1 with the observed stellar parameters, and compute the resulting SNR correlation. As written, the §5 analysis does not demonstrate that the observed SNR correlation is astrophysical; it only shows that the observed SNR correlation is not consistent with randomly permuted SNRs, which is the same flawed approach used in Z20.","section":"§5, SNR correlation test"},{"comment":"The claim that the observed CKS period ratios are 'inconsistent with random period ratios' is based on comparing the data to a uniform distribution in log period ratio after applying the cut log(Pj)+log(Pj+1)<4. While the demonstration that this cut induces an anti-correlation in a uniform sample is mathematically correct, the uniform distribution is not a realistic null for Kepler's detection and dynamical stability constraints. The actual prior over period ratios is strongly shaped by the requirement of detectability and by stability, so the induced anti-correlation in the uniform null may not correspond to what a realistic random model would predict. To support the conclusion that the observed R=0.15 (rather than the expected -0.5 under the uniform null) is evidence for regularity, the authors should repeat the experiment with a forward model that draws periods from a realistic period distribution and applies the actual detection efficiency, or at least test the sensitivity of the induced correlation to the assumed period prior.","section":"§4, period-ratio analysis"},{"comment":"The paper states that discounting undetected planets rather than redrawing 'did not differ significantly' from the redraw procedure, but no quantitative support is given. Since the choice of redraw-until-detected affects the effective radius distribution and hence the null Pearson-R, the paper should either show the comparison or state that this is a claim from previous work.","section":"§3.1, footnote 4"}],"minor_comments":[{"comment":"The text reports the observed period-ratio correlation after the Z20 cut as R=0.15 with p<10^-5, while the caption of Figure 4 states R=0.15, p=0.04. These values are inconsistent and should be reconciled.","section":"§4 and Figure 4"},{"comment":"The statement that in the 10 systems with four or more planets all smaller than 1.8 R_Earth the correlation 'cannot be explained by detection bias' is presented as an argument but is not a statistical test. This is a reasonable motivating example, but the wording overstates its evidential weight.","section":"§2, first paragraph"},{"comment":"The detection model in Eq. (1) ignores several effects (impact parameter, eccentricity, finite observing window) at the stated level of ~10%. The paper argues these are unimportant because 70% of CKS multis have SNR>20. That argument is reasonable, but a reader may wonder whether the redraw-until-detected procedure is sensitive to the exact threshold behavior at SNR~10; a brief discussion of this sensitivity would strengthen the presentation.","section":"§3.1, Eq. (1)"},{"comment":"The manuscript is a reply to Zhu (2019) and would benefit from explicit statements of the sample size and the exact definition of Pearson-R (e.g., whether radii are logarithmically transformed) used in the null tests, as these details are left implicit.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is a direct reply to Zhu (2019) and is likely to be of interest to the journal's readership. The central methodological point about Eq. (4) is sound, but the paper's own quantitative conclusion (the >10-sigma rejection) and the Section 5 SNR test both need substantial additional work. I would urge the editor to require the authors to address the sensitivity of their null test to the radius prior and to correct the circularity in the SNR analysis before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper is that its central methodological point is solid and quite elegant. Eq. 4, writing the resampled radius as R_obs times sqrt(SNR_new/SNR_obs), makes the flaw in Zhu's procedure obvious: sampling SNR cannot produce independent radii because every new radius is proportional to the observed one. The mock-universe demonstration and the period-ratio dependency analysis reinforce the same point. That part of the paper deserves a careful referee and probably publication.\n\nThe soft spot is the paper's own headline claim. The >10-sigma rejection of detection bias comes from a null test that draws radii from a log-normal with sigma=1 and redraws until SNR>=10. That choice of prior matters more than the authors acknowledge. As the stress-test note explains, conditioning on detection makes the detected radius inherit a common stellar/detection factor within each system. If the intrinsic radius scatter is narrow, that common factor can produce correlated synthetic radii all by itself. The paper does not vary the prior, nor does it check whether its synthetic detected radii reproduce the observed CKS radius distribution. Without those robustness tests, saying they \"rule out detection bias\" is an overstatement; they have shown their chosen null fails, not that every reasonable null fails.\n\nAlso minor: the paper defends the authors' own earlier work, which is fine, but it would be stronger if the code for the resampling were included. And the period-ratio result, while correct as a critique of Zhu's cut, yields a fairly modest observed correlation (R=0.15) once the dependency is removed; that might be worth a level of caution rather than a strong astrophysical claim.\n\nWho is this for? Exoplanet astronomers working on Kepler architectures and anyone citing Zhu (2019). The critique is the contribution; the >10-sigma claim should be recast as conditional on the assumed radius distribution.\n\nRecommendation: send it to peer review, yes. It has a real mathematical insight and identifies a genuine error in a published method. But the referee should push for a sensitivity analysis over the prior width and a comparison of the synthetic detected radii to the observed distribution before accepting the strong conclusion.","headline":"The critique of Zhu's SNR-resampling is correct and worth publishing, but the paper's own >10-sigma null-test claim is hostage to an untested width of the radius prior.","tokens_in":9013,"tokens_out":2381,"would_cite":true,"duration_ms":27139,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Kepler's 'peas in a pod' pattern is real, not a detection artifact.","keywords":["Kepler multi-planet systems","peas in a pod","planet radius correlation","detection bias","bootstrap resampling","transit signal-to-noise","period ratios","planetary system architecture"],"falsifier":"Obtain Kepler's empirically measured detection completeness as a function of SNR and orbital period, use it in place of the SNR $\\ge 10$ step function, and draw radii from a distribution calibrated to the CKS occurrence rates; if the null then yields Pearson $R\\simeq0.62$, the claim that detection bias is ruled out is overturned.","tokens_in":8070,"feed_emoji":"🪐","tokens_out":10714,"duration_ms":87903,"temperature":0.7,"pith_summary":"The paper argues that the Kepler 'peas in a pod' pattern—planets around the same star having similar sizes and regular orbital spacing—is astrophysical. It challenges a recent claim that detection bias produces the pattern, showing that a resampling test based on transit signal-to-noise (SNR) does not generate random, independent planet radii. Because each resampled radius is the observed radius multiplied by $\\sqrt{\\mathrm{SNR}_{\\mathrm{new}}/\\mathrm{SNR}_{\\mathrm{obs}}}$, the resampled radii inherit the original correlation. When the authors instead sample radii directly from a log-normal distribution and apply a simple SNR $\\ge 10$ detection threshold, the null hypothesis of random radii yields Pearson $R=0.023\\pm0.044$, far below the observed $R=0.62$, ruling out detection bias at $>10\\sigma$ confidence.","feed_headline":"Kepler's peas-in-a-pod pattern is real, not a detection artifact","feed_subtitle":"SNR resampling fails; direct radius resampling excludes detection bias at >10 sigma.","key_machinery":"The central object is the radius-SNR identity $R_{p,\\mathrm{new}} = R_{p,\\mathrm{obs}}\\sqrt{\\mathrm{SNR}_{\\mathrm{new}}/\\mathrm{SNR}_{\\mathrm{obs}}}$, derived by rearranging Kepler's SNR equation. This identity is what makes SNR resampling invalid for null-hypothesis testing: the observed radius appears as a multiplicative factor, so the resampled radii are not independent of the observed radii. The paper's alternative machinery is a bootstrap null test that draws planet radii directly from a log-normal distribution with $\\mu=0$ and $\\sigma=1$, redrawing until SNR $\\ge 10$, and compares the Pearson-$R$ correlation of adjacent radii across 1000 trials.","core_discovery":"The central claim is that a bootstrap test which draws transit SNR and converts it to radius cannot test whether the observed correlation in planet radii is a detection artifact. The identity $R_{p,\\mathrm{new}} = R_{p,\\mathrm{obs}}\\sqrt{\\mathrm{SNR}_{\\mathrm{new}}/\\mathrm{SNR}_{\\mathrm{obs}}}$ shows why: the new radius is proportional to the observed radius, so the correlation structure of the observed radii survives resampling. This is demonstrated explicitly by applying the SNR-resampling method to a mock universe in which all planets in a system are identical by construction, and finding that the resampled radii still show a correlation. The paper's own null test resamples radius directly, redrawing until a synthetic planet passes SNR $\\ge 10$, and finds essentially no correlation in 1000 trials per system. It also shows that a period-ratio cut based on the product of adjacent ratios being less than 25 induces an artificial anti-correlation, while the CKS data show positively correlated period ratios in compact systems (period ratio < 4).","pith_inferences":["A concrete extension: re-run the radius-resampling null with Kepler's full measured completeness map instead of a fixed SNR=10 step to test whether the $>10\\sigma$ exclusion survives a more realistic sensitivity function.","The same radius-resampling logic could be applied to planets detected by the TESS mission, where the detection threshold differs, to see whether peas-in-a-pod is a universal feature of close-in multi-planet architectures.","The SNR-resampling artifact is a general caution: any bootstrap that reuses the observed target quantity to generate a synthetic realization will bias the null toward the observed pattern, beyond the planet-radius case.","The significance is tied to the log-normal prior with $\\sigma=1$; if the true underlying radius distribution is much steeper, the exclusion strength could weaken."],"forward_implications":["The observed radius correlation ($R=0.62$) is not reproduced by a random-radius null convolved with Kepler's detection threshold; the null gives $R=0.023\\pm0.044$, so detection bias is excluded at $>10\\sigma$.","Any future test of whether the peas-in-a-pod pattern is biased must resample the parameter of interest (radius or period ratio), not a derived detection statistic such as SNR.","For compact systems with period ratio below 4, adjacent period ratios in CKS multis are correlated, and the apparent randomness argued in the counter-analysis comes from a cut that mathematically forces an anti-correlation.","The astrophysical origin of the pattern reinforces forward-modeling studies in which correlated sizes and regular spacings are needed to reproduce the Kepler multis."],"supporting_citations":[{"why":"Defined the peas-in-a-pod pattern and the original radius-resampling null test that this paper refines.","marker":"W18"},{"why":"The recent SNR-resampling analysis this paper argues is invalid; provides the SNR-to-radius conversion used in the critique.","marker":"Z20"},{"why":"Independent finding of correlated radii and masses in TTV systems, supporting the astrophysical interpretation.","marker":"Millholland et al. (2017)"},{"why":"Defines the CDPP photometric precision metric used in the SNR equations the paper relies on.","marker":"Christiansen et al. (2012)"},{"why":"Introduced the 'redraw until detected' technique for populating synthetic multi-planet systems used in the null test.","marker":"Xie et al. (2016)"}],"fun_headline_variants":["Peas-in-a-pod pattern isn't a Kepler artifact","SNR resampling can't explain Kepler's peas","Kepler's peas are real: detection bias excluded","Astrophysical, not artifact: Kepler's peas","Resampling flaw debunks detection-bias claim"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The $>10\\sigma$ rejection assumes that Kepler's detection is completely described by a fixed SNR $\\ge 10$ cutoff and that the true underlying radius distribution is a log-normal with $\\mu=0$, $\\sigma=1$; if either is far from reality, the null distribution of Pearson $R$ could shift enough to change the significance.","fun_headline_variants_meta":{"raw":{"variants":["Peas-in-a-pod pattern isn't a Kepler artifact","SNR resampling can't explain Kepler's peas","Kepler's peas are real: detection bias excluded","Astrophysical, not artifact: Kepler's peas","Resampling flaw debunks detection-bias claim"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000161,"raw_usage":{"total_tokens":1239,"prompt_tokens":951,"completion_tokens":288,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":211}},"tokens_in":567,"tokens_out":288,"duration_ms":3064,"temperature":1.0,"reasoning_tokens":211,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:03:00.008866+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Obtain Kepler's empirically measured detection completeness as a function of SNR and orbital period, use it in place of the SNR $\\ge 10$ step function, and draw radii from a distribution calibrated to the CKS occurrence rates; if the null then yields Pearson $R\\simeq0.62$, the claim that detection bias is ruled out is overturned.","supporting_citations":[],"review_version":1}