{"id":"f22bcb3f-66b0-43e5-b1dd-c2d137a35c23","arxiv_id":"1908.09077","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A pilot design that fits a prognostic model on a held-out control subset and matches on propensity and prognostic scores improves MSE and sensitivity-power in large observational samples.","lead":"Researchers propose using part of an observational dataset as a 'pilot set' to learn what predicts outcomes in untreated people, then exclude that set from the final analysis. Matching on this prognostic information plus treatment propensity reduces estimation error and makes results harder to explain away by hidden bias in simulations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pilot selection in Algorithm 1 may violate the 'appropriate pilot' condition of Theorem 2 under prognostic model misspecification, leaving the central claim untested in a key regime.","rationale":"I examined the argument for the central claim. The claimed 12–36% MSE reduction and doubled Γ hinge on the pilot-fitted prognostic model providing scores that improve matching in the analysis set. The weakest point is the pilot set selection. Algorithm 1 aims to keep the pilot near the treated group, but this creates a distribution shift between pilot and analysis controls. Under correct specification this shift is benign; under misspecification it biases the pseudo-true prognostic score toward the pilot distribution. Theorem 2 delegates this to an unverified 'appropriate' condition, and the simulations never misspecify the functional form, so the paper's central claim is not established in the regime where double robustness is most needed. The factor-of-2 error in Theorem 1 (Var(τ̂) should be 2σ²/n_T, not 4σ²/n_T) is real but not load-bearing: it is an illustrative moment bound, and the simulation comparisons do not depend on that constant. The reader's conditional verdict is appropriate: the method is plausible and well-supported in linear correctly-specified settings, but requires either a proof that Algorithm 1's pilot selection is 'appropriate' or simulations covering misspecified prognostic models before the claim can be accepted without qualification. My concern aligns with the reader's weakest assumption, so I do not change the verdict.","tokens_in":23581,"tokens_out":7950,"duration_ms":87082,"concrete_test":"Run the Section 4.2 simulation with the outcome model changed to Y_i = τ T_i + Ψ(X_i) + X_2^2 + ε_i (or another nonlinearity), while Algorithm 1 still fits a linear prognostic model, for n=2000, n_T≈100, ρ∈{0,0.5,0.9}, and matching ratios k∈{1,5}. Compare MSE and median Γ for (a) Algorithm 1 pilot selection, (b) a random split of the same pilot size, and (c) propensity score matching. If Algorithm 1's MSE or median Γ worsens relative to (b) or loses its advantage over (c) substantially, the pilot-selection assumption is the load-bearing gap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the prognostic model fitted on the pilot set yield accurate prognostic scores for the analysis set. Algorithm 1 constructs the pilot set by 1:2 Mahalanobis matching of each treated unit to two controls and randomly selecting one control per pair, so the pilot covariate distribution is a function of the treated distribution and the Mahalanobis metric, not a random sample of the analysis-set control distribution. When the prognostic model is correctly specified, selection on covariates is harmless. But when it is misspecified, the fitted model converges to the pseudo-true parameter weighted by the pilot distribution; that approximation need not be accurate for analysis-set units, especially controls far from the treated group that may enter matches for larger k or poor overlap. Theorem 2 explicitly conditions on the pilot selection being 'appropriate for consistent estimation of θ̃' (Section 3.4.2) but does not prove Algorithm 1 satisfies this. The simulations in Section 4.2 generate outcomes from linear models and fit linear (or lasso) prognostic models, so this misspecification regime is not exercised. The paper's own Section 5.3.3 acknowledges match-quality trade-offs but does not test misspecified functional form.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a 'pilot design' for observational studies: before the analysis phase, a subset of observations is set aside as a pilot set, outcomes in the pilot set are used to fit a prognostic model, and the estimated propensity and prognostic scores are then used jointly to match units in the remaining analysis set. The authors introduce Assignment-Control (AC) plots as a diagnostic, state two theorems about the behavior of matching estimators in this setting, and present simulations comparing prognostic pilot matching with propensity score matching and Mahalanobis distance matching. The central empirical claim is that pilot matching reduces mean squared error of the estimated treatment effect and substantially increases the median Gamma value in Rosenbaum sensitivity analyses relative to standard matching. The paper also argues that pilot designs are broadly useful for design-phase decisions beyond prognostic score construction.","tokens_in":23805,"tokens_out":6865,"duration_ms":70566,"significance":"If the claims hold, the paper makes a useful practical contribution to the growing literature on design-phase decisions in observational studies. The simulation study is extensive, covers several important robustness scenarios, and the authors provide publicly available code, which is a strength. The AC plot is a genuinely useful visualization concept. However, the theoretical support is not fully solid: Theorem 1 contains an incorrect variance formula, and Theorem 2 relies on an unverified condition about the pilot set. The empirical claims are also confined to correctly specified or over-specified linear models, so the key regime of functional-form misspecification is not tested. These issues are fixable but need attention before the paper can be accepted.","major_comments":[{"comment":"The variance formula in Eq. (3) is incorrect by a factor of 2. Under the stated assumptions, each pair difference is D_i = tau + (Psi(X_i) - Psi(X_{j(i)})) + (epsilon_i - epsilon_{j(i)}), so conditional on the matched pair, Var(D_i) = Var(Psi(X_i)-Psi(X_{j(i)})) + 2 sigma^2, not + 4 sigma^2. Equation (3) should therefore read Var(tau_hat) = [Var(Psi(X_i)-Psi(X_{j(i)})) + 2 sigma^2] / n_T, and Eqs. (5) and (6) should be adjusted accordingly. The qualitative message that variance and MSE depend on the mean and variance of prognostic score differences remains, but the theorem as printed is mathematically wrong and should be corrected.","section":"Section 3.4.1, Eq. (3)"},{"comment":"Theorem 2 conditions on the pilot selection being 'appropriate for consistent estimation of theta-tilde', but the paper does not prove that Algorithm 1 satisfies this condition, nor does it give a verifiable sufficient condition. Algorithm 1 selects pilot controls by 1:2 Mahalanobis matching to treated units, so the pilot covariate distribution is a function of the treated distribution and the Mahalanobis metric, rather than a random sample from the analysis-set control distribution. If the prognostic model is misspecified, the fitted coefficients converge to a selection-weighted pseudo-true parameter that need not provide accurate prognostic scores for analysis-set units. This is precisely the regime in which the paper's double-robustness claim is invoked. Please either prove the condition for Algorithm 1 under the theorem's maintained assumptions, or state a weaker theorem with an explicit assumption that can be checked, and qualify the repeated 'doubly robust' language in Sections 5 and 6 accordingly.","section":"Section 3.4.2, Theorem 2"},{"comment":"The simulation study does not exercise a misspecified functional form for the prognostic or propensity model. The main simulations and all robustness checks use linear generating models with linear or lasso fits, so the fits are either correctly specified or over-specified. Since the pilot-set selection concern is fundamentally a misspecification phenomenon, the central claims that pilot matching reduces MSE and increases median Gamma are untested in the regime where the fitted prognostic model is not a correct parametric description of the analysis-set outcomes. Please add simulations with a nonlinear prognostic score (for example, including a quadratic or interaction term that the pilot model omits), or explicitly temper the conclusions in Sections 5 and 6 to the correctly specified or over-specified case.","section":"Section 4.2 and Section 5.3.4"}],"minor_comments":[{"comment":"Algorithm 1 says 'Fit a linear propensity model' and 'Fit a linear prognostic model', but the simulations and Supplementary Figure 1 also use lasso fits; please clarify that the algorithm is a template and that the model family is a design choice made separately in each application.","section":"Section 3.3, Algorithm 1"},{"comment":"The displayed rate expression in Eq. (9) is garbled in the current text; it should be written with explicit parentheses, for example tau_hat - tau_hat(theta_hat) = O_p(n_analysis^{-1/2}) + o_p(R(n_pilot)), and the notation tau vs. tau_hat should be defined consistently.","section":"Section 3.4.2, Eq. (9)"},{"comment":"The text reports median Gamma values of approximately 5 for pilot matching and 2.5 for propensity score matching without specifying the exact value of k and rho at which these numbers are read; please state the simulation cell used for this comparison.","section":"Section 5.1, Figure 3B"},{"comment":"There are several small typographical issues: 'King and Nielson' should be 'King and Nielsen', the R package names 'sensitivtymv' and 'sensitivtyfull' appear to be misspelled, and the square-root symbol in '1√n' is not rendered properly.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The factor-of-2 error in Theorem 1 is concrete and should be fixed; the unverified pilot-selection condition in Theorem 2 is the main scientific risk. The authors' own Section 5.3.3 acknowledges match-quality trade-offs, so a simulation with misspecified functional form would substantially strengthen the revision. There is no concern about novelty disclosure or inappropriate citation practice beyond the usual self-citation to the authors' stratamatch package."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on matching or design sensitivity. The pilot design idea is genuinely useful: hold out a pilot set, fit a prognostic model there, then match on propensity plus prognosis. This is a real, actionable extension of Leacy and Stuart and Antonelli et al., and the AC plots are a nice visualization. The simulations are extensive, with public code, and they show convincing MSE and gamma improvements in the linear settings they study. The authors are also honest about trade-offs in sample size, match quality, and overlap, which is rarer than it should be.\n\nNow the soft spots. Theorem 1 has a factor-of-two mistake in the variance formula. For matched pairs, D_i = tau + (Psi_i - Psi_j) + (epsilon_i - epsilon_j), so the variance of the difference is Var(Psi diff) + 2 sigma^2, not 4 sigma^2. The displayed formula should have 2 sigma^2/n_T, not 4 sigma^2/n_T. This does not sink the paper—the qualitative message that prognostic score variation drives variance and MSE still stands—but it is a real error in a stated theorem and should be corrected.\n\nMore importantly, Theorem 2's double-robustness claim is a near-immediate generalization of Antonelli et al., with proof deferred to the supplement, and it conditions on the pilot selection being 'appropriate for consistent estimation of theta-tilde.' The paper does not prove Algorithm 1 satisfies that condition. The stress-test note is right: the pilot set is not a random sample of controls; it is selected by 1:2 Mahalanobis matching to treated units, so under prognostic model misspecification the fitted model targets a pilot-weighted pseudo-true parameter, not necessarily the analysis-set quantity. The simulations only use linear or lasso models on linear DGPs, so this misspecification regime is never exercised. This is a genuine gap, but it is a gap in theoretical support rather than a contradiction in the empirical results. The authors seem aware that harder prognostic models hurt performance (Section 5.3.4), and they frame Theorem 2 as motivating rather than as the entire basis for the method.\n\nIf I were refereeing, I would ask for a corrected Theorem 1, a more careful statement of what 'appropriate pilot selection' requires, and at least one misspecification simulation with a nonlinear outcome. None of these requires a overhaul. The core idea is sound, the simulations are reproducible, and the paper is worth engaging in peer review rather than desk rejection.\n\nBring it to reading group if you discuss matching or design sensitivity.","headline":"A practical, well-simulated pilot-matching extension of prognostic-score design, with a small theorem bug and a genuine gap between Algorithm 1 and Theorem 2.","tokens_in":24341,"tokens_out":2471,"would_cite":true,"duration_ms":27775,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62D20"],"pacs":[],"model":"deepseek-v4-flash","headline":"A pilot design that spends some observations before the analysis—fitting a prognostic model on a held-out set and then matching on both propensity and prognostic scores—reduces estimation error and roughly doubles the…","keywords":["causal inference","observational studies","pilot design","prognostic score","propensity score","matching","sensitivity analysis","Assignment-Control plots"],"falsifier":"Re-run the main simulation (n=2000, rho=0.5) with the pilot set selected uniformly at random from the control pool instead of by 1:2 Mahalanobis matching: if the MSE reduction and the Gamma advantage over propensity matching persist, the benefit comes from joint-score matching itself; if they vanish, the pilot allocation scheme is the load-bearing component.","tokens_in":23431,"feed_emoji":"📊","tokens_out":7639,"duration_ms":72742,"temperature":0.7,"pith_summary":"The paper argues that in observational studies with many control units but limited high-quality controls, the best use of some observations is not estimation but design: hold out a pilot set, fit a prognostic model on it, and match treated units to remaining controls jointly on propensity and prognostic scores. In simulations this \"prognostic pilot matching\" reduced mean squared error by 12% to 36% relative to propensity-score matching and 8% to 34% relative to Mahalanobis distance matching at 1:1, and raised the median Gamma at which the effect would be explained away from about 2.5 to about 5. The reason is that matching on prognosis shrinks within-set heterogeneity in the control potential outcome, which improves precision and, uniquely for observational studies, strengthens sensitivity analyses. A reader should care because data-rich but control-poor studies are increasingly common, and this reframes sample size as a design resource rather than only an analysis resource.","feed_headline":"Pilot design cuts error and makes findings harder to overturn","feed_subtitle":"Holding out a pilot control set to fit a prognostic model cuts MSE up to 36% and roughly doubles robustness to unobserved bias.","key_machinery":"The central object is the prognostic score, defined as any function of the covariates such that the control potential outcome is independent of the covariates once that score is known; the paper's version models the expected outcome under no treatment. The proposed machine is Algorithm 1: fit a propensity model on the full data, construct a 1:2 Mahalanobis match of each treated unit to two controls, keep one control from each pair uniformly at random as the pilot set, fit a linear prognostic model on the pilot outcomes, then Mahalanobis-match the remaining analysis set on the estimated propensity and prognostic scores. Supporting this are two theorems: Theorem 1 writes the bias, variance, and mean squared error of any pair-matching estimator in terms of the mean and variance of within-pair prognostic-score differences, and Theorem 2 gives doubly robust consistency—estimation remains consistent as long as at least one of the two score models is correctly specified. The paper also introduces Assignment-Control (AC) plots, which display treated and control units in the two-dimensional space of propensity and prognostic scores to show what each matching strategy optimizes.","core_discovery":"The central claim is that deliberately removing a subset of controls before estimation—using them only to build a prognostic model, then matching the remaining treated and control units jointly on estimated propensity and prognostic scores—can produce a better study than using every observation in the analysis phase. In the paper's simulations, 1:1 pilot matching lowered mean squared error by 12% to 36% compared with propensity score matching and 8% to 34% compared with Mahalanobis distance matching, depending on the correlation between treatment-assignment and outcome variation. The same designs raised the median Gamma in a Rosenbaum-style sensitivity analysis from about 2.5 to about 5, meaning an unobserved confounder would need to be roughly twice as strong to explain away the result. The mechanism is that prognostic matching reduces within-pair differences in the control potential outcome, and lower unit heterogeneity in matched sets directly boosts design sensitivity.","pith_inferences":["The pilot-design principle should generalize beyond matching—the same held-out-split logic could inform outcome selection, instrumental-variable characterization, or analysis-plan prespecification—but the paper demonstrates quantitative gains only for matching.","A testable extension: in settings with a large control reserve, a cross-validated or repeated-split version of the pilot design could estimate the population average treatment effect rather than the sample average treatment effect among the treated, and might recover some of the sample-size cost the authors document.","The Gamma gain is a statement about design sensitivity, not bias: as the paper's own Figure 4 shows, all methods remain biased when a confounder is actually present, so the result should be read as robustness to explaining-away rather than protection from confounding.","If treatment assignment is hard to model but the control outcome is comparatively easy to model, the paper's framework suggests the prognostic score will carry most of the adjustment burden; the converse should hold when prognosis is the harder model."],"forward_implications":["In data-rich settings, moving a fraction of controls from the analysis set into a pilot set can lower estimator mean squared error even though the analysis sample shrinks.","The gains are largest when propensity and prognosis are weakly correlated, because then propensity matching alone leaves within-pair prognostic differences large.","Matching on the prognostic score raises the Gamma value in sensitivity analyses, so an unobserved confounder must be roughly twice as strong to explain away the pilot-matching result.","The estimator is doubly robust: misspecifying one of the two score models does not by itself destroy consistency as long as the other is correctly specified.","Full matching on both scores uses every control in the analysis set exactly once and still slightly outperforms propensity-based full matching in the simulations."],"supporting_citations":[{"why":"Defines the prognostic score and argues for matching jointly on propensity and prognostic scores.","marker":"[8]"},{"why":"Supplies the doubly robust matching estimator result that Theorem 2 generalizes.","marker":"[10]"},{"why":"Provides simulation evidence that joint propensity-prognostic matching performs well even under model misspecification.","marker":"[9]"},{"why":"Establishes the Gamma sensitivity analysis used to quantify robustness to unobserved confounding.","marker":"[2]"},{"why":"Demonstrates that reducing unit heterogeneity in matched sets increases design sensitivity.","marker":"[6]"},{"why":"Argues propensity score matching can be variance-increasing, motivating the heterogeneity-reduction approach.","marker":"[4]"},{"why":"Provides large-sample properties for matching estimators on which Theorem 2's rate statement relies.","marker":"[7]"},{"why":"Shows that same-sample prognostic stratification biased treatment effect estimates, motivating the held-out pilot.","marker":"[12]"}],"fun_headline_variants":["Pilot design cuts error up to 36% and doubles bias tolerance","Prognostic pilot design lowers MSE and doubles design sensitivity","Hold out a pilot set to cut error and double bias robustness","Using some data for design cuts error and boosts sensitivity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole procedure assumes the pilot set is representative enough that the prognostic model fitted on it remains accurate for the controls left in the analysis set; if the pilot controls are atypical or the model is misspecified, the computed scores are off and the claimed gains in error and robustness can shrink or reverse.","fun_headline_variants_meta":{"raw":{"variants":["Pilot design cuts error up to 36% and doubles bias tolerance","Prognostic pilot design lowers MSE and doubles design sensitivity","Hold out a pilot set to cut error and double bias robustness","Using some data for design cuts error and boosts sensitivity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001251,"raw_usage":{"total_tokens":5143,"prompt_tokens":971,"completion_tokens":4172,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":4102}},"tokens_in":587,"tokens_out":4172,"duration_ms":27885,"temperature":1.0,"reasoning_tokens":4102,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:23:11.096669+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the main simulation (n=2000, rho=0.5) with the pilot set selected uniformly at random from the control pool instead of by 1:2 Mahalanobis matching: if the MSE reduction and the Gamma advantage over propensity matching persist, the benefit comes from joint-score matching itself; if they vanish, the pilot allocation scheme is the load-bearing component.","supporting_citations":[{"cited_title":"The prognostic analogue of the propensity score.Biometrika 2008; 95(2): 481–488","cited_arxiv_id":null,"evidence_quote":"Defines the prognostic score and argues for matching jointly on propensity and prognostic scores."},{"cited_title":"Doubly robust matching estimators for high dimensional confounding adjustment.Biometrics2018; 74(4): 1171–1179","cited_arxiv_id":null,"evidence_quote":"Supplies the doubly robust matching estimator result that Theorem 2 generalizes."},{"cited_title":"On the joint use of propensity and prognostic scores in estimation of the average treatment effect on the treated: a simulation study.Statistics in medicine2014; 33(20): 3488–3508","cited_arxiv_id":null,"evidence_quote":"Provides simulation evidence that joint propensity-prognostic matching performs well even under model misspecification."},{"cited_title":"Sensitivity analysis in observational studies.Encyclopedia of statistics in behavioral science2005; 4: 1809–1814","cited_arxiv_id":null,"evidence_quote":"Establishes the Gamma sensitivity analysis used to quantify robustness to unobserved confounding."},{"cited_title":"Heterogeneity and causality: Unit heterogeneity and design sensitivity in observational studies.The American Statistician2005; 59(2): 147–152","cited_arxiv_id":null,"evidence_quote":"Demonstrates that reducing unit heterogeneity in matched sets increases design sensitivity."},{"cited_title":"Copyathttp://j.mp/1sexgVwDownloadCitation BibTex Tagged XML Download Paper2016; 378","cited_arxiv_id":null,"evidence_quote":"Argues propensity score matching can be variance-increasing, motivating the heterogeneity-reduction approach."},{"cited_title":"Large sample properties of matching estimators for average treatment effects.econometrica 2006; 74(1): 235–267","cited_arxiv_id":null,"evidence_quote":"Provides large-sample properties for matching estimators on which Theorem 2's rate statement relies."},{"cited_title":"Endogenous stratification in randomized experiments.Review of Economics and Statistics2018; 100(4): 567–580","cited_arxiv_id":null,"evidence_quote":"Shows that same-sample prognostic stratification biased treatment effect estimates, motivating the held-out pilot."}],"review_version":1}