{"id":"0f4532bc-9756-4f6f-aae3-8077fb8829f1","arxiv_id":"2608.06512","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":5,"one_line_summary":"For target-weighted group average treatment effects under population shift, the optimal two-stage design allocates groups in proportion to sqrt(deployment share) times the sum of arm standard deviations, with the Neyman split within groups.","lead":"This paper designs a two-stage experiment that splits samples across groups and treatment arms to estimate treatment effects for a target population that differs from the study population. TWNA allocates more samples to groups that are both common in deployment and hard to measure, and it beats standard alternatives in simulations and real-data benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Transportability of group-specific effects is the load-bearing assumption: Theorem 1 is internally correct, but TWNA optimizes precision for estimands that are only identified under composition shift, an untestable condition.","rationale":"The reader's weakest-assumption diagnosis matches my read: the untestable transportability condition is the softest point in the chain from the paper's framing to its practical claim. I checked the oracle derivation, the plug-in regret proof, and the weight-robust propositions; I found no algebraic error or hidden internal assumption beyond those stated. The pilot-consistency arguments are standard and the simulation design distinguishes the two allocation margins cleanly. Because the transportability condition is stated explicitly as 'composition shift only' and the theoretical claims are conditional on it, the concern does not invalidate the central theorem; it defines the scope. The paper already flags this in the Method section and Discussion. I would therefore keep the reader's ACCEPT verdict unchanged, while noting that a semi-synthetic violation test would sharpen the boundary of the practical claim.","tokens_in":29858,"tokens_out":12990,"duration_ms":122469,"concrete_test":"Construct a semi-synthetic deployment population Q by reweighting within each fixed group S=k on a covariate that interacts with treatment (e.g., x1 in the calibrated benchmark, or pre-treatment earnings in LaLonde), so that E_Q[Y(1)-Y(0)|S=k] differs from E_P[Y(1)-Y(0)|S=k]. Run TWNA under this Q and compute realized target-weighted MSE against the true Q-GATEs. If the bias term dominates the variance reduction from the optimal allocation, the transportability premise is confirmed as load-bearing for the practical claim; if a reweighting variable that changes group effects cannot be found or the bias is small, the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central derivation (Theorem 1) is correct given its objective: minimizing B_Q(ρ,e) yields e*_k = σ1k/(σ1k+σ0k) and ρ*_k ∝ √q_k(σ1k+σ0k) by two tight Cauchy–Schwarz steps, and the regret bound in Theorem 2 follows from consistency of the pilot variance estimates. The load-bearing premise is the composition-shift assumption stated in the Method section: E_Q{Y(1)-Y(0)|S=k} = E_P{Y(1)-Y(0)|S=k}. If this fails, the target estimands τ_k are not identified from the experimental sample; TWNA then minimizes the target-weighted risk of biased quantities. This is not an internal inconsistency—the paper is explicit about the scope—but it is the condition on which the practical claim 'experiments run in one population to guide decisions in another' rests, and it cannot be tested from the experimental data alone. Every simulation and benchmark generates Y(a) with the same group-level conditional distributions in P and Q, so the evaluation never exercises the failure of this assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes TWNA, a two-stage stratified experimental design for estimating group average treatment effects (GATEs) when the deployment population differs from the experimental population in group composition. The oracle design minimizes the leading term of the target-weighted GATE risk and takes the closed form e*_k = σ1k/(σ1k + σ0k) and ρ*_k ∝ √q_k(σ1k + σ0k), with the usual Neyman treatment split within each group. The plug-in rule replaces the oracle variances with pilot estimates, and the paper proves consistency and an O_p(n0^{-1/2}) regret bound (with an O_p(n0^{-1}) refinement at an interior optimum). It also extends the design to deployment-weight uncertainty via average-case, minimax, and full-ignorance rules, and introduces a winsorized 'robust TWNA' variant for small or heavy-tailed pilot cells. The empirical evaluation includes a full simulation grid, an end-to-end validation with realized assignment, a misspecified-weight study, and three covariate-rich benchmarks (calibrated synthetic, IHDP, LaLonde), all showing gains for TWNA over uniform, deployment-only, variance-only, and ATE-optimal designs.","tokens_in":30060,"tokens_out":14239,"duration_ms":126526,"significance":"The central derivation is correct and complete: Theorem 1 is proved by two tight Cauchy–Schwarz steps, the regret proof in Appendix A is careful about clipping and floor constraints, and Proposition 1 gives a clean minimax treatment of weight uncertainty. The oracle allocation is parameter-free given the variances and weights, and the plug-in rule is explicitly evaluated against oracle and external baselines. The empirical work is unusually extensive: a full budget-by-pilot grid, paired tests, machine-reproducible code, and stress tests outside the theorem's clean conditions. The main limitation is that the target estimands are identified only under the explicitly stated composition-shift assumption (E_Q{Y(1)−Y(0)|S=k} = E_P{Y(1)−Y(0)|S=k}), which is untestable from the experimental data alone; this is a scope condition rather than an internal inconsistency, and it does not affect the validity of the derivations conditional on that assumption.","major_comments":[],"minor_comments":[{"comment":"The estimands τ_k are defined without a subscript indicating the distribution; since the paper later uses E_Q{Y(1)−Y(0)|S=k} = E_P{Y(1)−Y(0)|S=k}, please introduce them explicitly under the deployment population Q and add a sentence in the Discussion noting that this transportability condition is untestable from the pilot and final experimental data alone.","section":"Method, estimand definition"},{"comment":"In the 'Variance heterog.', 'Aligned shift+var', and 'Anti-aligned' rows, the entry 'σ σ' is ambiguous; define σ as the common value σ1k = σ0k and define σ rev as its reversal, so the reader does not have to consult Table 5 to parse the main-text table.","section":"Table 1 caption"},{"comment":"The footnote describing Cytrynbaum's v2 and v3 is confusing and appears to reference two different arXiv versions with different titles and objectives; please verify the citation and clarify whether the description applies to the published or arXiv version actually cited.","section":"Related Work, footnote 1"},{"comment":"The values 42.87, 48.86, 43.68, and 1.88% are quoted without stating their units or the exact objective (B_q(ρ,e⋆)) used in the exact frontier; please define these quantities in the main text or move the numerical discussion entirely to Appendix B.","section":"Robustness to Deployment-Weight Uncertainty"},{"comment":"The repository is described by file names only; include a URL or explicit data-availability statement so the code and data can be located by readers.","section":"Appendix D"}],"recommendation":"minor_revision","confidential_remarks":"This is a strong methods paper with a clean proof, thorough simulations, and reproducible code. The transportability assumption is explicit, and the paper would be even more useful if a short sensitivity experiment under its violation were added, but I do not see this as blocking publication. I would route the minor comments above to the authors and otherwise recommend acceptance after minor revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The central result is Theorem 1: under composition shift, the target-weighted GATE risk is minimized by the within-group Neyman split and cross-group shares proportional to sqrt(q_k)(sigma1k+sigma0k). That derivation is correct, two Cauchy-Schwarz steps with no fitted constants. The paper is also honest about the lineage: Cytrynbaum already had the ATE version and Shi-Lin had q-linear rules; the sqrt(q) weighting is the genuine new element. That makes it a useful extension rather than a paradigm shift.\n\nWhat the paper does well: the plug-in procedure and regret bound are standard but carefully done, with explicit positivity assumptions, clipping, and an allocation floor. The empirical work is unusually thorough: a simulation grid over budget and pilot size, an end-to-end check with realized sampling rather than first-order draws, real-covariate benchmarks on IHDP and LaLonde, and a battery of stress tests including heavy tails and contaminated pilots. The results behave the way the theory says, and the paired comparisons isolate where each margin contributes. The 13-44% effective sample-size savings is a fair way to state the gain.\n\nSoft spots. The transportability condition—group-specific treatment effects must be the same in P and Q—is the load-bearing assumption. If it fails, the target estimands are not identified from the experimental sample and TWNA optimizes precision for the wrong quantities. The paper states this explicitly, but every simulation and benchmark generates data under that assumption, so the failure mode is never exercised. That is a genuine limitation, though shared by the transportability literature rather than a flaw in the math. Minor: the repository is referenced but no link is provided and no code is shipped, so the reproducibility appendix is weaker than it looks. The learned-partition proxy baseline also carries a known composition bias, which makes it a soft comparison.\n\nWho should read this: anyone designing stratified experiments for target-weighted GATEs with credible deployment weights and pilot variance estimates. The paper is honest about the gap between what it delivers and the harder problem of non-compositional shift. It deserves a serious referee and, after the caveat is moved forward and the code is shared, publication.","headline":"Correct, well-scoped closed-form allocation for target-weighted GATE precision; the transportability assumption is the main caveat but does not undermine the result.","tokens_in":30652,"tokens_out":2121,"would_cite":true,"duration_ms":18091,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proves the closed-form optimal allocation for target-weighted group treatment effects under population shift: within each group, treat at the Neyman fraction σ₁ₖ/(σ₁ₖ+σ₀ₖ), and across groups, allocate sample proportional to…","keywords":["experimental design","heterogeneous treatment effects","population shift","generalization","Neyman allocation","stratified randomized experiments","two-stage design","deployment-weighted precision"],"falsifier":"Evaluate the oracle formula against numerical grid-search minimization of B_Q(ρ,e) on thousands of random (q,σ) draws; any mismatch falsifies Theorem 1, and observing in a deployment population a group whose treatment effect differs from the experimental estimate would falsify the transportability premise the design rests on.","tokens_in":29632,"feed_emoji":"🎯","tokens_out":6208,"duration_ms":48309,"temperature":0.7,"pith_summary":"The paper asks how to split a fixed experimental budget across pre-specified groups and treatment arms when the experiment will guide decisions in a deployment population whose group composition differs from the experimental one. It shows that the design minimizing target-weighted mean squared error of group average treatment effects has a closed form: within each group, assign treatment in the Neyman proportion σ₁ₖ/(σ₁ₖ+σ₀ₖ); across groups, allocate sample proportional to √qₖ(σ₁ₖ+σ₀ₖ), the geometric compromise between deployment importance and measurement difficulty. A two-stage procedure that first runs a small pilot to estimate group–arm variances then plugs them in provably converges to this oracle as the pilot grows, and the rule degrades gracefully when deployment weights are misspecified. This matters because standard designs that allocate by experimental proportions, deployment proportions, or variance alone can lose 30–60% relative precision exactly when deployment importance and statistical difficulty coincide.","feed_headline":"Optimal trial design: sample ∝ √q(σ₁+σ₀) per group","feed_subtitle":"For experiments that inform a different population, this two-stage design provably cuts group-effect error.","key_machinery":"The central object is the target-weighted group average treatment effect risk B_Q(ρ,e) = Σₖ (qₖ/ρₖ)(σ₁ₖ²/eₖ + σ₀ₖ²/(1−eₖ)), the leading-order variance of the stratified difference-in-means estimator for fixed groups, weighted by deployment shares qₖ. Minimizing it factorizes: the within-group Neyman treatment fraction e*ₖ = σ₁ₖ/(σ₁ₖ+σ₀ₖ) is obtained first by Cauchy–Schwarz on each group, leaving the group-allocation problem Σₖ qₖsₖ²/ρₖ whose minimizer ρ*ₖ ∝ √qₖsₖ, with sₖ = σ₁ₖ+σ₀ₖ, is the equality case of the Engel form of Cauchy–Schwarz. All extensions—the weight-robust average/worst-case/ignorance designs, the plug-in regret bound, and the saddle-point proofs—reuse the same factorization.","core_discovery":"The paper's central discovery is a complete solution to a design problem: minimize B_Q(ρ,e) = Σₖ (qₖ/ρₖ)(σ₁ₖ²/eₖ + σ₀ₖ²/(1−eₖ)) over group shares ρ and treatment probabilities e, the leading-order variance of the stratified difference-in-means estimator for a fixed vector of group effects weighted by deployment shares. Two Cauchy–Schwarz steps give the unique minimizer: e*ₖ = σ₁ₖ/(σ₁ₖ+σ₀ₖ) and ρ*ₖ = √qₖ(σ₁ₖ+σ₀ₖ)/Σₗ √qₗ(σ₁ₗ+σ₀ₗ). The result specializes classical Neyman allocation to a target-weighted GATE loss, introducing the √qₖ factor that balances deployment importance against statistical difficulty. The paper further shows that the plug-in two-stage rule recovers the oracle as pilot variance estimates stabilize, that deployment-weight uncertainty is handled by mean, least-favorable, or equalizing weights with closed forms, and that the proportional gains are largest when deployment importance and variance difficulty are aligned.","pith_inferences":["A natural extension is to relax the transportability assumption itself, for instance by estimating group-specific drift between experimental and deployment populations from auxiliary observational data and inserting those shifted estimands into the same closed-form allocation; the √qₖ mechanism would survive, but the weights would target shifted quantities.","The same Cauchy–Schwarz allocation logic should apply to other deployment-weighted objectives, such as weighted policy regret or weighted average treatment effects, where the √qₖ factor would be modified by the objective's curvature; the paper's weight-robust Proposition 1 already provides the template for convex ambiguity sets.","The equalizer rule ρₖ ∝ sₖ² suggests a testable design principle: when the deployment mix is genuinely unknown, allocating by squared total variance is a no-regret choice that an always-on platform with fixed traffic composition could benchmark against post-hoc correction.","The plug-in's tail fragility in spike-mixture settings, with coefficient of variation near 6 in the aligned platform cell, indicates that practical deployments should pair the plug-in rule with a robust scale estimator whenever pilot cells can miss rare high-variance events; the paper's robust variant is one such fix, and other shrinkage or Bayesian estimators are worth experimenting with."],"forward_implications":["If the oracle rule is correct, a designer who knows or can estimate deployment shares and group–arm variances should allocate sample proportionally to √qₖ(σ₁ₖ+σ₀ₖ) and split treatment within each group by the Neyman fraction, with deployment-important and hard-to-measure groups receiving more units.","The two-stage plug-in procedure converges to the oracle allocation as pilot size grows, with design regret O_p(n₀^{-1/2}) in general and O_p(n₀^{-1}) at an interior optimum, so a few dozen observations per group–arm cell capture most of the attainable gain.","Misspecifying deployment weights cannot cost more than discarding them entirely: with uniform input weights the rule reduces to the q-free HHK plug-in, and for any partial misspecification the √q margin remains strictly beneficial.","Under complete ignorance of the deployment composition, the equalizing rule ρₖ ∝ (σ₁ₖ+σ₀ₖ)² keeps target-weighted risk constant across all compositions, and the minimax rule traces the lower envelope of worst-case risk.","The cross-group allocation margin composes with within-group stratification: applying a stratification-tree refinement on TWNA's composition performs no worse than on uniform composition in the paper's benchmarks, with the gain coming from the cross-group margin."],"supporting_citations":[{"why":"Supplies the classical stratified-sampling treatment allocation that TWNA specializes to the target-weighted GATE loss.","marker":"Neyman (1992)"},{"why":"Provides the pilot-adaptive two-stage template and the propensity-score clipping used for the plug-in defaults.","marker":"Hahn, Hirano, and Karlan (2011)"},{"why":"Closest prior work jointly optimizing sampling and treatment assignment, yielding the same within-stratum Neyman fraction.","marker":"Cytrynbaum (2023)"},{"why":"The transportable-design baseline that TWNA is benchmarked against and whose fixed-treatment allocation is adapted to the GATE loss.","marker":"Shi and Lin (2022)"},{"why":"Documents small-pilot concerns that motivate the pilot-robust variant and the conservative regret analysis.","marker":"Cai and Rafi (2024)"},{"why":"Provides the stratification-tree baseline used for the within-group refinement in the benchmarks.","marker":"Tabord-Meehan (2023)"}],"fun_headline_variants":["TWNA: Optimal sample allocation for shifted populations","Formula: sample ∝ √q(σ₁+σ₀) per group","Two-stage design for precise target-weighted GATEs","Balancing deployment importance and variance: TWNA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The design assumes that group-specific treatment effects transport unchanged from the experimental population to the deployment population; if a group's effect differs in deployment, the allocation optimizes precision for the wrong target quantities, and this transportability premise is untestable from experimental data alone.","fun_headline_variants_meta":{"raw":{"variants":["TWNA: Optimal sample allocation for shifted populations","Formula: sample ∝ √q(σ₁+σ₀) per group","Two-stage design for precise target-weighted GATEs","Balancing deployment importance and variance: TWNA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000347,"raw_usage":{"total_tokens":1915,"prompt_tokens":975,"completion_tokens":940,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":872}},"tokens_in":591,"tokens_out":940,"duration_ms":8072,"temperature":1.0,"reasoning_tokens":872,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:32:34.565763+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the oracle formula against numerical grid-search minimization of B_Q(ρ,e) on thousands of random (q,σ) draws; any mismatch falsifies Theorem 1, and observing in a deployment population a group whose treatment effect differs from the experimental estimate would falsify the transportability premise the design rests on.","supporting_citations":[],"review_version":2}