{"id":"c38dc13b-8766-43cf-8858-182bdac2bdf1","arxiv_id":"1908.00654","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Two new adjustment methods for treatment switching in cancer trials, stratified RPSFTM and random-forest-based prediction, are proposed and evaluated only in a simulation study with mixed results.","lead":"Patients in the control arm of cancer trials often switch to other treatments, which masks the true survival benefit of the experimental drug. This paper proposes two adjustment methods, stratified RPSFTM and a random forest prediction approach, and tests them only in simulated data, with mixed results.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The random-forest method treats censored control-arm non-switchers' observed times as true counterfactual survival times; at realistic censoring rates this biases predicted U and confounds the claimed simulation benefit.","rationale":"I read the paper as proposing two practical estimators for multi-level treatment switching, with a simulation-based claim that the new methods outperform existing ones. The stratified RPSFTM is a natural extension of RPSFTM and is plausible, but the random-forest method has a serious statistical flaw that is load-bearing for the central claim. The reader's weakest assumption identifies the same problem: the training set includes censored non-switchers and the method assumes the same model predicts counterfactual survival regardless of censoring. This is not merely a strong assumption; it is violated by construction in the simulation whenever censoring is present. Because the simulation's censoring rates reach 75%, the reported behavior of the random-forest method, good at low censoring and worse at high censoring, is exactly what would be expected from regressing on censored outcomes, so the simulation cannot distinguish genuine adjustment performance from this artifact. The paper contains an in-text limitation statement in Section 5 acknowledging that the assumption is strong and hard to check, which supports the concern. The proposed test would isolate the censoring artifact by comparing the proposed estimator with an oracle version using true counterfactual times, which are available in the simulation. No code is provided, so the exact random-forest implementation cannot be independently audited. Given the reader's verdict is already CONDITIONAL and this concern points to the same central weakness, I would not change the verdict, but the revision should address the censoring issue directly and temper the blanket claim in Section 4.3.2.","tokens_in":16222,"tokens_out":4692,"duration_ms":49334,"concrete_test":"Re-run the paper's simulation for, say, HR=0.6, switch proportion 50%, and censoring 75%, with two variants of the random-forest step: (A) the proposed method, training on observed times of all control-arm non-switchers with censored times treated as events; and (B) an oracle version, training on the true simulated counterfactual times U for the same non-switchers. Compare estimated HR bias and MSE over the 500 replicates. If variant B reduces the bias of variant A by a practically important amount (for example, more than 50%), the proposed method's reported performance is an artifact of ignoring censoring, and the central claim should be restricted or the method revised to handle censored training labels, for example by using a survival forest or inverse-probability weighting of the training data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing defect is in the random-forest method of Section 3.2.2. Step 1 uses all non-switchers in the control arm as the training set and takes their observed time as the counterfactual event time U. For censored non-switchers, the observed time is min(U,C), not U; treating it as an event time changes the regression target from E[U|X] to E[min(U,C)|X], which is systematically smaller under independent censoring. The paper explicitly assumes that the same model predicts U regardless of censoring status, but this is false whenever any training observation is censored. At the 75% censoring level used in the simulation, a large fraction of control-arm non-switchers contribute censoring times, so the predicted counterfactual survival times for switchers are biased downward; the later max(predicted, switch time) floor does not repair that bias. The reported simulation pattern, in which the random-forest method looks good at 25% censoring and degrades at higher censoring, is exactly what this misspecification predicts, so the simulation does not validate the method. The Section 4.3.2 claim that the proposed methods 'show benefit over current existing methods in all scenarios' therefore relies on a method whose apparent low-censoring performance is confounded by a censoring artifact. The paper itself acknowledges in Section 5 that the assumption is strong and hard to check, but it never tests it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper considers the problem of estimating the overall survival treatment effect in randomized trials when control-arm patients switch to the experimental treatment or to other salvage therapies. The authors review simple and advanced adjustment methods (ITT, exclusion, censoring, IPCW, RPSFTM, IPE) and propose two extensions: a stratified RPSFTM that allows different acceleration factors for different treatment levels, and a random-forest-based prediction method that imputes counterfactual survival times for switchers from covariates. The evaluation is a simulation study with 500 replications per scenario, varying the true hazard ratio (0.4, 0.6, 0.8), censoring rate (25%, 50%, 75%) and switch proportion (25%, 50%, 75%). The paper reports bias, MSE and coverage for all methods and concludes that the two proposed methods generally outperform existing methods.","tokens_in":16435,"tokens_out":9552,"duration_ms":88448,"significance":"If the proposed methods performed as claimed, they would be practically useful for health technology assessments of oncology drugs, where treatment switching is common and multi-level switching is increasing. The paper provides a clear taxonomy of existing methods and a thorough simulation grid. However, the evidence is entirely simulation-based, and the simulation has serious internal problems: the random-forest training target treats censored observations as events, the data-generating model is the same AFT family assumed by the RPSFTM approach, and some summary tables contain impossible values. These issues currently prevent the paper from establishing the validity of either new method.","major_comments":[{"comment":"The random forest method trains on all non-switchers in the control arm and uses their observed survival time as the counterfactual event time U_i. For censored non-switchers, the observed time is min(U_i, C_i), not U_i, so the regression target is E[min(U,C)|X] rather than E[U|X]. Under independent censoring this target is systematically smaller, and the bias grows with the censoring rate; the paper's explicit assumption in Section 3.2.2 that the same model predicts U regardless of censoring status is therefore false whenever any training observation is censored. Table 2 shows exactly the predicted pattern: for HR=0.4 with 50% switched, the RF bias goes from -0.612 at 25% censoring to -8.787 at 75% censoring, and the MSE in Table 1 rises from 0.357 to 1.967. The Section 4.3.2 claim that the proposed methods 'show benefit over current existing methods in all scenarios' is not supported for the random forest method, because its apparent advantage at low censoring is a training-set artifact rather than evidence of validity.","section":"Section 3.2.2, Table 2"},{"comment":"The simulation generates data from a Weibull AFT model, log(S)=α0+α1 arm+α2 age+α3 ECOG+α4 risk+ε, which is exactly the structural family assumed by RPSFTM and stratified RPSFTM. Evaluating RPSFTM-type estimators on data generated from their own assumed model measures internal consistency, not general performance. The paper's conclusions that stratified RPSFTM outperforms other methods would be more convincing if the simulation also included data generated under a Cox proportional hazards model, a time-varying treatment effect, or non-log-linear covariate effects. As it stands, the favorable performance of the RPSFTM family is at least partly a self-referential feature of the simulation design.","section":"Section 4.2 vs Sections 2.2.2 and 3.1"},{"comment":"In Table 2, the bias rows for ITT (75% Switched) and RPSFTM (75% Switched) at HR=0.4 contain the same numbers as the corresponding MSE entries in Table 1 (5.768, 4.876, 4.876 for ITT; 0.591, 0.758, 0.758 for RPSFTM). These values are impossible as biases: a bias of 5.768 with true HR=0.4 would imply an estimated HR of -5.368. The table appears to have the wrong columns copied, and these rows cannot be used to evaluate the methods. The authors should correct Table 2 and re-verify the simulation summaries.","section":"Table 2, rows for ITT and RPSFTM under 75% Switched"},{"comment":"The statement in Section 4.3.2 that 'In all scenarios our newly proposed methods show benefit over current existing methods among the three numerical metrics' is contradicted by Table 4, which recommends ITT for all HR=0.8 scenarios, and by the coverage results in Table 3: SRP has 54.4% coverage in the HR=0.4, 75% switched, 25% censored cell, and RF has 70.4% coverage in the HR=0.4, 75% switched, 75% censored cell. Coverage far below the nominal 95% is a serious deficiency that is not reflected in the bias and MSE summaries. The recommendation table and the summary claim need to be reconciled.","section":"Section 4.3.2 and Table 3, Table 4"},{"comment":"The data-generating algorithm for switchers is not precisely defined. The text first generates a switch time from Uniform(0,365), but then defines T_i,treatment = T_i,control(no switch) × exp(-ψ_k)F and T_i = T_i,control(before switch) + T_i,treatment. It is unclear how the switch time enters this calculation, how T_i,control(no switch) relates to the generated no-switch survival time, and what happens when the switch time exceeds the no-switch survival time. Because the RPSFTM structural model requires splitting the observed time into T_control and T_treatment at the actual switch time, the equations as written do not establish that the simulated data follow the assumed counterfactual model. A precise generative algorithm or code is needed before the simulation results can be interpreted.","section":"Section 4.2, switching-time generation"}],"minor_comments":[{"comment":"The sign convention for the acceleration factor is stated as exp(-ψ) in the RPSFTM equation but as exp(ψ_k) in Section 3.1; the equivalence between 'beneficial treatment' and ψ<0 should be defined once and used consistently.","section":"Section 2.2.2"},{"comment":"The floor rule 'the maximum of predicted value and time of switching is used' is an ad hoc correction; the paper should explain whether this is a modelling assumption or a computational trick, and report the sensitivity of results to this rule.","section":"Section 3.2.2"},{"comment":"Coverage percentages are reported with inconsistent precision (e.g., '78%' alongside '93.4%'); use a uniform format.","section":"Table 3"},{"comment":"The motivating study is described as having approximately 20% events, but the simulation censoring rates are 25%, 50% and 75%; the relationship between the real event rate and the simulated censoring scenarios should be clarified.","section":"Section 4.1"},{"comment":"The phrase 'the two new switched? treatments' contains an apparent typo; also, the authors should state whether simulation code is available for reproducibility.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"Given the copy-paste error in Table 2 and the absence of code, I would ask the authors to share the simulation code and corrected tables. The random forest method, as currently specified, is not a valid estimator for censored survival outcomes, and the simulation design is too favorable to the RPSFTM family. These are fixable with additional work, but the current version is not publishable without major changes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper tackles a real problem: estimating the true survival benefit of an experimental drug when control-arm patients switch to various treatments, including treatments that are not the trial drug. That is exactly the situation payers and reimbursement agencies face, and the existing toolkit (RPSFTM, IPE, IPCW) is genuinely limited there. What the paper does well is to extend RPSFTM to multiple treatment levels—stratified RPSFTM is a natural, useful generalization with multiple acceleration factors, and the accompanying simulation covers a wide grid of scenarios (HR 0.4/0.6/0.8, 25/50/75% censoring, 25/50/75% switching). The authors are also unusually candid about limitations: they explicitly flag the strong assumption behind the random-forest method and the computational burden of g-estimation as the number of treatment levels grows.\n\nThe main soft spot is the random-forest method. Step 1 trains on all non-switchers in the control arm, taking their observed time as the counterfactual event time U. For censored patients, that observed time is min(U, C), not U. So the regression target is E[min(U,C)|X], which is systematically smaller than E[U|X] when any training patient is censored. The paper assumes the same model regardless of censoring status, but that is exactly what fails. The max(predicted, switch time) floor does not repair the bias. It is telling that the simulation shows RF doing well at 25% censoring and degrading at 75%—that is precisely the signature of this misspecification. The authors acknowledge the assumption is strong and hard to check, but they never test it.\n\nAlso, the summary claim in Section 4.3.2 that the proposed methods show benefit over current existing methods among the three numerical metrics is overstated. The tables show stratified RPSFTM does not always win; for example, at HR=0.4 with 75% switched and 25% censored, RPSFTM has MSE 0.591 vs SRP 0.988. The recommendation table is more honest, but the text overclaims. And the simulation generates data from a Weibull AFT model, which directly favors the AFT-based methods. That is not a fatal flaw, but it limits the generalizability. No code or implementation details are given, so the results are not independently reproducible from the text.\n\nFor whom is this paper? Biostatisticians working on treatment-switching adjustment, especially in health technology assessment. Stratified RPSFTM is worth a serious referee; it could be genuinely useful after refinement and a more efficient estimation algorithm. The random-forest method, as it stands, is not sound until censoring is handled properly—e.g., via random survival forests or inverse-probability weighting within the training step. I would send this to peer review, but with the expectation of major revision and a re-analysis of the RF method.","headline":"A serious methods paper with one solid extension (stratified RPSFTM) and one flawed machine-learning approach that needs major rework before it can support the paper's claims.","tokens_in":823,"tokens_out":1114,"would_cite":false,"duration_ms":39464,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62N02","62P10"],"pacs":[],"model":"deepseek-v4-flash","headline":"Two new adjustment methods, stratified RPSFTM and random-forest prediction, recover the true overall survival benefit when control-arm patients switch to multiple treatments.","keywords":["treatment switching","overall survival","RPSFTM","random forest","IPCW","iterative parameter estimation","hazard ratio","clinical trials"],"falsifier":"Re-run the paper's simulation with switch status and switching time driven partly by an unmeasured baseline covariate that also predicts survival, so that non-switchers and switchers differ in prognosis beyond the measured covariates; if the random-forest adjusted hazard ratio drifts systematically from the true value, the same-model assumption is refuted.","tokens_in":15911,"feed_emoji":"📊","tokens_out":9466,"duration_ms":84846,"temperature":0.7,"pith_summary":"Oncology trials face a problem: when patients on the control arm switch to the experimental drug or other cancer therapies after progression, the intent-to-treat estimate understates the experimental drug's true overall survival benefit, and existing adjustment methods assume switches happen only to one treatment. This paper proposes two ways to recover the unobservable no-switch survival time: a stratified rank-preserving structural failure time model (RPSFTM) that allows several different levels of treatment effect, and a random-forest-based prediction that imputes, for each switcher, the survival time they would have had without switching, using baseline covariates. In a simulation calibrated to a phase III blood-cancer trial, with true hazard ratios 0.4, 0.6, and 0.8 and censoring and switch rates from 25 to 75 percent, the new methods produce lower bias, lower mean squared error, and better confidence-interval coverage than the existing methods in the scenarios studied. If the result holds in practice, health-technology assessors would have usable estimators for the common case in which control patients switch to several different treatments.","feed_headline":"Two new methods recover survival benefit after treatment switches","feed_subtitle":"Simulation shows stratified RPSFTM and random-forest prediction beat existing switching adjustments in most scenarios.","key_machinery":"The core object is the rank-preserving structural failure time model (RPSFTM), extended into a stratified form that models a counterfactual survival time $U_i = T_{i,\\text{control}} + \\exp(\\psi_k) T_{i,\\text{treated}}$ with a separate parameter $\\psi_k$ for each treatment-effect level, replacing RPSFTM's single 'common treatment effect'. The second machine is the random forest, a tree-ensemble learner trained on non-switchers in the control arm to predict each switcher's counterfactual survival time from baseline covariates, with predicted values never allowed to be earlier than the switching time. The estimation engine for stratified RPSFTM is g-estimation over a grid of candidate $\\psi_k$ values, selecting the combination that makes the counterfactual-time distributions equal across arms.","core_discovery":"The central claim is that multi-level treatment switching can be adjusted by replacing the single acceleration factor of RPSFTM with a small set of treatment-level acceleration parameters estimated by g-estimation, and alternatively by training a random forest on control-arm non-switchers to predict each switcher's counterfactual survival time. The paper shows by simulation that both approaches estimate the true hazard ratio more accurately and more stably than ITT, censoring switchers, IPCW, RPSFTM, and IPE when control patients switch to two treatment levels, one matching the experimental effect and one 30 percent less effective. Stratified RPSFTM is generally the best except at low censoring rates, where the random forest is more stable; both new methods keep coverage near nominal in most scenarios, while IPCW and censoring-based methods fail badly with high switching and censoring.","pith_inferences":["The paper treats the switched treatments' effect levels as known in simulation; in a real trial these must be guessed from external evidence, and misspecifying a level would likely push both new methods back toward bias.","Because the random-forest approach trains only on control-arm non-switchers, an internal exchangeability check, such as comparing predicted against observed survival in a validation set of late switchers, would test its central assumption on real data.","If g-estimation for stratified RPSFTM were replaced by a faster optimization routine, the method could scale beyond the reported 'fewer than four levels' limit and cover real regimens with many lines of therapy.","The simulation's covariates are generated independently and weakly; real datasets with correlated, strongly prognostic covariates could change the relative performance of IPCW and the random-forest method."],"forward_implications":["Health-technology assessments can estimate the no-switch treatment effect even when control patients switch to more than one therapy, using stratified RPSFTM for moderate-to-high censoring and random-forest prediction for low censoring.","The simulation supports avoiding IPCW and simple censoring adjustments when the switch proportion is high, because their bias and variance are the worst in those scenarios.","When the true treatment effect is small (hazard ratio near 0.8), an ITT analysis is adequate and switching adjustments add little.","RPSFTM and IPE remain the strongest of the existing methods, so the new methods should be judged against them and improve on them in multi-level switching settings."],"supporting_citations":[{"why":"Supplies the inverse probability of censoring weighting framework that the paper evaluates as a comparator.","marker":"[1]"},{"why":"Defines marginal structural models, the setting in which IPCW is applied.","marker":"[2]"},{"why":"Introduces the rank-preserving structural failure time model that stratified RPSFTM extends.","marker":"[5]"},{"why":"Provides the g-estimation algorithm used to estimate the acceleration parameters.","marker":"[7]"},{"why":"Supplies the re-censoring rule needed when counterfactual times are derived for censored patients.","marker":"[8]"},{"why":"Introduces the iterative parameter estimation method used as a comparator.","marker":"[11]"},{"why":"Defines random forests, the prediction algorithm at the core of the second proposed method.","marker":"[12]"}],"fun_headline_variants":["Unmasking true survival benefit amid treatment switches","New models correct survival estimates for treatment changes","Stratified RPSFTM and random forest outperform old methods","Multi-level switching handled by two new survival methods","Counterfactual survival: beating switching adjustments"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The random-forest method rests on the premise that the model learned from control-arm patients who never switch predicts what would have happened just as accurately for patients who did switch, including those whose records were censored rather than ended in death.","fun_headline_variants_meta":{"raw":{"variants":["Unmasking true survival benefit amid treatment switches","New models correct survival estimates for treatment changes","Stratified RPSFTM and random forest outperform old methods","Multi-level switching handled by two new survival methods","Counterfactual survival: beating switching adjustments"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00058,"raw_usage":{"total_tokens":2717,"prompt_tokens":915,"completion_tokens":1802,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":1729}},"tokens_in":531,"tokens_out":1802,"duration_ms":16846,"temperature":1.0,"reasoning_tokens":1729,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:40:54.753515+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the paper's simulation with switch status and switching time driven partly by an unmeasured baseline covariate that also predicts survival, so that non-switchers and switchers differ in prognosis beyond the measured covariates; if the random-forest adjusted hazard ratio drifts systematically from the true value, the same-model assumption is refuted.","supporting_citations":[{"cited_title":", one can re-censor all patients at 𝑈2∗=minB𝑈2,C","cited_arxiv_id":null,"evidence_quote":"Defines marginal structural models, the setting in which IPCW is applied."},{"cited_title":"no unmeasured confounders","cited_arxiv_id":null,"evidence_quote":"Introduces the rank-preserving structural failure time model that stratified RPSFTM extends."},{"cited_title":"Correcting for non-compliance in randomized trials using rank preserving structural failure time models[J]","cited_arxiv_id":null,"evidence_quote":"Provides the g-estimation algorithm used to estimate the acceleration parameters."},{"cited_title":"G-estimation of the effect of prophylaxis therapy for Pneumocystis carinii pneumonia on the survival of AIDS patients[J]","cited_arxiv_id":null,"evidence_quote":"Supplies the re-censoring rule needed when counterfactual times are derived for censored patients."},{"cited_title":"Assessing methods for dealing with treatment switching in clinical trials: A follow-up simulation study[J]","cited_arxiv_id":null,"evidence_quote":"Introduces the iterative parameter estimation method used as a comparator."},{"cited_title":"Estimating a treatment effect in survival studies in which patients switch treatment[J]","cited_arxiv_id":null,"evidence_quote":"Defines random forests, the prediction algorithm at the core of the second proposed method."}],"review_version":1}