{"id":"b8ba9a2f-dc6e-435c-b5f0-8717cefb0871","arxiv_id":"1908.05810","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper's central estimate is that the PROWESS treatment killed roughly two participants for every three it saved.","lead":"This paper develops a statistical model that claims to estimate how many participants in a clinical trial were saved, killed, or unaffected by a treatment, using only the four summary death counts. Applied to the PROWESS sepsis trial, it reports that the drug killed two patients for every three it saved, a claim with large uncertainty.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The saved/killed split is unidentified: the multinomial likelihood depends only on sums such as t3+t4 and t2+t4, so the reported 103 killed and 12%/18% ratio are artifacts of the least-squares weighting, not the data.","rationale":"The reader's weakest assumption—that the least squares objective identifies the true saved/killed split—is exactly the load-bearing weakness. The paper is transparent, carefully written, and provides reproducible code and a Monte Carlo design, but these virtues do not overcome the fundamental lack of identification. The observed aggregate data in a binary randomized trial identify only the three independent marginal sums of the four potential-outcome types, leaving a one-dimensional continuum of equally likely t vectors. The transformation t + a*(1,-1,-1,1) leaves the multinomial cell probabilities unchanged, so the likelihood is flat and the MLE is not unique. The least squares estimator selects a point from this continuum through a weighting scheme that is not derived from the data-generating process and has no identification justification. The reported standard error for the killed count (132) already signals that the estimate is not statistically significant, and the Monte Carlo calibrates the truth to the estimator's own output, providing no independent validation. Because the paper's headline claim—the ratio of killed to saved and the in-trial death count—depends entirely on this non-identified split, the central scientific claim fails. The verdict of REJECT is appropriate and unchanged by this stress test.","tokens_in":11175,"tokens_out":6652,"duration_ms":63631,"concrete_test":"Compute the likelihood L(t|g,p) from Section 2 for the PROWESS data at the reported estimate t=(964,308,205,213) and at t'=(1169,103,0,418), obtained by adding 205*(1,-1,-1,1) so that all four sums s1..s4 are unchanged. If L(t|g)=L(t'|g) to numerical precision while the least squares objective S differs between the two, this confirms that the data do not identify the saved/killed split and the reported ratio is an artifact of the estimator's weighting. A direct way to see this is to re-express the likelihood as Multinomial(N; p(t3+t4)/N, p(t1+t2)/N, (1-p)(t2+t4)/N, (1-p)(t1+t3)/N) and observe that the transformation leaves every cell probability unchanged.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that PROWESS killed 103 participants in-trial and killed two for every three saved—requires identifying t(2) (saved) and t(3) (killed) separately from the observed cell counts g=(210,640,259,581). This is not possible from the data. The four observed cells are multinomial with cell probabilities proportional to p(t3+t4), p(t1+t2), (1-p)(t2+t4), (1-p)(t1+t3), where p=1/2 (Section 2). These probabilities depend on t only through the three independent sums s1=t3+t4, s2=t1+t2, s3=t2+t4, and s4=t1+t3 with s1+s2=s3+s4=N. Consequently, the likelihood in Eq. (1)-(3) is invariant under the transformation t -> t + a*(1,-1,-1,1), which preserves all four sums and hence all cell probabilities. The maximum likelihood estimator is therefore set-valued over this continuum. The least squares estimator of Section 3.2 minimizes S(N|p), a weighted sum of squared randomization errors that depends on the unobserved matrix N(i,j); the observed data impose only column-sum constraints, leaving the row split free. The reported point estimates t(2)=308 and t(3)=205 are thus selected by the arbitrary weighting 1/[p(1-p) sum_{i in I} t(i)], not by evidence. The bootstrap standard error for t(3) is 132 (Figure 2), so t(3) is not statistically significant, and the Monte Carlo (Section 5) calibrates the true t to the very estimates in question, so it cannot establish identification. The central ethical claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops a model of a two-arm randomized experiment with a binary outcome, classifying participants into four potential-outcome types: would live regardless, would be saved, would be killed, and would die regardless. It derives an expression for the probability of the observed cell counts as a function of the type counts and the randomization probability, then proposes a maximum likelihood estimator and a computationally tractable least squares estimator. Applying the least squares estimator to aggregate data from the PROWESS trial (cell counts g = (210, 640, 259, 581), p = 1/2), the paper reports that 18% of participants would be saved, 12% would be killed, that the intervention killed two participants for every three saved, and that 103 trial participants were killed by the intervention. A Monte Carlo simulation calibrates the true type counts to these point estimates and reports bias and RMSE.","tokens_in":11445,"tokens_out":2150,"duration_ms":22019,"significance":"Were the estimates identified by the data, the ethical and clinical implications would be substantial, because the paper claims to quantify the number of trial participants harmed by a life-saving intervention. The paper is also transparent about the computational difficulty of the maximum likelihood problem and clearly explains the potential-outcome taxonomy. However, the central quantitative claim is not supported: the observed data identify only the reduced-form difference between saved and killed counts, not the split between them, so the reported 103 killed and the 2:3 ratio are artifacts of an arbitrary objective function rather than evidence. The Monte Carlo exercise cannot repair this because it assumes the disputed point estimates as the truth.","major_comments":[{"comment":"The likelihood expression is invariant to the transformation t -> t + a*(1, -1, -1, 1), which preserves all four sums t3+t4, t1+t2, t2+t4, and t1+t3 that enter the data probabilities. Consequently, the observed cell counts identify only the marginal mortality rates in the two arms, and the split between participants who would be saved and those who would be killed is unidentified. The maximum likelihood estimator is therefore set-valued over a continuum of type-count vectors, and the likelihood surface is flat in the saved/killed dimension.","section":"Section 2, Eqs. (1)-(3)"},{"comment":"The least squares estimator minimizes a weighted sum of squared randomization errors over the unobserved matrix N(i,j), subject only to the column-sum constraints that reproduce the observed cell counts. Because those constraints leave the row split (equivalently, the saved/killed decomposition) free, the minimum of S(N|p) is determined by the particular weighting scheme rather than by features of the data. The reported t(2)=308 and t(3)=205 are therefore selected by the objective function, not identified by the experiment.","section":"Section 3.2, objective S(N|p)"},{"comment":"The bootstrapped standard error for the estimated number killed, t(3)=205, is reported as 132, so the estimate is not statistically distinguishable from zero at conventional levels. The claim that the intervention killed 103 participants within the trial relies on an intermediate cell estimate n(3,1)=103 with standard error 66, which is also not statistically significant. The paper's headline ratio and absolute killed count are thus unsupported by its own reported precision.","section":"Section 4.2, Figure 2"},{"comment":"The Monte Carlo simulation sets the true potential-outcome type vector t equal to the paper's own point estimates from PROWESS and then evaluates the estimator's bias and RMSE. This exercise can at most quantify finite-sample noise around a chosen oracle value; it cannot establish that the observed PROWESS data identify the saved/killed split, because the simulation never varies the split along the unidentified dimension. The reported mean bias of about 42 participants and RMSE of about 124 participants are therefore not evidence for the validity of the point estimates.","section":"Section 5, Monte Carlo design"}],"minor_comments":[{"comment":"There is a duplicated word: 'about half of those participants were were randomized into the control group' should read 'were randomized'.","section":"Introduction, paragraph after the abstract results"},{"comment":"The phrase 'effectively a separate randomized experiment within each potential outcome outcome type i' contains a duplicated word 'outcome' and should be corrected.","section":"Section 2, paragraph before Eq. (1)"},{"comment":"The manuscript states it has been combined with another paper and superseded by a later arXiv posting. If this version is still under consideration, the relationship to the superseding paper should be clarified in the submission letter or a revision.","section":"Title page note"},{"comment":"The reduced form is reported as '6 percentage points', while the precise calculation 210/850 - 259/840 equals approximately -0.0602; stating the exact value in the text would avoid ambiguity.","section":"Section 4.1, first paragraph"}],"recommendation":"reject","confidential_remarks":"The central identification failure is not a local fix: the likelihood is flat over the saved/killed dimension, and the least squares objective picks an arbitrary point on that flat region. The bootstrap standard errors already show the killed count is insignificantly different from zero, and the Monte Carlo is circular. Given the manuscript's own note that it has been superseded by arXiv:1912.06739, I see no path to publication for this version in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does one thing right: it writes down a correct likelihood for the four-type potential outcomes model, using the independence of randomization within each type to express the observed cell counts as a convolution of binomials. It is also transparent about the MLE being computationally intractable, and it describes the PROWESS data and the reduced-form logic clearly. The least squares estimator is a new construction, and the Monte Carlo is carefully described.\n\nThe problem is that the headline claim is not identified. The stress-test note is exactly right: the likelihood depends on t only through the three sums t3+t4, t1+t2, t2+t4, and t1+t3, so the transformation t -> t + a*(1,-1,-1,1) leaves every observable probability unchanged. The four observed cells pin down the two marginal mortality rates and nothing else; the split between saved and killed is a continuum. The least squares objective S(N|p) weights randomization errors within subsets of types, but those subsets include unobserved within-type intervention counts, so the objective is not a function of the data alone. The reported t(2)=308 and t(3)=205, and therefore the \"two killed for every three saved\" ratio, are chosen by the particular weighting scheme, not by evidence.\n\nThe empirical results confirm this. The bootstrap standard error for t(3) is 132, so the number killed is not statistically distinguishable from zero. The Monte Carlo calibrates the true t to the paper's own point estimates, so it cannot validate identification; it only shows that the estimator recovers its assumed inputs on average. That is circular in exactly the sense the reader flagged.\n\nThis is a useful teaching example of how an arbitrary estimator can produce a definite-looking answer from unidentified parameters, but it is not a scientific claim about PROWESS. The author needed either an identifying assumption (e.g., monotonicity, which would set the killed count to zero) or an explicit acknowledgment that only the difference is estimable. Without that, the central ethical conclusion is unsupported.\n\nI would not send this to peer review as a substantive contribution. If it crossed my desk as an editor, I would desk-reject on identification grounds, though I might suggest the authors reframe it as a cautionary note. A serious referee would catch the flaw on page two, and the paper as written provides no fix.","headline":"A clearly written paper whose headline claim is an artifact of an arbitrary least-squares weighting, not a measured quantity: the saved/killed split is unidentified from the aggregate trial data.","tokens_in":12010,"tokens_out":1986,"would_cite":false,"duration_ms":20905,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62P10","62F10"],"pacs":[],"model":"deepseek-v4-flash","headline":"Reanalysis of the PROWESS sepsis trial estimates the drug killed two participants for every three it saved.","keywords":["randomized experiment","potential outcomes","PROWESS trial","treatment harm","least squares estimation","mortality","sepsis","causal inference"],"falsifier":"Take the reported estimates (t(2)=308, t(3)=205) and form an alternative vector with t(2)=300, t(3)=197, adjusting t(1) and t(4) to keep the expected cell counts identical; both vectors predict the same observed data, so no estimator using only these counts can distinguish them.","tokens_in":10865,"feed_emoji":"⚖️","tokens_out":7171,"duration_ms":63401,"temperature":0.7,"pith_summary":"This paper develops a method to estimate how many participants in a randomized trial were saved by a treatment and how many were killed by it, using only the aggregate counts of deaths and survivals in each arm and the fact that randomization balances unobserved types. Applied to the PROWESS sepsis trial, the method estimates that 18% of participants would be saved and 12% would be killed by the drug, a ratio of two killed for every three saved. The paper argues that the standard reduced-form analysis, which showed a net mortality reduction and supported FDA approval, misses this harm. The estimates imply that about 103 trial participants were actually killed by the intervention.","feed_headline":"Reanalysis: sepsis drug killed 2 for every 3 saved","feed_subtitle":"Aggregate trial data and a new estimator suggest 103 patients died from the drug while 155 were saved by it.","key_machinery":"The key object is the 4x4 matrix N(i,j) assigning each potential outcome type i to observed outcome group j, with eight cells forced to zero by logic. The observed data are expressed as a convolution of four independent binomial random variables, one for each type, with success probability p (the intended intervention fraction). The proposed least squares estimator minimizes a sum of squared randomization errors across all subsets of types, weighting each by the inverse of its variance; this objective, S(N|p), is what selects a specific point along the continuum of type vectors consistent with the aggregate data.","core_discovery":"The paper's central claim is that the numbers of trial participants in each of four potential outcome types—live regardless, saved, killed, die regardless—can be estimated from the experiment's aggregate outcome counts and the intended randomization fraction, without any individual-level covariates or assumptions beyond random assignment. Within the PROWESS trial, it finds 964 participants would live regardless, 308 would be saved, 205 would be killed, and 213 would die regardless, implying a ratio of two killed per three saved and 103 participants actually killed inside the trial. The author presents this as a decomposition of the reduced form, which itself only identifies the net difference between saved and killed.","pith_inferences":["Because the observed data identify only the net difference between saved and killed, the specific 2:3 ratio depends on the arbitrary weighting of randomization errors; a different defensible weighting could shift the split while leaving the reduced form unchanged.","The method's reliance on aggregate counts means it cannot distinguish between participants killed by the drug's mechanism (e.g., bleeding) and those who would have died of sepsis anyway but were classified as killed by the potential-outcome definition; the interpretation of 'killed' is definitional, not causal at the individual level.","The approach could be extended to instrumental-variable settings or to trials with non-binary outcomes by generalizing the type matrix and the binomial convolution.","A bootstrap or sensitivity analysis over different weighting schemes could quantify how much of the estimated ratio is driven by the objective function rather than the data."],"forward_implications":["If the PROWESS estimates are correct, the trial's net benefit of about 6 percentage points masks a gross harm of 12% killed and a gross benefit of 18% saved.","The method provides a template for decomposing reduced-form effects in other randomized trials into saved and killed counts, which could inform risk-benefit assessments.","The finding implies that even a trial that supports regulatory approval can cause substantial harm to a minority of participants, raising ethical questions about informed consent and monitoring.","The paper's estimates are consistent with the subsequent voluntary withdrawal of the drug in 2011 after a confirmatory trial failed to show survival benefit."],"supporting_citations":[{"why":"Supplies the PROWESS trial data and the reduced-form mortality estimates that the method decomposes.","marker":"Bernard et al., 2001"},{"why":"Establishes the potential outcomes framework that defines the four participant types.","marker":"Rubin, 1974"},{"why":"Formalizes causal inference from experiments, providing the basis for treating the reduced form as the difference between killed and saved.","marker":"Holland, 1986"},{"why":"Provides the BARON global optimizer used to compute the least squares estimates.","marker":"Sahinidis, 2018"}],"fun_headline_variants":["Sepsis drug killed 2 for every 3 saved in PROWESS reanalysis","New estimator: sepsis drug killed 103 while saving 155","PROWESS reanalysis: two killed per three saved","Hidden deaths: sepsis drug killed 103 in PROWESS","Potential outcomes model reveals sepsis trial's hidden harm"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The estimator's objective function is assumed to identify the true saved/killed split, but the observed aggregate counts only fix sums such as the number saved plus the number who die regardless, so many different splits fit the data equally well; the specific 2:3 ratio is a consequence of the chosen weighting scheme, not of the data alone.","fun_headline_variants_meta":{"raw":{"variants":["Sepsis drug killed 2 for every 3 saved in PROWESS reanalysis","New estimator: sepsis drug killed 103 while saving 155","PROWESS reanalysis: two killed per three saved","Hidden deaths: sepsis drug killed 103 in PROWESS","Potential outcomes model reveals sepsis trial's hidden harm"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000671,"raw_usage":{"total_tokens":2996,"prompt_tokens":824,"completion_tokens":2172,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":440,"completion_tokens_details":{"reasoning_tokens":2095}},"tokens_in":440,"tokens_out":2172,"duration_ms":16607,"temperature":1.0,"reasoning_tokens":2095,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:05:16.347685+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the reported estimates (t(2)=308, t(3)=205) and form an alternative vector with t(2)=300, t(3)=197, adjusting t(1) and t(4) to keep the expected cell counts identical; both vectors predict the same observed data, so no estimator using only these counts can distinguish them.","supporting_citations":[{"cited_title":"R., J.-L","cited_arxiv_id":null,"evidence_quote":"Supplies the PROWESS trial data and the reduced-form mortality estimates that the method decomposes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the BARON global optimizer used to compute the least squares estimates."}],"review_version":1}