Post-treatment problems: What can we say about the effect of a treatment among sub-groups who (would) respond in some way?

Chad Hazlett; Nina McMurry; Tanvi Shinkre

Post-treatment problems: What can we say about the effect of a treatment among sub-groups who (would) respond in some way?

T0 review · 3 major / 2 minor · reviewed 2026-05-22 · grok-4.3

Challenge this review Download PDF Read on arXiv ↗

Pith's one-line read The effect of a treatment among those who would respond in a particular way can be bounded by making assumptions only about the non-responding group.

desk verdict The paper defines TRACE as the treatment effect among units that would show a specific post-treatment response and bounds it by reasoning about the non-reactive complement, but the bounds may rest on assumptions close to those it critiques. read the letter →

arxiv 2505.06754 v4 submitted 2025-05-10 stat.ME

Chad Hazlett , Nina McMurry , Tanvi Shinkre This is my paper

classification stat.ME

keywords causalinferencepost-treatmentbiassubgroupeffectsprincipalstratificationtreatmentresponseaverageeffectboundingmethods

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Many researchers want to know a treatment's effect specifically among units that would show a certain post-treatment response, such as actually implementing an intervention or experiencing a mechanistic change. Conditioning directly on the observed response creates bias, and stronger identification strategies often rest on assumptions that are hard to defend. This paper defines the Treatment Reactive Average Causal Effect as the average effect of treatment among the subgroup that would realize a given value of the post-treatment variable if treated. It shows that plausible ranges for this quantity can be recovered by placing bounds or point values on the effect within the complementary non-reactive subgroup. The approach is illustrated in three applied settings involving policing, community interventions, and canvassing.

What carries the argument

The Treatment Reactive Average Causal Effect (TRACE), which isolates the average causal effect of treatment inside the latent subgroup that would exhibit a specified post-treatment response under treatment.

What would settle it

In the Liberia community-policing example, a new randomized trial that directly measures implementation fidelity and outcomes separately in reactive and non-reactive communities would falsify the approach if the resulting point estimate for the TRACE lies outside the bounds previously reported.

Watch

Extended reading notes

Core claim

The paper defines the Treatment Reactive Average Causal Effect (TRACE) as the total effect of treatment among units that would realize a particular value of a post-treatment variable if assigned to treatment. It establishes that bounds on the TRACE are identified once bounds or point values are placed on the treatment effect inside the non-reactive subgroup, without requiring point identification of the TRACE itself or additional data on the post-treatment variable in the control arm.

Load-bearing premise

Plausible bounds or point values can be placed on the treatment effect inside the non-reactive subgroup without data or assumptions that would themselves deliver point identification of the TRACE.

Editorial extensions

If this is right

In settings such as perceived race during traffic stops, the method can produce point identification rather than bounds when the non-reactive effect is credibly zero.
Applied researchers can report ranges for the effect of an intervention among communities that would meaningfully implement it, rather than among all communities or only observed implementers.
Studies of mechanisms, such as canvassing that shifts feelings toward a group, can bound the effect among those whose feelings would change without assuming the effect is the same for those whose feelings would not change.
The method avoids post-treatment bias while remaining feasible with existing data and modest auxiliary assumptions about the non-reactive group.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the non-reactive effect is itself the object of separate theoretical interest, the same bounding logic could be reversed to learn about that group from knowledge of the reactive group.
The approach may extend to continuous post-treatment variables by discretizing the response or by placing functional-form restrictions on how the effect varies with the post-treatment value.
In panel or repeated-intervention designs, within-unit changes in reactivity could tighten the bounds by providing additional information on the composition of the reactive subgroup.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

read the letter

This paper defines the Treatment Reactive Average Causal Effect (TRACE) as the total effect of treatment among units that would, if treated, realize a particular post-treatment value. It then bounds this quantity by placing restrictions on the effect in the non-reactive group rather than conditioning directly on the observed post-treatment variable or invoking the strong assumptions of principal stratification or instrumental variables approaches. The three examples—police-perceived race and violence, community policing implementation in Liberia, and canvassing effects on transgender attitudes—show the authors have concrete applied settings in view and want to avoid bias from post-treatment selection. The framing is distinct enough from standard tools that it could be useful for researchers who care about mechanistic or implementation responses but cannot defend the usual point-identification assumptions. The paper does a clear job stating the bias problem with direct conditioning and why existing strategies often demand indefensible restrictions. The applications help make the case that this is meant as a practical tool rather than a purely theoretical exercise. The soft spot is the bounding step itself. If credible restrictions on the non-reactive effect can be justified from observables or mild domain knowledge without essentially assuming away the identification problem, the ranges will be informative. If those restrictions turn out to be as strong as the ones the authors criticize elsewhere, the method mainly relabels the difficulty rather than relaxing it. The abstract does not show the derivations or any simulation checks, so the full paper must demonstrate that the bounds are non-vacuous and rest on weaker conditions. This is for applied causal inference people in social science who routinely face post-treatment variables they substantively care about. Readers working on partial identification or new estimands will find the most value. It has enough of a distinct proposal and real examples to deserve serious referee time rather than a desk reject. I would send it for peer review and ask referees to focus on the identification argument and whether the bounds deliver usable ranges in the examples.

Referee Report

3 major / 2 minor

Summary. The paper defines the Treatment Reactive Average Causal Effect (TRACE) as the total effect of treatment among units that would realize a specific value of a post-treatment variable if treated. It argues that direct conditioning on observed post-treatment values is biased and that prior identification strategies rely on indefensible assumptions. The central proposal is that bounds on the TRACE can be obtained by placing plausible restrictions on the treatment effect within the complementary non-reactive subgroup, with the approach illustrated in three empirical examples involving police violence, community policing in Liberia, and canvassing on transgender rights.

Significance. If the bounding procedure can be formalized with explicit, non-circular restrictions that are weaker than those required for point identification, the framework would supply a practical tool for addressing post-treatment conditioning bias in applied causal inference. The three examples demonstrate relevance to policy-relevant questions in criminology and political science where post-treatment variables (implementation, attention, mechanistic response) are common but hard to handle.

major comments (3)

[Abstract and §2 (Identification)] The abstract and introduction assert that reasoning about the non-reactive group yields identifiable ranges for the TRACE, yet no explicit identification result, set of assumptions, or derivation is referenced. The claim that this approach avoids the 'indefensible assumptions' of existing methods is load-bearing and requires a formal proof or theorem statement showing how bounds are obtained from observables alone.
[§3 (Bounding strategy) and Example (i)] The weakest assumption listed—that plausible bounds or point values can be assigned to the treatment effect in the non-reactive subgroup—risks circularity if those bounds are themselves justified by cross-world or monotonicity restrictions equivalent in strength to those criticized in prior work. A concrete example or sensitivity analysis demonstrating that the resulting TRACE interval is strictly wider than point-identified alternatives under weaker conditions is needed.
[Example (i)] In the police-perceived race example, the paper states that point identification 'may be possible.' The conditions under which the bounds collapse to a point (and whether those conditions are substantively weaker than standard monotonicity or exclusion restrictions) should be stated explicitly, with a comparison to existing estimators.

minor comments (2)

[§2] Notation for the non-reactive subgroup and the TRACE parameter should be introduced with a clear table or diagram relating observed and counterfactual quantities.
[Examples section] The three empirical illustrations would benefit from a side-by-side comparison table showing the width of the TRACE bounds relative to naive conditioning and to at least one existing alternative method.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the detailed and constructive report. The comments identify important opportunities to strengthen the formal presentation of the identification strategy and to clarify comparisons with existing approaches. We address each major comment below and will incorporate revisions accordingly.

read point-by-point responses

Referee: [Abstract and §2 (Identification)] The abstract and introduction assert that reasoning about the non-reactive group yields identifiable ranges for the TRACE, yet no explicit identification result, set of assumptions, or derivation is referenced. The claim that this approach avoids the 'indefensible assumptions' of existing methods is load-bearing and requires a formal proof or theorem statement showing how bounds are obtained from observables alone.

Authors: We agree that the manuscript would be improved by an explicit theorem statement. In the revision we will add a formal identification result in Section 2 that states the minimal assumptions on the non-reactive subgroup under which the TRACE bounds are identified from observables, together with the derivation. This will also make explicit how the maintained restrictions differ in strength from those required for point identification in the literature we critique. revision: yes
Referee: [§3 (Bounding strategy) and Example (i)] The weakest assumption listed—that plausible bounds or point values can be assigned to the treatment effect in the non-reactive subgroup—risks circularity if those bounds are themselves justified by cross-world or monotonicity restrictions equivalent in strength to those criticized in prior work. A concrete example or sensitivity analysis demonstrating that the resulting TRACE interval is strictly wider than point-identified alternatives under weaker conditions is needed.

Authors: We acknowledge the risk of circularity and will address it directly. The revision will include a sensitivity analysis in the police-perceived-race example that varies the width of the interval placed on the non-reactive subgroup effect and shows that the resulting TRACE bounds remain strictly wider than point-identified alternatives obtained under stronger monotonicity or exclusion restrictions. We will also add text clarifying that the restrictions are justified by substantive knowledge about the non-reactive units rather than cross-world assumptions that apply to the full population. revision: yes
Referee: [Example (i)] In the police-perceived race example, the paper states that point identification 'may be possible.' The conditions under which the bounds collapse to a point (and whether those conditions are substantively weaker than standard monotonicity or exclusion restrictions) should be stated explicitly, with a comparison to existing estimators.

Authors: We will revise the discussion of Example (i) to state the precise conditions under which the bounds collapse to a point (namely, when the treatment effect in the non-reactive subgroup is restricted to a singleton or very narrow interval on substantive grounds). We will compare these conditions explicitly to standard monotonicity and exclusion restrictions, noting that they apply only to the non-reactive subgroup and therefore impose weaker requirements on the data-generating process. A short comparison table with existing estimators will be added. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: TRACE bounds derived from independent reasoning on non-reactive subgroup

full rationale

The paper defines TRACE as the total treatment effect among units that would realize a specific post-treatment value if treated. It then states that bounds on TRACE follow from separate reasoning about the treatment effect in the complementary non-reactive subgroup. No equations, fitted parameters, or self-citations are shown that reduce the claimed bounds back to the TRACE itself or to the reactive-group effect by construction. The identification strategy therefore rests on external restrictions or observable implications that are not presupposed to equal the target quantity, rendering the derivation self-contained rather than circular.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

Abstract-only review prevents exhaustive enumeration; the approach implicitly relies on standard causal assumptions plus a new modeling choice for the non-reactive effect.

assumptions (2)

domain assumption No unmeasured confounding for the treatment-outcome relationship within strata defined by potential post-treatment response.
Standard for any causal effect claim; invoked when defining the total effect within the reactive group.
ad hoc to paper Plausible values or bounds can be assigned to the treatment effect in the non-reactive group.
This is the key step that converts the problem into a range for TRACE; location not specified in abstract.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Post-treatment problems: What can we say about the effect of a treatment among sub-groups who (would) respond in some way?." pith.science (2026). https://pith.science/paper/2505.06754

@misc{pith2026250506754,
  author       = {Pith},
  title        = {Pith review of: Post-treatment problems: What can we say about the effect of a treatment among sub-groups who (would) respond in some way?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2505.06754}},
  note         = {Machine review of arXiv:2505.06754}
}

read the original abstract

Investigators are often interested in how a treatment affects an outcome for units responding to treatment in a certain way. We may wish to know the effect among units that, for example, meaningfully implemented an intervention, passed an attention check, or demonstrated some important mechanistic response. Simply conditioning on the observed value of the post-treatment variable introduces problematic biases. Further, the identification assumptions required of several existing strategies are often indefensible. We propose the Treatment Reactive Average Causal Effect (TRACE), which we define as the total effect of treatment in the group that, if treated, would realize a particular value of the relevant post-treatment variable. By reasoning about the effect among the "non-reactive" group, we can identify and estimate the range of plausible values for the TRACE. We demonstrate the use of this approach with three examples: (i) learning the effect of police-perceived race on police violence during traffic stops, a case where point identification may be possible; (ii) estimating effects of a community-policing intervention in Liberia, in communities that meaningfully implemented it, and (iii) studying how in-person canvassing affects support for transgender rights, among participants for whom the intervention would result in more positive feelings towards transgender people.

Discussion (0). Sign in to comment.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

TRACE ≡ E[Y(1)−Y(0)|M(1)=1] ... By reasoning about the effect among the non-reactive group, we can identify and estimate the range of plausible values for the TRACE.
IndisputableMonolith/Foundation/AbsoluteFloorClosure.lean absolute_floor_iff_bare_distinguishability unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

no-assumption trimming bounds ... monotonicity-based trimming bounds

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Pith tools

Reviewed May 22, 2026 · model on record in the stance chip above.