REVIEW 2 major objections 4 minor 17 references
Disentangling Causal Mechanisms in Conjoint Experiments Using Mediation
T0 review · 2 major / 4 minor · reviewed 2026-07-12 · grok-4.5
Pith's one-line read One extra belief experiment lets conjoint designs recover direct and indirect effects of attributes under standard mediation assumptions.
desk verdict Clean design fix for a real conjoint problem; principal ignorability remains the load-bearing soft spot, and the sensitivity step leans on an untested symmetry. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mediation formula for the average nested potential outcome α(t,t′), which multiplies the outcome regression from the Y(T,M) arm by the mediator propensity from the M(T) arm and averages over covariates and auxiliary factors; its influence-function representation supplies the doubly robust estimator.
What would settle it
A large, systematic gap between the marginal means obtained from an auxiliary Y(T) experiment and the within-world nested outcomes α(t,t) recovered from the mediation formula that cannot be closed by improving the predictive models for outcome or mediator.
Extended reading notes
Core claim
Combining a standard conjoint in which both treatment and mediator are randomized with a second experiment that elicits respondents’ beliefs about the mediator given the treatments identifies the average nested potential outcomes under principal ignorability and the manipulation exclusion restriction; all average marginal direct, indirect, and total effects then follow as linear combinations of those nested outcomes and admit doubly robust estimators.
Load-bearing premise
Once covariates are controlled for, a respondent’s potential outcomes for any fixed treatment and mediator value must not systematically differ by which principal stratum that respondent belongs to.
Editorial extensions
If this is right
- Researchers can recover total effects of a focal attribute even when a mediator is deliberately randomized for realism.
- Indirect effects become estimable, revealing whether an attribute works mainly by changing beliefs about another attribute.
- An optional pure Y(T) arm supplies a partial falsification test and a data-driven sensitivity analysis for the untestable assumptions.
- Heterogeneous mediation effects can be summarized by ordinary linear regression on the influence-function pseudo-outcomes, with valid standard errors.
Reading between the lines
- Existing conjoint datasets can be re-analyzed for mediation by fielding only the inexpensive M(T) arm on a comparable sample, provided conditional exchangeability across samples can be defended.
- When several mediators are theoretically relevant, collapsing them into a single joint mediator keeps the identification strategy intact but raises sample-size demands.
- The design principle extends beyond conjoints: any experiment that randomizes a bundled treatment can unlock mechanism questions by adding a separate belief-elicitation arm.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that standard conjoint designs identify only controlled effects and cannot recover total or indirect effects of one attribute that operate through beliefs about another. It proposes combining a conventional Y(T,M) conjoint with a separate M(T) experiment that elicits respondents’ beliefs about the mediator, and shows that under principal ignorability (A1), manipulation exclusion (A2), and design-based ignorability/positivity (A3–A5) the average nested potential outcomes α(t,t′) are identified by a mediation formula (Eq. 1). All AMIE, AMDE and AMCE quantities are then linear combinations of these α’s and can be estimated by a doubly robust influence function (Eq. 2) with cross-fitted machine-learning nuisance models. A pre-registered Prolific experiment (N≈4 500) replicating Kirkland & Coppock (2018) illustrates the approach, decomposing candidate-attribute effects through party and supplying a sensitivity analysis that uses the auxiliary Y(T) arm.
Significance. If the identifying assumptions hold, the design fills a genuine gap: researchers can now recover theoretically central mediation quantities that existing conjoint protocols cannot identify even under sequential ignorability. The influence-function estimator, the linear-regression representation of all contrasts (Appendix C), and the partial falsification test via the Y(T) arm are practical contributions. The pre-registered application is well-powered, balance-checked, and shows that eliminated effects need not coincide with indirect effects—an important caution for the literature. Double robustness, cross-fitting, and an explicit sensitivity analysis further strengthen the package. These features make the paper a useful methodological advance for both political science and marketing applications of conjoint analysis.
major comments (2)
- Section 5 and Appendix D: The only design-based check compares the diagonal terms α(t,t) with the non-parametrically identified ϕ(t) from the Y(T) arm. Every AMIE, however, is a contrast of off-diagonal nested counterfactuals α(t,t′) with t≠t′. The subsequent sensitivity analysis therefore extrapolates the estimated bias function γ from the observed diagonal discrepancies to the unobserved cross-world cells by imposing the additional, untested symmetry restriction γ(t,t′,m,x,s)=γ(t′,t,m,x,s). When the diagonal already exhibits material discrepancies (Figure 4, race and experience), the credibility of the adjusted AMIEs rests on this symmetry assumption rather than on principal ignorability alone. A clearer statement of this limitation, or an alternative sensitivity that does not rely on symmetry, is needed before the application claims can be fully trusted.
- Assumption A1 (principal ignorability) and its role in Eq. 1: The identification of the cross-world terms that enter every AMIE requires that potential outcomes Y_i(t,m,s) are independent of principal-stratum membership G_i given X_i for all strata, including those that generate t≠t′. While the paper correctly notes that a strong predictive model for M reduces the scope for violations and that the Y(T) arm supplies a partial test, the manuscript does not quantify how large residual dependence would have to be to overturn the reported AMIEs, nor does it provide simulation evidence under realistic conjoint designs. Strengthening this discussion (or adding a simple simulation) would make the load-bearing assumption more transparent.
minor comments (4)
- Notation for multi-valued treatments and mediators is introduced only in the application; a brief statement in Section 3 that the binary exposition extends immediately would help readers.
- Figure 1 and Figure 2 use the same colour scheme for respondent party; a small legend clarification or distinct line types would improve readability when printed in black-and-white.
- Appendix B.7 derives the influence function; a one-sentence pointer in the main text to the fact that the known propensity scores can be plugged in (rather than estimated) would be useful for practitioners.
- A few typographical inconsistencies remain (e.g., “Micha¨ el”, occasional missing spaces around em-dashes). A final proof-reading pass would clean these up.
Circularity Check
No circularity: identification of nested potential outcomes and mediation effects follows from standard assumptions plus two independent randomized experiments; estimation is orthogonal double ML with no free parameters fitted to the target quantities.
full rationale
The paper's central identification result (Eq. 1) is the classic mediation formula adapted to a split-sample design that separately randomizes (T,M,S) for the outcome regression and (T,S) for the mediator propensity; under the explicitly stated Assumptions A1–A5 it recovers the average nested potential outcomes α(t,t′) by construction of the potential-outcome algebra, not by fitting the targets themselves. All AMIE/AMDE/AMCE quantities are then linear combinations of those α’s (Table 1). The influence function (Eq. 2) is the standard doubly-robust score for this functional (Kennedy 2024; Farbmacher et al. 2022), estimated by cross-fitting nuisance functions that are never themselves the objects of interest. The optional sensitivity analysis (Appendix D) post-hoc extrapolates a bias function γ from the observable diagonal discrepancies ϕ(t)–α(t,t) under an additional symmetry restriction; it does not redefine or force the primary estimates. There are no self-definitional loops, no parameters fitted to a subset and then “predicted,” no load-bearing uniqueness theorems imported from the authors’ prior work, and no renaming of known empirical patterns. The derivation is therefore self-contained against the external literature on mediation and double machine learning.
Assumptions & free parameters
free parameters (1)
- linear coefficients θ in the sensitivity function γ
assumptions (4)
- domain assumption Principal ignorability (A1): Y_i(t,m,s) ⊥ G_i | X_i
- domain assumption Manipulation exclusion restriction (A2): potential outcomes do not depend on which experimental arm the unit is assigned to
- standard math Ignorability and positivity of experiment type, treatments, and mediator (A3–A5)
- standard math SUTVA, consistency, and composition of potential outcomes
Cite this review
Pith. "Pith review of Disentangling Causal Mechanisms in Conjoint Experiments Using Mediation." pith.science (2026). https://pith.science/paper/NLAJC2BK
@misc{pith2026260703508,
author = {Pith},
title = {Pith review of: Disentangling Causal Mechanisms in Conjoint Experiments Using Mediation},
year = {2026},
howpublished = {\url{https://pith.science/paper/NLAJC2BK}},
note = {Machine review of arXiv:2607.03508}
}
read the original abstract
Conjoint experiments provide an attractive way to assess the role of multiple attributes simultaneously on decision-making. However, the randomization of multiple attributes prevents understanding the causal mechanisms that, critically, depend on the relationship between attributes -- e.g., how one attribute affects the respondent's belief as to another attribute. This is because conjoint experiments recover controlled effects whereas a substantively important estimand may be the total or indirect effect of one attribute. Unfortunately, existing experimental designs for conjoint experiments cannot estimate these effects. We provide an alternative framework that requires one additional, simple experiment to learn the relationship between attributes among respondents alongside the standard assumptions for causal mediation. Estimation of the relevant effects can be done in a doubly robust fashion using machine learning methods. We illustrate this by conducting a pre-registered experiment on candidate choice and disentangle the effect of different attributes by understanding their mediation through the candidate's party.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
General Targeted Machine Learning for Modern Causal Mediation Analysis
“General Targeted Machine Learning for Modern Causal Mediation Analysis.”arXiv preprint arXiv:2408.14620. Montgomery, Jacob M and Santiago Olivella
-
[2]
InProceedings of the Seventeenth Confer- ence on Uncertainty in Artificial Intelligence
Direct and Indirect Effects. InProceedings of the Seventeenth Confer- ence on Uncertainty in Artificial Intelligence. Morgan Kaufmann pp. 411–420. Rao, Vithala R. 2014.Applied Conjoint Analysis. Berlin Heidelberg: Springer. Rathbun, Brian C., Christopher Sebastian Parker and Caleb Pomeroy
2014
-
[3]
Semiparametric estimation for causal mediation analysis with multiple causally ordered mediators
“Semiparametric estimation for causal mediation analysis with multiple causally ordered mediators.”Journal of the Royal Statistical Society Series B: Statistical Methodology84(3):794–821. 44 A Additional Illustration There are many scenarios in political science and marketing where the primary treatment variable affects beliefs about other characteristics...
2018
-
[4]
finds that it is highly variable whether an indirect effect accompanies an eliminated effect. Thus, while we do not wish to speak directly to this debate, it shows that estimating me- diation quantities (e.g., indirect effects) is relevant to current empirical work; our paper provides an analytical and empirical framework (given the requisite assumptions)...
2010
-
[5]
Further, assume that they only wish to vote for Black candidates if they are Democrats
shows the difference between a controlled direct effect and a natural direct effect: Imagine there is a respondent who believes a candidate is a Democrat if and only if they are Black. Further, assume that they only wish to vote for Black candidates if they are Democrats. A controlled direct effect exists: If we tell them the candidate is a Democrat, the ...
2014
-
[6]
where one notes that (t,s) and (t ′,s) defines a contrast of (high-dimensional) treatments. Marginalizing over unitsiand auxiliary treatmentssgives a decomposition of the average marginal total effect into an AMCE (i.e., an average marginal controlled direct effects), an average marginal reference interaction, an average marginal mediated interaction, and...
2014
-
[7]
No Interactions
provides the identifiable estimand and analogous application of Kennedy (2024)’s results would produce an influence function. B.4 Effect of “No Interactions” As the text notes, without anM(T)-study, even principal ignorability (or sequential ignora- bility) is insufficient to identify a direct and indirect effect. This is because no information is learned...
2024
-
[8]
E[Yi(t, m,s)|1{M i(t′,s) =m},X i =x]× Pr (Mi(t′,s) =m|X i =x)×Pr (X i =x)f(S=s) # X s,x,m
from theY(T)-experiment and the controlled direct effect or average marginal component effect onTfrom theY(T, M)-experiment (Imai, Tingley and Yamamoto, 2013; Acharya, Blackwell and Sen, 2018). The cost of doing so is assuming that the controlled direct effect equals the direct effect, i.e., assuming away the existence of a reference interaction (VanderWe...
2013
Show all 17 references
-
[9]
These are known as associative strata (Frangakis and Rubin, 2002; Forastiere, Mattei and Ding, 2018). Equation 10 decomposes the average marginal indirect effectδ(t) as follows δ(t) =E[Y i(t, Mi(1),S i)−Y i(t, Mi(0),S i)] = X g E[Yi(t, Mi(1),S i)−Y i(t, Mi(0),S i)|Gi =g]×Pr(G ...
2002
-
[10]
Our proposed influence function thus nearly exactly coincides with Farbmacher et al. (2022) once we adjust for the propensity score of being in each experiment, i.e.,A i ∈ {0,1}and account for the missing data in each experiment recall that ifA i = 0,Y i is missing and ifA i =...
2022
-
[11]
We also consider the case where some other distributionfoverS i is used
Given this influence function, the estimator proposed in the main text follows as either a one-step estimator or by solving the implied estimation equation forα(t, t ′) 6 (Kennedy, 2024). We also consider the case where some other distributionfoverS i is used. Given that fis a...
2024
-
[12]
no carry-over
that is likely more efficient but requires Monte Carlo integration to deal with the third term. For simplicity, we thus prefer the first option in our analyses. 7 B.8 Crossover Design Principal ignorability (Assumption A1) is often viewed as a strong assumption. We note that t...
2013
-
[13]
¯ξcan be found similarly, e.g., 1/|T |in the (t, t∗)-position for allt ∗ ∈ Tand−1/|T |in the (t ′, t∗)-position for allt ∗ ∈ T
wherez 1 has a ‘1’ in the position of (t, t′) and zero otherwise.ξ(t ∗) =α(t, t ∗)−α(t ′, t∗) can be found by azwith ‘1’ in the (t, t∗)-position and ‘-1’ in the (t ′, t∗)-position. ¯ξcan be found similarly, e.g., 1/|T |in the (t, t∗)-position for allt ∗ ∈ Tand−1/|T |in the (t ...
2022
-
[14]
C.1 Heterogeneous Effects Section 6.2 notes that the best linear approximation to the conditional expectation function can be found using linear regression
Simple argumentation also shows that these results apply whenthas more than two levels. C.1 Heterogeneous Effects Section 6.2 notes that the best linear approximation to the conditional expectation function can be found using linear regression. To establish this, we define the...
2021
-
[15]
15It can also be shown, somewhat tediously, that an identical estimator can be obtained using results from Kennedy (2024) to find the influence function of the corresponding population linear regression coefficient of the conditional effect function as the outcome andw i as th...
2024
-
[16]
The proof is as follows: Theorem 1 states three equivalent ways of estimatingδ(1; 1)−δ(1; 0). Approach 1 in this instance provides a single regression estimator that is equivalent to a single regression with linear predictor as follows, noting that since the sum in Equation 19...
2021
-
[17]
Prolific
by supposing the existence of a functionγ(t, t ′, m,x,s) that captures the magnitude of the bias that is driven by (i) differences in average potential outcomes between principal strata and (ii) differences in average potential outcomes due to direct manipulation of the mediat...
2018
Reviewed July 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.