{"id":"377a7154-f9d0-42ed-bd72-8b7e2b504d08","arxiv_id":"2607.03508","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Adding an M(T) belief experiment to a standard Y(T,M) conjoint identifies average marginal direct and indirect effects via doubly-robust machine learning under principal ignorability.","lead":"The paper introduces a conjoint design that adds one simple belief-elicitation experiment so researchers can recover total, direct, and indirect effects of attributes under standard mediation assumptions. This lets analysts separate, for example, whether race affects candidate choice directly or only by changing beliefs about party.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Principal ignorability is only partially testable for the cross-world nested outcomes that identify AMIEs; the sensitivity analysis then relies on an untested symmetry restriction.","rationale":"The reader correctly isolates principal ignorability as the weakest link and notes that it is only partially testable. The refinement above simply makes precise which part of A1 is untested (the cross-world cells that identify AMIEs) and which extra assumption the paper’s own sensitivity procedure then introduces. Because the paper already flags A1, supplies the partial falsification test, and reports a sensitivity analysis, the methodological contribution remains intact under the stated conditions; the verdict therefore stays CONDITIONAL. No deeper inconsistency in the identification argument or the influence-function derivation was found.","tokens_in":37698,"tokens_out":569,"duration_ms":15985,"concrete_test":"Re-run the sensitivity analysis of Appendix D on the pre-registered race and experience contrasts while replacing the symmetry restriction with two alternative extrapolations of γ (e.g., γ(t,t′)=0 for t≠t′, and γ(t,t′)=2·γ(t,t)−γ(t′,t′)). If either alternative moves any of the party-specific AMIEs for Black/white or for political experience across zero or changes their statistical significance relative to the original sensitivity results in Figure 5, the robustness claim for the indirect-effect estimates is weaker than presented.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central identification result (Eq. 1 / mediation formula for α(t,t′)) and the influence function (Eq. 2) both require principal ignorability (A1) for all principal strata, including those that generate the cross-world nested counterfactuals Y_i(t,M_i(t′),s) with t≠t′. The only design-based check the paper supplies (Section 5) compares the diagonal terms α(t,t) against the non-parametrically identified ϕ(t) from the auxiliary Y(T) arm. That check is silent on the off-diagonal terms that enter every AMIE. The subsequent sensitivity analysis (Appendix D) therefore extrapolates the estimated bias function γ from the observed diagonal discrepancies to the unobserved cross-world cells by imposing the additional, untested restriction γ(t,t′,m,x,s)=γ(t′,t,m,x,s). When the diagonal already shows material discrepancies (as it does for race and experience in the application, Figure 4), the credibility of the adjusted AMIEs rests on this symmetry assumption rather than on A1 alone.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"The paper argues that standard conjoint designs identify only controlled effects and cannot recover total or indirect effects of one attribute that operate through beliefs about another. It proposes combining a conventional Y(T,M) conjoint with a separate M(T) experiment that elicits respondents’ beliefs about the mediator, and shows that under principal ignorability (A1), manipulation exclusion (A2), and design-based ignorability/positivity (A3–A5) the average nested potential outcomes α(t,t′) are identified by a mediation formula (Eq. 1). All AMIE, AMDE and AMCE quantities are then linear combinations of these α’s and can be estimated by a doubly robust influence function (Eq. 2) with cross-fitted machine-learning nuisance models. A pre-registered Prolific experiment (N≈4 500) replicating Kirkland & Coppock (2018) illustrates the approach, decomposing candidate-attribute effects through party and supplying a sensitivity analysis that uses the auxiliary Y(T) arm.","tokens_in":37957,"tokens_out":1029,"duration_ms":17381,"significance":"If the identifying assumptions hold, the design fills a genuine gap: researchers can now recover theoretically central mediation quantities that existing conjoint protocols cannot identify even under sequential ignorability. The influence-function estimator, the linear-regression representation of all contrasts (Appendix C), and the partial falsification test via the Y(T) arm are practical contributions. The pre-registered application is well-powered, balance-checked, and shows that eliminated effects need not coincide with indirect effects—an important caution for the literature. Double robustness, cross-fitting, and an explicit sensitivity analysis further strengthen the package. These features make the paper a useful methodological advance for both political science and marketing applications of conjoint analysis.","major_comments":[{"comment":"Section 5 and Appendix D: The only design-based check compares the diagonal terms α(t,t) with the non-parametrically identified ϕ(t) from the Y(T) arm. Every AMIE, however, is a contrast of off-diagonal nested counterfactuals α(t,t′) with t≠t′. The subsequent sensitivity analysis therefore extrapolates the estimated bias function γ from the observed diagonal discrepancies to the unobserved cross-world cells by imposing the additional, untested symmetry restriction γ(t,t′,m,x,s)=γ(t′,t,m,x,s). When the diagonal already exhibits material discrepancies (Figure 4, race and experience), the credibility of the adjusted AMIEs rests on this symmetry assumption rather than on principal ignorability alone. A clearer statement of this limitation, or an alternative sensitivity that does not rely on symmetry, is needed before the application claims can be fully trusted.","section":null},{"comment":"Assumption A1 (principal ignorability) and its role in Eq. 1: The identification of the cross-world terms that enter every AMIE requires that potential outcomes Y_i(t,m,s) are independent of principal-stratum membership G_i given X_i for all strata, including those that generate t≠t′. While the paper correctly notes that a strong predictive model for M reduces the scope for violations and that the Y(T) arm supplies a partial test, the manuscript does not quantify how large residual dependence would have to be to overturn the reported AMIEs, nor does it provide simulation evidence under realistic conjoint designs. Strengthening this discussion (or adding a simple simulation) would make the load-bearing assumption more transparent.","section":null}],"minor_comments":[{"comment":"Notation for multi-valued treatments and mediators is introduced only in the application; a brief statement in Section 3 that the binary exposition extends immediately would help readers.","section":null},{"comment":"Figure 1 and Figure 2 use the same colour scheme for respondent party; a small legend clarification or distinct line types would improve readability when printed in black-and-white.","section":null},{"comment":"Appendix B.7 derives the influence function; a one-sentence pointer in the main text to the fact that the known propensity scores can be plugged in (rather than estimated) would be useful for practitioners.","section":null},{"comment":"A few typographical inconsistencies remain (e.g., “Micha¨ el”, occasional missing spaces around em-dashes). A final proof-reading pass would clean these up.","section":null}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid methods contribution that sits comfortably in stat.ME / political-methodology venues. The principal-ignorability and symmetry issues are real but of the sort that mediation papers routinely navigate; with clearer caveats and a modest expansion of the sensitivity discussion the manuscript should be publishable. No concerns about novelty disclosure or citation patterns."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This paper gives conjoint people a practical way to recover total, direct, and indirect effects of one attribute through another. The new piece is the paired design: keep the usual Y(T,M) conjoint, add a cheap M(T) arm that elicits beliefs about the mediator, then identify the nested potential outcomes under standard mediation assumptions plus a manipulation exclusion restriction. They derive the corresponding doubly-robust influence function that accounts for the missing natural mediator and show how to get average and low-dimensional heterogeneous effects with cross-fitting. The pre-registered Prolific replication of Kirkland & Coppock (N≈4500) is well-powered, balanced, and actually informative: race is heavily mediated by party, experience is mostly direct, and the eliminated effect does not always track the AMIE.\n\nWhat works: identification under A1–A5 is cleanly written (Appendix B), the influence function is standard double-ML adapted to the missing-mediator structure, and they supply a partial falsification check plus a sensitivity analysis that uses the auxiliary Y(T) arm. Citations are appropriate; no circular free parameters drive the main estimates.\n\nThe soft spot is real but ordinary for mediation. Principal ignorability is only partially testable: the Y(T) check speaks only to the diagonal α(t,t). Every AMIE uses the off-diagonal nested terms, and the sensitivity analysis extrapolates the bias function γ by imposing symmetry γ(t,t′)=γ(t′,t). When the diagonal already shows discrepancies (race and experience in Figure 4), the adjusted AMIEs rest on that extra restriction. They are transparent about it and the qualitative story largely survives, but it is the place a referee should press. No public code/data yet is a minor practical gap, not a conceptual one.\n\nThis is for anyone who already runs conjoints and wants mechanism language that is more than “we controlled for party.” It deserves a serious referee. I would engage with it and expect to cite the design and the estimator.","headline":"Clean design fix for a real conjoint problem; principal ignorability remains the load-bearing soft spot, and the sensitivity step leans on an untested symmetry.","tokens_in":38557,"tokens_out":494,"would_cite":true,"duration_ms":6059,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"One extra belief experiment lets conjoint designs recover direct and indirect effects of attributes under standard mediation assumptions.","keywords":["causal inference","factorial design","conjoint experiment","mediation analysis","double machine learning","principal ignorability","average marginal component effect"],"falsifier":"A large, systematic gap between the marginal means obtained from an auxiliary Y(T) experiment and the within-world nested outcomes α(t,t) recovered from the mediation formula that cannot be closed by improving the predictive models for outcome or mediator.","tokens_in":38603,"feed_emoji":"🗳️","tokens_out":815,"duration_ms":13895,"temperature":0.7,"pith_summary":"Conjoint experiments randomize many attributes at once for realism, but that same randomization only recovers controlled effects that deliberately block pathways through other attributes. The paper shows that a second, short experiment—asking respondents which mediator value they associate with each profile—supplies the missing treatment-to-mediator link. With that information plus the usual causal-mediation assumptions, total, direct, and indirect effects become identifiable and can be estimated with doubly robust machine-learning methods. A pre-registered candidate-choice study illustrates the payoff: race effects largely travel through beliefs about party, while political-experience effects are mostly direct. The design therefore gives experimenters a practical route to the quantities they actually care about rather than only the quantities pure randomization delivers.","feed_headline":"One extra belief experiment unlocks direct and indirect effects","feed_subtitle":"Standard conjoints only give controlled effects; a short mediator-belief arm recovers the total and mechanism quantities that matter.","key_machinery":"The mediation formula for the average nested potential outcome α(t,t′), which multiplies the outcome regression from the Y(T,M) arm by the mediator propensity from the M(T) arm and averages over covariates and auxiliary factors; its influence-function representation supplies the doubly robust estimator.","core_discovery":"Combining a standard conjoint in which both treatment and mediator are randomized with a second experiment that elicits respondents’ beliefs about the mediator given the treatments identifies the average nested potential outcomes under principal ignorability and the manipulation exclusion restriction; all average marginal direct, indirect, and total effects then follow as linear combinations of those nested outcomes and admit doubly robust estimators.","pith_inferences":["Existing conjoint datasets can be re-analyzed for mediation by fielding only the inexpensive M(T) arm on a comparable sample, provided conditional exchangeability across samples can be defended.","When several mediators are theoretically relevant, collapsing them into a single joint mediator keeps the identification strategy intact but raises sample-size demands.","The design principle extends beyond conjoints: any experiment that randomizes a bundled treatment can unlock mechanism questions by adding a separate belief-elicitation arm."],"forward_implications":["Researchers can recover total effects of a focal attribute even when a mediator is deliberately randomized for realism.","Indirect effects become estimable, revealing whether an attribute works mainly by changing beliefs about another attribute.","An optional pure Y(T) arm supplies a partial falsification test and a data-driven sensitivity analysis for the untestable assumptions.","Heterogeneous mediation effects can be summarized by ordinary linear regression on the influence-function pseudo-outcomes, with valid standard errors."],"fun_headline_variants":["Belief experiment recovers mediation paths in conjoints","Second arm yields total and indirect conjoint effects","Extra belief task disentangles attribute mechanisms","Nested outcomes identify direct effects via belief data","One simple belief survey unlocks conjoint mediation"],"cache_read_input_tokens":32896,"weakest_assumption_plain":"Once covariates are controlled for, a respondent’s potential outcomes for any fixed treatment and mediator value must not systematically differ by which principal stratum that respondent belongs to.","fun_headline_variants_meta":{"raw":{"variants":["Belief experiment recovers mediation paths in conjoints","Second arm yields total and indirect conjoint effects","Extra belief task disentangles attribute mechanisms","Nested outcomes identify direct effects via belief data","One simple belief survey unlocks conjoint mediation"]},"model":"grok-4.5","effort":"low","cost_usd":0.003358,"raw_usage":{"total_tokens":1085,"prompt_tokens":692,"num_sources_used":0,"completion_tokens":48,"cost_in_usd_ticks":33580000,"prompt_tokens_details":{"text_tokens":692,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":345,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":692,"tokens_out":48,"duration_ms":3565,"temperature":1.0,"reasoning_tokens":345,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T01:59:26.436646+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"A large, systematic gap between the marginal means obtained from an auxiliary Y(T) experiment and the within-world nested outcomes α(t,t) recovered from the mediation formula that cannot be closed by improving the predictive models for outcome or mediator.","supporting_citations":[],"review_version":1}