{"id":"2625ddfe-35e5-4a15-9579-2c68a9efd7da","arxiv_id":"2607.05792","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":8.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":3,"one_line_summary":"Logging the features and relative probability of one unexposed item alongside the exposed item identifies causal effects of content features from stochastic algorithms even with unobserved confounders.","lead":"This paper introduces a method for estimating causal effects of content features (like chatbot tone) from data generated by stochastic algorithms such as recommendation systems or LLMs. By logging the features of one unexposed item and the relative probability of the exposed vs. unexposed pair, platforms can turn routine user interactions into valid micro-experiments that identify causal effects even with unobserved confounders.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Identification is clean; the load-bearing concern is whether within-pair assignment is truly randomized in production systems, as the reader identified. No additional concern found.","rationale":"The identification argument is correct and cleanly proven. I verified the key step in Theorem 1: for discordant pairs, P(h=δ|pair) = P(h=-δ|pair) = 1/2 by Assumption 1, so the distribution of pairs is identical across the two conditioning groups regardless of confounding in pair selection. The reader correctly identified Assumption 1 as the load-bearing premise, and I agree with the reader's assessment that the theoretical contributions are substantial and the proposed data structures are practically feasible. The concern about Assumption 1 is a practical one, not a theoretical flaw. In the LLM case, the pair structure is created by design (two API calls, random selection), so Assumption 1 holds by construction. This makes the method a randomized experiment, with the novelty lying in the estimand and data structure rather than in the identification per se. In recommendation systems, Assumption 1 might fail if the final selection incorporates additional signals correlated with outcomes. The C-TACE's local nature (conditional on discordant pairs) is a limitation but is clearly acknowledged and carefully worked through in the linear example (Section 10), which shows when the C-TACE coincides with the per-feature effect and when it captures a bundled effect. The lack of empirical validation is a reasonable concern, but the theoretical framework is sound. The paper provides a diagnostic for Assumption 2 (Section 9.4) but not for Assumption 1; a balance check on pre-treatment covariates would fill this gap. The reader's verdict of ACCEPT with HIGH confidence is appropriate.","tokens_in":32381,"tokens_out":17368,"duration_ms":773875,"concrete_test":"Among discordant pairs (D_i^h ≠ 0), test whether pre-treatment covariates X_i are balanced between the D_i^h = 1 and D_i^h = -1 groups. By Lemma 3, D_i^h ⊥ (X_i, g(V_i, V_i^N)) | D_i^h ≠ 0 under Assumption 1. A standard two-sample balance check (e.g., standardized mean differences) on X_i between the two groups provides a necessary condition: if covariates are imbalanced, Assumption 1 is violated and identification fails. This is analogous to balance checks in randomized experiments and can be implemented on any dataset with logged covariates. The paper's placebo test (Section 10.8) checks a related but distinct condition; this balance check directly tests the randomization assumption itself.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader correctly identifies Assumption 1 as the load-bearing premise. The identification argument (Theorem 1) is correct: for discordant pairs (h ≠ 0), the within-pair randomization ensures that the distribution of pairs is identical in the h=δ and h=-δ groups, so E[Y_i|h=δ] - E[Y_i|h=-δ] identifies E[τ_i|h=δ]. The proof is clean and I verified the key step: for each pair with |h*|=δ, P(h=δ|pair)=P(h=-δ|pair)=1/2 by Assumption 1, so the distribution of pairs—and hence potential outcomes—is the same in both groups regardless of confounding in pair selection. The concern is whether Assumption 1 holds in practice. In the LLM case (Section 9), the pair is created by the analyst (two API calls, random selection), so Assumption 1 holds by construction—but this means the method is essentially a randomized experiment, and the contribution lies in the estimand (C-TACE) and data structure (LCE) rather than in the identification itself. In recommendation systems, the within-pair selection might incorporate real-time signals that correlate with outcomes, violating Assumption 1. One subtle point: the C-TACE is a local estimand for the subpopulation with discordant pairs. This subpopulation is selected by the algorithm's behavior, which can be confounded. The 'even in the presence of unobserved confounders' claim concerns internal validity (the within-pair comparison is valid), not external validity (the C-TACE may not generalize to the full population). The paper is transparent about this, but the distinction merits emphasis. The paper provides a diagnostic for Assumption 2 (Section 9.4) but no analogous diagnostic for Assumption 1.","agreement_with_reader":"agree"},"referee_report":{"model":"glm-5.2","summary":"This paper proposes a new method for estimating causal effects of content features using observational data generated by stochastic algorithms. The key insight is that when a stochastic algorithm selects content for a user, the randomization within the unit-specific treatment set can be exploited for causal identification, provided two additional data elements are logged: the identity of at least one unexposed item and the relative probability of the exposed versus unexposed item. The authors introduce the Logged Counterfactual Exposures (LCE) data structure and the Conditional Treatment-Averaged Causal Effect (C-TACE) estimand, which averages unit-level treatment effects over both users and the pairs of items the algorithm generates. The paper proves identification (Theorems 1-4) under a randomized assignment assumption (Assumption 1), extends the framework to unequal assignment probabilities and estimated probabilities via replays (Section 9), and provides a linear data-generating example (Section 10) that concretely illustrates the estimand's properties relative to naive regression, unconfoundedness adjustment, and negative-sample approaches.","tokens_in":33186,"tokens_out":1385,"duration_ms":178573,"significance":"The paper makes a substantive contribution to causal inference for stochastic algorithm outputs. The identification results (Theorems 1-4) are clean and the derivations are verifiable. The linear example (Section 10) with replication code in the supplement is a strength, as it makes the subtle interpretation of the C-TACE estimand concrete and demonstrates the properties of the estimator relative to alternatives. The inverse sampling design for estimated assignment probabilities (Section 9.3, Lemma 5, Theorem 4) is an elegant contribution that provides exact unbiasedness of the weight. The practical discussion of how LCE data can be obtained from LLM log-probabilities, replayed mechanisms, and recommendation system candidate sets adds real-world relevance. The core claim—that causal effects of content features are identified even with unobserved confounders in pair selection, given within-pair randomization and the logged relative probability—is well-supported by the proofs.","major_comments":[{"comment":"Section 9.2, trimming population definition: The trimmed population P(κ) is defined in terms of q_i, the probability that the display mechanism produces the high-feature side. However, q_i is a function of the unit-specific treatment set A_i and potential outcomes, which are not fully observed. The paper does not clearly specify how q_i is to be evaluated or estimated for the purpose of defining the trimmed population in practice. The estimator q̂_i from replays is introduced, but the relationship between trimming on the true q_i (which defines the estimand) and trimming on the estimated q̂_i (which is what would be feasible) is not fully addressed. This is load-bearing for the estimated-probability regime (Theorem 4), where the feasible estimator's properties are established on P(κ). Clarifying whether trimming is on q_i or q̂_i, and what the consequences are for the former case, would ","section":null},{"comment":"Section 10: The linear example is valuable for illustrating the estimand's properties, but it is purely a simulation/theoretical exercise. The paper lacks any empirical evaluation on real data, even from a small-scale or illustrative setting. Given that the paper's motivation heavily emphasizes practical applicability to LLMs and recommendation systems, and that the authors claim LCE data is 'practically obtainable at scale' (Section 13), at least one demonstration on real or semi-synthetic data would substantially strengthen the paper's contribution and address concerns about feasibility. This is particularly relevant for the replay-based estimation (Section 9), where practical issues like API costs, model version drift, and the accuracy of the diagnostic (Section 9.4) are discussed but not empirically tested.","section":null}],"minor_comments":[{"comment":"Assumption 1 (Section 3) states that the assignment probability is 1/2 conditional on the full set of variables including potential outcomes. The text later refers to 'random assignment' and 'randomized assignment' in ways that could be read as unconditional. Making the conditional nature of the randomization more explicit in the prose surrounding Assumption 1 would improve clarity.","section":null},{"comment":"Section 5, Definition of pseudo potential outcomes: The definitions of V^h_i(-1), V^h_i(1), Y^h_i(-1), Y^h_i(1) involve case distinctions based on the sign of h(V_i(0), V_i(1)). The notation is somewhat dense and the connection to the observed data (Y_i, V_i, V^N_i) could be made more explicit. A brief remark explaining how these pseudo potential outcomes relate to the observed quantities for a unit with D^h_i != 0 would help readers.","section":null},{"comment":"Section 7 (Multiple TCFs): The discussion of what can and cannot be learned with multiple treatment-characterizing functions is insightful but somewhat informal. A more formal statement of the identification results (or lack thereof) for the multi-TCF case, perhaps as a theorem or proposition, would improve clarity.","section":null},{"comment":"Section 12.1: The expansion of the unconditional difference in means Y_{V_i=1} - Y_{V_i=0} into three terms is algebraically detailed but could benefit from a more concise summary of the key takeaway. The main point—that the causal contrast is contaminated by imbalance and incomparability bias from tied pairs—could be stated more prominently before the derivation.","section":null},{"comment":"Typo in Section 10.5: The unconfoundedness regression coefficient is denoted β̂_unc in some places and the text refers to 'the unconfoundedness analysis of Section 12' before Section 12 is formally introduced. Consider forward-referencing more carefully.","section":null},{"comment":"Figure captions: Figures 1-6 are referenced in the text but the captions themselves are somewhat terse. Adding more detail to the figure captions, particularly regarding what the axes and different lines represent, would improve readability for readers who skim the figures.","section":null}],"recommendation":"minor_revision","confidential_remarks":"The paper is by senior and well-known authors in the causal inference field. The core identification result is correct and the contribution is genuine. The main gap is the lack of empirical validation, which is unusual for a paper that emphasizes practical applicability so heavily. I recommend minor revision because the identification theory is sound and the linear example partially addresses the gap, but the authors should be encouraged to add at least a small real-data illustration if at all possible. The self-citation pattern (Athey et al. 2026, Agrawal et al. 2026) appears to be to companion/related papers and is not problematic."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive report. The referee raises two major comments: (1) clarification of the trimmed population definition in Section 9.2, specifically whether trimming is on the true q_i or the estimated q̂_i, and (2) the absence of an empirical evaluation on real data. We agree that both points warrant attention and will revise the manuscript accordingly.","responses":[{"response":"The referee correctly identifies an ambiguity in Section 9.2 that we will resolve in the revision. To clarify the intended logic: the estimand is defined on the population P(κ) = {i : κ ≤ q_i ≤ 1−κ}, where q_i is the true probability that the display mechanism produces the high-feature side for unit i's configuration. This is the population-level quantity that defines the target of inference. In practice, q_i is not directly observed, but it can be estimated via replays (Regime III) or from logged counterfactual exposures (Regime II). The feasible estimator trims on the estimated q̂_i rather than the true q_i, and the referee is right that the manuscript does not currently make this distinction explicit or discuss the consequences. We will revise Section 9.2 to address this in three ways. First, we will state clearly that the estimand is defined by trimming on the true q_i, while the feasible estimator trims on q̂_i. Second, we will note that when q̂_i is obtained from a fixed number M of replays, misclassification of the trimming boundary can occur for units whose true q_i is near κ or 1−κ. This introduces a boundary effect: the feasible trimmed population differs from the target population by units whose q_i is within the estimation error of the threshold. The resulting estimand is defined on a slightly different population, but the difference vanishes as M grows (or as r grows in the sequential design of Section 9.3), because q̂_i converges to q_i. Third, and most importantly, we will clarify that the inverse sampling design of Section 9.3 sidesteps this issue entirely: under the sequential design, the weight T_i/r is exactly unbiased for 1/P_i at every r ≥ 1 (Lemma 5), and Theorem 4 establishes that the feasible estimator identifies the same estimand on P(κ) without","revision_made":"no","referee_comment":"Section 9.2, trimming population definition: The trimmed population P(κ) is defined in terms of q_i, the probability that the display mechanism produces the high-feature side. However, q_i is a function of the unit-specific treatment set A_i and potential outcomes, which are not fully observed. The paper does not clearly specify how q_i is to be evaluated or estimated for the purpose of defining the trimmed population in practice. The estimator q̂_i from replays is introduced, but the relationship between trimming on the true q_i (which defines the estimand) and trimming on the estimated q̂_i (which is what would be feasible) is not fully addressed. This is load-bearing for the estimated-probability regime (Theorem 4), where the feasible estimator's properties are established on P(κ). Clarifying whether trimming is on q_i or q̂_i, and what the consequences are for the former case, would."},{"response":"We agree with the referee that an empirical demonstration would strengthen the paper and address practical feasibility concerns. We will add a real-data illustration in the revision. Specifically, we plan to implement the LCE framework using an open-weight LLM (e.g., Llama or Mistral) in a setting where we generate pitches for a lending scenario similar to the running example in the paper. Because we control the generation pipeline, we can log the exposed response, sample counterfactual responses via replays at the same prompt and temperature, extract features (e.g., concreteness, tone), and compute relative probabilities from token-level log-probabilities. This setup will allow us to demonstrate the full pipeline: constructing LCE data, estimating the C-TACE, implementing the replay-based estimation of Section 9.3, and applying the diagnostic of Section 9.4. We will report API/compute costs, the number of replays used, and the behavior of the diagnostic, directly addressing the practical concerns the referee raises. We note that a fully observational dataset from a production system is not available to us, which is why we focus on a setting where we control the stochastic algorithm. However, we believe this semi-synthetic demonstration, where the data-generating mechanism is a real LLM rather than a parametric simulation, will substantively address the feasibility question. We will also add discussion of model version drift: since we use an open-weight model hosted locally, the algorithm remains fixed across replays, which is the ideal case for Assumption 2. We will discuss how the diagnostic of Section 9.4 can detect violations when this condition is not met, as would be the case with a hosted API whose model version may change. We believe this addition will address","revision_made":"no","referee_comment":"Section 10: The linear example is valuable for illustrating the estimand's properties, but it is purely a simulation/theoretical exercise. The paper lacks any empirical evaluation on real data, even from a small-scale or illustrative setting. Given that the paper's motivation heavily emphasizes practical applicability to LLMs and recommendation systems, and that the authors claim LCE data is 'practically obtainable at scale' (Section 13), at least one demonstration on real or semi-synthetic data would substantially strengthen the paper's contribution and address concerns about feasibility. This is particularly relevant for the replay-based estimation (Section 9), where practical issues like API costs, model version drift, and the accuracy of the diagnostic (Section 9.4) are discussed but not empirically tested."}],"tokens_in":32240,"tokens_out":1227,"duration_ms":209496,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"The main thing to know: this paper introduces a genuinely new identification strategy for causal effects of content features using data from stochastic algorithms. The core insight is that if you log the identity of one unexposed item alongside the exposed one, plus the relative probability of the pair, you can identify a causal effect even with unobserved confounders driving which items the algorithm considered. The identification (Theorem 1) is clean and I checked the key step: for discordant pairs, within-pair randomization ensures the distribution of pairs is identical in the h=δ and h=-δ groups, so the difference in means identifies the within-pair causal effect. No circularity. The estimand (C-TACE) is a local contrast averaged over the pairs the algorithm actually produces, which the paper is transparent about. The linear example in Section 10 is well done — it makes the estimand concrete, shows what the naive regression and unconfoundedness approaches recover, and demonstrates the placebo test cleanly. The practical discussion of how to obtain LCE data from LLM APIs (token log-probabilities, replays) is timely and concrete. The inverse-sampling design (Section 9.3) using negative binomial stopping is a nice touch — exact unbiasedness for the weight at any r≥1. The related literature is thorough and the positioning relative to off-policy evaluation, negative sampling, and compound treatment literatures is accurate. The distinction from embedding-level propensity methods (Saito-Joachims) is clearly drawn. The soft spots are real but proportionate. First, no empirical evaluation on real data. The linear example is illustrative, not validating. For a paper claiming practical feasibility at scale, this is a gap. Second, the stress-test note raises a point I agree with: in the LLM case, the analyst creates the pair by making two API calls and randomly selecting, so Assumption 1 holds by construction — but then the method is essentially a randomized experiment, and the contribution is the estimand and data structure, not the identification. In recommender systems, within-pair selection may incorporate real-time signals that correlate with outcomes, violating Assumption 1, and the paper provides a diagnostic for Assumption 2 (reproducible randomization) but not for Assumption 1. Third, the C-TACE is local to the discordant subpopulation, which is algorithm-selected and potentially confounded. The paper acknowledges this but the external validity limitation is sharper than the tone suggests. These are the kind of gaps a good referee process should push on, not reasons to dismiss the paper. The identification is correct, the estimand is well-motivated, and the practical relevance to LLMs and recommender systems is clear. This is a theory paper from authors who know what they are doing, and the core contribution — using relative pair probabilities rather than marginal propensities for identification — is new. It deserves a serious referee.","headline":"New identification result for causal effects from stochastic algorithm outputs; clean theory, no empirical validation, but the core insight is genuinely novel and the proofs hold up.","tokens_in":33189,"tokens_out":1289,"would_cite":true,"duration_ms":48789,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Two extra log fields turn AI randomness into causal evidence","keywords":["causal inference","stochastic algorithms","counterfactual exposures","treatment effects","recommendation systems","large language models","identification","micro experiments"],"falsifier":"If the within-pair display decision is correlated with the potential outcomes conditional on the pair—for example, because a final ranking step uses real-time engagement predictions that also drive the outcome—then the two subpopulations (those who received the higher-feature item and those who received the lower-feature item from a discordant pair) are no longer comparable, and the difference in their mean outcomes does not have a causal interpretation. The placebo test in Section 10.8 can detect this: a nonzero coefficient on the non-shown item's features after partialling out the shown item","tokens_in":32452,"feed_emoji":"🎲","tokens_out":1865,"duration_ms":183813,"temperature":0.7,"pith_summary":"This paper proves that the causal effect of content features (like the tone of an LLM response or the price of a recommended product) can be recovered from routine user interactions with stochastic algorithms, without running purpose-built experiments and without observing all confounders, provided that two additional pieces of information are logged per interaction: the identity of at least one item the algorithm could have shown but did not, and an estimate of the relative probability of the shown versus unshown item. The central object is Logged Counterfactual Exposures (LCE) data, and the central mechanism is that the stochastic algorithm's internal randomization over which item to display, conditional on having selected a pair, acts as a valid experiment of size one (a micro experiment) even when the selection of the pair itself is confounded by unobserved user traits. The estimand, called the Conditional Treatment-Averaged Causal Effect (C-TACE), averages unit-level causal contrasts over both users and the pairs of items the algorithm naturally produces. Theorem 1 establishes that this estimand is identified as a simple difference in mean outcomes between two subpopulations: those who received the higher-feature item from their pair and those who received the lower-feature item, among pairs where the feature differs. The paper further shows that when exact assignment probabilities are not logged, they can be estimated by replaying the same stochastic algorithm (e.g., re-querying an LLM at the same prompt and temperature), and an inverse-sampling design yields exactly unbiased weights with modest replay costs. The practical consequence is that platforms already running LLMs or recommendation systems can, at the cost of a second API call or logging a candidate set, generate causal evidence about content features from data they already collect in the ordinary course of business.","feed_headline":"Two extra log fields turn AI randomness into causal evidence","feed_subtitle":"Logging what an algorithm could have shown, plus the relative probability, identifies causal effects of content features even with unobsired","key_machinery":"The treatment characterizing function (TCF) h(v, v') is an anti-symmetric function that maps the features of the exposed and unexposed items to a scalar contrast. It defines subpopulations of comparable units (those with h = δ versus h = -δ). The pseudo-experiment construction transforms the observed quadruple (Y_i, V_i, V_i^N, X_i) into data resembling a standard randomized experiment by assigning a three-valued pseudo-treatment based on the sign of the TCF. Under Assumption 1 (randomized within-pair assignment), the pseudo-treatment is independent of potential outcomes, and the C-TACE reduces to a difference in conditional expectations of the observed outcome.","core_discovery":"The paper's core discovery is that the randomization a stochastic algorithm uses to decide which item to display, given a set of candidates, is sufficient to identify causal effects of content features even in the presence of unobserved confounders that affect both which candidates the algorithm considers and how users respond. The identification does not require a model of the assignment mechanism, observation of all content features, or adjustment for user characteristics. It requires only that, for each user, the features of one unexposed alternative and the relative display probability of the pair are recorded. The C-TACE estimand averages over the pairs the algorithm actually produces,禀","pith_inferences":["The C-TACE's interpretation as a bundled effect (averaging over correlated features that co-vary with the target feature in the algorithm's output) means it answers a policy question that standard factorial experiments cannot: not 'what is the effect of changing this feature in isolation' but 'what happens if we steer the algorithm toward this feature as it currently operates.' This makes it direc","The framework could be extended to sequential or multi-turn interactions, where the 'content feature' of interest is a property of the conversation trajectory rather than a single response. The micro-experiment logic would still apply if the algorithm's stochastic branching points are logged, though the pair structure would need to generalize to paths.","The reproducibility requirement (Assumption 2) creates a natural tension with the pace of model deployment: as LLM providers deprecate older model versions, the window for valid replays closes. This suggests a practical imperative to log counterfactual exposures at the time of original interaction rather than relying on retroactive replay, even though the paper shows both regimes are valid in prin"],"forward_implications":["Platforms deploying LLMs can estimate the causal effect of interpretable content features (e.g., friendliness, concreteness) on user engagement by making a second API call per interaction to log the unexposed response and its relative probability, converting routine traffic into causal evidence.","The placebo test (Section 10.8) provides a practical diagnostic: if the non-shown item's features predict the outcome after controlling for the shown item's features, the analyst has evidence that selection bias is present and the identification assumptions may be violated.","The inverse-sampling design (Section 9.3) means that even when exact probabilities are unavailable, a platform can replay its algorithm a small number of times per interaction (r between 5 and 10) and obtain unbiased causal estimates with only a few percent variance inflation, making the method feasible for production-scale systems.","The framework extends naturally to comparing LLMs: by randomly forwarding one of two models' responses to each user and logging features of both, the relative quality of models can be assessed from single-response ratings rather than head-to-head rankings."],"fun_headline_variants":["Algorithm randomness enables causal inference from recommendation logs","Two logged fields unlock causal estimates from stochastic algorithm data","Unexposed alternatives identify content feature effects without full confounder control","Logging near-miss items turns stochastic algorithms into causal engines","Relative display probabilities suffice for causal identification under unobserved confound"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The load-bearing premise is that, once the algorithm has selected its candidate pair for a given user, the final choice of which item to display is a fair coin flip (or a flip with known probability) that is independent of how the user would respond to either item. In production systems, the final ranking step often incorporates additional real-time signals or context that could correlate with user outcomes, which would break this assumption. A second premise, relevant when概率","fun_headline_variants_meta":{"raw":{"variants":["Algorithm randomness enables causal inference from recommendation logs","Two logged fields unlock causal estimates from stochastic algorithm data","Unexposed alternatives identify content feature effects without full confounder control","Logging near-miss items turns stochastic algorithms into causal engines","Relative display probabilities suffice for causal identification under unobserved confounders"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":703,"prompt_tokens":622,"completion_tokens":81,"prompt_tokens_details":null},"tokens_in":622,"tokens_out":81,"duration_ms":14213,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T23:48:38.759468+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If the within-pair display decision is correlated with the potential outcomes conditional on the pair—for example, because a final ranking step uses real-time engagement predictions that also drive the outcome—then the two subpopulations (those who received the higher-feature item and those who received the lower-feature item from a discordant pair) are no longer comparable, and the difference in their mean outcomes does not have a causal interpretation. The placebo test in Section 10.8 can detect this: a nonzero coefficient on the non-shown item's features after partialling out the shown item","supporting_citations":[],"review_version":1}