{"id":"90f8229e-d047-45f3-90a3-00b6f3f2d5f7","arxiv_id":"2607.26309","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Masking treatment-defining words before learning adjustment representations preserves overlap and reduces bias in text-as-treatment causal inference.","lead":"This paper shows that when treatment in a text study is defined by particular words, learned text representations can accidentally encode those words and destroy the statistical overlap needed for causal adjustment. Masking treatment-defining words before representation learning keeps treated and control documents comparable, with proofs for topic models and empirical support for large language models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 4's overlap guarantee relies on an untestable conditional-independence assumption that the LLM variant explicitly violates; the non-estimable sensitivity bound leaves the practical claim supported only by simulation and a penalty that mechanically enforces overlap.","rationale":"The reader's weakest assumption—that a treatment-removing transformation with T⊥W−|Z exists—is exactly the hinge on which Theorem 4 turns. My concern does not displace the reader's; it sharpens it by noting that the paper's own LLM method abandons this assumption, leaving only a non-estimable bound (Theorem 6) and a penalty that mechanically ensures the reported overlap rates. This is a scope limitation, not an internal inconsistency: the theorem is sound, the simulations are extensive, and the limitations section is honest. The proposed incomplete-lexicon test would empirically probe how load-bearing the conditional-independence assumption is. If the guarantee is brittle, the paper should be read as proving a narrow result; if robust, the broader claim gains support. Either way, the CONDITIONAL verdict remains appropriate.","tokens_in":26667,"tokens_out":9918,"duration_ms":95585,"concrete_test":"Run the Section 4.2 LDA simulation with an intentionally incomplete masking lexicon: randomly drop p = 10%, 20%, and 30% of true treatment/control words before deletion, and record the mean fraction of propensity scores in [0.1,0.9] (the Table 4 metric). If overlap remains near the fully-masked level, the conditional-independence assumption is not load-bearing; if the fraction decays toward the unmasked case (99.98% extreme), Theorem 4's guarantee is fragile to small violations of T⊥Wdel|Z, and the practical deletion claim needs an explicit robustness condition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central formal result, Theorem 4, is conditional on T⊥W−|Z. In Section 3.2 the paper states this independence 'holds whenever Z captures any remaining treatment-correlated variation in non-lexicon text'—an untestable assertion. If out-of-lexicon words (e.g., 'great', 'terrible' not in the sentiment lexicon) carry treatment signal, Wdel still predicts T given Z, and the overlap guarantee collapses. For replacement masking—the method actually used with LLMs—the paper concedes T⊥Wmask|Z does not hold, and the only formal support is Theorem 6's sensitivity bound, whose surplus εM is explicitly 'not directly estimable' (Appendix A.5). Thus the LLM method has no testable overlap guarantee; the reported 87–98% in-band propensities are partly produced by the adaptive overlap penalty Lpenalty, which mechanically pushes propensity scores into the band. This is a load-bearing gap between the theorem's restrictive hypothesis and the paper's general 'masking preserves overlap' claim—a gap the paper candidly acknowledges in Section 7 but does not close.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies causal inference when the treatment is a latent linguistic property encoded in the text itself. It argues that representations learned from full text can encode treatment so well that overlap fails, even when the underlying causal model satisfies overlap, and proposes masking treatment-defining lexical tokens before representation learning: deletion masking for bag-of-words/topic models and replacement masking for large language models. It formalizes representation-induced overlap failure (Theorem 2), proves that deletion masking preserves overlap under a conditional-independence assumption (Theorem 4), and gives a sensitivity bound for replacement masking (Theorem 6). Simulations with LDA and DistilBERT on semi-synthetic Amazon reviews, plus a CFPB application, show improved MSE, bias, and overlap statistics relative to unmasked baselines.","tokens_in":1721,"tokens_out":1850,"duration_ms":62657,"significance":"If taken at face value, the paper identifies a genuinely important failure mode and offers a simple, interpretable remedy for lexicon-based text treatments. The formal results are modest but correctly proved under explicit assumptions; the LDA simulation is thorough, with topic-misspecification and parameter-sensitivity checks, and the LLM section includes valuable ablations that separate the contributions of masking and the overlap penalty. The central conceptual contribution—separating treatment-defining lexical signal from adjustment-relevant text—is useful. However, the strongest advertised guarantee applies only to deletion masking when a strong, untestable conditional-independence condition holds. The LLM variant has no formal overlap guarantee: its sensitivity bound depends on an unestimable quantity, and the reported in-band propensity fractions are partly produced by an adaptively tuned penalty that mechanically pushes propensity scores into the band. These caveats are acknowledged in places but not reflected in the abstract and several result presentations.","major_comments":[{"comment":"The theorem's conclusion is entirely conditional on T⊥W⁻|Z. The sentence after the theorem states that this 'holds whenever Z captures any remaining treatment-correlated variation in non-lexicon text,' which is close to restating the assumption and is untestable in practice. If out-of-lexicon words carry treatment information beyond Z, e_b⁻(W) can fall outside [α⋆,1−α⋆]. The LDA simulation enforces the condition by construction—treatment/control topic words are exactly the deleted lexicon—so it cannot validate the general claim in the abstract that deletion masking preserves overlap for bag-of-words/topic-model representations. Please either develop an empirical check for residual dependence or explicitly restrict the claim to settings in which the conditional independence is known or plausibly holds.","section":"Section 3.2, Theorem 4"},{"comment":"Replacement masking is presented as a relaxation with no formal overlap guarantee. Theorem 6 bounds the fraction of propensity scores outside [α⋆−η,1−α⋆+η] by εM/η, but εM is defined via an unobservable conditional expectation eedel_M and is explicitly 'not directly estimable'; α⋆ is also unknown. Thus the bound is structural only and cannot underwrite the reported overlap improvements. Moreover, the 87–98% in-band fractions in Table 2 are partly produced by Lpenalty, whose weight is adaptively updated to target a 90% validation overlap rate (Section 5.1 and Appendix D.1). The ablation in D.6.1 shows masking without the penalty has only 16–27% in-band, so the headline overlap diagnostic largely measures the penalty, not the masking transformation. The authors acknowledge parts of this in Section 7, but the abstract and Section 5.2 still attribute the overlap improvement to masking. Pleas","section":"Section 5 and Appendix A.5"},{"comment":"The adaptive penalty introduces a tuning target (90% validation overlap) and an initial weight λP. Appendix D.8 shows that MSE is non-monotonic in λP^(0): for binary high confounding, the default λP^(0)=100 produces a spike in MSE, while neighboring values perform better. The authors report this sensitivity, but the absence of a principled criterion for choosing the overlap target or the initial penalty means the LLM method's performance depends on a hyperparameter whose default is not derived from the formal theory. This reinforces the gap between the claimed overlap guarantee and the actual procedure.","section":"Section 5.1 and Appendix D.8"}],"minor_comments":[{"comment":"The abstract says deletion masking 'prove[s] that deletion masking preserves overlap for bag-of-words/topic-model representations' without the conditional-independence qualifier. Please add the assumption to the abstract or to the contribution list.","section":"Abstract, Section 4.1"},{"comment":"The table formatting for the grouped columns (full/masked/leace under IPW, OR, AIPW) is hard to read; the column headers appear to be misaligned with the numeric entries. Please use a clearer layout with explicit subcolumn labels.","section":"Table 1"},{"comment":"The Bayes error R_b is unobservable, and the paper notes that any classifier gives an upper bound. This is fine, but the subsequent diagnostic uses estimated propensity scores, not the true e_b. Please clarify in the text that the empirical overlap fraction is an estimate of the quantity in Theorem 2 and is subject to estimation error.","section":"Section 3.1"},{"comment":"References [11] and [12] appear to be the same paper (Gui and Veitch, 'Causal estimation for text data with (apparent) overlap violations') in preprint and published form. Duplicate citation is confusing; retain one or clearly label as preprint/version.","section":"References [11] and [12]"},{"comment":"The auto-calibration of λQ says the outcome and cross-entropy losses 'contribute equally to the gradient,' but the total loss also contains Lpenalty and LU. Please clarify that equalization is only between Lout and LCE, not all terms.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest and the core Theorem 4 is correct under its stated assumptions. The main problem is that the LLM method is presented as preserving overlap while its formal support is a non-estimable sensitivity bound and the reported in-band propensity scores are largely a consequence of the adaptively tuned penalty. If the authors reframe the contribution as (i) a formal result for deletion masking under exact conditional independence and (ii) an empirical heuristic for LLMs with transparent diagnostics and explicit limitations, I would support publication. The current version overstates the overlap guarantee for the LLM setting, so major revision is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nWhat you should know: this paper names a real failure mode—representations learned from full text can encode the treatment and kill overlap—and offers a simple fix (mask the words that define treatment). The formal core is Theorem 4: if T is independent of the transformed text given Z, any representation built from the masked text preserves overlap. That's a clean, correct result, and it's genuinely useful for bag-of-words and topic-model settings. Theorem 2's bound on overlap in terms of Bayes error is also a nice formalization.\n\nThe paper is at its best in the LDA section. The simulations are careful: train/test splits, varied parameters, topic-count misspecification, and multiple estimators. The results are believable and the comparison to LEACE is instructive—it shows that erasing all treatment signal throws out the confounders you need.\n\nThe soft spot is the LLM extension. The independence assumption behind Theorem 4 explicitly does not hold for replacement masking, and the formal fallback (Theorem 6) is a sensitivity bound that depends on an unestimable surplus. The paper says this plainly, but it means the headline guarantee only covers the lexical, bag-of-words case. Also, the adaptive overlap penalty is tuned to target a 90% validation overlap rate, so the reported 87–98% in-band propensities are partly manufactured by the loss, not discovered. To the authors' credit, they ablate this: masking alone removes the degenerate point masses that TI-estimator produces, so the practical story survives even without the penalty. Still, the LLM method is a heuristic with good empirical support, not a theorem.\n\nTwo smaller issues: no code is released, which will matter for applied readers who want to adopt the pipeline. And the real-data application (CFPB) is a single null result—fine as a sanity check, but not much evidence either way.\n\nWho is this for? Anyone doing text-as-treatment with a lexicon-defined treatment. It's not transformative, but it's a clear, practical improvement over methods that ignore the lexical shortcut. I'd want this in the literature, and I'd cite it. Send it to peer review—the theory is sound where it claims to be, the limitations are disclosed, and the simulations are thorough. I'd ask for code and a sharper demarcation of where the guarantee ends and the heuristic begins.\n\nOverall: worthy of a serious referee, likely conditional accept after moderate revision.","headline":"A solid, honestly-scoped method paper: the formal overlap result for deletion masking is correct, the LLM extension is a useful heuristic with a disclosed but non-estimable sensitivity bound.","tokens_in":27445,"tokens_out":1510,"would_cite":true,"duration_ms":18411,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62D20","68T50"],"pacs":[],"model":"deepseek-v4-flash","headline":"Masking treatment-defining words before representation learning prevents the overlap collapse that makes text-based causal adjustment fail.","keywords":["causal inference from text","overlap assumption","treatment-encoding representations","masking","latent text treatments","propensity score","topic models","large language models"],"falsifier":"In a simulation where the true confounders Z are known, compute the Bayes error R⋆ of T from Z and the Bayes error R_b of T from the masked representation. Theorem 2 and Theorem 4 jointly imply R_b ≥ R⋆ whenever masking preserves overlap. A single configuration—for example, a corpus where product-category words are correlated with sentiment and survive deletion—where a classifier on the masked bag-of-words achieves R_b < R⋆ would show the independence premise fails and the guarantee breaks.","tokens_in":26571,"feed_emoji":"🔤","tokens_out":7403,"duration_ms":70803,"temperature":0.7,"pith_summary":"The paper's claim is that in text-as-treatment settings where treatment is encoded by identifiable words—sentiment terms, partisan framing, hedging—the standard practice of learning adjustment representations from full text can break causal inference not by missing confounders but by encoding treatment directly, making treated and control documents separable and destroying overlap. It proposes masking: delete or replace treatment-defining words before representation learning. For bag-of-words/topic models, deletion masking gives a formal guarantee—any representation based on the masked text has propensity scores bounded away from 0 and 1 whenever the true confounders capture remaining treatment-related variation. For large language models, replacement masking is a deliberate relaxation, paired with an adaptive penalty that keeps estimated propensities inside an overlap band. In simulations the masking pipeline keeps 87–98% of propensity scores in a safe range where full-text representations put nearly all scores outside it, and it reduces error relative to both unmasked and concept-erased baselines.","feed_headline":"Mask treatment words first to keep overlap in text causal inference","feed_subtitle":"Full-text embeddings can make treated and control documents separable; masking the defining words keeps them comparable.","key_machinery":"The carrying mechanism is the treatment-removing transformation of the text performed before representation learning. Deletion masking removes treatment-defining tokens so the representation is a function only of remaining words; the proof shows the induced propensity score is a conditional expectation of the true propensity e⋆(Z) given the representation, inheriting the [α⋆,1−α⋆] bounds. Replacement masking for language models replaces the tokens with [MASK] and adds a training-time penalty that activates when estimated propensities leave [α,1−α]; its sensitivity bound (Theorem 6) says overlap degrades smoothly with residual leakage rather than collapsing. The encompassing device is the com","core_discovery":"The central result is Theorem 4: if W− is a treatment-removing transformation with T⊥W−|Z, then any representation b−(W)=h(W−) induces a propensity score P(T=1|b−(W)) inside [α⋆,1−α⋆], the same overlap margin assumed for the true confounders Z. In words, once the lexical shortcut is removed, any remaining treatment predictability in the representation is attributable to confounders and is legitimate for adjustment. Deletion of lexicon words realizes this for bag-of-words/topic models. Replacement masking—substituting [MASK] tokens—does not satisfy the strict independence, so the paper instead provides a sensitivity bound: the fraction of propensities leaving a widened band is at most a treat","pith_inferences":["Inference: The core mechanism—removing the named textual cue before representation learning—should transfer to any treatment with a discoverable cue, such as politeness markers or sentence length, even without a fixed lexicon; the paper tests only lexicon-defined treatments.","Inference: The adaptive overlap penalty alone, without any masking, may be a general-purpose safeguard for text-as-treatment settings where no lexicon exists; the paper's ablation suggests it works but relegates it to an appendix.","Inference: A direct test of the mechanism would compare the empirical Bayes error of masked-text representations to the Bayes error of the true confounders in a simulation; if masked representations ever predict treatment better than the oracle confounders, the assumed independence fails and the guarantee does not apply.","Inference: The non-estimability of the sensitivity bound suggests a practical design rule the paper does not state: in real applications, tune the overlap penalty to hold out a validation set and stop when the in-band fraction stabilizes, since the bound itself cannot be checked."],"forward_implications":["Any full-text representation in a lexicon-defined text-treatment study should be checked for overlap collapse; Theorem 2 shows the fraction of documents with propensities inside [α,1−α] is at most Bayes error divided by α, and Corollary 3 gives exact collapse when treatment is recoverable.","Analysts already using topic-model adjustments can adopt deletion masking with minimal change and obtain a provable overlap guarantee, with simulated gains in IPW particularly large.","Concept-erasure approaches that strip all treatment signal restore overlap but discard confounder information; in the paper's simulations they perform no better than the naive difference in means.","For large language models, replacement masking plus an adaptive overlap penalty improves overlap diagnostics and lowers bias relative to unmasked training, while full-text evaluation of the mask-trained encoder is better than masked evaluation.","Replacement masking carries no formal guarantee; the honest reading is that it converts overlap collapse into a smooth degradation controlled by a non-estimable surplus, so practitioners should report the fraction of propensities in the overlap band."],"fun_headline_variants":["Mask treatment words to avoid overlap failure in text causal inference","Overlap trap: full-text embeddings encode treatment; masking fixes it","Preserve causal overlap by masking treatment tokens","Remove lexical treatment signal to keep propensity scores in check","Masking treatment words stabilizes causal estimates from text"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire overlap guarantee rests on the existence of a transformation with T⊥W−|Z—for deletion masking, that after removing the defining words, no remaining text predicts treatment except through the true confounders; the paper concedes replacement masking does not satisfy this and falls back on a sensitivity bound.","fun_headline_variants_meta":{"raw":{"variants":["Mask treatment words to avoid overlap failure in text causal inference","Overlap trap: full-text embeddings encode treatment; masking fixes it","Preserve causal overlap by masking treatment tokens","Remove lexical treatment signal to keep propensity scores in check","Masking treatment words stabilizes causal estimates from text"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00018,"raw_usage":{"total_tokens":1132,"prompt_tokens":727,"completion_tokens":405,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":471,"completion_tokens_details":{"reasoning_tokens":327}},"tokens_in":471,"tokens_out":405,"duration_ms":3942,"temperature":1.0,"reasoning_tokens":327,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T00:11:43.296247+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In a simulation where the true confounders Z are known, compute the Bayes error R⋆ of T from Z and the Bayes error R_b of T from the masked representation. Theorem 2 and Theorem 4 jointly imply R_b ≥ R⋆ whenever masking preserves overlap. A single configuration—for example, a corpus where product-category words are correlated with sentiment and survive deletion—where a classifier on the masked bag-of-words achieves R_b < R⋆ would show the independence premise fails and the guarantee breaks.","supporting_citations":[],"review_version":1}