{"id":"f2e2b390-6aea-4736-a81c-40b0d4eb538f","arxiv_id":"2501.13777","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A survey-weighted pseudo-likelihood extension of the Mixture-of-Unigrams model reduces bias in topic proportions under informative sampling.","lead":"This paper adds survey weights to a classic text topic model so that open-ended survey responses represent the population instead of the sample. The authors show in a simulation that the weighted version sharply reduces bias in estimated topic shares, and they apply it to 2016 and 2020 ANES responses.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Weighted-MoU bias correction is demonstrated only under oracle weights tied to the true latent topic; the paper does not show the pseudo-likelihood still corrects topic proportions under realistic estimated or proxy weights.","rationale":"The reader's weakest assumption is that the sample design is fully captured by reported weights and that the pseudo-likelihood is valid for mixture models. I agree with the spirit, but I sharpen it: even granting the pseudo-likelihood construction, the simulation validates only oracle weights derived from the true latent topic. The weighted score-equation logic behind the method is sound when weights are true inverse probabilities, so I do not regard the paper as internally inconsistent or the central idea as implausible. The gap is that the paper presents the method for real surveys while testing it only in the most favorable possible setting. The hMoU is a further unvalidated contribution: no simulation checks whether group-level topic proportions or their uncertainties are recovered. The reader's CONDITIONAL verdict is therefore appropriate: release code, add a robustness simulation with realistic weights, and simulate the hMoU. My concern does not move the verdict; it explains why the conditions should be required.","tokens_in":15986,"tokens_out":13031,"duration_ms":137431,"concrete_test":"Re-run the Section 3 simulation under a design where inclusion probabilities depend on an observed covariate X_d that is correlated with the latent topic (e.g., logit pi_d = gamma X_d with X_d generated from a topic-dependent distribution), estimate weights by fitting a propensity model to X_d to mimic real survey weights, and compare RMSE and absolute bias of theta against the unweighted MoU and the oracle-weighted MoU. If the estimated-weight version does not approach the oracle's RMSE of 0.0458, the paper's bias-correction claim is limited to oracle weights and should be qualified accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3, the sampling design is generated as a function of the latent topic z_d, and the weights used in the pseudo-likelihood are inverse probabilities of that oracle design. Under such a design, the weighted score equation sum_d omega_d * d log L_d(theta,phi)/d theta is design-unbiased for the population score, so bias reduction is expected and is not a stringent test of the method. The paper imports the exponentiated pseudo-likelihood from Parker et al. (2023) and Savitsky and Toth (2016) without deriving the conditions under which it is valid for mixture models. In real surveys, weights are estimated and are functions of observed auxiliary variables, not of latent topic membership. If selection is informative with respect to topic even after conditioning on those auxiliary variables, the reported weights are not the inverse probabilities needed for unbiasedness, and the weighted MoU may only partially correct the bias. The central claim that incorporating survey weights 'corrects the bias' is therefore established only in the oracle-weight, single-design case. The ANES application provides no credible intervals and no sensitivity analysis for weight misspecification. The hierarchical MoU is introduced with no simulation at all, so the group-level comparisons in Figures 4-8 and 12-16 have no calibration check. This is a boundary-condition concern, not an internal inconsistency: the point-estimation logic is sound under oracle weights, but the paper's practical claims are stronger than the evidence provided.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a survey-weighted pseudolikelihood version of the Mixture of Unigrams (MoU) topic model, in which each document's likelihood is raised to the power of its scaled survey weight. The authors argue that this accounts for informative sampling and reduces bias in estimates of topic proportions. A simulation study under a selection mechanism that depends on the true latent topic shows large reductions in RMSE and absolute bias for the topic-proportion vector θ when weights are used. The paper also introduces a hierarchical MoU (hMoU) model in which topic proportions are linked to document-level fixed and random effects through a multinomial logit form, and it applies both models to ANES 2016 and 2020 open-ended responses on the most important national problem. The application compares weighted and unweighted MoU topic proportions and reports hMoU topic proportions across age, gender, race, and state groups.","tokens_in":16257,"tokens_out":3554,"duration_ms":35866,"significance":"If the pseudolikelihood approach is valid beyond the oracle-weight simulation, the paper addresses a real gap: open-ended survey text is common, and ignoring informative sampling can distort topic proportion estimates. The simulation is a clean proof-of-concept: selection depends directly on the latent topic, and the reported weights are the inverse selection probabilities, so the large bias reduction (θ RMSE from 0.2419 to 0.0458; absolute bias from 0.2202 to 0.004) is a meaningful demonstration of mechanism. The hMoU extension is a natural and potentially useful way to study how topics vary across respondent groups. However, the paper's central practical claim is conditional on assumptions that are not stated, the hMoU is never validated in simulation, and the ANES results lack uncertainty quantification. These issues limit the strength of the conclusions but are addressable in revision.","major_comments":[{"comment":"The hierarchical MoU model is a central contribution, but it is never validated in simulation. The paper states that the ANES application 'demonstrates the effectiveness' of hMoU, yet all reported group comparisons are point estimates with no credible intervals. Consequently, statements such as 'Topic 1 is more prominent among females than males' or 'Topic 6 is notably more prominent among the Black race group' cannot be distinguished from sampling noise, and the hMoU estimation procedure itself could be biased or poorly identified without detection. Please add a simulation study that generates data from the hMoU generative process with known group differences and checks point estimation and interval coverage, and report posterior intervals (or at least posterior standard deviations) for the key topic-proportion comparisons in the ANES application.","section":"Section 4, Figures 4-8 and 12-16"},{"comment":"The pseudolikelihood p(w_d | θ, φ) = [Σ_z p(z|θ) Π_n p(w_{d,n}|z, φ)]^{ω_d} is imported from Parker et al. (2023) and Savitsky and Toth (2016) without a statement of the conditions under which this exponentiated likelihood is valid for mixture models. In the simulation, the selection probabilities are functions of the true latent topic indicator z_d, and the weights are the oracle inverse probabilities of that design. Under such a design, bias reduction is expected from standard design-based weighting. In real surveys, weights are estimated from auxiliary variables and may not equal the inverse of selection probabilities conditional on latent topic membership. To make the practical claim that the weighted MoU 'corrects the bias' under informative sampling, please state the required assumptions (e.g., sampling ignorability given the weights or a specified selection model) and provide at least one additional simulation in which weights are estimated from observed covariates or are partially misspecified, to assess the robustness of the bias correction.","section":"Section 2.2, Eq. (2); Section 3"},{"comment":"The comparisons between the weighted and unweighted MoU models, and the cross-group comparisons in the hMoU application, are based only on point estimates of topic proportions. For example, the paper reports that the COVID/healthcare topic proportion is 0.3996 under the unweighted MoU and 0.5320 under the weighted MoU, and that this is 'an important finding', but no uncertainty interval is provided for either estimate. Similarly, Figures 4-8 and 12-16 show posterior mean proportions without intervals. Because MCMC output is already available, adding 95% credible intervals or posterior intervals for the displayed proportions and for selected group differences would materially strengthen the inference and prevent readers from interpreting posterior noise as substantive differences.","section":"Section 5.1 and 5.2"}],"minor_comments":[{"comment":"The text refers to 'Table 3' when comparing the models, but the table is labeled Table 1; please correct the cross-reference.","section":"Section 3, Table 1"},{"comment":"Table 1 reports a single RMSE, absolute bias, and interval score for the vector θ, but θ has multiple components (topics). Please clarify whether these metrics are averaged over topics or computed for the vector as a whole, and consider reporting per-topic values, which are more informative.","section":"Section 3"},{"comment":"The identifiability constraint is stated as 'we constrain βJ and ηJ to be equal to the zero vector', but η is a scalar concentration parameter in the Dirichlet prior for φ; the constraint should presumably be on γ_J (the random effects vector). Please fix this typo and state all identifiability constraints completely.","section":"Section 4"},{"comment":"There are several typographical errors and informal phrases, including 'the the', 'accross', 'distince', 'hyperparamter', 'unfordable' in Figure 9, and the repeated phrase 'MoU model accounting for informative sampling' where a shorter reference would improve readability. A careful proofread is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable application of an established pseudolikelihood idea to topic models, with a clean but narrow simulation and a descriptive application to public ANES data. The main concern for scope is that the hMoU contribution—a substantial part of the title and abstract—has no validation beyond an illustrative application with point estimates only. I would advise the editor that a revision with a hMoU simulation study and uncertainty quantification in the applications would substantially raise the paper's value to the journal's readership. The absence of code or reproducibility information is also worth noting, though not disqualifying."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the weighted MoU is a real, contained contribution and the simulation supports the main claim, but the hierarchical model is illustrated, not validated, and the ANES results have no uncertainty attached. I'd send it to review, expecting revision.\n\nWhat's actually new: applying the survey-weight pseudo-likelihood to the Mixture of Unigrams, and the hierarchical extension with document-level fixed and random effects. The ingredients are known—Parker et al. and Savitsky-Toth supply the weighting; Nigam et al. supply MoU—but the combination for free-text survey responses is not in the literature. That's incremental but useful for survey agencies.\n\nThe simulation in Section 3 is the strongest part. It's clean: a population, an informative design, weights that are true inverse probabilities, and 100 replications. The RMSE for theta drops from 0.24 to 0.046 and absolute bias from 0.22 to 0.004. That's the central claim, and it holds in that setting.\n\nThe soft spots are real, though. First, the stress-test note is on target: the weights are oracle weights, built from the latent topic assignment. So the simulation is a best-case demonstration, not a test of how the method behaves when weights are estimated from auxiliary variables and selection is only partially explained. The pseudo-likelihood is imported from earlier work without derivation for mixture models, which is acceptable as a modeling choice, but the boundary conditions should be stated.\n\nSecond, the hMoU is introduced in Section 4 and then applied to ANES with no simulation at all. Figures 4-8 and 12-16 are point estimates without credible intervals, so the group differences (race, gender, state) cannot be separated from noise. The authors report interval scores in the simulation but do not give intervals for the application. Third, no code or data are released, and the preprocessing is described in one sentence.\n\nThe ANES application itself is fine as an illustration: two datasets, sensible topics, and a plausible difference between weighted and unweighted results. But it's not a validation.\n\nBottom line: this is a competent paper for survey statisticians who want a simple weighted topic model. The authors need to add a simulation for the hMoU, provide uncertainty quantification for the applied results, and acknowledge the oracle-weight limitation. With those, it would be a solid applied methods paper. I'd take it for peer review but not without revision.","headline":"Weighted MoU is a sound small contribution with clean simulation support, but the hierarchical extension and applied claims outrun the evidence.","tokens_in":16805,"tokens_out":3223,"would_cite":false,"duration_ms":27181,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62D05","62F15","62P25"],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding survey weights to a Mixture-of-Unigrams model removes bias in estimated topic proportions.","keywords":["topic modeling","Mixture of Unigrams","informative sampling","survey weights","pseudo-likelihood","open-ended survey responses","hierarchical topic model","ANES data"],"falsifier":"Run the same simulation but make selection depend on a specific keyword inside a topic (for example, over-sampling documents that contain 'covid' within the health topic) while keeping the reported weights as in the paper. If the weighted MoU still recovers the true topic-proportion vector $\\theta$ with bias near 0.004, the correction is robust; if bias grows to the level of the unweighted model, the pseudo-likelihood fails when selection is word-level rather than topic-level.","tokens_in":15735,"feed_emoji":"📊","tokens_out":6902,"duration_ms":55246,"temperature":0.7,"pith_summary":"Open-ended survey responses come from complex sampling designs in which some respondents are more likely to be selected than others, yet standard topic models treat every document as equally representative. The paper proposes a Mixture-of-Unigrams model that corrects for this by raising each document's likelihood to the power of its survey weight, a pseudo-likelihood adjustment imported from design-based inference. In the simulation, the correction cuts the RMSE of the topic-proportion vector from 0.2419 to 0.0458 and the absolute bias from 0.2202 to 0.004, while estimation of the word-probability matrix is essentially unchanged. A hierarchical extension with fixed and random effects in the topic proportions allows comparisons across gender, race, age, and state. The payoff, if the method is right, is that surveys like the ANES can produce unbiased statements about what the population thinks is the most important problem, rather than what the realized sample says.","feed_headline":"Survey weights cut topic-proportion bias from 0.22 to 0.004","feed_subtitle":"Weighted Mixture-of-Unigrams fixes open-ended survey analysis; RMSE falls from 0.2419 to 0.0458.","key_machinery":"The object that carries the argument is the exponentiated pseudo-likelihood: each document's Mixture-of-Unigrams likelihood is raised to the power of its scaled survey weight $\\omega_d$. This re-weights every document's contribution to the posterior so that the target of inference is the population rather than the realized sample. The same exponentiated weight sits at the document level of the hierarchical MoU, where topic proportions are modeled through a multinomial logit link on fixed and random effects, with a symmetric Dirichlet prior on the word distributions $\\phi_j$; the weighting is what does the bias correction, while the hierarchy supplies group-level structure.","core_discovery":"The central claim is that when selection probabilities depend on the latent topic structure, the standard MoU likelihood misestimates the population topic-proportion vector $\\theta$ because the sample over-represents some topics. The proposed fix replaces the document likelihood with a survey-weighted pseudo-likelihood $p(\\mathbf{w}_d \\mid \\theta, \\phi) = \\left[\\sum_z p(z\\mid\\theta) \\prod_{n=1}^{N_d} p(w_{d,n}\\mid z,\\phi)\\right]^{\\omega_d}$, where $\\omega_d$ are the scaled survey weights. Under this model the simulation recovers $\\theta$ with RMSE 0.0458 and absolute bias 0.004, compared with 0.2419 and 0.2202 for the unweighted MoU. The paper also claims that the same pseudo-likelihood can be embedded in a hierarchical MoU, with topic proportions built from fixed effects for gender, race, and age plus state-level random effects, so that group comparisons account for the sampling design.","pith_inferences":["Editorial inference: the exponentiated-likelihood construction is modular, so the same survey-weight correction could be applied to other document-level topic models such as LDA; the paper does not demonstrate this.","Editorial inference: the correction is only as good as the weights. A decisive test is to simulate selection that depends on a rare word within a topic rather than on topic membership alone; if the weighted model then still recovers $\\theta$, the approach is robust, and if not, the assumption that weights capture all selection dependence is violated.","Editorial inference: the state-level random effects make the hierarchical model a natural tool for small-area estimation of topic prevalence, a direction the paper notes but does not develop with design-based uncertainty measures.","Editorial inference: comparing weighted topic proportions against human coding of the same open-ended responses would test whether the correction improves substantive accuracy, not just numerical recovery in simulation."],"forward_implications":["Analysts can report population-level topic proportions from open-ended survey questions instead of sample-level proportions that over-weight oversampled subgroups.","In the ANES 2020 application, the leading COVID/healthcare/inequality topic rises from 0.3996 unweighted to 0.5320 weighted, showing that design correction can change substantive conclusions.","Word-topic distributions $\\phi$ are largely unaffected by informative sampling in the simulation, so the main correction is needed for topic proportions, not for the topical vocabulary.","The hierarchical weighted MoU gives a way to compare topic prevalence across age, gender, race, and state while still accounting for the sample design.","The method transfers directly to confidential micro-data from complex surveys, since it only requires the reported survey weights and the text."],"supporting_citations":[{"why":"Supplies the exponentiated pseudo-likelihood construction for unit-level survey models that this paper imports into the Mixture-of-Unigrams likelihood.","marker":"Parker et al. (2023)"},{"why":"Establishes Bayesian estimation under informative sampling via survey-weighted likelihoods, the posterior basis for the weighted MoU.","marker":"Savitsky and Toth (2016)"},{"why":"Introduces the pseudo-likelihood approach for complex surveys, the general estimation strategy behind the weighting.","marker":"Binder (1983)"},{"why":"Provides the survey-weight reweighting formulation that turns the sample likelihood into a population likelihood.","marker":"Skinner (1989)"},{"why":"Defines the Mixture-of-Unigrams model that the paper extends to informative sampling.","marker":"Nigam et al. (2000)"},{"why":"Supplies the interval score used to compare predictive intervals across the weighted and unweighted models.","marker":"Gneiting and Raftery (2007)"}],"fun_headline_variants":["Weighted MoU cuts topic bias from 0.22 to 0.004","Survey-weighted topic model eliminates sampling bias","Pseudolikelihood saves topic models from survey skew","Fix for biased topic proportions in complex surveys","Topic modeling that respects survey weights"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported survey weights fully capture the sampling design and that raising each document's MoU likelihood to the power of its weight is a valid basis for posterior inference; if the weights are miscalibrated or selection depends on words inside a document beyond topic membership, the correction is incomplete.","fun_headline_variants_meta":{"raw":{"variants":["Weighted MoU cuts topic bias from 0.22 to 0.004","Survey-weighted topic model eliminates sampling bias","Pseudolikelihood saves topic models from survey skew","Fix for biased topic proportions in complex surveys","Topic modeling that respects survey weights"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000577,"raw_usage":{"total_tokens":2759,"prompt_tokens":1020,"completion_tokens":1739,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":1665}},"tokens_in":636,"tokens_out":1739,"duration_ms":10607,"temperature":1.0,"reasoning_tokens":1665,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:35:41.766209+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same simulation but make selection depend on a specific keyword inside a topic (for example, over-sampling documents that contain 'covid' within the health topic) while keeping the reported weights as in the paper. If the weighted MoU still recovers the true topic-proportion vector $\\theta$ with bias near 0.004, the correction is robust; if bias grows to the level of the unweighted model, the pseudo-likelihood fails when selection is word-level rather than topic-level.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes Bayesian estimation under informative sampling via survey-weighted likelihoods, the posterior basis for the weighted MoU."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the pseudo-likelihood approach for complex surveys, the general estimation strategy behind the weighting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the survey-weight reweighting formulation that turns the sample likelihood into a population likelihood."},{"cited_title":"K., Thrun, S., and Mitchell, T","cited_arxiv_id":null,"evidence_quote":"Defines the Mixture-of-Unigrams model that the paper extends to informative sampling."},{"cited_title":"and Raftery, A","cited_arxiv_id":null,"evidence_quote":"Supplies the interval score used to compare predictive intervals across the weighted and unweighted models."}],"review_version":1}