Estimating Propensities of Selection for Big Datasets via Data Integration

Anders Holmberg; Bronwyn Loong; Lyndon Ang; Robert Clark

REVIEW 4 major objections 6 minor 2 references

Estimating Propensities of Selection for Big Datasets via Data Integration

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proposes the Overlap-Excluded propensity score estimator, and claims that with a correctly specified selection model the resulting inverse-probability-weighted estimates are unbiased and more efficient than four standard…

desk verdict A clean, incremental variant of propensity weighting for linked non-probability data; the simulation is promising but the central efficiency theorem is missing and even its stated setting doesn't match the simulation design. read the letter →

arxiv 2501.04185 v1 pith:6XQF2S3Q submitted 2025-01-07 stat.ME

Lyndon Ang , Robert Clark , Bronwyn Loong , Anders Holmberg This is my paper · ORCID

classification stat.ME MSC 62D0562J12

keywords bigdataselectionbiaspropensityscoreestimationinverseprobabilityweightingintegrationnon-probabilitysamplesreferencesampleoverlap-excludedestimator

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Big-data sources such as administrative and sensor records cover the population unevenly, so using them directly gives biased estimates of means and totals. The paper's route to correction is to integrate the big dataset with a small probability reference sample, estimate each big-data unit's propensity to be included, then weight by the inverse of that propensity. The proposed Overlap-Excluded (OE) estimator builds the propensity model from two pieces: every unit in the big dataset contributes its own inclusion log-likelihood, and the reference units that are not also in the big dataset contribute the non-inclusion log-likelihood, weighted by their design weights. The paper claims that, under a correctly specified selection model, the OE inverse-probability-weighted estimator is unbiased and, in simulations with large big-data samples, has lower relative root mean squared error than four established alternatives while keeping coverage near nominal levels. If that holds, statistical agencies can get more precise official statistics from large non-probability datasets without collecting new probability samples.

What carries the argument

The central object is the Overlap-Excluded (OE) pseudo-likelihood, defined as $l_{OE}(\theta) = \sum_{i \in B} \log \pi^B_i + \sum_{i \in A\setminus B} d^A_i \log(1 - \pi^B_i)$, where $\pi^B_i$ is the propensity of inclusion in the big dataset $B$, $A$ is the probability reference sample, and $d^A_i$ are its design weights. The first sum uses all big-data units directly; the second uses only reference units that are not in $B$, which is the key difference from previous methods that use all of $A$ for the second term. Its score equation sets a weighted sum of covariate vectors to zero and is solved by Newton-Raphson; the solution plugs into a logistic model $\pi^B_i = \exp(x_i^\top \theta)/(1 + \exp(x_i^\top \theta))$ and then into the inverse-probability-weighted mean estimator. The mechanism that carries the argument is the removal of the overlap $A \cap B$ from the non-inclusion term, which avoids double-counting the already observed big-data units and makes the estimating equation closer to the population likelihood.

What would settle it

In the paper's simulation design, deliberately misclassify 5% of the overlap indicators in the reference sample and compare the OE IPW estimates with those from CLW under the same corrupted linkage; if OE's bias exceeds CLW's, the accurate-linkage assumption is the limiting premise.

Watch

Extended reading notes

Core claim

The central claim is that the Overlap-Excluded (OE) pseudo-likelihood estimator is a valid and efficient way to estimate selection propensities for a big non-probability dataset when a probability reference sample can be linked to it. OE maximizes a pseudo-log-likelihood that keeps the term over B for included units and uses only the reference units outside B (the set difference A \ B), weighted by design weights, for the non-included term. Under a logistic propensity model, solving the OE score equation gives propensities that, when plugged into the inverse-probability-weighted mean estimator, yield unbiased population-mean estimates whenever the model is correct. In the paper's simulations with a population of 200,000, OE showed the lowest relative root mean squared error among the compared estimators at big dataset sizes of 50,000 and 140,000, and its 95% confidence intervals had coverage between 0.946 and 0.953. The authors further state that theoretical results, not included in the paper, confirm OE is at least as efficient as the CLW and KW estimators under Poisson sampling of the reference sample, and more efficient than the ALP estimator when propensities are large.

Load-bearing premise

The load-bearing premise is that every unit in the reference sample A can be correctly classified as either present or absent in the big dataset B; if this linkage is wrong, the A without B set is misidentified and the OE score equation is misspecified, making the resulting propensity scores and IPW estimates biased.

Editorial extensions

If this is right

At big-data sizes of 50,000 and 140,000 in the simulation, the OE estimator had the lowest percentage relative root mean squared error of all compared estimators, with clear gains over the CLW, KW, WVL, and VD approaches.
When the propensity model is correctly specified, the OE-based IPW estimator is unbiased; this is claimed as a property of the method, not just an observed simulation outcome.
Coverage probabilities of 95% confidence intervals from the plug-in variance estimator stay close to nominal (0.946 to 0.953) across all simulated scenarios.
A theoretical comparison, reported but not derived in the paper, says OE is at least as efficient as CLW and KW under Poisson sampling of the reference sample, and more efficient than the ALP estimator when the true propensities are large.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A design implication not pursued in the paper is that, if overlap membership is known, the reference sample could be deliberately drawn to maximize overlap with the big dataset, so that the A without B portion carries more information and the survey cost could be reduced.
The method's reliance on exact linkage suggests a natural stress test: corrupt a known fraction of overlap indicators and measure the bias; the paper does not quantify how sensitive OE is to linkage errors.
The same pseudo-likelihood construction could be extended to doubly robust estimation of totals and quantiles, using OE propensities as the fitted weights in a regression-assisted estimator.
If the reference sample uses a complex design with unequal inclusion probabilities, the plug-in variance may need an additional term; exploring this would make the method applicable beyond the simulation settings considered.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a useful small step, not a breakthrough. The idea is to estimate propensities into a non-probability big data set using a reference probability sample, but to drop the reference sample units that also appear in the big data from the non-inclusion term. That twist (A\B instead of all A, and modeling pi directly rather than the WVL p) is real and clearly explained. The simulation is clean and the results are sensible: OE has the lowest RRMSE for large B; WVL wins at NB=2,000. Coverage of the plug-in variance is close to nominal. So the paper has value as a careful comparison and a plausible alternative estimator.

The soft spots are the ones the stress-test flags. Section 6 promises a theoretical proof of the efficiency claim but doesn't give it or a reference. That in itself is a completeness gap, not a fatal flaw, but the bigger mismatch is that the promised result is for Poisson sampling of the reference sample, while the simulation uses PPS. So even if the theorem holds, it would not verify the headline simulation. The central efficiency claim for large B rests on one well-specified DGP, one population, one nA, perfect linkage, and no misspecification. The alternative variance estimator in Eq (20) is derived without proof and not tested in the simulation, so we don't know if it preserves coverage. These are fixable with more simulation and an actual proof or a clear statement that efficiency is empirical.

I was a bit worried about circularity, but there's none: the estimator is fitted to a pseudo-likelihood and evaluated on simulated external baselines. The authors are honest about the linkage assumption, and they compare against the right literature.

Bottom line: this deserves a serious referee. It's not ready as is — the missing proof and the design mismatch have to be addressed, and the simulation should include at least one misspecified model or linkage-error scenario. But the core idea is sound and likely useful for official statistics. I'd send it to review, and if I were working on data integration methods I'd probably cite the OE estimator even in its current form.

Referee Report

4 major / 6 minor

Summary. The paper proposes an Overlap-Excluded (OE) propensity score estimator for correcting selection bias when estimating a finite-population mean from a non-probability sample B with the help of a probability reference sample A. The method assumes that membership of units in A in B can be identified exactly. The OE pseudo-likelihood in Eq. (11) uses B for the positive term and A\B, weighted by the survey design weights, for the negative term. The paper derives the corresponding score and Newton-Raphson updating equations, defines an inverse probability weighted estimator (16), and presents plug-in variance estimators in Eqs. (17)-(20). The method is compared with the CLW, KW, WVL, and VD estimators in a simulation study with N=200,000, nA=5,000, and NB equal to 2,000, 50,000, or 140,000. The reported simulation results show OE with low bias and the lowest RRMSE for the two larger values of NB, while WVL is best for the smallest NB. The paper concludes that when the propensity model is correctly specified, the OE-based IPW estimator is unbiased and efficient relative to the alternatives, particularly for large B.

Significance. If the claimed efficiency properties were rigorously established, the OE estimator would be a practically useful addition to the toolkit for integrating big non-probability data with probability samples, especially in settings where common identifiers or high-quality record linkage make the overlap between A and B known. The paper has clear strengths: the setup is clean, the simulation design is described in sufficient detail to be reproducible, and the comparison includes four established alternative estimators. The Monte Carlo results are plausible and suggest that using A\B rather than all of A in the pseudo-likelihood can improve efficiency. However, the paper's central theoretical claims are currently only asserted, not proved, and the simulation evidence is limited to a single correctly specified logistic model, one reference-sample size, one population, and perfect linkage. Because the manuscript's headline message is an efficiency ranking, the missing proof and the limited evidence are load-bearing weaknesses. The method itself is not circular or incoherent; the gaps are in the supporting theory and the breadth of the empirical evaluation.

major comments (4)

[Section 6] The concluding efficiency claim rests on an unproved assertion. The text states that 'while not included in this paper, it can be shown theoretically (confirming the simulation results in this paper) that when the reference sample is taken by Poisson sampling the OE IPW estimator is at least as efficient as' the CLW and KW estimators. No proof or reference is supplied. Since the abstract and Section 6 present efficiency relative to alternatives as a main message, this is a central claim rather than a side remark. Moreover, the promised theorem is stated for a Poisson-sampled reference sample, whereas the Section 5 simulation states that the reference sample A is drawn by 'randomised systematic probability proportional to size sampling (PPS)'. A Poisson-sampling theorem would therefore not confirm the simulation results actually reported. The authors should either provide a proof that matches the simulation design or explicitly label the efficiency claim as a conjecture supported only by the simulation.
[Section 5] The simulation evidence is too narrow to support the general efficiency claim in Section 6. Table 1 covers a single data-generating process, a single population (N=200,000), a single reference sample size (nA=5,000), one logistic propensity model, and perfect linkage between A and B. The ranking in Table 1 (OE best for NB=50,000 and 140,000, WVL best for NB=2,000) may depend on these choices. In particular, the paper does not examine misspecification of the propensity model, varying nA, varying overlap rates, or linkage errors. At minimum, the conclusion should be restricted to the design actually simulated, or the simulation should be extended to settings that probe the claimed conditions. Without this, the sentence in Section 6 that the estimates are efficient 'relative to the other approaches we have considered' overstates what the evidence shows.
[Section 4.2] The alternative plug-in variance estimator in Eq. (20) is introduced without derivation and is not evaluated in the simulation. Eq. (20) is intended to estimate the component in Eq. (19) without requiring design weights for units in B, which would be important in practice because those weights may be unavailable. The paper gives no argument for its unbiasedness or consistency, and Table 2 does not state whether the coverage probabilities were computed using Eqs. (17)-(19) or Eq. (20). The authors should either derive Eq. (20), provide a reference, or study it in the simulation; as written, the paper contains an untested variance estimator that is presented as an alternative to a component of the main variance formula.
[Section 4 and Section 6] The unbiasedness claim in Section 6 ('When the model for π_i^B is correctly specified, the IPW estimator formed from the OE propensity scores is unbiased') is not established in the paper. Section 4 derives the pseudo-likelihood and score equation heuristically, but no formal consistency or unbiasedness theorem is given for the maximum pseudo-likelihood estimator θ-hat from Eq. (13), nor for the resulting IPW estimator (16). The simulation results at NB=2,000 show small biases, but a theoretical statement should be proved or explicitly labelled as a conjecture. The authors should add a theorem with regularity conditions, or revise the wording to avoid making an unproved theoretical assertion.

minor comments (6)

[Title] The title in the manuscript text has an extra space in 'D ata Integration'; this should be corrected to 'Data Integration'.
[Section 4.2, Eqs. (17)-(19)] The displayed equations for the plug-in variance estimator are split by line numbers in a way that makes them hard to read; the plus signs appear on separate lines from the terms they connect. Please reformat the display as a single equation or a clearly aligned multi-line equation.
[Section 5.1, Table 2] The caption of Table 2 should state explicitly which of the two plug-in variance estimators, Eqs. (17)-(19) or Eq. (20), was used to construct the confidence intervals, since the paper presents both.
[Section 5.1, Table 1] The text reports that WVL has the lowest RRMSE for NB=2,000, but does not discuss why the ranking reverses as NB grows; a brief explanation or at least an acknowledgment would help the reader interpret the headline claim.
[Section 6] The phrase 'H´ajek-like estimator' contains a typographical issue with the diacritic; the name should be rendered as 'Hájek-like'.
[Section 2] Assumptions A1-A3 are stated but are not used in any formal derivation; the paper should either state that they are assumptions for the theoretical properties, or connect them to the estimating equation (13) and the variance estimators.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the OE estimator is derived from a decomposition of the population log-likelihood and evaluated against external simulation baselines.

full rationale

The paper's derivation chain is self-contained and does not reduce to its inputs. The proposed Overlap-Excluded pseudo-likelihood in Eq. (11) is constructed directly from the population log-likelihood in Eq. (2), replacing the unknown population component with a design-weighted sum over A\B. The score function in Eq. (13) and the IPW estimator in Eq. (16) follow standard maximum pseudo-likelihood and inverse probability weighting arguments. No parameter is fitted to the simulation outcomes, and the efficiency comparison in Section 5 is against independently defined competitors (CLW, KW, WVL, VD) on externally generated data. The paper contains no load-bearing self-citations and does not invoke a uniqueness theorem from the authors' prior work. Section 6 admits that the theoretical efficiency proof is 'not included in this paper' and states that it 'can be shown theoretically,' but this is an omitted or unsupported claim rather than a circular step: nothing in the paper defines the OE estimator in terms of the claimed efficiency result, and the simulation is not used to derive the estimator. The mismatch between the promised Poisson-sampling proof and the PPS reference design used in Section 5 is a completeness and corroboration gap, not a reduction of the central claim to its own assumptions. The perfect-linkage assumption is stated explicitly as a premise and is not smuggled in as a conclusion. Accordingly, no specific circular step can be quoted or exhibited, and the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The estimator itself has no fitted or hand-tuned parameters; it depends on the stated assumptions A1-A3, the logistic model for π_B, and perfect linkage between A and B. No new entities are introduced.

assumptions (5)

domain assumption A1 Ignorability: δ_i and y_i are independent conditional on x_i.
Section 2. Needed for the propensity score to fully capture selection bias; if unmeasured confounders affect both inclusion and y, the IPW estimator is biased.
domain assumption A2 Positivity: P(δ_i=1|x_i)>0 for all i.
Section 2. Required for inverse probability weights to be finite; the paper itself notes it can fail for platform-based big data.
domain assumption A3 Independence: δ_i ⊥ δ_j for i≠j given x.
Section 2. Justifies the pseudo-likelihood and plug-in variance; may be violated under clustering in big data.
domain assumption Logistic model for propensities: π_i^B = exp(x_i^T θ)/(1+exp(x_i^T θ)).
Section 4. The OE pseudo-likelihood is defined under this model; misspecification would bias the estimated propensities.
domain assumption Perfect linkage between A and B: the membership δ_i for each unit in A is observed without error.
Section 2. The A\B sample used in Eq. (11) must be correctly identified; linkage errors would misclassify the second component of the pseudo-score.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Estimating Propensities of Selection for Big Datasets via Data Integration." pith.science (2026). https://pith.science/paper/6XQF2S3Q

@misc{pith2026250104185,
  author       = {Pith},
  title        = {Pith review of: Estimating Propensities of Selection for Big Datasets via Data Integration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6XQF2S3Q}},
  note         = {Machine review of arXiv:2501.04185}
}

read the original abstract

Big data presents potential but unresolved value as a source for analysis and inference. However,selection bias, present in many of these datasets, needs to be accounted for so that appropriate inferences can be made on the target population. One way of approaching the selection bias issue is to first estimate the propensity of inclusion in the big dataset for each member of the big dataset, and then to apply these propensities in an inverse probability weighting approach to produce population estimates. In this paper, we provide details of a new variant of existing propensity score estimation methods that takes advantage of the ability to integrate the big data with a probability sample. We compare the ability of this method to produce efficient inferences for the target population with several alternative methods through an empirical study.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

2 extracted references · 2 canonical work pages

[1]

R., West, B

Andridge, R. R., West, B. T., Little, R. J. A., Boonstra, P . S. & Alvarado-Leiton, F. (2019), ‘Indices of Non-Ignorable Selection Bias for Propo rtions Estimated from Non- Probability Samples’, Journal of the Royal Statistical Society Series C: Applied Statistics 68(5), 1465–1483. Australian Bureau of Statistics (2024 a), ‘Australian Agriculture: Broad- ...

work page 2019
[1782]

Chen, Y ., Li, P . & Wu, C. (2020), ‘Doubly Robust Inference Wit h Nonprobability Survey Samples’, Journal of the American Statistical Association 115(532), 2011–2021. Golini, N. & Righi, P . (2024), ‘Integrating probability and big non-probability samples data to produce Ofﬁcial Statistics’, Statistical Methods & Applications 33, 555–580. Kim, J. K. & Wa...

work page 2020

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

R., West, B

Andridge, R. R., West, B. T., Little, R. J. A., Boonstra, P . S. & Alvarado-Leiton, F. (2019), ‘Indices of Non-Ignorable Selection Bias for Propo rtions Estimated from Non- Probability Samples’, Journal of the Royal Statistical Society Series C: Applied Statistics 68(5), 1465–1483. Australian Bureau of Statistics (2024 a), ‘Australian Agriculture: Broad- ...

work page 2019

[2] [1782]

Chen, Y ., Li, P . & Wu, C. (2020), ‘Doubly Robust Inference Wit h Nonprobability Survey Samples’, Journal of the American Statistical Association 115(532), 2011–2021. Golini, N. & Righi, P . (2024), ‘Integrating probability and big non-probability samples data to produce Ofﬁcial Statistics’, Statistical Methods & Applications 33, 555–580. Kim, J. K. & Wa...

work page 2020