Pith. sign in

REVIEW 4 major objections 6 minor 15 references

This paper claims that a single, reusable LLM-based calibration of ML score buckets can replace per-experiment LLM labeling for daily prevalence measurement across hundreds of A/B experiments.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 22:39 UTC pith:IO3CNALW

load-bearing objection The paper's real contribution is an operational pattern, not a new estimator, and its headline validation numbers are explicitly illustrative — worth reviewing but with serious caveats. the 4 major comments →

arxiv 2602.16111 v2 pith:IO3CNALW submitted 2026-02-18 stat.AP cs.AI

Calibrate Globally, Measure Everywhere: Scaling LLM-Based Prevalence Measurement Across A/B Experiments

Zehao Xu , Tony Paek , Kevin O'Sullivan , Attila Dobi This is my paper
classification stat.AP cs.AI MSC 62D0562F03
keywords prevalence estimationsurrogate measurementLLM labelingscore bucketingA/B testingcalibrationexperimentation platformHansen-Hurwitz estimator
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that expensive LLM labeling need only be done once per content category on a recurring global sample, and the resulting per-score-bucket prevalence calibration can be reused to estimate content prevalence in every experiment arm and segment from impression logs alone. On the deployed platform, this yields daily per-arm prevalence for over 20x as many concurrent arms under the same labeling budget, with zero incremental labeling cost. The authors argue it is not merely a cheaper approximation: across roughly 300 production audits, the surrogate's 95% confidence interval contains the LLM reference point estimate in 92% of evaluations, and day-level delta aggregation detects 2-5% relative shifts that no single per-arm LLM measurement can detect. A sympathetic reader should care because this converts prevalence measurement from a rare, expensive, one-shot audit into an always-on, log-only experiment metric.

Core claim

The central claim is that the prevalence of a content category in any experiment arm can be written as a weighted sum of bucket-level prevalences, where the weights are the arm's impression shares over model-score buckets (Equation 10). The bucket-level prevalences are estimated once, offline, from an LLM-labeled, impression-times-score weighted PPSWOR sample, then treated as a fixed calibration table reused across all experiments. The arm-specific bucket shares are computed from logs alone. Because bucket-level prevalences are claimed to be reasonably stable across segments, the same global calibration serves every experiment, and the day-level delta between arms isolates the distributional

What carries the argument

The key object is the score-bucket surrogate: model scores are discretized into B buckets, each bucket b is assigned an estimated prevalence P_{k,b} (the probability an impression in bucket b belongs to category k), estimated from a global LLM-labeled sample. Arm prevalence is then computed as the sum over buckets of (impression share in bucket b) times (bucket prevalence), per Equation 10. The calibration pipeline uses PPSWOR sampling with weights proportional to impression count times model score, so high-score buckets are not starved of labeled examples. The online layer is a deterministic SQL query plus a Monte Carlo extension that draws bucket-level prevalences from a logit-normal distr

Load-bearing premise

The entire system rests on the claim that a score bucket's prevalence is the same for every experiment arm and segment; if a treatment changes which items fall inside a score bucket, the surrogate's estimated delta inherits that change as bias.

What would settle it

Run an experiment that filters items inside a single score bucket (e.g., removes items with score in [0.70, 0.80) while leaving scores at or above 0.80 untouched) and compare the surrogate's treatment-control delta with an LLM-labeled reference. If the bucket-level prevalence is truly segment-invariant, the surrogate should stay unbiased; if the surrogate diverges from the LLM reference, the invariance assumption fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Prevalence can be measured daily per arm at near-log-query cost, enabling always-on guardrails for hundreds of concurrent experiments.
  • A single recurring LLM-labeled calibration stream replaces per-experiment, per-arm, per-segment, per-day labeling jobs; its cost is amortized across all downstream experiments.
  • Day-level delta aggregation with a sign test can recover 2-5% relative prevalence shifts that per-arm LLM measurements, which are typically one-shot reads, cannot statistically resolve.
  • The surrogate agrees with the LLM reference in absolute level: in 92% of production audit evaluations, the surrogate's 95% confidence interval contains the LLM point estimate.
  • The pattern is portable to teams without the same infrastructure: they can run a recurring calibration-labeling workflow once and reuse it across experiments.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If bucket-level prevalence is truly segment-invariant, the surrogate becomes a power amplifier: it converts a single expensive cross-section into a daily time series of deltas, and could be applied to segments the LLM never sampled, such as fine demographic or surface intersections.
  • The method's delta sensitivity depends on the treatment shifting impression mass across score buckets; a treatment that re-ranks items within a bucket or changes the composition inside a bucket without changing the bucket's overall impression share would be invisible to the surrogate, so complementary spot audits on such experiments would be prudent.
  • The analytic variance treats bucket-level estimators as independent, but they share the same global calibration sample; the true uncertainty of the surrogate prevalence is likely larger than reported. A bootstrap over calibration draws would provide a more honest interval.
  • As LLM inference becomes cheaper, the authors expect the surrogate's role to shrink for absolute level estimation, but the core ideas of reusing calibration and separating absolute levels from deltas should remain useful; one testable extension is applying the same bucket-calibration idea to other surrogate signals, such as content embeddings or context features.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a surrogate-based prevalence measurement system for A/B experiments on media platforms. The core idea is to maintain a global calibration of ML score buckets using a recurring LLM-labeled PPSWOR sample, then reuse the resulting bucket-level prevalences P_hat_{k,b} across arbitrary experiment arms and segments by combining them with impression-log bucket shares c_{k,b}(S). The authors derive a simple prevalence estimator (Eq. 10), an analytic variance approximation (Eq. 11), and a calibration pipeline based on a single global sample. They report validation against LLM-based reference measurements in two production experiments, a day-level delta-aggregation method with sign tests, and an implementation strategy with SQL and Monte Carlo variants. The abstract further claims coverage of reference point estimates in 92% of roughly 300 audits and over 20x more arms under the same labeling budget.

Significance. If the central claims hold, the paper describes a practically valuable system: it decouples expensive LLM labeling from per-experiment measurement and enables daily per-arm prevalence estimates at log-query cost. The method is not estimator-novel, but the system-level contribution—a reusable global calibration with per-experiment SQL metrics—could be useful to applied communities. Strengths include the clear PPSWOR/Hansen–Hurwitz recap, the use of impression-times-score weighting to increase bucket-level sample sizes, the inclusion of a null experiment (Experiment B) as a control, and the explicit description of implementation details. However, the quantitative evidence is largely illustrative/modified for confidentiality, and the two load-bearing statistical assumptions—segment-invariance of bucket-level prevalences and independence of bucket-level estimators—are not adequately supported. These issues are central to the paper's empirical and inferential claims.

major comments (4)
  1. [Section 4.1, Table 1 footnote] The footnote states: 'All thresholds, prevalence values, statistical summaries and are illustrative and have been modified for confidentiality. They do not represent actual platform metrics.' This applies to the only quantitative validation of the method, and presumably to Table 2, Figure 3, and Experiment C in Section 6.1.2. As a result, the paper's core empirical claims—that the surrogate matches the LLM reference, that CI coverage is 92%, and that day-level aggregation detects 2–5% shifts—cannot be audited. Please provide real, non-illustrative evaluation data (with sample sizes, dates, and audit definitions) or clearly reframe the contribution as methodological and state that no production validation is included.
  2. [Section 3.2, Eq. (11)] The variance formula assumes that bucket-level estimators are independent across buckets 'reasonable given disjoint buckets.' This is not correct: all bP_k,b are estimated from the same global PPSWOR sample (Eqs. 13–15), sharing the same sampling randomness and the marginal denominator P(impression in b). Disjoint buckets do not imply independence of the estimators. Algorithm 1 also draws each P*_k,b independently from a logit-normal distribution, ignoring correlation. This invalidates the stated confidence-interval coverage and affects the day-level p-values. Please derive the covariance structure or use a bootstrap/calibration-draw procedure that propagates the joint sampling distribution of the bucket-level prevalences.
  3. [Section 3.2, Eq. (10)] The reuse of global bucket-level prevalences for arbitrary segments relies on the claim that bP_k,b(S) is 'fairly stable across segments,' but no numerical evidence is provided. The assumption fails exactly in the experiments the paper targets: in Experiment A, Treatment 1 filters items with m_{k1} >= 0.70, which changes the composition of the [0.7, 0.8) bucket and hence the bucket-level prevalence. Table 1 shows surrogate treatment–control deltas 47% and 58% larger than the LLM reference for k1 and k2, a pattern consistent with within-bucket bias rather than random noise. The day-level sign test in Section 6.1.2 inherits this bias: the p≈0 result for Experiment C may be an artifact of a biased surrogate delta, especially since the LLM aggregate p-value for the same experiment is ≈0.31. Please provide segment-level bucket-prevalence comparisons, or demonstrate invariance on data not gene
  4. [Abstract and Section 6.1.2] The abstract claims 'roughly 300 production audits' with 92% CI containment and 'over 20x as many concurrent arms,' but the body reports no audit protocol, no count of audits, and no coverage calculation. The day-level sign test also lacks an exact definition: 'p-value close to 0' and the sign-test p-value are not quantified, and the test ignores temporal autocorrelation of daily deltas and the use of a fixed calibration snapshot. Please add a dedicated validation section with the audit definition, coverage computation, number of positive/negative days, exact p-values, and a discussion of whether the daily deltas are treated as independent.
minor comments (6)
  1. [Section 2.3, Eq. (6)] The notation U_i^{1/w_i,k(S)} is clear but would benefit from stating that ties are broken deterministically; otherwise the reservoir order statistic is ambiguous.
  2. [Section 4.1, first paragraph] Typo: 'The null hypotheses are that filtering on k_l for l in {1,2} do not reduce prevalences' should be 'does not reduce prevalence.'
  3. [Section 5.1, Eq. (13)] The indicator symbol ⊮ is nonstandard; use bold 1_{i in b} or I(i in b) and define it clearly.
  4. [Section 5.1, after Eq. (15)] The phrase 'a sample of 10,000 items' appears without a reference to the calibration window or category; clarify whether this is a single category's sample or the total across categories.
  5. [Section 6.1.1] The sentence 'With a sufficiently large calibration sample (e.g., millions of labeled items per bucket)' seems inconsistent with the earlier sample sizes; please clarify the realistic calibration budget and per-bucket sample sizes used in the deployment.
  6. [Section 4.1, Table 1] The p-values are reported as 0.00; provide the test statistic and the exact numerical p-value, or state the test used (e.g., two-sample z-test on the HH estimates).

Circularity Check

0 steps flagged

No significant circularity: the surrogate is calibrated from an external LLM-labeled global sample and applied to impression-log bucket shares; the invariance assumption is testable rather than definitional.

full rationale

The paper's central derivation is not circular. Equation (10) estimates segment prevalence as a post-stratified sum of an externally calibrated bucket-level prevalence P-hat_{k,b} and impression-log bucket shares c_{k,b}(S). The bucket-level prevalences are estimated from a real LLM-labeled global PPSWOR sample (Eqs. 12–15), an input that is distinct from the per-arm LLM reference estimates used for validation. The arm-level prediction is not equal to the reference by construction unless the empirical invariance assumption in Section 3.2 holds; if within-bucket composition changes under treatment, the surrogate and reference can and do differ, as Table 1 shows. The paper does not hide this: Section 6.1.1 explicitly states that the confidence intervals answer a calibration-conditional question, and the lack of numerical support for 'fairly stable across segments' is a missing-evidence limitation, not a circular reduction. The only self-citation, Farooq et al. [5], is used for background engineering details of the production prevalence pipeline and is not load-bearing for the surrogate derivation. The shared LLM-labeling pipeline between calibration and validation is a real external-validity concern—both may share systematic LLM bias—but it does not make the surrogate's arm-level prediction a restatement of the reference, nor does it reduce the derivation to its inputs. The day-level sign test may inherit calibration bias, but that is a statistical validity threat, not a circularity. No step in the derivation chain is self-definitional or fitted-input-called-prediction.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The estimator is a post-stratification: bucket-level prevalences learned from a global LLM-labeled sample, reweighted by per-arm bucket shares. No new quantities are postulated; the burden is carried by design choices (B=10, impression×score weighting), the LLM-label ground-truth assumption, segment-invariance of bucket prevalence, and a variance formula that ignores shared-sample correlations.

free parameters (4)
  • bucket count B (and boundaries) = 10 equal-width buckets over [0,1]
    B=10 equal-width buckets chosen as a default; paper notes bias-variance tradeoffs and flags adaptive bucketization as future work (Sections 4.1, 6.2).
  • calibration sampling weight function = w ∝ I_i · m_{i,k}
    Impression-times-score weighting chosen by inspection of bucket balance (Figure 2); a design choice, not derived from any optimality criterion.
  • calibration window length = unspecified
    Section 5.1 says 'over a calibration window' without stating its length; the drift-adaptivity claim depends on this choice.
  • logit-normal variance transform = delta method
    Algorithm 1 Step 1 converts bucket binomial variance to logit scale via the delta method; an approximation used for the Monte Carlo extension.
axioms (5)
  • domain assumption LLM labels Z_{i,k} are true category labels (unbiased reference)
    Section 2.4 substitutes LLM labels into the Hansen–Hurwitz estimator and calls the result unbiased for prevalence; any systematic LLM label error is inherited by both the calibration and the reference.
  • domain assumption Bucket-level prevalence is approximately segment-invariant: P(Z=1|m∈b,S) ≈ P(Z=1|m∈b)
    Section 3.2 Eq (10) reuses a global calibration for arbitrary arms; stated as an empirical observation without supporting numbers. Fails when treatment changes within-bucket item composition.
  • domain assumption Bucket-level estimators are independent across buckets for Eq (11)
    Justified as 'reasonable given disjoint buckets,' but the estimators are computed from one shared global sample (Eq 13), so they are correlated; the variance formula ignores this covariance.
  • standard math Hansen–Hurwitz estimator unbiasedness under PPSWOR
    Standard survey result (Hansen & Hurwitz 1943, cited [6]); requires correct inclusion probabilities and known denominators.
  • domain assumption Model score m_{i,k} is a valid surrogate for category membership
    The entire surrogate relies on the score carrying bucket-level information about Z; the score model's training labels are not described, so its bias relative to LLM labels is unknown.

pith-pipeline@v1.3.0-alltime-deepseek · 11629 in / 22338 out tokens · 197442 ms · 2026-08-02T22:39:40.827883+00:00 · methodology

0 comments
read the original abstract

Online media platforms track the share of impressions associated with content attributes, or prevalence, to evaluate trade-offs and set guardrails in A/B experiments. LLM-based labeling provides a high-fidelity reference measurement, but is cost-prohibitive to run per experiment, per arm, per segment, and per day on a platform with hundreds of concurrent experiments. We describe a surrogate-based prevalence measurement system deployed in Pinterest's experimentation platform. The contribution is system-level rather than estimator-level: the system maintains a single global calibration of ML score buckets, continuously refreshed from a recurring LLM-labeled stream, and reuses the resulting bucket-level prevalences across every experiment via a per-experiment SQL metric and a delta-focused dashboard. Because the calibration is derived from the platform's daily-batch prevalence samples, it remains representative of production traffic as distributions drift, and in Pinterest's deployment it incurs zero incremental labeling cost. Teams without such infrastructure can instantiate the same pattern with a recurring calibration-labeling workflow whose cost is amortized across all downstream experiments rather than paid per experiment, arm, segment, and day. The system serves ~100 experiments and ~250 arms per day across six calibrated content categories, including a holdout program. Relative to per-experiment LLM labeling, which in practice yields a one-shot read per arm on a small subset of experiments, the surrogate provides daily per-arm prevalence on over 20$\times$ as many concurrent arms under the same labeling budget. Across roughly 300 production audits, the surrogate's 95\% confidence interval contains the LLM-based reference point estimate in 92\% of evaluations, and day-level delta aggregation recovers 2--5\% relative shifts that no single per-arm LLM measurement can detect.

Figures

Figures reproduced from arXiv: 2602.16111 by Attila Dobi, Kevin O'Sullivan, Tony Paek, Zehao Xu.

Figure 1
Figure 1. Figure 1: Bucket-level impression-share shifts for categories [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of score-bucket distributions under two sampling schemes. Left: sampling with weights proportional to [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Relative prevalence reduction vs. calendar day for [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 1 canonical work pages

  1. [1]

    Susan Athey, Raj Chetty, Guido W Imbens, and Hyunseung Kang. 2025. The Surrogate Index: Combining Short-Term Proxies to Estimate Long-Term Treat- ment Effects More Rapidly and Precisely.The Review of Economic Studiesrdaf087 (2025)

    2025

  2. [2]

    George EP Box and Mervin E Muller. 1958. A note on the generation of random normal deviates.The annals of mathematical statistics29, 2 (1958), 610–611

    1958

  3. [3]

    Wilfrid J Dixon and Alexander M Mood. 1946. The statistical sign test.J. Amer. Statist. Assoc.41, 236 (1946), 557–566

    1946

  4. [4]

    Pavlos S Efraimidis and Paul G Spirakis. 2006. Weighted random sampling with a reservoir.Information processing letters97, 5 (2006), 181–185

    2006

  5. [5]

    Faisal Farooq, Aravindh Manickavasagam, and Attila Dobi. 2025. How Pinterest Built a Real-Time Radar for Violative Content using AI. https://medium.com/pinterest-engineering/how-pinterest-built-a-real- time-radar-for-violative-content-using-ai-d5a108e02ac2. Pinterest Engineering Blog

    2025

  6. [6]

    Morris H Hansen and William N Hurwitz. 1943. On the theory of sampling from finite populations.The Annals of Mathematical Statistics14, 4 (1943), 333–362

    1943

  7. [7]

    Horvitz and Donovan J

    Daniel G. Horvitz and Donovan J. Thompson. 1952. A Generalization of Sampling Without Replacement from a Finite Universe.J. Amer. Statist. Assoc.47, 260 (1952), 663–685. doi:10.1080/01621459.1952.10483446

    arXiv 1952

  8. [8]

    Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M. Henne. 2009. Controlled experiments on the web: survey and practical guide.Data Mining and Knowledge Discovery18 (2009), 140–181. doi:10.1007/s10618-008-0114-1

    doi:10.1007/s10618-008-0114-1 2009

  9. [9]

    Himabindu Lakkaraju, Jon Kleinberg, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. 2017. The Selective Labels Problem: Evaluating Algorithmic Predic- tions in the Presence of Unobservables. InProceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). 275–284. doi:10.1145/3097983.3098066

    arXiv 2017

  10. [10]

    Xin Liu, Yichen Zhu, Jindong Gu, Yunshi Lan, Chao Yang, and Yu Qiao. 2023. MM- SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models.arXiv preprint(2023). arXiv:2311.17600 https://arxiv.org/abs/2311.17600

    Pith/arXiv arXiv 2023

  11. [11]

    Gary W Oehlert. 1992. A note on the delta method.The American Statistician46, 1 (1992), 27–29

    1992

  12. [12]

    2023.Using GPT-4 for content moderation

    OpenAI. 2023.Using GPT-4 for content moderation. https://openai.com/blog/ using-gpt-4-for-content-moderation

    2023

  13. [13]

    Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré

    Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. 2020. Snorkel: rapid training data creation with weak supervision. The VLDB Journal29 (2020), 709–730. doi:10.1007/s00778-019-00552-1

    work page doi:10.1007/s00778-019-00552-1 2020

  14. [14]

    1992.Model Assisted Survey Sampling

    Carl-Erik Särndal, Bengt Swensson, and Jan Wretman. 1992.Model Assisted Survey Sampling. Springer

    1992

  15. [15]

    Xing, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv preprint(2023). arXiv:2306.05685 https://arxiv.org/abs/2306.05685

    Pith/arXiv arXiv 2023