{"id":"dc3412eb-50a7-4e66-ad72-70164d091a4f","arxiv_id":"2602.16111","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A reusable score-bucket calibration replaces per-experiment LLM labeling, giving daily per-arm content-prevalence estimates from impression logs at a fraction of the cost.","lead":"This paper describes a Pinterest system that replaces costly per-experiment LLM labeling with one reusable ML-score calibration, letting teams estimate how often users see content categories from impression logs alone. If it works as claimed, platforms can monitor content-exposure guardrails daily across hundreds of A/B experiments on a fixed labeling budget.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Load-bearing assumption that bucket-level prevalences are segment-invariant is unsupported; threshold interventions can shift within-bucket composition and bias surrogate deltas.","rationale":"The reader identified the segment-invariance of bucket-level prevalences as the weakest load-bearing premise, and I agree. The paper's central system-level claim—that one global calibration suffices for all arms and detects small deltas—fails if this assumption is violated. The evidence in Table 1, although explicitly illustrative, points in exactly that direction, and no robustness evidence is provided. The other concerns (shared-sample covariance in Eq. 11, the daily sign test's assumptions) are secondary: they affect the width of confidence intervals and the interpretation of p-values, but the invariance assumption affects the point estimates themselves. If the point estimates are biased, the entire comparison with the LLM reference collapses. The paper is honest about some limitations (e.g., Section 6.1.1 on calibration-dominated uncertainty), but it nowhere provides the segment-stability evidence that would secure the main reuse argument. Therefore the paper needs a CONDITIONAL acceptance: the methodology is sound if the invariance can be empirically established, but the current evidence does not support the strongest claims. I recommend no change to the reader's verdict.","tokens_in":12215,"tokens_out":2407,"duration_ms":25036,"concrete_test":"Using the LLM-labeled sample already collected for Experiment A, estimate the bucket-level prevalence separately for Control and Treatment arms: \\hat P_{k,b}(Control) and \\hat P_{k,b}(Treatment) for each score bucket b, with particular focus on the buckets adjacent to the filtering thresholds (e.g., [0.7, 0.8) for k1 and [0.1, 0.2) for k2). Compare these to the global \\hat P_{k,b} used in Eq. (10), accounting for sampling error (e.g., via a two-sample z-test). If the per-arm estimates differ materially from the global calibration, the invariance assumption is false for this experiment. A second check: recompute the surrogate delta using segment-specific \\hat P_{k,b}(S) in place of the global calibration; if the resulting delta moves toward the LLM reference delta, the bias is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of the paper is that a single global calibration of bucket-level prevalences can be reused across arbitrary arms and segments to yield LLM-comparable prevalence and delta estimates at zero incremental labeling cost. This rests on the invariance assumption in Eq. (10): P(Z=1 | m_k in b, impression in S) is the same for all segments S. Section 3.2 justifies this with the statement that bucket-level prevalences are \"fairly stable across segments,\" but provides no supporting numerical evidence. This assumption is fragile exactly in the experiments the paper targets. In Experiment A, Treatment 1 filters items with m_{k1} >= 0.70. Within a bucket such as [0.7, 0.8), the treatment removes a subset of items—those with scores above 0.70—which may have a different conditional prevalence than the rest of the bucket. The remaining items could have systematically lower or higher true prevalence, so the bucket-level prevalence changes, invalidating the global calibration. Table 1 shows the surrogate's treatment–control deltas are ~47% larger for k1 (1.03% vs 0.70%) and ~58% larger for k2 (0.49% vs 0.31%) than the LLM reference, a pattern consistent with within-bucket bias rather than random noise. The day-level sign test in Section 6.1.2 inherits this bias: the p≈0 result in Experiment C may simply reflect a biased daily delta, especially since the LLM aggregate over the entire window found no significant change (p≈0.31). Thus the system's signature capability—detecting 2–5% relative shifts—is only as reliable as the unverified invariance assumption, and the paper's own validation data suggest the assumption may be violated in threshold-based interventions.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a surrogate-based prevalence measurement system for A/B experiments on media platforms. The core idea is to maintain a global calibration of ML score buckets using a recurring LLM-labeled PPSWOR sample, then reuse the resulting bucket-level prevalences P_hat_{k,b} across arbitrary experiment arms and segments by combining them with impression-log bucket shares c_{k,b}(S). The authors derive a simple prevalence estimator (Eq. 10), an analytic variance approximation (Eq. 11), and a calibration pipeline based on a single global sample. They report validation against LLM-based reference measurements in two production experiments, a day-level delta-aggregation method with sign tests, and an implementation strategy with SQL and Monte Carlo variants. The abstract further claims coverage of reference point estimates in 92% of roughly 300 audits and over 20x more arms under the same labeling budget.","tokens_in":12429,"tokens_out":4505,"duration_ms":44700,"significance":"If the central claims hold, the paper describes a practically valuable system: it decouples expensive LLM labeling from per-experiment measurement and enables daily per-arm prevalence estimates at log-query cost. The method is not estimator-novel, but the system-level contribution—a reusable global calibration with per-experiment SQL metrics—could be useful to applied communities. Strengths include the clear PPSWOR/Hansen–Hurwitz recap, the use of impression-times-score weighting to increase bucket-level sample sizes, the inclusion of a null experiment (Experiment B) as a control, and the explicit description of implementation details. However, the quantitative evidence is largely illustrative/modified for confidentiality, and the two load-bearing statistical assumptions—segment-invariance of bucket-level prevalences and independence of bucket-level estimators—are not adequately supported. These issues are central to the paper's empirical and inferential claims.","major_comments":[{"comment":"The footnote states: 'All thresholds, prevalence values, statistical summaries and are illustrative and have been modified for confidentiality. They do not represent actual platform metrics.' This applies to the only quantitative validation of the method, and presumably to Table 2, Figure 3, and Experiment C in Section 6.1.2. As a result, the paper's core empirical claims—that the surrogate matches the LLM reference, that CI coverage is 92%, and that day-level aggregation detects 2–5% shifts—cannot be audited. Please provide real, non-illustrative evaluation data (with sample sizes, dates, and audit definitions) or clearly reframe the contribution as methodological and state that no production validation is included.","section":"Section 4.1, Table 1 footnote"},{"comment":"The variance formula assumes that bucket-level estimators are independent across buckets 'reasonable given disjoint buckets.' This is not correct: all bP_k,b are estimated from the same global PPSWOR sample (Eqs. 13–15), sharing the same sampling randomness and the marginal denominator P(impression in b). Disjoint buckets do not imply independence of the estimators. Algorithm 1 also draws each P*_k,b independently from a logit-normal distribution, ignoring correlation. This invalidates the stated confidence-interval coverage and affects the day-level p-values. Please derive the covariance structure or use a bootstrap/calibration-draw procedure that propagates the joint sampling distribution of the bucket-level prevalences.","section":"Section 3.2, Eq. (11)"},{"comment":"The reuse of global bucket-level prevalences for arbitrary segments relies on the claim that bP_k,b(S) is 'fairly stable across segments,' but no numerical evidence is provided. The assumption fails exactly in the experiments the paper targets: in Experiment A, Treatment 1 filters items with m_{k1} >= 0.70, which changes the composition of the [0.7, 0.8) bucket and hence the bucket-level prevalence. Table 1 shows surrogate treatment–control deltas 47% and 58% larger than the LLM reference for k1 and k2, a pattern consistent with within-bucket bias rather than random noise. The day-level sign test in Section 6.1.2 inherits this bias: the p≈0 result for Experiment C may be an artifact of a biased surrogate delta, especially since the LLM aggregate p-value for the same experiment is ≈0.31. Please provide segment-level bucket-prevalence comparisons, or demonstrate invariance on data not gene","section":"Section 3.2, Eq. (10)"},{"comment":"The abstract claims 'roughly 300 production audits' with 92% CI containment and 'over 20x as many concurrent arms,' but the body reports no audit protocol, no count of audits, and no coverage calculation. The day-level sign test also lacks an exact definition: 'p-value close to 0' and the sign-test p-value are not quantified, and the test ignores temporal autocorrelation of daily deltas and the use of a fixed calibration snapshot. Please add a dedicated validation section with the audit definition, coverage computation, number of positive/negative days, exact p-values, and a discussion of whether the daily deltas are treated as independent.","section":"Abstract and Section 6.1.2"}],"minor_comments":[{"comment":"The notation U_i^{1/w_i,k(S)} is clear but would benefit from stating that ties are broken deterministically; otherwise the reservoir order statistic is ambiguous.","section":"Section 2.3, Eq. (6)"},{"comment":"Typo: 'The null hypotheses are that filtering on k_l for l in {1,2} do not reduce prevalences' should be 'does not reduce prevalence.'","section":"Section 4.1, first paragraph"},{"comment":"The indicator symbol ⊮ is nonstandard; use bold 1_{i in b} or I(i in b) and define it clearly.","section":"Section 5.1, Eq. (13)"},{"comment":"The phrase 'a sample of 10,000 items' appears without a reference to the calibration window or category; clarify whether this is a single category's sample or the total across categories.","section":"Section 5.1, after Eq. (15)"},{"comment":"The sentence 'With a sufficiently large calibration sample (e.g., millions of labeled items per bucket)' seems inconsistent with the earlier sample sizes; please clarify the realistic calibration budget and per-bucket sample sizes used in the deployment.","section":"Section 6.1.1"},{"comment":"The p-values are reported as 0.00; provide the test statistic and the exact numerical p-value, or state the test used (e.g., two-sample z-test on the HH estimates).","section":"Section 4.1, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper has practical potential, and the system-level framing is useful. However, the central empirical evidence is explicitly illustrative, and the two main statistical assumptions (segment invariance and between-bucket independence) are not supported by the manuscript. The authors should be encouraged to release anonymized real audit data or a reproducible synthetic-data evaluation, and to fix the variance propagation. It may also be worth asking whether the paper's scope is more suited to a systems/applied track than a statistical methodology venue, given the emphasis on deployment details."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing you should know before reading: this is not a new estimator. It's a description of a production system that reuses a globally calibrated score-bucket table to estimate prevalence in many A/B experiments at near-zero marginal labeling cost. The estimator (Eq 10) is textbook post-stratification/model-assisted estimation (they cite Särndal). The contribution is operational: a single recurring LLM-labeled sample, bucket-level prevalence calibrated once, and a per-experiment SQL metric with a delta-focused daily aggregation. That pattern is genuinely useful for content-governance teams, and the paper is honest that the value is at the system level.\n\nWhat it does well: The writing is clear, the estimator algebra is correct, and the deployment details (PPSWOR sampling, impression×score weights, logit-normal draws) are concrete. The paper explicitly concedes in Section 6.1.1 that calibration uncertainty dominates arm-level CIs, which is candid.\n\nNow the soft spots. First, the headline validation numbers are not auditable: Table 1's footnote says all thresholds, prevalence values, and statistical summaries are illustrative and modified for confidentiality. So the 92% coverage and the effect-size reproductions cannot be checked. That is a big problem for the central claim. Second, Eq (11) ignores the covariance between bucket-level estimators induced by the shared global calibration sample. The justification that buckets are disjoint is wrong — the estimators are driven by a common sample. Third, the load-bearing assumption of segment-invariant bucket-level prevalences (Section 3.2) is asserted with \"fairly stable\" and no numbers. Experiment A's surrogate deltas are ~50% larger than the LLM reference for both categories, which is what you'd expect if threshold filtering changes the within-bucket composition. Fourth, the day-level sign test treats days as exchangeable and in Experiment C overrules the LLM reference (which found no significant effect) on the basis of a design prior. That is a strong inference from a weak instrument.\n\nNone of this is fatal to the system concept, but it means the paper's claims about 'LLM-comparable' and '2–5% shifts' are not supported by the evidence as presented. The methodology is sound enough to warrant peer review, though the authors should be asked to provide real validation data (even with irreducible noise) and to address the covariance and invariance issues.\n\nWho is it for: practitioners running content-governance measurement at scale. A reader who wants a new statistical estimator will be disappointed; a reader who wants a template for building a cheap prevalence metric will find value.\n\nRecommendation: send to review, but flag the need for real validation numbers and a corrected variance treatment.","headline":"The paper's real contribution is an operational pattern, not a new estimator, and its headline validation numbers are explicitly illustrative — worth reviewing but with serious caveats.","tokens_in":13108,"tokens_out":2417,"would_cite":false,"duration_ms":24763,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62D05","62F03"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a single, reusable LLM-based calibration of ML score buckets can replace per-experiment LLM labeling for daily prevalence measurement across hundreds of A/B experiments.","keywords":["prevalence estimation","surrogate measurement","LLM labeling","score bucketing","A/B testing","calibration","experimentation platform","Hansen-Hurwitz estimator"],"falsifier":"Run an experiment that filters items inside a single score bucket (e.g., removes items with score in [0.70, 0.80) while leaving scores at or above 0.80 untouched) and compare the surrogate's treatment-control delta with an LLM-labeled reference. If the bucket-level prevalence is truly segment-invariant, the surrogate should stay unbiased; if the surrogate diverges from the LLM reference, the invariance assumption fails.","tokens_in":11948,"feed_emoji":"📊","tokens_out":3949,"duration_ms":39638,"temperature":0.7,"pith_summary":"The paper tries to establish that expensive LLM labeling need only be done once per content category on a recurring global sample, and the resulting per-score-bucket prevalence calibration can be reused to estimate content prevalence in every experiment arm and segment from impression logs alone. On the deployed platform, this yields daily per-arm prevalence for over 20x as many concurrent arms under the same labeling budget, with zero incremental labeling cost. The authors argue it is not merely a cheaper approximation: across roughly 300 production audits, the surrogate's 95% confidence interval contains the LLM reference point estimate in 92% of evaluations, and day-level delta aggregation detects 2-5% relative shifts that no single per-arm LLM measurement can detect. A sympathetic reader should care because this converts prevalence measurement from a rare, expensive, one-shot audit into an always-on, log-only experiment metric.","feed_headline":"Calibrate once, measure everywhere: score buckets scale A/B prevalence","feed_subtitle":"One LLM-labeled score calibration replaces per-experiment labeling and detects 2-5% shifts that single per-arm LLM reads miss.","key_machinery":"The key object is the score-bucket surrogate: model scores are discretized into B buckets, each bucket b is assigned an estimated prevalence P_{k,b} (the probability an impression in bucket b belongs to category k), estimated from a global LLM-labeled sample. Arm prevalence is then computed as the sum over buckets of (impression share in bucket b) times (bucket prevalence), per Equation 10. The calibration pipeline uses PPSWOR sampling with weights proportional to impression count times model score, so high-score buckets are not starved of labeled examples. The online layer is a deterministic SQL query plus a Monte Carlo extension that draws bucket-level prevalences from a logit-normal distr","core_discovery":"The central claim is that the prevalence of a content category in any experiment arm can be written as a weighted sum of bucket-level prevalences, where the weights are the arm's impression shares over model-score buckets (Equation 10). The bucket-level prevalences are estimated once, offline, from an LLM-labeled, impression-times-score weighted PPSWOR sample, then treated as a fixed calibration table reused across all experiments. The arm-specific bucket shares are computed from logs alone. Because bucket-level prevalences are claimed to be reasonably stable across segments, the same global calibration serves every experiment, and the day-level delta between arms isolates the distributional","pith_inferences":["If bucket-level prevalence is truly segment-invariant, the surrogate becomes a power amplifier: it converts a single expensive cross-section into a daily time series of deltas, and could be applied to segments the LLM never sampled, such as fine demographic or surface intersections.","The method's delta sensitivity depends on the treatment shifting impression mass across score buckets; a treatment that re-ranks items within a bucket or changes the composition inside a bucket without changing the bucket's overall impression share would be invisible to the surrogate, so complementary spot audits on such experiments would be prudent.","The analytic variance treats bucket-level estimators as independent, but they share the same global calibration sample; the true uncertainty of the surrogate prevalence is likely larger than reported. A bootstrap over calibration draws would provide a more honest interval.","As LLM inference becomes cheaper, the authors expect the surrogate's role to shrink for absolute level estimation, but the core ideas of reusing calibration and separating absolute levels from deltas should remain useful; one testable extension is applying the same bucket-calibration idea to other surrogate signals, such as content embeddings or context features."],"forward_implications":["Prevalence can be measured daily per arm at near-log-query cost, enabling always-on guardrails for hundreds of concurrent experiments.","A single recurring LLM-labeled calibration stream replaces per-experiment, per-arm, per-segment, per-day labeling jobs; its cost is amortized across all downstream experiments.","Day-level delta aggregation with a sign test can recover 2-5% relative prevalence shifts that per-arm LLM measurements, which are typically one-shot reads, cannot statistically resolve.","The surrogate agrees with the LLM reference in absolute level: in 92% of production audit evaluations, the surrogate's 95% confidence interval contains the LLM point estimate.","The pattern is portable to teams without the same infrastructure: they can run a recurring calibration-labeling workflow once and reuse it across experiments."],"fun_headline_variants":["Calibrate once, reuse everywhere: LLM-surrogate prevalence at scale","One bucket calibration, all arms: daily prevalence without per-arm LLM","Global calibration, per-arm prevalence: 20x arms under same budget","Reuse LLM calibration across A/B arms: detects 2-5% shifts","Score buckets shared: LLM prevalence for every experiment arm"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire system rests on the claim that a score bucket's prevalence is the same for every experiment arm and segment; if a treatment changes which items fall inside a score bucket, the surrogate's estimated delta inherits that change as bias.","fun_headline_variants_meta":{"raw":{"variants":["Calibrate once, reuse everywhere: LLM-surrogate prevalence at scale","One bucket calibration, all arms: daily prevalence without per-arm LLM","Global calibration, per-arm prevalence: 20x arms under same budget","Reuse LLM calibration across A/B arms: detects 2-5% shifts","Score buckets shared: LLM prevalence for every experiment arm"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000164,"raw_usage":{"total_tokens":1137,"prompt_tokens":849,"completion_tokens":288,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":191}},"tokens_in":593,"tokens_out":288,"duration_ms":3721,"temperature":1.0,"reasoning_tokens":191,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T22:39:40.827883+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an experiment that filters items inside a single score bucket (e.g., removes items with score in [0.70, 0.80) while leaving scores at or above 0.80 untouched) and compare the surrogate's treatment-control delta with an LLM-labeled reference. If the bucket-level prevalence is truly segment-invariant, the surrogate should stay unbiased; if the surrogate diverges from the LLM reference, the invariance assumption fails.","supporting_citations":[],"review_version":1}