Pith. sign in

REVIEW 4 major objections 6 minor 31 references

SERP Interference Network and Its Applications in Search Advertising

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Cluster-randomized experiments built from SERP interference networks estimate a bidding algorithm's true lift, where product-split tests overstate it.

desk verdict Practical cluster-randomized design for SERP interference built from the ad publisher's own query reports; the construction is new and credible, but the validation of the 'good estimate of actual lift' claim rests on one non-concurrent comparison and censored graph data. read the letter →

arxiv 2506.21598 v1 pith:F265F4GW submitted 2025-06-19 cs.IR stat.ME

Purak Jain , Sandeep Appala This is my paper · ORCID
classification cs.IRstat.ME
keywords SERPinterferencenetworksponsoredsearchA/Btestingclusterrandomizedexperimentunipartitegraphprojectionbiddingalgorithmcensoredobservationaldatapaidadvertising
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Search advertisers cannot randomize A/B tests on customers because search users are anonymous to them, and randomizing on products instead violates the no-interference assumption when two products in different groups appear in the same auction. This paper claims that clustering products that co-appear on search results pages, and randomizing whole clusters, restores a trustworthy comparison. The authors build the clusters from the advertiser's own search query reports, using a cheap weighted graph projection and a scalable graph partitioning algorithm. In their application, the cluster-randomized estimate of a new bidding algorithm's effect was consistent with the lift observed after full rollout, whereas an earlier product-split test had overstated the lift. If the design works as claimed, search marketing teams can run faster and more reliable A/B tests without geo-cloning campaigns.

What carries the argument

The central object is the SERP interference network: an undirected bipartite graph between search queries and products, built from the ad publisher's query reports over a year of impressions. The paper's computational device is a weighted one-mode projection with edge weight $$W_{\rm uni}(a,b)=\sum_i \frac{1}{\log_e(f_{sq_i})}\frac{\min(W_{\rm bi}(sq_i,a),W_{\rm bi}(sq_i,b))}{\max(W_{\rm bi}(sq_i,a),W_{\rm bi}(sq_i,b))}\mathbb{I}[sq_i,a,b],$$ where $f_{sq_i}$ is the number of distinct products query $i$ drives impressions to and the indicator is one exactly when the query triggered impressions for both products. This weight suppresses contributions from broad upper-funnel queries while scoring products that repeatedly co-appear on the same results page as strongly interfering. The projection is partitioned with a multilevel graph partitioning algorithm that runs in $O(|E|)$ time and balances cluster sizes while minimizing the weighted edge cut between clusters, and the resulting clusters become the randomization units.

What would settle it

The clearest falsifier is a meta-experiment that randomizes between a Bernoulli product-split design and the cluster design on the same marketplace and compares the two estimated total average treatment effects; if they are statistically indistinguishable, the cluster design provides no bias correction over product splits.

Watch

Extended reading notes

Core claim

The paper's central claim is that a product-cluster randomized control trial formed from a SERP interference network yields a near-unbiased estimate of a bidding algorithm's causal effect. The interference network is a bipartite graph whose edges connect search queries to the products that the ad publisher showed for them; a weighted one-mode projection places an edge between any two products that co-appear for a query, with weight down-weighted for generic queries. Partitioning that product graph into balanced clusters (10,000 clusters in the reported run) and randomizing clusters, rather than individual products, makes SUTVA approximately hold because spillover is concentrated inside clusters and both treatment and control bids are absent in the same auction. The reported online A/B test, analyzed with difference-in-differences and cluster-robust standard errors, showed treatment-group lift consistent with the lift seen after the new bidding model was rolled out, while a previous naive product-split test showed inflated lift of roughly 44% versus actual approximately 24%.

Load-bearing premise

The design assumes the advertiser's search query reports faithfully represent the interference structure, even though they only show auctions the advertiser won; if losing auctions carry substantial spillover, the projected product graph and clusters are biased.

Editorial extensions

If this is right

  • Search marketers can run online A/B tests on bidding algorithms without cloning whole ad accounts or waiting out long switchback adjustment periods, because randomization happens on product clusters rather than individual products.
  • The earlier naive product-split experiment overestimated the bidding model's impact (roughly 44% lift versus roughly 24% realized), so the cluster design is what brought the online estimate in line with post-rollout performance.
  • The leakage-versus-cluster-count curve gives an operational rule for choosing the number of clusters: pick the largest $k$ before between-cluster edge weight climbs, balancing statistical power against bias.
  • Cluster-robust standard errors, applied after difference-in-differences estimation, prevent within-cluster error correlation from producing misleadingly small p-values.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The most direct test of the design would be the meta-experiment the paper mentions but could not run: randomizing one arm by Bernoulli product split and another by cluster split, then testing whether the two estimated treatment effects differ significantly.
  • Because the query reports only record auctions the advertiser won, the true interference graph likely has more edges than the projected one; adding SERP-level observation data would probably raise the measured between-cluster leakage and could change cluster boundaries.
  • The log-frequency down-weighting in the projection is one defensible choice; weighting by query position, ad slot, or auction presence might sharpen the interference model, but that extension would need its own validation.
  • If clusters can be refreshed as query reports stream in, the approach would generalize from one-off experiments to continuous experimentation, conditional on the cluster-stability analysis the paper lists as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper addresses interference in search-advertising A/B tests when the unit of randomization is a product. It proposes constructing a bipartite graph between search queries and products from the advertiser's daily search query reports, weighting a one-mode projection by Eq. (1), partitioning the product graph with METIS into balanced clusters, and randomizing clusters to treatment and control for a difference-in-differences evaluation of a new bidding model. The authors report that the cluster-randomized online test produced a lift 'consistent with' the post-rollout lift, whereas an earlier product-split test overstated lift (~44% vs ~24%). They also describe a SageMaker-based system architecture for deploying the design. The paper is candid that the query-report data are censored to won auctions and that a formal meta-experiment was not run.

Significance. If the central claim were established, the paper would provide a practical and scalable solution to a real industrial problem, with a computationally efficient weighting function and a production architecture. The manuscript is honest about its main limitations (censored data, absence of a meta-experiment), and it reports a real deployment rather than a simulation. However, the evidence for the headline claim—that the cluster-randomized design 'gives a good estimate of the actual lift'—is thin: the supporting quantitative comparison is non-concurrent and lacks uncertainty quantification, the censoring problem is acknowledged but not mitigated, and the DID results are reported only qualitatively. The methodology is coherent enough to warrant revision, but not yet sufficient to support the causal claim as stated.

major comments (4)
  1. [Section 6 and Section 3, Eq. (1)] Section 6 explicitly concedes that 'such observational data is censored, that is, we only have data when we win the auction.' This concession applies to the entire construction in Section 3: the bipartite graph, the weights in Eq. (1), and the reported 36% leakage statistic are computed from impressions on won auctions only. Since interference for a bidding change operates at the auction stage before winners are determined, losing auctions are exactly the cases most likely to carry spillover, and missing edges are not missing at random. The paper provides no argument or bound showing that clusters built from censored data control true interference; the claim that the design 'gives a good estimate of the actual lift' is therefore not established. Please provide either a formal sensitivity analysis under a model of censoring, a comparison against an uncensored SERP data source (e.g., the seoClarity data mentioned in Section 6), or a synthetic-data study demonstrating that the censoring does not materially change the estimated spillover structure.
  2. [Section 3, 'Magnitude of Spillover' and Section 6] The only quantitative support for bias reduction is the comparison between a prior product-split experiment (~44% lift) and the actual post-rollout lift (~24%). The authors themselves note in Section 3 that the proper validation is a meta-experiment randomizing over two designs (citing Saveski et al.), and they did not run it. Moreover, the 44% and 24% figures come from different time periods and no confidence intervals are reported. As a consequence, the claimed 'overstatement' is not statistically characterized, and this comparison cannot validate the cluster-DID estimator. Please report confidence intervals for both lifts, or replace the comparison with a formal bias bound connecting the leakage L to the bias of the cluster-randomized DID estimator.
  3. [Section 4, 'Measurement'] The DID analysis is described only as showing 'a lift in click-through-rate for the treatment group which was consistent with the lift observed post roll out.' No point estimate, standard error, confidence interval, or p-value is reported. Without these, 'consistent' is not assessable; a sufficiently wide confidence interval could make even the 44% product-split estimate consistent with the rollout. Please report the cluster-robust DID estimate and its confidence interval, and specify what 'consistent' means in terms of a pre-specified equivalence margin or similar criterion.
  4. [Section 3, 'Optimal number of clusters'] The choice of k=10,000 clusters and the associated 36% edge-weight leakage is selected from an elbow plot, but no sensitivity analysis is reported. Because both the number of clusters and the leakage tolerance are free design choices, and because the leakage is computed from the censored graph, the paper should show that the main conclusion is robust to reasonable variations in k and L. Without this, it is unclear whether the reported result is an artifact of a particular tuning choice rather than a property of the interference structure.
minor comments (6)
  1. [Section 3, Eq. (1)] Equation (1) is difficult to parse as printed: the summation appears as '𝑛∑︀𝑖=1' without the summed expression being clearly aligned, and the roles of n, 𝑊bi, 𝑓𝑠𝑞𝑖, and the indicator function are only given in the itemized list. Please rewrite the equation with standard notation and define all quantities before first use.
  2. [Section 3, 'Graph Partitioning Methodology'] The term 'network dismantling algorithms' is used to describe METIS graph partitioning, but these are distinct families of methods; please adjust the terminology to avoid confusion.
  3. [Section 5] There is a grammatical error: 'The serves as a blueprint' should read 'This serves as a blueprint.'
  4. [Section 6] There is a typo: 'we can than use' should be 'we can then use.'
  5. [Abstract and Section 3] The abstract mentions a tripartite network, but the paper only develops the bipartite search-query/product network; please either remove the tripartite mention or clarify that it is planned future work.
  6. [Section 2] The auction format is described as 'second price Vickrey–Clarke–Groves (VCG)', but VCG is not generally a second-price auction; please rephrase to avoid an inaccurate characterization.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the cluster design is built from pre-treatment impressions, the DID estimate uses experimental outcomes, and the validation compares to an external post-rollout lift.

full rationale

Walking the claimed derivation chain, I find no step that reduces to its inputs by construction. The interference graph is built from pre-treatment search-query impression reports (Section 3, Eq. 1 defines W_uni from the bipartite impression graph), and the clusters are formed by METIS edge-cut minimization with k chosen from a leakage-vs-k elbow. None of these steps uses the experimental outcome or the later rollout lift as an input, so the cluster structure is not fitted to the quantity it is used to estimate. The DID estimate in Section 4 is computed from post-randomization outcomes and compared externally to 'the lift observed post roll out of the new bidding model' in Section 6; that is an external benchmark, not a re-statement of a fitted parameter. The only prior result from the same team is the earlier product-split estimate of ~44% lift versus ~24% actual lift, which is used as motivation for interference bias, not as evidence that the cluster design works; it is anecdotal and non-concurrent, but anecdotal weakness is not circularity. There are no load-bearing self-citations: references [1]-[31] are external works, and no uniqueness theorem is imported from the authors' own prior publications. The paper itself flags the censoring limitation in Section 6: 'such observational data is censored, that is, we only have data when we win the auction.' This is a genuine validity threat to the graph's completeness, because losing-auction co-occurrences are unobserved, but it is a data-availability limitation explicitly acknowledged by the authors, not a circular derivation. The strongest inferential overreach is labeling the design as giving 'a good estimate of the actual lift' based on consistency with a single rollout, but that is weak external validation, not a reduction of the claim to its own assumptions. No quoted reduction of the form Eq. X = Eq. Y, or fitted parameter renamed as prediction, exists in the paper. Therefore the correct circularity score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new physical or theoretical entities are introduced. The SERP interference network is a data-derived graph rather than a postulated entity. The central claim rests on three hand-chosen parameters, an ad hoc weighting function, and several domain assumptions about censored observational data. These choices are disclosed but not independently validated.

free parameters (3)
  • log-frequency downweighting in W_uni = Not fitted; base-e log chosen by hand
    Equation (1) divides each edge contribution by log_e(f_sq_i) to downweight generic queries. No derivation or sensitivity analysis is provided.
  • number of product clusters k = 10,000
    Section 3 'Optimal number of clusters' selects k by an elbow on the leakage curve, but no formal criterion or uncertainty is given.
  • leakage tolerance = 36% between-cluster edge weight
    Section 3 'Optimal number of clusters' accepts 36% edge weight across clusters as acceptable, but no analysis connects this leakage level to the bias of the treatment effect estimator.
assumptions (5)
  • domain assumption The advertiser's search query report accurately reflects the SERP co-appearance structure that generates interference.
    Section 3 'Modeling Network Interference' uses daily ad publisher reports as ground truth for query-product edges, even though these reports are censored to auctions the advertiser won.
  • ad hoc to paper W_uni(a,b) is monotonically related to the magnitude of interference between products a and b.
    Equation (1) is introduced without derivation, comparison to alternative weighting functions, or empirical calibration. It is the core modeling choice of the design.
  • domain assumption Minimizing weighted edgecut across clusters controls SUTVA violation bias.
    Section 3 'Graph Partitioning Methodology' defines leakage L as the fraction of edge weight crossing clusters and assumes minimizing this leakage is the relevant objective for reducing bias.
  • domain assumption The 44% versus 24% lift discrepancy in the earlier product-split test was caused by interference, not by other confounds.
    Section 3 'Magnitude of Spillover' attributes the difference to spillover without a concurrent control or meta-experiment, leaving time, seasonality, and other changes as alternative explanations.
  • standard math DID with cluster-robust standard errors identifies the treatment effect under cluster randomization.
    Section 4 'Measurement' invokes difference-in-differences estimation and White cluster-robust standard errors, which is a standard assumption under correct specification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SERP Interference Network and Its Applications in Search Advertising." pith.science (2026). https://pith.science/paper/F265F4GW

@misc{pith2026250621598,
  author       = {Pith},
  title        = {Pith review of: SERP Interference Network and Its Applications in Search Advertising},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F265F4GW}},
  note         = {Machine review of arXiv:2506.21598}
}
read the original abstract

Search Engine marketing teams in the e-commerce industry manage global search engine traffic to their websites with the aim to optimize long-term profitability by delivering the best possible customer experience on Search Engine Results Pages (SERPs). In order to do so, they need to run continuous and rapid Search Marketing A/B tests to continuously evolve and improve their products. However, unlike typical e-commerce A/B tests that can randomize based on customer identification, their tests face the challenge of anonymized users on search engines. On the other hand, simply randomizing on products violates Stable Unit Treatment Value Assumption for most treatments of interest. In this work, we propose leveraging censored observational data to construct bipartite (Search Query to Product Ad or Text Ad) SERP interference networks. Using a novel weighting function, we create weighted projections to form unipartite graphs which can then be use to create clusters to randomized on. We demonstrate this experimental design's application in evaluating a new bidding algorithm for Paid Search. Additionally, we provide a blueprint of a novel system architecture utilizing SageMaker which enables polyglot programming to implement each component of the experimental framework.

Figures

Figures reproduced from arXiv: 2506.21598 by the authors.

Figure 1
Figure 1. Components of interest on a Search Engine Results Page [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Note the two Shopping Ad advertisements shown in the same carousel. If these two products have bids from different bidders (i.e. control and treatment) then Stable Unit Treatment Value Assumption is violated. 3. Product-Cluster Randomized Control Trial Design To address the challenge of interference in experimental designs, we propose a preemptive modeling strategy that incorporates interference networks during the … view at source ↗
Figure 3
Figure 3. Birds-eye view of our experiment design. First, we fetch the search query reports from the ad publisher and create a search-query, product bipartite graph. Then we take a weighted one mode projection and finally use a clustering algorithm to cluster products. We then do a stratified random split of product clusters. Graph Partitioning Methodology Constructing a prod￾uct graph following the above approach then allows… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Birds-eye view of Product-cluster split generation sys￾tem. 1. Graph Generation which requires parsing >1 year of daily search query report data and Keyword data to build a graph edge list - thus requiring spark’s distributed compute. We use a cluster of memory￾optimiz…
Figure 5
Figure 5. Figure 5: Evolving Large Language Model powered SERPs with components of interest highlighted in rectangle boxes [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 28 canonical work pages

  1. [1]

    Kohavi, A

    R. Kohavi, A. Deng, B. Frasca, T. Walker, Y. Xu, N. Pohlmann, Online controlled experiments at large scale, in: Proceedings of the 19th ACM SIGKDD In- ternational Conference on Knowledge Discovery and Data Mining, 2013, pp. 1168–1176

    work page 2013

  2. [2]

    Vaver, J

    J. Vaver, J. Koehler, Measuring ad effectiveness using geo experiments, Google Inc (2011)

    work page 2011

  3. [3]

    Rubin, Causal inference using potential outcomes, J

    D. Rubin, Causal inference using potential outcomes, J. Amer. Statist. Assoc. 100 (2005) 322–331. URL: https: //doi.org/10.1198/016214504000001880. doi:10.1198/ 016214504000001880

    doi:10.1198/016214504000001880 2005

  4. [4]

    T. D. Cook, D. L. DeMets, Introduction to Statistical Methods for Clinical Trials, Chapman and Hall/CRC, 2007

    work page 2007

  5. [5]

    Bojinov, D

    I. Bojinov, D. Simchi-Levi, J. Zhao, Design and analysis of switchback experiments, Management Science 69 (2023) 3759–3777

    work page 2023

  6. [6]

    Roberts, S

    C. Roberts, S. A. Roberts, Design and analysis of clini- cal trials with clustering effects due to treatment, Clin- ical Trials 2 (2005) 152–162

    work page 2005

  7. [7]

    P. M. Aronow, C. Samii, Estimating average causal effects under general interference, with application to a social network experiment (2017)

    work page 2017

  8. [8]

    Backstrom, J

    L. Backstrom, J. Kleinberg, Network bucket testing, in: Proceedings of the 20th International Conference on World Wide Web, 2011, pp. 615–624

    work page 2011

Show all 31 references
  1. [9]

    Ugander, B

    J. Ugander, B. Karrer, L. Backstrom, J. Kleinberg, Graph cluster randomization: Network exposure to multiple universes, in: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2013, pp. 329–337

    2013

  2. [10]

    Holtz, R

    D. Holtz, R. Lobel, I. Liskovich, S. Aral, Reducing interference bias in online marketplace pricing experi- ments, arXiv preprint arXiv:2004.12489 (2020)

    2020 arXiv

  3. [11]

    G. W. Basse, H. A. Soufiani, D. Lambert, Randomiza- tion and the pernicious effects of limited budgets on auction experiments, in: Artificial Intelligence and Statistics, PMLR, 2016, pp. 1412–1420

    2016

  4. [12]

    Ostrovsky, M

    M. Ostrovsky, M. Schwarz, Reserve prices in internet advertising auctions: A field experiment, in: Pro- ceedings of the 12th ACM Conference on Electronic Commerce, 2011, pp. 59–60

    2011

  5. [13]

    Johari, H

    R. Johari, H. Li, I. Liskovich, G. Y. Weintraub, Experi- mental design in two-sided platforms: An analysis of bias, Management Science 68 (2022) 7069–7089

    2022

  6. [14]

    Bajari, B

    P. Bajari, B. Burdick, G. W. Imbens, L. Masoero, J. Mc- Queen, T. Richardson, I. M. Rosen, Multiple randomiza- tion designs, arXiv preprint arXiv:2112.13495 (2021)

    2021

  7. [15]

    Vickrey, Counterspeculation, auctions, and com- petitive sealed tenders, The Journal of Finance 16 (1961) 8–37

    W. Vickrey, Counterspeculation, auctions, and com- petitive sealed tenders, The Journal of Finance 16 (1961) 8–37

    1961

  8. [16]

    E. H. Clarke, Multipart pricing of public goods, Public Choice (1971) 17–33

    1971

  9. [17]

    Groves, Incentives in teams, Econometrica: Journal of the Econometric Society (1973) 617–631

    T. Groves, Incentives in teams, Econometrica: Journal of the Econometric Society (1973) 617–631

    1973

  10. [18]

    L. H. Moulton, Covariate-based constrained random- ization of group-randomized trials, Clinical Trials 1 (2004) 297–305

    2004

  11. [19]

    M. G. Hudgens, M. E. Halloran, Toward causal in- ference with interference, Journal of the American Statistical Association 103 (2008) 832–842

    2008

  12. [20]

    Stram, P

    R. Stram, P. Reuss, K.-D. Althoff, Weighted one mode projection of a bipartite graph as a local similarity mea- sure, in: Case-Based Reasoning Research and Devel- opment: 25th International Conference, ICCBR 2017, Trondheim, Norway, June 26-28, 2017, Proceedings 25, Springer, ...

    2017

  13. [21]

    Braunstein, L

    A. Braunstein, L. Dall’Asta, G. Semerjian, L. Zdeborová, Network dismantling, Proceedings of the National Academy of Sciences 113 (2016) 12368–12373

    2016

  14. [22]

    M. E. Newman, Detecting community structure in networks, The European Physical Journal B 38 (2004) 321–330

    2004

  15. [23]

    Girvan, M

    M. Girvan, M. E. Newman, Community structure in social and biological networks, Proceedings of the National Academy of Sciences 99 (2002) 7821–7826

    2002

  16. [24]

    M. E. Newman, Fast algorithm for detecting commu- nity structure in networks, Physical Review E 69 (2004) 066133

    2004

  17. [25]

    F. Lin, W. W. Cohen, Power iteration clustering (2010)

    2010

  18. [26]

    Karypis, V

    G. Karypis, V. Kumar, Metis: A software package for partitioning unstructured graphs, partitioning meshes, and computing fill-reducing orderings of sparse ma- trices (1997)

    1997

  19. [27]

    Saveski, J

    M. Saveski, J. Pouget-Abadie, G. Saint-Jacques, W. Duan, S. Ghosh, Y. Xu, E. M. Airoldi, Detecting network effects: Randomizing over randomized exper- iments, in: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2017, pp. 1027–1035

    2017

  20. [28]

    J. D. Angrist, J.-S. Pischke, Mostly Harmless Economet- rics: An Empiricist’s Companion, Princeton University Press, 2009

    2009

  21. [29]

    White, Asymptotic Theory for Econometricians, Academic Press, 2014

    H. White, Asymptotic Theory for Econometricians, Academic Press, 2014

    2014

  22. [30]

    A. V. Joshi, A. V. Joshi, Amazon’s machine learning toolkit: Sagemaker, Machine Learning and Artificial Intelligence (2020) 233–243

    2020

  23. [31]

    Arellano, Computing robust standard errors for within-groups estimators, Oxford Bulletin of Eco- nomics & Statistics 49 (1987)

    M. Arellano, Computing robust standard errors for within-groups estimators, Oxford Bulletin of Eco- nomics & Statistics 49 (1987). A. Components of Interest on Recent LLM Powered SERPs Figure 5:Evolving Large Language Model powered SERPs with components of interest highlighted...

    1987

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.