Pith. sign in

REVIEW 4 major objections 4 minor 31 references

Fine-tuned vision-language models can replace human relevance judgments in online A/B experiments, with query-level sDCG@K error within 0.03 and a 6x reduction in minimum detectable effect.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 07:11 UTC pith:ZC3JHJYJ

load-bearing objection Worth refereeing: a credible industry-scale VLM relevance pipeline with real MDE gains, though the validation section needs better hygiene before the headline alignment numbers can be taken at face value. the 4 major comments →

arxiv 2608.02446 v1 pith:ZC3JHJYJ submitted 2026-08-03 cs.IR cs.LG

Advancing Relevance Measurement with Vision-Language Models for Web-Scale Search

Han Wang , Alex Whitworth , Pak Ming Cheung , Zhenjie Zhang , Krishna Kamath , Xi Chen , Roberto Konow , Kurchi Subhra Hazra This is my paper
classification cs.IR cs.LG
keywords vision-language modelsrelevance measurementA/B testingsearch evaluationstratified samplingsDCG@KLLM-as-a-judgehuman annotation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that a fine-tuned vision-language model can take over the job of judging search relevance for online A/B experiments. On a five-level rating scale, the model's labels match human annotations 82.9% of the time, and when aggregated into the query-level sDCG@K metric the mean error stays below 0.03, with paired control-treatment differences essentially unbiased. Because automated labeling is cheap and fast, the paper can expand the query sample and switch to stratified sampling, cutting the minimum detectable effect roughly sixfold. If correct, search platforms can guard against relevance degradation in experiments without waiting days for human annotation.

Core claim

The paper's central discovery is that VLM-generated relevance labels, when aggregated into query-level sDCG@K, are accurate enough to support live A/B experiment decisions. The mean query-level error is 0.021 overall, paired difference error is 0.000, and the P10–P90 error range is tight (roughly ±0.1). This errors-in-variables calibration, not a new model architecture, is what lets the authors trust automated labels as a guardrail metric. It also unlocks a sampling redesign: using the VLM to label hundreds of thousands of pairs at 1/50,000th the per-label cost and 20x faster turnaround, the system moves from simple random sampling to stratified query sampling, reducing MDE from 1.3–1.5% to

What carries the argument

The mechanism is a fine-tuned open-source vision-language model that takes a query, a pin image, and pin text metadata (title, description, board titles, engaged queries) and outputs a 1–5 relevance label by argmax over the logits. These labels feed the sDCG@K metric, and a paired query-sampling design with stratification by query popularity and interest category blocks between-query variance. The MDE reduction is decomposed through the formula MDE = (z_alpha/2 + z_beta) * sqrt(2 sigma^2/n) / mu, showing that stratification removes between-stratum variance and sample-size increases further shrink the detectable lift.

Load-bearing premise

The load-bearing premise is that the human-labeled pairs used to validate the VLM are genuinely disjoint from the pairs used to fine-tune and select the model; if they overlap, the reported agreement and error bounds are inflated.

What would settle it

Take the set of human-labeled query-pin pairs from live experiment traffic and check whether any of them appear in the 0.8M training pairs or 20K model-selection pairs; if the overlap is nonzero, recompute the 82.9% exact match and 0.021 mean error on a truly held-out set. A second check: run an A/A experiment with the VLM pipeline and verify that paired sDCG@K differences are centered at zero, which would fail if VLM bias is asymmetric.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Relevance evaluation for search A/B tests can run in hours instead of days, at a fraction of the human-labeling cost.
  • Stratified query sampling, previously impractical with human labels, becomes routine and cuts the minimum detectable effect roughly sixfold.
  • The VLM's tight error distribution, not just low mean error, is what makes the metric trustworthy for experiment guardrails.
  • The approach extends to non-English markets, with moderate-to-strong rank correlation and paired-difference errors near zero.
  • Scaling query sets and increasing evaluation frequency becomes feasible, enabling broader and more frequent relevance measurement.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the validation pairs overlap with fine-tuning or model-selection pairs, the reported alignment would be optimistic; a fresh disjoint human-labeled set would settle this.
  • One could stress-test the pipeline by running an A/A experiment and checking whether VLM-derived paired differences are centered at zero; if not, label bias is leaking into the estimate.
  • The MDE gains may transfer to other platforms only if the VLM is fine-tuned on that platform's content; zero-shot generalization is not established.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper describes a deployed pipeline at Pinterest Search in which a fine-tuned Qwen3-VL-4B model assigns 5-level relevance labels to query–Pin pairs and computes a query-level sDCG@K metric for online A/B experiments. The authors report that the VLM labels align closely with human labels (82.9% exact match, QWK 0.507, mean query-level sDCG@K error below 0.03), that paired-difference errors are near zero, and that VLM labeling enables stratified query sampling and larger sample sizes, yielding a 6× reduction in Minimum Detectable Effects. They also report validation in three non-English markets. The paper frames its contribution as an end-to-end industrial deployment rather than a new modeling architecture.

Significance. If the validation evidence is sound, this is a practically significant demonstration that VLM-generated relevance labels can serve as a guardrail metric in industrial search A/B testing. The strengths of the paper are its clear system description, the explicit MDE decomposition in Section 4.2, the discussion of why CUPED is unsuitable (Section 3.3), and the honest disclosure of the prior text-only workshop version in footnote 1. The claims are not overdrawn in scope, and the practical effect—scaling relevance measurement while reducing MDEs—is valuable. However, the central evidence depends on properties of the validation set and statistical reporting that are currently not established, so the strength of the contribution is moderated by missing disclosure and missing baselines.

major comments (4)
  1. [§4.1 / §3.2] The validation set is described only as 'query-Pin pairs from live experiment traffic.' The paper never states that these pairs are disjoint from the 0.8M fine-tuning pairs and the ~20K model-selection pairs described in Section 3.2. If the same or near-duplicate pairs were used, the reported 82.9% exact match, QWK 0.507, and the error distributions in Table 1 would reflect memorization rather than generalization. Please disclose the validation-set construction, include an explicit disjointness statement, and report the validation sample size.
  2. [§4.1] The sentence 'which is comparable to the inter-annotator agreement' is unsupported. No human–human agreement statistic (exact match, adjacent match, or QWK) is reported for the same rating scale and validation pairs. Without this baseline, 'comparable' is uncheckable, and the sufficiency of QWK 0.507 cannot be assessed. Add a human–human agreement benchmark for the same validation data or revise the claim.
  3. [§4.1, Tables 1–3] No confidence intervals or sample sizes are given for QWK, Kendall's tau, Spearman's rho, or the error percentiles. The central claim that mean error 'remains within 0.03' depends on point estimates with no measure of precision. Without n and interval estimates, the reader cannot judge whether the differences across popularity segments or non-English markets are meaningful. Report n per segment and bootstrap or analytic CIs for the key alignment metrics.
  4. [§4.1, Figure 3 and Table 1] The claim that paired differences 'eliminate VLM label bias' assumes that VLM mislabeling is uncorrelated with treatment assignment. A ranking change that shifts the distribution of error-prone content (e.g., by image type, query category, or popularity) could alter the paired-difference error distribution and bias the treatment effect. The paper should state this caveat explicitly or provide a falsifiable check, such as error stratified by treatment/control and content characteristics.
minor comments (4)
  1. [§4.2, Table 4] The row 'Stratified SRS 5000' is difficult to interpret: if the SRS estimator is applied to a stratified sample, the variance reduction shown is not an isolated effect of stratification. A cleaner factorial decomposition—SRS/SRS at both n and Stratified/Stratified at both n—would make the attribution clearer.
  2. [§4.3, Table 3] For non-English markets, only point estimates are reported. The claim that the model 'generalizes effectively' would be strengthened by CIs and sample sizes, particularly for BR where the P90 query-level error is 0.140, noticeably larger than in the US.
  3. [§3.3, Eq. (1)] The notation V(Y_SRS) in Eq. (1) is nonstandard: the quantity displayed is a decomposition of the variance of the sample mean under SRS. Consider rewriting to clarify that the between-strata term is removed by the stratified estimator, not by the population variance decomposition.
  4. [Figure 3] Axis labels and legends are not legible in the submitted version; please ensure high-resolution figures in the camera-ready version.

Circularity Check

0 steps flagged

No circularity: the VLM validation and MDE decomposition are grounded in human-labeled live-traffic comparisons and standard statistical identities; self-citations are not load-bearing.

full rationale

The paper's central derivation is not circular. The VLM is fine-tuned on ~0.8M human-annotated pairs and evaluated against query-Pin pairs from live experiment traffic, with agreement reported as exact match, QWK, rank correlations, and sDCG@K error. This is a supervised-learning validation, not an identity: the validation metric is computed from labels the model did not see during training, and nothing in the equations defines sDCG@K in terms of VLM outputs alone. The 6× MDE reduction is derived from the standard MDE formula and the stratified-sampling variance decomposition in Equation 1, both of which are independent statistical facts rather than consequences of the model's predictions. The self-citations [25, 26] are transparently identified as prior work and are not used to justify the main empirical claims: [26] is a preliminary text-only workshop version, and [25] provides a text-only baseline. Neither is invoked as a uniqueness theorem or as an alternative to evidence. The only concern that could be raised is the lack of an explicit statement that the live-traffic validation pairs are disjoint from the training and model-selection pairs, but the paper does not state or rely on any such overlap, and the absence of an explicit disjointness claim is a data-hygiene reporting gap, not a circular-reasoning step. Accordingly, no load-bearing reduction of the claimed result to its own inputs is exhibited.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 0 invented entities

The central claims rest on standard statistical formulas and on assumptions about label quality and data hygiene. The most consequential unverified premises are the independence of validation labels from training data (Sec 4.1 vs 3.2) and the out-of-sample validity of the stratum choice (Sec 3.3).

free parameters (3)
  • Query stratum definitions (interest category × popularity segment) = DistilBERT query-interest model; cross-product with popularity segment
    Selected after comparing BERTopic and DistilBERT on the same dataset; reported variance reduction (Table 4) is not split-sample validated.
  • Neyman sample allocation ratios = derived from estimated stratum variances
    Optimal allocation uses variance estimates from the sample; noise in these estimates can make MDE reductions optimistic.
  • K in sDCG@K = 25
    K=25 is a manually chosen design constant for the headline relevance metric; results may change with K.
axioms (6)
  • domain assumption Human 5-level relevance labels are ground truth for semantic relevance.
    The entire validation treats human labels as the benchmark; labeled data and guidelines in Sec 3.1/4.1.
  • domain assumption Validation labels from live experiment traffic are independent of the training and model-selection data.
    Disjointness is not explicitly stated; needed for the alignment statistics in Sec 4.1.
  • domain assumption Aggregated 'highly engaged queries' used as Pin text metadata are content annotations rather than leakage of the outcome being predicted.
    Sec 3.2: engagement-derived queries are added to model input; if they encode the same relevance/engagement signal being measured, alignment could be inflated for engagement-correlated queries.
  • domain assumption Paired treatment-control sampling blocks between-query variance and causes VLM label bias to cancel in paired differences.
    Sec 3.4 and Table 1 rely on paired differences eliminating bias; this fails if error is correlated with treatment.
  • ad hoc to paper The chosen query strata yield variance-reduction estimates that generalize out of sample.
    The stratum definitions were selected after comparing options on the data used to report Table 4; no split-sample validation is described.
  • standard math Variance decomposition Eq (1) and MDE Eq (3) apply under the deployed sampling scheme.
    Both are textbook formulas (Thompson Ch. 11; Bloom 1995); the application is standard.

pith-pipeline@v1.3.0-daily-deepseek · 9004 in / 12614 out tokens · 110291 ms · 2026-08-04T07:11:54.259836+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Advancing Relevance Measurement with Vision-Language Models for Web-Scale Search." pith.science (2026). https://pith.science/paper/ZC3JHJYJ

@misc{pith2026260802446,
  author       = {Pith},
  title        = {Pith review of: Advancing Relevance Measurement with Vision-Language Models for Web-Scale Search},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZC3JHJYJ}},
  note         = {Machine review of arXiv:2608.02446}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Relevance evaluation plays a crucial role in personalized search systems, serving as a guardrail alongside user engagement metrics to ensure that search results align with user queries and intent. While human annotation is the traditional method for relevance evaluation, its high cost and long turnaround time limit its scalability. In this work, we present a VLM-based automated relevance evaluation pipeline deployed within Pinterest Search for online A/B experiments. We rigorously validate the alignment between VLM-generated judgments and human annotations, demonstrating that VLMs can provide reliable relevance measurement for experiments while greatly improving the evaluation efficiency. Leveraging VLM-based labeling further unlocks opportunities to expand the query set, optimize sampling design, and efficiently assess a wider range of search experiences at scale. This approach leads to higher-quality relevance metrics and significantly reduces the Minimum Detectable Effects (MDEs) in online experiment measurements.

Figures

Figures reproduced from arXiv: 2608.02446 by Alex Whitworth, Han Wang, Krishna Kamath, Kurchi Subhra Hazra, Pak Ming Cheung, Roberto Konow, Xi Chen, Zhenjie Zhang.

Figure 1
Figure 1. Figure 1: The fine-tuned VLM takes query text, Pin text meta [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The VLM-based relevance measurement pipeline at Pinterest Search, consisting of search result sampling with [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Query-level 𝒔𝑫𝑪𝑮@𝑲 error distribution for single group (left) and paired differences (right) in the US market relevance evaluation. We next evaluate Qwen3-VL across different model sizes and compare against a text-only XLM-RoBERTa-large baseline aug￾mented with BLIP-generated image captions [25], to assess the accuracy-efficiency tradeoff and the value of visual inputs. Results for the US market are summar… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 12 linked inside Pith

  1. [1]

    Zahra Abbasiantaeb, Chuan Meng, Leif Azzopardi, and Mohammad Aliannejadi

  2. [2]

    Negar Arabzadeh and Charles LA Clarke. 2025. Benchmarking LLM-based Rel- evance Judgment Methods. InProceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. 3194–3204

    2025

  3. [3]

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al . 2025. Qwen3-VL Technical Report.arXiv preprint arXiv:2511.21631(2025)

    Pith/arXiv arXiv 2025

  4. [4]

    Yoav Benjamini and Yosef Hochberg. 1995. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.Journal of the Royal statistical society: series B (Methodological)57, 1 (1995), 289–300

    1995

  5. [5]

    Howard S Bloom. 1995. Minimum Detectable Effects: A Simple Way to Report the Statistical Power of Experimental Designs.Evaluation Review19, 5 (1995), 547–556

    1995

  6. [6]

    Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.Psychological Bulletin70 (1968), 213–220

    1968

  7. [7]

    Alex Deng, Ya Xu, Ron Kohavi, and Toby Walker. 2013. Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data.Proceedings of the Sixth ACM International Conference on Web Search and Data Mining(2013)

    2013

  8. [8]

    Guglielmo Faggioli, Laura Dietz, Charles LA Clarke, Gianluca Demartini, Matthias Hagen, Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Potthast, Benno Stein, et al. 2023. Perspectives on Large Language Models for Relevance Judgment. InProceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval. 39–50

    2023

  9. [9]

    Ramin Giahi, Kehui Yao, Sriram Kollipara, Kai Zhao, Vahid Mirjalili, Jianpeng Xu, Topojoy Biswas, Evren Korpeoglu, and Kannan Achan. 2025. VL-CLIP: Enhancing Multimodal Recommendations via Visual Grounding and LLM-Augmented CLIP Embeddings. InProceedings of the Nineteenth ACM Conference on Recommender Systems. 482–491

    2025

  10. [10]

    Maarten Grootendorst. 2022. BERTopic: Neural topic modeling with a class-based TF-IDF procedure.arXiv preprint arXiv:2203.05794(2022)

    Pith/arXiv arXiv 2022

  11. [11]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al . 2024. A Survey on LLM-as-a-Judge.arXiv preprint arXiv:2411.15594(2024)

    Pith/arXiv arXiv 2024

  12. [12]

    Alpa Jain and Gilad Mishne. 2010. Organizing Query Completions for Web Search. InProceedings of the 19th ACM International Conference on Information and Knowledge Management. 1169–1178

    2010

  13. [13]

    Sean MacAvaney and Luca Soldaini. 2023. One-Shot Labeling for Automatic Rel- evance Estimation. InProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2230–2235

    2023

  14. [14]

    Chuan Meng, Negar Arabzadeh, Arian Askari, Mohammad Aliannejadi, and Maarten de Rijke. 2025. Query Performance Prediction Using Relevance Judg- ments Generated by Large Language Models.ACM Transactions on Information Systems43, 4 (2025), 1–35

    2025

  15. [15]

    Luke W Miratrix, Jasjeet S Sekhon, and Bin Yu. 2013. Adjusting Treatment Effect Estimates by Post-Stratification in Randomized Experiments.Journal of the Royal Statistical Society Series B: Statistical Methodology75, 2 (2013), 369–396

    2013

  16. [16]

    Jerzy Neyman. 1934. On the Two Different Aspects of the Representative Method: The Method of Stratified Sampling and the Method of Purposive Selection.Journal of the Royal Statistical Society97, 4 (1934), 158–606

    1934

  17. [17]

    Catarina Pires, Sérgio Nunes, and Luís Filipe Teixeira. 2025. Expanding Relevance Judgments for Medical Case-based Retrieval Task with Multimodal LLMs.arXiv preprint arXiv:2506.17782(2025)

    Pith/arXiv arXiv 2025

  18. [18]

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Dis- tilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108(2019)

    Pith/arXiv arXiv 2019

  19. [19]

    Shreya Shankar, JD Zamfirescu-Pereira, Björn Hartmann, Aditya Parameswaran, and Ian Arawjo. 2024. Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. InProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology. 1–14

    2024

  20. [20]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530(2024)

    Pith/arXiv arXiv 2024

  21. [21]

    Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2024. Large Language Models can Accurately Predict Searcher Preferences. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1930–1940

    2024

  22. [22]

    2012.Sampling

    Steven K Thompson. 2012.Sampling. John Wiley & Sons, Chapter 11, 141–170

    2012

  23. [23]

    Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Daniel Campos, Nick Craswell, Ian Soboroff, Hoa Trang Dang, and Jimmy Lin. 2024. A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look. arXiv preprint arXiv:2411.08275(2024)

    Pith/arXiv arXiv 2024

  24. [24]

    Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Nick Craswell, and Jimmy Lin. 2024. UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor.arXiv preprint arXiv:2406.06519(2024)

    Pith/arXiv arXiv 2024

  25. [25]

    Han Wang, Mukuntha Narayanan Sundararaman, Onur Gungor, Yu Xu, Krishna Kamath, Rakesh Chalasani, Kurchi Subhra Hazra, and Jinfeng Rao. 2024. Improv- ing Pinterest Search Relevance Using Large Language Models.arXiv preprint arXiv:2410.17152(2024)

    Pith/arXiv arXiv 2024

  26. [26]

    Han Wang, Alex Whitworth, Pak Ming Cheung, Zhenjie Zhang, and Krishna Kamath. 2025. LLM-based Relevance Assessment for Web-Scale Search Evaluation at Pinterest.arXiv preprint arXiv:2509.03764(2025)

    arXiv 2025

  27. [27]

    Huizhi Xie and Juliette Aurisset. 2016. Improving the Sensitivity of Online Controlled Experiments: Case Studies at Netflix. InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 645– 654

    2016

  28. [28]

    Jheng-Hong Yang and Jimmy Lin. 2024. Toward Automatic Relevance Judgment using Vision–Language Models for Image–Text Retrieval Evaluation.arXiv preprint arXiv:2408.01363(2024)

    Pith/arXiv arXiv 2024

  29. [29]

    Chengcan Ye, Ting Peng, Tim Chang, Zhiyi Zhou, and Feng Wang. 2023. Query- aware Multi-modal based Ranking Relevance in Video Search. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track. 322–330

    2023

  30. [30]

    Xing, Haotong Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Haotong Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena.ArXivabs/2306.05685 (2023)

    Pith/arXiv arXiv 2023

  31. [2024]

    Can We Use Large Language Models to Fill Relevance Judgment Holes? arXiv preprint arXiv:2405.05600(2024)

    Pith/arXiv arXiv 2024