Pith. sign in

REVIEW 3 major objections 5 minor 39 references

SAGE claims that human product judgment about search relevance can be formalized into a versioned rubric and scaled by a calibrated, distilled LLM judge that outperforms engagement proxies and lifts a core business metric.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

SAGE co-evolves a relevance policy, expert-curated precedents, and a distilled LLM judge to grade search relevance at production scale, reporting 0.72–0.73 linear Cohen's kappa against humans and a 0.25% DAU lift at LinkedIn.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection The integration is real and the candor is refreshing, but the headline kappas are measured against a benchmark the judge itself helped edit, so treat those numbers as human-machine agreement, not independent human agreement. the 3 major comments →

arxiv 2602.07840 v4 pith:AHGE6C64 submitted 2026-02-08 cs.IR cs.AI

SAGE: Scalable AI Governance & Evaluation

Benjamin Le , Xueying Lu , Nick Stern , Wenqiong Liu , Igor Lapchuk , Xiang Li , Baofen Zheng , Kevin Rosenberg
show 12 more authors
Jiewen Huang Zhe Zhang Abraham Cabangbang Satej Milind Wagle Jianqiang Shen Raghavan Muthuregunathan Abhinav Gupta Mathew Teoh Andrew Kirk Thomas Kwan Jingwei Wu Wenjing Zhang
This is my paper
classification cs.IR cs.AI
keywords AI governancesearch relevanceLLM-as-a-judgeknowledge distillationbidirectional calibrationpolicy precedentproduction evaluationCohen's kappa
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that human product judgment about search relevance can be turned into a machine-executable, versioned rubric through a bidirectional calibration loop in which a natural-language Policy, a small curated set of Precedent examples, and an LLM judge co-evolve. It then compresses the calibrated teacher judge into an 8-billion-parameter student model that scores query-document pairs at over ten million judgments per day, achieving 0.72–0.73 linear weighted Cohen's kappa against expert labels at 92× lower cost than the teacher. Deployed in the job-search and people-search systems of a large professional network, this evaluation signal guided offline candidate selection and online monitoring, caught quality regressions that engagement metrics missed, and drove a 0.25% lift in daily active users through two online experiments. If correct, SAGE shows that human judgment can become a scalable governance commodity rather than a bottleneck for AI development.

Core claim

SAGE's central discovery is that relevance governance can be formalized as a three-object co-evolution: a natural-language Policy, a few hundred expertly curated Precedent cases, and an LLM Surrogate Judge. Bidirectional feedback lets the judge correct precedent and expose policy gaps, not just be calibrated by humans. After this loop, the teacher judge reached 0.77 linear weighted Cohen's kappa against expert human judgment, and distillation into an 8-billion-parameter student preserved 0.72 (Job Search) and 0.73 (People Search) agreement with expert labels at roughly 1/92 of the teacher's cost. In production, the student-powered pipeline ranked candidates, gated releases, monitored live tr

What carries the argument

The load-bearing mechanism is the bidirectional calibration loop, an iterative process with four feedback vectors: Human→Policy (intuition gaps), Human→Human (inter-rater kappa for ambiguity detection), Judge→Precedent (adversarial audit that corrects human labeling errors), and Judge→Policy (edge-case discovery). The formal target is alignment divergence Δ(J,H) between surrogate judge J and latent human utility H; the loop minimizes Δ while updating the benchmark itself, and teacher–student distillation then transfers the calibrated judgment into a compact student with the same prompt and output schema, preserving alignment at 92× lower cost.

Load-bearing premise

The load-bearing premise is that when the judge disagrees with an expert-labeled precedent, the judge is more often right than wrong — because those judge-corrected precedents are then used as the human benchmark for the headline agreement scores; if judge disagreements instead reflect systematic bias, the calibration loop silently fits 'ground truth' to the judge and the reported kappas overstate alignment with independent human judgment.

What would settle it

Have an independent pool of expert annotators who took no part in calibration label a held-out set of query-document pairs without seeing judge rationales, then compute the student judge's linear weighted Cohen's kappa against that set; if agreement falls below the 0.7 substantial-agreement threshold, the near-human alignment claim is falsified, leaving only the platform's DAU experiments as evidence that the co-evolved relevance standard is correct.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Evaluation capacity rises from sparse manual review to tens of millions of graded judgments per day, making relevance a continuously monitored quantity rather than a sampled one.
  • Release gating can happen offline: candidate models are scored on counterfactual traffic before users see them, cutting candidate selection cycles from two weeks to three days and catching a -16% Good Recall@10 regression before exposure.
  • Policy-based metrics can surface silent degradations: a 40% week-over-week rise in Poor Match Rate@10 occurred while the closest engagement-based proxy stayed flat.
  • Enforcing a policy-based relevance bar in the job-alert pipeline lowered Poor Match Rate@10 by roughly 29–51%, improved NDCG@10 and Good Recall@10, and drove a combined +0.25% lift in daily active users in two statistically significant experiments.
  • Student–teacher agreement of 0.81 and human-expert agreement ceilings of 0.78–0.83 imply the student is near the practical fidelity ceiling, so further gains would come from improving the human standard, not from a larger student.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the co-evolution loop is what carries the result, the recipe should transfer to other domains with subjective quality judgments (content moderation, support triage, recommender review) wherever a compact expert-curated precedent set can be assembled; a testable extension is reproducing a 0.7+ kappa in a fresh domain with an independent expert panel.
  • Because the benchmark and the calibration share the same precedent, the reported kappas measure agreement with the co-evolved standard, not with independent human judgment; the DAU experiments are the only external check that the co-evolved notion of relevance is right.
  • The online model's compressed 1-token reasoning and the student judge's fixed training data both risk drift as language, job postings, and member behavior evolve; a natural extension is scheduled recalibration on drifted traffic samples to keep the policy-to-judge alignment current.
  • The paper implies a division of labor shift: human effort moves from producing labels to resolving ambiguity and codifying norms, which if adopted broadly would change the cost structure and talent needs of large-scale evaluation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents SAGE, a framework for scalable relevance evaluation at LinkedIn Search. It combines a natural-language Policy, a curated Precedent set, and an LLM Surrogate Judge in a 'bidirectional calibration' loop, where the Judge critiques human labels and the Policy is revised accordingly. After calibration, a GPT-o3 Teacher Judge is distilled into an 8B Student Judge, which is further compressed into an online model. The paper reports Teacher–Human linear weighted Cohen's kappa of 0.77 for People Search and Student–Human kappa of 0.72 (Job Search) and 0.73 (People Search), at 92× lower cost than the Teacher. It also reports two online A/B experiments in the Job Search Alert campaign that together produced a +0.25% lift in LinkedIn daily active users. The central claim is that SAGE operationalizes human product judgment at production scale and detects relevance regressions that engagement metrics miss.

Significance. If the reported kappas and DAU lift are valid, SAGE is a valuable industrial case study: it demonstrates a practical path from expensive frontier-model judges to high-throughput student judges, with explicit policy versioning and decomposition into interpretable relevance attributes. The work has notable strengths: the distillation results show monotone improvements across model iterations (Table 3, Figure 2), cost comparisons are normalized and clearly presented (Table 2), and the production deployment covers offline candidate selection, online experimentation, and continuous monitoring. The reported detection of a regression invisible to engagement metrics (Figure 5) is an important, falsifiable observation. However, the validity of the headline kappa numbers depends on the independence of the human benchmark, which is threatened by the Judge→Precedent feedback loop. The statistical reporting also lacks the detail needed to assess the reliability of the point estimates.

major comments (3)
  1. [§4.2, Table 1; §5.1, Figure 3] The headline kappas are computed against the Precedent set that was modified through the Judge→Precedent feedback loop. Table 1, Iteration 1 ('Corrected human labeling errors identified by Judge disagreement') shows that Judge disagreements directly change Precedent labels 'upon expert validation.' The same Precedent is then used as the benchmark for Teacher–Human and Student–Human kappa in Figure 3. The manuscript does not report the rejection rate of Judge-proposed corrections, nor a blinded re-validation of the corrected labels. If the Judge's disagreements reflect systematic bias rather than genuine human error, the calibration loop silently fits the Precedent to the Judge, inflating reported kappas. I recommend adding an independent, blinded human-validation set that was never exposed to Judge reasoning, and reporting kappa on that set separately. This is the load-bearing issue for
  2. [§5.3, Table 3; Figure 3; §6.5, Table 4] All kappa values are point estimates without confidence intervals or sample sizes. For example, Table 3 reports linear kappa rising from 0.5429 to 0.7351, but no uncertainty is given, making it impossible to judge whether the final value is statistically above the 0.7 threshold. Similarly, Figure 3 shows Teacher–Human and Student–Human kappas without error bars. §6.5 states 'All reported metrics are statistically significant (p<0.05)' but does not provide the test used, the number of observations, the number of comparisons, or confidence intervals. Given that the DAU lift is a small percentage, this lack of statistical detail undermines the quantitative headline. Please report CIs, N, and test details for all kappa and online-experiment metrics.
  3. [§6.5, Table 4] The attribution of the +0.25% DAU lift to SAGE is not fully substantiated. The paper describes two sequential online experiments but does not report how users were randomized, the duration of each experiment, the size of the treatment and control groups, whether the two iterations were overlapping or sequential, or any correction for multiple testing. The claim that this was 'the largest step-function change in growth the LinkedIn Jobs Search product has driven in years' is also unquantified. Please provide a detailed experimental protocol and a sensitivity analysis for the DAU metric. Without this, the causal link between the SAGE-guided relevance changes and the business metric remains suggestive rather than established.
minor comments (5)
  1. [§5.1] The target is described as 'linear kappa ≥ 0.7, an industry-standard threshold for substantial agreement [19].' Landis and Koch define substantial agreement as 0.61–0.8; the specific 0.7 threshold is not standard in the cited reference. Please clarify the origin of this threshold or cite a different source.
  2. [Figure 2] The arrow labels are missing spaces (e.g., '0.66SFT from 200K examples') and '0:3' should be written as '0–3' for consistency with Table 3. The figure also mixes model and policy updates without clear visual distinction.
  3. [§6.2] The definitions of Good Recall and Poor Match Rate use 'min(K,G)' and 'min(K,N)' but the notation is not introduced clearly. In the displayed equations, the summation limits are ambiguous. Please rewrite the equations with explicit set notation and define all symbols (K, N, G, s_i) before use.
  4. [Table 2] The label 'Minimum Lower-Bound Human Judge' is unclear. It would be clearer to state 'Human annotator (MTurk lower bound)' and to describe the assumptions behind the 154× figure (e.g., annotations per hour, pay rate, adjudication costs).
  5. [§6.5] The statement 'All reported metrics are statistically significant (p<0.05)' is too broad if some metrics are not primary endpoints. Please specify the primary metric for each experiment and report the test statistic and p-value for each row of Table 4.

Circularity Check

1 steps flagged

Reported 0.72–0.77 'human agreement' kappas are measured against a Precedent that the Judge itself helped correct via Judge→Precedent updates.

specific steps
  1. fitted input called prediction [§4.2 Bidirectional Calibration Loop (Table 1); reported in §4.2, §5.3, Figure 3]
    "Judge→Precedent (Adversarial Audit): ... When the Judge disagrees with a human label, we audit the reasoning trace ... Upon expert validation, we update the Precedent, leveraging the Judge to surface potential errors for human correction. ... Table 1: Iteration 1 0.71 Judge→Precedent Corrected human labeling errors identified by Judge disagreement"

    The headline kappas (Teacher 0.77; Student 0.72/0.73) are computed on the same Precedent used in the calibration loop, and that Precedent was revised when the Judge disagreed with human labels ('Upon expert validation, we update the Precedent'). For every accepted Judge disagreement, the benchmark label moves toward the Judge's output, so subsequent agreement on that item is raised by construction. The paper reports no rejection rate, no held-out subset, and no blinded re-validation; §7.2 states 'The Judge Can Audit the Ground Truth.' Thus the reported 'human agreement' is partly the calibration target, not an independent measure of human alignment.

full rationale

The central quantitative claim—near-human-level agreement (0.77 Teacher, 0.72/0.73 Student)—is not an independent evaluation: the Precedent against which these kappas are measured was co-edited by the Judge in the Judge→Precedent feedback loop (Table 1, Iteration 1). Because accepted Judge corrections convert disagreements into agreements by construction, and no blinded/holdout validation is reported, the kappa figures partially measure the Judge against its own proposed labels. This is a genuine, though partial, circularity. However, the paper also includes external evidence that is not circular: two online A/B experiments showing statistically significant PMR/NDCG/GR improvements and a +0.25% DAU lift from deploying the distilled judge-based pipeline. That external business metric indicates the co-evolved relevance concept has real-world validity, so the paper is not entirely self-referential. I found no load-bearing self-citation: reference [3] is cited only for implementation techniques (pruning, reasoning internalization), not for the core claims. The other patterns (uniqueness imported from authors, ansatz via citation, renaming known results) do not apply. Score 6 reflects one central benchmark-contamination loop that partially forces the headline agreement numbers, while the DAU A/B results provide independent support.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

No new physical or conceptual entities are postulated; the Student Judge, Online Model, and 'Reasoning Internalization' are trained artifacts/techniques, not invented theoretical constructs. The ledger burden sits on the five axioms and the three hand-chosen parameter groups listed above.

free parameters (3)
  • Attribute weighting heuristics = undisclosed
    §4.1 Decision 1: final relevance grades are produced by 'weighted heuristics' over decomposed attribute scores; the weights encode product-owner priority (e.g., Responsibilities outweigh Company Size) but their values are never stated, yet every downstream metric derives from them.
  • Good/Poor relevance thresholds = s ≥ 3 and s ≤ 1 on the 0–4 scale
    §6.2 defines operational metrics GR@K and PMR@K from these hand-set cutoffs; they determine what counts as a 'regression' in Fig. 5 and what the online models enforce.
  • Evaluation depth K = chosen per surface as expected impressions
    §6.2: 'K is chosen to be the number of documents expected to be impressed to the member'; a manual choice that changes metric sensitivity, with no sensitivity analysis reported.
axioms (5)
  • domain assumption A latent human utility function H(q,d) exists, is what search should optimize, and can be approximated by a surrogate judge J minimizing alignment divergence Δ(J,H).
    §3 defines the governance problem this way; H is never measured directly — agreement is always against product-owner precedent labels, which are themselves revised during calibration.
  • domain assumption Relevance decomposes into orthogonal attributes whose independent 0–4 scores can be recombined by fixed weighted heuristics.
    §4.1 Decision 1. The entire rubric, audit trail, and failure attribution rest on this decomposition, but no evidence of orthogonality (e.g., inter-attribute correlation or ablation) is presented.
  • domain assumption Linear weighted Cohen's kappa ≥ 0.7 is a production-grade reliability threshold.
    §5.1 imports Landis & Koch [19] descriptive labels ('substantial agreement') as a launch-gating standard; this is a statistical convention, not a validated bar for release decisions.
  • domain assumption A few hundred expert-curated precedent examples represent the unbounded query–document space.
    §4.1 Decision 3 deliberately trades quantity for fidelity; no coverage analysis of the precedent set against the long-tail traffic distribution is reported.
  • ad hoc to paper When the Judge and a human expert disagree on a Precedent label, the Judge is usually correcting genuine human error (e.g., overlooked evidence) rather than imposing its own bias.
    §4.2 Judge→Precedent Adversarial Audit and Table 1 Iteration 1; this premise is what legitimizes editing the ground truth using the judge that will then be graded against that ground truth. The paper provides only anecdotal support ('we found that the Judge frequently correctly identified...').

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of SAGE: Scalable AI Governance & Evaluation." pith.science (2026). https://pith.science/paper/AHGE6C64

@misc{pith2026260207840,
  author       = {Pith},
  title        = {Pith review of: SAGE: Scalable AI Governance & Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AHGE6C64}},
  note         = {Machine review of arXiv:2602.07840}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Evaluating relevance in large-scale search systems is fundamentally constrained by the governance gap between nuanced, resource-constrained human oversight and the high-throughput requirements of production systems. While traditional approaches rely on engagement proxies or sparse manual review, these methods often fail to capture the full scope of high-impact relevance failures. We present \textbf{SAGE} (Scalable AI Governance \& Evaluation), a framework that operationalizes high-quality human product judgment as a scalable evaluation signal. At the core of SAGE is a bidirectional calibration loop where natural-language \emph{Policy}, curated \emph{Precedent}, and an \emph{LLM Surrogate Judge} co-evolve. SAGE systematically resolves semantic ambiguities and misalignments, transforming subjective relevance judgment into an executable, multi-dimensional rubric with near human-level agreement. To bridge the gap between frontier model reasoning and industrial-scale inference, we apply teacher-student distillation to transfer high-fidelity judgments into compact student surrogates at \textbf{92$\times$} lower cost. Deployed within LinkedIn Search ecosystems, SAGE guided model iteration through simulation-driven development, distilling policy-aligned models for online serving and enabling rapid offline evaluation. In production, it powered policy oversight that measured ramped model variants and detected regressions invisible to engagement metrics. Collectively, these drove a \textbf{0.25\%} lift in LinkedIn daily active users.

Figures

Figures reproduced from arXiv: 2602.07840 by Abhinav Gupta, Abraham Cabangbang, Andrew Kirk, Baofen Zheng, Benjamin Le, Igor Lapchuk, Jianqiang Shen, Jiewen Huang, Jingwei Wu, Kevin Rosenberg, Mathew Teoh, Nick Stern, Raghavan Muthuregunathan, Satej Milind Wagle, Thomas Kwan, Wenjing Zhang, Wenqiong Liu, Xiang Li, Xueying Lu, Zhe Zhang.

Figure 1
Figure 1. Figure 1: SAGE framework: Bidirectional Calibration produces a calibrated Teacher Judge, which distills into a scalable Student [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Linear Cohen’s kappa of Student Judge across major [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Our distillation cascade: a frontier Teacher Judge [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: PMR@10 vs. Dismiss to Apply Ratio (D2A) in the [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 11 linked inside Pith

  1. [1]

    Amazon. 2026. Amazon Mechanical Turk Pricing. https://requester.mturk.com/ pricing. Accessed: 2026-02-06

    2026

  2. [2]

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional AI: Harmlessness from AI Feedback.arXiv preprint arXiv:2212.08073(2022). doi:10.48550/arXiv.2212.08073

    Pith/arXiv arXiv doi:10.48550/arxiv.2212.08073 2022

  3. [3]

    Fedor Borisyuk, Sriram Vasudevan, Muchen Wu, Guoyao Li, Benjamin H. Le, Shaobo Zhang, Kay Qianqi Shen, Yuchin Juan, Kayhan Behdin, Liming Dong, Kaixu Yang, Shusen Jing, Ravi Teja Pothamsetty, Rajat Arora, Sophie Yanying Sheng, Vitaly Abdrashitov, Yang Zhao, Lin Su, Xiaoqing Wang, Chujie Zheng, Sarang Metkar, Rupesh Gupta, Igor Lapchuk, David N. Racca, Mad...

    2026

  4. [4]

    Bertil Braun and Martin Forell. 2025. (Towards) Scalable Reliable Automated Evaluation with Large Language Models. InProceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM). 320–336

    2025

  5. [5]

    Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2023. RA- GAS: Automated Evaluation of Retrieval Augmented Generation.arXiv preprint arXiv:2309.15217(2023). doi:10.48550/arXiv.2309.15217

    Pith/arXiv arXiv doi:10.48550/arxiv.2309.15217 2023

  6. [6]

    Naghmeh Farzi and Laura Dietz. 2025. Criteria-Based LLM Relevance Judgments. arXiv preprint arXiv:2507.09488(2025)

    Pith/arXiv arXiv 2025

  7. [7]

    Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets. InProceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). do...

    doi:10.18653/v1/d19-1107 2019

  8. [8]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al . 2024. A survey on llm-as-a-judge.The Innovation(2024)

    2024

  9. [9]

    Rupesh Gupta, Chujie Zheng, and Haojun Li. 2025. Retrieval for Semantic People Search. InProceedings of the 48th ACM SIGIR. 4229–4233

    2025

  10. [10]

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the Knowledge in a Neural Network.arXiv preprint arXiv:1503.02531(2015)

    Pith/arXiv arXiv 2015

  11. [11]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685(2021)

    Pith/arXiv arXiv 2021

  12. [12]

    Renjun Hu, Yi Cheng, Libin Meng, Jiaxin Xia, Yi Zong, Xing Shi, and Wei Lin. 2025. Training an LLM-as-a-Judge Model: Pipeline, Insights, and Practical Lessons. In Companion Proceedings of the ACM on Web Conference 2025. 228–237

    2025

  13. [13]

    David R Hunter. 2004. MM algorithms for generalized Bradley-Terry models.The annals of statistics32, 1 (2004), 384–406

    2004

  14. [14]

    Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated Gain-Based Evaluation of IR Techniques.ACM Transactions on Information Systems20, 4 (2002), 422–446. doi:10.1145/582415.582418

    arXiv 2002

  15. [15]

    Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017. Unbiased learning-to-rank with biased feedback. InProceedings of the tenth ACM interna- tional conference on web search and data mining. 781–789

    2017

  16. [16]

    Yuchin Juan, Jianqiang Shen, Shaobo Zhang, Qianqi Shen, Caleb Johnson, Luke Simon, Liangjie Hong, and Wenjing Zhang. 2025. Scaling Retrieval for Web- Scale Recommenders: Lessons from Inverted Indexes to Embedding Search. In Proceedings of the Nineteenth ACM RecSys. 1066–1069

    2025

  17. [17]

    Krishnaram Kenthapadi, Benjamin Le, and Ganesh Venkataraman. 2017. Person- alized job recommendation system at linkedin: Practical challenges and lessons learned. InProceedings of the eleventh ACM RecSys. 346–347

    2017

  18. [18]

    Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo

  19. [19]

    J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data.Biometrics33, 1 (1977), 159–174

    1977

  20. [20]

    Weiyuan Li, Xintao Wang, Siyu Yuan, Rui Xu, Jiangjie Chen, Qingqing Dong, Yanghua Xiao, and Deqing Yang. 2025. Curse of knowledge: When complex evaluation context benefits yet biases llm judges.arXiv preprint arXiv:2509.03419 (2025)

    arXiv 2025

  21. [21]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). doi:10.18653/v1/2023.emnlp-main.153

    doi:10.18653/v1/2023.emnlp-main.153 2023

  22. [22]

    Xingyu Lu, Tianke Zhang, Chang Meng, Xiaobei Wang, Jinpeng Wang, Yi-Fan Zhang, Shisong Tang, Changyi Liu, Haojie Ding, Kaiyu Jiang, et al. 2025. Vlm as policy: Common-law content moderation framework for short video platform. InProceedings of the 31st KDD V. 2. 4682–4693

    2025

  23. [23]

    Navid Mehrdad, Hrushikesh Mohapatra, Mossaab Bagdouri, Prijith Chandran, Alessandro Magnani, Xunfan Cai, Ajit Puthenputhussery, Sachin Yadav, Tony Lee, ChengXiang Zhai, and Ciya Liao. 2024. Large Language Models for Relevance Judgment in Product Search. arXiv:2406.00247 [cs.IR] https://arxiv.org/abs/2406. 00247

    Pith/arXiv arXiv 2024

  24. [24]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. InSC20: Inter- national Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1–16

    2020

  25. [25]

    Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. InProceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). doi:10.18653/v1/2020.acl-main.442

    doi:10.18653/v1/2020.acl-main.442 2020

  26. [26]

    Rudd, Christopher Andrews, and Philip Tully

    Ethan M. Rudd, Christopher Andrews, and Philip Tully. 2025. A Practical Guide for Evaluating LLMs and LLM-Reliant Systems.arXiv preprint arXiv:2506.13023 (2025)

    Pith/arXiv arXiv 2025

  27. [27]

    Jaidev Shah, Iman Barjasteh, Amey Barapatre, Rana Forsati, Gang Luo, Fan Wu, Yuan Fang, Xue Deng, Blake Shepard, Ronak Shah, et al . 2025. Towards Web- scale Recommendations with LLMs: From Quality-aware Ranking to Candidate Generation. InProceedings of the 31st KDD Volume-1. 2514–2524

    2025

  28. [28]

    Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2023. Large Language Models Can Accurately Predict Searcher Preferences.arXiv preprint arXiv:2309.10621(2023)

    Pith/arXiv arXiv 2023

  29. [29]

    Nguyen Vo, Hongwei Shang, Zhen Yang, Juexin Lin, Seyed Danial Mohseni Taheri, and Changsung Kang. 2024. Knowledge Distillation for Efficient and Effective Relevance Search on E-commerce. InSIGIR Workshop on eCommerce

    2024

  30. [30]

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023. Large Language Models are not Fair Evaluators.arXiv preprint arXiv:2305.17926(2023). doi:10.48550/arXiv.2305.17926

    Pith/arXiv arXiv doi:10.48550/arxiv.2305.17926 2023

  31. [31]

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners.arXiv preprint arXiv:2109.01652(2021)

    Pith/arXiv arXiv 2021

  32. [32]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems35 (2022), 24824–24837

    2022

  33. [33]

    Dezhi Ye, Jie Liu, Junwei Hu, Jiabin Fan, Bowen Tian, Haijin Liang, and Jin Ma

  34. [34]

    Dawei Yin, Yuening Hu, Jiliang Tang, Tim Daly Jr., Mianwei Zhou, Hua Ouyang, Jianhui Chen, Changsung Kang, Hongbo Deng, Chikashi Nobata, Jean-Marc Langlois, and Yi Chang. 2016. Ranking Relevance in Yahoo Search. InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 323–332. doi:10.1145/2939672.2939677

    arXiv 2016

  35. [35]

    Xiaoyu Zhang, Yishan Li, Jiayin Wang, Bowen Sun, Weizhi Ma, Peijie Sun, and Min Zhang. 2024. Large Language Models as Evaluators for Recommendation Explanations. InProceedings of the 18th ACM Conference on Recommender Systems. 33–42. doi:10.1145/3640457.3688075

    arXiv 2024

  36. [36]

    Xing, Hao Zhang, Preprint, 2025, Preprint Le, Lu, and Stern et al

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Preprint, 2025, Preprint Le, Lu, and Stern et al. Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT- Bench and Chatbot Arena.arXiv preprint arXiv:2306.05685(2023). doi:10.48550/ arXiv.2306.05685

    Pith/arXiv arXiv doi:10.48550/arxiv.2306.05685 2025

  37. [37]

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. LIMA: Less Is More for Alignment.Advances in Neural Information Processing Systems36 (2023)

    2023

  38. [2024]

    InProceedings of the Twelfth International Conference on Learning Repre- sentations (ICLR)

    Prometheus: Inducing Fine-grained Evaluation Capability in Language Models. InProceedings of the Twelfth International Conference on Learning Repre- sentations (ICLR)

  39. [2025]

    In Proceedings of the 31st KDD Volume-2

    Applying Large Language Model For Relevance Search In Tencent. In Proceedings of the 31st KDD Volume-2. 5171–5181

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.