Pith. sign in

REVIEW 4 major objections 5 minor 35 references

Fine-tuned LLM-generated relevance labels, added at scale to ranker training, improve both textual and behavioral relevance and lift app-store conversion by 0.24%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 20:24 UTC pith:ZX2X47P4

load-bearing objection Solid industrial validation of LLM-as-judge with a real online lift; the offline Pareto claim overreaches its tiny NDCG deltas. the 4 major comments →

arxiv 2602.23234 v5 pith:ZX2X47P4 submitted 2026-02-26 cs.IR cs.AIcs.LG

Scaling Search Relevance: Augmenting App Store Ranking with LLM-Generated Judgments

Evangelia Christakopoulou , Vivekkumar Patel , Hemanth Velaga , Sandip Gaikwad , Sean Suchter , Venkat Sundaranatha This is my paper
classification cs.IR cs.AIcs.LG
keywords LLM-as-a-judgetextual relevancebehavioral relevancemulti-objective learning to rankPareto frontierdata augmentationtail queriesapp-store search
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that a fine-tuned, in-house 3-billion-parameter language model can act as a scalable offline judge, generating millions of textual relevance labels that erase the historical scarcity of expert judgments in a commercial app-store search ranker. The authors show that augmenting the ranker's multi-objective training data with these labels moves the behavioral-textual relevance Pareto frontier outward: offline NDCG rises for both objectives, and the production model's operating point is strictly dominated. A worldwide A/B test reports a statistically significant +0.24% increase in conversion rate, with the largest gains on tail queries where behavioral signals are unreliable. If true, this offers a practical blueprint for any search system facing relevance-label scarcity.

Core claim

On the paper's own terms, the central discovery is that LLM-as-a-Judge works at industrial scale when the judge is fine-tuned on the in-house human rubric: a 3B fine-tuned model reaches F1 0.800 against held-out human judgments, far above a pretrained 30B model's 0.382. Using that model as a force multiplier, the authors generate millions of pointwise textual relevance labels across storefronts and languages. Adding these labels to the ranker's scalarized multi-objective training improves NDCG@1/3/7 for both textual and behavioral relevance, a true Pareto improvement over the production model, and the online A/B test confirms the direction with a +0.24% conversion lift.

What carries the argument

The load-bearing mechanism is the multi-objective ranker trained by scalarized data mixing: behavioral labels (clicks/downloads from aggregated logs) and textual labels (human plus LLM-generated judgments) are mixed in the training set at a tunable ratio, so the same query-app pair can appear once with each label type, and gradients for each objective are computed strictly separately. The label generator is a fine-tuned 3B in-house LLM, prompted few-shot with the same query and app metadata that human judges see, instructed to output textual labels; the fine-tuning is done on a small set of existing human judgments. The tunable mixing ratio is what lets the authors traverse—rather than only

Load-bearing premise

The offline Pareto claim depends on the held-out human judgments used for NDCG evaluation being entirely disjoint from the human judgments used to fine-tune and validate the LLM judge, and from the LLM-generated labels used in ranker training—the paper asserts this separation but provides no overlap statistics.

What would settle it

Compute the intersection between the query-app pairs in the held-out textual NDCG evaluation set and the query-app pairs on which the LLM judge was fine-tuned or on which it generated training labels; any non-trivial overlap would mean the reported textual-relevance gains could reflect the ranker learning the LLM's own rubric rather than a genuinely better semantic ranking, and the Pareto claim would need to be re-evaluated.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A fine-tuned 3B model outperforms a pretrained 30B model for relevance labeling, so smaller, task-adapted LLMs can be a cost-effective replacement for much larger general-purpose ones in offline annotation.
  • Adding millions of LLM-generated labels to ranker training improves both textual and behavioral NDCG at ranks 1, 3, and 7, making the production model's operating point strictly dominated.
  • The online A/B test shows a statistically significant +0.24% conversion-rate lift on worldwide traffic, with gains in 89% of storefronts, validating the offline Pareto shift in a live environment.
  • Tail queries—those with sparse behavioral data—receive the largest conversion improvements, indicating that LLM-generated textual labels close the relevance gap where click/download signals are weakest.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the strict separation between evaluation judgments and LLM training/generation data holds, the offline Pareto gain implies that rankers can be improved without any new human annotation beyond the small seed set used to fine-tune the judge.
  • The +0.24% conversion lift may be a lower bound for the approach: the paper reports a single data-mixing point on the new Pareto frontier, and sweeping the mixing ratio could uncover operating points with even larger gains at modest trade-offs.
  • Because the LLM judge itself only reaches F1 0.8 against human judgments, the ceiling of this pipeline is capped by judge quality; an ensemble of judges or an iterative 'generate-train-rejudge' loop is a natural next step to push the frontier further.
  • The result suggests a general recipe for other search verticals: fine-tune a modest LLM on a few hundred expert judgments, generate labels offline for historical query-item pairs, and mix those labels into the existing multi-objective trainer, rather than attempting real-time LLM reranking.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper describes an industrial-scale application of the LLM-as-a-judge paradigm to App Store search. The authors fine-tune a 3B in-house LLM on human textual-relevance judgments, use it to generate millions of pointwise textual relevance labels for historical query-app pairs, and augment the training data of a production multi-objective ranker that optimizes both behavioral and textual relevance. Offline NDCG evaluations (Table 2) and a worldwide A/B test (Table 3, Figure 2) are reported, with the claimed contributions being an outward shift of the behavioral-textual Pareto frontier and a statistically significant +0.24% conversion-rate gain, particularly on tail queries.

Significance. If the claims hold, this is a practically important demonstration that LLM-generated relevance labels can be used as a scalable substitute for scarce human labels in a large commercial search system, improving both relevance objectives and yielding measurable online gains. The paper's strengths include its industrial-scale A/B test, the use of a fine-tuned smaller model over a much larger pretrained model, and its explicit attempt to separate LLM-judge training/validation judgments from ranker evaluation judgments. However, the load-bearing claims of 'strict dominance' and 'statistically significant' online gains are not backed by the statistical evidence presented; uncertainty quantification and control conditions are missing, and the separation between training/evaluation sets is asserted without overlap statistics.

major comments (4)
  1. [§4.3.2, Table 2] The claim that the llm-augmented model 'strictly dominates' prod rests on NDCG differences of 0.001–0.005 (e.g., textual NDCG@1 from 0.867 to 0.868). No confidence intervals, standard errors, paired significance tests, or evaluation-set sizes are reported. Without knowing the evaluation-set size and variance, these differences may be within noise. Please provide uncertainty quantification and specify the number of queries/query-app pairs in the ranker evaluation set. The phrase 'strictly dominating' should be qualified accordingly.
  2. [§5, Table 3] The A/B test claims a 'statistically significant +0.24%' conversion-rate increase, but no confidence interval, p-value, number of sessions, or hypothesis-testing method is given. The statement that gains were observed in 89% of storefronts implies 11% saw no improvement or a decrease; please report the per-storefront effect distribution and significance, and provide a forest plot or CI table. Figure 2's conversion-rate-difference curve also lacks axis labels and error bars, making it impossible to assess the claimed tail-query effect.
  3. [§4.3.1] The experimental design does not include a control that adds the same volume of non-LLM labels (e.g., additional human labels if available, or randomly selected behavioral labels) to the training data. Without such a control, the observed NDCG improvements could be attributed to increased training-data volume rather than to the quality of LLM-generated textual labels. Please add a data-volume-matched baseline, or explicitly justify why the existing prod comparison is sufficient.
  4. [§4.3.1] The paper states that the textual-judgment set used for NDCG evaluation of the ranker is distinct from the set used to fine-tune/validate the LLM judge, but it provides no overlap statistics between the ranker's evaluation set, the ranker's training set (including LLM-generated labels), and the LLM judge's training/validation sets. If the evaluation query-app pairs appear in the LLM-generated training data, the 'textual relevance' NDCG could be measuring the ranker's ability to reproduce the LLM's rubric rather than genuine semantic quality. Please report the overlap (e.g., percentage of evaluation pairs that appear in LLM-generated labels) and, ideally, evaluate on a held-out set with zero overlap.
minor comments (5)
  1. [Table 3] The 'Model' column in Table 3 shows 'prod' and 'llm-augmented' but the 'Conversion rate' entry is empty for prod; either remove the column or fill in the baseline value.
  2. [Figure 2] The axes are labeled 'Log Query frequency' and 'Conversion rate difference', but the y-axis does not specify units (absolute percentage points vs. relative). Add axis titles and error bars; also clarify the bucket definitions.
  3. [§2, References] Reference [14] is listed as 'Qi Liu1, ...' with superscript '1' in the author names; this should be cleaned up. Reference [21] in the bibliography lacks author names and page numbers.
  4. [§3.1] The prompt in Figure 1 includes placeholders like 'query_1', 'app_1', and 'label_1'. It would be clearer to show a concrete example or at least mention that these are anonymized placeholders for real query-app pairs.
  5. [§4.2.1] The text says 'We experimented with various prompt configurations (zero-shot vs. few-shot, textual vs. numeric labels)' but then omits results for prompt design. While this is acknowledged, the statement 'few-shot prompts with textual labels performed best' would be more convincing if accompanied by even a small table or figure.

Circularity Check

0 steps flagged

No circularity: the LLM-generated labels are evaluated against held-out human judgments and online conversion, not against their own training labels.

full rationale

The paper's derivation chain is an empirical pipeline: (1) fine-tune an in-house 3B LLM on human textual-relevance judgments; (2) generate millions of pointwise textual relevance labels; (3) augment the production ranker's multi-objective training data with these labels; (4) evaluate offline NDCG against a held-out set of human judgments; (5) validate online via a worldwide A/B test on conversion rate. No step defines its target in terms of its own inputs. The LLM judge is evaluated on a validation set distinct from its fine-tuning set (Table 1), and the paper explicitly states that 'the judgments used for computing the NDCG@k for the textual relevance are not included in the ones we used for finetuning and validation of the LLM relevance labels, as they are two distinct sets.' The ranker's textual NDCG is therefore computed from human judgments, not from the LLM-generated training labels. The online metric, conversion rate, is entirely external to the LLM labels and the offline NDCG computation. The only self-citation with overlapping authors ([28], which includes two of the present authors) appears in related work as inspiration on alignment techniques and is not load-bearing; it is not used to justify the central claim. The lack of confidence intervals or significance tests on the offline NDCG deltas is a statistical-reporting weakness about the robustness of the claimed Pareto improvement, not a circularity. Thus no circular step is present.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 0 invented entities

The central claim rests on a chain of human-judgment-driven supervision: human labels fine-tune the LLM, LLM labels train the ranker, and human labels evaluate it. The reliability of that chain depends on the independence of the evaluation judgments and on the assumption that conversion measures relevance. No fundamentally new entities are introduced.

free parameters (3)
  • Data mixing ratio (behavioral:textual)
    Section 3.2 introduces mixing ratios such as 90-10, 70-30, 50-50 as a tunable hyperparameter to position the model on the Pareto frontier; the selected ratio for the reported model is not disclosed.
  • Few-shot prompt configuration
    Section 4.2.2 states few-shot prompts with textual labels performed best but omits the shot count, example selection, and label vocabulary details.
  • LLM fine-tuning hyperparameters
    The 3B model is fine-tuned on human judgments (§4.2.1) but no training details (epochs, learning rate, data size) are given.
axioms (6)
  • domain assumption Human judgments are the ground truth for textual relevance.
    The entire validation and fine-tuning loop treats human judgments as correct; no inter-judge agreement is reported (§4.1, §4.2).
  • domain assumption The held-out human judgments used for ranker NDCG are independent of the labels used to fine-tune and validate the LLM judge.
    Stated in §4.3.1 but not empirically shown; the whole offline Pareto claim rests on this separation.
  • domain assumption Aggregated click/download logs provide valid behavioral relevance labels.
    Behavioral NDCG is computed from these logs, and they are used as training targets (§3.2, §4.1).
  • standard math NDCG@k captures relevance quality.
    Standard IR metric [11]; used for both objectives with different gain definitions.
  • domain assumption Conversion rate (session with at least one download) is the appropriate online metric.
    Used as the A/B primary metric (§5); assumes it reflects the ranker's relevance objectives.
  • domain assumption The LLM-generated labels align with the human rubric across all storefronts and languages.
    The LLM was evaluated on a validation set (presumably English), but labels are generated across 'multiple storefronts and languages'; no per-language quality check is reported (§1, §3.1).

pith-pipeline@v1.3.0-alltime-deepseek · 8218 in / 14401 out tokens · 125282 ms · 2026-08-02T20:24:16.734648+00:00 · methodology

0 comments
read the original abstract

Large-scale commercial search systems optimize for relevance to drive successful sessions that help users find what they are looking for. To maximize relevance, we leverage two complementary objectives: behavioral relevance (results users tend to click or download) and textual relevance (a result's semantic fit to the query). A persistent challenge is the scarcity of expert-provided textual relevance labels relative to abundant behavioral relevance labels. We first address this by systematically evaluating LLM configurations, finding that a specialized, fine-tuned model significantly outperforms a much larger pre-trained one in providing highly relevant labels. Using this optimal model as a force multiplier, we generate millions of textual relevance labels to overcome the data scarcity. We show that augmenting our production ranker with these textual relevance labels leads to a significant outward shift of the Pareto frontier: offline NDCG improves for behavioral relevance while simultaneously increasing for textual relevance. These offline gains were validated by a worldwide A/B test on the App Store ranker, which demonstrated a statistically significant +0.24% increase in conversion rate, with the most substantial performance gains occurring in tail queries, where the new textual relevance labels provide a robust signal in the absence of reliable behavioral relevance labels.

Figures

Figures reproduced from arXiv: 2602.23234 by Evangelia Christakopoulou, Hemanth Velaga, Sandip Gaikwad, Sean Suchter, Venkat Sundaranatha, Vivekkumar Patel.

Figure 1
Figure 1. Figure 1: Example few-shot prompt for query (query_target) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Conversion rate absolute difference of llm-augmented vs prod model vs query frequency bucket. The lower-numbered buckets correspond to lower frequency - more tail queries, while the higher-numbered buckets correspond to higher frequency-more head queries. 6 Conclusion and future work By systematically evaluating both large-scale pretrained and spe￾cialized fine-tuned in-house models, we demonstrate that LL… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 25 linked inside Pith

  1. [1]

    Abdelrahman Abdallah, Bhawna Piryani, Jamshid Mozafari, Mohammed Ali, and Adam Jatowt. 2025. How Good Are LLM-based Rerankers? An Empirical Study. CoRRabs/2508.16757 (2025). arXiv:2508.16757 https://arxiv.org/abs/2508.16757

    Pith/arXiv arXiv 2025

  2. [2]

    Negar Arabzadeh and Charles L. A. Clarke. 2025. Benchmarking LLM-based Relevance Judgment Methods.CoRRabs/2504.12558 (2025). arXiv:2504.12558 Scaling Search Relevance: Augmenting App Store Ranking with LLM-Generated Judgments Preprint, February 2026, USA https://arxiv.org/abs/2504.12558

    Pith/arXiv arXiv 2025

  3. [3]

    Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. 2022. InPars: Data Augmentation for Information Retrieval using Large Language Models. arXiv:2202.05144 [cs.CL] https://arxiv.org/abs/2202.05144

    Pith/arXiv arXiv 2022

  4. [4]

    Yiqun Chen, Qi Liu, Yi Zhang, Weiwei Sun, Xinyu Ma, Wei Yang, Daiting Shi, Jiaxin Mao, and Dawei Yin. 2024. TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategy.CoRRabs/2406.11678 (2024). arXiv:2406.11678 https://arxiv.org/abs/2406.11678

    Pith/arXiv arXiv 2024

  5. [5]

    Cheng-Han Chiang and Hung-yi Lee. 2023. Can Large Language Models Be an Alternative to Human Evaluations?. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers). Association for Computational Linguistics, Toronto, Canada, 15607–15631. https://aclanthology.org/2023.acl-long.870

    2023

  6. [6]

    Nayoung Choi, Youngjune Lee, Gyu-Hwung Cho, Haeyu Jeong, Jungmin Kong, Saehun Kim, Keunchan Park, Sarah Cho, Inchang Jeong, Gyohee Nam, et al

  7. [7]

    Gabriele Faggioli, Giorgia Marenzi, Stefano Mizzaro, and Tetsuya Sakai. 2023. Perspectives on Large Language Models for Relevance Judgment.CoRR abs/2304.09161 (2023). arXiv:2304.09161 https://arxiv.org/abs/2304.09161

    Pith/arXiv arXiv 2023

  8. [8]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2024. A Survey on LLM-as- a-Judge.CoRRabs/2411.15594 (2024). arXiv:2411.15594 https://arxiv.org/abs/ 2411.15594

    Pith/arXiv arXiv 2024

  9. [9]

    Zihan Hong, Yushi Wu, Zhiting Zhao, Shanshan Feng, Jianghong Ma, Jiao Liu, and Tianjun Wei. 2025. Multi-Objective Recommendation in the Era of Generative AI: A Survey of Recent Progress and Future Prospects.CoRRabs/2506.16893 (2025). arXiv:2506.16893 https://arxiv.org/abs/2506.16893

    Pith/arXiv arXiv 2025

  10. [10]

    Kasra Hosseini, Thomas Kober, Josip Krapac, Roland Vollgraf, Weiwei Cheng, and Ana Peleteiro Ramallo. 2024. Retrieve, Annotate, Evaluate, Repeat: Lever- aging Multimodal LLMs for Large-Scale Product Retrieval Evaluation.CoRR abs/2409.11860 (2024). arXiv:2409.11860 https://arxiv.org/abs/2409.11860

    Pith/arXiv arXiv 2024

  11. [11]

    Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques.ACM Transactions on Information Systems (TOIS)20, 4 (2002), 422–446

    2002

  12. [12]

    Qi Liu, Atul Singh, Jingbo Liu, Cun Mu, and Zheng Yan. 2024. Towards More Relevant Product Search Ranking Via Large Language Models: An Empirical Study.arXiv preprint arXiv:2409.17460(2024)

    Pith/arXiv arXiv 2024

  13. [13]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. CoRRabs/2303.16634 (2023). arXiv:2303.16634 https://arxiv.org/abs/2303.16634

    Pith/arXiv arXiv 2023

  14. [14]

    Qi Liu1, Haozhe Duan1, Yiqun Chen1, Quanfeng Lu, Weiwei Sun, and Jiaxin Mao1

  15. [15]

    Sean MacAvaney and Luca Soldaini. 2023. One-Shot Labeling for Automatic Relevance Estimation.CoRRabs/2302.11266 (2023). arXiv:2302.11266 https: //arxiv.org/abs/2302.11266

    Pith/arXiv arXiv 2023

  16. [16]

    Debabrata Mahapatra, Chaosheng Dong, and Michinari Momma. 2023. Querywise Fair Learning to Rank through Multi-Objective Optimiza- tion. https://www.amazon.science/publications/querywise-fair-learning-to- rank-through-multi-objective-optimization

    2023

  17. [17]

    Yue Meng, Cheng Guo, Yi Cao, Tong Liu, and Bo Zheng. 2025. A Generative Re-ranking Model for List-level Multi-objective Optimization at Taobao.CoRR abs/2505.07197 (2025). arXiv:2505.07197 https://arxiv.org/abs/2505.07197

    Pith/arXiv arXiv 2025

  18. [18]

    Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. 2023. RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models.CoRRabs/2309.15088 (2023). arXiv:2309.15088 https://arxiv.org/abs/ 2309.15088

    Pith/arXiv arXiv 2023

  19. [19]

    Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. 2023. Rankzephyr: Effective and robust zero-shot listwise reranking is a breeze!arXiv preprint arXiv:2312.02724(2023)

    Pith/arXiv arXiv 2023

  20. [20]

    Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Ji- aming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. 2023. Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting.CoRRabs/2306.17563 (2023). arXiv:2306.17563 https://arxiv.org/abs/2306.17563

    Pith/arXiv arXiv 2023

  21. [21]

    Rahul Raja, Arpita Vats, and Sudipta Roy. 2025. Aligning Prompts with Ranking Goals: Prompt Engineering for LLM-Based Recommendations.Preprints.org (2025). https://www.preprints.org/manuscript/202509.1959/v1

    arXiv 2025

  22. [22]

    Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. 2023. ARES: Automated Evaluation for Retrieval-Augmented Generation Systems.CoRR abs/2311.09476 (2023). arXiv:2311.09476 https://arxiv.org/abs/2311.09476

    Pith/arXiv arXiv 2023

  23. [23]

    Nilanjan Sinhababu, Andrew Parry, Debasis Ganguly, and Pabitra Mitra. 2025. Modeling Ranking Properties with In-Context Learning.CoRRabs/2505.17736 (2025). arXiv:2505.17736 https://arxiv.org/abs/2505.17736

    Pith/arXiv arXiv 2025

  24. [24]

    Jie Tang, Huiji Gao, Liwei He, and Sanjeev Katariya. 2024. Multi-Objective Learn- ing to Rank by Model Distillation.CoRRabs/2407.07181 (2024). arXiv:2407.07181 https://arxiv.org/abs/2407.07181

    Pith/arXiv arXiv 2024

  25. [25]

    Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2023. Large Lan- guage Models Can Accurately Predict Searcher Preferences.CoRRabs/2309.10621 (2023). arXiv:2309.10621 https://arxiv.org/abs/2309.10621

    Pith/arXiv arXiv 2023

  26. [26]

    Zhichao Xu, Mohammad Aliannejadi, Ben Carterette, and James Allan. 2024. Do Large Language Models Rank Fairly? An Empirical Study. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics. https://aclanthology.org/ 2024.naacl-long.319.pdf

    2024

  27. [27]

    Guo Yu, Lianbo Ma, Xilu Wang, Wei Du, Wenli Du, and Yaochu Jin. 2024. Towards Fairness-Aware Multi-Objective Optimization.Complex & Intelligent Systems (2024). https://link.springer.com/article/10.1007/s40747-024-01668-w

    doi:10.1007/s40747-024-01668-w 2024

  28. [28]

    Kai Yuan, Anthony Zheng, Jia Hu, Divyanshu Sheth, Hemanth Velaga, Kylee Kim, Matteo Guarrera, Besim Avci, Jianhua Li, Xuetao Yin, Rajyashree Mukher- jee, and Sean Suchter. 2026. Unifying Ranking and Generation in Query Auto- Completion via Retrieval-Augmented Generation and Multi-Objective Alignment. arXiv:2602.01023 [cs.IR] https://arxiv.org/abs/2602.01023

    Pith/arXiv arXiv 2026

  29. [29]

    Yang Zhao, Yixin Wang, and Mingzhang Yin. 2024. Permutative Preference Alignment from Listwise Ranking of Human Judgments.CoRRabs/2410.04346 (2024). arXiv:2410.04346 https://arxiv.org/abs/2410.04346

    arXiv 2024

  30. [30]

    Zheng, J

    C. Zheng, J. Wang, S. A. Zhang, A. Kishore, and S. Singh. 2024. Semantic Search Evaluation.CoRRabs/2410.21549 (2024). arXiv:2410.21549 https://arxiv.org/abs/ 2410.21549

    Pith/arXiv arXiv 2024

  31. [31]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685 [cs.CL] https://arxiv.org/abs/2306.05685

    Pith/arXiv arXiv 2023

  32. [32]

    Honglei Zhuang, Zhen Qin, Kai Hui, Junru Wu, Le Yan, Xuanhui Wang, and Michael Bendersky. 2023. Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels.CoRRabs/2310.14122 (2023). arXiv:2310.14122 https://arxiv.org/abs/2310.14122

    Pith/arXiv arXiv 2023

  33. [33]

    Shengyao Zhuang, Honglei Zhuang, Bevan Koopman, and Guido Zuccon. 2023. A Setwise Approach for Effective and Highly Efficient Zero-Shot Ranking with Large Language Models.CoRRabs/2310.09497 (2023). arXiv:2310.09497 https: //arxiv.org/abs/2310.09497

    Pith/arXiv arXiv 2023

  34. [2024]

    InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track

    RRADistill: Distilling LLMs’ Passage Ranking Ability for Long-Tail Queries Document Re-Ranking on a Search Engine. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track. 627–641

    2024

  35. [2025]

    arXiv:2504.07439 https://arxiv.org/abs/2504.07439

    LLM4Ranking: An Easy-to-Use Framework of Utilizing Large Language Models for Document Reranking.CoRRabs/2504.07439 (2025). arXiv:2504.07439 https://arxiv.org/abs/2504.07439

    Pith/arXiv arXiv 2025