pith. sign in

arxiv: 2406.00247 · v2 · pith:VBIPYLD5new · submitted 2024-06-01 · 💻 cs.IR · cs.AI

Large Language Models for Relevance Judgment in Product Search

Navid Mehrdad , Hrushikesh Mohapatra , Mossaab Bagdouri , Prijith Chandran , Alessandro Magnani , Xunfan Cai , Ajit Puthenputhussery , Sachin Yadav
show 3 more authors
Tony Lee ChengXiang Zhai Ciya Liao
This is my paper
classification 💻 cs.IR cs.AI
keywords relevanceproductsearchjudgmentllmsmodelsattributeevaluators
0
0 comments X
read the original abstract

High relevance of retrieved and re-ranked items to the search query is the cornerstone of successful product search, yet measuring relevance of items to queries is one of the most challenging tasks in product information retrieval, and quality of product search is highly influenced by the precision and scale of available relevance-labelled data. In this paper, we present an array of techniques for leveraging Large Language Models (LLMs) for automating the relevance judgment of query-item pairs (QIPs) at scale. Using a unique dataset of multi-million QIPs, annotated by human evaluators, we test and optimize hyper parameters for finetuning billion-parameter LLMs with and without Low Rank Adaption (LoRA), as well as various modes of item attribute concatenation and prompting in LLM finetuning, and consider trade offs in item attribute inclusion for quality of relevance predictions. We demonstrate considerable improvement over baselines of prior generations of LLMs, as well as off-the-shelf models, towards relevance annotations on par with the human relevance evaluators. Our findings have immediate implications for the growing field of relevance judgment automation in product search.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. A Cascaded Generative Approach for e-Commerce Recommendations

    cs.AI 2026-05 unverdicted novelty 6.0

    A cascaded generative merchandising framework with placement theme generation, constrained keyword generation, and teacher-student fine-tuning achieves a 2.7% lift in cart adds per page view over a strong baseline in ...

  2. SHE: Stepwise Hybrid Examination Reinforcement Learning Framework for E-commerce Search Relevance

    cs.AI 2025-10 unverdicted novelty 5.0

    SHE is a new RL framework using stepwise hybrid examination rewards to improve reasoning quality and accuracy in large-scale e-commerce query-product relevance prediction.

  3. Joint Optimization of Relevance and Engagement in Multi-Task Ranking for E-Commerce with Efficient LLM Supervision

    cs.IR 2026-05 unverdicted novelty 4.0

    Production multi-task e-commerce ranking model uses LLM-generated three-level ordinal relevance labels and a unified value model to balance semantic quality against engagement signals.

  4. A Cascaded Generative Approach for e-Commerce Recommendations

    cs.AI 2026-05 unverdicted novelty 4.0

    A cascaded generative system for e-commerce recommendations using theme and keyword generation with teacher-student fine-tuning achieves a 2.7% lift in cart adds per page view.

  5. K-CARE: Knowledge-driven Symmetrical Contextual Anchoring and Analogical Prototype Reasoning for E-commerce Relevance

    cs.IR 2026-04 unverdicted novelty 4.0

    K-CARE uses behavior-derived anchoring and expert prototype analogies to ground LLMs and improve relevance on knowledge-intensive e-commerce cases.