REVIEW 1 cited by
Semantic Search Evaluation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Semantic Search Evaluation
read the original abstract
We propose a novel method for evaluating the performance of a content search system that measures the semantic match between a query and the results returned by the search system. We introduce a metric called "on-topic rate" to measure the percentage of results that are relevant to the query. To achieve this, we design a pipeline that defines a golden query set, retrieves the top K results for each query, and sends calls to GPT 3.5 with formulated prompts. Our semantic evaluation pipeline helps identify common failure patterns and goals against the metric for relevance improvements.
Forward citations
Cited by 1 Pith paper
-
Scaling Search Relevance: Augmenting App Store Ranking with LLM-Generated Judgments
Using a fine-tuned 3B LLM to generate millions of textual relevance labels for App Store search improves the ranker's behavioral/textual Pareto frontier and lifts conversion by 0.24%.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.