{"id":"43ab60e7-f0d1-4e78-8853-5d32c4223f76","arxiv_id":"2412.14454","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"No single LLM recommendation prompt is best across datasets; validation-based prompt selection with a ratio-based RPI indicator beats fixed prompts in most tested cases.","lead":"This paper compares 90 prompt designs for LLM-based recommendation systems on five shopping datasets and finds that no single prompt wins everywhere, so it selects prompts with a small validation set. A practical takeaway is that prompt choice should be tuned per dataset, and a cheap model can narrow the search before using a stronger model.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The RPI-selection result hinges on an unstated protocol detail: whether the Section 4.3 RPI analysis used only the 100 validation users. If it reused the Section 4.1 test users, Tables 3–5 leak test information and the reported gains are in-sample.","rationale":"The paper's main contribution is a practical prompt-selection recipe: use RPI on a small validation set to narrow to two candidates, then pick the better one on validation. This recipe is only valid if all RPI guidance comes from validation users. Section 4.2's RPI tables are computed on the same 100 users later used for test evaluation; Section 4.3 says a validation set was prepared but does not explicitly say that the RPI analysis for candidate construction was recomputed on it. Because the Table 5 RPI prompts differ from the Tables 3–4 best-RPI prompts, the likely intent is a recomputation, but the omission is exactly the kind of protocol detail that determines whether Tables 3–5 leak test information. Without code or a clear statement, the headline result is not independently verifiable. The secondary concern about a single 100-user split is also material: with no error bars, differences of 0.01–0.02 nDCG@10 between RPI and baselines could flip under resampling. Both concerns point to the same remedy: release the split definitions and reproduce the validation-only protocol. This does not refute the paper; it makes the current evidence insufficient for unconditional acceptance, which is consistent with the reader's CONDITIONAL verdict.","tokens_in":11129,"tokens_out":7864,"duration_ms":61418,"concrete_test":"Request the released code/logs or ask the authors to state, for each dataset, which user IDs were used for the RPI component analysis in Section 4.3. Then rerun the full selection pipeline using only the 100 validation users to construct the two RPI candidate prompts, evaluate on the held-out test users, and compare the selected prompts and nDCG@10 against Table 5. If the selected prompts differ or the RPI row changes by more than about 0.01, the reported result is contaminated. As a secondary check, repeat the selection with 10 different 100-user validation splits to confirm the 'highest in three of five' ranking is stable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 computes RPI tables on the 100 users from Section 4.1, who are later treated as the test set. Section 4.3 then introduces 100 different validation users and says 'we conducted an RPI analysis ... created two prompts with the best combinations' — but it never states explicitly that this RPI analysis was recomputed on the validation data only. The selected prompts in Table 5 differ from the best-RPI prompts in Tables 3–4 (e.g., Music: SR-30-TCD vs L-20-TCD / SL-30-TCD), which suggests it was rerun, but the text does not say so and no code is released. If the candidate prompts were instead selected using RPI on the test users, then the validation step only chooses between two prompts already known to be good on the test set, and the RPI row in Table 5 is an in-sample comparison. This would invalidate the central claim that a 100-user validation set plus RPI selects near-optimal prompts. Independently of that ambiguity, the evaluation rests on one 100-user split with no error bars or repeated splits; differences among top methods are often under 0.02 nDCG@10, so the 'highest in three of five' claim may not be stable. Both issues are resolvable by releasing the split definitions and rerunning validation-only selection.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether prompt design choices for LLM-based recommender systems can be selected systematically. It standardizes an inference prompt template, varies the user-information component along three dimensions (sampling format Random/Latest/Extract, sampling size k, item attributes T/C/D combinations, and summarization), and evaluates 90 prompts on 100-user test sets from five Amazon categories using gpt-4o-mini. The authors introduce the Relative Performance Indicator (RPI), an average accuracy-ratio summary over the prompt grid, to identify which component choices help. They then propose selecting prompts with a separate 100-user validation set, either by grid search (GS) or by constructing two RPI-guided candidate prompts and choosing between them (RPI). They report that RPI-guided selection ranks first in three datasets and second in one (Table 5), and that a gpt-4o-based cost-aware variant (GS*) is best or tied for best in four of five datasets (Table 6). The paper also reports API costs and discusses limitations around model generality and exploration cost.","tokens_in":11402,"tokens_out":5340,"duration_ms":40117,"significance":"If the findings are stable, the paper provides a useful empirical map of a largely unexplored design space and a low-cost validation procedure for prompt selection. The strengths are the systematic 90-prompt grid, the standardized template that isolates the user-information component, the explicit accounting of API calls and cost, and a clear statement of limitations (Section 5). The central claims, however, rest on two things the manuscript does not yet establish: a clean separation between the data used to derive the RPI-guided candidates and the data used to evaluate them, and evidence that the reported differences are not noise from a single 100-user split. Neither issue is a conceptual showstopper, and both can be fixed with additional reporting or experiments, but they are load-bearing for the headline 'validation set is enough' claim.","major_comments":[{"comment":"The RPI-guided selection protocol is not fully specified. Section 4.2 reports RPI tables and component-level conclusions computed on the 100 users selected in Section 4.1; Section 4.1 does not state that these users are a test set, but Table 5 later evaluates selected prompts on exactly these users. Section 4.3 introduces 100 different validation users and says 'we conducted an RPI analysis ... created two prompts with the best combinations,' but it never states explicitly that this second RPI analysis was recomputed on the validation users only. If the candidate prompts in Table 5 were selected using RPI rankings from the test users, then the validation step only chooses between two prompts already known to be good on test data, and the RPI row in Table 5 is an in-sample comparison. The same ambiguity affects the cost-reduction proposal in Section 5, which suggests excluding randomly sampled, k=5, C, CD, and Extract prompts on the basis of RPI patterns from Section 4.2. Please state, for each RPI table, the exact user set used, and ideally release the split definitions; the core claim that a 100-user validation set suffices depends on this.","section":"§4.2–4.3, Tables 3–5"},{"comment":"All reported nDCG@10 values are point estimates from a single 100-user split, with no variance, confidence intervals, significance tests, or repeated seeds for the random negative sampling described in Section 3.1. Differences between top methods are often small, e.g., Table 5 Grocery RPI and GS both 0.438, Table 5 Movie RPI 0.624 vs. L-10-T 0.646, and Table 6 Music RPI 0.723 vs. GS* 0.723. Without an estimate of split-to-split variability, the claims that RPI 'ranked first in three out of five cases' (Section 4.3) and that GS* 'outperformed previous research in four out of five datasets' (Section 4.4) are not yet supported. Please add repeated splits, bootstrap confidence intervals, or paired significance tests for the main comparisons.","section":"Tables 2–6, §4.3"},{"comment":"The RPI is defined as an average of accuracy ratios over the grid (Figure 3) and is reported as a single number. The text interprets positive values as indicating that a component is 'statistically superior' (Section 4.2), but no variance estimate or test is provided; moreover, the 72 or 18 grid entries are not independent because they share prompts, datasets, and users. This interpretation matters because the RPI-guided candidate construction in Section 4.3 relies on these point estimates to pick components (e.g., L-20-TCD, SR-30-TCD). Please either add uncertainty estimates or soften the statistical language to 'associated with higher average accuracy in this grid.'","section":"§4.2, Figure 3, Tables 3–4"},{"comment":"The GS* procedure is not fully described. It is said to select the prompt with highest accuracy on validation data using gpt-4o among the four previous prompts, GS, and RPI; however, the validation accuracy numbers and the exact candidate set are not reported, and no repeated selection is shown. Since Table 6 is the basis for the cost-efficiency claim, please report the validation-set accuracies and show that the gpt-4o selection is stable across splits or at least report the variability of the chosen prompt.","section":"§4.4, Table 6"}],"minor_comments":[{"comment":"'Kusano at al.' should be 'Kusano et al.'","section":"Page 2, author line"},{"comment":"Several table captions and headings contain formatting artifacts such as 'T able' and 'T able 1'; these should be corrected.","section":"Tables 1–6"},{"comment":"The process for generating the 18 negative items per user is described as random, but the seeds and the relationship between the negative sampling and the evaluation split are not reported; please document this for reproducibility.","section":"Section 4.1"},{"comment":"The abbreviations SR, SL, and SE are used in Table 4 before their component notation is fully explained in the text; a one-line definition in Section 3.2 would help.","section":"Section 3.2 and Table 4"},{"comment":"The phrase 'the latest two items' is used for positive examples, while Figure 1 shows ten viewed items; the relationship between the number of history items, the sampling size k, and the candidate list should be clarified.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about validation/test leakage is, in my reading, not resolved by the current text: Section 4.2 computes RPI on the users who later serve as the test set, and Section 4.3 never explicitly says that the RPI analysis for candidate construction was rerun on the validation users only. This is the central load-bearing issue and should be fixed before publication. In addition, the manuscript would benefit from releasing the split definitions and negative-sampling seeds; for a benchmark-style empirical claim, the absence of any variance or repetition is a serious gap. The paper is otherwise within the scope of a cs.IR venue and the contribution is incremental but useful."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The paper is a solid empirical map of prompt design choices for zero-shot LLM recommenders, and it earns its place as a reference for practitioners. The 90-prompt grid across five datasets with titles, categories, and descriptions is the most systematic comparison I've seen in this niche, and the observation that no single prompt dominates—and that category/description value is dataset-dependent—is credible and useful. The Relative Performance Indicator is a simple way to decompose component contributions, and the cost analysis for gpt-4o-mini vs gpt-4o is honest and practical.\n\nThe soft spots are real, and they matter in proportion to how much weight the paper puts on its selection method. The strongest concern is the Section 4.3 protocol. The paper introduces 100 validation users, then says 'we conducted an RPI analysis... created two prompts with the best combinations.' It never states that this RPI was recomputed on the validation users only. The selected prompts in Table 5 differ from the best-RPI prompts in Tables 3–4 (e.g., Music: SR-30-TCD vs L-20-TCD), which strongly suggests it was rerun on validation, but the text should say so. If it reused the test users, Tables 3–5 are in-sample and the central claim collapses. I think it's an omission, not a deliberate leak, but it's load-bearing and must be fixed.\n\nSecond, the evaluation rests on one 100-user split per dataset with no variance, no seeds for negative sampling, and no significance tests. In Table 5, the differences between GS and RPI are often under 0.02 nDCG@10 (Clothes: 0.011, Book: 0.012, Grocery: tie). With a single split, those rankings could flip. The 'highest in three of five' claim is not statistically grounded. The authors do not release code or exact prompt templates, which further limits reproducibility.\n\nOn the positive side, the paper flags its own limitations: Section 5 admits transfer to other LLMs is unverified. That is a minor limitation, honestly stated. The dataset statistics in Table 1 are used meaningfully to explain when categories/descriptions help.\n\nWho should read this? Anyone working on prompt selection or prompt engineering for LLM-based recommenders. It's a useful benchmark and a practical starting point, but I would not treat the RPI-based selection method as established until the protocol is clarified and error bars are added.\n\nRecommendation: this deserves a serious referee, not a desk reject. I would send it to review and ask for a revision that (1) states explicitly which data feeds the Section 4.3 RPI analysis, (2) releases the split definitions and seeds (or repeated splits), and (3) reports variance or confidence intervals. If those are addressed, it could become a citable reference. As is, I'd be hesitant to rely on its headline quantitative claims.","headline":"A useful large empirical grid for LLM prompt selection, but the selection method's advantage over simple grid search is not yet demonstrated because the validation protocol is ambiguous and there are no error bars.","tokens_in":11958,"tokens_out":3928,"would_cite":false,"duration_ms":28778,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that no single prompt design dominates for LLM-based recommendation and that selecting a prompt per dataset with a small validation set, guided by a relative-performance analysis, beats fixed baselines and full grid…","keywords":["prompt selection","large language models","recommendation systems","relative performance indicator","zero-shot ranking","validation data","nDCG@10","cost-efficient inference"],"falsifier":"Read the code or protocol to determine which 100 users produced the RPI rankings; if those rankings used the test users, the reported gains are in-sample rather than forward-looking. A second check is to rerun the selection on ten fresh 100-user validation splits: if the chosen prompt and the resulting test nDCG@10 change substantially across splits, the single-split protocol does not support the claimed advantage.","tokens_in":10868,"feed_emoji":"🎯","tokens_out":7224,"duration_ms":54356,"temperature":0.7,"pith_summary":"The paper asks whether longer or more elaborate prompts are always better in LLM-based recommendation systems, and answers no. Across 450 experiments with 90 prompt variants and five datasets, it finds that the best prompt changes with the dataset, with format, sampling size, and item attributes all interacting. It proposes selecting prompts with a small validation set: a relative-performance analysis narrows the field to two candidate prompts, and the better one on 100 validation users is used for test inference. This selection ranks first in nDCG@10 on three of five datasets and second on one, beating four previous prompt designs and even full grid search. A cost-aware variant, which searches with a cheap LLM and rescoring where needed, is highest or tied on four of five datasets.","feed_headline":"No single prompt wins for LLM recommenders; pick per dataset","feed_subtitle":"RPI-guided selection with 100 validation users tops three of five datasets and a cost-aware variant four of five.","key_machinery":"The load-bearing mechanism is the relative performance indicator (RPI). For each prompt component (sampling format, sample size, item attributes), the paper fixes the other components, computes the ratio of nDCG@10 when the component changes, and averages these ratios across formats; RPI is that average minus one, expressed as a percentage, with positive values meaning the component helps. RPI turns a 90-variant prompt grid into ranked component preferences and lets the authors construct two candidate prompts, one for sampling-based and one for summarization-based prompts. The second mechanism is validation-set prompt selection: the candidate that scores higher on 100 held-out users is chosen, and the paper's cost-aware variant uses a cheaper LLM for exploration and a stronger LLM to rescore candidates on validation before final inference.","core_discovery":"The central discovery is that prompt effectiveness in LLM recommendation is conditional, not intrinsic: no component consistently wins, and accuracy differences are large, with the minimum near random in some datasets. The paper's constructive result is an automatic prompt-selection procedure. It first builds a grid of 90 prompts by crossing item-attribute sets (title, category, description and their combinations) with sampling schemes (Random, Latest, Extract), sample sizes $k = 5, 10, 20, 30$, and summarization. On validation data it computes a relative performance indicator (RPI) for each component, constructing one prompt from the best sampling-side combination and one from the best summarization-side combination; the better of the two on validation is then used for test inference. In the paper's experiments this RPI-guided pick outperformed the four prior prompt designs in most datasets and, notably, outperformed selecting the single best prompt on validation (grid search), which the paper attributes to overfitting the validation set. It also shows that a stronger LLM can be reserved for final inference while a cheaper LLM does the search, because rescoring the few candidate prompts on validation with the stronger model yields the best or tied-best accuracy in four of five datasets.","pith_inferences":["The RPI factorization is not specific to recommendation: any prompt family with independent components (format, size, content) could be ranked the same way, so the method may transfer to other LLM tasks.","The paper does not test repeated validation splits; a natural next experiment is to resample the 100-user validation set many times and check whether the selected prompt and test nDCG@10 are stable, since large variance would undermine the single-split protocol.","The dataset statistics reported in the paper (title length, category informativeness, description availability) correlate with which components help, suggesting a future 'prompt predictor' that skips validation entirely by mapping dataset statistics to prompt components.","The pruning suggestion in the discussion implies an even cheaper protocol: run 36 prompts instead of 90 on the cheap LLM, then rescore survivors with the strong LLM; this is not evaluated in the paper."],"forward_implications":["A practitioner can select a competitive prompt for a new dataset by running about 100 validation users instead of testing all 90 prompts on the full test set.","Title-only prompts, the focus of much earlier work, are not universally best; adding categories or descriptions improves some datasets, while categories alone consistently hurt.","Full grid search on validation data can overfit; a component-based aggregation such as RPI can beat it on test data.","Exploration cost can be cut by doing the search with a cheaper LLM and the final inference with a stronger one; the paper's cost-aware strategy is best or tied in four of five datasets.","Because no prompt transfers across datasets or LLMs, prompt choice should be treated as a per-dataset, per-model decision rather than a fixed template."],"supporting_citations":[{"why":"Supplies the base task-prompt structure and the R-10-T comparison baseline.","marker":"[18]"},{"why":"Supplies the summarization-prompt family and the SR-100-T baseline.","marker":"[29]"},{"why":"Supplies the E-30-T aggregated-demonstration baseline and the writing style for the task portion.","marker":"[30]"},{"why":"Supplies Latest-k sampling and the L-10-T baseline.","marker":"[10]"},{"why":"Provides the sentence embeddings used by Extract-k sampling.","marker":"[8]"},{"why":"Provides the five domain datasets and the title, category, and description item metadata.","marker":"[19]"},{"why":"Supplies the role-setting phrase and the Random-k sampling reference.","marker":"[3]"},{"why":"Motivates the Extract-k sampling idea by analogy to in-context example selection.","marker":"[17]"}],"fun_headline_variants":["LLM recommenders: no universal prompt, selection is key","Prompt pick beats grid search in LLM recommenders","Smarter prompt selection for LLM recommender systems","RPI-guided prompt selection trumps grid search for LLM Recs","For LLM recommenders, prompt choice depends on dataset"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains depend on a protocol detail the paper never states clearly: the relative-performance analysis that picks the two candidate prompts must be run on the 100 validation users, not on the 100 test users, and one 100-user split must be enough to rank 90 prompt designs reliably.","fun_headline_variants_meta":{"raw":{"variants":["LLM recommenders: no universal prompt, selection is key","Prompt pick beats grid search in LLM recommenders","Smarter prompt selection for LLM recommender systems","RPI-guided prompt selection trumps grid search for LLM Recs","For LLM recommenders, prompt choice depends on dataset"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000492,"raw_usage":{"total_tokens":2467,"prompt_tokens":1046,"completion_tokens":1421,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":662,"completion_tokens_details":{"reasoning_tokens":1338}},"tokens_in":662,"tokens_out":1421,"duration_ms":8891,"temperature":1.0,"reasoning_tokens":1338,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:13:51.405036+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Read the code or protocol to determine which 100 users produced the RPI rankings; if those rankings used the test users, the reported gains are in-sample rather than forward-looking. A second check is to rerun the selection on ten fresh 100-user validation splits: if the chosen prompt and the resulting test nDCG@10 change substantially across splits, the single-split protocol does not support the claimed advantage.","supporting_citations":[{"cited_title":"In: NAACL-HLT","cited_arxiv_id":null,"evidence_quote":"Supplies the E-30-T aggregated-demonstration baseline and the writing style for the task portion."},{"cited_title":"In: ECIR (2)","cited_arxiv_id":null,"evidence_quote":"Supplies Latest-k sampling and the L-10-T baseline."},{"cited_title":"In: EMNLP (1)","cited_arxiv_id":null,"evidence_quote":"Provides the sentence embeddings used by Extract-k sampling."},{"cited_title":"In: EMNLP/IJCNLP (1)","cited_arxiv_id":null,"evidence_quote":"Provides the five domain datasets and the title, category, and description item metadata."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the role-setting phrase and the Random-k sampling reference."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the Extract-k sampling idea by analogy to in-context example selection."}],"review_version":1}