{"id":"068a06d3-2f55-4676-bb61-5f492d022157","arxiv_id":"2509.09066","paper_version":1,"verdict":"REJECT","confidence":"LOW","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A manuscript claims instructional prompt engineering improves LLM-based cold-start recommendation, but provides no reproducible evidence.","lead":"The paper claims that structuring prompts with examples and user metadata improves few-shot LLM recommendations for cold-start users, citing gains of up to 18.7% in precision@5 and 21.3% in NDCG@10. It presents no code, data, or experimental detail, and the references are largely irrelevant, so the results cannot be verified.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground truth for cold-start test users is never defined; precision@5 and NDCG@10 cannot be computed without a relevance set, making the reported gains uninterpretable.","rationale":"The reader's weakest assumption correctly identifies the most load-bearing flaw: the evaluation protocol is incompletely or inconsistently specified. The paper's central claim is purely empirical — that prompt optimization improves Precision@5 and NDCG@10. For these metrics to be computable, each test user needs a known set of relevant items. The paper explicitly says test users have no interaction history, yet never explains the origin of the ground-truth relevance sets. This is an internal inconsistency, not merely a missing baseline comparison or a quality-control issue. If the ground truth is absent, the reported numbers are not just noisy; they are undefined. This single concern is sufficient to reject the paper's central claim. Other issues (irrelevant references, vague baselines, lack of error bars, unclear optimization procedure) also exist, but they are secondary once the evaluation metrics cannot be interpreted. My proposed check — asking for the split code or attempting a reconstruction — would definitively settle whether the concern lands. The verdict already being REJECT, I see no reason to change it.","tokens_in":7450,"tokens_out":4174,"duration_ms":44042,"concrete_test":"Request the authors' data preprocessing and split code. Verify whether test users have any held-out interaction records used as ground truth. If no ground-truth set exists, the metrics are undefined and the empirical claim fails. If a hold-out set exists, recompute Precision@5 and NDCG@10 on Amazon Reviews using exactly that ground-truth set; if the numbers differ by more than a small margin (e.g., 5%) from the paper's reported values, the results are not reproducible. If no code is available, attempt a standard cold-start split on MovieLens-1M (e.g., leave-one-out with no training interactions) and check whether the claimed improvements can be reproduced.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim (abstract, Section 4, Tables 2–3) is that optimized instructional prompts improve Precision@5 and NDCG@10 on cold-start users. However, Section 3.4 states that in all datasets, 'cold-start splits were formed retaining no interaction history of test users,' and Section 4 repeats that 'all test users had no prior interaction data within the training set.' Precision@5 and NDCG@10 are ranking metrics that require a ground-truth set of relevant items for each test user. The paper never specifies where this ground truth comes from. If test users have no interaction history at all, there is no observable evidence of their preferences, so the metrics are undefined. If some interactions are held out for evaluation, the text contradicts itself and the split protocol is missing. No data split code, exact prompts, or ground-truth construction details are provided. Without a defined relevance set, the reported improvements (e.g., +18.7% P@5 on Amazon Reviews) cannot be verified, and the central claim is unsupported by the described methodology.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes prompt optimization for frozen LLMs in few-shot cold-start recommendation. It introduces a context-conditioned prompt P(u, Ds) composed of an instructional header, a support set of exemplar users, and target-user metadata, and reports improvements in Precision@5, NDCG@10, and semantic coherence over baselines on Amazon Reviews, Last.fm, and MovieLens 1M using BioGPT, LLaMA-2, and GPT-4. The central claim is that optimal exemplar injection and instruction structuring can significantly improve ranking metrics without fine-tuning. The manuscript, however, leaves the optimization module unspecified, does not define the ground truth for cold-start users, and does not describe baselines or exact prompts; the evaluation therefore cannot be audited from the text as written.","tokens_in":7769,"tokens_out":4719,"duration_ms":50583,"significance":"The high-level question addressed here—whether prompt structure and few-shot exemplars can mitigate the cold-start problem for frozen LLM recommenders—is timely and of potential practical interest. The paper also explicitly acknowledges limitations of prompt instability and support-set selection (Section V), which is a plus. If the results were reproducible, they would provide a useful empirical data point for a lightweight, model-agnostic adaptation strategy. As presented, however, the contribution cannot be assessed: the proposed Prompt Optimization Module is never defined, the ground-truth construction is missing, and the reported gains rest on unnamed baselines and metric definitions that are internally inconsistent with the stated cold-start protocol.","major_comments":[{"comment":"The cold-start evaluation as described is not computable. Section 3.4 states that 'cold-start splits were formed retaining no interaction history of test users,' and Section 4 repeats that 'all test users had no prior interaction data within the training set.' Precision@5 and NDCG@10 require a ground-truth set of relevant items for each test user. No source for this ground truth is specified. If test users truly have no interaction history at all, the metrics are undefined; if a held-out portion of their history is used for evaluation, that protocol must be stated and the 'no interaction history' wording reconciled. This is load-bearing: every reported gain in Tables 1–3 depends on these metrics.","section":"Section 3.4 and Section 4"},{"comment":"The central method, the Prompt Optimization Module (POM), and 'token-level alignments and embedding space regularization' are never defined. The only formal content in Section 3.3 is a standard attention expression, and even that is written incorrectly as `(Q, K, V) = softmax(QK^T / sqrt(d_k))`, which is not a valid definition of Q, K, V. No algorithm, objective, or inference procedure for POM is given, and the claimed 'embedding space regularization' is never formalized. The abstract's statement that the pipeline 'uses token-level alignments and embedding space regularization' is therefore unverifiable.","section":"Abstract and Sections 3.2–3.3"},{"comment":"The comparisons are uninterpretable because the baseline is not identified. Tables 2 and 3 report gains over 'Best Baseline,' but the text never states which baseline this is (zero-shot LLM? collaborative filtering? a hybrid model?), what prompts it used, or how it was configured. Without this information, the headline improvements (e.g., +18.7% Precision@5 on Amazon Reviews) cannot be checked or attributed.","section":"Tables 2 and 3"},{"comment":"Prompt hyperparameters appear to be tuned on the same data used for the final evaluation. The paper says the exemplar count and prompt length were 'parameterically altered to find optimum combinations' (Section 3.4) and that '6 to 8 exemplars produced the best tradeoff' (Section 4), but there is no mention of a validation split. Consequently, the finding about optimal exemplar density is a selection artifact rather than empirical evidence. A separate validation set or nested evaluation is required before the 'optimal exemplar injection' claim can be supported.","section":"Section 3.4 and Section 4"},{"comment":"The manuscript contains garbled and contradictory formal notation. Section 3.1 writes `P(u,Ds) = I0Ds00(u)` and Section 2 contains `(?) = 2`; Section 3.3's attention formula is incomplete; Section 4 refers to 'our pings' where 'prompts' is intended. These are not merely typos—they prevent a reader from reconstructing the method. Additionally, Section 5's claims about 'immediate decoder pathways' and 'attention scales' are made without any formal analysis or ablation supporting them.","section":"Section 3.1 and throughout"}],"minor_comments":[{"comment":"The example prompt is illustrative only. To make the work reproducible, the exact prompt templates for each dataset and model should be provided in an appendix or public repository.","section":"Section 3.4"},{"comment":"The same comparisons appear with inconsistent numbers: Amazon Precision@5 gain is 18.7 in Table 2 but 18.8 in Table 3; MovieLens is 14.2 vs 14.3. These discrepancies should be reconciled.","section":"Tables 2 and 3"},{"comment":"Figures 1–3 are referenced in the text but are not present in the provided manuscript; ensure they are included and legible, with axis labels and captions.","section":"Figures"},{"comment":"Many cited references appear unrelated to the paper's topic (e.g., warehouse robotics, patent retrieval, AR overlays), while standard references for the datasets and baselines are missing. The reference list should be revised to support the claims made.","section":"References"},{"comment":"The manuscript has numerous typographical and grammatical errors, including 'widebounИround' (Section 1), 'timely composition' (Abstract), and 'movie lens' spacing. A thorough proofreading pass is needed.","section":"General"}],"recommendation":"reject","confidential_remarks":"The manuscript appears to be an early draft with severe formatting and internal-consistency problems. Beyond the missing methodological details, the undefined ground truth and unnamed baselines mean the central empirical claim cannot be reviewed. The references also contain many spurious entries, which raises concerns about the care taken in preparing the submission. I would not invite a revision unless the authors can supply the full experimental protocol and correct the internal contradictions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me cut to the chase: this paper is not in a reviewable state. The central empirical claim—that optimized instructional prompts improve Precision@5 and NDCG@10 for cold-start users—is unsupported because the evaluation is undefined. The text says test users have no interaction history in the training set, but those metrics require ground-truth relevant items for each user. Where does that ground truth come from? It never says. Without a relevance set, the metrics can't be computed. That's a load-bearing flaw, not a nit.\n\nWhat is actually new here? Very little. The recipe—instructional header plus a few exemplar profiles plus user metadata—is standard few-shot prompting. The 'Prompt Optimization Module' is a name, not a module; the abstract's talk of 'token-level alignments and embedding space regularization' is never defined in the methodology. So there is no new algorithm to evaluate.\n\nThat said, the direction is not crazy. Using frozen LLMs with a few-shot prompt to handle cold-start users is a plausible idea, and the paper's choice of datasets (Amazon, Last.fm, MovieLens) is reasonable. But the execution is too thin: no exact prompts, no baseline details beyond a single citation, no error bars, and the hyperparameter tuning (exemplar count 6–8, prompt length 1024) is done on the same datasets used for final numbers, which means those findings are fitting, not discovery.\n\nThe reference list is a red flag: most citations are unrelated, some are self-citations, and none seem to substantiate the methodological choices. The prose is also garbled in places, which makes me suspect the work was assembled hastily.\n\nThe stress-test note is on target. Without a defined ground truth for cold-start users, the reported +18.7% precision gain is just a number. The paper should be desk rejected. If the authors can provide the actual prompts, a defined relevance set (e.g., held-out interactions or an explicit surrogate), and code, a revised version might deserve a look—but as it stands, it doesn't earn referee time.\n\nWho would get value from this? Someone writing a survey on prompt engineering for recsys might skim the abstract, but nobody can reproduce or build on the results. I would not cite it, and I would not bring it to reading group except as a cautionary example of missing ground-truth definitions.","headline":"A prompt-engineering recipe whose evaluation is undefined makes the reported gains uninterpretable; the method is also not actually specified.","tokens_in":8149,"tokens_out":3337,"would_cite":false,"duration_ms":31017,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that optimizing instructional prompts—adding curated exemplar users and structured headers—can substantially improve precision@k and NDCG scores of frozen LLMs in cold-start recommendation without fine-tuning.","keywords":["Cold-Start Recommendation","Instructional Prompting","Few-Shot Learning","Large Language Models","Prompt Optimization","In-Context Learning","LLM-based Recommender Systems","Semantic Coherence"],"falsifier":"Inspect the evaluation protocol: if the ground-truth relevant items used to compute Precision@5 and NDCG@10 are taken from the test users' own held-out interactions, then those users are not actually cold-start, and the reported gains do not apply to the cold-start setting the paper claims to address.","tokens_in":7418,"feed_emoji":"🎯","tokens_out":5461,"duration_ms":50012,"temperature":0.7,"pith_summary":"The paper tries to establish that prompt engineering alone can mitigate the cold-start problem in LLM-based recommenders. It introduces a context-conditioned prompt formulation that injects an instructional header, a support set of exemplar users with item rankings, and target-user metadata into a frozen LLM. Across Amazon Reviews, Last.fm, and MovieLens 1M, and with BioGPT, LLaMA-2, and GPT-4, it reports consistent gains in Precision@5, NDCG@10, and semantic coherence over zero-shot and baseline models. If true, this would make prompt-based adaptation a scalable, fine-tuning-free route to personalization for new users.","feed_headline":"Prompt optimization lifts cold-start recommendations up to 21%","feed_subtitle":"Carefully structured prompts with a few examples beat fine-tuning for new users in Amazon, Last.fm, and MovieLens tests.","key_machinery":"The central object is the context-conditioned prompt formulation P(u, Ds) → R̂, where u is a cold-start user profile, Ds is a curated support set of exemplar users and their item rankings, and R̂ is the predicted ranked list. The prompt is built from an instructional header I, the support set Ds, and user metadata φ(u), and is fed to a frozen autoregressive LLM. The paper also invokes token-level alignments and embedding-space regularization as part of the optimization, though the detailed mechanism is only sketchily described; the empirical lever is the joint variation of prompt length l∈[256,2048] and exemplar density k∈[2,10].","core_discovery":"On the paper's own terms, the core discovery is that the composition of the prompt is not merely a syntactic wrapper but a functional control mechanism: how the instructional header, the exemplar set, and the metadata are arranged and worded steers the model's attention and decoder behavior, and thereby determines recommendation quality in low-data settings. The paper demonstrates this through a Prompt Optimization Module that varies prompt length and exemplar density, finding peak performance around 1024 tokens and 6-8 exemplars, with gains up to +18.7% precision@5 and +21.3% NDCG@10 on Amazon Reviews compared with zero-shot LLM baselines.","pith_inferences":["If the ground-truth gap is resolved, the method could be deployed at user onboarding, using only demographic or tag metadata to generate warm-start recommendations before any interaction history exists.","The paper's claim that prompt structure controls attention scales could be tested directly by inspecting attention maps, an analysis the paper does not provide.","The reliance on embedding-similarity-based exemplar selection suggests that more adaptive selection (e.g., active learning or reinforcement learning) could push gains further, but also implies performance may degrade in metadata-sparse domains.","The observed semantic coherence gains hint that prompt optimization might double as a lightweight domain-adaptation step before fine-tuning, potentially reducing the amount of labeled data needed for downstream training."],"forward_implications":["Prompt optimization can serve as a fine-tuning-free alternative for cold-start recommendation in content-rich domains like e-commerce and music.","The reported gains hold across multiple model families, suggesting the approach is model-agnostic and does not require instruction tuning to benefit.","Exemplar density around 6-8 and prompt length around 1024 tokens give the best tradeoff; beyond that, returns diminish, offering a practical operating point.","Instructional headers are especially important for less instruction-tuned models such as BioGPT, where removing them notably degrades performance.","The semantic coherence metric indicates the method captures thematic alignment, not just item relevance, which is valuable when behavioral data is absent."],"fun_headline_variants":["Prompt structure boosts LLM recommendations for new users","How prompt wording steers LLM recommendations in cold-start","Optimal prompts lift cold-start recs by 21%","LLM prompt design drives cold-start recommendation gains","Crafting prompts improves LLM cold-start performance"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the test users truly have no interaction history yet there is a well-defined ground-truth list of relevant items to score them against; the paper never states where that ground truth comes from.","fun_headline_variants_meta":{"raw":{"variants":["Prompt structure boosts LLM recommendations for new users","How prompt wording steers LLM recommendations in cold-start","Optimal prompts lift cold-start recs by 21%","LLM prompt design drives cold-start recommendation gains","Crafting prompts improves LLM cold-start performance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000208,"raw_usage":{"total_tokens":1234,"prompt_tokens":729,"completion_tokens":505,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":428}},"tokens_in":473,"tokens_out":505,"duration_ms":5229,"temperature":1.0,"reasoning_tokens":428,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T19:42:28.805024+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the evaluation protocol: if the ground-truth relevant items used to compute Precision@5 and NDCG@10 are taken from the test users' own held-out interactions, then those users are not actually cold-start, and the reported gains do not apply to the cold-start setting the paper claims to address.","supporting_citations":[],"review_version":1}