Pith. sign in

REVIEW 3 major objections 4 minor 107 references

Revisiting Prompt Engineering: A Comprehensive Evaluation for LLM-based Personalized Recommendation

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that prompt effectiveness in LLM-based personalized recommendation depends on the model tier: for cost-efficient LLMs, prompts that rephrase the task, step back to background principles, or structure reasoning reliably…

desk verdict A solid, practice-relevant benchmark study of prompt engineering for LLM recommendation, but the high-performance LLM 'simple is better' claim is underpowered and should be read as a cost argument, not an accuracy result. read the letter →

arxiv 2507.13525 v1 pith:3NMZJWXJ submitted 2025-07-17 cs.IR

Genki Kusano , Kosuke Akimoto , Kunihiro Takeoka This is my paper
classification cs.IR
keywords LLM-basedrecommendationpromptengineeringevaluationstatisticalanalysisreasoningmodelsinferencecostzero-shotrankingreproducibility
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that prompt engineering for LLM-based recommendation is not a one-size-fits-all toolbox: for cost-efficient LLMs, prompts that rephrase the task, step back to background principles, or structure reasoning through observation, thought, and action (Rephrase, Step-Back, ReAct) reliably improve ranking accuracy. For high-performance LLMs, the same complex prompts add little accuracy over a simple baseline while costing several times more, so the simple baseline is the most reasonable choice when accuracy is the priority. It also claims that NLP-style prompting techniques such as step-by-step reasoning, role-play, and emotional stimuli do not transfer well to recommendation, and that reasoning models or thinking mode can actually reduce accuracy via overthinking. If correct, the paper gives practitioners a concrete cost-accuracy rule for choosing prompts and models without assuming that more sophisticated prompts are better.

What carries the argument

The central objects are the three effective prompt designs: Rephrase (the LLM rewrites and expands the instruction before answering), Step-Back (consider the principles and theories behind the question), and ReAct (proceed through Observation, Thought, and Action steps). The analysis machinery is a linear mixed-effects model, which treats the prompt as a fixed effect and user, LLM, and metric as random effects, together with one-sided Wilcoxon signed-rank tests, to attribute accuracy differences to prompt type while controlling for other sources of variation.

What would settle it

Re-run the Section 5.2 comparison on the full 1,600-user sample using a paired significance test for claude-3.7-sonnet's Baseline versus Step-Back nDCG@3; if Step-Back shows a significant, reproducible gain across datasets, the claim that the simple baseline is the most reasonable accuracy choice for high-performance LLMs would need to be revised.

Watch

Extended reading notes

Core claim

Across 23 prompt types, 8 datasets, and 12 LLMs in a single-user zero-shot ranking setup, the paper finds that prompt effectiveness is tier-dependent. For cost-efficient LLMs like gpt-4.1-mini and llama3.3-70b, the prompts Rephrase, Step-Back, and ReAct significantly outperform the baseline under a linear mixed-effects model and Wilcoxon tests, while prompts popular in NLP such as Step-by-Step, RolePlay, and Emotion do not. For high-performance models, claude-3.7-sonnet with the simple Baseline reaches nDCG@3 of 0.621 and with Step-Back 0.628, a difference the paper calls minimal, so the simple configuration is judged the most reasonable. Reasoning models and thinking mode produced lower accuracy than the baseline in several cases and higher inference cost, and some complex prompts failed to follow the output format, which the error analysis identifies as a key cause of reduced accuracy.

Load-bearing premise

The conclusion that simple prompts match complex ones on high-performance models rests on a small evaluation of 160 users with no significance test on the small accuracy gap between the baseline and Step-Back prompts.

Editorial extensions

If this is right

  • For cost-efficient LLMs, adopting Rephrase, Step-Back, or ReAct is a low-risk accuracy improvement over a simple baseline across datasets and user types.
  • NLP-style prompts such as step-by-step reasoning, role-play, and emotional stimuli should not be assumed to transfer to ranking tasks, since several reduced accuracy in the recommendation setting.
  • With high-performance LLMs, the simple baseline prompt is the cost-effective default: complex prompts add little or no accuracy while multiplying inference spend.
  • Reasoning models and thinking modes are not a free accuracy boost for recommendation; they can lower accuracy due to overthinking and cost several times more.
  • Practitioners should test at least four prompts (Baseline, Rephrase, Step-Back, ReAct) and inspect inference logs for incomplete rankings, since format-following failures are a major source of degraded accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The tier-dependent pattern suggests prompt value is a function of model capability: weaker models need scaffolding to parse the task, while stronger models already internalize it, so prompt complexity beyond a baseline is redundant rather than helpful.
  • Format-adherence failures act as a hidden confound in any prompt comparison; evaluations that do not report partial-ranking or failure rates may attribute accuracy losses to the prompt technique when the true cause is instruction-following breakdown.
  • The concrete dollar figures and cost class cutoffs are time-sensitive: as API prices fall, the boundary between cost-efficient and high-performance tiers will shift, so the qualitative tier rule likely persists while the specific model recommendations may age.
  • A testable extension is whether the same tier-dependent ordering holds for other ranking and decision tasks, such as search result re-ranking or candidate selection in conversational systems, where output format constraints are similarly strict.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper reports a large-scale empirical comparison of prompt designs for single-user, zero-shot LLM-based re-ranking, covering 23 prompt types (21 evaluated in the main Section 4 comparison), 8 datasets, and 12 LLMs. The authors use Wilcoxon signed-rank tests and linear mixed-effects models to identify prompts that beat a baseline, plus cost accounting in USD. They conclude that for cost-efficient LLMs such as gpt-4.1-mini and llama3.3-70b, the Rephrase, Step-Back, and ReAct prompts are most effective, while for high-performance LLMs a simple baseline prompt is competitive at much lower cost; they also report that step-by-step reasoning and reasoning-model thinking modes often fail to improve or even reduce accuracy. Code and prompt templates are released.

Significance. The study is a useful, unusually broad empirical resource: it compares more prompts, datasets, and LLMs than most prior work, provides the exact prompt templates in the appendix, reports failure modes and token/cost statistics, and releases code for reproducibility. If the conclusions survive revision, the practical guidance for choosing among a small set of prompts on cost/accuracy grounds would be a genuinely useful contribution to LLM-based recommendation. However, the headline claim about high-performance LLMs is currently not supported by the paper's own numbers, and the statistical analysis has multiple-comparison and aggregation issues that need to be addressed before the central claims can be taken at face value.

major comments (3)
  1. [§5.2, Table 13; abstract; §1] The claim that for high-performance LLMs 'simple prompts often outperform more complex ones' and that 'if accuracy is the top priority, using the simple baseline prompt with claude-3.7-sonnet is more effective' is not supported by the reported point estimates. In Table 13, for claude-3.7-sonnet the nDCG@3 of Step-Back is 0.628 versus 0.621 for Baseline, and the Hit@3 values in Table 29 are 0.744 versus 0.738. For gpt-4.1, ReAct improves nDCG@3 from 0.526 to 0.561 and Hit@3 from 0.619 to 0.644. These comparisons are based on only 160 users and are not significance-tested. The paper should either provide significance tests for these differences and reconcile the gpt-4.1 result, or restrict the conclusion to a cost-efficiency statement for claude-3.7-sonnet rather than an accuracy-based generalisation to high-performance LLMs.
  2. [§4.1.1, Tables 7–8] The model-wise analysis performs a very large number of one-sided Wilcoxon signed-rank tests (21 prompts across multiple LLMs, datasets, and user-type subsets) and marks every p<0.05 result as significant without any multiple-comparison correction. With roughly a hundred or more tests in Table 7 and Table 8, several apparent 'significant' improvements are expected by chance alone. The robust-prompt claims in RQ1 and RQ2 should be based on adjusted p-values (e.g., FDR or Bonferroni within each LLM/column) or explicitly labeled as exploratory.
  3. [§4.2, Table 9; §4.4] The LMEM statement that ReAct, Rephrase, and Step-Back are effective 'across all datasets and LLMs' overstates what the model can show. The model has random intercepts for user, LLM, and metric but no prompt-by-LLM or prompt-by-dataset interaction terms, so the positive fixed effect is an average over LLMs. Table 8 shows that ReAct has relative improvement -13.8% for gpt-4o-mini and -5.3% for amazon-nova-lite, and Step-Back has -11.1% for amazon-nova-lite. The authors acknowledge the influence of gpt-4.1-mini and llama3.3-70b in the text, but the 'model-agnostic' wording in Section 4.4 and in the abstract-level generalization should be revised to 'effective on average, with strong LLM-specific variation.'
minor comments (4)
  1. [Abstract, §4, Figure 1] The paper alternately says 23 prompt types and 21 prompt types. The abstract says 23 prompts, while Section 4 and Figure 1 say 21 prompts including the baseline and 20 previous prompts. Please reconcile this count.
  2. [§1] The statement that gpt-4.1-mini or llama3.3-70b with an effective prompt reaches about 90% of claude-3.7-sonnet's accuracy at 'less than one fifth of the cost' is not consistent with Table 34/Table 35: gpt-4.1-mini with Rephrase costs $1.15 per 160 users versus $3.85 for claude-3.7-sonnet Baseline, which is about 30%, not below 20%. Please clarify the exact comparison used.
  3. [References] Reference [13] has a placeholder year '5555' and volume '01'; it should be corrected to the actual publication year and volume.
  4. [§3.4, §4.3] The treatment of unranked items by random completion is a reasonable choice, but since failure and partial-ranking rates are high for several prompt-LLM pairs (Table 10), the paper could state more explicitly how random completion interacts with the significance tests, especially for prompts like ReAct in gpt-4o-mini where partial outputs exceed 20%.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: an empirical benchmark of externally sourced prompts whose single self-citation is not load-bearing.

full rationale

The paper contains no derivation chain that reduces to its own inputs. Every evaluated prompt is adopted from external prior literature (e.g., Rephrase [12], Step-Back [63], ReAct [60], Step-by-Step [23]), and effectiveness is measured on fixed candidate sets across 8 public datasets and 12 LLMs, so the central claims that Rephrase/Step-Back/ReAct help cost-efficient models and that Baseline suffices for high-performance models are empirical findings from actual model outputs, not consequences of prompt definitions. The relative improvement r_p = (m_p - m_Baseline)/m_Baseline (Section 4.1.1) and the linear mixed-effects model (Section 4.2.1) are standard inferential tools applied to measured accuracy; no fitted parameter is renamed as a prediction, and the paper never claims held-out generalization of its selected prompts. Identifying effective prompts from the same data later recommended in Section 6 is ordinary post-hoc empirical inference, explicitly framed as a starting guideline, not a validated forecast. The only self-citation, [26] (Kusano, RecSys 2024), supports the design of Generate-Item, a prompt that was evaluated but is not among the recommended three, so it is not load-bearing. The paper itself flags its constraints, including the reduced 160-user evaluation in Section 5.1 (footnote 11) and the restriction to Baseline for reasoning models in Section 5.2; these limitations, plus the absence of significance tests on the Table 13 gaps (e.g., Step-Back 0.628 vs Baseline 0.621 for claude-3.7-sonnet, ReAct 0.561 vs 0.526 for gpt-4.1), weaken the high-performance-LLM generalization as a correctness and statistical-power matter, but they are not circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No theoretical derivation is attempted; the study's claims rest on experimental design choices. The key assumptions are that the single-user evaluation protocol is valid, that the most recent item is a reliable positive, that random negatives are appropriate, and that list prices approximate true inference cost. These assumptions are stated or implied in Sections 3.1 through 3.4.

assumptions (5)
  • domain assumption Single-user interaction history is sufficient context to infer preferences
    The paper restricts to a single-user setting (Section 1) and uses only the target user's history in prompts, so findings may not generalize to settings with other-user signals.
  • domain assumption The most recent interaction is a valid positive and random non-interacted items are valid negatives
    Section 3.1 constructs the ranking task from the most recent item as positive and 9 random negatives; if recency is a noisy preference signal, such as in news, the accuracy estimates shift.
  • domain assumption Random completion of incomplete rankings preserves relative prompt comparisons
    Section 3.4 randomly ranks unmentioned items when LLMs return partial rankings; this adds noise but is applied uniformly across prompts.
  • domain assumption Listed API token prices reflect actual inference cost
    Section 3.2 and Table 1 use public list prices as of March 13, 2025 to compute USD costs; actual costs can differ, especially for reasoning models with hidden reasoning tokens.
  • standard math Wilcoxon and LMEM assumptions hold for nDCG@3 observations
    Sections 4.1.1 and 4.2.1 apply paired Wilcoxon tests and a linear mixed-effects model; these require exchangeability and random-effects assumptions across users, LLMs, and metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Revisiting Prompt Engineering: A Comprehensive Evaluation for LLM-based Personalized Recommendation." pith.science (2026). https://pith.science/paper/3NMZJWXJ

@misc{pith2026250713525,
  author       = {Pith},
  title        = {Pith review of: Revisiting Prompt Engineering: A Comprehensive Evaluation for LLM-based Personalized Recommendation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3NMZJWXJ}},
  note         = {Machine review of arXiv:2507.13525}
}
read the original abstract

Large language models (LLMs) can perform recommendation tasks by taking prompts written in natural language as input. Compared to traditional methods such as collaborative filtering, LLM-based recommendation offers advantages in handling cold-start, cross-domain, and zero-shot scenarios, as well as supporting flexible input formats and generating explanations of user behavior. In this paper, we focus on a single-user setting, where no information from other users is used. This setting is practical for privacy-sensitive or data-limited applications. In such cases, prompt engineering becomes especially important for controlling the output generated by the LLM. We conduct a large-scale comparison of 23 prompt types across 8 public datasets and 12 LLMs. We use statistical tests and linear mixed-effects models to evaluate both accuracy and inference cost. Our results show that for cost-efficient LLMs, three types of prompts are especially effective: those that rephrase instructions, consider background knowledge, and make the reasoning process easier to follow. For high-performance LLMs, simple prompts often outperform more complex ones while reducing cost. In contrast, commonly used prompting styles in natural language processing, such as step-by-step reasoning, or the use of reasoning models often lead to lower accuracy. Based on these findings, we provide practical suggestions for selecting prompts and LLMs depending on the required balance between accuracy and cost.

Figures

Figures reproduced from arXiv: 2507.13525 by the authors.

Figure 1
Figure 1. Recommendation accuracy (nDCG@3) and infer￾ence cost (USD for processing 1,600 users from 8 datasets) for 21 prompts, including the baseline, evaluated with five LLMs in Section 4. Hollow and filled markers represent the baseline and 20 prompts from previous studies, respectively. A new approach that can address these limitations is the use of large language models (LLMs). When a recommendation task is described in … view at source ↗
Figure 2
Figure 2. Multi-turn conversation [𝑡sys, 𝑡𝑢1, 𝑡𝑎1, 𝑡𝑢2] used in the Rephrase prompt as input to the LLM, with output text 𝑡𝑎2. 3.4 Evaluation Metrics and Output Formatting In the numerical experiments, each prompt produced a ranking of 10 candidate items. We evaluated the ranking performance using two standard metrics: Hit@3 and nDCG@3 5 . In some cases, LLMs returned only a partial ranking, including just a few top-ranked it… view at source ↗
Figure 3
Figure 3. nDCG@3 and inference cost. For non-reasoning models, hollow and filled markers represent the Baseline and the highest-scoring prompt in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Baseline [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Rephrase [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: ReAct [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Step-Back 𝑡sys = (default_system_inst) -------------------------------------------------------------------------------------------------------------------------------- 𝑡𝑢1 = (user_info + candidate_info) ------------------------------------------------------------------…
Figure 8
Figure 8. Figure 8: Step-by-Step 𝑡sys = You are an AI assistant that pretends to be a person who has interacted with the following items. # Logs: (user_item_history) ---------------------------------------------------------------------------------------------------------------------------…
Figure 9
Figure 9. Figure 9: Pretend-User [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Summarize-Item [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Rephrase → ReAct [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: SelfRefine (Step-Back) [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Incorrect output generated by the ReAct prompt [PITH_FULL_IMAGE:figures/full_fig_p028_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

107 extracted references · 63 canonical work pages

  1. [1]

    Arkadeep Acharya, Brijraj Singh, and Naoyuki Onoe. 2023. LLM Based Gener- ation of Item-Description for Recommendation System. In RecSys. ACM, 1204– 1207

    2023

  2. [2]

    Keqin Bao, Jizhi Zhang, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He

  3. [3]

    Oren Barkan and Noam Koenigstein. 2016. Item2vec: Neural Item Embedding for Collaborative Filtering. In RecSys Posters (CEUR Workshop Proceedings, Vol. 1688) . CEUR-WS.org

    2016

  4. [4]

    Filippo Betello, Antonio Purificato, Federico Siciliano, Giovanni Trappolini, An- drea Bacciu, Nicola Tonellotto, and Fabrizio Silvestri. 2025. A Reproducible Analysis of Sequential Recommender Systems. IEEE Access 13 (2025), 5762–5772

    2025

  5. [5]

    Jiangjie Chen, Xintao Wang, Rui Xu, Siyu Yuan, Yikai Zhang, Wei Shi, Jian Xie, Shuang Li, Ruihan Yang, Tinghui Zhu, Aili Chen, Nianqi Li, Lida Chen, Caiyu Hu, Siye Wu, Scott Ren, Ziquan Fu, and Yanghua Xiao. 2024. From Persona to Personalization: A Survey on Role-Playing Language Agents. TMLR (2024), 2835–8856

    2024

  6. [6]

    Benjamin Clavié, Alexandru Ciceu, Frederick Naylor, Guillaume Soulié, and Thomas Brightwell. 2023. Large Language Models in the Workplace: A Case Study on Prompt Engineering for Job Type Classification. InNLDB (Lecture Notes in Computer Science, Vol. 13913). Springer, 3–17

    2023

  7. [7]

    Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. In RecSys. ACM, 191–198

    2016

  8. [8]

    Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. 2019. Are we really making much progress? A worrying analysis of recent neural recommen- dation approaches. In RecSys. ACM, 101–109

    2019

Show all 107 references
  1. [9]

    Sunhao Dai, Ninglu Shao, Haiyuan Zhao, Weijie Yu, Zihua Si, Chen Xu, Zhongxi- ang Sun, Xiao Zhang, and Jun Xu. 2023. Uncovering ChatGPT’s Capabilities in Recommender Systems. In RecSys. ACM, 1126–1132

    2023

  2. [10]

    Alexander Dallmann, Daniel Zoller, and Andreas Hotho. 2021. A Case Study on Sampling Strategies for Evaluating Neural Sequential Item Recommendation Models. In RecSys. ACM, 505–514

    2021

  3. [11]

    McAuley, Anton Korikov, Scott Sanner, Arnau Ramisa, René Vidal, Maheswaran Sathiamoorthy, Atoosa Kasirzadeh, and Silvia Milano

    Yashar Deldjoo, Zhankui He, Julian J. McAuley, Anton Korikov, Scott Sanner, Arnau Ramisa, René Vidal, Maheswaran Sathiamoorthy, Atoosa Kasirzadeh, and Silvia Milano. 2024. A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys). In KDD. ACM, 6448–6458

    2024

  4. [12]

    Yihe Deng, Weitong Zhang, Zixiang Chen, and Quanquan Gu. 2023. Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves. CoRR abs/2311.04205 (2023)

    2023 arXiv

  5. [13]

    Wenqi Fan, Zihuai Zhao, Jiatong Li, Yunqing Liu, Xiaowei Mei, Yiqi Wang, Jiliang Tang, and Qing Li. 5555. Recommender Systems in the Era of Large Language Models (LLMs). TKDE 01 (5555), 1–20

  6. [14]

    Yi Fang, Wenjie Wang, Yang Zhang, Fengbin Zhu, Qifan Wang, Fuli Feng, and Xiangnan He. 2025. Reason4Rec: Large Language Models for Recommendation with Deliberative User Preference Alignment. CoRR abs/2502.02061 (2025)

    2025 arXiv

  7. [15]

    Yunfan Gao, Tao Sheng, Youlin Xiang, Yun Xiong, Haofen Wang, and Jiawei Zhang. 2023. Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System. CoRR abs/2303.14524 (2023)

    2023 arXiv

  8. [16]

    Mihajlo Grbovic, Vladan Radosavljevic, Nemanja Djuric, Narayan Bhamidipati, Jaikit Savla, Varun Bhagwan, and Doug Sharp. 2015. E-commerce in Your Inbox: Product Recommendations at Scale. In KDD. ACM, 1809–1818

    2015

  9. [17]

    Danny Halawi, Jean-Stanislas Denain, and Jacob Steinhardt. 2024. Overthinking the Truth: Understanding how Language Models Process False Demonstrations. In ICLR. OpenReview.net

    2024

  10. [18]

    Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural Collaborative Filtering. In WWW. ACM, 173–182

    2017

  11. [19]

    Zhankui He, Zhouhang Xie, Rahul Jha, Harald Steck, Dawen Liang, Yesu Feng, Bodhisattwa Prasad Majumder, Nathan Kallus, and Julian J. McAuley. 2023. Large Language Models as Zero-Shot Conversational Recommenders. In CIKM. ACM, 720–730

    2023

  12. [20]

    McAuley, and Wayne Xin Zhao

    Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian J. McAuley, and Wayne Xin Zhao. 2024. Large Language Models are Zero-Shot Rankers for Recommender Systems. In ECIR (2) (Lecture Notes in Computer Science, Vol. 14609). Springer, 364–381

    2024

  13. [21]

    Jianchao Ji, Zelong Li, Shuyuan Xu, Wenyue Hua, Yingqiang Ge, Juntao Tan, and Yongfeng Zhang. 2024. GenRec: Large Language Model for Generative Recommendation. In ECIR (3) (Lecture Notes in Computer Science, Vol. 14610) . Springer, 494–502

    2024

  14. [22]

    Muhammad Murad Khan, Roliana Ibrahim, and Imran Ghani. 2017. Cross Domain Recommender Systems: A Systematic Literature Review. ACM Comput. Surv. 50, 3 (2017), 36:1–36:34

    2017

  15. [23]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large Language Models are Zero-Shot Reasoners. In NeurIPS

    2022

  16. [24]

    Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, Xin Zhou, Enzhi Wang, and Xiaohang Dong. 2024. Better Zero-Shot Reasoning with Role-Play Prompting. In NAACL-HLT. ACL, 4099–4113. Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Kusano et al

    2024

  17. [25]

    Bell, and Chris Volinsky

    Yehuda Koren, Robert M. Bell, and Chris Volinsky. 2009. Matrix Factorization Techniques for Recommender Systems. Computer 42, 8 (2009), 30–37

    2009

  18. [26]

    Genki Kusano. 2024. Data Augmentation using Reverse Prompt for Cost-Efficient Cold-Start Recommendation. In RecSys. ACM, 861–865

    2024

  19. [27]

    Cheng Li, Jindong Wang, Yixuan Zhang, Kaijie Zhu, Wenxin Hou, Jianxun Lian, Fang Luo, Qiang Yang, and Xing Xie. 2023. Large Language Models Understand and Can be Enhanced by Emotional Stimuli. CoRR abs/2307.11760 (2023)

    2023 arXiv

  20. [28]

    Lei Li, Yongfeng Zhang, and Li Chen. 2023. Prompt Distillation for Efficient LLM-based Recommendation. In CIKM. ACM, 1348–1357

    2023

  21. [29]

    Lei Li, Yongfeng Zhang, Dugang Liu, and Li Chen. 2024. Large Language Mod- els for Generative Recommendation: A Survey and Visionary Discussions. In LREC/COLING. ELRA and ICCL, 10146–10159

    2024

  22. [30]

    Jianghao Lin, Xinyi Dai, Yunjia Xi, Weiwen Liu, Bo Chen, Xiangyang Li, Chenxu Zhu, Huifeng Guo, Yong Yu, Ruiming Tang, and Weinan Zhang. 2024. How Can Recommender Systems Benefit from Large Language Models: A Survey. ACM Trans. Inf. Syst. (2024). Just Accepted

    2024

  23. [31]

    Junling Liu, Chao Liu, Renjie Lv, Kang Zhou, and Yan Zhang. 2023. Is ChatGPT a Good Recommender? A Preliminary Study. CoRR abs/2304.10149 (2023)

    2023 arXiv

  24. [32]

    Jiahao Liu, Xueshuo Yan, Dongsheng Li, Guangping Zhang, Hansu Gu, Peng Zhang, Tun Lu, Li Shang, and Ning Gu. 2025. Improving LLM-powered Recom- mendations with Personalized Information. CoRR abs/2502.13845 (2025)

    2025 arXiv

  25. [33]

    Yucong Luo, Mingyue Cheng, Hao Zhang, Junyu Lu, and Enhong Chen. 2024. Unlocking the Potential of Large Language Models for Explainable Recommen- dations. In DASFAA (5) (Lecture Notes in Computer Science, Vol. 14854) . Springer, 286–303

    2024

  26. [34]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-Refine: It...

    2023

  27. [35]

    Bodhisattwa Prasad Majumder, Shuyang Li, Jianmo Ni, and Julian J. McAuley

  28. [36]

    Tong Man, Huawei Shen, Xiaolong Jin, and Xueqi Cheng. 2017. Cross-Domain Recommendation: An Embedding and Mapping Approach. In IJCAI. ijcai.org, 2464–2470

    2017

  29. [37]

    Raja Sekhar Reddy Mekala, Yasaman Razeghi, and Sameer Singh. 2024. EchoPrompt: Instructing the Model to Rephrase Queries for Improved In-context Learning. In NAACL (Short Papers). ACL, 399–432

    2024

  30. [38]

    Aleksandr Milogradskii, Oleg Lashinin, Alexander P, Marina Ananyeva, and Sergey Kolesnikov. 2024. Revisiting BPR: A Replicability Study of a Common Recommender System Baseline. In RecSys. ACM, 267–277

    2024

  31. [39]

    Jianmo Ni, Jiacheng Li, and Julian J. McAuley. 2019. Justifying Recommendations using Distantly-Labeled Reviews and Fine-Grained Aspects. In EMNLP/IJCNLP (1). ACL, 188–197

    2019

  32. [40]

    Petrov and Craig Macdonald

    Aleksandr V. Petrov and Craig Macdonald. 2022. A Systematic Review and Replicability Study of BERT4Rec for Sequential Recommendation. In RecSys. ACM, 436–447

    2022

  33. [41]

    Anderson

    Steffen Rendle, Walid Krichene, Li Zhang, and John R. Anderson. 2020. Neural Collaborative Filtering vs. Matrix Factorization Revisited. In RecSys. ACM, 240– 248

    2020

  34. [42]

    Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. 2024. A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications. CoRR abs/2402.07927 (2024)

    2024 arXiv

  35. [43]

    Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, and Xia Ben Hu

  36. [44]

    Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang

  37. [45]

    Zhu Sun, Di Yu, Hui Fang, Jie Yang, Xinghua Qu, Jie Zhang, and Cong Geng

  38. [46]

    Yu-Min Tseng, Yu-Chao Huang, Teng-Yun Hsiao, Wei-Lin Chen, Chao-Wei Huang, Yu Meng, and Yun-Nung Chen. 2024. Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization. In EMNLP (Findings). ACL, 16612– 16631

    2024

  39. [47]

    Shubham Vatsal and Harsh Dubey. 2024. A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Tasks.CoRR abs/2407.12994 (2024)

    2024 arXiv

  40. [48]

    BERT4Rec: Sequential Recommendation with Bidirectional Encoder Repre- sentations from Transformer. In CIKM. ACM, 1441–1450

  41. [49]

    Lei Wang and Ee-Peng Lim. 2024. The Whole is Better than the Sum: Using Aggre- gated Demonstrations in In-Context Learning for Sequential Recommendation. In NAACL-HLT. ACL, 876–895

    2024

  42. [50]

    Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023. Plan-and-Solve Prompting: Improving Zero-Shot Chain- of-Thought Reasoning by Large Language Models. In ACL (1). ACL, 2609–2634

    2023

  43. [51]

    Noah Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, Man Zhang, Zhaoxiang Zhang, Wanli Ouyang, Ke Xu, Wenhao Huang, Jie Fu, and Junran Peng. 2024. RoleLLM: Benchmarking, Eliciting, and Enhancing Role-P...

    2024

  44. [52]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In ICLR. OpenReview.net

    2023

  45. [53]

    Lei Wang and Ee-Peng Lim. 2023. Zero-Shot Next-Item Recommendation using Large Pretrained Language Models. CoRR abs/2304.03153 (2023)

    2023 arXiv

  46. [54]

    Yinwei Wei, Xiang Wang, Qi Li, Liqiang Nie, Yan Li, Xuanping Li, and Tat-Seng Chua. 2021. Contrastive Learning for Cold-Start Recommendation. In ACM Multimedia. ACM, 5382–5390

    2021

  47. [55]

    Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, and Ming Zhou. 2020. MIND: A Large-scale Dataset for News Recommendation. In ACL. ACL, 3597–3606

    2020

  48. [56]

    Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, Hui Xiong, and Enhong Chen

  49. [57]

    Lanling Xu, Junjie Zhang, Bingqian Li, Jinpeng Wang, Sheng Chen, Wayne Xin Zhao, and Ji-Rong Wen. 2025. Tapping the Potential of Large Language Models as Recommender Systems: A Comprehensive Framework and Empirical Analysis. ACM Transactions on Knowledge Discovery from Data (2...

    2025

  50. [58]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In NeurIPS

    2022

  51. [59]

    Le, Denny Zhou, and Xinyun Chen

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. 2024. Large Language Models as Optimizers. In ICLR. OpenReview.net

    2024

  52. [60]

    Narasimhan, and Yuan Cao

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In ICLR. OpenReview.net

    2023

  53. [61]

    Zhenrui Yue, Sara Rabhi, Gabriel de Souza Pereira Moreira, Dong Wang, and Even Oldridge. 2023. LlamaRec: Two-Stage Recommendation using Large Language Models for Ranking. CoRR abs/2311.02089 (2023)

    2023 arXiv

  54. [62]

    Tianzi Zang, Yanmin Zhu, Haobing Liu, Ruohan Zhang, and Jiadi Yu. 2023. A Survey on Cross-domain Recommendation: Taxonomies, Methods, and Future Directions. ACM Trans. Inf. Syst. 41, 2 (2023), 42:1–42:39

    2023

  55. [63]

    Chi, Quoc V

    Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2024. Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models. In ICLR. OpenReview.net

    2024

  56. [64]

    Xiaohan Xu, Chongyang Tao, Tao Shen, Can Xu, Hongbo Xu, Guodong Long, Jian-Guang Lou, and Shuai Ma. 2024. Re-Reading Improves Reasoning in Large Language Models. In EMNLP. ACL, 15549–15575

    2024

  57. [65]

    Joyce Zhou, Yijia Dai, and Thorsten Joachims. 2024. Language-Based User Profiles for Recommendation. CoRR abs/2402.15623 (2024)

    2024 arXiv

  58. [66]

    Zhihui Zhou, Lilin Zhang, and Ning Yang. 2023. Contrastive Collaborative Filtering for Cold-Start Item Recommendation. In WWW. ACM, 928–937

    2023

  59. [67]

    (omitted)

    Ziwei Zhu, Shahin Sefati, Parsa Saadatpanah, and James Caverlee. 2020. Rec- ommendation for New Users and New Items via Randomized Training and Mixture-of-Experts Transformation. In SIGIR. ACM, 1121–1130. Revisiting Prompt Engineering for Personalized Recommendation Conference...

    2020

  60. [70]

    Zhi Zheng, Wenshuo Chao, Zhaopeng Qiu, Hengshu Zhu, and Hui Xiong. 2024. Harnessing Large Language Models for Text-Rich Sequential Recommendation. In WWW. ACM, 3207–3216

    2024

  61. [74]

    The presence of action elements, particularly in the context of the Bourne series

  62. [75]

    High ratings and positive reviews, as the user has previously favored products with a perfect rating of 5.0

  63. [76]

    After analyzing the candidate products, I will provide a ranked list indicating the order of preference for the user

    The relevance of the genre to the user 's interests, with a focus on action and comedy. After analyzing the candidate products, I will provide a ranked list indicating the order of preference for the user. Now, let 's proceed to rank the candidate products based on these crite...

  64. [77]

    Seven Year Itch VHS (Comedy) (omitted)

  65. [78]

    Now, let 's rank the products

    Love Streams (Drama) Based on the analysis of these products, the final ranking will be provided in the specified format. Now, let 's rank the products. ### Ranked List:

  66. [79]

    (omitted)

    **The Bourne Identity** (Action) - Strong alignment with the user 's interest in the Bourne series. (omitted)

  67. [80]

    Tyler Perry 's A Madea Christmas

    **Cooking French, Vol. 2** (Special Interests) - Least relevant to the user 's interests. ### Final Output: [4, 9, 8, 6, 1, 3, 2, 7, 10, 5] -------------------------------------------------------------------------------------------------------------------------------- 𝑡𝑢2 = (f...

    2018

  68. [81]

    (omitted)

    **Seven Year Itch VHS**: Comedy, but lacks a strong description or notable acclaim. (omitted)

  69. [82]

    The Bourne Identity,

    **Love Streams**: Drama, not aligned with the user 's preferences. ### Observation: - **The Bourne Identity**: Strongly matches user preferences as it is an action film from a series they enjoy. - **Underworld: Awakening**: Action-oriented, but may not be as appealing as the B...

    2018

  70. [84]

    **Ratings**: Products with higher ratings (5.0 in this case) are more favorable

  71. [85]

    **Familiarity with Series**: The user has already shown interest in the Bourne series, so products related to that series should be prioritized

  72. [86]

    Now, let 's analyze the candidate products:

    **Overall Appeal**: Products that are critically acclaimed or have a strong following may also be ranked higher. Now, let 's analyze the candidate products:

  73. [87]

    (omitted)

    **The Bourne Identity** (4) - This is directly related to the Bourne series and fits the user 's established interest in action and espionage. (omitted)

  74. [88]

    **Love Streams** (10) - A drama that does not fit the user 's established preferences for action and thrillers. Based on this analysis, the ranked list of candidate products is: [4, 8, 9, 2, 3, 6, 7, 1, 5, 10] -------------------------------------------------------------------...

  75. [89]

    **Genre**: The user seems to prefer action and thriller movies, particularly those related to the Bourne series

  76. [90]

    **Rating**: Higher-rated products are more likely to be recommended

  77. [91]

    Now, let 's analyze the candidate products:

    **Description**: Products with engaging descriptions that align with the user 's interests will be prioritized. Now, let 's analyze the candidate products:

  78. [92]

    It has a compelling description and fits the action/thriller genre

    **The Bourne Identity** (Product 4): This is directly related to the Bourne series, which the user has shown a strong preference for. It has a compelling description and fits the action/thriller genre. (omitted)

  79. [93]

    Bourne Ultimatum

    **Love Streams** (Product 10): This is a drama that may not resonate with the user 's action/thriller interests. Based on this analysis, the ranked list of candidate products is: [4, 8, 9, 2, 3, 6, 1, 7, 5, 10] ------------------------------------------------------------------...

    2018

  80. [94]

    (omitted)

    **Bourne Ultimatum** - Action-packed with a perfect rating of 5.0. (omitted)

  81. [95]

    # Candidate Products: You will evaluate the following 10 candidate products to determine which ones best align with the user 's preferences for high-rated action and comedy films:

    **The Bourne Legacy** - A thrilling addition to the Bourne franchise, rated 5.0. # Candidate Products: You will evaluate the following 10 candidate products to determine which ones best align with the user 's preferences for high-rated action and comedy films:

  82. [96]

    (omitted)

    **Seven Year Itch VHS** - Comedy genre. (omitted)

  83. [97]

    **Love Streams** - A drama film. # Objective: Your task is to rank these 10 candidate products based on how closely they align with the user 's preferences for action and comedy films, particularly those that are highly rated. The ranking should be presented in the format of i...

  84. [98]

    (omitted)

    **The Bourne Identity** - Strong match for action preference. (omitted)

  85. [99]

    ### Answer: Based on the evaluations, the ranked list of candidate products is: **[4, 9, 8, 3, 2]**

    **Chicago** - Critically acclaimed but not action or comedy. ### Answer: Based on the evaluations, the ranked list of candidate products is: **[4, 9, 8, 3, 2]**. -------------------------------------------------------------------------------------------------------------------...

    2018

  86. [100]

    **Genre and Themes**: The user has shown a preference for action and thriller movies, particularly those in the Bourne series, which are characterized by intense action and espionage themes

  87. [101]

    **Ratings**: Higher-rated products (5.0 ratings) are prioritized

  88. [102]

    Given these criteria, let 's analyze the candidate products:

    **Relevance**: Products that are part of the same franchise or have similar themes to the Bourne series will be ranked higher. Given these criteria, let 's analyze the candidate products:

  89. [103]

    It has a similar action and espionage theme

    **The Bourne Identity** (Product 4): This is directly related to the Bourne series, which the user has shown a strong preference for. It has a similar action and espionage theme. (omitted)

  90. [104]

    **Love Streams** (Product 10): This is a drama and does not fit the user 's action-oriented preferences. Based on this analysis, the ranked list of candidate products is as follows: [4, 8, 9, 2, 3, 6, 7, 1, 5, 10] ---------------------------------------------------------------...

  91. [105]

    For instance, if the user enjoys a mix of genres, including comedy or drama, those products could be ranked higher

    **User Preference Analysis**: While the analysis focused on action and thriller genres, it could benefit from a deeper understanding of the user's overall preferences. For instance, if the user enjoys a mix of genres, including comedy or drama, those products could be ranked h...

  92. [106]

    This would allow for continuous learning and improvement based on user responses

    **Feedback Loop**: Implementing a feedback mechanism where users can indicate their satisfaction with recommendations could help refine future predictions. This would allow for continuous learning and improvement based on user responses. By incorporating these considerations, ...

    2018

  93. [107]

    (omitted)

    **Emmet Otter 's Jug-Band Christmas** - Not related to fitness. (omitted)

  94. [108]

    ### Observation: Among the candidate products, only **Tracy Anderson: Precision Toning** is related to fitness

    **Royal Paintbox** - Not related to fitness. ### Observation: Among the candidate products, only **Tracy Anderson: Precision Toning** is related to fitness. However, based on the user 's previous experiences with Tracy Anderson 's workouts, they found them too easy. The other ...

    2018

  95. [2019]

    In EMNLP/IJCNLP (1)

    Generating Personalized Recipes from Historical User Preferences. In EMNLP/IJCNLP (1). ACL, 5975–5981

  96. [2020]

    In RecSys

    Are We Evaluating Rigorously? Benchmarking Recommendation for Re- producible Evaluation and Fair Comparison. In RecSys. ACM, 23–32

  97. [2023]

    In RecSys

    TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation. In RecSys. ACM, 1007–1014

  98. [2024]

    World Wide Web (WWW) 27, 5 (2024), 60

    A survey on large language models for recommendation. World Wide Web (WWW) 27, 5 (2024), 60

    2024

  99. [2025]

    CoRR abs/2503.16419 (2025)

    Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models. CoRR abs/2503.16419 (2025)

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.