REVIEW 3 major objections 5 minor 35 references
Conversational recommenders work better when they switch from attribute questions early to item choices later.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-10 21:45 UTC pith:5QFIQZIO
load-bearing objection Stage-aware elicitation (attributes early, items later) is a real, usable design rule for CRS, backed by a fresh user study, the InPE annotations, and a competent MoE baseline that wins offline; annotation agreement and single-domain offline eval are the soft spots, not the core claim. the 3 major comments →
When and How to Ask: Dynamic Preference Elicitation Strategies for Conversational Recommendation
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Optimal preference elicitation is stage-dependent and context-sensitive: attribute-based questions dominate early dialogue stages when preferences are abstract, while item-based strategies become preferred as preferences grow concrete; hybrid strategies stay useful throughout. Explicitly modeling this timing and choice with a routed mixture of experts improves both recommendation recall and pairwise response preference over strong baselines.
What carries the argument
COPE (Conversational Preference Elicitation via Mixture of Experts): a hierarchical router that first chooses among elicitation, recommendation, or general chat, then (if eliciting) chooses attribute, item, or hybrid strategy, each executed by a lightweight expert adapter on a shared frozen language-model backbone.
Load-bearing premise
That majority-vote crowd labels of preferred strategy and response quality on movie dialogues are a reliable enough signal of real multi-turn user utility.
What would settle it
A live multi-turn user study in which systems forced to follow the stage-dependent strategy schedule (attribute early, item later) fail to outperform fixed-strategy or random-routing controls on actual recommendation success or user satisfaction.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that preference elicitation in conversational recommender systems (CRSs) is stage-dependent: attribute-based strategies work best early when preferences are abstract, while item-based (and hybrid) strategies become preferable as preferences refine. Supporting evidence comes from a preliminary crowdsourced user study (Figure 1) and from InPE, a re-annotation of the INSPIRED movie dialogues that labels elicitation necessity, strategy type (attribute/item/hybrid), and response preference. The authors then introduce COPE, a hierarchical Mixture-of-Experts architecture on a frozen LLM backbone with LoRA experts and a two-stage router (task then strategy). Offline experiments on InPE report gains in Recall@k, task/strategy accuracy, and pairwise response win-rate over neighborhood, KG, and LLM baselines, plus mechanistic analysis of router activations that recovers the same stage-wise pattern.
Significance. If the stage-dependent pattern and the value of explicit strategy modeling hold, the work supplies both a useful empirical regularity for CRS design and a concrete modeling recipe (structured action space + hard-routed MoE). The public InPE annotations and the released code are genuine community resources that enable strategy-aware training and evaluation, which prior CRS datasets largely lack. The offline gains of COPE over strong LLM baselines (Tables 6–7) and the router transition analysis (Figure 5) are concrete, reproducible contributions even if live-user transfer remains open.
major comments (3)
- Table 4 reports Krippendorff’s α = 0.244 for the binary elicitation-necessity decision (Q1) and only moderate α = 0.531 for strategy type (Q2). Because these majority-vote labels supervise both the hierarchical router (Eq. 7, Algorithm 1) and the pairwise preference judgments used for win-rate (Tables 3 and 7), the central claim that attribute-based strategies are optimal early and item-based later rests on a noisy ground-truth signal. Majority voting and expert adjudication mitigate but do not remove the risk that reported stage-wise patterns and COPE gains largely reflect annotator consensus rather than multi-turn user utility. A reliability analysis (e.g., sensitivity to low-agreement turns, or a second independent annotation round) is needed before the labels can be treated as reliable supervision.
- All quantitative claims (Tables 6–9, Figure 5) are offline and confined to the single-domain INSPIRED movie corpus. The paper never demonstrates that a model trained on InPE labels improves live recommendation success, conversation length, or user satisfaction in an interactive setting, nor that the stage-dependent pattern transfers outside movies. Given that the weakest modeling assumption is precisely the transfer of these offline labels to real multi-turn utility, at least a small-scale online or simulated interactive evaluation (or an explicit discussion of this limitation with a concrete transfer experiment) is required to support the claim that “context-aware preference elicitation strategies are beneficial for conversational recommendation.”
- Table 8 shows that the “w/o all” (single-adapter) variant obtains higher Task Accuracy, Strategy Accuracy, and Recall@10 than the full multi-expert COPE, while under-performing only on pairwise win-rate. This raises a load-bearing question about whether the hierarchical MoE specialization itself is necessary, or whether the performance edge of the full model is largely attributable to the strategy labels rather than to expert separation. A clearer analysis of when specialization helps versus when a unified adapter suffices would strengthen the architectural claim.
minor comments (5)
- Minor numerical inconsistencies appear between text and tables (e.g., 60.02 % vs. 60.07 % for elicitation-yes turns; hybrid proportions 46.97 % vs. 47.03 %). Please reconcile.
- Figure 3 and Figure 5 would benefit from absolute counts or confidence intervals alongside the normalized proportions, so readers can judge sample size per turn.
- The GitHub URL uses the handle “juanfacabian” while the author list is Xia/Zhang/Wang; a brief note on repository ownership would avoid confusion.
- Hyper-parameter choices (LoRA rank, loss weights in Table 5, InfoNCE temperature) are reported but not ablated; a short sensitivity paragraph would help reproducibility.
- In §5.1 the action triplet ⟨a_t, σ_t, ϕ_t⟩ is introduced, yet ϕ_t (recommendation objective) receives little subsequent modeling detail compared with a_t and σ_t.
Circularity Check
Mild circularity: router predictions recover annotation stage patterns by construction, and SFT on preferred responses makes pairwise likelihood win-rate largely forced.
specific steps
-
fitted input called prediction
[Abstract; §6.4 Mechanistic Analysis / Figure 5]
"the analysis of the predicted strategies uncovers consistent stage-wise tendencies in dialogue progression, providing empirical evidence of common interaction patterns in conversational recommendation systems."
Ground-truth strategy labels already exhibit the stage-dependent distribution (Figure 3). The router is trained by supervised cross-entropy on exactly those labels (Eq. 7; teacher-forced in Algorithm 1). Accurate prediction therefore recovers the identical stage-wise pattern by construction of the fit; presenting the recovered distribution as independent empirical evidence is circular.
-
fitted input called prediction
[§5.2.2 Multi-Task Learning (Lsft Eq. 8); §6.1 (iii) Response Quality / Table 7 / Eq. 10]
"We evaluate response quality using human-annotated response pairs (𝑦𝑤, 𝑦𝑙 ), where𝑦𝑤 is the preferred response and 𝑦𝑙 is the less preferred one. We calculate (1)Pair-wise Win Rate by checking whether the model assigns a higher log-likelihood to the preferred response 𝑦𝑤 than to 𝑦𝑙 ; and (2)the Log-likelihood Margin"
Task experts are updated via supervised fine-tuning Lsft (Eq. 8) whose targets are the annotator-selected preferred (strategy-aware) responses that define the positive side of the contrastive pairs. Measuring whether the resulting model assigns higher normalized log-likelihood to those same yw than to yl is statistically forced by the SFT objective rather than an independent test of preference.
full rationale
The paper's central stage-dependence claim has independent non-circular support from a separate preliminary user study (328 participants, Figure 1) showing attribute-based preference early and item-based later; this is not derived from the later model. InPE annotations (Figure 3, Table 1) and COPE training/eval are self-contained offline ML experiments against baselines, with no self-definitional equations, no uniqueness theorems imported via self-citation, and no ansatz smuggling. However, two analysis/eval steps partially reduce to fitted inputs: (1) the hierarchical router is teacher-forced on the exact InPE strategy labels (Eq. 7, Algorithm 1) that already encode the stage-wise distribution, so accurate predictions necessarily reproduce it—yet Abstract and §6.4 present the recovered pattern in Figure 5 as new 'empirical evidence of common interaction patterns'; (2) experts receive Lsft (Eq. 8) on the annotator-preferred (strategy-aware) responses that define the positive pairs, after which pairwise win-rate/margin (Table 7, Eq. 10) simply checks higher likelihood on those same yw vs yl. These do not collapse the modeling contribution or the Recall gains, but they mean the 'uncovering' and response-quality claims are not fully independent of the annotation inputs used for supervision. Score 3 reflects partial/mild circularity only on those secondary claims; primary empirical pattern and architecture remain non-circular.
Axiom & Free-Parameter Ledger
free parameters (4)
- LoRA rank r and alpha
- Multi-task loss weights (w_sft, w_task, w_eli, w_rec)
- InfoNCE temperature τ
- Learning rate and warmup ratio
axioms (4)
- domain assumption User preferences evolve from abstract to concrete across dialogue stages, making attribute-based elicitation preferable early and item-based later.
- domain assumption Majority-vote crowd annotations of elicitation necessity and preferred strategy constitute a valid supervisory signal for real multi-turn utility.
- ad hoc to paper A frozen LLM backbone plus lightweight LoRA experts can specialize into distinct CRS behaviors without mutual interference under hard routing.
- standard math Standard cross-entropy and InfoNCE losses are appropriate objectives for routing and item ranking respectively.
invented entities (2)
-
InPE dataset
independent evidence
-
COPE hierarchical MoE architecture
no independent evidence
read the original abstract
Conversational Recommender Systems (CRSs) are interactive systems that use multi-turn natural language dialogue to understand evolving user preferences and provide personalized recommendations. To achieve this goal, CRSs rely on preference elicitation strategies to actively gather informative preference cues from users; however, the timing and selection of these strategies during a conversation remain largely unexplored. While many existing studies emphasize eliciting explicit item attributes and tend to adopt relatively static elicitation strategies, the use of item-based preference elicitation and how it varies across different dialogue stages remains less explored. In this work, we conduct a systematic investigation of preference elicitation strategies from a stage-aware perspective. We provide empirical evidence that optimal preference elicitation strategies are stage-dependent and context-sensitive: attribute-based inquiries are effective in early stages, while item-based strategies become superior as preferences refine. To support this paradigm, we introduce InPE, a dataset enriched with fine-grained annotations for elicitation necessity and strategy selection. With this dataset, we propose COPE (COnversational Preference Elicitation via Mixture of Experts), a novel architecture for strategy modeling. Extensive offline evaluation on our dataset indicates that context-aware preference elicitation strategies are beneficial for conversational recommendation. In addition, the analysis of the predicted strategies uncovers consistent stage-wise tendencies in dialogue progression, providing empirical evidence of common interaction patterns in conversational recommendation systems. Our dataset is available at https://github.com/juanfacabian/InPE.
Figures
Reference graph
Works this paper leans on
-
[1]
Laura Blédaité and Francesco Ricci. 2015. Pairwise preferences elicitation and exploitation for conversational collaborative filtering. InProceedings of the 26th ACM Conference on Hypertext and Social Media. 231–236
work page 2015
-
[2]
Qibin Chen, Junyang Lin, Yichang Zhang, Ming Ding, Yukuo Cen, Hongxia Yang, and Jie Tang. 2019. Towards knowledge-based recommender dialog system. arXiv preprint arXiv:1908.05391(2019)
work page internal anchor Pith review Pith/arXiv arXiv 2019
-
[3]
Konstantina Christakopoulou, Filip Radlinski, and Katja Hofmann. 2016. Towards conversational recommender systems. InProceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 815–824
work page 2016
-
[4]
Yang Deng, Yaliang Li, Fei Sun, Bolin Ding, and Wai Lam. 2021. Unified conversa- tional recommendation policy learning via graph-based reinforcement learning. InProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1431–1441
work page 2021
-
[5]
Chongming Gao, Wenqiang Lei, Xiangnan He, Maarten De Rijke, and Tat-Seng Chua. 2021. Advances and challenges in conversational recommender systems: A survey.AI open2 (2021), 100–126
work page 2021
-
[6]
Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. 2022. Recommendation as language processing (rlp): A unified pretrain, personalized prompt and predict paradigm (p5). InProceedings of the 16th ACM Conference on Recommender Systems. 299–315
work page 2022
-
[7]
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schel- ten, Alex Vaughan, et al . 2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783(2024)
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[8]
Shuyu Guo, Shuo Zhang, Weiwei Sun, Pengjie Ren, Zhumin Chen, and Zhaochun Ren. 2023. Towards explainable conversational recommender systems. InProceed- ings of the 46th International ACM SIGIR conference on research and development in information retrieval. 2786–2795
work page 2023
-
[9]
Shirley Anugrah Hayati, Dongyeop Kang, Qingxiaoyang Zhu, Weiyan Shi, and Zhou Yu. 2020. Inspired: Toward sociable recommendation dialog systems.arXiv preprint arXiv:2009.14306(2020)
work page internal anchor Pith review Pith/arXiv arXiv 2020
-
[10]
Zhankui He, Zhouhang Xie, Rahul Jha, Harald Steck, Dawen Liang, Yesu Feng, Bodhisattwa Prasad Majumder, Nathan Kallus, and Julian McAuley. 2023. Large language models as zero-shot conversational recommenders. InProceedings of the 32nd ACM International Conference on Information and Knowledge Management. 720–730
work page 2023
-
[11]
Diana C Hernandez-Bocanegra and Jürgen Ziegler. 2023. Explaining recommen- dations through conversations: Dialog model and the effects of interface type and degree of interactivity.ACM Transactions on Interactive Intelligent Systems 13, 2 (2023), 1–47
work page 2023
-
[12]
Chenhao Hu, Shuhua Huang, Yansen Zhang, and Yubao Liu. 2022. Learning to infer user implicit preference in conversational recommendation. InProceedings of the 45th International ACM SIGIR conference on research and development in information retrieval. 256–266
work page 2022
-
[13]
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models.ICLR1, 2 (2022), 3
work page 2022
-
[14]
Ivica Kostric, Krisztian Balog, and Filip Radlinski. 2024. Generating usage-related questions for preference elicitation in conversational recommender systems. ACM Transactions on Recommender Systems2, 2 (2024), 1–24
work page 2024
-
[15]
2018.Content analysis: An introduction to its methodology
Klaus Krippendorff. 2018.Content analysis: An introduction to its methodology. Sage publications
work page 2018
-
[16]
Wenqiang Lei, Xiangnan He, Yisong Miao, Qingyun Wu, Richang Hong, Min- Yen Kan, and Tat-Seng Chua. 2020. Estimation-action-reflection: Towards deep interaction between conversational and recommender systems. InProceedings of the 13th ACM International Conference on Web Search and Data Mining. 304–312
work page 2020
-
[17]
Wendi Li, Wei Wei, Xiaoye Qu, Xian-Ling Mao, Ye Yuan, Wenfeng Xie, and Dangyang Chen. 2023. Trea: Tree-structure reasoning schema for conversational recommendation.arXiv preprint arXiv:2307.10543(2023)
work page internal anchor Pith review Pith/arXiv arXiv 2023
-
[18]
Kevin McCarthy, James Reilly, Lorraine McGinty, and Barry Smyth. 2004. On the dynamic generation of compound critiques in conversational recommender systems. InInternational Conference on Adaptive Hypermedia and Adaptive Web- Based Systems. Springer, 176–184
work page 2004
-
[19]
Kevin McCarthy, Yasser Salem, and Barry Smyth. 2010. Experience-based cri- tiquing: Reusing critiquing experiences to improve conversational recommenda- tion. InInternational Conference on Case-Based Reasoning. Springer, 480–494
work page 2010
-
[20]
Bilih Priyogi. 2019. Preference elicitation strategy for conversational recom- mender system. InProceedings of the twelfth ACM International Conference on Web Search and Data Mining. 824–825
work page 2019
-
[21]
Xuhui Ren, Hongzhi Yin, Tong Chen, Hao Wang, Zi Huang, and Kai Zheng
-
[22]
Learning to ask appropriate questions in conversational recommendation. InProceedings of the 44th international ACM SIGIR Conference on Research and Development in Information Retrieval. 808–817
-
[23]
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt- Thieme. 2012. BPR: Bayesian personalized ranking from implicit feedback.arXiv preprint arXiv:1205.2618(2012)
work page internal anchor Pith review Pith/arXiv arXiv 2012
-
[24]
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538(2017)
work page internal anchor Pith review Pith/arXiv arXiv 2017
-
[25]
Yueming Sun and Yi Zhang. 2018. Conversational recommender system. InThe 41st international acm sigir conference on research and development in information retrieval. 235–244
work page 2018
-
[27]
Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua. 2019. Kgat: Knowledge graph attention network for recommendation. InProceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 950–958
work page 2019
-
[28]
Xiaolei Wang, Kun Zhou, Ji-Rong Wen, and Wayne Xin Zhao. 2022. Towards unified conversational recommender systems via knowledge-enhanced prompt learning. InProceedings of the 28th ACM SIGKDD conference on knowledge dis- covery and data mining. 1929–1937
work page 2022
-
[29]
Zhouhang Xie, Junda Wu, Hyunsik Jeon, Zhankui He, Harald Steck, Rahul Jha, Dawen Liang, Nathan Kallus, and Julian McAuley. 2024. Neighborhood-based collaborative filtering for conversational recommendation. InProceedings of the 18th ACM Conference on Recommender Systems. 1045–1050
work page 2024
-
[30]
Zhihui Xie, Tong Yu, Canzhe Zhao, and Shuai Li. 2021. Comparison-based conversational recommender system with relative bandit feedback. InProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1400–1409
work page 2021
-
[31]
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report.arXiv preprint arXiv:2505.09388(2025)
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[32]
Ting Yang and Li Chen. 2024. Unleashing the retrieval potential of large language models in conversational recommender systems. InProceedings of the 18th ACM Conference on Recommender Systems. 43–52
work page 2024
-
[33]
Tong Yu, Yilin Shen, and Hongxia Jin. 2019. A visual dialog augmented interactive recommender system. InProceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 157–165
work page 2019
-
[34]
Yiming Zhang, Lingfei Wu, Qi Shen, Yitong Pang, Zhihua Wei, Fangli Xu, Bo Long, and Jian Pei. 2022. Multiple choice questions based multi-interest policy learning for conversational recommendation. InProceedings of the ACM Web Conference 2022. 2153–2162
work page 2022
-
[35]
Kun Zhou, Wayne Xin Zhao, Shuqing Bian, Yuanhang Zhou, Ji-Rong Wen, and Jingsong Yu. 2020. Improving conversational recommender systems via knowl- edge graph based semantic fusion. InProceedings of the 26th ACM SIGKDD international conference on knowledge discovery and data mining. 1006–1014
work page 2020
-
[36]
Jie Zou, Yifan Chen, and Evangelos Kanoulas. 2020. Towards question-based rec- ommender systems. InProceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval. 881–890
work page 2020
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.