REVIEW 3 major objections 5 minor 33 references
User Willingness-aware Sales Talk Dataset
T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Explicitly labeling user willingness at the utterance level lets fine-tuned sales dialogue systems raise the rate of improving purchase intention from 0.23 to 0.44, and adding a stage-based willingness strategy strengthens the effect.
desk verdict A genuinely new willingness-annotated sales dialogue dataset, but the headline effectiveness claim is confounded by training-data mismatch and lacks significance testing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the dataset's utterance-level willingness annotation: for every sales-side utterance, user participants assign positive/neutral/negative labels for three willingness types (continuing dialogue CD, providing information PI, goal-acceptance GA). This turns an unobservable mental state into per-turn training supervision. The second piece is attribute-conditioned supervised fine-tuning, which feeds the three willingness labels as attributes during training and generation; the stage-based variant (GPT-3.5WD) switches the active attribute from CD to PI to GA across dialogue turns. Together they let the model be steered toward the willingness profile that the dataset analysis associates with successful sales.
What would settle it
A direct test would fix the training data: fine-tune one GPT-3.5 model on all 109 dialogues with no willingness labels and another on the same 109 dialogues with the willingness attributes, then run the same 48-participant evaluation. If the no-label model matches the 0.44 success rate, the willingness labels are not the cause; if the label model still wins, the claim is supported. A secondary check is whether the 0.23-to-0.44 gap reaches statistical significance with a larger sample.
Extended reading notes
Core claim
The central claim is that user willingness is a learnable, utterance-level signal that directly improves a sales system's ability to raise purchase intention. The paper's dataset is the first sales-talk corpus annotated with three willingness types per sales utterance — continuing the dialogue, providing information, and accepting the salesperson's objective — together with pre/post seven-point purchase-intention ratings. Analysis shows that successful dialogues avoid negative willingness utterances, and that willingness scores rise right after the dialogue opens and just before it closes, with goal-acceptance willingness climbing at the end. On this basis, the authors fine-tune GPT-3.5 with attribute-conditioned supervised fine-tuning, first conditioning generation on all three willingness labels set to 'positive' (GPT-3.5W), then conditioning on a stage schedule that activates one willingness at a time (GPT-3.5WD). In a 48-participant evaluation, GPT-3.5WD achieves a 0.44 success rate versus 0.23 for the baseline, while an untuned GPT-4o reaches 0.58 — a reference the paper treats as showing headroom rather than as the dataset's comparison.
Load-bearing premise
The paper's key comparison assumes that fine-tuning only on the 63 successful dialogues is a fair control for models fine-tuned on all 109 dialogues, so the difference in success rates is attributed to the willingness labels rather than to the different training sets.
Editorial extensions
If this is right
- If the claim holds, sales dialogue systems can be improved without hand-crafted persuasion rules, simply by training on utterance-level willingness annotations plus a stage schedule.
- Willingness labels provide a fine-grained training signal that coarse dialogue success or failure labels cannot: a baseline trained only on successful dialogues underperforms.
- The stage-based strategy (early engagement, middle information gathering, late goal acceptance) generalizes from dataset analysis into system behavior.
- Fine-tuned models still trail an untuned GPT-4o in success rate (0.44 vs 0.58), so the paper's contribution is better understood as a training signal that lifts smaller models, not as state-of-the-art persuasion.
- The dataset enables future work on real-time willingness prediction and dynamic strategy switching, which the paper names as next step.
Reading between the lines
- The willingness labels could double as a reward signal for reinforcement learning or as a ranking filter at inference, two uses the paper does not explore.
- Because the dataset analysis emphasizes avoiding negative willingness over maximizing positive, a model trained specifically to minimize negative jumps might replicate the strategic benefit without explicit stage scheduling.
- The annotation recipe should transfer to other product categories and languages, but the paper only demonstrates Japanese earbud sales, so the transfer claim is untested.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces a Japanese sales-talk dataset of 109 Wizard-of-Oz dialogues in which user-side participants rate each sales utterance for three types of willingness (continue dialogue, provide information, accept goal). The authors report corpus statistics, dialogue- and utterance-level analyses, and derive stagewise strategy recommendations. They then fine-tune GPT-3.5 in three configurations (baseline, willingness-conditioned, and willingness-plus-stage-strategy), evaluate with 48 new users, and report that willingness conditioning improves the purchase-intention success rate from 0.23 to 0.33, with the stage-strategy variant reaching 0.44.
Significance. The dataset is a potentially valuable contribution: to my knowledge it is the first sales-talk corpus with utterance-level user willingness annotations, and the Wizard-of-Oz setup with free user exit approximates deployment conditions more closely than prior human-to-human sales corpora. The released data and code are concrete strengths, as are the analyses of willingness distributions and negative-utterance counts, which generate testable hypotheses about what makes sales dialogues succeed. However, the experimental evidence for the paper's headline effectiveness claim is currently confounded, and Section 7.1 overstates the causal support for the value of willingness labels.
major comments (3)
- [Section 6.1 and Table 3] The baseline GPT-3.5 is fine-tuned on the 63 successful dialogues only, whereas GPT-3.5W and GPT-3.5WD are fine-tuned on all 109 dialogues. The reported success-rate differences therefore conflate the presence of willingness labels with a change in training-set composition and size. To support the claim that utterance-level willingness labels are responsible for the improvement, the authors need a control trained on all 109 dialogues without willingness conditioning (and ideally also a model trained on the 63 successful dialogues with willingness labels), or another design that holds the training data fixed while toggling labels.
- [Section 6.2 and Table 3] No significance tests, confidence intervals, or effect sizes are reported for the success-rate comparisons with n=48 participants. Given the small sample and four conditions measured within the same users, a paired test (e.g., McNemar's test) or a nonparametric bootstrap is needed before the 0.23 versus 0.33 versus 0.44 differences can be interpreted as reliable.
- [Section 6.1 (GPT-3.5WD)] The stage schedule (positive CD until turn 3, PI on turns 4 to 6, GA from turn 7) is selected post hoc from the same corpus analyzed in Section 5.2, and the three turn thresholds are free parameters. The comparison between GPT-3.5W and GPT-3.5WD therefore does not show that the derived strategy generalizes beyond this particular threshold choice; a sensitivity analysis or held-out validation of the strategy parameters is needed before claiming that incorporating the stage-based strategy 'further increases the effectiveness.'
minor comments (5)
- [Section 6.2] The prose contains several garbled passages, e.g., 'We For this experimental evaluation', 'same process and settings used for in the data collection', and 'The In addition'; these require copyediting.
- [Section 5.2] Footnote 4 renders as 'in each stage4' with a missing space; please fix the formatting.
- [Table 4] The sales utterance 'The willingness to continue dialogue is increasing.' reads as an unnatural leakage of the internal label. If this is an actual model output, clarifying its provenance would help; if it is a constructed illustration, state so explicitly.
- [Section 7.3] The failure-case annotation uses only 40 dialogues (10 per model) and does not report inter-annotator agreement; please report agreement coefficients and details of the sampling procedure.
- [Appendix A] The claim that the absence of users with the lowest pre-purchase intention score supports dataset quality assumes that the seven-point Likert scale is treated as interval data; this assumption should be stated.
Circularity Check
No significant circularity: the dataset, strategies, and models are evaluated on held-out users, and no central result reduces to its own inputs by construction.
full rationale
The paper's central derivation is dataset construction, analysis, strategy extraction, and system evaluation. None of these steps reduces to its inputs by definition. The Section 5.2 strategies (improve early willingness, maintain information-provision willingness in the middle, increase goal-acceptance willingness at the end) are empirical generalizations from the collected dialogues, but they are then implemented in GPT-3.5WD and tested on 48 new users; the success metric is a measured change in purchase intention, not a re-readout of the willingness labels or the strategy rules. Figure 6's correspondence between GPT-3.5WD's willingness trajectory and its designed strategy is a manipulation check, not a forced prediction. The main weakness is the confound noted in Section 6.1: the GPT-3.5 baseline is fine-tuned only on the 63 successful dialogues, while GPT-3.5W and GPT-3.5WD use all 109 dialogues, so the success-rate differences in Table 3 may partly reflect training-data quantity rather than willingness conditioning. This is a genuine experimental-control concern, but it is not circularity: the comparison is not equivalent to the input by construction, and no fitted parameter is renamed as a prediction. The paper also does not rely on any load-bearing self-citation or imported uniqueness theorem; the only nearby self-citation (Tsuta et al. 2023, used as motivation for utterance-level evaluation) is not load-bearing. The acknowledged limitations (product category, language, fictional purchase) are external-validity caveats, not circular steps. Therefore the paper's derivation chain is self-contained enough on the circularity axis, and the confound should be treated as a correctness-risk issue rather than a circularity finding.
Assumptions & free parameters
free parameters (2)
- Turn thresholds for strategy schedule in GPT-3.5WD =
3, 6 (turns)
- Willingness label mapping =
+1, 0, -1
assumptions (3)
- domain assumption Self-reported utterance-level willingness ratings accurately reflect users' true willingness during the dialogue
- domain assumption Change in the 7-point purchase-intention Likert score is a valid proxy for sales success
- domain assumption The Wizard-of-Oz environment with trained salespeople produces user behavior similar to interactions with a real automated system
Cite this review
Pith. "Pith review of User Willingness-aware Sales Talk Dataset." pith.science (2026). https://pith.science/paper/M5H6AZHU
@misc{pith2026241219490,
author = {Pith},
title = {Pith review of: User Willingness-aware Sales Talk Dataset},
year = {2026},
howpublished = {\url{https://pith.science/paper/M5H6AZHU}},
note = {Machine review of arXiv:2412.19490}
}
read the original abstract
User willingness is a crucial element in the sales talk process that affects the achievement of the salesperson's or sales system's objectives. Despite the importance of user willingness, to the best of our knowledge, no previous study has addressed the development of automated sales talk dialogue systems that explicitly consider user willingness. A major barrier is the lack of sales talk datasets with reliable user willingness data. Thus, in this study, we developed a user willingness-aware sales talk collection by leveraging the ecological validity concept, which is discussed in the field of human-computer interaction. Our approach focused on three types of user willingness essential in real sales interactions. We created a dialogue environment that closely resembles real-world scenarios to elicit natural user willingness, with participants evaluating their willingness at the utterance level from multiple perspectives. We analyzed the collected data to gain insights into practical user willingness-aware sales talk strategies. In addition, as a practical application of the constructed dataset, we developed and evaluated a sales dialogue system aimed at enhancing the user's intent to purchase.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
- [3]
- [4]
-
[5]
Egon Brunswik. 1940. Thing constancy as measured by correlation coefficients. Psychological Review, 47(1):69
work page 1940
-
[6]
Nils Dahlb \"a ck, Arne J \"o nsson, and Lars Ahrenberg. 1993. Wizard of oz studies: why and how. In Proceedings of the 1st international conference on Intelligent user interfaces, pages 193--200
work page 1993
-
[7]
Yasuharu Den, Junpei Nakamura, Toshinobu Ogiso, and Hideki Ogura. 2008. http://www.lrec-conf.org/proceedings/lrec2008/pdf/258_paper.pdf A proper approach to J apanese morphological analysis: Dictionary, model, and evaluation . In Proceedings of the Sixth International Conference on Language Resources and Evaluation ( LREC '08) , Marrakech, Morocco. Europe...
work page 2008
-
[8]
Yi Dong, Zhilin Wang, Makesh Sreedhar, Xianchao Wu, and Oleksii Kuchaiev. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.754 S teer LM : Attribute conditioned SFT as an (user-steerable) alternative to RLHF . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11275--11288, Singapore. Association for Computational Linguistics
Show all 33 references
-
[9]
Dubinsky
Alan J. Dubinsky. 1981. https://doi.org/10.1080/08853134.1981.10754192 A factor analytic study of the personal selling process . Journal of Personal Selling & Sales Management, 1(1):26--33
1981
-
[10]
Yannick Frommherz and Alessandra Zarcone. 2021. https://doi.org/10.3389/fcomp.2021.686050 Crowdsourcing ecologically-valid dialogue data for german . Frontiers in Computer Science, 3
2021
-
[11]
Sarik Ghazarian, Behnam Hedayatnia, Alexandros Papangelis, Yang Liu, and Dilek Hakkani-Tur. 2022. https://doi.org/10.18653/v1/2022.findings-acl.331 What is wrong with you?: Leveraging user sentiment for automatic dialog evaluation . In Findings of the Association for Computati...
2022 doi
-
[12]
Sarik Ghazarian, Ralph Weischedel, Aram Galstyan, and Nanyun Peng. 2020. https://doi.org/10.1609/aaai.v34i05.6283 Predictive engagement: An efficient metric for automatic evaluation of open-domain dialogue systems . Proceedings of the AAAI Conference on Artificial Intelligence...
2020 doi
-
[13]
Jana Götze, Maike Paetzel-Prüsmann, Wencke Liermann, Tim Diekmann, and David Schlangen. 2022. https://aclanthology.org/2022.lrec-1.433 The slurk Interaction Server Framework : Better Data for Better Dialog Models . In Proceedings of the Thirteenth Language Resources and Evalua...
2022
-
[14]
Takuya Hiraoka, Graham Neubig, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2016 a . Construction and analysis of a persuasive dialogue corpus. Situated Dialog in Speech-Based Human-Computer Interaction, pages 125--138
2016
-
[15]
Takuya Hiraoka, Graham Neubig, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2016 b . https://doi.org/10.1016/j.specom.2016.09.002 Learning cooperative persuasive dialogue policies using framing . Speech Communication, 84:83--96
2016 doi
-
[16]
Hite and Joseph A
Robert E. Hite and Joseph A. Bellizzi. 1985. https://doi.org/10.1080/08853134.1985.10754398 Differences in the importance of selling techniques between consumer and industrial salespeople . Journal of Personal Selling & Sales Management, 5(2):19--30
1985
-
[17]
Yeonkwon Jung. 2022. https://doi.org/10.1007/978-981-19-0051-8_5 Sales Talk , pages 87--114. Springer Nature Singapore, Singapore
2022 doi
-
[18]
J. F. Kelley. 1984. https://doi.org/10.1145/357417.357420 An iterative design methodology for user-friendly natural language office information applications . ACM Trans. Inf. Syst., 2(1):26–41
1984
-
[19]
Taku Kudo, Kaoru Yamamoto, and Yuji Matsumoto. 2004. https://aclanthology.org/W04-3230 Applying conditional random fields to J apanese morphological analysis . In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 230--237, Barcelona,...
2004
-
[20]
Levitt and John A
Steven D. Levitt and John A. List. 2007. https://doi.org/10.1257/jep.21.2.153 What do laboratory experiments measuring social preferences reveal about the real world? Journal of Economic Perspectives, 21(2):153–174
2007 doi
-
[21]
Lidiya Murakhovs ' ka, Philippe Laban, Tian Xie, Caiming Xiong, and Chien-Sheng Wu. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.657 Salespeople vs S ales B ot: Exploring the role of educational value in conversational recommender systems . In Findings of the Associat...
2023 doi
-
[22]
John Hill Sean Dwyer and Warren Martin. 2000. https://doi.org/10.1080/08853134.2000.10754235 An empirical investigation of critical success factors in the personal selling process for homogenous goods . Journal of Personal Selling & Sales Management, 20(3):151--159
2000
-
[23]
Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. https://doi.org/10.18653/v1/N19-1170 What makes a good conversation? how controllable attributes affect human judgments . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association f...
2019 doi
-
[24]
Iulian Vlad Serban, Ryan Lowe, Peter Henderson, Laurent Charlin, and Joelle Pineau. 2018. https://doi.org/10.5087/dad.2018.101 A Survey of Available Corpora For Building Data - Driven Dialogue Systems . Dialogue & Discourse, 9(1):1--49
2018 doi
-
[25]
Shiwei Sun, Jin Zhang, Yiwei Zhu, Mian Jiang, and Shuhui Chen. 2022. https://doi.org/10.1016/j.techfore.2022.121596 Exploring users' willingness to disclose personal information in online healthcare communities: The role of satisfaction . Technological Forecasting and Social C...
2022
-
[26]
Abhisek Tiwari, Abhijeet Khandwe, Sriparna Saha, Roshni Ramnani, Anutosh Maitra, and Shubhashis Sengupta. 2023. https://doi.org/10.1016/j.eswa.2022.118775 Towards personalized persuasive dialogue generation for adversarial task oriented dialogue setting . Expert Systems with A...
2023
-
[27]
Yuma Tsuta, Naoki Yoshinaga, Shoetsu Sato, and Masashi Toyoda. 2023. https://doi.org/10.18653/v1/2023.ijcnlp-srw.8 Rethinking response evaluation from interlocutor ' s eye for open-domain dialogue systems . In Proceedings of the 13th International Joint Conference on Natural L...
2023 doi
-
[28]
Harm De Vries, Dzmitry Bahdanau, and Christopher D. Manning. 2020. https://api.semanticscholar.org/CorpusID:220845833 Towards ecologically valid research on language user interfaces . ArXiv, abs/2007.14435
2020 arXiv
-
[29]
Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. 2019. https://doi.org/10.18653/v1/P19-1566 Persuasion for good: Towards a personalized persuasive dialogue system for social good . In Proceedings of the 57th Annual Meeting of the Associ...
2019 doi
-
[30]
Wilson and Daniel L
Elizabeth J. Wilson and Daniel L. Sherrell. 1993. https://doi.org/10.1007/bf02894421 Source effects in communication and persuasion research: A meta-analysis of effect size . Journal of the Academy of Marketing Science, 21(2):101–112
1993 doi
-
[31]
C. G. Wolf, J. M. Carroll, T. K. Landauer, B. E. John, and J. Whiteside. 1989. https://doi.org/10.1145/67449.67500 The role of laboratory experiments in hci: help, hindrance, or ho-hum? In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '89, pag...
1989
-
[32]
Zhou Yu, Leah Nicolich-Henkin, Alan W Black, and Alexander Rudnicky. 2016. https://doi.org/10.18653/v1/W16-3608 A W izard-of- O z study on a non-task-oriented dialog systems that reacts to user engagement . In Proceedings of the 17th Annual Meeting of the Special Interest Grou...
2016 doi
-
[33]
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. https://doi.org/10.18653/v1/P18-1205 Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational ...
2018 doi
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.