{"id":"2dbb0eb8-3aeb-45e5-b927-dcb9388ce213","arxiv_id":"2412.19490","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new sales-talk dataset with utterance-level user willingness annotations is introduced, and GPT-3.5 models fine-tuned on it show higher purchase-intention success than a baseline in a user study.","lead":"This paper builds a Japanese sales-dialogue dataset where users rated each salesperson utterance for three kinds of willingness, and shows that fine-tuned GPT-3.5 models conditioned on those ratings can raise purchase-intention success rates. It matters because it gives dialogue researchers a new annotated resource, though the experiment has a training-data confound that limits the strength of the evidence.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The willingness-label effect is confounded with training-data size: GPT-3.5 baseline uses only 63 successful dialogues, while GPT-3.5W/WD use all 109, so success-rate gains in Table 3 may reflect data quantity, not willingness labels.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the baseline is not a valid control because it is trained on a different subset of the data. I agree with this assessment. The dataset construction and the within-condition comparison between GPT-3.5W and GPT-3.5WD (which share training data) provide some internal support, but the central claim that willingness labels cause higher success is not established without a training-data-matched baseline. The reader's CONDITIONAL verdict is therefore appropriate; my stress-test does not move it. I would keep the condition: the authors should add the missing control and report significance tests before the effectiveness claim is accepted.","tokens_in":11664,"tokens_out":2786,"duration_ms":26839,"concrete_test":"Run a control model fine-tuned on all 109 dialogues exactly like GPT-3.5W but without willingness attributes (same data, no attribute-conditioned SFT). Evaluate it on the same 48-participant counterbalanced protocol. If its success rate matches ~0.33–0.44, the label effect evaporates; if it stays near 0.23, the labels matter. Also report a paired test (e.g., McNemar or bootstrap CI) for baseline vs GPT-3.5W success rates.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6.1 defines three GPT-3.5 models. The baseline is fine-tuned on only the 63 successful dialogues; GPT-3.5W and GPT-3.5WD are fine-tuned on all 109 dialogues, with willingness labels injected via attribute-conditioned SFT. Table 3 then credits the labels for the success-rate gap (0.23 vs 0.33/0.44). But the manipulation changes two factors at once: (i) the presence of utterance-level willingness labels and (ii) the training set, from a success-only subset to the full corpus. Unsuccessful dialogues may contain useful negative examples or simply more data; either could improve a fine-tuned model independently of willingness conditioning. No experiment holds the training data fixed toggling only labels. The GPT-4o result underscores that model choice/swamping matters. Additionally, no confidence intervals or significance tests are reported for n=48 users, so the 0.23 vs 0.33 gap is not shown to be reliable. Thus the paper's headline claim—explicit willingness consideration improves success—is not yet supported by the controlled evidence. The dataset contribution stands, but the effectiveness claim needs the missing control.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces a Japanese sales-talk dataset of 109 Wizard-of-Oz dialogues in which user-side participants rate each sales utterance for three types of willingness (continue dialogue, provide information, accept goal). The authors report corpus statistics, dialogue- and utterance-level analyses, and derive stagewise strategy recommendations. They then fine-tune GPT-3.5 in three configurations (baseline, willingness-conditioned, and willingness-plus-stage-strategy), evaluate with 48 new users, and report that willingness conditioning improves the purchase-intention success rate from 0.23 to 0.33, with the stage-strategy variant reaching 0.44.","tokens_in":11918,"tokens_out":3051,"duration_ms":30543,"significance":"The dataset is a potentially valuable contribution: to my knowledge it is the first sales-talk corpus with utterance-level user willingness annotations, and the Wizard-of-Oz setup with free user exit approximates deployment conditions more closely than prior human-to-human sales corpora. The released data and code are concrete strengths, as are the analyses of willingness distributions and negative-utterance counts, which generate testable hypotheses about what makes sales dialogues succeed. However, the experimental evidence for the paper's headline effectiveness claim is currently confounded, and Section 7.1 overstates the causal support for the value of willingness labels.","major_comments":[{"comment":"The baseline GPT-3.5 is fine-tuned on the 63 successful dialogues only, whereas GPT-3.5W and GPT-3.5WD are fine-tuned on all 109 dialogues. The reported success-rate differences therefore conflate the presence of willingness labels with a change in training-set composition and size. To support the claim that utterance-level willingness labels are responsible for the improvement, the authors need a control trained on all 109 dialogues without willingness conditioning (and ideally also a model trained on the 63 successful dialogues with willingness labels), or another design that holds the training data fixed while toggling labels.","section":"Section 6.1 and Table 3"},{"comment":"No significance tests, confidence intervals, or effect sizes are reported for the success-rate comparisons with n=48 participants. Given the small sample and four conditions measured within the same users, a paired test (e.g., McNemar's test) or a nonparametric bootstrap is needed before the 0.23 versus 0.33 versus 0.44 differences can be interpreted as reliable.","section":"Section 6.2 and Table 3"},{"comment":"The stage schedule (positive CD until turn 3, PI on turns 4 to 6, GA from turn 7) is selected post hoc from the same corpus analyzed in Section 5.2, and the three turn thresholds are free parameters. The comparison between GPT-3.5W and GPT-3.5WD therefore does not show that the derived strategy generalizes beyond this particular threshold choice; a sensitivity analysis or held-out validation of the strategy parameters is needed before claiming that incorporating the stage-based strategy 'further increases the effectiveness.'","section":"Section 6.1 (GPT-3.5WD)"}],"minor_comments":[{"comment":"The prose contains several garbled passages, e.g., 'We For this experimental evaluation', 'same process and settings used for in the data collection', and 'The In addition'; these require copyediting.","section":"Section 6.2"},{"comment":"Footnote 4 renders as 'in each stage4' with a missing space; please fix the formatting.","section":"Section 5.2"},{"comment":"The sales utterance 'The willingness to continue dialogue is increasing.' reads as an unnatural leakage of the internal label. If this is an actual model output, clarifying its provenance would help; if it is a constructed illustration, state so explicitly.","section":"Table 4"},{"comment":"The failure-case annotation uses only 40 dialogues (10 per model) and does not report inter-annotator agreement; please report agreement coefficients and details of the sampling procedure.","section":"Section 7.3"},{"comment":"The claim that the absence of users with the lowest pre-purchase intention score supports dataset quality assumes that the seven-point Likert scale is treated as interval data; this assumption should be stated.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The dataset itself is a credible contribution, and the authors' design choices for ecological validity are thoughtful. The main risk is that the experimental section, as written, will be read as causal evidence for willingness-label effectiveness, when the training-set confound leaves that claim unsupported. I would treat the requested control experiment (all 109 dialogues without labels) and the significance testing as necessary conditions for acceptance, not as optional robustness checks."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The dataset is the real contribution here; the paper's headline claim that utterance-level willingness labels improve sales success is not yet supported by the experiments as designed.\n\nThe genuinely new thing is the corpus: 109 Japanese sales dialogues with per-utterance, three-way willingness annotations (continue dialogue, provide information, accept the goal), collected in a Wizard-of-Oz setup that makes a real effort to mimic a product-page chat. That resource does not exist in prior sales-talk datasets, and the authors did the collection carefully: multiple sales experts, crowd users, counterbalanced evaluation later. The correlation analysis is interesting—negative willingness ratings correlate with failure to improve purchase intent, and the stage-wise strategy (early CD, middle PI, late GA) is plausible and grounded in the data. They also ship the data and code, which is strong practice.\n\nThe soft spot is the main experiment. GPT-3.5 baseline was fine-tuned only on the 63 successful dialogues, while GPT-3.5W and GPT-3.5WD were trained on all 109. So the comparison in Table 3 changes two things at once: presence of willingness labels and size/composition of the training set. The success-rate gap (0.23 vs 0.33/0.44) could reflect extra training data or negative examples rather than the willingness annotation. The paper needs a control: a model fine-tuned on all 109 dialogues without willingness labels, and ideally another with labels but only on the 63. Also no significance tests or confidence intervals are reported for the 48-user evaluation, so the 0.10 and 0.21 gaps may not be reliable. That's not a minor quibble; the central claim about label value rests on this comparison.\n\nSecondary concerns: the GPT-3.5WD stage strategy is derived from the same dataset's successful dialogues and then evaluated on new users, which gives some independent grounding, but it is still a hand-picked schedule, not a learned policy with held-out validation. The untuned GPT-4o outperforming all fine-tuned models is itself interesting and suggests the practical upside of willingness-conditioned FT is modest. The limitations section is honest about the fictional product, Japanese-only, and no real purchases.\n\nBottom line: the dataset deserves serious refereeing and likely adoption, but the effectiveness claim needs a clean controlled comparison and statistical tests before I would believe it. I'd recommend acceptance pending those changes.","headline":"A genuinely new willingness-annotated sales dialogue dataset, but the headline effectiveness claim is confounded by training-data mismatch and lacks significance testing.","tokens_in":12431,"tokens_out":2860,"would_cite":true,"duration_ms":28613,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Explicitly labeling user willingness at the utterance level lets fine-tuned sales dialogue systems raise the rate of improving purchase intention from 0.23 to 0.44, and adding a stage-based willingness strategy strengthens the effect.","keywords":["sales dialogue","user willingness","utterance-level annotation","purchase intention","ecological validity","persuasive dialogue","dialogue dataset","fine-tuning"],"falsifier":"A direct test would fix the training data: fine-tune one GPT-3.5 model on all 109 dialogues with no willingness labels and another on the same 109 dialogues with the willingness attributes, then run the same 48-participant evaluation. If the no-label model matches the 0.44 success rate, the willingness labels are not the cause; if the label model still wins, the claim is supported. A secondary check is whether the 0.23-to-0.44 gap reaches statistical significance with a larger sample.","tokens_in":11459,"feed_emoji":"🛒","tokens_out":5894,"duration_ms":53132,"temperature":0.7,"pith_summary":"This paper tries to establish that a sales dialogue system performs better when it is trained on fine-grained, utterance-level judgments of the user's willingness: willingness to keep talking, to volunteer information, and to accept the seller's goal. To get reliable labels, the authors built a dataset of 109 Japanese sales chats for wireless earbuds using a Wizard-of-Oz setup designed for ecological validity, with users rating each sales utterance. They then fine-tuned GPT-3.5 with these labels and a stage-dependent strategy (early: keep dialogue; middle: gather information; late: push toward purchase). In a user study, the willingness-and-strategy model improved purchase intention in 44% of dialogues, versus 23% for a baseline fine-tuned only on successful dialogues. The point that matters: utterance-level willingness data, not just dialogue outcome, may be the actionable signal for building persuasive sales systems.","feed_headline":"Willingness labels lift sales-chat success from 23% to 44%","feed_subtitle":"Utterance-level willingness ratings give fine-tuned dialogue models a sharper signal than sales outcomes alone.","key_machinery":"The load-bearing object is the dataset's utterance-level willingness annotation: for every sales-side utterance, user participants assign positive/neutral/negative labels for three willingness types (continuing dialogue CD, providing information PI, goal-acceptance GA). This turns an unobservable mental state into per-turn training supervision. The second piece is attribute-conditioned supervised fine-tuning, which feeds the three willingness labels as attributes during training and generation; the stage-based variant (GPT-3.5WD) switches the active attribute from CD to PI to GA across dialogue turns. Together they let the model be steered toward the willingness profile that the dataset analysis associates with successful sales.","core_discovery":"The central claim is that user willingness is a learnable, utterance-level signal that directly improves a sales system's ability to raise purchase intention. The paper's dataset is the first sales-talk corpus annotated with three willingness types per sales utterance — continuing the dialogue, providing information, and accepting the salesperson's objective — together with pre/post seven-point purchase-intention ratings. Analysis shows that successful dialogues avoid negative willingness utterances, and that willingness scores rise right after the dialogue opens and just before it closes, with goal-acceptance willingness climbing at the end. On this basis, the authors fine-tune GPT-3.5 with attribute-conditioned supervised fine-tuning, first conditioning generation on all three willingness labels set to 'positive' (GPT-3.5W), then conditioning on a stage schedule that activates one willingness at a time (GPT-3.5WD). In a 48-participant evaluation, GPT-3.5WD achieves a 0.44 success rate versus 0.23 for the baseline, while an untuned GPT-4o reaches 0.58 — a reference the paper treats as showing headroom rather than as the dataset's comparison.","pith_inferences":["The willingness labels could double as a reward signal for reinforcement learning or as a ranking filter at inference, two uses the paper does not explore.","Because the dataset analysis emphasizes avoiding negative willingness over maximizing positive, a model trained specifically to minimize negative jumps might replicate the strategic benefit without explicit stage scheduling.","The annotation recipe should transfer to other product categories and languages, but the paper only demonstrates Japanese earbud sales, so the transfer claim is untested."],"forward_implications":["If the claim holds, sales dialogue systems can be improved without hand-crafted persuasion rules, simply by training on utterance-level willingness annotations plus a stage schedule.","Willingness labels provide a fine-grained training signal that coarse dialogue success or failure labels cannot: a baseline trained only on successful dialogues underperforms.","The stage-based strategy (early engagement, middle information gathering, late goal acceptance) generalizes from dataset analysis into system behavior.","Fine-tuned models still trail an untuned GPT-4o in success rate (0.44 vs 0.58), so the paper's contribution is better understood as a training signal that lifts smaller models, not as state-of-the-art persuasion.","The dataset enables future work on real-time willingness prediction and dynamic strategy switching, which the paper names as next step."],"supporting_citations":[{"why":"Supplies the ecological validity concept that motivates the realistic data-collection design.","marker":"(Brunswik, 1940, 1952)"},{"why":"Provides the Wizard-of-Oz method used to collect near-realistic user-side data.","marker":"(Dahlbäck et al., 1993)"},{"why":"Earlier camera-sales persuasive dialogue corpus that this dataset extends by adding willingness labels.","marker":"(Hiraoka et al., 2016a)"},{"why":"Earlier personalized persuasive dialogue dataset that also lacks willingness data.","marker":"(Tiwari et al., 2023)"},{"why":"Attribute-conditioned SFT is the training method that injects willingness labels into GPT-3.5.","marker":"(Dong et al., 2023)"},{"why":"Ecological-validity methodology for language user interfaces that shapes the collection pipeline.","marker":"(Vries et al., 2020)"},{"why":"Supports the claim that human-to-system dialogues differ from human-to-human, justifying the Wizard-of-Oz setup.","marker":"(Serban et al., 2018)"}],"fun_headline_variants":["Willingness signals boost sales-chat success from 23% to 44%","First sales-talk dataset with utterance-level willingness labels","Fine-tuning on user willingness nearly doubles sales success","Sales AI learns to read willingness, lifting purchase intent"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's key comparison assumes that fine-tuning only on the 63 successful dialogues is a fair control for models fine-tuned on all 109 dialogues, so the difference in success rates is attributed to the willingness labels rather than to the different training sets.","fun_headline_variants_meta":{"raw":{"variants":["Willingness signals boost sales-chat success from 23% to 44%","First sales-talk dataset with utterance-level willingness labels","Fine-tuning on user willingness nearly doubles sales success","Sales AI learns to read willingness, lifting purchase intent"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000664,"raw_usage":{"total_tokens":3032,"prompt_tokens":947,"completion_tokens":2085,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":2017}},"tokens_in":563,"tokens_out":2085,"duration_ms":17797,"temperature":1.0,"reasoning_tokens":2017,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:16:27.606906+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would fix the training data: fine-tune one GPT-3.5 model on all 109 dialogues with no willingness labels and another on the same 109 dialogues with the willingness attributes, then run the same 48-participant evaluation. If the no-label model matches the 0.44 success rate, the willingness labels are not the cause; if the label model still wins, the claim is supported. A secondary check is whether the 0.23-to-0.44 gap reaches statistical significance with a larger sample.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ecological validity concept that motivates the realistic data-collection design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Earlier personalized persuasive dialogue dataset that also lacks willingness data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the claim that human-to-system dialogues differ from human-to-human, justifying the Wizard-of-Oz setup."}],"review_version":1}