REVIEW 3 major objections 6 minor 24 references
Enhancing Impression Change Prediction in Speed Dating Simulations Based on Speakers' Personalities
T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read The paper claims that whether an utterance improves a speed-dating partner's impression depends on both speakers' personalities, and that a model using this signal outperforms generic dialogue-coherence selection in human evaluation.
desk verdict Novel task formulation for per-utterance impression change, but the training labels rely on an unvalidated temporal attribution that should be addressed before the results are taken at face value. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is a binary classifier built on a Longformer transformer encoder, fine-tuned on per-utterance love-scale change labels. The input format concatenates the dialogue partner's personality, the speaker's personality, the target utterance, and the last ten utterances of dialogue history, separated by special tokens [CLS], [P-SEP], [D-SEP], and [SEP]; a fully-connected layer on the max-pooled final hidden state outputs the probability that the utterance increases the love score. In simulation, P+D-based selection uses multiple cross-validation classifiers, picks the candidate receiving the most increase votes, and falls back to cohesion scoring when no candidate wins. This machinery carries the argument because both the ablation and the A/B test turn on the presence of personality features in this input.
What would settle it
Take the corpus's rating-change timestamps and re-assign each change to an utterance one, two, or three utterances earlier, or to a random utterance in the preceding ten, then retrain the P+D classifier and measure accuracy and F1 on held-out pairs; if the shuffled or lagged labels perform as well as the original labels, the model is not detecting utterance-level causes. Alternatively, have independent raters watch the dialogues and mark which utterance they believe caused each rating change, and compare their assignments with the original second-by-second assignment.
Extended reading notes
Core claim
The central claim is that whether an utterance improves a partner's impression of the speaker depends on both speakers' personalities, and that a classifier using this information can predict per-utterance impression changes well enough to improve dialogue simulation. The paper shows this in two stages: a supervised classifier that reads the partner's personality, the speaker's personality, the target utterance, and the dialogue history outperforms classifiers using only one of these sources; and when this classifier picks among generated utterances in simulated speed-dating dialogues, human participants judge the resulting dialogues as giving a better impression of the speaker 75% of the time against both a dialogue-history-only selector and a cohesion-based baseline. The task is framed as predicting increase versus non-increase, and the paper states as a limitation that it cannot model utterances that decrease the love score because the corpus contains too few of them.
Load-bearing premise
The load-bearing premise is that each love-scale score change is caused by the utterance being spoken at that second, with no meaningful lag, accumulation, or attribution to the other speaker; if that assignment is wrong, the classifier's target labels are wrong even if the model fits them.
Editorial extensions
If this is right
- If the central claim is right, utterance selection for social dialogue simulation should condition on both participants' personality profiles, not just coherence or dialogue context.
- Per-utterance impression labels make it possible to optimize dialogue generation directly for downstream social outcomes rather than text-quality metrics.
- The 75% preference rate in the A/B comparison suggests that personality-aware selection can produce visibly better first impressions in short dialogues.
- Because dialogue-history-only models performed no better than the cohesion baseline, dialogue history alone is not sufficient; personality features are the part that carries the improvement.
Reading between the lines
- The per-utterance label assignment assumes the partner's rating change at a given second is caused by the utterance then being spoken; if love-score changes lag or accumulate over several utterances, the classifier may be learning post-hoc attribution rather than genuine utterance effects, and a shuffled or lagged-label test would separate these.
- Because the corpus contains only 287 increasing and 155 decreasing utterances against 14,784 unchanged ones, the model is effectively trained to detect a rare positive event; a natural extension is to rank candidates by predicted impression change rather than binary increase or non-increase.
- The personality input includes a 25-item profile and 32 psychological scales, so the model may be using personalities partly as a proxy for speaker identity and stylistic preferences; a held-out-personality evaluation would show how much of the gain transfers to new people.
- The same personality-plus-dialogue selector could be applied to other high-stakes first-impression settings, such as job interviews or initial customer-service exchanges, whenever per-utterance impression labels or a proxy such as engagement can be collected.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a method for predicting whether an utterance improves a dialogue partner's impression during speed dating, using both speakers' personalities and the dialogue history. The authors annotate the MMSD corpus with per-utterance love-scale changes obtained from retrospective continuous self-ratings, train a Longformer-based classifier on a balanced subset of these labels, and evaluate with an ablation study (P+D vs D vs P vs None) and a human A/B test on simulated dialogues. The central claim is that personalities are useful for per-utterance impression-change prediction and that the proposed P+D-based utterance selection produces dialogues that are significantly preferred over those selected by a cohesion baseline or a D-only selector.
Significance. If the per-utterance labels are valid, the paper introduces a novel task and a practical demonstration: it is, to my knowledge, the first to condition utterance-level impression-change prediction on both speakers' personalities and dialogue context, and the human A/B evaluation provides direct evidence for downstream usefulness. The authors ship a concrete empirical comparison with significance testing, which is a strength. However, the validity of the label construction is the main scientific risk; if the temporal attribution of rating changes is unreliable, both research questions lose their foundation. The paper also leaves the cross-validation protocol underspecified, which weakens reproducibility.
major comments (3)
- [Section 2, Corpus] The label assignment procedure attributes every love-scale change to the utterance at the timestamp of the change, but the paper reports no validation that the change is actually caused by that utterance. A rating change could be a delayed reaction to an earlier utterance, an accumulated effect over several turns, or a reaction to the other speaker's utterance; the 1-second rating granularity does not by itself establish causal attribution. Because both RQ1 (Eq. 1 and Table 1) and RQ2 (Table 2) inherit these labels, an unvalidated temporal model could render the learned signal largely spurious, especially since only 287 of roughly 15,000 utterances are positive. The Limitation section mentions data sparsity for decreases but not this label-validity threat. Please add a validation study (e.g., inter-annotator agreement on the alignments, test-retest reliability, or a lag-aware analysis showing that results are robust to alternate alignments) or explicitly discuss why the current attribution is trustworthy.
- [Section 4.1.1, Experimental setting] The cross-validation description reads: 'For cross-validation, 48 pairs were held constant, while the two excluded pairs were evenly divided for validation and testing.' This is ambiguous: it does not state how many folds are used, whether the 549 training instances are split at the pair level or the utterance level, or how the 50 pairs are partitioned into training/validation/test across folds. Without a precise protocol, the reported standard deviations and the superscript significance tests in Table 1 are not reproducible. Please specify the exact k-fold or leave-pairs-out scheme and the paired vs unpaired test procedure.
- [Section 4.2.1, Experimental setting] The paper does not state whether the 20 participants in the A/B test overlap with the 10 annotators who created the love-scale labels or with the participants whose utterances were used to train the selection classifiers. If any of the test participants contributed to label annotation or to the training data in the cross-validation, the human evaluation could be inflated by leakage of that participant's preferences into the trained model. Please clarify the disjointness of the human-evaluation participants from the annotation and training sets, and if they are not fully disjoint, discuss the implications or re-run the evaluation with excluded participants.
minor comments (6)
- [Section 3.2, Classification model] The input format description uses special tokens [CLS], [SEP], [P-SEP], and [D-SEP] but does not explain how the personality features are encoded (e.g., as text strings, categorical IDs, or numeric vectors). Please give a concrete example or table of the input construction.
- [Table 1] Several metrics have very large standard deviations (e.g., recall of 0.73 ± 0.34 for D-only and 0.85 ± 0.15 for P-only). Reporting per-fold results or confidence intervals would help assess the stability of the differences, especially given the modest number of positive instances.
- [Section 4.1.2, Results] The claim that 'personalities are useful' is chiefly supported by the P+D condition being significantly better than D-only and the None baseline, while P-only is not significantly better than None. The abstract and conclusion state the claim more strongly than the ablation justifies; please qualify it as 'personalities are useful when combined with dialogue history' unless additional evidence is provided.
- [Appendix B] The 13 love-scale questions are adapted from Rubin's scale, but the paper does not report whether the internal consistency (e.g., Cronbach's alpha) of the averaged 13-item score was checked in this continuous rating setting. Please report this if available.
- [Limitations] The limitation section mentions only data sparsity for decreasing utterances. I recommend also acknowledging the label-validity risk and the potential for temporal misalignment, which is a more fundamental threat to the proposed approach.
- [Appendix C] There is a typo: 'descrived' should be 'described'. Also, the list of pre-training corpora is interesting but some entries (e.g., 'Corp. Website Crawl') are not referenced; please provide details or citations for reproducibility.
Circularity Check
No significant circularity: the personality-usefulness and dialogue-simulation claims are empirical and evaluated on held-out data and human preference; self-citations are data/tooling citations, not load-bearing.
full rationale
The paper's derivation chain is not circular. RQ1 trains T=f(P,u,D) from per-utterance labels and compares feature sets in cross-validation (Sec. 4.1); whether personalities help is an empirical result, not an input. RQ2 selects utterances with classifiers trained on held-out folds and tests the resulting dialogues by human A/B preference (Sec. 4.2), so the outcome is not forced by the model construction. The MMSD corpus (Ishii et al., 2023), the PPP generation model (Otsuka et al., 2024), and the cohesion baseline (Sugiyama, 2021) are cited from overlapping-author work, but they supply data, a generation tool, and a baseline, respectively; none of them is used to define the target prediction or to justify the conclusion that personalities improve prediction. The one nontrivial validity concern, flagged by the skeptical reading, is that Section 2 assigns a retrospective love-scale change to the utterance at the time of change without validating causality or inter-annotator reliability; however, this is a label-validity/measurement issue, not a circular reduction of the claimed prediction to its input features. No equation is equivalent to its input by construction, and no fitted parameter is renamed as a prediction. Score reflects only minor, non-load-bearing self-citations.
Assumptions & free parameters
assumptions (4)
- domain assumption A love-scale score change observed at time t is caused by the utterance being spoken at that time.
- domain assumption The 25-item profile and 32 psychological scales capture the personality information relevant to impression change.
- domain assumption Rubin's love scale is a suitable operationalization of 'impression' for the task.
- domain assumption The participant-specific generation models produce utterances similar enough to the real speakers that selection quality, not generation artifacts, determines the A/B outcome.
Cite this review
Pith. "Pith review of Enhancing Impression Change Prediction in Speed Dating Simulations Based on Speakers' Personalities." pith.science (2026). https://pith.science/paper/N2Y2ZTVV
@misc{pith2026250204706,
author = {Pith},
title = {Pith review of: Enhancing Impression Change Prediction in Speed Dating Simulations Based on Speakers' Personalities},
year = {2026},
howpublished = {\url{https://pith.science/paper/N2Y2ZTVV}},
note = {Machine review of arXiv:2502.04706}
}
read the original abstract
This paper focuses on simulating text dialogues in which impressions between speakers improve during speed dating. This simulation involves selecting an utterance from multiple candidates generated by a text generation model that replicates a specific speaker's utterances, aiming to improve the impression of the speaker. Accurately selecting an utterance that improves the impression is crucial for the simulation. We believe that whether an utterance improves a dialogue partner's impression of the speaker may depend on the personalities of both parties. However, recent methods for utterance selection do not consider the impression per utterance or the personalities. To address this, we propose a method that predicts whether an utterance improves a partner's impression of the speaker, considering the personalities. The evaluation results showed that personalities are useful in predicting impression changes per utterance. Furthermore, we conducted a human evaluation of simulated dialogues using our method. The results showed that it could simulate dialogues more favorably received than those selected without considering personalities.
Figures
Reference graph
Works this paper leans on
-
[1]
If [loved one] were feeling badly, my first duty would be to cheer him/her up
-
[2]
I feel that I can confide in [loved one] about virtually everything
-
[3]
I find it easy to ignore [loved one]’s faults
-
[4]
Exploring personalized neural conversational models. In Proceedings of the 26th International Joint Conference on Artificial Intelligence , pages 3728–3734. Jiwei Li, Michel Galley, Chris Brockett, Georgios Sp- ithourakis, Jianfeng Gao, and William B Dolan
-
[5]
I feel very possessive toward [loved one]
-
[6]
If I could never be with [loved one], I would feel miserable
-
[7]
If I were lonely, my first thought would be to seek [loved one] out
-
[8]
One of my primary concerns is [loved one]’s welfare
Show all 24 references
-
[9]
I would do almost anything for [loved one]
-
[10]
I feel responsible for [loved one]’s well be- ing
-
[11]
When I am with [loved one], I spend a good deal of time just looking at him/her
-
[12]
I would greatly enjoy being confided in by [loved one]
-
[13]
C Detail of training setting Using this dataset descrived in Sec
It would be hard for me to get along without [loved one]. C Detail of training setting Using this dataset descrived in Sec. 4.1.1, we fine- tuned a pre-trained Longformer model ( Beltagy et al. , 2020) with 81M parameters for 15 epochs. During fine-tuning, we utilized AdamW with...
2020
-
[14]
I would forgive [loved one] for practically anything
-
[19]
What is your hobby?
(Preparation) We manually set the initial ut- terance, “What is your hobby?”
-
[20]
Using X’s LoRA and the input utterance, we generate 20 replies
-
[21]
An utterance is chosen using the baseline
-
[22]
The same process is repeated with Y’s LoRA
-
[23]
Steps 2-4 are repeated until 10 utterances are generated
-
[24]
Mr. Ramen shop
We create three patterns of the final 10 ut- terances. For each pattern, Steps 2-5 are re- peated with a different method for Step 3. 6 Profile Psychological scale Age Rosenberg’s Self Esteem Scale (RSES) Final Education Self-Consciousness Scale Department Immersion Scale Reside...
-
[2016]
Not True
A persona-based neural conversation model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 994–1003. Ryan Lowe, Michael Noseworthy, Iulian V Serban, Nicolas Angelard-Gontier, Y oshua Bengio, and Joelle ...
2017 arXiv
-
[2017]
Psycholog- ical science, 28(10):1478–1489
Is romantic desire predictable? machine learn- ing applied to initial romantic attraction. Psycholog- ical science, 28(10):1478–1489. Satwik Kottur, Xiaoyu Wang, and Vitor R Carvalho
-
[2020]
arXiv preprint arXiv:2004.05150
Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 . Paul W Eastwick, Samantha Joel, Kathleen L Carswell, Daniel C Molden, Eli J Finkel, and Shelley A Blozis
2004 arXiv
-
[2022]
European Journal of Per- sonality, page 08902070221085877
Predicting romantic interest during early rela- tionship development: A preregistered investigation using machine learning. European Journal of Per- sonality, page 08902070221085877. Ryuichiro Higashinaka, Masahiro Mizukami, Hidetoshi Kawabata, Emi Y amaguchi, Noritake Adachi,...
2018
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.