Pith. sign in

REVIEW 5 major objections 5 minor 15 references

PicPersona-TOD : A Dataset for Personalizing Utterance Style in Task-Oriented Dialogue with Image Persona

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read User images can serve as a task-oriented dialogue persona: PicPersona-TOD pairs photos with dialogues and rewrites both sides of the conversation to match the user's apparent age, formality, and emotion, and its trained model Pictor…

desk verdict First visual-persona TOD dataset with a mostly sound pipeline, but the emotion-alignment statistics are internally inconsistent and need an audit before the dataset premise is fully trusted. read the letter →

arxiv 2504.17390 v1 pith:RC5MNDSE submitted 2025-04-24 cs.CL

Jihyun Lee , Yejin Jeon , Seungyeon Seo , Gary Geunbae Lee This is my paper · ORCID
classification cs.CL
keywords task-orienteddialoguepersonalizationvisualpersonamultimodalLLMdatasetdistillationnaturallanguagegenerationretrieval-augmentedhumanevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PicPersona-TOD is a new dataset for task-oriented dialogue in which a user photo is part of the persona: each of 18,148 dialogues is paired with an image, and both user and system utterances are rewritten so that age, formality, greetings, and emotional tone match what the photo conveys. The paper's claim is that this visual-persona signal produces measurably more engaging system responses than generic or text-only personalization, without losing the underlying task information. Human raters scored the dataset close to 4 on a 4-point scale for style appropriateness, semantic consistency, and satisfaction, and preferred it over two text-persona baselines. A companion model, Pictor, trained on the dataset, keeps that personalization when tested in bus, movie, and home domains it never saw during training.

What carries the argument

The load-bearing mechanism is first-impression-guided prompting. For each image, an LLM writes a one-sentence first impression covering age, gender, emotion, outfit, and overall vibe; that impression is then inserted into prompts that restyle the user's utterances and generate personalized system utterances. The pipeline also aligns images to dialogues through shared emotion labels, filters dialogues by embedding-based personalization strength and direction vectors, and grounds recommendations in retrieved reviews and encyclopedia entries. Pictor reproduces this at inference time by taking the image, the generated impression, turn progress, dialogue policy, and database results as inputs to a vision-language model.

What would settle it

Re-annotate a random sample of the claimed image-dialogue pairs with independent human raters or a different emotion classifier: if the image's emotion and the dialogue's emotion match no more often than chance, or if the labels cannot be reproduced, then the alignment stage contributes no persona coherence and the dataset premise is unsupported. A simpler check is arithmetic: the stated distribution (50.92% positive, 52.44% neutral, 0.55% negative) sums to 103.91%, so the counts behind it must be re-derived before the pairing criterion can be trusted.

Watch

Extended reading notes

Core claim

The central discovery the paper argues for is that user images can carry the persona in task-oriented dialogue: a system that first forms a first impression from the image and then uses that impression, the dialogue policy, and retrieved facts to rewrite its utterances can tailor formality, vocabulary, greetings, and emotional awareness to the individual user. The paper builds PicPersona-TOD to demonstrate this, starting from MultiWOZ-2.2 and SGD dialogues, pairing them with FFHQ images by matching predicted emotion labels, restyling user utterances, personalizing system responses with first-impression prompts and retrieval-augmented recommendations, and filtering by personalization strength, direction, semantics, and naturalness. The demonstration includes human evaluations showing high preference over text-based personalization, DST and policy results comparable to MultiWOZ, and zero-shot personalization by Pictor in held-out domains. The intended conclusion is that a vision-based persona is a workable, information-preserving route to personalized task-oriented dialogue.

Load-bearing premise

The dataset's premise rests on the claim that two independently computed emotion labels—one for an image and one for a dialogue—are reliable enough that matching them produces coherent persona-dialogue pairs; the paper's reported emotion distribution sums to 103.91%, so that reliability is not demonstrated.

Editorial extensions

If this is right

  • If the central claim is right, a task-oriented dialogue system that can see its user can adjust formality, word difficulty, greetings, and emotional tone to the user's apparent age and mood.
  • Personalization does not have to come at the cost of task accuracy: DST joint-goal accuracy and policy entity-F1 on PicPersona-TOD stay close to MultiWOZ, the human-curated source dataset.
  • The visual signal generalizes: Pictor's zero-shot scores in bus, movie, and home domains are near 4 on a 4-point scale, suggesting the model learned a transferable mapping from image to response style rather than domain-specific templates.
  • An automated LLM pipeline with filtering can produce a large, multi-domain personalized TOD dataset (18,148 dialogues across 18 services) at far lower cost than human annotation.
  • Retrieval-grounded recommendation generation is a workable guard against hallucinated personalized suggestions, since the model is told to connect recommendations to real review and entry information.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the reported emotion distribution sums to 103.91%, the emotion-alignment stage may not be doing what it appears to do; a clean test is to compare PicPersona-TOD built with emotion-matched pairs against a version built with randomly paired images and see whether downstream personalization scores change.
  • The style-transfer results might be driven mainly by the first-impression prompt rather than by the image itself; training Pictor with text-only impressions (no image input) would separate the contribution of raw pixels from the contribution of the inferred persona description.
  • The requirement of a camera for deployment could be relaxed by using images only as a training-time supervisor and letting the deployed model infer persona from ordinary descriptions, which would extend the personalization to text-only interactions.
  • The same automatic pipeline should transfer to other service domains or privacy-safe synthetic faces; the filtering thresholds would need recalibration, but the core first-impression and retrieval components are domain-agnostic.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces PicPersona-TOD, a task-oriented dialogue dataset that pairs FFHQ user images with dialogues from MultiWOZ 2.2 and SGD, rewrites user and system utterances for style personalization using GPT-4o, filters the result with style-strength, style-direction, semantic, and naturalness checks, and trains an NLG baseline, Pictor, on the dataset. The authors report human evaluations showing high dataset quality and personalization preference, lexical and emotional analyses, and a generalization experiment on unseen domains. The central claim is that this is the first visual-persona TOD dataset and that it enables personalized responses without compromising task information.

Significance. If the dataset is sound, it fills a clear gap: existing TOD datasets use only textual personas, while prior visual-persona dialogue work targets chit-chat. PicPersona-TOD is large (18,148 dialogues, 18 services), grounded with external knowledge, and released with a public repository, which are practical strengths. The paper also deserves credit for including human evaluation with strong inter-rater agreement, for comparing against textual-personalization baselines, and for candidly discussing its limitations, including the absence of a direct human-curated benchmark. The main risks are the emotion-based image-dialogue alignment step, the lack of baselines/error bars in the generalization and model-comparison claims, and the reliance on GPT-4-family models for both generation and evaluation. These issues are local and fixable, but they are load-bearing for the dataset premise and for the headline empirical claims.

major comments (5)
  1. [2.2 (and Appendix B)] The reported emotion distribution in the dataset is 50.92% positive, 52.44% neutral, and 0.55% negative; these sum to 103.91%, not 100%. Since the paper states that image-dialogue pairs are accepted only when the predicted emotion labels match, these mutually exclusive labels underlie the entire alignment step, and the inconsistent sum means the labels or their aggregation cannot be fully trusted. This opens the possibility that a non-trivial fraction of image-dialogue pairs are effectively arbitrary. I request a full audit of the emotion labels (including the dialogue sentiment classifier output), a corrected distribution, and a direct pair-coherence test, for example human judges rating whether each image and dialogue plausibly belong to the same person, on a random sample. Without this evidence, the central dataset premise that user images are genuinely part of the persona is not verified.
  2. [6.2, Table 4] The unseen-domain generalization claim is presented without any baseline. The reported scores of 3.51-3.96 on a 4-point scale are only interpretable relative to a comparison system, such as Pictor trained on seen domains only, a non-personalized NLG, or the original unpersonalized system responses from MultiWOZ/SGD. I recommend adding such a baseline and reporting annotator-level agreement or confidence intervals, since the current table does not establish that the high scores are due to personalization rather than to the generic quality of the generated utterances.
  3. [6.1, Figure 9] The model comparison in Section 6.1 is adjudicated by GPT-4, which belongs to the same model family as the GPT-4o generator used to create the training data. Human evaluation in Section 4.1 mitigates this concern for dataset quality, but for the specific comparison between Pictor and GPT-4o-mini there is no human validation. The reported superiority could partly reflect a judge-side preference for GPT-4o-style generations. I recommend adding a human evaluation on the sampled 100 dialogues or, failing that, reporting the human-GPT agreement on this specific comparison.
  4. [Tables 5 and 6] Table 5 states that each experiment was repeated three times and averaged, but no variance, standard deviation, or significance test is reported, so the claimed differences, such as the impression component improving style scores, are not statistically supported. Table 6 similarly reports DST and policy differences without error bars or significance tests, which weakens the conclusion that PicPersona-TOD maintains information accuracy on par with MultiWOZ. Please report mean and variance (or bootstrap confidence intervals) and, where feasible, a paired significance test for the key comparisons.
  5. [2.5] The style-strength and style-direction filtering thresholds, 2.5×IQR and 4.5×IQR, are introduced without justification or sensitivity analysis. Because filtering removes 1.49% and 1.98% of the dataset in these stages, the final dataset properties depend on these choices. I ask for a short sensitivity analysis showing how the final dataset size, personalization strength, and downstream evaluation results change under a range of threshold values, or at least a rationale for why these specific multipliers are appropriate.
minor comments (5)
  1. [Various prompt templates (Appendix I)] Typographical errors are frequent in the appendix prompts: 'Dose' should be 'Does', 'Promps' in the Figure 14 caption should be 'Prompts', and 'sLLM' in Section 6.1 should be 'small LLM' or similar. These should be corrected for a camera-ready version.
  2. [Throughout] The model name 'LLaV A' appears with an erroneous space throughout the paper and should be 'LLaVA'.
  3. [Introduction] The phrase 'lacks details and concurrency about the users' appears to use 'concurrency' where 'consistency' or 'currency' was intended; please revise.
  4. [Appendix G] The paper specifies that 100 dialogues are sampled for Section 6.1 and 100 per domain for Section 6.2, but it does not state whether the Section 6.1 sample overlaps with the Section 6.2 unseen-domain samples; please clarify to avoid ambiguity about the evaluation protocol.
  5. [Limitations] The Limitation paragraph 'Lack of Direct Benchmark Comparison' is directly relevant to Section 6.2's generalization claim; consider moving or cross-referencing it there so that the absence of a baseline is acknowledged at the point of the claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation detected: dataset quality and personalization claims rest on human evaluation, and no equation or fitted parameter is renamed as a prediction.

full rationale

Walking the paper's derivation chain, the central contributions are an automatically constructed dataset and a separately trained NLG model, Pictor. The dataset pipeline (Section 2) uses GPT-4o, image/utterance alignment by emotion labels, style transfer, response personalization, and filtering; none of these steps defines its output in terms of the later evaluation metric. The quality and personalization claims in Sections 4.1 and 6.2 are supported by external human ratings on four-point Likert scales, not by a formula that folds the dataset back into the evaluation. The reported emotion distribution in Section 2.2 (50.92% positive + 52.44% neutral + 0.55% negative = 103.91%) is a serious correctness red flag that should be audited, but an internal inconsistency in label aggregation is not a circular derivation. The DST/policy evaluation in Section 6.4 is performed on data that was filtered for DST/policy consistency, so it is a consistency check rather than an independent prediction; it does not reduce to the labels by construction because the filtering is LLM-based and could have failed. The unseen-domain test in Section 6.2 uses dialogues built with the same GPT-4o pipeline as the training data, which is an evaluation-design limitation that weakens the generalization claim, but the reported human scores are measured, not derived from the training input. No load-bearing self-citation is present: the cited prior work is not by the present authors, and no uniqueness theorem or ansatz is imported from the authors' own work. The GPT-4o-as-generator and GPT-4-as-judge setup in Section 6.1 is a same-family evaluation risk, but the paper also reports human evaluation and a human-model correlation of 0.84, providing external grounding. Under the strict standard of exhibiting a specific reduction, such as Eq. X equaling Eq. Y by construction or a fitted parameter being renamed as a prediction, no circular step is established in this paper.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on domain assumptions about face-image inferences, emotion-based image-dialogue pairing, and LLM output fidelity. The only hand-set numeric parameters are the two IQR filter thresholds in Section 2.5. No invented entities are introduced.

free parameters (2)
  • PS threshold multiplier = 2.5 × IQR
    Section 2.5 removes dialogues whose personalization strength is below 2.5 times the interquartile range; this multiplier is chosen by the authors and affects which dialogues remain.
  • PV direction threshold multiplier = 4.5 × IQR
    Section 2.5 removes 1.98% of dialogues whose personalization vector distance from the class mean exceeds 4.5 times the interquartile range; no principled justification is given.
assumptions (4)
  • domain assumption Face images from FFHQ carry reliable persona signals (age, gender, formality, emotion) that are appropriate to drive dialogue style.
    Sections 2.1 and 2.4 extract metadata and first impressions from images and use them to rewrite system responses; the paper provides no validation that these inferences are accurate or socially appropriate.
  • domain assumption Emotion labels computed for images and dialogues are consistent enough that matching identical labels yields coherent persona-dialogue pairs.
    Section 2.2 pairs each dialogue with an image only when the predicted sentiment matches; the reported distribution sums to over 100%, so the label pipeline is not trustworthy as reported.
  • ad hoc to paper GPT-4o style transfer preserves dialogue state, policy labels, and factual content.
    The construction in Sections 2.3 and 2.4 rewrites every user and system utterance with GPT-4o; DST and policy compatibility are filtered by LLM checks and sampled human evaluation, not guaranteed.
  • domain assumption Retrieved Google Maps reviews and Wikipedia entries are relevant and sufficient to ground recommendation personalization.
    Section 2.4 uses Sentence-BERT retrieval with the three most similar reviews; retrieval quality is not measured, so factual grounding is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PicPersona-TOD : A Dataset for Personalizing Utterance Style in Task-Oriented Dialogue with Image Persona." pith.science (2026). https://pith.science/paper/RC5MNDSE

@misc{pith2026250417390,
  author       = {Pith},
  title        = {Pith review of: PicPersona-TOD : A Dataset for Personalizing Utterance Style in Task-Oriented Dialogue with Image Persona},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RC5MNDSE}},
  note         = {Machine review of arXiv:2504.17390}
}
read the original abstract

Task-Oriented Dialogue (TOD) systems are designed to fulfill user requests through natural language interactions, yet existing systems often produce generic, monotonic responses that lack individuality and fail to adapt to users' personal attributes. To address this, we introduce PicPersona-TOD, a novel dataset that incorporates user images as part of the persona, enabling personalized responses tailored to user-specific factors such as age or emotional context. This is facilitated by first impressions, dialogue policy-guided prompting, and the use of external knowledge to reduce hallucinations. Human evaluations confirm that our dataset enhances user experience, with personalized responses contributing to a more engaging interaction. Additionally, we introduce a new NLG model, Pictor, which not only personalizes responses, but also demonstrates robust performance across unseen domains https://github.com/JihyunLee1/PicPersona.

Figures

Figures reproduced from arXiv: 2504.17390 by the authors.

Figure 1
Figure 1. Example of PicPersona-TOD: Unlike existing [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An overview of the automatic pipeline for generating PicPersona-TOD dataset. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Visualizations of personalization strength and [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Examples of filtered-out results: style strength filtering (left) and style direction filtering (right). [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Lexical analysis of PicPersona-TOD. "Word [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Human evaluation results for dataset quality [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: The winning characteristics analysis across [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Overview of the proposed model Pictor. (e.g., start, middle, end), user’s utterance (u˜t), di￾alogue policy (polt), DB results (DBt), and the user’s image (Imgi ). Similar to the process used in constructing the PicPersona-TOD dataset, we first generate the user’s impr…
Figure 9
Figure 9. Figure 9: Performance comparison of LLama 3-8B, Pictor 7B with LLaVA 7B, InstrucBLIP 7B, and GPT￾4o mini across various user scenarios. optimal performance in personalized scenarios. 6.2 Generalization Performance Evaluation Domain Natural. Fluency Personalize Semantic User Sati…
Figure 10
Figure 10. Figure 10: Examples of greeting personalization (left) and recommendation personalization (right). [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 12
Figure 12. Figure 12: A screenshot of the Comparison with Other [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: PicPersona-TOD dialogue sample [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: Promps templates [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 8 canonical work pages

  1. [1]

    Dose the paraphrase user utterance is well matched to user description?

    work page

  2. [2]

    Dose the user paraphrased user utterance is semantically equivalent to the original user utterance ?

    work page

  3. [3]

    Dose the paraphrase system utterance is well personalized (style, tone, formality) to user description? ‐ 1: Not at all (The sentence paraphrased system utterance is not personalized with specific words, phrases, or style to user description) ‐ 2: A little (changes formality or tone for according to user description (Please tell your plan ‐> Could I know ...

    work page

  4. [4]

    arXiv preprint arXiv:2310.13420

    Conversation chronicles: Towards diverse tem- poral and relational dynamics in multi-session con- versations. arXiv preprint arXiv:2310.13420. Chaitanya K Joshi, Fei Mi, and Boi Faltings. 2017. Per- sonalization in goal-oriented dialog. arXiv preprint arXiv:1706.07503. Tero Karras, Samuli Laine, and Timo Aila. 2019. A style-based generator architecture fo...

    arXiv 2017

  5. [5]

    Does the system’s utterance enhance the overall user experience, compared to system_reference? assess the dataset in four scales

    work page

  6. [6]

    MinTL: Minimalist Transfer Learning for Task-Oriented Dialogue Systems

    Emous: Simulating user emotions in task- oriented dialogues. In Proceedings of the 46th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2526– 2531. Zhaojiang Lin, Andrea Madotto, Genta Indra Winata, and Pascale Fung. 2020. Mintl: Minimalist transfer learning for task-oriented dialogue systems. arXiv preprin...

    work page Pith review arXiv 2020

  7. [7]

    arXiv preprint arXiv:2010.11853

    Star: A schema-guided dialog dataset for trans- fer learning. arXiv preprint arXiv:2010.11853. Atsumoto Ohashi and Ryuichiro Higashinaka. 2023. Enhancing task-oriented dialogue systems with gen- erative post-processing networks. In Proceedings of the 2023 Conference on Empirical Methods in Natu- ral Language Processing, pages 3815–3828. OpenAI. Gpt-4o min...

    arXiv 2010

  8. [9]

    "" if previous_system: prompt += f

    Multiwoz 2.2: A dialogue dataset with addi- tional annotation corrections and state tracking base- lines. In Proceedings of the 2nd Workshop on Natu- ral Language Processing for Conversational AI, ACL 2020, pages 109–117. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018a. Per- sonalizing dialogue agents: I have ...

    arXiv 2020

Show all 15 references
  1. [13]

    Dose the system paraphrased system utterance is semantically equivalent to the original system utterance?

  2. [15]

    "" Prompt for Dialogue Personalization and Paraphrase Evaluation. (Section 6.1) prompt = f

    4: A lot **Original Dialogue** {dial1} **User Image Description:** {user_impression} **Changed Dialogue:** {dial2} Your answer must be in the following format/. Below is just an example, not the actual answer.: Score : (Q1:3, Q2:4, Q3:2, Q4:4, Q5:3) Score : (Q1:3, Q2:3, Q3:4, ...

  3. [2018]

    arXiv preprint arXiv:1801.04871

    Building a conversational agent overnight with dialogue self-play. arXiv preprint arXiv:1801.04871. Xiaoteng Shen, Rui Zhang, Xiaoyan Zhao, Jieming Zhu, and Xi Xiao. 2024. Pmg: Personalized multimodal generation with large language models. In Proceed- ings of the ACM on Web Co...

    2024 arXiv

  4. [2020]

    arXiv preprint arXiv:2005.00547

    Goemotions: A dataset of fine-grained emo- tions. arXiv preprint arXiv:2005.00547. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv...

    2005 arXiv

  5. [2021]

    arXiv preprint arXiv:2109.07506

    Dialogue state tracking with a language model using schema-driven prompting. arXiv preprint arXiv:2109.07506. Young-Jun Lee, Byungsoo Ko, Han-Gyu Kim, Jongh- wan Hyeon, and Ho-Jin Choi. 2024a. Dialogcc: An automated pipeline for creating high-quality multi- modal dialogue data...

    2024 arXiv

  6. [2023]

    In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 14150–14164

    Multimodal persona based generation of comic dialogs. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 14150–14164. Jaewoo Ahn, Yeda Song, Sangdoo Yun, and Gun- hee Kim. 2023. Mpchat: Towards multimodal...

    2023 arXiv

  7. [2024]

    arXiv preprint arXiv:2408.02417

    Infusing emotions into task-oriented dialogue systems: Understanding, management, and genera- tion. arXiv preprint arXiv:2408.02417. Mauajama Firdaus, Hardik Chauhan, Asif Ekbal, and Pushpak Bhattacharyya. 2020. Meisd: A multimodal multi-label emotion, intensity and sentiment ...

    2020 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.