pith. sign in

arxiv: 2606.30312 · v1 · pith:NDDZTFBMnew · submitted 2026-06-29 · 💻 cs.CL

DialogPII: A multilingual dataset of synthetic dialog transcripts to detect personal information

Roland Roller , Vera Czehmann , Derya Erman , Luke Flanagan , Ibrahim Baroud , Fr\'ed\'eric Blain , Viviana Cotik , Eletta Giusto
show 8 more authors
Akhil Juneja Mariana Neves Maria S{\l}owi\'nska Christine Hovhannisyan Aaron Louis Eidt Lisa Raithel Sebastian M\"oller Maija Poikela
This is my paper

Pith reviewed 2026-06-30 06:21 UTC · model grok-4.3

classification 💻 cs.CL
keywords personal information detectionde-identificationmultilingual datasetsynthetic dialogsnamed entity recognitionprivacy protectionspeech transcriptionconversational data
0
0 comments X

The pith

A new multilingual dataset of synthetic dialogs supports the detection and removal of personal information in conversational data.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces DialogPII to address privacy barriers that prevent sharing conversational data from domains such as healthcare. It supplies synthetic dialogs across eight scenarios, nineteen entity types, and eleven languages so that de-identification systems can be developed and tested without using real private records. The dialogs were produced with large language models, manually checked and localized, then rendered as speech and transcribed to create paired written and audio-derived resources. Baseline sequence-labeling models for named entity recognition are released together with validation checks on annotation quality and translation fidelity. This resource directly enables training and benchmarking of tools that locate and remove sensitive details while preserving the utility of the conversations.

Core claim

DialogPII consists of synthetic dialogs in eight interaction scenarios that include emergency calls, medical anamnesis interviews, therapy sessions, insurance communication, customer support, clinical interviews, police reports, and group therapy discussions. It covers nineteen entity types across eleven languages and supplies both written transcripts and speech-derived versions obtained through text-to-speech synthesis followed by automatic transcription and manual annotation correction. The paper also releases baseline multilingual named entity recognition models along with inter-annotator agreement, translation quality, and benchmark results on transformer sequence labeling.

What carries the argument

The DialogPII dataset of aligned written and speech-derived dialog transcripts annotated for personal information entities across multiple languages and scenarios.

If this is right

  • De-identification systems can be trained on conversational data from healthcare, legal, and customer-service domains without exposing actual private records.
  • The speech-derived transcripts allow direct measurement of how transcription noise affects entity detection performance.
  • Localization to country- and city-specific contexts supports evaluation of de-identification tools for international use.
  • The released baseline models provide an initial performance reference for future multilingual and multimodal de-identification work.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Synthetic datasets of this kind could lower the cost and ethical overhead of creating training resources for privacy tasks in additional sensitive domains.
  • The projection-and-correction annotation method may be reusable for extending the resource to further languages or entity types.
  • Performance gaps between written and transcribed versions could guide improvements in handling automatic speech recognition output for privacy applications.

Load-bearing premise

The semi-automatically generated synthetic dialogs are sufficiently plausible, diverse, and representative of real sensitive conversations to serve as effective training and evaluation data.

What would settle it

Models trained on DialogPII achieve substantially lower precision and recall on a test set of real de-identified conversations from the same domains than models trained directly on those real conversations.

read the original abstract

Conversational data collected in domains such as healthcare or social sciences is a valuable resource for research and automated analysis. However, responsible data sharing requires the detection and removal of personally identifiable and sensitive information to protect individual privacy. To support the development and evaluation of automatic de-identification systems, we present DialogPII, a multilingual dataset of synthetic dialogs and speech-derived transcripts for personal information detection. DialogPII covers eight interaction scenarios (emergency calls, medical anamnesis interviews, therapy sessions, insurance communication, customer support, clinical interviews regarding an AI-supported dashboard, police reports, and group therapy discussions), 19 entity types, and 11 languages (English, Arabic, Finnish, French, German, Hindi, Italian, Polish, Portuguese, Spanish, and Turkish). Dialogs were generated semi-automatically using large language models, manually curated for plausibility and diversity, and localized to country- and city-specific contexts. All dialogs were additionally converted to speech via text-to-speech synthesis, transcribed with Whisper, and annotated through automatic projection and manual correction, yielding aligned written and speech-derived resources across all languages. We further release baseline multilingual named entity recognition models and provide technical validation through inter-annotator agreement analysis, translation quality evaluation, annotation projection assessment, and benchmark experiments with transformer-based sequence labeling models.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The paper presents DialogPII, a multilingual dataset of synthetic dialogs and speech-derived transcripts for personal information detection. It covers eight interaction scenarios, 19 entity types, and 11 languages. Dialogs are generated semi-automatically with LLMs, manually curated and localized, converted to speech via TTS, transcribed with Whisper, and annotated via projection with manual correction. The work also releases baseline multilingual NER models and reports technical validations including inter-annotator agreement, translation quality, annotation projection assessment, and benchmark experiments with transformer sequence labeling models.

Significance. If the synthetic dialogs prove representative of real sensitive conversations, the resource would meaningfully support development of de-identification systems for conversational data in healthcare, social sciences, and other domains, with particular value in its multilingual scope and aligned text-speech modalities. The release of baseline models and multiple internal validation steps (IAA, WER, benchmarks) are positive contributions to reproducibility.

major comments (1)
  1. [Abstract, §3] Abstract and §3: The described validation pipeline addresses internal consistency (IAA, translation quality, Whisper WER, annotation projection) and reports benchmark experiments, but provides no quantitative external validation such as comparisons of PII entity co-occurrence statistics, dialog act sequences, or linguistic markers against any real (even de-identified) conversational corpora in the target domains or languages. This is load-bearing for the central claim that the curated synthetic dialogs are sufficiently plausible, diverse, and representative to serve as effective training and evaluation data for de-identification systems.
minor comments (2)
  1. [Abstract] The abstract would benefit from stating the total number of dialogs, average length per language/scenario, and exact train/dev/test splits to allow immediate assessment of scale.
  2. Clarify in the methods whether the dataset and baseline models will be released with persistent identifiers and access instructions, as this directly affects usability.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback highlighting the importance of validating the representativeness of our synthetic dialogs. We address the major comment below and have revised the manuscript accordingly where feasible.

read point-by-point responses

  1. Referee: [Abstract, §3] Abstract and §3: The described validation pipeline addresses internal consistency (IAA, translation quality, Whisper WER, annotation projection) and reports benchmark experiments, but provides no quantitative external validation such as comparisons of PII entity co-occurrence statistics, dialog act sequences, or linguistic markers against any real (even de-identified) conversational corpora in the target domains or languages. This is load-bearing for the central claim that the curated synthetic dialogs are sufficiently plausible, diverse, and representative to serve as effective training and evaluation data for de-identification systems.

    Authors: We acknowledge that the manuscript does not include quantitative external validation against real conversational corpora, as no such direct comparisons (e.g., entity co-occurrence or dialog act sequences) are reported. Obtaining access to real de-identified PII-containing dialog data from sensitive domains like healthcare is severely constrained by privacy regulations and ethical review processes, even for research purposes; publicly available de-identified corpora in the target languages and scenarios are not readily accessible for this type of analysis. Our validation strategy instead emphasizes expert manual curation for plausibility and diversity, combined with the reported internal metrics and benchmarks. We have added an expanded discussion of this limitation, the rationale for the synthetic approach, and qualitative evidence from the curation process to §3 and a new Limitations section in the revised manuscript. revision: partial

Circularity Check

0 steps flagged

No circularity: dataset release with direct construction and empirical validation

full rationale

The paper presents a new multilingual synthetic dialog dataset for PII detection, generated semi-automatically via LLMs, manually curated, localized, TTS-synthesized, transcribed with Whisper, and annotated. No mathematical derivations, predictions, fitted parameters, or uniqueness theorems are claimed. Baseline NER models are trained on the released data in the standard supervised manner. Validation steps (IAA, translation quality, projection assessment) are independent empirical checks with no reduction to self-defined inputs or self-citation chains. The contribution is self-contained as a data resource without any load-bearing circular steps.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Dataset release paper with no free parameters, axioms, or invented entities; the contribution rests on the assumption that synthetic data generation plus curation produces usable labeled resources.

pith-pipeline@v0.9.1-grok · 5834 in / 1136 out tokens · 26796 ms · 2026-06-30T06:21:16.729794+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 4 canonical work pages

  1. [1]

    Journal of the American Medical Informatics Association14(5), 550–563 (2007)

    Uzuner, ¨O., Luo, Y., Szolovits, P.: Evaluating the state-of-the-art in automatic de-identification. Journal of the American Medical Informatics Association14(5), 550–563 (2007)

    2007

  2. [2]

    Journal of biomedical informatics 58, 20–29 (2015)

    Stubbs, A., Uzuner, ¨O.: Annotating longitudinal clinical narratives for de- identification: The 2014 i2b2/UTHealth corpus. Journal of biomedical informatics 58, 20–29 (2015)

    2014

  3. [3]

    Journal of the American Medical Informat- ics Association24(3), 596–606 (2017)

    Dernoncourt, F., Lee, J.Y., Uzuner, O., Szolovits, P.: De-identification of patient notes with recurrent neural networks. Journal of the American Medical Informat- ics Association24(3), 596–606 (2017)

    2017

  4. [4]

    Scientific Data10(1), 207 (2023)

    Richter-Pechanski, P., Wiesenbach, P., Schwab, D.M., Kiriakou, C., He, M., Allers, M.M., Tiefenbacher, A.S., Kunz, N., Martynova, A., Spiller, N.,et al.: A distributable German clinical corpus containing cardiovascular clinical routine doctor’s letters. Scientific Data10(1), 207 (2023)

    2023

  5. [5]

    In: German Medical Data Sciences 2024: Health–Thinking, Researching and Acting Together

    Lohr, C., Matthies, F., Faller, J., Modersohn, L., Riedel, A., Hahn, U., Kiser, R., Boeker, M., Meineke, F.: De-identifying grascco–a pilot study for the de- identification of the german medical text project (gemtex) corpus. In: German Medical Data Sciences 2024: Health–Thinking, Researching and Acting Together. Proceedings of the 69th Annual Meeting of t...

    2024

  6. [6]

    In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Industry Papers), pp

    Cohn, I., Laish, I., Beryozkin, G., Li, G., Shafran, I., Szpektor, I., Hartman, T., Hassidim, A., Matias, Y.: Audio de-identification-a new entity recognition task. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Industry Papers), pp. 197–204 (2019)

    2019

  7. [7]

    In: [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, vol

    Godfrey, J.J., Holliman, E.C., McDaniel, J.: SWITCHBOARD: Telephone speech corpus for research and development. In: [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 1, pp. 517–520 (1992). IEEE

    1992

  8. [8]

    In: LREC, vol

    Cieri, C., Miller, D., Walker, K.: The Fisher corpus: A resource for the next generations of speech-to-text. In: LREC, vol. 4, pp. 69–71 (2004)

    2004

  9. [9]

    In: 2022 International Joint Conference on Neural Networks (IJCNN), pp

    Baril, G., Cardinal, P., Koerich, A.L.: Named entity recognition for audio de-identification. In: 2022 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2022). IEEE

    2022

  10. [10]

    In: Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020), pp

    Kaplan, M.: May i ask who’s calling? named entity recognition on call center 25 transcripts for privacy law compliance. In: Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020), pp. 1–6 (2020)

    2020

  11. [11]

    In: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp

    Gouvˆ ea, E., Dadgar, A., Jalalvand, S., Chengalvarayan, R., Jayakumar, B., Price, R., Ruiz, N., McGovern, J., Bangalore, S., Stern, B.: Trustera: A live conversa- tion redaction system. In: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5 (2023). IEEE

    2023

  12. [12]

    In: ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp

    Aggazzotti, C., Garg, A., Cai, Z., Andrews, N.: Content anonymization for pri- vacy in long-form audio. In: ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 13412–13416 (2026). IEEE

    2026

  13. [13]

    In: Proceedings of the Workshop on Compu- tational Approaches to Language Data Pseudonymization (CALD-pseudo 2024), pp

    Gardiner, S., Habib, T., Humphreys, K., Azizi, M., Mailhot, F., Paling, A., Thomas, P., Zhang, N.: Data anonymization for privacy-preserving large language model fine-tuning on call transcripts. In: Proceedings of the Workshop on Compu- tational Approaches to Language Data Pseudonymization (CALD-pseudo 2024), pp. 64–75 (2024)

    2024

  14. [14]

    In: Proceedings of the Second Workshop in South East Asian Language Processing, pp

    Abdjul, R.N., Lestari, D.P., Purwarianti, A., Mawalim, C.O., Sakti, S., Unoki, M.: Indonesian speech content de-identification in low resource transcripts. In: Proceedings of the Second Workshop in South East Asian Language Processing, pp. 63–71 (2025)

    2025

  15. [15]

    In: Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pp

    Malmasi, S., Fang, A., Fetahu, B., Kar, S., Rokhlenko, O.: SemEval-2022 task 11: Multilingual complex named entity recognition (MultiCoNER). In: Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pp. 1412–1437 (2022)

    2022

  16. [16]

    In: Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pp

    Fetahu, B., Kar, S., Chen, Z., Rokhlenko, O., Malmasi, S.: SemEval-2023 task 2: Fine-grained multilingual named entity recognition (MultiCoNER 2). In: Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pp. 2247–2265 (2023)

    2023

  17. [17]

    arXiv preprint arXiv:2603.08879 (2026)

    Baroud, I., Otto, C., Czehmann, V., Hovhannisyan, C., Raithel, L., M¨ oller, S., Roller, R.: MultiGraSCCo: A Multilingual Anonymization Benchmark with Annotations of Personal Identifiers. arXiv preprint arXiv:2603.08879 (2026)

    work page arXiv 2026

  18. [18]

    In: 2025 IEEE International Conference on Data Mining Workshops (ICDMW), pp

    Meena, B., Skubisz, J., Rajgarhia, H., Dave, N., Ganesh, K., Dalmia, S., Mukherji, A., Sundarababu, V., Pospelova, O.: Scalable multilingual PII annotation for responsible AI in LLMs. In: 2025 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 367–375 (2025). IEEE

    2025

  19. [19]

    Qualitative research15(5), 616–632 (2015)

    Saunders, B., Kitzinger, J., Kitzinger, C.: Anonymising interview data: Challenges and compromise in practice. Qualitative research15(5), 616–632 (2015)

    2015

  20. [20]

    In: Pro- ceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp

    Hiebel, N., Ferret, O., Fort, K., N´ ev´ eol, A.: Can synthetic text help clinical 26 named entity recognition? a study of electronic health records in French. In: Pro- ceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2320–2338 (2023)

    2023

  21. [21]

    In: Ku, L.-W., Martins, A., Srikumar, V

    Long, L., Wang, R., Xiao, R., Zhao, J., Ding, X., Chen, G., Wang, H.: On LLMs- Driven Synthetic Data Generation, Curation, and Evaluation: A Survey. In: Ku, L.-W., Martins, A., Srikumar, V. (eds.) Findings of the Association for Com- putational Linguistics: ACL 2024. Association for Computational Linguistics, Bangkok, Thailand (2024)

    2024

  22. [22]

    Computational Linguistics48(4), 1053–1101 (2022)

    Pil´ an, I., Lison, P., Øvrelid, L., Papadopoulou, A., S´ anchez, D., Batet, M.: The text anonymization benchmark (tab): A dedicated corpus and evaluation framework for text anonymization. Computational Linguistics48(4), 1053–1101 (2022)

    2022

  23. [23]

    In: Zhao, D

    Klie, J.-C., Bugert, M., Boullosa, B., Castilho, R., Gurevych, I.: The INCEp- TION Platform: Machine-Assisted and Knowledge-Oriented Interactive Anno- tation. In: Zhao, D. (ed.) Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations, pp. 5–9. Association for Computational Linguistics, Santa Fe, New Mexico (2018)

    2018

  24. [24]

    INTERSPEECH 2023 (2023)

    Bain, M., Huh, J., Han, T., Zisserman, A.: WhisperX: Time-Accurate Speech Transcription of Long-Form Audio. INTERSPEECH 2023 (2023)

    2023

  25. [25]

    audio 2.1 speaker diarization pipeline: principle, bench- mark, and recipe

    Bredin, H.: pyannote. audio 2.1 speaker diarization pipeline: principle, bench- mark, and recipe. In: 24th Interspeech Conference (INTERSPEECH 2023), pp. 1983–1987 (2023). ISCA

    2023

  26. [26]

    In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp

    Warner, B., Chaffin, A., Clavi´ e, B., Weller, O., Hallstr¨ om, O., Taghadouini, S., Gallagher, A., Biswas, R., Ladhak, F., Aarsen, T.,et al.: Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long con- text finetuning and inference. In: Proceedings of the 63rd Annual Meeting of the Association for Computation...

    2025

  27. [27]

    arXiv preprint arXiv:2509.06888 , year=

    Marone, M., Weller, O., Fleshman, W., Yang, E., Lawrie, D., Durme, B.V.: mmBERT: A Modern Multilingual Encoder with Annealed Language Learning (2025). https://arxiv.org/abs/2509.06888

    work page arXiv 2025

  28. [28]

    arXiv preprint arXiv:2011.06993 (2020)

    Schweter, S., Akbik, A.: Flert: Document-level features for named entity recog- nition. arXiv preprint arXiv:2011.06993 (2020)

    work page arXiv 2011

  29. [29]

    Technical report, Linguistic Data Consortium, Philadelphia (1996)

    Canavan, A., Zipperlen, G.: CALLFRIEND American English-Non-Southern Dialect LDC96S46. Technical report, Linguistic Data Consortium, Philadelphia (1996)

    1996

  30. [30]

    TalkBank (2006)

    Yaeger-Dror, M., Linguistic Data Consortium: CABank English CallFriend 27 Northern US Corpus. TalkBank (2006). https://doi.org/10.21415/T5B61M . https://doi.org/10.21415/T5B61M

    work page doi:10.21415/t5b61m 2006

  31. [31]

    In: German Medi- cal Data Sciences 2022–Future Medicine: More Precise, More Integrative, More Sustainable!, pp

    Modersohn, L., Schulz, S., Lohr, C., Hahn, U.: GRASCCO—the first publicly shareable, multiply-alienated german clinical text corpus. In: German Medi- cal Data Sciences 2022–Future Medicine: More Precise, More Integrative, More Sustainable!, pp. 66–72. IOS Press, ??? (2022)

    2022

  32. [32]

    Journal of Medical Internet Research 28, 83195 (2026) 28

    Sassi, Z., Eickmann, S., Roller, R., Osmanodja, B., Spencker, J.J., ¨Omero˘ glu, ¨O.E., Burchardt, A., Hahn, M., Dabrock, P., M¨ oller, S.,et al.: Human-AI Inter- action in Kidney Transplant Decision Support Systems: Qualitative Study of Patient and Support Person Expectations. Journal of Medical Internet Research 28, 83195 (2026) 28

    2026