Pith. sign in

REVIEW 3 major objections 2 minor 19 references

A dual-encoder model with asymmetric attention classifies when spoken journal entries contradict their written emotional tone.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-05-07 09:22 UTC

load-bearing objection The paper introduces a novel CADD task and TTS dataset for detecting text-voice dissonance in journaling, but its results may not generalize beyond synthetic speech. the 3 major comments →

arxiv 2604.27517 v1 submitted 2026-04-30 cs.HC

I'm Fine, But My Voice Isn't: Cross-Modal Affective Dissonance Detection for Reflective Journaling

Sumin Lee This is my paper
classification cs.HC
keywords cross-modal affective dissonanceemotion regulationreflective journalingdual-encoder modelasymmetric attentionTTS datasethuman-computer interactionaffective computing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper defines Cross-Modal Affective Dissonance Detection to spot cases where the acoustic delivery of a journal entry reveals different feelings than the text alone. It supplies a controlled TTS dataset that holds sentence content fixed while varying acoustics and trains a model that separates masking, coping, and congruent cases. If the approach holds, users could receive signals about their own emotion regulation during private writing. The authors also measure how far synthetic training falls short on real speech and list concrete steps needed to close that gap.

Core claim

We formalize Cross-Modal Affective Dissonance Detection (CADD) as a directional three-way classification distinguishing Masking (positive text, negative acoustics), Coping (negative text, positive acoustics), and Congruent utterances, grounded in Gross's process model of emotion regulation. We present CADD-Journal, a 1,800-sample TTS dataset with shared-sentence-pool design that isolates acoustic signal from textual content, and DACM, a dual-encoder model with asymmetric cross-modal attention that resolves gradient degeneracy in pooled fusion and reaches macro-F1 0.711. Four-step ablation shows asymmetric attention drives the largest gain while domain-shift tests on naturalistic corpora flag

What carries the argument

DACM, the dual-encoder model that applies asymmetric cross-modal attention to direct gradients between text and acoustic encoders instead of symmetric pooling.

Load-bearing premise

The TTS-generated dataset with fixed sentence content and varied acoustics sufficiently captures the cross-modal affective dissonance that occurs in ordinary human journaling.

What would settle it

Train the same DACM architecture on a corpus of real human voice recordings paired with their original journal text, label the three dissonance classes by multiple annotators, and check whether macro-F1 remains within a few points of 0.711.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Asymmetric attention accounts for the bulk of the performance lift, adding 0.242 to macro-F1 in ablation.
  • The shared-sentence-pool design isolates acoustic variation from lexical content by construction.
  • Zero-shot transfer to three naturalistic speech corpora reveals a clear drop relative to the TTS test set.
  • Two specific requirements for future in-the-wild data collection are stated to reduce the observed domain gap.
  • The ReflectJournal iOS app supplies an immediate deployment vehicle for gathering labeled real-user examples.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the classifier generalizes, journaling apps could surface personal patterns of emotional masking over weeks or months.
  • The same three-class framing could be tested on other paired modalities such as text plus facial video or text plus physiological signals.
  • Collecting the naturalistic corpora the paper calls for would also allow direct comparison of human versus model disagreement on dissonance labels.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The paper formalizes Cross-Modal Affective Dissonance Detection (CADD) as a directional three-class task (Masking: positive text/negative acoustics; Coping: negative text/positive acoustics; Congruent) grounded in Gross's emotion regulation model. It introduces the CADD-Journal dataset (1,800 TTS samples via shared-sentence-pool design to isolate acoustics), the DACM dual-encoder model with asymmetric cross-modal attention (macro-F1 0.711), a four-step ablation attributing +0.242 gain to asymmetric attention and +0.033 to DIM on cross-modal features, a zero-shot domain-gap analysis on naturalistic corpora, and the ReflectJournal iOS app for future data collection.

Significance. If the performance and ablation results generalize beyond TTS data, the work could meaningfully advance affective computing and human-computer interaction by addressing authenticity gaps in digital journaling through cross-modal analysis. The explicit ablation isolating asymmetric attention, the domain-gap quantification with concrete future requirements, and the deployment app are strengths that support reproducibility and extensibility. The shared-sentence-pool dataset design is a clear methodological contribution for controlling textual content.

major comments (3)
  1. [Abstract / Dataset Construction] Abstract and Dataset section: The shared-sentence-pool design is presented as provably isolating acoustic signal from textual content, yet the manuscript provides no human perceptual validation study or inter-rater reliability metrics confirming that the three CADD classes (Masking, Coping, Congruent) are distinguishable by listeners in the TTS data at rates supporting the model's distinctions. Without this, the macro-F1 of 0.711 and ablation gains risk being TTS-specific artifacts rather than measures of real cross-modal affective dissonance.
  2. [Results / Ablation Study] Results and Ablation: The four-step ablation reports asymmetric attention as the dominant driver (+0.242) and DIM effective only on cross-modal features (+0.033), but no error bars, statistical significance tests (e.g., McNemar or paired t-tests), or full training details (learning rates, batch sizes, convergence criteria) are provided. This undermines assessment of whether the reported performance differences are robust or load-bearing for the central claim.
  3. [Evaluation / Domain Gap Analysis] Evaluation section: A substantial zero-shot performance drop is reported on three naturalistic corpora, but the paper does not include a human-recorded validation set or any direct comparison of model predictions against human judgments on real speech. This is load-bearing for the claim that DACM detects genuine cross-modal dissonance, as the TTS training distribution may lack micro-variations, disfluencies, and physiological coupling present in human emotion regulation.
minor comments (2)
  1. [Abstract] The acronym 'DIM' appears in the ablation description without prior expansion in the abstract; define all acronyms at first use and ensure the model architecture diagram or equations clarify its role.
  2. [Abstract / Methods] The abstract states the model 're-solves a gradient degeneracy in pooled fusion' but does not provide the specific equation or derivation showing the degeneracy; include this in the methods section for clarity.

Simulated Author's Rebuttal

3 responses · 1 unresolved

We thank the referee for their constructive and detailed feedback, which identifies key areas for strengthening the validation and robustness of our work on Cross-Modal Affective Dissonance Detection. We address each major comment point by point below, indicating revisions where appropriate.

read point-by-point responses

  1. Referee: [Abstract / Dataset Construction] Abstract and Dataset section: The shared-sentence-pool design is presented as provably isolating acoustic signal from textual content, yet the manuscript provides no human perceptual validation study or inter-rater reliability metrics confirming that the three CADD classes (Masking, Coping, Congruent) are distinguishable by listeners in the TTS data at rates supporting the model's distinctions. Without this, the macro-F1 of 0.711 and ablation gains risk being TTS-specific artifacts rather than measures of real cross-modal affective dissonance.

    Authors: We agree that a human perceptual validation study would provide valuable empirical support for the distinguishability of the CADD classes in the TTS data. The shared-sentence-pool design ensures identical textual content across classes, with differences arising solely from controlled TTS acoustic renderings. To address this concern directly, we will add a small-scale human listening study to the revised manuscript, reporting inter-rater reliability (e.g., Fleiss' kappa) on listeners' ability to identify the intended Masking, Coping, and Congruent classes from the TTS samples. This will help confirm alignment between the dataset construction and human perception. revision: yes

  2. Referee: [Results / Ablation Study] Results and Ablation: The four-step ablation reports asymmetric attention as the dominant driver (+0.242) and DIM effective only on cross-modal features (+0.033), but no error bars, statistical significance tests (e.g., McNemar or paired t-tests), or full training details (learning rates, batch sizes, convergence criteria) are provided. This undermines assessment of whether the reported performance differences are robust or load-bearing for the central claim.

    Authors: We acknowledge that the absence of error bars, statistical significance testing, and complete training details limits the ability to assess the robustness of the ablation results. In the revised manuscript, we will include standard deviations from multiple training runs with different random seeds, apply McNemar's test for pairwise significance between ablation configurations, and add a detailed appendix specifying all hyperparameters including learning rates, batch sizes, optimizer settings, maximum epochs, and convergence criteria based on validation performance. revision: yes

  3. Referee: [Evaluation / Domain Gap Analysis] Evaluation section: A substantial zero-shot performance drop is reported on three naturalistic corpora, but the paper does not include a human-recorded validation set or any direct comparison of model predictions against human judgments on real speech. This is load-bearing for the claim that DACM detects genuine cross-modal dissonance, as the TTS training distribution may lack micro-variations, disfluencies, and physiological coupling present in human emotion regulation.

    Authors: We recognize that direct comparison to human judgments on real speech would strengthen claims about detecting genuine dissonance. No existing human-recorded dataset with CADD-specific labels is available, which is the motivation for developing the ReflectJournal app to enable future in-the-wild collection. The zero-shot analysis already quantifies the domain gap and specifies concrete requirements for future corpora. In the revision, we will expand the discussion to explicitly acknowledge this limitation and detail how the app supports the necessary human validation in subsequent work. revision: partial

standing simulated objections not resolved
  • Absence of a human-recorded validation set with CADD labels and direct model predictions versus human judgments on real speech, as no such labeled naturalistic dataset currently exists and requires new data collection.

Circularity Check

0 steps flagged

No significant circularity; empirical results on constructed dataset are independent measurements

full rationale

The paper defines a new task (CADD) grounded in an external psychological model (Gross), constructs a TTS dataset via shared-sentence-pool design to control for text, trains a dual-encoder model (DACM), reports macro-F1 and ablation deltas on that dataset, and quantifies zero-shot drop on separate naturalistic corpora. None of these steps reduce by construction to the inputs: the F1 score is a measured classification accuracy, the ablation isolates component contributions via standard removal experiments, and the domain-gap analysis uses external data. No self-definitional equations, fitted parameters renamed as predictions, or load-bearing self-citations appear in the provided text. The derivation chain is self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 3 invented entities

Abstract-only view provides no explicit free parameters, axioms, or invented physical entities; new task and model components are introduced as definitional contributions.

invented entities (3)
  • CADD no independent evidence
    purpose: Formalize authenticity gap in digital journaling as directional three-way classification
    New task definition grounded in Gross's process model
  • DACM no independent evidence
    purpose: Dual-encoder model with asymmetric cross-modal attention
    Proposed architecture to address gradient degeneracy in pooled fusion
  • CADD-Journal no independent evidence
    purpose: 1,800-sample TTS dataset with shared-sentence-pool design
    Dataset to isolate acoustic signal from textual content

pith-pipeline@v0.9.0 · 5529 in / 1227 out tokens · 49215 ms · 2026-05-07T09:22:22.029684+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of I'm Fine, But My Voice Isn't: Cross-Modal Affective Dissonance Detection for Reflective Journaling." pith.science (2026). https://pith.science/paper/2604.27517

@misc{pith2026260427517,
  author       = {Pith},
  title        = {Pith review of: I'm Fine, But My Voice Isn't: Cross-Modal Affective Dissonance Detection for Reflective Journaling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2604.27517}},
  note         = {Machine review of arXiv:2604.27517}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Digital journaling creates an authenticity gap: users consciously translate raw emotions into text, often sanitizing narratives even in private writing. We formalize this as Cross-Modal Affective Dissonance Detection (CADD), a directional three-way classification distinguishing Masking (positive text, negative acoustics), Coping (negative text, positive acoustics), and Congruent utterances, grounded in Gross's process model of emotion regulation. We present three further contributions: (i) CADD-Journal, a 1,800-sample TTS dataset with a shared-sentence-pool design that provably isolates acoustic signal from textual content; (ii) DACM, a dual-encoder model with asymmetric cross-modal attention that re-solves a gradient degeneracy in pooled fusion, achieving macro-F1 0.711 - with a four-step ablation demonstrating that asymmetric attention is the dominant driver (+ 0.242) while the DIM is effective only on cross-modal features (+0.033); and (iii) a domain gap quantification: zero-shot evaluation across three naturalistic corpora reveals a substantial gap between TTS-trained models and real speech, and we identify two concrete requirements for future in-the-wild corpus construction. ReflectJournal, a proof-of-concept iOS application, operationalizes the framework and provides a deployment platform for naturalistic data collection.

Figures

Figures reproduced from arXiv: 2604.27517 by Sumin Lee.

Figure 1
Figure 1. Figure 1: DACM architecture. Frozen text (blue, XLM view at source ↗
Figure 2
Figure 2. Figure 2: ReflectJournal: (A/B) recording interface, (C) jour view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references · 19 canonical work pages

  1. [1]

    Chang, Sungbok Lee, and Shrikanth S

    Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N. Chang, Sungbok Lee, and Shrikanth S. Narayanan. 2008. IEMOCAP: Interactive emotional dyadic motion capture database.Language Resources and Evaluation42, 4 (2008), 335–359

    work page 2008

  2. [2]

    Jingyi Chen, Zhimeng Guo, Jiyun Chun, Pichao Wang, Andrew Perrault, and Micha Elsner. 2026. Do audio llms really listen, or just transcribe? measuring lexical vs. acoustic emotion cues reliance. InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 5848–5877

    work page 2026

  3. [3]

    Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, and Jian Wu. 2022. WavLM: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing16, 6 (2022), 1505–1518

    work page 2022

  4. [4]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guil- laume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. InProceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics (ACL). Association for Com...

    work page 2020

  5. [5]

    Kamel, and Fakhri Karray

    Moataz El Ayadi, Mohamed S. Kamel, and Fakhri Karray. 2011. Survey on speech emotion recognition: Features, classification schemes, and databases.Pattern Recognition44, 3 (2011), 572–587

    work page 2011

  6. [6]

    ElevenLabs. 2025. Eleven v3: Generative Text-to-Speech. https://elevenlabs.io/ docs/overview/models#eleven-v3 Accessed: April 2026

    work page 2025

  7. [7]

    Scherer, Björn W

    Florian Eyben, Klaus R. Scherer, Björn W. Schuller, Johan Sundberg, Elisabeth André, Carlos Busso, Laurence Y. Devillers, Julien Epps, Petri Laukka, Shrikanth S. Narayanan, and Khiet P. Truong. 2016. The Geneva minimalistic acoustic param- eter set (GeMAPS) for voice research and affective computing.IEEE Transactions on Affective Computing7, 2 (2016), 190–202

    work page 2016

  8. [8]

    James J. Gross. 1998. The emerging field of emotion regulation: An integrative review.Review of General Psychology2, 3 (1998), 271–299

    work page 1998

  9. [9]

    Larson and Robert L

    Dale G. Larson and Robert L. Chastain. 1990. Self-concealment: Conceptual- ization, measurement, and health implications.Journal of Social and Clinical Psychology9, 4 (1990), 439–455

    work page 1990

  10. [10]

    Leary and Robin M

    Mark R. Leary and Robin M. Kowalski. 1990. Impression management: A liter- ature review and two-component model.Psychological Bulletin107, 1 (1990), 34–47

    work page 1990

  11. [11]

    Sijie Mai, Ying Zeng, Shuangjia Zheng, and Haifeng Hu. 2022. Hybrid contrastive learning of tri-modal representation for multimodal sentiment analysis.IEEE transactions on affective computing14, 3 (2022), 2276–2289

    work page 2022

  12. [12]

    Pennebaker

    James W. Pennebaker. 1997. Writing about emotional experiences as a therapeutic process.Psychological Science8, 3 (1997), 162–166

    work page 1997

  13. [13]

    Joshua M. Smyth. 1998. Written emotional expression: Effect sizes, outcome types, and moderating variables.Journal of Consulting and Clinical Psychology 66, 1 (1998), 174–184

    work page 1998

  14. [14]

    Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Multimodal transformer for unaligned multimodal language sequences. InProceedings of the 57th annual meeting of the association for computational linguistics. 6558–6569

    work page 2019

  15. [15]

    Jiangnan Yang, Ying Luo, Bing Zeng, and Soujanya Poria. 2023. ConFEDE: Contrastive feature decomposition for emotion recognition in conversations. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (ACL), Volume 1: Long Papers. Association for Computational Linguistics, Toronto, Canada, 7184–7197

    work page 2023

  16. [16]

    Lin, Andy T

    Shu Wen Yang, Po Han Chi, Yung Sung Chuang, Cheng I.Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan Ting Lin, Tzu Hsien Huang, Wei Cheng Tseng, Ko Tik Lee, Da Rong Liu, Zili Huang, Shuyan Dong, Shang Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung Yi Lee. 2021. SUPERB: Speech processing Universal PERformance Be...

    work page doi:10.21437/interspeech.2021-1775 2021

  17. [17]

    Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou, and Kaisheng Yang. 2020. CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of Modality. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). Association for Computational Linguistics, Online, 3718–3727

    work page 2020

  18. [18]

    Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Copenhagen, Denmark, 1103–1114

    work page 2017

  19. [19]

    AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis- Philippe Morency. 2018. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL). Association for Computational Linguistics, Melbourne, Aust...

    work page 2018