Pith. sign in

REVIEW 2 major objections 5 minor 33 references

Joint training on speech and unpaired text lets an end-to-end spoken dialogue state tracker generalize to new domains without any spoken training data from them.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 19:44 UTC pith:HSSQV6XU

load-bearing objection Useful, well-ablated result on adapting spoken DST with unpaired text — but the 'cross-domain' claim outruns the experiments, which shift datasets rather than hold out domains. the 2 major comments →

arxiv 2511.22503 v2 pith:HSSQV6XU submitted 2025-11-27 cs.CL cs.SDeess.AS

Joint Speech and Text Training for LLM-Based End-to-End Spoken Dialogue State Tracking

Katia Vendrame , Bolaji Yusuf , Santosh Kesiraju , \v{S}imon Sedl\'a\v{c}ek , Old\v{r}ich Plchot , Jan \v{C}ernock\'y This is my paper
classification cs.CL cs.SDeess.AS
keywords spoken dialogue state trackingjoint speech-text trainingunpaired text datacross-domain generalizationLLMLoRASpokenWOZMultiWOZ
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

End-to-end spoken dialogue state tracking usually needs paired speech-and-labels data for every domain it is deployed on. This paper argues that the expensive speech side can be partly replaced with cheap text: by training an LLM-based speech DST model jointly on speech from a source domain and unpaired written DST data from target domains, the model learns to track dialogue states in the target domains without ever hearing them. On SpokenWOZ and MultiWOZ, joint speech-plus-text training substantially improves target-domain joint goal accuracy—recovering up to 79% of the gap to a fully supervised speech model in one configuration—and the text encoder used during training is discarded at inference, so there is no added cost at test time. The result matters because spoken DST annotation is scarce, while textual DST data is abundant across domains.

Core claim

The central discovery is that a shared connector and LoRA-tuned LLM can absorb dialogue-state knowledge from written DST data and apply it to speech input. Feeding text user turns through a text encoder into the same connector and LLM used for speech, and training with three cross-entropy losses (speech DST, text DST, text DST from the transcript), closes a large fraction of the joint-goal-accuracy gap to full speech supervision—64.7% on MultiWOZ with Gemma-3-1B and 79% on SpokenWOZ with OLMo-1B—even when target text is a small part of a larger corpus. Ablations show the text encoder itself contributes beyond mere LM exposure to target states.

What carries the argument

A text-encoding branch added to the E2E speech-to-DST model: written user input is tokenized, passed through a Transformer text encoder, then through the same connector and LoRA-adapted LLM that process speech embeddings. The connector and LoRA layers are shared across modalities, so text-DST gradients reshape the shared projection speech also uses; at inference the text encoder is dropped, leaving the original speech pipeline. Three cross-entropy losses (speech DST, text DST, text DST from the speech transcription) are summed each step, with the text loss weight as a tunable knob.

Load-bearing premise

The experiments treat SpokenWOZ and MultiWOZ as source/target domains, but the two datasets share many slot keys and, in several conditions, overlapping values; MultiWOZ's train set is also from a different city than its test set. If the gains come from recognizing already-seen keys/values rather than learning genuinely novel target-domain slots, the 'cross-domain generalization' claim is weaker than presented.

What would settle it

Take a source speech training set covering only hotel/restaurant slots and a target text training set covering only train/attraction slots with disjoint value sets and no city overlap with the source. If adding the target text does not improve joint goal accuracy on the target spoken test set over the no-text baseline, the paper's cross-domain claim is refuted. The same test run in both directions (SW source/MW target and MW source/SW target) with slot-disjoint splits would isolate whether the transfer mechanism actually handles novel domains.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Training spoken DST models on new domains no longer requires collecting spoken DST data for those domains; text data alone can transfer the domain knowledge.
  • The method closes a large, and in the 12B-parameter case almost complete, fraction of the performance gap to full speech supervision on the target domain.
  • The added text encoder is discarded at inference, so cross-domain capability is obtained without any increase in inference cost.
  • Text-loss weighting gives a practical control to balance source-domain and target-domain accuracy according to deployment needs.
  • Text-only DST techniques such as slot augmentation and paraphrasing, which are hard to apply to speech, can now indirectly benefit spoken DST through the joint text branch.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The results suggest the main bottleneck for cross-domain spoken DST may be the LLM's knowledge of target-domain ontologies and slot values rather than the speech encoder's ability to recognize new words; text supplies that ontology cheaply.
  • A stricter test would use source and target domains with disjoint slot keys and value sets; the current datasets share many slot names and, in some settings, city-dependent values, so the headline 'cross-domain' claim may overstate true novelty.
  • If the mechanism generalizes, it implies that the size and diversity of the text-only corpus matters more than the amount of unlabeled speech, opening a path to zero-shot spoken DST from large text dialogue collections.
  • One testable extension: measure JGA as a function of target-text volume to see whether gains saturate, which would reveal how much text is needed to match a given speech-data budget.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper addresses data scarcity in end-to-end spoken dialogue state tracking by proposing joint training with unpaired textual DST data. The method extends the E2E speech-to-DST framework of [7] with a text encoder that is used only during training; the connector and LoRA parameters are shared between speech and text pipelines. Experiments on SpokenWOZ and Speech-aware MultiWOZ show that adding text from the target dataset to speech training improves target-domain joint goal accuracy, including when the target text is mixed with a larger corpus, and the paper ablates the text encoder, sweeps the text loss weight, and reports results with four language models. The central advertised result is that spoken DST can be performed on domains for which only textual training data is available.

Significance. If the central claim holds, the method is practically valuable: it would reduce the need to collect paired speech-DST data for every domain, leveraging abundant text DST corpora. The paper has concrete strengths: it uses two public benchmarks and the standard JGA metric, builds on a strong published baseline, reports an oracle topline, an ablation without the text encoder, a loss-weight sweep, and multi-LLM scaling results, and is transparent about the city-mismatch between MultiWOZ train and test. The main weakness is that the experiments do not actually test held-out domains, so the significance is currently conditional on an additional experiment.

major comments (2)
  1. [§3.1, Tables 1–3] The central claim is not established by the experimental design. 'Cross-domain' is operationalized as training speech on one dataset and text on another (SpokenWOZ and MultiWOZ), but both are multi-domain benchmarks with largely overlapping domain ontologies. No condition holds the ontology fixed and tests on spoken data from domains never seen in either speech or text (e.g., speech on hotel+restaurant, text on train+attraction, spoken evaluation on train/attraction). The observed gains could reflect adaptation to the target dataset's schema, JSON formatting, and shared slot keys rather than generalization to novel domains. A disjoint-domain experiment is needed before the §4 conclusion is warranted.
  2. [§3.6] The slot-recall analysis in this section supports the concern above. On MultiWOZ, where the text training set contains different slot values (different city) than the test set, joint training improves key recall by 31.8–41.5% but value recall by only 2.6–19.2%. On SpokenWOZ, where values overlap, value recall improves by 36.0–58%. This asymmetry suggests the method transfers the target schema and surface patterns more than the ability to extract novel slot values from speech. The paper should test and report value-level generalization to held-out domains/values rather than aggregate JGA alone.
minor comments (5)
  1. [All tables] Every table entry is a single run with no error bars, seeds, or significance tests. Several comparisons are small (e.g., Table 1 A1 vs A2 on SW: 36.1 vs 36.3; B1 vs B4 on SW: 20.5 vs 20.5). Reporting variance across at least a few seeds would strengthen the quantitative claims.
  2. [Figure 2] The legend notation 'X, Y→Z' is defined for paired speech data from X and unpaired text from Y, tested on Z, but the listed entries such as 'SW, MW→MW' and 'MW, SW→SW' are easy to misread as dataset lists; a short example or clearer variable names would help.
  3. [Title] The title contains a typo: 'STA TE TRACKING' should be 'STATE TRACKING'.
  4. [§3.1] The paper alternates between 'Speech-aware MultiWOZ' and 'Speech-Aware MultiWOZ'; use one form consistently.
  5. [§3.3, Table 1] The toplines C1/C2/C4 are clearly labeled as such, but the prose around them (e.g., C1 description) could more explicitly separate oracle behavior from the operational generalization result A2/B2.

Circularity Check

0 steps flagged

No significant circularity: target-domain text is used as training data by design, and all reported gains are measured on held-out speech validation/test sets.

full rationale

The paper's central claim is empirical: adding unpaired target-domain text to speech+text joint training improves held-out spoken DST. The target text is part of the training data by design, not a fitted parameter; evaluation is on validation/test splits not used for early stopping or checkpoint selection. The text loss weight (0.5) is a hand-set hyperparameter with a validation-based sweep in Fig. 2, which is conventional. The self-cited baseline [7] supplies the architectural scaffold and prior baseline numbers, but the paper re-implements and compares against it rather than deriving the main result from it; no uniqueness theorem or ansatz is imported from [7] to force the conclusion. The topline experiments C1/C2/C4 explicitly include target text and are labeled as oracle/topline, not as evidence for the unsupervised claim. The skeptical concern that 'domain' is operationalized as dataset rather than disjoint ontology slot sets is a threat to external validity/interpretation of the cross-domain claim, not circularity: the inference from training text to test speech is not equivalent to the input by construction. No parameter is fitted to the reported test metrics, so no circular step can be exhibited.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The paper introduces a train-only text encoder, which is a standard architectural component rather than a postulated entity. Its contribution is empirically validated by the Table 2 ablation, so no invented-entity ledger entries are needed. The main assumptions are about the validity of the cross-dataset proxy and the transfer mechanism.

free parameters (2)
  • text loss weight λ = 0.5 (default; swept in Fig. 2)
    The cross-entropy loss on unpaired text batches is multiplied by 0.5 by default. Fig. 2 sweeps this value and shows a trade-off, so the qualitative conclusion is not tied to one value, but the absolute JGA numbers depend on this choice.
  • LoRA rank/alpha = 32/32
    Hyperparameters chosen by hand following the prior baseline [7]; no sensitivity analysis is reported, so the central results could depend on this unexamined choice.
axioms (4)
  • domain assumption Transfer between SpokenWOZ and MultiWOZ datasets is a valid proxy for cross-domain generalization.
    The paper never tests within-dataset domain splits; the entire empirical case rests on SW↔MW transfer. If the gains come from dataset overlap (city, slot keys), the generality of the cross-domain claim weakens. This is set up in §3.1.
  • domain assumption Sharing connector and LoRA between speech and text encoders transfers textual DST knowledge to the speech modality, despite asymmetric encoder outputs (text bypasses the conv layers).
    The method's mechanism depends on this compatibility (§2.2 and Fig. 1c); it is only indirectly supported by the Table 2 ablation, not by any direct analysis of the embedding spaces.
  • domain assumption External resources (WavLM, Whisper re-transcriptions, Fisher/Librispeech/CommonVoice/VoxPopuli for ASR pretraining, MultiWOZ evaluation script) are sufficiently accurate and unbiased.
    The pipeline's inputs and metric come from these external resources (§3.1–3.2); the paper does not validate their error characteristics or potential contamination with the test sets.
  • domain assumption JGA with fuzzy matching against the ontology is the correct and comparable measure of DST quality.
    Evaluation uses the MultiWOZ evaluation script [25]; this is standard practice in the DST literature and not introduced by this paper.

pith-pipeline@v1.3.0-alltime-deepseek · 8154 in / 15956 out tokens · 135701 ms · 2026-08-03T19:44:39.683783+00:00 · methodology

0 comments
read the original abstract

End-to-end spoken dialogue state tracking (DST) is made difficult by the tandem of having to handle speech input and data scarcity. Combining speech foundation encoders and large language models has been proposed in recent work as to alleviate some of this difficulty. Although this approach has been shown to result in strong spoken DST models, achieving state-of-the-art performance in realistic multi-turn DST, it struggles to generalize across domains and requires annotated spoken DST training data for each domain of interest. However, collecting such data for every target domain is both costly and difficult. Noting that textual DST data is more easily obtained for various domains, in this work, we propose jointly training on available spoken DST data and written textual data from other domains as a way to achieve cross-domain generalization. We conduct experiments which show the efficacy of our proposed method for getting good cross-domain DST performance without relying on spoken training data from the target domains.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 4 linked inside Pith

  1. [1]

    Yes, it should leave on Sunday. It should leave after 13:45

    INTRODUCTION Task-oriented dialogue (ToD) systems are conversational agents that assist users in achieving certain defined goals such as making reser- vations, locating attractions, requesting information, etc. As the di- alogues typically comprise multiple user-agent turns in unstructured natural language, a key component of ToD systems is dialogue state...

  2. [2]

    Everybody has a master is out here oh yeah well it seems […]

    METHODS Our DST system is based on the E2E framework from [7], which uses a model comprising a speech encoder, a large language model and a connector module used to transform the outputs of the former 1Note that by unpaired, here and throughout the paper, we mean DST data with no spoken utterances as opposed to text data with no DST labels. arXiv:2511.225...

    Pith/arXiv arXiv 2025

  3. [3]

    Datasets and metrics We conduct our experiments on two spoken dialogue state track- ing datasets: SpokenWOZ (SW) [9] and Speech-aware MultiWOZ (MW) [18]

    EXPERIMENTS 3.1. Datasets and metrics We conduct our experiments on two spoken dialogue state track- ing datasets: SpokenWOZ (SW) [9] and Speech-aware MultiWOZ (MW) [18]. In most of our experiments, we utilize a training set from one of them as the paired speech-text data and the other as unpaired text data, e.g., training on a combination of SpokenWOZ sp...

  4. [4]

    Our method entails augmenting an LLM-based spoken DST model with a text encoder which allows the model to be trained on arbitrary textual DST data

    CONCLUSIONS In this paper, we have proposed a joint speech and text training method for end-to-end spoken dialogue state tracking. Our method entails augmenting an LLM-based spoken DST model with a text encoder which allows the model to be trained on arbitrary textual DST data. We show empirically on the SpokenWOZ and MultiWOZ datasets that this method is...

  5. [5]

    DSTC-11: Speech aware task- oriented dialog modeling track,

    Hagen Soltau, Izhak Shafran, Mingqiu Wang, Abhinav Ras- togi, Wei Han, and Yuan Cao, “DSTC-11: Speech aware task- oriented dialog modeling track,” inProceedings of the Eleventh Dialog System Technology Challenge, Prague, Czech Repub- lic, Sept. 2023, pp. 226–234

    2023

  6. [6]

    Speech-aware multi-domain dialogue state generation with ASR error correction modules,

    Ridong Jiang et al., “Speech-aware multi-domain dialogue state generation with ASR error correction modules,” inPro- ceedings of the Eleventh Dialog System Technology Challenge, Prague, Czech Republic, Sept. 2023, pp. 105–112, Association for Computational Linguistics

    2023

  7. [7]

    Adapting text-based dialogue state tracker for spoken dialogues,

    Jaeseok Yoon, Seunghyun Hwang, Han Ran, Jeong-Uk Bang, and Kee-Eung Kim, “Adapting text-based dialogue state tracker for spoken dialogues,” inProceedings of the Eleventh Dialog System Technology Challenge, Prague, Czech Repub- lic, Sept. 2023, pp. 81–88, Association for Computational Lin- guistics

    2023

  8. [8]

    Integrating dialog history into end-to- end spoken language understanding systems,

    Jatin Ganhotra et al., “Integrating dialog history into end-to- end spoken language understanding systems,” inInterspeech 2021, 2021, pp. 1254–1258

    2021

  9. [9]

    ConvKT: Conversation- Level Knowledge Transfer for Context Aware End-to-End Spo- ken Language Understanding,

    Vishal Sunder, Eric Fosler-Lussier, Samuel Thomas, Hong- Kwang J Kuo, and Brian Kingsbury, “ConvKT: Conversation- Level Knowledge Transfer for Context Aware End-to-End Spo- ken Language Understanding,” inInterspeech 2023, 2023, pp. 1129–1133

    2023

  10. [10]

    Retrieval augmented end-to-end spoken dialog models,

    Mingqiu Wang et al., “Retrieval augmented end-to-end spoken dialog models,” inICASSP. IEEE, 2024, pp. 12056–12060

    2024

  11. [11]

    Approach- ing Dialogue State Tracking via Aligning Speech Encoders and LLMs,

    ˇSimon Sedl´aˇcek, Bolaji Yusuf, J´an ˇSvec, Pradyoth Hegde, San- tosh Kesiraju, Oldˇrich Plchot, and Jan ˇCernock´y, “Approach- ing Dialogue State Tracking via Aligning Speech Encoders and LLMs,” inInterspeech 2025, 2025, pp. 1748–1752

    2025

  12. [12]

    LoRA: Low-rank adaptation of large lan- guage models,

    Edward J Hu et al., “LoRA: Low-rank adaptation of large lan- guage models,” inInternational Conference on Learning Rep- resentations, 2022

    2022

  13. [13]

    SpokenWOZ: A large-scale speech- text benchmark for spoken task-oriented dialogue agents,

    Shuzheng Si et al., “SpokenWOZ: A large-scale speech- text benchmark for spoken task-oriented dialogue agents,” in Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023

    2023

  14. [14]

    TOD-DA: Towards Boosting the Robust- ness of Task-oriented Dialogue Modeling on Spoken Conver- sations,

    Xin Tian et al., “TOD-DA: Towards Boosting the Robust- ness of Task-oriented Dialogue Modeling on Spoken Conver- sations,”arXiv preprint arXiv:2112.12441, 2021

    Pith/arXiv arXiv 2021

  15. [15]

    Adapting document-grounded dialog systems to spo- ken conversations using data augmentation and a noisy channel model,

    David Thulke, Nico Daheim, Christian Dugast, and Hermann Ney, “Adapting document-grounded dialog systems to spo- ken conversations using data augmentation and a noisy channel model,”arXiv preprint arXiv:2112.08844, 2021

    Pith/arXiv arXiv 2021

  16. [16]

    Improving speech translation by understanding and learning from the auxiliary text translation task,

    Yun Tang, Juan Pino, Xian Li, Changhan Wang, and Dmitriy Genzel, “Improving speech translation by understanding and learning from the auxiliary text translation task,” in59th An- nual Meeting of the Association for Computational Linguistics, Online, Aug. 2021, pp. 4252–4261, Association for Computa- tional Linguistics

    2021

  17. [17]

    MAESTRO: Matched Speech Text Representations through Modality Matching,

    Zhehuai Chen and others, “MAESTRO: Matched Speech Text Representations through Modality Matching,” inInterspeech 2022, 2022, pp. 4093–4097

    2022

  18. [18]

    USTED: Im- proving ASR with a unified speech and text encoder-decoder,

    Bolaji Yusuf, Ankur Gandhe, and Alex Sokolov, “USTED: Im- proving ASR with a unified speech and text encoder-decoder,” inICASSP. IEEE, 2022, pp. 8297–8301

    2022

  19. [19]

    Towards reducing the need for speech train- ing data to build spoken language understanding systems,

    Samuel Thomas, Hong-Kwang J Kuo, Brian Kingsbury, and George Saon, “Towards reducing the need for speech train- ing data to build spoken language understanding systems,” in ICASSP 2022-2022 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 7932–7936

    2022

  20. [20]

    Written Term Detection Im- proves Spoken Term Detection,

    Bolaji Yusuf and Murat Sarac ¸lar, “Written Term Detection Im- proves Spoken Term Detection,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 3213– 3223, 2024

    2024

  21. [21]

    Speechless: Speech Instruction Training Without Speech for Low Resource Languages,

    “Speechless: Speech Instruction Training Without Speech for Low Resource Languages,” inInterspeech 2025, 2025, pp. 3239–3243

    2025

  22. [22]

    Speech Aware Dialog System Technology Challenge (DSTC11),

    Hagen Soltau et al., “Speech Aware Dialog System Technology Challenge (DSTC11),” inInterspeech 2023, 2023, pp. 4668– 4672

    2023

  23. [23]

    DialogStudio: Towards richest and most diverse unified dataset collection for conversational AI,

    Jianguo Zhang et al., “DialogStudio: Towards richest and most diverse unified dataset collection for conversational AI,” inFindings of the Association for Computational Linguistics: EACL 2024, St. Julian’s, Malta, Mar. 2024, pp. 2299–2315, Association for Computational Linguistics

    2024

  24. [24]

    Robust speech recognition via large-scale weak supervision,

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever, “Robust speech recognition via large-scale weak supervision,” inInternational conference on machine learning. PMLR, 2023, pp. 28492– 28518

    2023

  25. [25]

    The Fisher corpus: A resource for the next generations of speech- to-text.,

    Christopher Cieri, David Miller, and Kevin Walker, “The Fisher corpus: A resource for the next generations of speech- to-text.,” inLREC, 2004, vol. 4, pp. 69–71

    2004

  26. [26]

    Librispeech: An ASR corpus based on public domain audio books,

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur, “Librispeech: An ASR corpus based on public domain audio books,” inICASSP, 2015, pp. 5206–5210

    2015

  27. [27]

    Common voice: A massively- multilingual speech corpus,

    Rosana Ardila et al., “Common voice: A massively- multilingual speech corpus,” inProceedings of the Twelfth Language Resources and Evaluation Conference, Marseille, France, May 2020, pp. 4218–4222, European Language Re- sources Association

    2020

  28. [28]

    V oxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation,

    Changhan Wang et al., “V oxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation,” inACL 2021-59th Annual Meet- ing of the Association for Computational Linguistics, 2021

    2021

  29. [29]

    Shades of BLEU, flavours of success: The case of MultiWOZ,

    Tom ´aˇs Nekvinda and Ond ˇrej Du ˇsek, “Shades of BLEU, flavours of success: The case of MultiWOZ,” inProceedings of the First Workshop on Natural Language Generation, Eval- uation, and Metrics (GEM), Online, Aug. 2021, pp. 34–46, As- sociation for Computational Linguistics

    2021

  30. [30]

    WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing,

    Sanyuan Chen et al., “WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing,”IEEE Jour- nal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  31. [31]

    Gemma 3 technical report,

    Gemma Team, “Gemma 3 technical report,”arXiv preprint arXiv:2503.19786, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    Olmo: Accelerating the science of language models,

    Dirk Groeneveld et al., “Olmo: Accelerating the science of language models,”Preprint, 2024

    2024

  33. [33]

    Decoupled weight decay regularization,

    Ilya Loshchilov and Frank Hutter, “Decoupled weight decay regularization,” inInternational Conference on Learning Rep- resentations, 2019

    2019