Pith. sign in

REVIEW 4 major objections 6 minor 24 references

The paper proposes a strictly causal, streaming framework that, every second, labels a speaker's high-level intent and low-level conversational behavior and produces a short rationale, at 0.74 s latency, and shows it transfers from syntheti

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-03 00:52 UTC pith:BEHGYXIW

load-bearing objection A well-built streaming system for hierarchical speech-act detection and rationale generation, but the headline claims are measured against LLM-generated labels and judges, so the empirical loop is not yet closed. the 4 major comments →

arxiv 2602.11065 v2 pith:BEHGYXIW submitted 2026-02-11 cs.CL cs.AI

S-MARC: Causal Streaming Reasoning for Full-Duplex Conversational Behavior Modeling

Dingkun Zhou , Shuchang Pan , Jiachen Lian , Siddharth Banerjee , Sarika Pasumarthy , Dhruv Hebbar , Siddhant Patel , Zeyi Austin Li
show 6 more authors
Kan Jen Cheng Sanay Bordia Krish Patel Akshaj Gupta Tingle Li Gopala Anumanchipalli
This is my paper
classification cs.CL cs.AI
keywords full-duplex dialogueconversational behavior modelingspeech actsstreaming causal inferenceGraph-of-Thoughtsinterpretable rationalessynthetic dialogue corpusConversationGoT-120h
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Human conversation runs on an implicit chain of thought: listeners perceive what kind of act a speaker is performing, decide whether to take the floor or stay silent, and then act. This paper tries to make that pathway explicit and machine-readable. It proposes a strictly causal, streaming model that every second classifies the speaker's high-level speech act (constative, directive, commissive, acknowledgment) and low-level behavior (continuation, turn-taking, interruption, backchannel), and then generates a concise rationale explaining the choice. Training on a 120-hour synthetic duplex corpus with causally constrained labels, the model detects behaviors with strong discrimination on core classes, transfers to real conversations with little degradation, and produces rationales at 0.74 s latency. A sympathetic reader would care because it reframes full-duplex spoken AI from next-token generation to next-behavior reasoning with auditable decisions.

Core claim

The paper's central claim is that conversation can be modeled as a two-level perception task — first grasp the speaker's communicative intent, then predict the concrete interaction move it licenses — and that a strictly causal, streaming system trained with this hierarchy can do both at one decision per second while also explaining itself. The authors argue that low-level behaviors are statistically coupled to high-level acts, e.g., acknowledgments predominantly pair with backchannels, and that this dependency can be exploited by feeding high-level context into low-level prediction. They demonstrate it with a hierarchical perceiver (two causal transformer decoders, with high-level states mod

What carries the argument

The load-bearing machinery is a two-part causal pipeline. A hierarchical Speech-Act Perceiver processes each 1-second audio block with frozen acoustic and semantic embeddings, fuses them by gating, and runs two causal transformer decoders: one for high-level speech acts and one for low-level behaviors, with the high-level state modulating the low-level stream through feature-wise conditioning. A Graph-of-Thoughts (GoT) reasoner then maintains a sliding-window graph with second-level and sentence-level nodes, uses a learned selector to pick anchor sentences (supervised by the teacher LLM's retrieval), linearizes the selected evidence into a causal past-to-present chain, and decodes a rational

Load-bearing premise

The load-bearing premise is that the LLM-generated dialogues and per-second annotations are a faithful stand-in for real human conversational behavior; if those labels diverge systematically from what human listeners perceive, then the reported accuracy, transfer numbers, and benchmark claims measure the teacher model's own judgments rather than human ground truth.

What would settle it

A concrete test: collect a set of real full-duplex conversations, have independent trained human annotators label each 1-second segment with the paper's eight-category taxonomy, and compare the model's predictions against those human labels. If per-class agreement for the long-tail classes (interruption, backchannel, commissive) is near chance, or if the human-labeled corpus yields materially different class distributions from the synthetic one, then the synthetic label protocol is not capturing human judgment and the transfer and benchmark claims are unsupported.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Full-duplex systems can emit an auditable rationale for every second of behavior, not just a label.
  • The conditional coupling between high-level and low-level acts justifies hierarchical prediction and can improve long-tail behavior detection.
  • A synthetic, causally annotated duplex corpus can serve as training data that transfers to real conversations with only slight degradation.
  • Rationale generation at 0.74 seconds makes explanations usable inside live dialogue, where multi-second or longer LLM thinking is infeasible.
  • The per-second, two-level evaluation protocol offers a template for benchmarking conversational reasoning in duplex spoken dialogue systems.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the teacher-generated labels are only quality-filtered, not independently re-labeled by humans, a head-to-head comparison against human-annotated per-second labels on real conversations would reveal whether the transfer numbers reflect human behavior or the teacher model's annotation style.
  • A natural audit extension: have humans judge whether each generated rationale is actually supported by the retrieved evidence chain, separating 'plausible-sounding' from 'genuinely grounded' explanations.
  • The same hierarchy could act as a controller in a full-duplex agent: high-level act predictions could gate the agent's speech generation (hold on backchannels, prepare answers on directives), turning behavior detection into a policy signal rather than just an annotation layer.
  • A cheap ablation would give a one-second lookahead to the perceiver; if performance jumps, the strict-causality constraint is doing the work; if not, the hierarchy and input features are what matter.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces S-MARC, a streaming, strictly causal, hierarchical framework for full-duplex conversational behavior modeling. At every 1-second tick it predicts a high-level speech act (constative, directive, commissive, acknowledgment) and a low-level interaction behavior (continuation, turn-taking, interruption, backchannel), while generating a concise natural-language rationale via a Graph-of-Thoughts (GoT) module. The system is trained on ConversationGoT-120h, a newly constructed corpus of GPT-4o/GPT-5-generated dialogues with per-second speech-act and rationale annotations produced by LLMs and only quality-filtered by five volunteers. The authors report in-domain and out-of-distribution (Candor) detection results, human-model agreement, rationale quality as judged by GPT-4o, and low latency (0.74 s). The central claims are that the model achieves robust behavior detection, produces interpretable reasoning chains, transfers to real duplex dialogues, and establishes a benchmark foundation for conversational reasoning.

Significance. If the empirical evaluation is accepted, the work provides a useful architectural template for real-time, interpretable behavior perception in spoken dialogue systems: the strict-causality protocol, the hierarchical label scheme, and the low-latency GoT reasoner are all sensible contributions. However, the significance is currently conditional: the detection and transfer numbers are measured against labels produced by the same LLM pipeline that generated the training data and judged the rationales. The paper explicitly acknowledges in the Impact Statement that distribution shift and artifacts are a key risk, but the evaluation does not mitigate that risk because the same annotation model is used for training labels, OOD labels, and rationale judgments. To substantiate the 'benchmark foundation' claim, the authors need to provide human-annotated gold labels or at least a credible human agreement study on a held-out sample of real conversations.

major comments (4)
  1. [§3.1 / Appendix A] The abstract and introduction state that the corpus pairs dialogue data with 'human-annotated labels,' but §3.1 describes GPT-4o/GPT-5 generation of speech acts and rationales with only five volunteers performing quality filtering, and Appendix A confirms that OOD Candor labels are also generated by GPT-5 under the same protocol. Thus Tables 4 and 5 measure agreement with GPT-5's taxonomy judgments, not with human ground truth. The 'successful transfer' claim to real dialogues is therefore not established. Concretely: annotate a random sample (e.g., 500 seconds) from Candor with multiple human coders using the proposed taxonomy, report per-class agreement, and use that as a gold set for Table 5. Without this, the claimed 'benchmark foundation' is circular.
  2. [§4.3, Tables 4 and 5] The long-tail F1 scores are low (e.g., Interruption 0.495 in-domain and 0.486 OOD; Directives 0.474 and 0.456; Commissives 0.474 and 0.445), which weakens the 'robust behavior detection' claim in the abstract and conclusion. Moreover, §4.2 states results are 'evaluated with statistical analysis over five independent runs,' but no confidence intervals, standard deviations, or significance tests are reported for any detection metric. Please report per-run variance and, at minimum, a paired significance test comparing against a majority-class baseline and against a zero-shot GPT-4o/GPT-5 causally constrained predictor.
  3. [§4.4, Table 7] Rationale quality is evaluated by GPT-4o as an automatic judge, which is part of the same LLM family that generated the training rationales and anchors. Human ratings are reported for the training rationales (Table 3), but not for the model-generated rationales in Table 7. The claim 'approaching GPT-5 quality' is therefore based on an LLM judge that may systematically favor LLM-style outputs. Please add a human evaluation of model-generated rationales on a held-out test set, including inter-annotator agreement (e.g., Krippendorff's alpha), and report the comparison with the random-selector and GPT-4o baselines.
  4. [§4.3] No baseline for speech-act detection is reported. The paper argues that cross-paper comparison is confounded by taxonomy and granularity differences, which is understandable, but it should still compare against a non-trivial baseline under the same taxonomic protocol. For instance, prompt a strong LLM (GPT-4o or GPT-5) with the current second's ASR hypothesis and the causal history, in a zero-shot or few-shot manner, and evaluate with the same metric. Without such a baseline, the absolute AUC/F1 numbers in Tables 4–5 cannot be interpreted as a contribution of the proposed architecture.
minor comments (6)
  1. [Title / Abstract] The arXiv title is 'S-MARC: Causal Streaming Reasoning...' but the full-text title is 'Conversational Behavior Modeling Foundation Model With Multi-Level Perception.' Please reconcile the title used on the abstract page and in the body.
  2. [§4.3, Table 6] The caption says 'mean and standard deviation over evaluated samples' but the table shows only means, no standard deviations. Either add the standard deviations or correct the caption.
  3. [§4.2] The claim of 'statistical analysis over five independent runs' is not reflected anywhere in the reported results; please make the analysis explicit (e.g., error bars, significance tests) or remove the statement.
  4. [§4.1 / Table 2] The heading 'V olunteers' has a typographical space; also the volunteer means, while lower on some dimensions, are within one point of GPT-4o—consider discussing the implications of this discrepancy rather than only saying 'model evaluation is more optimistic.'
  5. [§4.1, Event Distribution] There is a typo: 'maean spacing' should be 'mean spacing.' Similarly, in §3.2, 'CosyV oice2' should be 'CosyVoice2.'
  6. [Appendix B, HMA definition] The HMA definition uses a formula but does not specify how disagreements are handled when the model's label is one of several plausible options; please clarify whether the volunteer is asked to judge the model's exact label or to select among alternatives.

Circularity Check

3 steps flagged

Detection and transfer claims are measured against GPT-5-generated labels rather than independent human ground truth, with rationale and human-agreement checks also self-referential.

specific steps
  1. fitted input called prediction [§3.1 Generation; Appendix A 'Strictly causal protocol'; Tables 4–5]
    "we then feed this topic chain together with the observable inputs of the current second into a reasoning GPT5 (Singh et al., 2025), so that it gradually derives the speech act (high/low) label for the current second ... when generating OOD annotations for the Candor dataset ... any large model or auxiliary component, including GPT-5, is forbidden from accessing future audio, future transcripts, or future semantic cues, ensuring that the OOD annotation process matches a real online perception setting."

    The training labels and the OOD 'real dialogue' labels are both produced by GPT-5 under the same protocol. Therefore Tables 4 and 5, and the claimed 'successful transfer to real duplex dialogues,' measure the model's agreement with GPT-5's taxonomy on Candor, not with independently established human conversational behavior. The 'real-world' evaluation target is defined by the same LLM that defined the training target, so the transfer result reduces to GPT-5 self-consistency across corpora.

  2. other [§4.4, Table 7]
    "For evaluation, we use GPT-4o as an automatic judge, scoring the generated outputs with a fixed rubric (Ruler) along four dimensions: Alignment, Justification, Caption Completeness, and Clarity."

    The rationales used to train GoT were generated by GPT-5, and the quality of the model's generated rationales is scored by GPT-4o, another LLM from the same family. No independent human gold-standard rating of the generated rationales is reported. Thus the claim 'produces interpretable reasoning chains' rests on LLM self-assessment: an OpenAI teacher produced the supervision, and an OpenAI judge validates the imitation, without external human grounding.

  3. other [§4.3, Table 6; HMA Definition in Appendix B]
    "Each volunteer independently annotates, at each level, whether they agree with the model’s decision at time t, yielding binary responses (1 if agree, otherwise 0)."

    The human evaluation protocol asks volunteers to ratify the model's per-second decisions rather than to independently label the conversation. High HMA values (e.g., HMAh=0.97) therefore show that humans can agree with the model when prompted with its choices, not that the model's labels match an independent human gold standard. This converts purported human validation into a conformity test, weakening the claim that the detected labels reflect human perception.

full rationale

The paper's core empirical claims—robust behavior detection, successful OOD transfer to real Candor dialogues, and interpretable reasoning chains—are evaluated through a largely self-referential chain: GPT-5 generates the training labels, GPT-5 also generates the OOD labels under the same causal protocol, GPT-4o judges the generated rationales, and the human evaluation asks volunteers only whether they agree with the model. The abstract's phrase 'human-annotated labels' is contradicted by §3.1, where the labels are produced by GPT-5 and only quality-filtered by five volunteers. This does not make the architecture itself circular: the hierarchical perceiver and GoT reasoner are genuinely trained on audio/text inputs and could in principle be tested against independent human annotations. However, as reported, the headline detection and transfer numbers measure agreement with an LLM-generated label space rather than with human conversational behavior. There is no load-bearing self-citation chain or imported uniqueness theorem; the circularity is partial and concentrated in the evaluation protocol, so a score of 5 is appropriate.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 0 invented entities

The central claim depends on approximately half a dozen hand-set hyperparameters and, more importantly, on the assumption that LLM-generated labels and rationales are faithful, causally valid ground truth. No new physical or theoretical entities are introduced; the reported performance is on the authors' own synthetic benchmark plus LLM-annotated real audio.

free parameters (7)
  • Alpha (positive-class weight) = N_neg/N_pos computed on training split
    Used in selector's weighted BCE (L_wbce); depends on training label distribution.
  • Lambda_count = 0.01
    Regularizer that constrains number of selected anchors; hand-chosen.
  • Lambda_rank = 0.1
    Regularizer that promotes positive anchor ranking; hand-chosen.
  • Selector temperature T = 1.0
    Scales selector logits before thresholding; hand-chosen in Appendix B.
  • Causal window W = 90 seconds
    Candidate sentence window [t−W,t); hand-chosen for context.
  • Temporal-neighborhood bias beta = not specified
    Causal attention penalty −β(i−j); value not reported.
  • Trained weights of SA Perceiver, GoT selector, T5 decoder = learned on ConversationGoT-120h
    All central predictors are fitted to the synthetic corpus; no external pretraining verification.
axioms (5)
  • domain assumption GPT-4o/GPT-5 generated dialogues, identity profiles, and topic chains are representative of human conversation
    Section 3.1 builds the entire 120h corpus this way; only quality-filtered, not per-label expert-checked.
  • domain assumption GPT-5's per-second speech-act and rationale annotations are valid causal ground truth
    Section 3.1 and Appendix 'Strictly causal protocol'; training supervision and OOD labels rely on these annotations.
  • ad hoc to paper The strict causal protocol prevents future information from leaking into labels
    Appendix asserts GPT-5 sees only current/past; if violated, the 'online' claims are undermined.
  • domain assumption GPT-4o rubric scores correlate with human rationale quality
    Section 4.4 uses GPT-4o as automatic judge; human ratings are on a different, smaller sample.
  • standard math Standard ML assumptions (cross-entropy training, AdamW, transformer generalization) hold
    Used throughout; no formal guarantees beyond empirical results.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of S-MARC: Causal Streaming Reasoning for Full-Duplex Conversational Behavior Modeling." pith.science (2026). https://pith.science/paper/BEHGYXIW

@misc{pith2026260211065,
  author       = {Pith},
  title        = {Pith review of: S-MARC: Causal Streaming Reasoning for Full-Duplex Conversational Behavior Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BEHGYXIW}},
  note         = {Machine review of arXiv:2602.11065}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Human conversation is organized by an implicit chain of thought and manifests as temporally structured conversational behaviors. Capturing this perceptual pathway is critical for building natural full-duplex interactive systems. We propose S-MARC (Streaming Causal Modeling and Reasoning for Conversation), a streaming, causal, and hierarchical framework for conversational behavior modeling and reasoning. By formalizing the intent-to-action pathway, S-MARC predicts high-level communicative functions and low-level interaction behaviors while modeling their causal and temporal dependencies. To support this setting, we construct a high-quality corpus that pairs controllable, event-rich duplex dialogue data with behavior labels. S-MARC organizes streaming predictions into a continuously evolving graph structure, generates concise justifications for its decisions, and dynamically optimizes its reasoning process. Experiments on synthetic and real duplex dialogues show that S-MARC achieves robust behavior detection, produces interpretable reasoning chains, and establishes a benchmark foundation for conversational reasoning in full-duplex spoken dialogue systems.

Figures

Figures reproduced from arXiv: 2602.11065 by Akshaj Gupta, Dhruv Hebbar, Dingkun Zhou, Gopala Anumanchipalli, Jiachen Lian, Kan Jen Cheng, Krish Patel, Sanay Bordia, Sarika Pasumarthy, Shuchang Pan, Siddhant Patel, Siddharth Banerjee, Tingle Li, Zeyi Austin Li.

Figure 1
Figure 1. Figure 1: Comparison of dialogue paradigms. (Traditional) Traditional duplex systems frame conversation as a direct sequence prediction task. (Ours) We propose a framework based on next-behavior perception and reasoning: the agent first perceives the speaker’s behaviors at multiple levels, then reasons via a Graph-of-Thoughts, and finally generates a response. Raux & Eskenazi, 2012; Khouzaimi et al., 2016; Marge et … view at source ↗
Figure 2
Figure 2. Figure 2: Causal streaming pipeline for conversational behavior modeling. At each 1 s tick, the model causally predicts hierarchical speech acts and generates an evidence-grounded rationale using a sliding-window Graph-of-Thoughts (GoT). information and topic sequence, we complete transitions between topics and generate the beginning and ending of the dialogue, making the overall dialogue more coherent. To provide s… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 1 canonical work pages

  1. [1]

    Talking turns: Benchmarking audio foundation models on turn-taking dynamics.arXiv preprint arXiv:2503.01174, 2025a

    Arora, S., Lu, Z., Chiu, C.-C., Pang, R., and Watanabe, S. Talking turns: Benchmarking audio foundation models on turn-taking dynamics.arXiv preprint arXiv:2503.01174, 2025a. Arora, S., Lu, Z., Chiu, C.-C., Pang, R., and Watanabe, S. Talking turns: Benchmarking audio foundation models on turn-taking dynamics, 2025b. URL https://arxiv. org/abs/2503.01174. ...

    Pith/arXiv arXiv

  2. [4]

    Cosyvoice 2: Scalable streaming speech synthesis with large language models.arXiv preprint arXiv:2412.10117,

    Du, Z., Wang, Y ., Chen, Q., Shi, X., Lv, X., Zhao, T., Gao, Z., Yang, Y ., Gao, C., Wang, H., et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models.arXiv preprint arXiv:2412.10117,

    Pith/arXiv arXiv

  3. [10]

    S., Wang, Q., Lian, J., Li, T., Watanabe, S., and yi Lee, H

    Lin, G.-T., Kuan, S.-Y . S., Wang, Q., Lian, J., Li, T., Watanabe, S., and yi Lee, H. Full-duplex-bench v1.5: Evaluating overlap handling for full-duplex speech mod- els, 2025a. URL https://arxiv.org/abs/2507. 23159. Lin, G.-T., Lian, J., Li, T., Wang, Q., Anumanchipalli, G., Liu, A. H., and yi Lee, H. Full-duplex-bench: A benchmark to evaluate full-duple...

    Pith/arXiv arXiv

  4. [11]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization.arXiv preprint arXiv:1711.05101,

    Pith/arXiv arXiv

  5. [13]

    Panayotov, V ., Chen, G., Povey, D., and Khudanpur, S

    URL https://arxiv.org/abs/2410.21276. Panayotov, V ., Chen, G., Povey, D., and Khudanpur, S. Librispeech: an asr corpus based on public domain au- dio books. In2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206–5210. IEEE,

    Pith/arXiv arXiv

  6. [14]

    Perez, E., Strub, F., De Vries, H., Dumoulin, V ., and Courville, A

    URL https: //arxiv.org/abs/2510.07355. Perez, E., Strub, F., De Vries, H., Dumoulin, V ., and Courville, A. Film: Visual reasoning with a general con- ditioning layer. InProceedings of the AAAI conference on artificial intelligence, volume 32,

    Pith/arXiv arXiv

  7. [15]

    Sacks, H., Schegloff, E

    URL https://arxiv.org/ abs/2203.00674. Sacks, H., Schegloff, E. A., and Jefferson, G. A simplest systematics for the organization of turn-taking for conver- sation.Language, 50(4):696–735,

    Pith/arXiv arXiv

  8. [17]

    Stivers, T., Enfield, N

    URL https://arxiv.org/abs/2601.03267. Stivers, T., Enfield, N. J., Brown, P., Englert, C., Hayashi, M., Heinemann, T., Hoymann, G., Rossano, F., de Ruiter, J. P., Yoon, K.-E., and Levinson, S. C. Universals and cultural variation in turn-taking in conversation.Pro- ceedings of the National Academy of Sciences, 106(26): 10587–10592,

    Pith/arXiv arXiv

  9. [19]

    Xu, Y ., Chen, H., Yu, J., Huang, Q., Wu, Z., Zhang, S., Li, G., Luo, Y ., and Gu, R

    URL https: //arxiv.org/abs/1804.00015. Xu, Y ., Chen, H., Yu, J., Huang, Q., Wu, Z., Zhang, S., Li, G., Luo, Y ., and Gu, R. Secap: Speech emo- tion captioning with large language model,

    Pith/arXiv arXiv

  10. [20]

    Yao, Y ., Li, Z., and Zhao, H

    URL https://arxiv.org/abs/2312.10381. Yao, Y ., Li, Z., and Zhao, H. Beyond chain-of-thought, effective graph-of-thought reasoning in language mod- els,

    Pith/arXiv arXiv

  11. [21]

    Zhang, W., He, Y ., Lin, G., Liu, Z., Sun, S., Wang, B., Zou, X., Wong, J

    URL https: //arxiv.org/abs/2504.09696. Zhang, W., He, Y ., Lin, G., Liu, Z., Sun, S., Wang, B., Zou, X., Wong, J. H. M., Wang, Q., Sailor, H. B., Chen, N. F., and Aw, A. T. Beyond classification: Towards speech emotion reasoning with multitask audiollms,

    arXiv

  12. [22]

    Zhi-Xuan, T., Ying, L., Mansinghka, V ., and Tenenbaum, J

    URL https://arxiv.org/abs/2506.06820. Zhi-Xuan, T., Ying, L., Mansinghka, V ., and Tenenbaum, J. B. Pragmatic instruction following and goal assistance via cooperative language-guided inverse planning,

    arXiv

  13. [23]

    filled in after seeing the future,

    URLhttps://arxiv.org/abs/2402.17930. 13 Conversational Behavior Modeling Foundation Model With Multi-Level Perception Table 8.Turn-taking event frequencies (per minute) and cumulative durations (%) for the simulation dataset, a human reference, and model baselines.Human, dGSLM, and Moshi values are reproduced from Fig. 2 of (Arora et al., 2025b). Number o...

    Pith/arXiv arXiv 1974

  14. [24]

    We then fuse them via element-wise gating: λi =σ(W bhB i +W ehE i ), e i = (1−λ i)⊙h B i +λ i ⊙h E i , and map to a shared latent space(˜zi =g shared(ei))

    to transcribe (Ui), then feeds the transcribed text into a frozen T5-Base encoder (Raffel et al., 2020), and applies mask-aware mean pooling to obtain(h E i ). We then fuse them via element-wise gating: λi =σ(W bhB i +W ehE i ), e i = (1−λ i)⊙h B i +λ i ⊙h E i , and map to a shared latent space(˜zi =g shared(ei)). We use two task-specific causal Transform...

    2020

  15. [1982]

    Explore theory of mind: Program-guided adversarial data generation for theory of mind reasoning.arXiv preprint arXiv:2412.12175,

    Sclar, M., Yu, J., Fazel-Zarandi, M., Tsvetkov, Y ., Bisk, Y ., Choi, Y ., and Celikyilmaz, A. Explore theory of mind: Program-guided adversarial data generation for theory of mind reasoning.arXiv preprint arXiv:2412.12175,

    Pith/arXiv arXiv

  16. [2011]

    Predic- tion of turn-taking using multitask learning with predic- tion of backchannels and fillers

    Hara, K., Inoue, K., Takanashi, K., and Kawahara, T. Predic- tion of turn-taking using multitask learning with predic- tion of backchannels and fillers. InInterspeech 2018, pp. 991–995,

    2018

  17. [2015]

    Nguyen, T

    URL https://arxiv.org/abs/ 1510.06807. Nguyen, T. A., Kharitonov, E., Copet, J., Adi, Y ., Hsu, W.-N., Elkahky, A., Tomasello, P., Algayres, R., Sagot, B., Mohamed, A., and Dupoux, E. Generative spoken dialogue language modeling, 2022a. URL https:// arxiv.org/abs/2203.16502. Nguyen, T. A., Kharitonov, E., Copet, J., Adi, Y ., Hsu, W.-N., Elkahky, A., Toma...

    Pith/arXiv arXiv

  18. [2018]

    doi: 10.21437/Interspeech.2018-1442. Hwang, J., Hira, M., Chen, C., Zhang, X., Ni, Z., Sun, G., Ma, P., Huang, R., Pratap, V ., Zhang, Y ., Kumar, A., Yu, C.-Y ., Zhu, C., Liu, C., Kahn, J., Ravanelli, M., Sun, P., Watanabe, S., Shi, Y ., Tao, Y ., Scheibler, R., Cornell, S., Kim, S., and Petridis, S. Torchaudio 2.1: Advancing speech recognition, self-sup...

    work page doi:10.21437/interspeech.2018-1442 2018

  19. [2020]

    Cheng, Z., Cheng, Z.-Q., He, J.-Y ., Sun, J., Wang, K., Lin, Y ., Lian, Z., Peng, X., and Hauptmann, A

    URL https:// arxiv.org/abs/2005.14165. Cheng, Z., Cheng, Z.-Q., He, J.-Y ., Sun, J., Wang, K., Lin, Y ., Lian, Z., Peng, X., and Hauptmann, A. Emotion- llama: Multimodal emotion recognition and reasoning with instruction tuning,

    Pith/arXiv arXiv 2005

  20. [2022]

    doi: 10.18653/v1/2022

    Association for Computational Linguistics. doi: 10.18653/v1/2022. sigdial-1.22. URL https://aclanthology.org/ 2022.sigdial-1.22/. Lian, J. and Anumanchipalli, G. Towards hierarchical spo- ken language disfluency modeling. InProceedings of the 9 Conversational Behavior Modeling Foundation Model With Multi-Level Perception 18th Conference of the European Ch...

    doi:10.18653/v1/2022 2022

  21. [2023]

    Inoue, K., Lala, D., Skantze, G., and Kawahara, T

    URL https://arxiv.org/abs/2310.17864. Inoue, K., Lala, D., Skantze, G., and Kawahara, T. Yeah, un, oh: Continuous and real-time backchannel prediction with fine-tuning of voice activity projection,

    Pith/arXiv arXiv

  22. [2024]

    org/abs/2406.11161

    URL https://arxiv. org/abs/2406.11161. Défossez, A., Mazaré, L., Orsini, M., Royer, A., Pérez, P., Jégou, H., Grave, E., and Zeghidour, N. Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037,

    Pith/arXiv arXiv

  23. [2025]

    Jurafsky, D

    URL https://arxiv.org/abs/2410.15929. Jurafsky, D. and Martin, J. H.Speech and Lan- guage Processing. Draft / Stanford University, 3 edi- tion,

    Pith/arXiv arXiv

  24. [2026]

    Watanabe, S., Hori, T., Karita, S., Hayashi, T., Nishitoba, J., Unno, Y ., Soplin, N

    URL https://arxiv.org/abs/2601.15668. Watanabe, S., Hori, T., Karita, S., Hayashi, T., Nishitoba, J., Unno, Y ., Soplin, N. E. Y ., Heymann, J., Wiesner, M., Chen, N., Renduchintala, A., and Ochiai, T. Espnet: End- 12 Conversational Behavior Modeling Foundation Model With Multi-Level Perception to-end speech processing toolkit,

    arXiv

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.