Pith. sign in

REVIEW 11 cited by

Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2305.15255 v4 pith:JANS4RF5 submitted 2023-05-24 cs.CL cs.LGcs.SDeess.AS

Eliya Nachmani , Alon Levkovitch , Roy Hirsch , Julian Salazar , Chulayuth Asawaroengchai , Soroosh Mariooryad , Ehud Rivlin , RJ Skerry-Ryan
show 1 more author
Michelle Tadmor Ramanovich
This is my paper
classification cs.CLcs.LGcs.SDeess.AS
keywords speechspokencontinuationspectronansweringapproachgithubhttps
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present Spectron, a novel approach to adapting pre-trained large language models (LLMs) to perform spoken question answering (QA) and speech continuation. By endowing the LLM with a pre-trained speech encoder, our model becomes able to take speech inputs and generate speech outputs. The entire system is trained end-to-end and operates directly on spectrograms, simplifying our architecture. Key to our approach is a training objective that jointly supervises speech recognition, text continuation, and speech synthesis using only paired speech-text pairs, enabling a `cross-modal' chain-of-thought within a single decoding pass. Our method surpasses existing spoken language models in speaker preservation and semantic coherence. Furthermore, the proposed model improves upon direct initialization in retaining the knowledge of the original LLM as demonstrated through spoken QA datasets. We release our audio samples (https://michelleramanovich.github.io/spectron/spectron) and spoken QA dataset (https://github.com/google-research-datasets/LLAMA1-Test-Set).

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. VITA-QinYu: Expressive Spoken Language Model for Role-Playing and Singing

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    VITA-QinYu is the first expressive end-to-end spoken language model supporting role-playing and singing alongside conversation, trained on 15.8K hours of data and outperforming prior models on expressiveness and conve...

  2. The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoning

    eess.AS 2026-03 conditional novelty 7.0 of 10

    Causal full-duplex speech LLMs improve response quality by recursively feeding soft vocabulary embeddings as latent thoughts during listening, trained by matching a non-causal expert posterior via ELBO.

  3. The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoning

    eess.AS 2026-03 unverdicted novelty 7.0 of 10

    FLAIR enables spoken dialogue AI to conduct continuous latent reasoning while perceiving speech through recursive latent embeddings and an ELBO-based finetuning objective.

  4. Efficient Chain-of-Modality Reasoning via Progressive Compression for Spoken Language Models

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Spoken math models that emit a 40%-compressed reasoning trace between question and answer beat full-reasoning baselines by ~3 accuracy points while using roughly one third of the text tokens.

  5. FacePlex: Full-Duplex Joint Speech-Facial Motion Generation for Conversational Avatars

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    FacePlex introduces a unified streaming model with Rolling Flow Matching and Rolling Cross-Attention to enable full-duplex joint real-time generation of speech and facial motion tokens.

  6. Audio Interaction Model

    cs.SD 2026-06 unverdicted novelty 6.0 of 10

    Audio-Interaction unifies offline and online audio tasks into one streaming model via the SoundFlow framework and a new 2.6M-item streaming corpus, enabling real-time instruction following and proactive responses.

  7. Learning When to Think While Listening in Large Audio-Language Models

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    A wait-think-answer controller for LALMs is trained via SFT followed by six-reward DAPO, raising row-weighted accuracy from 67.6% to 70.3% and cutting post-endpoint thinking length by 14% on synthetic spoken QA while ...

  8. VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agents

    cs.SD 2025-10 conditional novelty 6.0 of 10

    A Chinese benchmark built on real human speech evaluates large audio language models across instruction following, knowledge, and robustness, revealing large performance gaps.

  9. Which Speech Representation Better Matches Text-Native Reasoning? A Study of Speech-Text Alignment on Frame Rate and Representation

    eess.AS 2026-06 unverdicted novelty 5.0 of 10

    Empirical sweep finds 4.17 Hz frame rate plus intermediate-layer alignment optimal for speech QA under frozen text LLM backbone.

  10. Enhancing Speech Large Language Models through Reinforced Behavior Alignment

    cs.CL 2025-08 unverdicted novelty 5.0 of 10

    Reinforced Behavior Alignment (RBA) uses self-synthesized data from a teacher LLM and reinforcement learning to close the instruction-following gap in SpeechLMs, outperforming distillation and reaching SOTA on spoken ...

  11. The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoning

    eess.AS 2026-03 unverdicted novelty 4.0 of 10

    FLAIR enables simultaneous latent reasoning during speech input in full-duplex dialogue models via recursive latent embeddings and an ELBO-based training objective without added latency.

Pith tools