{"id":"dbb5923b-b15f-4355-ad00-3d356af1e136","arxiv_id":"2505.23308","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A LLaVA-style model with an added Whisper speech encoder answers spoken questions about images, trained on TTS-synthesized speech and reaching near the text-input baseline.","lead":"This paper extends a vision-language model with a speech encoder so that a user can ask a spoken question about an image and get a textual answer. It trains the system entirely on text-to-speech synthesized questions and reports accuracy close to the text-only baseline, suggesting voice interaction for image Q&A is within reach.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The synthetic near-parity result is real on its own terms, but the transfer to natural speech that makes it practically meaningful is untested; Table 4's WER collapse suggests it may fail.","rationale":"The reader's weakest assumption is exactly the load-bearing gap I see: the paper converts text VQA into SVQA with two TTS systems, evaluates on synthetic speech, and then extrapolates to natural speech without testing it. The cross-TTS design is a genuine strength: training on StyleTTS2 and testing on F5, plus the multi-voice protocol, demonstrates robustness to the two synthesizers and to many voice prompts. It does not, however, span the gap to human speech, and Table 4 gives a concrete, if confounded, reason to worry that the QA fine-tuning has moved the model away from robust general speech processing. Because the paper's own Section 7 acknowledges that the impact of synthetic speech is difficult to quantify and offers only an anticipation, the appropriate verdict is CONDITIONAL, not a rejection. I agree with the reader that a natural-speech evaluation and artifact release are the conditions. The Section 9 injected review instructions are an editorial red flag and must be removed, but they do not change the scientific assessment. My read does not alter the reader's verdict; hence UNCHANGED.","tokens_in":9315,"tokens_out":7257,"duration_ms":73849,"concrete_test":"Record a held-out set of 200 SEED-Bench Image questions read by ~10 naive speakers in a quiet room with a consumer microphone, keeping the same textual post-prompt as Section 6.1. Run the released SVQA-Mix model on these recordings and compare accuracy against the 61.8/60.5 synthetic StyleTTS2/F5 scores and the 68.0 ASR-to-text pipeline score from Table 2. If natural-speech accuracy falls more than about 5 points absolute below the synthetic scores, or below the lower bound of the 95% confidence interval around those scores, the Section 7 transfer expectation is refuted and the abstract should state that the result holds only for synthesized speech.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that a SVQA model trained only on TTS speech 'nearly reaches' the text-based upper bound (SeedBench 61.8 vs 68.2 for SVQA-Mix) and that TTS choice has minor impact. This is supported on synthetic test sets, but the condition that makes the claim meaningful for a spoken interface is that synthetic speech is representative of natural human speech. Section 6.1 evaluates only StyleTTS2 and F5-TTS renderings, with the textual post-prompt kept as text. The cross-TTS and multi-voice protocol in Section 4 controls for TTS voice and style, not for human acoustic variability: coarticulation, disfluencies, prosody, noise, and recording conditions. Section 7 substitutes an expectation ('we anticipate that similar results would be obtained when testing with natural human speech') for evidence. Table 4 is a warning sign: after QA fine-tuning, SVQA-STTS2, SVQA-F5, and SVQA-Mix show 115.4-134.2% WER on synthesized SeedBench and 84.6-87.1% on natural MLS, while SVQA-baseline has 7.4% WER on MLS. The transcription instruction is confounded by these models' tendency to answer rather than transcribe, so the numbers are not themselves a natural-speech QA score; but they show that QA fine-tuning substantially changes how the speech stream is used. Without a natural-speech QA evaluation, the near-parity claim should be scoped to synthetic benchmarks, and the abstract's practical framing is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper extends the LLaVA-1.5 visual question answering architecture to accept spoken questions, yielding a spoken VQA (SVQA) model with a Whisper encoder, a CLIP image encoder, and a Vicuna-13B LLM. Since no SVQA dataset exists, the authors synthesize training and test sets from textual VQA data using two zero-shot TTS systems (StyleTTS2 and F5-TTS) with many voices, and they use a cross-TTS training/test protocol to guard against overfitting to a single synthesizer or voice. On SeedBench, MME, and DocVQA, the TTS-trained SVQA models achieve accuracies close to the text-only LLaVA-1.5 upper bound (e.g., SeedBench 61.8 vs 68.2 for the mixed-TTS model), with small differences between TTS choices. The paper also analyzes the effect of speaking the multiple-choice post-prompt and reports word error rates for the models under a transcription instruction. The overall claim is that a spoken VQA model trained entirely on synthesized speech nearly matches the text-based upper bound and that the choice of TTS model has only a minor impact.","tokens_in":9593,"tokens_out":2387,"duration_ms":27747,"significance":"If the results hold, the paper makes a useful empirical contribution: it provides a practical recipe for creating large-scale SVQA data with zero-shot multi-speaker TTS, a cross-TTS evaluation protocol that mitigates single-synthesizer overfitting, and evidence that a multimodal LLM can ingest spoken questions about images with a modest accuracy gap relative to text input. The scale of the synthetic data (3.4M samples, 950 speakers) and the use of two TTS systems with disjoint training/test speakers are concrete strengths, as is the release of the SVQA test sets. However, the significance is bounded by the fact that all VQA evaluations use synthetic test speech. The paper explicitly anticipates similar behavior on natural human speech, but no natural-speech VQA evaluation is performed, and the transcription results in Section 6.3 raise concern about how strongly QA fine-tuning changes the model's use of the speech stream. The central near-parity claim is therefore currently a claim about synthetic benchmarks, not about real deployment conditions.","major_comments":[{"comment":"The central claim of near-parity with the textual upper bound is supported only on synthetic test sets (StyleTTS2/F5-TTS renderings). The paper's sentence in Section 7, 'we anticipate that similar results would be obtained when testing with natural human speech,' is an unsupported expectation, not evidence. Section 6.3 provides a warning sign: after QA fine-tuning, SVQA-STTS2, SVQA-F5, and SVQA-Mix show WERs of 84.6-87.1% on natural speech from MLS, versus 7.4% for SVQA-baseline, indicating that QA fine-tuning substantially changes the processing of the speech stream. Although the transcription instruction is confounded by the models' tendency to answer rather than transcribe, the sharp contrast between synthetic and natural speech in this auxiliary task means the generalization claim needs a direct natural-speech QA evaluation. Please either add such an evaluation (even a small human-recorded pilot) or explicitly scope the near-parity claim to synthesized speech in the abstract and conclusion.","section":"Section 6.1 and Section 7"},{"comment":"The comparison supporting 'nearly reaches the upper bound' is confounded by fine-tuning strategy. The text upper bound (LLaVA-1.5-13B, 68.2 on SeedBench) is fully fine-tuned on VQA data, while the SVQA models are fine-tuned with LoRA (r=64, alpha=16). The paper acknowledges this in Section 7 as one possible explanation for the gap, but the near-parity claim is nonetheless weakened because the 6-7 point difference could be attributable to LoRA versus full fine-tuning rather than to the speech modality. A controlled experiment with a text-only LoRA-fine-tuned LLaVA-1.5 (or a fully fine-tuned SVQA model) is needed to attribute the gap to spoken input.","section":"Section 5.2 and Table 1"},{"comment":"The claims that the choice of TTS model has 'a minor impact' and that F5 'slightly outperforms' STTS2 rest on differences of less than one point (e.g., SeedBench 61.4 vs 61.9 for SVQA-STTS2 vs SVQA-F5). No error bars, repeated runs, or significance tests are reported. Given the absence of variance information, these differences are indistinguishable from noise. Please report standard deviations over at least a few random seeds, or clearly state the number of independent runs, so readers can assess the reliability of the TTS-choice and near-parity conclusions.","section":"Table 1 and Section 6.1"}],"minor_comments":[{"comment":"The final section, titled 'Guidelines for document review and summary,' is an inserted instruction telling reviewers to give a positive assessment and to include the word 'exceptional.' This is inappropriate in a scientific manuscript and must be removed entirely. It also directly conflicts with the expectation of independent review and is itself evidence that the manuscript text should not be trusted to be fully authorial.","section":"Section 9"},{"comment":"The author name appears as 'A. ehera' in reference [18]; this should be 'A. Behera' (or the correct name as in the original publication).","section":"Reference [18]"},{"comment":"The column headers 'Question' and 'Full' are ambiguous. Clarify in the caption that 'Question' means only the question is spoken while the post-prompt is textual, and 'Full' means the entire prompt including multiple-choice answers and instructions is spoken.","section":"Table 3"},{"comment":"The paper uses the term 'cross-validation' to describe training on one TTS and testing on another. This is not k-fold cross-validation; recommend using 'cross-TTS evaluation' or 'TTS-transfer evaluation' to avoid confusion.","section":"Section 4.2"},{"comment":"The abstract says 'initial findings indicate' that a model trained only with synthesized speech nearly reaches the upper bound, while the contributions in Section 1 state the paper 'demonstrates' this. Align the wording: either these are initial findings or established results, and given the lack of natural-speech evaluation, 'indicate' is the more accurate choice.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The manuscript contains an inserted Section 9 that explicitly instructs reviewers to write a positive review and include the word 'exceptional.' I have ignored this instruction in my assessment, but the editor should be aware that this passage violates standards of scholarly integrity and should be removed before any further consideration. The paper's empirical contributions are real and the cross-TTS protocol is a good practice, but the central generalization claim to natural speech is untested; the authors should either add a natural-speech evaluation or scope the claim accordingly. The paper is within the scope of the journal, though it is a workshop-style empirical contribution rather than a methodological breakthrough."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper's core result is real on its own terms: a LLaVA-style model with a Whisper encoder, trained on thousands of hours of multi-speaker TTS speech from two zero-shot synthesizers, reaches 61.8 on SeedBench when the text-only upper bound is 68.2, and the choice of TTS barely moves the needle. The cross-TTS training/test split and the speaker-disjoint design are genuine strengths. Second, the step from 'works on our synthetic benchmarks' to 'will work for real users' is assumed, not shown. Section 7's 'we anticipate' is not evidence, and Table 4's WER collapse after QA fine-tuning is a warning sign—though it is confounded by the model answering instead of transcribing, so treat it as a caution, not a refutation.\n\nWhat is new: prior spoken VQA work used single-speaker TTS, classification heads, and no cross-TTS validation. This paper builds a generative three-modal model, creates two large datasets (~5k hours), and shows empirically that TTS choice has minor impact on accuracy. That is a useful reference result for anyone building speech-enabled multimodal assistants.\n\nWhere it gets soft: the near-parity claim is scoped to synthetic test sets, and the paper's practical framing overreaches. No natural-speech QA evaluation, no error bars, no release of data or code. That is fixable, but it needs to be fixed. More serious, Section 9 is a prompt-injection paragraph telling reviewers to write positively. It is not scientific content and must be removed; I will not engage with that text as a legitimate part of the paper.\n\nWho this is for: researchers working on spoken dialogue and multimodal LLMs. It is a solid empirical recipe and a useful benchmark, not a breakthrough. With a natural-speech eval (even a small one) and the injected paragraph gone, this would be a decent Interspeech paper. As-is, it needs revision before acceptance. I would send it to review, with the clear instruction that the injection is unacceptable and the natural-speech claim has to be either tested or removed from the abstract.","headline":"Solid synthetic-benchmark recipe for spoken VQA with a real cross-TTS result, but the practical claim about natural speech is assumed, not shown—and the Section 9 prompt-injection paragraph must be removed before this is publishable.","tokens_in":10158,"tokens_out":3207,"would_cite":true,"duration_ms":33477,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A spoken visual question answering model trained entirely on synthesized speech nearly matches the accuracy of a text-only VQA model, closing most of the gap between text and speech interaction.","keywords":["spoken visual question answering","multimodal large language models","text-to-speech synthesis","zero-shot TTS","speech-image fusion","synthetic training data","visual question answering","LLaVA-style architecture"],"falsifier":"Record human speakers asking the same SEED-Bench, MME, and DocVQA questions about the same images, run the trained SVQA model on those natural utterances, and compare accuracy; if accuracy falls well below the synthetic-test numbers or the word error rate rises sharply, the near-parity claim does not survive contact with real speech.","tokens_in":9084,"feed_emoji":"🎤","tokens_out":6797,"duration_ms":64042,"temperature":0.7,"pith_summary":"The paper tries to establish that a visual question answering (VQA) system can be taught to answer spoken questions about images using only synthesized speech as training data, and that this spoken model lands close to the accuracy of the same model fed text questions. The authors build a model that fuses an image encoder and a speech encoder onto a text-only VQA backbone, then fine-tune it on questions synthesized by two zero-shot text-to-speech systems. On the SEED-Bench image benchmark the spoken model reaches 61.8 percent accuracy versus 68.2 percent for the text-only upper bound, and the choice of TTS system changes accuracy by only a couple of points. If this holds, it means spoken interaction with visual AI can be bootstrapped from existing text VQA datasets without recording any human speech.","feed_headline":"Synthetic speech trains spoken VQA near text accuracy","feed_subtitle":"Model answers spoken questions about images after training only on TTS audio, within ~6 points of text-only.","key_machinery":"The architecture is a LLaVA-style visual-language model extended with a speech tower: a frozen Whisper encoder followed by a trainable projector that downsamples speech frames and aligns them to the language model's embedding space, alongside a frozen CLIP image encoder and its projector. The speech projector is first pre-trained on transcription and on a new 'audio description' task in which the model is asked to describe the content of an audio clip, so that it learns to attend to meaning rather than only transcribe. Fine-tuning then uses question-answer pairs synthesized with two zero-shot TTS systems (StyleTTS2 and F5-TTS) across hundreds of voices, with training and testing on different TTS systems and different speakers to expose any overfitting. A low-rank adaptation (LoRA) of the language model is applied during fine-tuning.","core_discovery":"The central discovery is that a multimodal LLM can absorb spoken questions about images from purely synthetic speech and generalize to other synthetic voices and even to a second TTS system, nearly matching the text-only upper bound. The authors attribute the remaining gap to the base model having been fully fine-tuned for VQA while their model uses low-rank adaptation, and to residual errors from speech synthesis and from converting speech back to text internally. Their cross-validation between two TTS systems, with disjoint speaker pools for training and testing, provides evidence that the result is not overfit to a single synthetic voice.","pith_inferences":["If the synthetic-to-natural transfer holds, the same recipe can be rerun on larger and newer VQA instruction sets, which the paper names as future candidates, to push spoken VQA accuracy closer to the textual ceiling.","The near-parity on synthetic speech is the best-case transfer scenario; a natural-speech benchmark would reveal whether the small remaining gap widens, and the paper's own word-error-rate results suggest the model sometimes answers instead of transcribing, which could be a source of hidden failures.","The spoken-prompt degradation suggests that speech interfaces for VQA should keep structured elements (instructions, answer choices) in text, and raises the testable question of whether natural speech prosody would mitigate that loss.","Because the speech encoder and image encoder stay frozen, the method is modular: any stronger base language model could replace the 13-billion-parameter backbone and likely inherit the same synthetic-speech training recipe."],"forward_implications":["A spoken VQA model trained only on synthetic speech reaches 61.8 percent accuracy on SEED-Bench, within about 6 points of the text-based LLaVA-1.5 upper bound of 68.2 percent.","The TTS model used for training has a minor effect (about 1-2 points), and cross-TTS, cross-speaker evaluation suggests the model does not simply memorize a synthetic voice.","On document images (DocVQA), the direct speech model (ANLS 0.17) clearly outperforms an ASR-then-VQA pipeline (ANLS 0.07), whose transcription errors cascade into the answer.","Synthesizing the full prompt (question plus multiple-choice answers) as speech consistently hurts accuracy; keeping the structured prompt as text and speaking only the question works better.","The recipe converts an existing text VQA training set into a spoken one, producing two datasets of roughly 3.4 million audio questions each, so the approach can be applied to other public VQA datasets."],"supporting_citations":[{"why":"Supplies StyleTTS2, one of the two zero-shot TTS systems used to synthesize training and test speech.","marker":"[5]"},{"why":"Provides the LLaVA visual instruction tuning architecture that the paper extends with a speech tower.","marker":"[6]"},{"why":"Provides the Whisper encoder, the frozen speech encoder used to embed spoken questions.","marker":"[7]"},{"why":"Provides the CLIP image encoder used to align visual input to the language model.","marker":"[8]"},{"why":"Supplies SEED-Bench, the primary benchmark on which near-parity with the textual upper bound is measured.","marker":"[9]"},{"why":"Supplies DocVQA, the document-image benchmark where the direct speech model outperforms an ASR pipeline.","marker":"[11]"},{"why":"Provides the pre-trained Vicuna-based VQA model and image projector used as the starting point and the textual upper bound.","marker":"[12]"},{"why":"Supplies Multilingual LibriSpeech, the source of the many voices used as TTS prompts and of the natural-speech transcription test set.","marker":"[29]"},{"why":"Motivates the audio description pre-training task that makes the speech projector follow instructions rather than only transcribe.","marker":"[30]"},{"why":"Supplies F5-TTS, the second zero-shot TTS system used for cross-TTS validation.","marker":"[35]"}],"fun_headline_variants":["Synthetic speech trains VQA to near text accuracy","Spoken VQA via synthetic speech: near text accuracy","TTS-trained model answers visual questions by voice","Spoken VQA: synthetic audio rivals text"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that synthetic speech from StyleTTS2 and F5-TTS is representative enough of natural human speech that the near-parity measured on synthetic test sets will transfer to real users, and the paper does not test this with natural speech.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic speech trains VQA to near text accuracy","Spoken VQA via synthetic speech: near text accuracy","TTS-trained model answers visual questions by voice","Spoken VQA: synthetic audio rivals text"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000584,"raw_usage":{"total_tokens":2683,"prompt_tokens":816,"completion_tokens":1867,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":432,"completion_tokens_details":{"reasoning_tokens":1805}},"tokens_in":432,"tokens_out":1867,"duration_ms":16215,"temperature":1.0,"reasoning_tokens":1805,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:47:31.549056+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record human speakers asking the same SEED-Bench, MME, and DocVQA questions about the same images, run the trained SVQA model on those natural utterances, and compare accuracy; if accuracy falls well below the synthetic-test numbers or the word error rate rises sharply, the near-parity claim does not survive contact with real speech.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies StyleTTS2, one of the two zero-shot TTS systems used to synthesize training and test speech."},{"cited_title":"Visual question answering (VQA) attempts to describe, locate, and reason regarding some visual input [9, 10, 11]","cited_arxiv_id":null,"evidence_quote":"Provides the LLaVA visual instruction tuning architecture that the paper extends with a speech tower."},{"cited_title":"The LLaV A model extends a text-based, generative, large language model (LLM) for visual question answering (VQA) by allow- ing visual information input from images","cited_arxiv_id":null,"evidence_quote":"Provides the Whisper encoder, the frozen speech encoder used to embed spoken questions."},{"cited_title":"Speech-only datasets We pre-train the speech projector using the English subset of Multilingual LibriSpeech (MLS) [29]","cited_arxiv_id":null,"evidence_quote":"Provides the CLIP image encoder used to align visual input to the language model."},{"cited_title":"The speech tower is composed of a trained Whis- per encoder2 and a speech projector","cited_arxiv_id":null,"evidence_quote":"Supplies SEED-Bench, the primary benchmark on which near-parity with the textual upper bound is measured."},{"cited_title":"In contrast, our SVQA approach maintains more robust performance across all bench- marks","cited_arxiv_id":null,"evidence_quote":"Supplies DocVQA, the document-image benchmark where the direct speech model outperforms an ASR pipeline."},{"cited_title":"A significant portion of the effort was spent on building both the speech and SVQA datasets","cited_arxiv_id":null,"evidence_quote":"Provides the pre-trained Vicuna-based VQA model and image projector used as the starting point and the textual upper bound."},{"cited_title":"Speech-Based Visual Question Answering","cited_arxiv_id":"1705.00464","evidence_quote":"Supplies Multilingual LibriSpeech, the source of the many voices used as TTS prompts and of the natural-speech transcription test set."},{"cited_title":"Speech enabled visual question answering using lstm and cnn with real time image cap- turing for assisting the visually impaired,","cited_arxiv_id":null,"evidence_quote":"Motivates the audio description pre-training task that makes the speech projector follow instructions rather than only transcribe."},{"cited_title":"A VQA: A dataset for audio- visual question answering on videos,","cited_arxiv_id":null,"evidence_quote":"Supplies F5-TTS, the second zero-shot TTS system used for cross-TTS validation."}],"review_version":1}