Multi-modal and Multi-scale Spatial Environment Understanding for Immersive Visual Text-to-Speech

Haizhou Li; Rui Liu; Shuwei He; Yifan Hu

REVIEW 4 major objections 5 minor 1 cited by

Multi-modal and Multi-scale Spatial Environment Understanding for Immersive Visual Text-to-Speech

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read M2SE-VTTS claims that fusing RGB, depth, and local-global scene context makes visual text-to-speech more immersive, beating five baselines.

desk verdict A plausible multi-modal VTTS extension with an honest architecture, but the evaluation has enough loose ends—especially Topk selection and missing error bars—that it needs committee scrutiny before the SOTA claim holds. read the letter →

arxiv 2412.11409 v3 pith:4WJ5WECE submitted 2024-12-16 cs.CV cs.AIcs.MM

Rui Liu , Shuwei He , Yifan Hu , Haizhou Li This is my paper · ORCID

classification cs.CVcs.AIcs.MM

keywords visualtext-to-speechreverberationspatialenvironmentunderstandingRGB-Dmulti-scaleattentionCLIPdiffusionTTSSoundSpaces-Speech

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Visual text-to-speech (VTTS) turns an image of a room into speech that sounds as if it were spoken there, complete with the room's reverberation. Previous systems read only the global RGB content of the image; this paper argues that local patches and depth information matter for acoustics—where the speaker stands, what surfaces absorb or reflect sound. To test this, M2SE-VTTS encodes both RGB and depth images with a frozen CLIP vision transformer, uses Gemini-generated captions to pick the most semantically relevant image patches, and then blends local patch-level features with global scene features through attention. The paper reports that on the SoundSpaces-Speech benchmark it beats all five baselines on every metric, including room reverb error (RTE 0.0744), mel cepstral distortion (MCD 4.4215), and listener-perceived quality (MOS 3.849 ± 0.025). If true, the result suggests that a TTS system needs to 'see' the room's geometry as well as its content to synthesize believable acoustics.

What carries the argument

The load-bearing device is the Top-k patch-selection mechanism with shared indices: the Topk RGB Regions Detector uses multi-head attention between the frozen CLIP caption embedding and RGB patch embeddings to produce attention weights, picks the k=140 highest-weight patches, and the Topk Depth Regions Selector then applies those same indices to the depth patch embeddings. This shared-index design is what the paper argues keeps RGB and depth spatially aligned and avoids redundancy. On top of it, Local-aware RGB/Depth Attention cross-attends local patches to the global [CLS] feature, and Semantic-Guided RGB/Depth Attention conditions the result on the caption embedding; the two modalities are combined as $H_V = \lambda_1 H_G^R + \lambda_2 H_G^D$ with $\lambda_1=\lambda_2=0.5$. This feature vector $H_V$ is fed into the ViT-TTS diffusion-transformer backbone in place of the usual global image feature.

What would settle it

Measure whether the Top-k depth-selected patches actually correspond to surfaces that dominate the room's measured reverberation (e.g., large reflective walls or absorbent carpets), by correlating patch locations with per-surface absorption data or by testing a depth-guided variant whose patch indices come from depth attention rather than RGB attention; if the depth stream adds no measurable gain over an RGB-only model that already has captions and local attention, the central multi-modal claim fails.

Watch

Extended reading notes

Core claim

The central discovery claimed here is that multi-modal and multi-scale spatial environment understanding improves immersive VTTS. The paper argues that global RGB features are not enough: local image patches identify sound-relevant objects (e.g., a table versus a carpet), while the depth channel encodes room geometry, speaker position, and object arrangement. M2SE-VTTS operationalizes this by selecting the Top-k patches from RGB attention guided by Gemini captions, reusing those same patch indices to select depth features, and then applying two rounds of cross-attention—local-aware attention between local patches and the global [CLS] token, and semantic-guided attention that injects caption semantics—before fusing RGB and depth representations with equal weight. The evaluation on SoundSpaces-Speech reports consistent gains over ProDiff, DiffSpeech, VoiceLDM, ViT-TTS-ResNet18, and ViT-TTS-CLIP, with the full model achieving the best RTE, MCD, and MOS on both seen and unseen rooms. Ablations attribute the gains to the RGB stream, the depth stream, the Gemini caption guidance, the local-region selection, and the two attention stages.

Load-bearing premise

The depth stream leans entirely on the assumption that the k image patches judged most relevant by RGB-text attention are exactly the patches whose depth content matters for acoustics; if that correspondence fails, the depth channel introduces no independent spatial knowledge.

Editorial extensions

If this is right

Visual TTS systems should ingest depth as well as RGB, since the paper's ablations show removing either modality hurts both objective and subjective scores.
Using a semantic caption to pick local regions is more effective than picking regions by attention alone; the "w/o LLM" ablation shows a clear drop.
Sharing patch indices across RGB and depth yields better results than independent selection for every tested Top-k value, suggesting spatial alignment between modalities is key.
The model's superiority over diffusion TTS baselines (ProDiff, DiffSpeech) indicates that content-only TTS models cannot capture environmental reverb; environment-aware conditioning is necessary.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural testable extension is to replace the Gemini captions with a stronger or weaker captioner, or to ablate caption quality, since the paper only reports a binary with/without caption condition; the sensitivity of the shared-index mechanism to caption quality is unknown.
The shared-index assumption could be stressed in rooms where the acoustic-critical surfaces (e.g., large glass walls) are visually low-salience in RGB; a dedicated benchmark with such rooms would clarify whether depth is truly being used or simply copied from RGB.
Because the fusion weights are fixed at 0.5, the method cannot adapt to rooms where one modality carries more acoustic information; learning $\lambda_1,\lambda_2$ from the input or using a gating mechanism could improve generalization to unseen environments.
The approach may transfer to audio-visual navigation or dereverberation, where room geometry estimated from depth is a known useful prior.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Rui, here's my read on M2SE-VTTS. The core idea is sound: take the ViT-TTS backbone and add depth images plus Gemini-generated captions to guide local patch selection, then fuse local and global multi-modal features. That combination is genuinely new relative to the ViT-TTS and VoiceLDM baselines, and the paper shows the architecture clearly. The ablations are thoughtful—removing depth, the LLM guidance, local selection, or the local-global interaction all hurt, which is the right kind of evidence that the pieces matter. Code and audio samples are promised, which is more than many papers in this area ship.

The soft spots are real but not fatal. The biggest one is the Topk selection. The paper says Topk is set to 140 after testing values from 20 to 240 in the 'Topk Index Sharing Comparative Study,' but it never states which split that sweep ran on. Since the ablation table and main results are on test-unseen, if the sweep used test-unseen, the headline numbers are not clean hold-out numbers. That needs to be stated explicitly. Relatedly, the objective metrics are computed on 50 samples with no error bars or significance tests. The RTE gap over ViT-TTS-ResNet18 is 0.0015—tiny—so without variance we can't tell if the depth stream is actually helping or just adding noise. The VoiceLDM comparison is also not a faithful reproduction; the authors say they concentrated on their own text-guidance approach, which makes that baseline comparison apples-to-oranges. And their own earlier multi-source preprint (He et al. 2024) is cited but never compared, which is a gap the authors should close or explain.

The stress-test concern about data snooping is worth taking seriously, but I don't think it's a smoking gun. Nothing in the paper explicitly says the sweep used test-unseen, and it's plausible they used the validation set. The design choice of sharing RGB-derived indices for depth selection is risky—it assumes the RGB-text alignment picks out the acoustically relevant regions—but the ablations suggest it works, and it's a modeling choice, not circular reasoning.

Bottom line: this is a solid, incremental contribution to a narrow subfield. It deserves a serious referee, but the authors should be asked to report split usage for hyperparameter selection, add error bars or significance tests, and either reproduce VoiceLDM faithfully or drop it. I'd bring it to reading group—the architecture discussion alone is worth forty minutes—and I'd cite it if I were working on VTTS, though I'd wait for the revision before trusting the exact numbers.

Referee Report

4 major / 5 minor

Summary. The paper proposes M2SE-VTTS, a visual text-to-speech system that conditions speech synthesis on both RGB and depth images of the target environment. The method extracts CLIP-ViT patch features, uses Gemini-generated environment captions to identify salient RGB regions, shares those region indices to select depth patches, and then fuses local and global features through two attention stages before feeding the resulting environment representation into a ViT-TTS backbone with BigVGAN. Experiments on SoundSpaces-Speech compare the model with five baselines and report MOS, RTE, and MCD on test-unseen and test-seen splits, together with ablations and a study of the Topk index-sharing strategy. The authors claim that M2SE-VTTS consistently outperforms all baselines on both splits.

Significance. If the reported results are clean, the paper makes a useful contribution to visual text-to-speech by showing that depth information and local/global multi-scale reasoning can improve reverberation modeling. The architecture is clearly motivated, the code and audio are promised to be public, and the ablation suite covers the main components. However, the significance depends on the integrity of the evaluation protocol, and several load-bearing details of that protocol are currently unspecified, most importantly the data split used for selecting the Topk hyperparameter. The paper is therefore of interest to the VTTS and audio-visual scene understanding communities, but the experimental evidence as presented is not yet conclusive.

major comments (4)

[Implementation Details and Topk Index Sharing Comparative Study, Eq. (2)] The manuscript does not state which data split was used to choose Topk=140. Section 'Implementation Details' says the patch number Topk is set to 140, while Figure 2 sweeps Topk values from 20 to 240; Table 1 then reports test-unseen and test-seen numbers. If any part of that sweep used test-unseen samples, the reported RTE 0.0744 and MCD 4.4215 are not honest hold-out results and the headline claim of consistent superiority is invalidated. Please state explicitly which split was used for the sweep. If test data were used, re-run the hyperparameter selection on a validation split and re-report Tables 1 and 2 with the resulting Topk.
[Evaluation Metrics, Tables 1 and 2] Objective metrics are computed on only 50 randomly selected samples, with no error bars, confidence intervals, or significance tests. The gaps against the strongest visual baseline are small (e.g., RTE gap of 0.0015 and MCD gap of 0.1717 versus ViT-TTS-ResNet18 on test-unseen). Without per-sample variance or paired significance tests, the claim that M2SE-VTTS 'consistently outperforms all baseline systems' is not statistically supported. Please report bootstrap confidence intervals or paired significance tests for RTE and MCD, and state the random seed or repeat the evaluation over multiple subsets.
[Baselines, VoiceLDM paragraph] The VoiceLDM baseline is not the original model as published. The paragraph states that, because of differences in environmental text descriptions, the authors 'choose to concentrate on the model's novel method of leveraging textual descriptions to guide the synthesis of reverberation speech during code reproduction.' This means the comparison in Table 1 may be against a modified reimplementation rather than the original VoiceLDM. Either reproduce VoiceLDM faithfully from its released code and settings, or clearly label the baseline as an adapted variant and explain why the modification is a fair comparison.
[Eq. (1)-(2), Topk RGB Regions Detector] The tensor shapes in the local spatial understanding block are inconsistent as written. In Eq. (1), MultiHead(F_C^S, F_R^P, F_R^P) uses the caption CLS token as the query and the M RGB patch features as keys and values, so the attention output is a single vector of shape (1, D). Yet Eq. (2) treats \hat{F}_R^P as a patch-level collection from which Topk rows are selected to form H_R_Topk of shape (Topk, D). Please clarify the actual implementation: are the selected indices applied to the original patch features F_R^P, or is \hat{F}_R^P defined differently? As written, the local feature selection step is undefined and the method cannot be reproduced from the paper.

minor comments (5)

[Figure 2] The figure has no visible axis labels or legend, and the phrases 'contribution parameter settings' and 'comparison settings' in the caption are unclear. Please label the axes and define the two line styles.
[Notation throughout] The notation for the hyperparameter is inconsistent: the text uses 'Topk', 'Top k', and 'T opk' interchangeably. Please use a single symbol, for example K, throughout.
[Eq. (8)] The fusion weights lambda1 and lambda2 are both fixed at 0.5 without any sensitivity analysis. A short experiment or statement that results are insensitive to these weights would strengthen the paper.
[Footnote and grammar] The footnote contains the typo 'Corrposending Author' and the abstract contains the phrase 'an spatial image'; these should be corrected.
[Appendix reference] The reference to Appendix A uses a short URL. Please provide the full appendix content or a stable pointer that reviewers and readers can access.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the model's components are forward-computed from external image and caption inputs and evaluated against held-out audio and external baselines.

full rationale

The paper makes no derivation that reduces to its own inputs. RGB and Depth patch features are extracted from a frozen CLIP encoder, Gemini captions are exogenous inputs, and Eqs. (1)-(8) are forward computations whose outputs are compared with held-out ground truth on SoundSpaces-Speech. The Topk depth selector in Eq. (3) does reuse RGB-derived patch indices, but this is an explicit modeling design documented in the text, not a disguised prediction of the evaluation metrics; the depth stream is not claimed to be independently selected. Self-citations (e.g., the ViT-TTS backbone, preprocessing conventions, and prior VTTS work) are used as implementation baselines or dataset construction references, and none carries a uniqueness theorem or is the sole justification for the reported advantage. The only validity concern visible in the text is that the Topk sweep in Figure 2 is not stated to be restricted to a validation split, which could affect whether the reported test numbers are clean hold-out estimates; that is an experimental-leakage and soundness issue, not circularity. Against external baselines and external metrics, the central comparison is self-contained.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The method is an empirical system, so the ledger is modest. The central claim rests on two hand-set hyperparameters and on four domain assumptions about depth encoding, cross-modal index sharing, Gemini caption reliability, and dataset realism. No new physical entities or conserved quantities are introduced.

free parameters (2)

Topk = 140
Number of local RGB and depth patches selected per image. Chosen by sweeping values from 20 to 240 and fixing at 140 where objective metrics plateau or decline. If the sweep was done on the test-unseen set, the reported numbers use a test-selected hyperparameter.
lambda1 and lambda2 fusion weights = 0.5 each
Weights for combining RGB and depth global features in Eq. (8). Set to 0.5 by hand with no tuning or sensitivity analysis.

assumptions (5)

domain assumption SoundSpaces-Speech simulated reverberation is a valid proxy for real room acoustics.
All evaluations use simulated audio. If the simulator's acoustics are unrealistic, gains may not transfer to real rooms. Invoked in the Dataset section.
domain assumption CLIP, pretrained on RGB-text pairs, produces meaningful patch features when applied to depth images.
The depth encoder is the same frozen CLIP-ViT as the RGB encoder. This transfer is assumed, not tested. Invoked in Multi-modal Features Extraction.
domain assumption RGB-attention-selected Topk patch indices identify depth regions that matter for acoustics.
Sharing indices across modalities is a design choice justified by three heuristic reasons, not by an experiment. Invoked in Eq. (3), Topk Depth Regions Selector.
domain assumption Gemini Pro Vision will be available at inference and its captions are accurate enough to guide local region selection.
The method imports a proprietary external model. No open captioner is evaluated or provided as a fallback. Invoked in LLM-based Spatial Semantic Understanding.
domain assumption High caption-attention patches correspond to surfaces and objects that influence sound reflection and absorption.
The Topk detector selects patches by attention to caption semantics, but there is no direct acoustic supervision tying these patches to RT60. Invoked in Topk RGB Regions Detector.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Multi-modal and Multi-scale Spatial Environment Understanding for Immersive Visual Text-to-Speech." pith.science (2026). https://pith.science/paper/4WJ5WECE

@misc{pith2026241211409,
  author       = {Pith},
  title        = {Pith review of: Multi-modal and Multi-scale Spatial Environment Understanding for Immersive Visual Text-to-Speech},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4WJ5WECE}},
  note         = {Machine review of arXiv:2412.11409}
}

read the original abstract

Visual Text-to-Speech (VTTS) aims to take the environmental image as the prompt to synthesize the reverberant speech for the spoken content. The challenge of this task lies in understanding the spatial environment from the image. Many attempts have been made to extract global spatial visual information from the RGB space of an spatial image. However, local and depth image information are crucial for understanding the spatial environment, which previous works have ignored. To address the issues, we propose a novel multi-modal and multi-scale spatial environment understanding scheme to achieve immersive VTTS, termed M2SE-VTTS. The multi-modal aims to take both the RGB and Depth spaces of the spatial image to learn more comprehensive spatial information, and the multi-scale seeks to model the local and global spatial knowledge simultaneously. Specifically, we first split the RGB and Depth images into patches and adopt the Gemini-generated environment captions to guide the local spatial understanding. After that, the multi-modal and multi-scale features are integrated by the local-aware global spatial understanding. In this way, M2SE-VTTS effectively models the interactions between local and global spatial contexts in the multi-modal spatial environment. Objective and subjective evaluations suggest that our model outperforms the advanced baselines in environmental speech generation. The code and audio samples are available at: https://github.com/AI-S2-Lab/M2SE-VTTS.

Figures

Figures reproduced from arXiv: 2412.11409 by the authors.

**Figure 2.** The horizontal axis represents the Topk values, with the left plot showing MCD and the right plot displaying RTE. Green lines denote contribution parameter settings, while blue lines indicate comparison settings. understanding component led to a reduction in all subjective and objective metrics. This demonstrates that spatial images, when analyzed by Gemini for semantic understanding, enable our model to achieve a … view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Towards Expressive Video Dubbing with Multiscale Multimodal Context Interaction
cs.MM 2024-12 conditional novelty 6.0 of 10

M2CI-Dubber improves dubbing prosody by extracting global sentence-level and local phoneme-level features from multimodal context and fusing them with the current text through attention and graph interaction.

Reference graph

Works this paper leans on

38 extracted references · 29 canonical work pages · cited by 1 Pith paper

[1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Chen, C.; Gao, R.; Calamia, P.; and Grauman, K. 2022. Visual acoustic matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 18858--18868

work page 2022
[4]

Chen, C.; Jain, U.; Schissler, C.; Gari, S. V. A.; Al-Halah, Z.; Ithapu, V. K.; Robinson, P.; and Grauman, K. 2020. Soundspaces: Audio-visual navigation in 3d environments. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part VI 16, 17--36. Springer

work page 2020
[5]

Chen, C.; Sun, W.; Harwath, D.; and Grauman, K. 2023. Learning audio-visual dereverberation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1--5. IEEE

work page 2023
[6]

Cheng, A.-C.; Yin, H.; Fu, Y.; Guo, Q.; Yang, R.; Kautz, J.; Wang, X.; and Liu, S. 2024 a . Spatial RGPT : Grounded Spatial Reasoning in Vision-Language Models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

work page 2024
[7]

Cheng, J.; Yin, W.; Wang, K.; Chen, X.; Wang, S.; and Yang, X. 2024 b . Adaptive fusion of single-view and multi-view depth for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10138--10147

work page 2024
[8]

Chowdhury, S.; Nag, S.; Dasgupta, S.; Chen, J.; Elhoseiny, M.; Gao, R.; and Manocha, D. 2024. Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time

work page 2024

Show all 38 references

[9]

Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Confere...

2021
[10]

Fu, R.; Liu, J.; Chen, X.; Nie, Y.; and Xiong, W. 2024. Scene-llm: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403.11401

2024 arXiv
[11]

Guo, M.-H.; Lu, C.-Z.; Hou, Q.; Liu, Z.; Cheng, M.-M.; and Hu, S.-M. 2022. Segnext: Rethinking convolutional attention design for semantic segmentation. Advances in Neural Information Processing Systems, 35: 1140--1156

2022
[12]

He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770--778

2016
[13]

He, S.; Liu, R.; and Li, H. 2024. Multi-Source Spatial Knowledge Understanding for Immersive Visual Text-to-Speech. arXiv preprint arXiv:2410.14101

2024 arXiv
[14]

Huang, R.; Zhao, Z.; Liu, H.; Liu, J.; Cui, C.; and Ren, Y. 2022. Prodiff: Progressive fast diffusion model for high-quality text-to-speech. In Proceedings of the 30th ACM International Conference on Multimedia, 2595--2605

2022
[15]

Im, J.; and Nam, J. 2024. DiffRENT: A Diffusion Model for Recording Environment Transfer of Speech. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 7425--7429. IEEE

2024
[16]

I Want to Figure Things Out

Jain, G.; Teng, Y.; Cho, D. H.; Xing, Y.; Aziz, M.; and Smith, B. A. 2023. "I Want to Figure Things Out": Supporting Exploration in Navigation for People with Visual Impairments. Proceedings of the ACM on Human-Computer Interaction, 7(CSCW1): 1--28

2023
[17]

Jiang, H.; Cheng, T.; Gao, N.; Zhang, H.; Lin, T.; Liu, W.; and Wang, X. 2024. Symphonize 3d semantic scene completion with contextual instance queries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 20258--20267

2024
[18]

Karthik, S.; Roth, K.; Mancini, M.; and Akata, Z. 2024. Vision-by-Language for Training-Free Compositional Image Retrieval. International Conference on Learning Representations (ICLR)

2024
[19]

Lee, S.-g.; Ping, W.; Ginsburg, B.; Catanzaro, B.; and Yoon, S. 2022. Bigvgan: A universal neural vocoder with large-scale training. arXiv preprint arXiv:2206.04658

2022 arXiv
[20]

Lee, Y.; Yeon, I.; Nam, J.; and Chung, J. S. 2024. VoiceLDM: Text-to-Speech with Environmental Context. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 12566--12571. IEEE

2024
[21]

Li, G.; Hou, W.; and Hu, D. 2023. Progressive spatio-temporal perception for audio-visual question answering. In Proceedings of the 31st ACM International Conference on Multimedia, 7808--7816

2023
[22]

Liu, H.; Chen, Z.; Yuan, Y.; Mei, X.; Liu, X.; Mandic, D.; Wang, W.; and Plumbley, M. D. 2023 a . AudioLDM: text-to-audio generation with latent diffusion models. In Proceedings of the 40th International Conference on Machine Learning, 21450--21474

2023
[23]

Liu, H.; Huang, R.; Lin, X.; Xu, W.; Zheng, M.; Chen, H.; He, J.; and Zhao, Z. 2023 b . V i T - TTS : Visual Text-to-Speech with Scalable Diffusion Transformer. In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of the 2023 Conference on Empirical Methods in Natural Lan...

2023
[24]

Liu, J.; Li, C.; Ren, Y.; Chen, F.; and Zhao, Z. 2022. Diffsinger: Singing voice synthesis via shallow diffusion mechanism. In Proceedings of the AAAI conference on artificial intelligence, volume 36, 11020--11028

2022
[25]

Liu, R.; Hu, Y.; Ren, Y.; Yin, X.; and Li, H. 2024 a . Emotion rendering for conversational speech synthesis with heterogeneous graph-based context modeling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 18698--18706

2024
[26]

Liu, R.; Sisman, B.; Gao, G.; and Li, H. 2024 b . Controllable Accented Text-to-Speech Synthesis With Fine and Coarse-Grained Intensity Rendering. IEEE/ACM Transactions on Audio, Speech, and Language Processing

2024
[27]

Majumder, S.; Chen, C.; Al-Halah, Z.; and Grauman, K. 2022. Few-shot audio-visual learning of environment acoustics. Advances in Neural Information Processing Systems, 35: 2522--2536

2022
[28]

W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al

Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748--8763. PMLR

2021
[29]

Ren, Y.; Hu, C.; Tan, X.; Qin, T.; Zhao, S.; Zhao, Z.; and Liu, T.-Y. 2021. FastSpeech 2: Fast and High-Quality End-to-End Text to Speech. In International Conference on Learning Representations (ICLR)

2021
[30]

Ren, Y.; Ruan, Y.; Tan, X.; Qin, T.; Zhao, S.; Zhao, Z.; and Liu, T.-Y. 2019. Fastspeech: Fast, robust and controllable text to speech. Advances in neural information processing systems, 32

2019
[31]

Somayazulu, A.; Chen, C.; and Grauman, K. 2024. Self-supervised visual acoustic matching. Advances in Neural Information Processing Systems, 36

2024
[32]

N.; Tran, S.; Yao, B.; Chilimbi, T.; and Shah, M

Swetha, S.; Yang, J.; Neiman, T.; Rizve, M. N.; Tran, S.; Yao, B.; Chilimbi, T.; and Shah, M. 2024. X-Former: Unifying Contrastive and Reconstruction Learning for MLLMs. In European Conference on Computer Vision (ECCV)

2024
[33]

Tan, D.; Zhang, G.; and Lee, T. 2022. Environment Aware Text-to-Speech Synthesis . In Proc. Interspeech 2022, 481--485

2022
[34]

I.; Burnell, R.; Bai, L.; Gulati, A.; Tanzer, G.; Vincent, D.; Pan, Z.; Wang, S.; et al

Team, G.; Georgiev, P.; Lei, V. I.; Burnell, R.; Bai, L.; Gulati, A.; Tanzer, G.; Vincent, D.; Pan, Z.; Wang, S.; et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530

2024 arXiv
[35]

Wang, Z.; Song, S.; Luo, C.; Deng, S.; Xie, W.; and Shen, L. 2024. Multi-scale Dynamic and Hierarchical Relationship Modeling for Facial Action Units Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1270--1280

2024
[36]

Xu, D.; Li, Z.; Feng, H.; Wu, F.; and Wang, Y. 2024. Multi-Scale Feature Fusion Network with Symmetric Attention for Land Cover Classification Using SAR and Optical Images. Remote Sensing, 16(6)

2024
[37]

C.; and YAN, S

Zhang, T.; Li, X.; Fei, H.; Yuan, H.; Wu, S.; Ji, S.; Loy, C. C.; and YAN, S. 2024. OMG - LL a VA : Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

2024
[38]

Zhu, L.; Wei, F.; and Lu, Y. 2024. Beyond text: Frozen large language models in visual signal comprehension. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 27047--27057

2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Chen, C.; Gao, R.; Calamia, P.; and Grauman, K. 2022. Visual acoustic matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 18858--18868

work page 2022

[4] [4]

Chen, C.; Jain, U.; Schissler, C.; Gari, S. V. A.; Al-Halah, Z.; Ithapu, V. K.; Robinson, P.; and Grauman, K. 2020. Soundspaces: Audio-visual navigation in 3d environments. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part VI 16, 17--36. Springer

work page 2020

[5] [5]

Chen, C.; Sun, W.; Harwath, D.; and Grauman, K. 2023. Learning audio-visual dereverberation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1--5. IEEE

work page 2023

[6] [6]

Cheng, A.-C.; Yin, H.; Fu, Y.; Guo, Q.; Yang, R.; Kautz, J.; Wang, X.; and Liu, S. 2024 a . Spatial RGPT : Grounded Spatial Reasoning in Vision-Language Models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

work page 2024

[7] [7]

Cheng, J.; Yin, W.; Wang, K.; Chen, X.; Wang, S.; and Yang, X. 2024 b . Adaptive fusion of single-view and multi-view depth for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10138--10147

work page 2024

[8] [8]

Chowdhury, S.; Nag, S.; Dasgupta, S.; Chen, J.; Elhoseiny, M.; Gao, R.; and Manocha, D. 2024. Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time

work page 2024