Pith. sign in

REVIEW 2 major objections 5 minor 266 references

A continuous 12.5 Hz audio VAE can reconstruct speech, music, and sound well while encoding 32 minutes of audio in 541 ms, making it a practical backbone for large-scale text-to-audio training.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 03:30 UTC pith:SXLICWPK

load-bearing objection Solid systems report on a 12.5 Hz continuous audio VAE: reconstruction and encoder speed are real; the “scalable T2A backbone” claim is only lightly evidenced. the 2 major comments →

arxiv 2607.11738 v1 pith:SXLICWPK submitted 2026-07-13 eess.AS

Qwen-Audio-VAE Technical Report

Ziyue Jiang , Dake Guo , Zekai Zhang , Hangrui Hu , Ting He , Xinfa Zhu , Xiong Wang , Yongqi Wang
show 6 more authors
Jiapeng Wang Wenxiang Guo Zhifang Guo Chenfei Wu Dayiheng Liu Jin Xu
This is my paper
classification eess.AS
keywords audio VAEcontinuous latenttext-to-audiolow frame rateencoder pruningmulti-discriminatorspeech music sound reconstructionlatent diffusion backbone
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

General audio generators do not model waveforms directly; they learn in the latent space of an autoencoder. That autoencoder therefore has to do three jobs at once: reconstruct diverse audio with high fidelity, keep the latent sequence short so diffusion transformers stay cheap, and encode audio fast enough that online latent extraction does not bottleneck training. This paper claims that a continuous variational autoencoder running at 12.5 Hz, trained on five million hours of mixed speech, music, and sound, can meet all three requirements. The design pairs a causal convolutional encoder–decoder with window Transformers placed only at the lowest frame rate, multi-discriminator adversarial training, an asymmetric decoder, and latency-aware encoder pruning. On public speech, sound, and music reconstruction benchmarks it is the strongest model among the low-frame-rate continuous systems compared, and the encoder speedup lets text-to-audio training use larger batches and reach higher CLAP scores under the same wall-clock budget.

Core claim

Qwen-Audio-VAE shows that a continuous diagonal-Gaussian audio latent at 12.5 Hz and 128 dimensions, produced by a causal encoder with window Transformers only at the bottleneck, multi-discriminator training, and an asymmetric decoder, reconstructs speech, music, and general sound competitively with higher-rate systems while remaining the strongest low-frame-rate continuous VAE on the reported benchmarks; latency-aware encoder pruning then cuts 64×30 s encoding from 1957 ms to 541 ms without degrading reconstruction, which in turn supports larger text-to-audio batches and better generation metrics under fixed training time.

What carries the argument

The 12.5 Hz continuous latent bottleneck flanked by window Transformers, together with asymmetric encoder–decoder capacity and first-layer-only channel reduction: the Transformers supply long-range context exactly where sequences are shortest, while pruning and asymmetry keep high-resolution encoding cheap without collapsing reconstruction.

Load-bearing premise

That strong reconstruction on three public benchmarks plus a single medium-size diffusion transformer trained on one filtered audio-caption set is enough to prove the latent is a scalable backbone for large-scale general text-to-audio generation.

What would settle it

Train a substantially larger multi-domain diffusion transformer on the same latents versus a matched higher-frame-rate continuous VAE under identical compute and data; if generation quality and sample efficiency do not improve or stay competitive, the claim that this compact fast latent is a superior generation backbone fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Downstream diffusion transformers can operate on roughly 4–6× shorter sequences than common 50–75 Hz codecs, cutting quadratic attention cost.
  • Online latent extraction no longer limits batch size: the same wall-clock step can process several times more audio.
  • A single continuous representation trained on mixed speech, music, and sound can serve as a shared backbone instead of domain-specific codecs.
  • Light KL regularization is preferable to strong prior matching if the goal is generative learnability of the latent.
  • Future generators can treat encoding throughput as a first-class design constraint alongside reconstruction fidelity.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Aligning this latent with a pretrained audio embedding (for example a contrastive language-audio model) may improve DiT fit more than further reconstruction tuning alone.
  • The same first-layer-only pruning pattern is likely portable to other high-rate continuous modalities where early layers dominate latency.
  • If high-frequency and transient errors remain the main residual gap, a hybrid continuous–discrete residual or multi-band latent could close it without lengthening the sequence.
  • The reported 5 M-hour multi-domain mix may be doing as much work as the architecture; ablations that hold data fixed while swapping only the VAE would isolate the contribution more cleanly.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. Qwen-Audio-VAE is a continuous 24 kHz mono audio VAE that maps waveforms to a 12.5 Hz, 128-d diagonal-Gaussian latent via a causal convolutional encoder, a continuous bottleneck, window Transformers at the low frame rate, and an asymmetric higher-capacity decoder. It is trained with multi-scale spectral losses plus a bank of multi-period, multi-resolution STFT, multi-scale STFT, and sub-band CQT discriminators on ~5M hours of speech/music/sound. Latency-aware encoder pruning (stride reallocation, residual-unit pruning, first-layer channel reduction) reduces 64×30 s encoding from 1957 ms to 541 ms. Reconstruction is evaluated on LibriSpeech, AudioCaps, and SongDescriber against discrete codecs and continuous VAEs; a controlled 150M DiT experiment on filtered LAION-Audio-630K shows that faster encoding enables larger batches and higher AudioCaps CLAP under fixed wall-clock time.

Significance. If the results hold, the paper supplies a practical continuous audio latent that jointly targets reconstruction fidelity, sequence length for DiT generators, and online encoding throughput—three requirements that prior codecs and VAEs usually trade off. The staged encoder acceleration (Table 8) with preserved reconstruction, the multi-domain 5M-hour training scale, and the explicit batch-size/throughput T2A experiment are concrete engineering contributions that other labs can adopt. Strengths include public-benchmark reconstruction tables grouped by frame rate, ablations isolating the encoder Transformer, latent dimension, and frame rate (Table 7), and transparent reporting that stronger KL regularization hurts generation (Table 10). The work is primarily systems/empirical rather than theoretical, but the efficiency–quality operating point is useful for scalable text-to-audio pipelines.

major comments (2)
  1. §4.5 / Tables 9–10: The abstract and conclusion present Qwen-Audio-VAE as a “scalable general audio generation” backbone, but the only generative evidence is a single 150M DiT on filtered LAION-Audio-630K (~4.3k h). The CLAP gain (0.29→0.33) is driven mainly by 4× larger batch from faster encoding, not by a demonstration that the 12.5 Hz/128-d latent is learnable or high-quality at large DiT scale or multi-domain generation. Either temper the backbone claim to match the reported experiment, or add larger-scale / multi-domain generation results.
  2. Tables 4–6 and §4.2: The claim of being “consistently the best low-frame-rate model” is only partially supported. On AudioCaps, Qwen-Audio-VAE has worse MR-STFT than Stable Audio Open (2.315 vs 1.533); on SongDescriber it trails Hunyuan-Foley on Mel/PESQ/STOI. The paper notes that spectral and perceptual metrics can diverge, but the abstract/conclusion still package a uniform “strongest low-frame-rate reconstructor” claim. Qualify the claim by metric and domain, or report additional perceptual/listening evidence where objective metrics conflict.
minor comments (5)
  1. Figure 1 caption and abstract: “541 ms for 32 min of audio” matches 64×30 s; state the batch/workload explicitly so readers do not misread it as single-stream real-time latency.
  2. §2.3 / Eq. (2): λ_mel is mentioned but never given a numeric value alongside λ_stft=20, λ_adv=λ_fm=1, λ_kl=10^{-6}; add it for reproducibility.
  3. Table 5: STOI on general sound is less standard than on speech; a short note on interpretation (or an alternative intelligibility/perceptual metric) would help.
  4. Appendix Table 11 vs §2.4: encoder first-layer width is listed as 24 after pruning; ensure the main configuration table and the acceleration narrative are consistent about pre- vs post-pruning channels.
  5. Title/header typography: “Qwen-Audio-V AE” appears with a space in several places; normalize to Qwen-Audio-VAE.

Circularity Check

0 steps flagged

No circularity: empirical systems paper whose claims are measured on external public benchmarks and wall-clock timings, not derived from self-defined quantities.

full rationale

Qwen-Audio-VAE is an engineering report: a causal convolutional VAE with window Transformers, multi-discriminator adversarial training, asymmetric encoder–decoder, and latency-aware pruning, trained on 5 M hours of multi-domain audio. Its load-bearing claims (best low-frame-rate reconstruction on LibriSpeech/AudioCaps/SongDescriber among continuous VAEs; 3.62× encoding speedup from 1957 ms to 541 ms; modest CLAP lift from larger batches) are established by direct measurement against public datasets and prior models (DAC, EnCodec, Stable Audio Open, etc.), not by algebraic reduction of inputs. Loss weights (λ_stft=20, λ_kl=10^{-6}), strides, channel widths, and pruning steps are design choices validated by ablations (Tables 1–2, 7–8) and external metrics (PESQ, STOI, Mel, MR-STFT, CLAP); none is a fitted parameter renamed as a prediction, nor is any uniqueness theorem or self-citation used to force the result. Self-citations (Qwen multimodal reports) appear only as background and do not underwrite the reconstruction or latency numbers. The paper therefore contains no self-definitional loops, fitted-input-as-prediction steps, or load-bearing self-citation chains; score 0 is the correct outcome for a self-contained empirical systems paper.

Axiom & Free-Parameter Ledger

8 free parameters · 5 axioms · 1 invented entities

This is an empirical ML systems paper. Load-bearing content is architectural choices, loss weights, and a large proprietary corpus rather than mathematical axioms or new physical entities. Free parameters are the usual hyperparameter surface of a VAE-GAN; domain assumptions are standard in latent generative audio; no new particles or forces are postulated.

free parameters (8)
  • Latent frame rate and dimension (12.5 Hz, 128-d)
    Chosen operating point; ablations show 50 Hz and 64-d alternatives change quality/cost. Central claims depend on this choice.
  • Loss weights λ_mel, λ_stft=20, λ_adv=1, λ_fm=1, λ_kl=1e-6
    Hand-set weights that define the training objective; KL weight is shown to affect generation (Table 10).
  • Encoder first-layer width reduced to 24 channels
    Latency-critical design choice; full-channel reduction collapses quality (Table 1), so only first layer is shrunk.
  • Encoder strides (8,5,4,3) and residual-unit pruning (single dilation-1 unit)
    Re-allocated for latency; intermediate speedups reported in Table 8.
  • Decoder base width 1536 and 3 residual units with dilations {1,3,9}
    Asymmetric capacity allocation not derived; chosen because decoding is off the critical path.
  • Window Transformer config (8 layers, dim 1024, 16 heads, window 72)
    Capacity and context hyperparameters placed at the bottleneck; removal hurts reconstruction (Table 7).
  • Discriminator bank periods/FFT sizes/CQT bins
    Adversarial critics fully specified in Appendix Table 13; composition is a design choice that shapes reconstruction.
  • Domain mixture and sampling weights over ~5M hours
    Speech/music/sound hours and quality stratification are curated choices that determine domain robustness claims.
axioms (5)
  • domain assumption Continuous diagonal-Gaussian latents without vector quantization are preferable for diffusion/flow generators because they avoid quantization error and match continuous trajectories.
    Stated in §1–2.2 as motivation versus discrete codecs; standard in latent diffusion audio but not proved here.
  • domain assumption Downstream DiT cost scales primarily with latent sequence length (O(L²) attention), so lower frame rate is the main lever on generator training cost.
    §2.1; true for full self-attention DiTs but depends on generator architecture (windowing, linear attention, etc.).
  • domain assumption Encoder latency, not decoder latency, is the critical-path bottleneck for large-scale online latent extraction in text-to-audio training.
    §2.4 and §4.4; motivates asymmetric capacity and encoder-only pruning.
  • domain assumption Multi-scale spectral ℓ1 plus multi-discriminator adversarial and feature-matching losses are adequate proxies for perceptual reconstruction quality across speech, music, and sound.
    §2.3; standard VAE-GAN practice; paper acknowledges single metrics can miss muffling.
  • standard math Standard VAE reparameterization and closed-form diagonal-Gaussian KL to N(0,I) are valid training machinery.
    Kingma & Welling formulation used in Eq. (1) and L_kl.
invented entities (1)
  • Qwen-Audio-VAE model family no independent evidence
    purpose: Named continuous audio autoencoder suite serving as the paper’s representation backbone.
    An engineered model, not a new physical or mathematical entity; listed for completeness of what the paper introduces.

pith-pipeline@v1.1.0-grok45 · 17279 in / 3950 out tokens · 37559 ms · 2026-07-14T03:30:26.599154+00:00 · methodology

0 comments
read the original abstract

We introduce \textbf{Qwen-Audio-VAE}, a suite of low-bitrate, fast-encoding continuous audio autoencoders designed for scalable general audio generation. The model is built around a simple but important principle: an audio VAE should not only reconstruct diverse audio with high fidelity, but also produce compact latent representations fast enough to support large-scale text-to-audio training. Qwen-Audio-VAE combines a causal encoder-decoder, window Transformer blocks, and multi-discriminator training to achieve a strong balance between reconstruction quality and compression rate. The model is trained at scale on 5 million hours of multi-domain audio, enabling robust reconstruction across heterogeneous acoustic conditions. To further improve computational efficiency, we adopt an asymmetric encoder-decoder backbone and introduce latency-aware encoder pruning to maximize encoding throughput. Experiments on public speech, music, and sound reconstruction benchmarks show that Qwen-Audio-VAE generalizes well across diverse audio domains and is particularly efficient, requiring only 541 ms to encode 32 minutes of audio. Overall, Qwen-Audio-VAE provides a high-quality, compact, and high-throughput representation backbone for efficient general audio generation.

Figures

Figures reproduced from arXiv: 2607.11738 by Chenfei Wu, Dake Guo, Dayiheng Liu, Hangrui Hu, Jiapeng Wang, Jin Xu, Ting He, Wenxiang Guo, Xinfa Zhu, Xiong Wang, Yongqi Wang, Zekai Zhang, Zhifang Guo, Ziyue Jiang.

Figure 1
Figure 1. Figure 1: Qwen-Audio-VAE: a low-bitrate, fast-encoding continuous audio autoencoder with high￾fidelity reconstruction across speech, music, and sound and fast encoding (541 ms for 32 min of audio), making it an efficient backbone for general audio generation. 1 Introduction Multimodal foundation models (OpenAI, 2024; Comanici et al., 2025; Qwen, 2026) have made text-to￾audio-video generation (Yang et al., 2025; Chen… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of Qwen-Audio-VAE. A causal convolutional encoder downsamples the waveform to [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Qwen-Audio-VAE data pipeline. The multi-domain raw audio pool flows through accessibility, [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Spectrogram reconstruction comparison on LibriSpeech. Columns: ground truth, Stable Audio [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

266 extracted references · 107 linked inside Pith

  1. [1]

    2024 , url =

    OpenAI , title =. 2024 , url =

    2024

  2. [2]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context , author =

  3. [3]

    arXiv preprint arXiv:2507.06261 , year=

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities , author=. arXiv preprint arXiv:2507.06261 , year=

    Pith/arXiv arXiv

  4. [4]

    2026 , eprint=

    Qwen3.5-Omni Technical Report , author=. 2026 , eprint=

    2026

  5. [5]

    International Conference on Learning Representations , volume=

    Cogvideox: Text-to-video diffusion models with an expert transformer , author=. International Conference on Learning Representations , volume=

  6. [6]

    arXiv preprint arXiv:2504.13074 , year=

    Skyreels-v2: Infinite-length film generative model , author=. arXiv preprint arXiv:2504.13074 , year=

    Pith/arXiv arXiv

  7. [7]

    arXiv preprint arXiv:2503.20314 , year=

    Wan: Open and advanced large-scale video generative models , author=. arXiv preprint arXiv:2503.20314 , year=

    Pith/arXiv arXiv

  8. [8]

    arXiv preprint arXiv:2604.14148 , year=

    Seedance 2.0: Advancing video generation for world complexity , author=. arXiv preprint arXiv:2604.14148 , year=

    Pith/arXiv arXiv

  9. [9]

    High-Fidelity Audio Compression with Improved

    Kumar, Rithesh and Seetharaman, Prem and Luebs, Alejandro and Kumar, Ishaan and Kumar, Kundan , booktitle=NIPS, year=. High-Fidelity Audio Compression with Improved

  10. [10]

    Ji, Shengpeng and Jiang, Ziyue and Wang, Wen and Chen, Yifu and Fang, Minghui and Zuo, Jialong and Yang, Qian and Cheng, Xize and Wang, Zehan and Li, Ruiqi and others , booktitle=ICLR, year=

  11. [11]

    IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , year=

    Stable Audio Open , author=. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , year=

  12. [12]

    and Markovi

    Wu, Yi-Chiao and Gebru, Israel D. and Markovi. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , year=

  13. [13]

    Cheng, Ho Kei and Ishii, Masato and Hayakawa, Akio and Shibuya, Takashi and Schwing, Alexander and Mitsufuji, Yuki , booktitle=CVPR, year=

  14. [14]

    Shan, Sizhe and Li, Qiulin and Cui, Yutao and Yang, Miles and Wang, Yuehai and Yang, Qun and Zhou, Jin and Zhong, Zhao , journal=

  15. [15]

    Wang, Jun and Zeng, Xijuan and Qiang, Chunyu and Chen, Ruilong and Wang, Shiyao and Wang, Le and Zhou, Wangjing and Cai, Pengfei and others , journal=

  16. [16]

    Kim, Chris Dongjoo and Kim, Byeongchang and Lee, Hyunmin and Kim, Gunhee , booktitle=NAACL, year=

  17. [17]

    Machine Learning for Audio Workshop, NeurIPS , year=

    The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluation , author=. Machine Learning for Audio Workshop, NeurIPS , year=

  18. [18]

    and Reiss, Joshua D

    Steinmetz, Christian J. and Reiss, Joshua D. , booktitle=. auraloss: Audio focused loss functions in

  19. [19]

    Kong, Jungil and Kim, Jaehyeon and Bae, Jaekyoung , booktitle=NIPS, year=

  20. [20]

    Taming Transformers for High-Resolution Image Synthesis , author=

  21. [21]

    Scalable Diffusion Models with Transformers , author=

  22. [22]

    Auto-Encoding Variational Bayes , author=

  23. [23]

    , booktitle=ICML, year=

    Liu, Haohe and Chen, Zehua and Yuan, Yi and Mei, Xinhao and Liu, Xubo and Mandic, Danilo and Wang, Wenwu and Plumbley, Mark D. , booktitle=ICML, year=

  24. [24]

    Neil Zeghidour and Alejandro Luebs and Ahmed Omran and Jan Skoglund and Marco Tagliasacchi , title =

  25. [25]

    arXiv:2210.13438 , year=

    High fidelity neural audio compression , author=. arXiv:2210.13438 , year=

    Pith/arXiv arXiv

  26. [26]

    CoRR , volume =

    Xin Zhang and Dong Zhang and Shimin Li and Yaqian Zhou and Xipeng Qiu , title =. CoRR , volume =

  27. [27]

    2015 IEEE international conference on acoustics, speech and signal processing (ICASSP) , pages=

    Librispeech: an asr corpus based on public domain audio books , author=. 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP) , pages=. 2015 , organization=

    2015

  28. [28]

    IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , year=

    Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation , author=. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , year=

  29. [29]

    arXiv preprint arXiv:2508.19205 , year=

    Vibevoice technical report , author=. arXiv preprint arXiv:2508.19205 , year=

    Pith/arXiv arXiv

  30. [30]

    Attention is All you Need , url =

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, ukasz and Polosukhin, Illia , booktitle =. Attention is All you Need , url =

  31. [31]

    Qwen3.5: Accelerating Productivity with Native Multimodal Agents , url =

    Qwen Team , month =. Qwen3.5: Accelerating Productivity with Native Multimodal Agents , url =

  32. [32]

    2025 , eprint=

    Qwen3-Omni Technical Report , author=. 2025 , eprint=

    2025

  33. [33]

    arXiv:2302.13971 , year=

    Llama: Open and efficient foundation language models , author=. arXiv:2302.13971 , year=

    Pith/arXiv arXiv

  34. [34]

    arXiv:2301.12597 , year=

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models , author=. arXiv:2301.12597 , year=

    Pith/arXiv arXiv

  35. [35]

    arXiv:2303.08774 , year=

    GPT-4 technical report , author=. arXiv:2303.08774 , year=

    Pith/arXiv arXiv

  36. [36]

    NeurIPS , year=

    Perception test: A diagnostic benchmark for multimodal video models , author=. NeurIPS , year=

  37. [37]

    arXiv:2405.21075 , year=

    Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis , author=. arXiv:2405.21075 , year=

    Pith/arXiv arXiv

  38. [38]

    NeurIPS , year=

    Egoschema: A diagnostic benchmark for very long-form video language understanding , author=. NeurIPS , year=

  39. [39]

    CVPR , year=

    Mvbench: A comprehensive multi-modal video understanding benchmark , author=. CVPR , year=

  40. [40]

    arXiv:2406.06462 , year=

    VCR: Visual Caption Restoration , author=. arXiv:2406.06462 , year=

    Pith/arXiv arXiv

  41. [41]

    2024 , journal=

    MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI , author=. 2024 , journal=

    2024

  42. [42]

    MMBench: Is Your Multi-modal Model an All-around Player? , year =

    Yuan Liu and Haodong Duan and Yuanhan Zhang, Bo Li and Songyang Zhang and Wangbo Zhao and Yike Yuan and Jiaqi Wang and Conghui He and Ziwei Liu and Kai Chen and Dahua Lin , journal =. MMBench: Is Your Multi-modal Model an All-around Player? , year =

  43. [43]

    2023 , journal=

    HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination & Visual Illusion in Large Vision-Language Models , author=. 2023 , journal=

    2023

  44. [44]

    Grok-1.5 vision preview , year =

  45. [45]

    Grok-2 Beta Release , year =

  46. [46]

    arXiv:2403.20330 , year=

    Are We on the Right Way for Evaluating Large Vision-Language Models? , author=. arXiv:2403.20330 , year=

    Pith/arXiv arXiv

  47. [47]

    ICML , year=

    Mm-vet: Evaluating large multimodal models for integrated capabilities , author=. ICML , year=

  48. [48]

    NeurIPS , year=

    Flamingo: a visual language model for few-shot learning , author=. NeurIPS , year=

  49. [49]

    and Stoica, Ion and Xing, Eric P

    Chiang, Wei-Lin and Li, Zhuohan and Lin, Zi and Sheng, Ying and Wu, Zhanghao and Zhang, Hao and Zheng, Lianmin and Zhuang, Siyuan and Zhuang, Yonghao and Gonzalez, Joseph E. and Stoica, Ion and Xing, Eric P. , year =. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90\ url =

  50. [50]

    arXiv:2309.17425 , year=

    Data filtering networks , author=. arXiv:2309.17425 , year=

    Pith/arXiv arXiv

  51. [51]

    arXiv:2306.09265 , year=

    LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models , author=. arXiv:2306.09265 , year=

    Pith/arXiv arXiv

  52. [52]

    NeurIPS , year=

    Language models are few-shot learners , author=. NeurIPS , year=

  53. [53]

    arXiv:2304.08485 , year=

    Visual instruction tuning , author=. arXiv:2304.08485 , year=

    Pith/arXiv arXiv

  54. [54]

    arXiv:2304.10592 , year=

    Minigpt-4: Enhancing vision-language understanding with advanced large language models , author=. arXiv:2304.10592 , year=

    Pith/arXiv arXiv

  55. [55]

    arXiv:2309.17421 , year=

    The dawn of lmms: Preliminary explorations with gpt-4v (ision) , author=. arXiv:2309.17421 , year=

    Pith/arXiv arXiv

  56. [56]

    Our World in Data , year =

    Hannah Ritchie and Veronika Samborska and Max Roser , title =. Our World in Data , year =

  57. [60]

    arXiv preprint arXiv:2307.06281 , year=

    Mmbench: Is your multi-modal model an all-around player? , author=. arXiv preprint arXiv:2307.06281 , year=

    Pith/arXiv arXiv

  58. [61]

    arXiv:2311.16502 , year=

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi , author=. arXiv:2311.16502 , year=

    Pith/arXiv arXiv

  59. [62]

    arXiv:2303.16199 , year=

    Llama-adapter: Efficient fine-tuning of language models with zero-init attention , author=. arXiv:2303.16199 , year=

    Pith/arXiv arXiv

  60. [63]

    arXiv:2305.06500 , year=

    Instructblip: Towards general-purpose vision-language models with instruction tuning , author=. arXiv:2305.06500 , year=

    Pith/arXiv arXiv

  61. [64]

    arXiv:2304.14178 , year=

    mplug-owl: Modularization empowers large language models with multimodality , author=. arXiv:2304.14178 , year=

    Pith/arXiv arXiv

  62. [65]

    arXiv:2304.15010 , year=

    Llama-adapter v2: Parameter-efficient visual instruction model , author=. arXiv:2304.15010 , year=

    Pith/arXiv arXiv

  63. [66]

    arXiv:2305.03726 , year=

    Otter: A multi-modal model with in-context instruction tuning , author=. arXiv:2305.03726 , year=

    Pith/arXiv arXiv

  64. [67]

    arXiv:2305.16355 , year=

    Pandagpt: One model to instruction-follow them all , author=. arXiv:2305.16355 , year=

    Pith/arXiv arXiv

  65. [68]

    arXiv:2305.12223 , year=

    What Makes for Good Visual Tokenizers for Large Language Models? , author=. arXiv:2305.12223 , year=

    Pith/arXiv arXiv

  66. [69]

    arXiv:2305.10355 , year=

    Evaluating object hallucination in large vision-language models , author=. arXiv:2305.10355 , year=

    Pith/arXiv arXiv

  67. [70]

    arXiv:1504.00325 , year=

    Microsoft coco captions: Data collection and evaluation server , author=. arXiv:1504.00325 , year=

    Pith/arXiv arXiv

  68. [71]

    International journal of computer vision , volume=

    Visual genome: Connecting language and vision using crowdsourced dense image annotations , author=. International journal of computer vision , volume=. 2017 , publisher=

    2017

  69. [72]

    arXiv:2210.08402 , year=

    Laion-5b: An open large-scale dataset for training next generation image-text models , author=. arXiv:2210.08402 , year=

    Pith/arXiv arXiv

  70. [73]

    , author=

    Laion coco: 600m synthetic captions from laion2b-en. , author=. https://laion.ai/blog/laion-coco/ , year=

  71. [74]

    arXiv:2304.14108 , year=

    DataComp: In search of the next generation of multimodal datasets , author=. arXiv:2304.14108 , year=

    Pith/arXiv arXiv

  72. [75]

    2022 , url =

    COYO-700M: Image-Text Pair Dataset , author =. 2022 , url =

    2022

  73. [76]

    CVPR , year=

    Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts , author=. CVPR , year=

  74. [77]

    ACL , year=

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning , author=. ACL , year=

  75. [78]

    NeurIPS , year=

    Im2text: Describing images using 1 million captioned photographs , author=. NeurIPS , year=

  76. [79]

    arXiv:2209.06794 , year=

    Pali: A jointly-scaled multilingual language-image model , author=. arXiv:2209.06794 , year=

    Pith/arXiv arXiv

  77. [80]

    ICML , year=

    Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework , author=. ICML , year=

  78. [81]

    NeurIPS , year=

    Training language models to follow instructions with human feedback , author=. NeurIPS , year=

  79. [82]

    arXiv:2305.10403 , year=

    Palm 2 technical report , author=. arXiv:2305.10403 , year=

    Pith/arXiv arXiv

  80. [83]

    ICCV , year=

    nocaps: novel object captioning at scale , author=. ICCV , year=

Showing first 80 references.