Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

ShaLa: Multimodal Shared Latent Space Modelling

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Trained on phase plots, a vision-language model beats top VLMs at physics formula discovery.

desk verdict The submitted manuscript has a critical metadata mismatch and the headline MSE is an in-sample fit, but the core idea—fine-tuning a VLM to read kinematic plots and propose symbolic structure—is worth a serious look if the evaluation is fixed. read the letter →

arxiv 2508.17376 v1 pith:Q5SIDXNG submitted 2025-08-24 cs.LG cs.CV

Jiali Cui , Yan-Ying Chen , Yanxia Zhang , Matthew Klenk This is my paper · ORCID
classification cs.LGcs.CV
keywords physicsformuladiscoveryvision-languagemodelsymbolicregressionphaseportraitcausalchain-of-thoughtreinforcementlearningPhysSymbol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a vision-language model can discover governing equations by looking at the same kinematic plots physicists use. It introduces a two-stage curriculum that first teaches the model to reason about phase-space and trajectory images, then sharpens the symbolic structure of its hypotheses with reinforcement learning. At inference, the model invokes an external symbolic regression tool to fit the residual between its guess and the data, an agentic step the authors call Symbolic Residual Realignment. On the new 5,000-instance PhysSymbol benchmark, the 7B model reaches a structural score of 0.812 and a final mean-squared error of 0.032, roughly three times lower than the best zero-shot VLM baseline. The claim matters because it suggests visual pattern recognition can supply the strong priors that make symbolic regression tractable.

What carries the argument

The engine of the method is a two-part training curriculum followed by an agentic inference step. Motion Structure Induction (MSI) is a supervised fine-tuning stage that teaches the model to read plots and produce reasoning chains plus hypotheses; Reward-Guided Symbolic Calibration (RGSC) uses GRPO reinforcement learning with a parameter-agnostic structural reward, a Jaccard similarity between skeletonized term sets, to favor topologically correct equations; and Symbolic Residual Realignment (SR²) composes the model's ansatz with a residual expression found by an external symbolic regression tool. The structural reward is what lets the model improve even when coefficients are wrong, and the residual step is what converts a structurally reasonable guess into a numerically accurate law.

What would settle it

Compute the Post-SR² MSE using coefficients fit on one trajectory from an equation and evaluate the formula on a second trajectory from the same equation with a different initial condition. If the error jumps to the baseline level, the reported 0.032 is an in-sample fit rather than evidence of law discovery.

Watch

Extended reading notes

Core claim

The paper's central claim is that visual induction plus symbolic residual refinement can replace blind search over equations. Given only a phase-space portrait, a time-series plot, and trajectory data, VIPER-R1 first produces a causal chain of thought and a symbolic ansatz; a structural reward then pushes the model toward topologically correct formulas; and finally the model composes its ansatz with a symbolic regression of the remaining residual to obtain the final law. The authors report that VIPER-R1-7B achieves a structural score of 0.812 and an accuracy score of 0.487 on PhysSymbol, outperforming all tested general VLMs, with the final Post-SR² MSE of 0.032.

Load-bearing premise

The headline error numbers assume that fitting the final equation to the same trajectory data used to build it tests discovery, when it actually measures how well the formula fits that particular trajectory.

Editorial extensions

If this is right

  • Symbolic regression no longer has to search cold; a vision-language model that reads plots can give it a structurally correct starting point, shrinking the search space dramatically.
  • Reinforcement learning with a structure-only reward raises both structural and exact-match scores, showing that coefficient-agnostic training can improve symbolic precision.
  • The residual-realignment step turns any plausible hypothesis into a low-MSE final law, making the quality of the initial guess the key bottleneck for discovery.
  • The same two-stage recipe transfers to the smaller 3B model, which already beats all zero-shot baselines, and the paper states the pipeline is intended to scale to chaotic systems, partial differential equations, and real experimental video.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The residual stage could be bolted onto any vision-language-model-generated ansatz, so the framework likely generalizes beyond the specific model and training recipe used in the paper.
  • A direct test would remove one modality at a time, such as supplying only trajectory numbers or only plots, to measure how much of the structural score actually comes from visual input.
  • Because the benchmark equations are built by sampling two to five terms from an eleven-category library, performance on equations outside that known family, such as partial differential equations or chaotic systems, remains open.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript, whose arXiv metadata and abstract describe a different paper ('ShaLa'), is in fact a full text titled 'VIPER-R1: Mimicking the Physicist's Eye.' It proposes a vision-language model for physics formula discovery: given phase-space and time-series plots plus trajectory data, the model generates a causal chain of thought and a symbolic ansatz, is trained with supervised fine-tuning and reinforcement learning, and at inference time invokes a symbolic regression tool to fit a residual to its ansatz. The authors introduce a synthetic PhysSymbol corpus of 5,000 instances and report structural, accuracy, and post-regression MSE improvements over several commercial VLMs, with VIPER-R1-7B achieving structural score 0.812, accuracy 0.487, and Post-SR² MSE 0.032.

Significance. If the empirical claims held, the idea of using a VLM's visual reasoning to seed a symbolic regression search would be a genuinely interesting contribution to AI-for-science, and the PhysSymbol corpus could be a useful resource. The two-stage training curriculum and the ablation study are clearly presented, and the structural-score gains over the base Qwen-VL-2.5 model are large. However, the headline end-to-end metric is computed in-sample on the same trajectory used for residual fitting, which undermines the central claim of improved physical-law discovery. The paper also lacks a documented train/test split, a standalone symbolic-regression baseline, and released code or data, so the external validity of the results is currently not established.

major comments (4)
  1. [§3.4, §4.1, Appendix B.4 (Algorithm 2, Eq. 6-8, Eq. 13)] The reported Post-SR² MSE is computed on the same trajectory that SR² uses to fit the residual. In Algorithm 2, f_residual is obtained by symbolic regression on target (a_GT - a_VLM) over the evidence trajectory's time points, f_final is the sum f_VLM + f_residual, and Eq. (13) evaluates f_final on those same N points. This makes Table 1's MSE a training error on the evaluation instance, not a measure of whether the discovered law generalizes to new observations. The conclusion that the method enables 'more precise discovery of physical laws' is not supported by this metric. The authors should evaluate f_final on held-out trajectories not used by either the VLM or the SR² step and report both the in-sample and the generalization MSE.
  2. [§4.1, Appendix C] The evaluation section refers to a 'PhysSymbol test set' (Figure 4, Table 1), but nowhere in Section 4.1 or Appendix C is the train/test split of the 5,000 instances defined. Because the same dataset is used for MSI and RGSC training, and because the RGSC reward (Eq. 4 and Eq. 11) directly uses the ground-truth equation, the absence of a split description raises a contamination risk. The authors must state exactly how many instances are held out, confirm that no training instance is used in evaluation, and ideally release the split indices.
  3. [§4.1, Table 1, §5] Table 1 compares VIPER-R1 only against VLM baselines, while the conclusion claims the method 'outperforms ... traditional methods.' No standalone symbolic-regression baseline (e.g., PySR, AI Feynman, or an LLM-SR variant) is reported. Without such a baseline, the Post-SR² MSE numbers do not establish that the VLM's ansatz improves over running symbolic regression directly on the trajectory data. Add this comparison, including the same residual-fitting protocol applied to a trivial or empty ansatz.
  4. [Title, abstract, and full text] The manuscript's title, arXiv metadata, and abstract describe 'ShaLa: Multimodal Shared Latent Space Modelling,' but the full text is a different paper, 'VIPER-R1: Mimicking the Physicist's Eye,' about physics formula discovery. This is not a superficial typo: the abstract's claims about multimodal VAEs and shared latent spaces are absent from the body, and the body's claims about formula discovery are absent from the abstract. The title, metadata, and abstract must be aligned with the actual content before the manuscript can be evaluated as a scientific submission.
minor comments (6)
  1. [§3.2, Eq. 1-2] Equations (1) and (2) contain garbled notation, including raw '...' in the summands, and should be typeset properly with well-defined indices.
  2. [Appendix B.4, Algorithm 1] Algorithm 1, line 8 says the model is updated by 'descending the gradient' of the log-likelihood; the update should ascend the log-likelihood or minimize its negative.
  3. [Table 1] Table 1 reports only point estimates; confidence intervals or standard deviations across repeated evaluations (or bootstrap intervals) should be provided, since some baseline differences are small.
  4. [Appendix C.6] Appendix C.6 refers to a 'three-stage training pipeline,' but the paper describes two training stages plus an inference stage; the terminology should be aligned.
  5. [§C.2, §4.1] The dataset includes stochastic perturbations, and Eq. (13) computes MSE on a single stochastic trajectory; this further confounds the in-sample metric, because the residual regressor can fit the realized noise rather than the underlying law.
  6. [Appendix B and C] The paper provides a project page but no code, data, or reproduction script; please release the PhysSymbol generation and evaluation code to support reproducibility.

Circularity Check

1 steps flagged · score 6.0 of 10

Post-SR² MSE is fit on the same trajectory used for evaluation, so the headline discovery metric is an in-sample fit.

  1. fitted input called prediction [Algorithm 2 (Steps 2–5), Section 3.4 Eqs. (6)–(8), Appendix B.5 Eq. (13)]
    "Algorithm 2: // Stage 2: Residual Field Calculation: r_GT(t_j) ... ← ExtractData(x, v, r = a_GT − f_VLM(t_j)); // Stage 3: Tool-Using for Residual Modeling: f_residual ← SR(inputs=(t), target=(r)); // Stage 4: Theory Realignment: f_final ← f_ansatz + f_residual. B.5: 'Post-SR² Mean Squared Error (MSE) ... measuring how well the final, refined formula fits the observed data. ... MSE = 1/N Σ_j=1^N (a_GT(t_j) − f_final(t_j))^2.'"

    The residual r is defined on the evaluation instance's own trajectory as a_GT(t_j) − f_VLM(t_j); SR fits f_residual on those same (t, r) points; Post-SR² MSE then evaluates f_final = f_ansatz + f_residual on the same a_GT(t_j), t_j. Therefore the headline MSE is the in-sample fit error of the residual regressor on the test instance, not a held-out or predictive measure of law discovery. An SR tool minimizing squared error on r drives this number down by construction; the reported advantage (0.032 vs 0.091) compares in-sample fits, not discovery quality. Structural and accuracy scores remain independent, but the 'enabling more precise discovery of physical laws' claim rests on this circular metric.

full rationale

The central derivation chain is otherwise self-contained: MSI and RGSC are supervised/RL training stages whose targets and rewards are ground-truth equations; that is label supervision, not circularity. The paper does not lean on a self-citation chain for its core premise, and the use of the ground-truth equation in the RGSC reward is a training signal rather than a fitted prediction. The one load-bearing circular step is Post-SR² MSE: by Algorithm 2 and Eq. (13), the final composed law is fit to the same trajectory on which it is evaluated, so the metric reduces to a training error. Structural score and accuracy score are computed from the raw VLM ansatz before SR² and are independent of this circularity; they provide genuine evidence of hypothesis-generation quality. However, the abstract and Section 4.2 explicitly connect 'more precise discovery of physical laws' to the Post-SR² MSE, and no held-out split, generalization check, or standalone symbolic-regression baseline is reported to show that the final fit outperforms simply running SR on the trajectory. The fitting step is thus presented as a prediction, which is partial circularity rather than total collapse.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on a synthetic benchmark generated from a small term library, on C-CoT annotations derived from ground truth, and on a residual fit performed on evaluation data. These are the main unverified inputs that drive the reported results.

free parameters (2)
  • Residual expression r_residual fit by SR² = Fitted by PySR on each test trajectory
    The residual is fit directly to the test instance's data (Equation 7), and the final MSE is computed on that same data, so the headline metric is determined by this fitted object.
  • Reward weights (λ_format, λ_structural, λ_accuracy) = Not stated numerically; chosen by hand
    The composite reward in Equation 4 depends on three hand-chosen weights, but no values are reported, preventing exact reproduction.
assumptions (3)
  • ad hoc to paper C-CoT annotations are generated by GPT-4o using the ground-truth equation and plots
    Section C.5: the reasoning supervision is produced from the answer itself, which preloads the target into the training signal.
  • domain assumption PhysSymbol equations are sampled from an 11-term library with a mandatory linear restoring force
    Section C.2: the benchmark distribution is narrow, so recovery within this family is not evidence of open-ended discovery.
  • ad hoc to paper Structural reward is computed as Jaccard similarity against the ground-truth equation
    Equation 11: the RL reward uses the exact target equation, which strongly shapes the model toward the benchmark's ground truth.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ShaLa: Multimodal Shared Latent Space Modelling." pith.science (2026). https://pith.science/paper/Q5SIDXNG

@misc{pith2026250817376,
  author       = {Pith},
  title        = {Pith review of: ShaLa: Multimodal Shared Latent Space Modelling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q5SIDXNG}},
  note         = {Machine review of arXiv:2508.17376}
}
read the original abstract

This paper presents a novel generative framework for learning shared latent representations across multimodal data. Many advanced multimodal methods focus on capturing all combinations of modality-specific details across inputs, which can inadvertently obscure the high-level semantic concepts that are shared across modalities. Notably, Multimodal VAEs with low-dimensional latent variables are designed to capture shared representations, enabling various tasks such as joint multimodal synthesis and cross-modal inference. However, multimodal VAEs often struggle to design expressive joint variational posteriors and suffer from low-quality synthesis. In this work, ShaLa addresses these challenges by integrating a novel architectural inference model and a second-stage expressive diffusion prior, which not only facilitates effective inference of shared latent representation but also significantly improves the quality of downstream multimodal synthesis. We validate ShaLa extensively across multiple benchmarks, demonstrating superior coherence and synthesis quality compared to state-of-the-art multimodal VAEs. Furthermore, ShaLa scales to many more modalities while prior multimodal VAEs have fallen short in capturing the increasing complexity of the shared latent space.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ForeSight: A Predictive-Scheduling Deterministic Database

    cs.DB 2025-08 unverdicted novelty 5.0 of 10

    ForeSight uses a learned conflict predictor and a matrix dependency scan to schedule database transactions in a conflict-aware order, claiming up to 2x throughput on skewed workloads.

Reference graph

Works this paper leans on

44 extracted references · 20 canonical work pages · cited by 1 Pith paper

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Alayrac, J.-B.; Donahue, J.; Luc, P.; Miech, A.; Barr, I.; Hasson, Y.; Lenc, K.; Mensch, A.; Millican, K.; Reynolds, M.; et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35: 23716--23736

    2022

  4. [4]

    J.; and Guerrero, P

    Anciukevi c ius, T.; Xu, Z.; Fisher, M.; Henderson, P.; Bilen, H.; Mitra, N. J.; and Guerrero, P. 2023. Renderdiffusion: Image diffusion for 3d reconstruction, inpainting and generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 12608--12618

    work page 2023

  5. [5]

    Aneja, J.; Schwing, A.; Kautz, J.; and Vahdat, A. 2021. A contrastive learning approach for training variational autoencoder priors. Advances in neural information processing systems, 34: 480--493

    work page 2021

  6. [6]

    Bao, F.; Nie, S.; Xue, K.; Cao, Y.; Li, C.; Su, H.; and Zhu, J. 2023. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 22669--22679

    2023

  7. [7]

    A.; et al

    Bie, F.; Yang, Y.; Zhou, Z.; Ghanem, A.; Zhang, M.; Yao, Z.; Wu, X.; Holmes, C.; Golnari, P.; Clifton, D. A.; et al. 2024. Renaissance: A survey into ai text-to-image generation in the era of large model. IEEE Transactions on Pattern Analysis and Machine Intelligence

    work page 2024

  8. [8]

    R.; Lin, C

    Chan, E. R.; Lin, C. Z.; Chan, M. A.; Nagano, K.; Pan, B.; De Mello, S.; Gallo, O.; Guibas, L. J.; Tremblay, J.; Khamis, S.; et al. 2022. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 16123--16133

    2022

Show all 44 references
  1. [9]

    X.; Funkhouser, T.; Guibas, L.; Hanrahan, P.; Huang, Q.; Li, Z.; Savarese, S.; Savva, M.; Song, S.; Su, H.; Xiao, J.; Yi, L.; and Yu, F

    Chang, A. X.; Funkhouser, T.; Guibas, L.; Hanrahan, P.; Huang, Q.; Li, Z.; Savarese, S.; Savva, M.; Song, S.; Su, H.; Xiao, J.; Yi, L.; and Yu, F. 2015. ShapeNet: An Information-Rich 3D Model Repository . Technical Report arXiv:1512.03012 [cs.GR], Stanford University --- Princ...

    2015 arXiv

  2. [10]

    Chen, Y.-C.; Li, L.; Yu, L.; El Kholy, A.; Ahmed, F.; Gan, Z.; Cheng, Y.; and Liu, J. 2020. Uniter: Universal image-text representation learning. In European conference on computer vision, 104--120. Springer

    2020

  3. [11]

    Cui, J.; and Han, T. 2024. Learning Latent Space Hierarchical EBM Diffusion Models. In Salakhutdinov, R.; Kolter, Z.; Heller, K.; Weller, A.; Oliver, N.; Scarlett, J.; and Berkenkamp, F., eds., Proceedings of the 41st International Conference on Machine Learning, volume 235 of...

    2024

  4. [12]

    M.; Chin-Cheong, K.; Palumbo, E.; and Vogt, J

    Daunhawer, I.; Sutter, T. M.; Chin-Cheong, K.; Palumbo, E.; and Vogt, J. E. 2021. On the limitations of multimodal vaes. arXiv preprint arXiv:2110.04121

    2021 arXiv

  5. [13]

    Ho, J.; Jain, A.; and Abbeel, P. 2020 a . Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33: 6840--6851

    2020

  6. [14]

    Ho, J.; Jain, A.; and Abbeel, P. 2020 b . Denoising Diffusion Probabilistic Models. In Advances in Neural Information Processing Systems

    2020

  7. [15]

    Hsu, W.-N.; and Glass, J. 2018. Disentangling by partitioning: A representation learning framework for multimodal sensory data. arXiv preprint arXiv:1805.11264

    2018 arXiv

  8. [16]

    Hwang, H.; Kim, G.-H.; Hong, S.; and Kim, K.-E. 2021. Multi-view representation learning via total correlation objective. Advances in Neural Information Processing Systems, 34: 12194--12207

    2021

  9. [17]

    Karras, T.; Laine, S.; Aittala, M.; Hellsten, J.; Lehtinen, J.; and Aila, T. 2020. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 8110--8119

    2020

  10. [18]

    P.; and Welling, M

    Kingma, D. P.; and Welling, M. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114

    2013 arXiv

  11. [19]

    H.; Pham, T.; Lee, S.; Clark, C.; Kembhavi, A.; Mandt, S.; Krishna, R.; and Lu, J

    Le, D. H.; Pham, T.; Lee, S.; Clark, C.; Kembhavi, A.; Mandt, S.; Krishna, R.; and Lu, J. 2025. One diffusion to generate them all. In Proceedings of the Computer Vision and Pattern Recognition Conference, 2671--2682

    2025

  12. [20]

    A.; Zhu, Y.; Srinivasan, K.; Shah, P.; Savarese, S.; Fei-Fei, L.; Garg, A.; and Bohg, J

    Lee, M. A.; Zhu, Y.; Srinivasan, K.; Shah, P.; Savarese, S.; Fei-Fei, L.; Garg, A.; and Bohg, J. 2019. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In 2019 International conference on robotics and automation (...

    2019

  13. [21]

    Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, 19730--19742. PMLR

    2023

  14. [22]

    Liu, Y.; Lin, C.; Zeng, Z.; Long, X.; Liu, L.; Komura, T.; and Wang, W. 2023. Syncdreamer: Generating multiview-consistent images from a single-view image. arXiv preprint arXiv:2309.03453

    2023 arXiv

  15. [23]

    Mo, S.; and Raj, B. 2023. Weakly-supervised audio-visual segmentation. Advances in Neural Information Processing Systems, 36: 17208--17221

    2023

  16. [24]

    H.; and Freeman, W

    Owens, A.; Isola, P.; McDermott, J.; Torralba, A.; Adelson, E. H.; and Freeman, W. T. 2016. Visually indicated sounds. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2405--2413

    2016

  17. [25]

    Palumbo, E.; Daunhawer, I.; and Vogt, J. E. 2023. MMVAE+: Enhancing the generative quality of multimodal VAEs without compromises. In The Eleventh International Conference on Learning Representations. OpenReview

    2023

  18. [26]

    Palumbo, E.; Manduchi, L.; Laguna, S.; Chopard, D.; and Vogt, J. E. 2024. Deep Generative Clustering with Multimodal Diffusion Variational Autoencoders. In International Conference on Learning Representations

    2024

  19. [27]

    Pandey, K.; Mukherjee, A.; Rai, P.; and Kumar, A. 2022. DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents. Transactions on Machine Learning Research

    2022

  20. [28]

    Ramesh, A.; Dhariwal, P.; Nichol, A.; Chu, C.; and Chen, M. 2022. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2): 3

    2022 arXiv

  21. [29]

    Ramesh, A.; Pavlov, M.; Goh, G.; Gray, S.; Voss, C.; Radford, A.; Chen, M.; and Sutskever, I. 2021. Zero-shot text-to-image generation. In International conference on machine learning, 8821--8831. Pmlr

    2021

  22. [30]

    Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022. High-Resolution Image Synthesis with Latent Diffusion Models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2022

  23. [31]

    L.; Ghasemipour, K.; Gontijo Lopes, R.; Karagol Ayan, B.; Salimans, T.; et al

    Saharia, C.; Chan, W.; Saxena, S.; Li, L.; Whang, J.; Denton, E. L.; Ghasemipour, K.; Gontijo Lopes, R.; Karagol Ayan, B.; Salimans, T.; et al. 2022. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing syste...

    2022

  24. [32]

    Shi, Y.; Paige, B.; Torr, P.; et al. 2019. Variational mixture-of-experts autoencoders for multi-modal deep generative models. Advances in neural information processing systems, 32

    2019

  25. [33]

    Sutter, T.; Daunhawer, I.; and Vogt, J. 2020. Multimodal generative learning utilizing jensen-shannon-divergence. Advances in neural information processing systems, 33: 6100--6110

    2020

  26. [34]

    M.; Daunhawer, I.; and Vogt, J

    Sutter, T. M.; Daunhawer, I.; and Vogt, J. E. 2021. Generalized multimodal ELBO. arXiv preprint arXiv:2105.02470

    2021 arXiv

  27. [35]

    Suzuki, M.; Nakayama, K.; and Matsuo, Y. 2016. Joint multimodal learning with deep generative models. arXiv preprint arXiv:1611.01891

    2016 arXiv

  28. [36]

    Sutter, J

    Thomas M. Sutter, J. E. V., Imant Daunhawer. 2021. Generalized Multimodal ELBO. In 9th International Conference on Learning Representations, ICLR

    2021

  29. [37]

    Vahdat, A.; Kreis, K.; and Kautz, J. 2021. Score-based Generative Modeling in Latent Space. In Neural Information Processing Systems (NeurIPS)

    2021

  30. [38]

    Vedantam, R.; Fischer, I.; Huang, J.; and Murphy, K. 2018. Generative Models of Visually Grounded Imagination. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net

    2018

  31. [39]

    Wah, C.; Branson, S.; Welinder, P.; Perona, P.; and Belongie, S. 2011. The caltech-ucsd birds-200-2011 dataset

    2011

  32. [40]

    Wu, M.; and Goodman, N. 2018. Multimodal generative models for scalable weakly-supervised learning. Advances in neural information processing systems, 31

    2018

  33. [41]

    Xu, X.; Wang, Z.; Zhang, E.; Wang, K.; and Shi, H. 2023. Versatile Diffusion: Text, Images and Variations All in One Diffusion Model . In 2023 IEEE/CVF International Conference on Computer Vision (ICCV)

    2023

  34. [42]

    Yu, A.; Ye, V.; Tancik, M.; and Kanazawa, A. 2021. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 4578--4587

    2021

  35. [43]

    Yuan, S.; Cui, J.; Li, H.; and Han, T. 2024. Learning Multimodal Latent Generative Models with Energy-Based Prior. In European Conference on Computer Vision (ECCV)

    2024

  36. [44]

    Zhang, C.; Zhang, C.; Zhang, M.; and Kweon, I. S. 2023. Text-to-image diffusion models in generative ai: A survey. arXiv preprint arXiv:2303.07909

    2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.