Pith. sign in

REVIEW 3 major objections 5 minor 40 references

Implicit Inversion turns CLIP into a Decoder

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper shows that a frozen, discriminative CLIP model can generate images from text—without any decoder, training, or fine-tuning—by inverting its image encoder through a frequency-aware implicit neural representation.

desk verdict A real inversion pipeline with a misleading 'no training' claim: the method depends on an offline-trained INR bank, but the core contribution is solid enough to referee. read the letter →

arxiv 2505.23161 v2 pith:FRYSE7P3 submitted 2025-05-29 cs.CV cs.AIcs.LG

Antonio D'Orazio , Maria Rosaria Briglia , Donato Crisostomi , Dario Loi , Emanuele Rodolà , Iacopo Masi This is my paper
classification cs.CVcs.AIcs.LG
keywords CLIPinversionimplicitneuralrepresentationtext-to-imagegenerationtraining-freesynthesisdecoder-freemodalitygapProcrustesalignmentcoarse-to-fine
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a frozen, discriminative CLIP model can be turned into a generative decoder by inverting its image encoder: one optimizes the weights of an implicit neural representation (INR) so that CLIP's embedding of the rendered image matches the embedding of a target text prompt, with no decoder network, no training, and no fine-tuning of CLIP. The author argues that the main obstacle—CLIP's modality gap between text and image embeddings—can be bridged by an orthogonal Procrustes projection, and that the optimization stays on the natural-image manifold if the INR is initialized with adversarially robust weights and guided by a blending loss toward retrieved natural images. If true, this would mean that the generative potential of contrastive vision-language models is hidden in plain sight, and that a single discriminative backbone can act as a text-to-image generator, an image editor, and a style-transfer engine. The practical consequence is that image synthesis no longer requires a dedicated latent-to-image decoder, shrinking the parameter count by orders of magnitude relative to diffusion pipelines.

What carries the argument

The machinery is an implicit neural representation (INR)—an MLP mapping pixel coordinates $(i, j)$ to RGB values—using FINER's variable-periodic activations, which assign different frequency bands to different network layers. This stratification lets the optimizer steer generation with a Gaussian learning-rate schedule: early iterations update only the low-frequency layers (coarse layout and color), later iterations the high-frequency layers (texture). Two stabilizers carry the argument: adversarial weight perturbation (AWP) during the INR's initial training on blurred images, which makes the weights robust so that inversion steps do not drift off the natural-image manifold, and an orthogonal Procrustes rotation computed from the $k$ nearest caption–image pairs, which projects the text embedding onto the image sub-manifold to overcome CLIP's modality gap. A blending loss toward retrieved natural images anchors the output to real-photo statistics.

What would settle it

Run CLIP−1 on prompts whose nearest captions in the retrieval database are distant (or remove the retrieval-based initialization entirely and start from random INR weights): if the outputs collapse into stripe artifacts and duplicated objects, then the 'CLIP alone' generation is in fact an unblurring of a retrieved seed rather than a generalizable inversion of the embedding. A parallel test is to measure the nearest-neighbor distance between generated images and their database seeds; if generations are near-duplicates at the blur level, the pipeline is deblurring, not generating.

Watch

Extended reading notes

Core claim

The central discovery is that CLIP's vision encoder is invertible enough to serve as a decoder, provided the inversion is performed over a frequency-aware implicit neural representation rather than over raw pixels. Starting from a natural image whose caption is most similar to the prompt, the method initializes an INR that was trained, under adversarial weight perturbation, on a blurred version of that image; this flattens the loss landscape and makes the low-frequency anchor stable. During refinement, a Gaussian learning-rate schedule updates layers from low to high frequency, producing coarse-to-fine generation akin to diffusion; image augmentations are averaged to enforce robustness; the prompt embedding is projected into the image sub-manifold with an orthogonal Procrustes matrix computed from the $k$ nearest caption–image pairs; and a blending loss pulls the output embedding toward a weighted combination of retrieved natural-image embeddings. The result is text-to-image generation, image reconstruction, controlled edits, and neural style transfer, all from a frozen CLIP ViT-B/32 with 150M parameters and no training.

Load-bearing premise

The load-bearing premise is that gradient descent on CLIP's cosine distance, anchored by a blurred robust initialization and a blending loss, keeps the generated images on the natural-image manifold; the paper itself states that this constraint is not explicitly enforced, and the ablations show that removing any stabilizer makes outputs drift into striping, duplicated objects, or harsh colors.

Editorial extensions

If this is right

  • Decoder-free text-to-image at 150M parameters: on 10k MS-COCO captions, CLIP−1 reaches FID 72.5, IS 9.5, and CLIPSIM 38.6—roughly half the FID and nearly double the IS of the training-free DAS baseline.
  • Zero-shot transfer: the same frozen pipeline performs image reconstruction, prompt-driven edits, and neural style transfer with no task-specific changes, indicating that the inversion front-loads a general decoding capability.
  • Each stabilizer earns its place: ablations on 1k captions show that removing the frequency schedule alone drives FID from 107.1 to 185.1 and produces stripe artifacts, while removing AWP lets the layout drift, removing Procrustes yields cluttered details, and removing the blending loss produces harsh colors and duplicated objects.
  • Plug-and-play compatibility: the same inversion procedure works with tuned CLIP variants such as CLIPAG and EB-CLIP, suggesting the decoding capability is intrinsic to the CLIP embedding space, not to one checkpoint.
  • Practical cost: the complete text-to-image synthesis runs in about 78 seconds on a single RTX 4060 with about 3.2 GB of VRAM after the offline data-preparation stage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not explore what happens for prompts far from its retrieval database; a natural test would be to measure FID/CLIPSIM on rare or invented concepts, which would reveal whether the implicit manifold prior generalizes or whether output quality tracks database coverage.
  • The AWP ablation suggests a testable link between robustness and invertibility: if flatter loss landscapes invert more faithfully, then adversarially robust training of other frozen discriminative encoders could act as a cheap generative-prior recipe.
  • Because the same frozen pipeline performs reconstruction, editing, and style transfer, the inversion may double as a diagnostic probe of CLIP's embedding geometry—exposing which text directions are decodable to images and which are not.
  • If the per-prompt Procrustes projection is what makes the modality gap traversable, then a scaled-up or better-structured retrieval database (rather than a bigger generator) could push quality further; this remains untested in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes CLIP−1, a method for text-to-image synthesis that inverts a frozen CLIP image encoder by optimizing an implicit neural representation (INR). The pipeline has three stages: an offline data-preparation stage that trains an INR on each blurred LAION image with adversarial weight perturbations (AWP); an initialization stage that retrieves the INR whose caption embedding is closest to the prompt and applies an orthogonal Procrustes projection to bridge the text-image modality gap; and an optimization stage that refines the INR layer-wise with a Gaussian learning-rate schedule, augmented CLIP embeddings, and a blending loss toward retrieved natural-image embeddings. The authors report FID 72.5, CLIPSIM 38.6, and IS 9.5 on 10,000 MS-COCO captions, outperforming the concurrent DAS baseline, and demonstrate zero-shot transfer to reconstruction, controlled editing, and style transfer. The central claim is that a frozen discriminative model can generate coherent images without any decoder, training, or fine-tuning.

Significance. If the central claim were accurate, the paper would be significant: it would show that a frozen discriminative vision-language model contains enough generative structure to produce semantically aligned images through INR optimization, with implications for interpretability and for repurposing frozen backbones. The paper ships code and models, reports comparisons against several decoder-free baselines, and includes qualitative and ablation evidence. The zero-shot task demonstrations are also valuable. However, the significance is currently overstated because the method relies on an offline trained INR bank and per-prompt optimization, and the ablation study does not isolate the contributions it claims to measure. The empirical contribution is still interesting, but the claims need to be re-scoped before the paper can be accepted.

major comments (3)
  1. [Abstract & §2.2, Table 1] The headline claim of image synthesis 'without any decoder, training, or fine-tuning' and the '0 trainable parameters' entry in Table 1 are contradicted by the method itself. Section 2.2 and Algorithms 1–2 describe an offline stage in which an INR is trained on each blurred LAION image using MSE, SSIM, and L1 losses with adversarial weight perturbations, with training hyperparameters given in §6.3. Section 2.3 then retrieves one of these trained INRs and refines it via 400 AdamW optimization steps. Thus the generative capability depends on a trained, image-specific INR bank, and the process is not 'CLIP alone' in any literal sense. Please rephrase the claim as 'CLIP is frozen and no CLIP training or fine-tuning is used,' report the offline INR training cost, and qualify the '0 trainable parameters' entry accordingly.
  2. [§3.3, Fig. 6] The ablation study does not isolate the four components it claims to evaluate. The components are listed as frequency scheduling, AWP, Procrustes alignment, and the blending loss, but rows iv and v remove frequency optimization together with Procrustes or blending loss, respectively. Consequently, the text's attributions—'eliminating the orthogonal Procrustes projection (iv)' and 'disabling the blending loss (v)'—are not supported, since any effect in those rows could be due to the absent frequency scheduling. Please add single-component ablations (w/o Procrustes only; w/o blending only) or restrict the claims to the joint variants.
  3. [§3.1, Eq. (5)] CLIPSIM is not an independent semantic-alignment metric for this method. It measures the cosine similarity between the CLIP image embedding of the generated image and the caption embedding, while the optimization in Eq. (5) minimizes the cosine distance between augmented output embeddings and the Procrustes-transformed caption embedding. The paper acknowledges this circularity for CLIP-Inv but not for CLIP−1, and the ablation rows show CLIPSIM rising when components are removed, exactly as the optimization objective would predict. Please temper the CLIPSIM-based conclusions or add a human evaluation or an external semantic metric.
minor comments (5)
  1. [§3.3, Fig. 6 caption] The caption labels columns as 'ii.frequency scheduling' and 'iii.AWP' without the 'w/o' prefix; as printed, they appear to list components that are present rather than ablated. Also, the text says the ablated variants are (i)–(iv), but the table reports five rows.
  2. [§6.3.1] The loss weights α1=0.85, α2=0.25, and α3=0.25 do not sum to one; clarify whether these are relative weights or whether additional normalization is applied in Algorithm 2.
  3. [Eq. (5)] Step (b) writes ϕ0 as the result of a min-max optimization, but that step is the offline training objective, not the initialization itself; align this notation with Algorithm 2, where the trained weights are returned.
  4. [References] The reference [13] cited for Stable Diffusion guidance is 'Measuring forgetting of memorized training examples,' which does not correspond to the classifier-free guidance method the sentence describes; please update the citation.
  5. [§2.2 and Algorithms 1–2] Equation (2) defines L as a generic reconstruction loss, while Algorithm 1 maximizes negative SSIM and Algorithm 2 uses a weighted MSE+SSIM+L1 combination; please specify which loss is used in Eq. (2) and why Algorithm 1 uses only SSIM.

Circularity Check

1 steps flagged · score 2.0 of 10

No central derivation collapses into its inputs; the only partial circularity is that CLIPSIM is essentially the same cosine objective the optimizer minimizes, a point the authors themselves concede for CLIPInvert. The 'without any training' wording overstates the offline INR pretraining stage, but that is a labeling/fairness issue rather than a circular reduction.

  1. other [Sec. 2.4 Eq. (5), Sec. 3.1 CLIPSIM definition and Table 1]
    "Since these metrics do not capture semantic alignment with the prompt, we also report CLIPSIM [11], which measures the cosine similarity between the CLIP embeddings of generated images and their corresponding captions. ... L is the cosine distance. ... CLIPInvert [14], whose higher CLIPSIM can be attributed to overfitting to the target embedding."

    The inversion objective (Eq. 5c/4) minimizes the cosine distance L between the generated INR rendering's CLIP embedding and the Procrustes-projected prompt embedding, while CLIPSIM is the cosine similarity between the generated image's CLIP embedding and the caption embedding. The reported CLIPSIM therefore partly measures the very quantity that the optimizer is constructed to maximize, so a high value is not an independent confirmation of semantic alignment. The paper itself concedes this failure mode for CLIPInvert, and its ablations show that removing realism components raises CLIPSIM while lowering FID. The circularity is partial because FID and IS use external Inception features.

full rationale

The central derivation does not reduce to its inputs: the INR weights are optimized against the frozen CLIP encoder, and the main quality claims are anchored by FID and IS, which are external to the optimization. The only partial circularity is the CLIPSIM metric, which is essentially the same cosine objective being optimized; this is acknowledged for CLIPInvert and is visible in the ablations, so it raises the score only to 2. Separately, the abstract's 'without any training' wording is contradicted by Sec. 2.2, where per-image INRs are trained on blurred LAION images with AWP (Algorithms 1-2) and then retrieved as initializations; this is a suppressed premise and a possible fairness concern against pixel-space baselines, but it is not a circular reduction because the final image is not by construction equal to the retrieved INR and the realism metrics are independent. The conclusion's admission that no explicit mechanism enforces the natural-image manifold is an honest limitation, not a circular step. No load-bearing self-citation or imported uniqueness theorem was found.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a large set of hand-chosen hyperparameters and on domain assumptions about CLIP's embedding geometry and FINER's frequency stratification. No new physical or mathematical entities are introduced.

free parameters (6)
  • INR architecture: 5 hidden layers x 256 units, first_omega=hidden_omega=25 = 5x256, omega 25
    Chosen by hand; no sensitivity analysis, and the layer-wise frequency assumption depends on this design.
  • AWP perturbation strength alpha/gamma = 0.01 (reported as alpha; algorithm uses gamma)
    Controls the robustness of initialization; no sensitivity analysis reported.
  • Loss weights alpha1, alpha2, alpha3 = 0.85, 0.25, 0.25
    Weights for MSE, SSIM, and L1 in INR training; chosen by hand.
  • Gaussian blur kernel size and sigma for blurred targets = kernel 101, sigma in [10,20]
    Determines the low-frequency anchor; not ablated.
  • Gaussian scheduling LR, gradient clipping, refresh period, focus layers = LR 2e-4, clip [1.0,0.5,0.2], refresh 70, layers [0,1,2]
    Core coarse-to-fine mechanism; values hand-selected, no sensitivity study.
  • Blending beta, neighbor counts k and p, augmentations n = beta=0.5, k=8, p=256, n=32
    Trade-off hyperparameters for the blending loss, Procrustes alignment, and augmentation averaging; no sensitivity analysis.
assumptions (5)
  • domain assumption CLIP's cosine embedding space is locally structured so that minimizing cosine distance between an augmented INR render and a Procrustes-aligned text embedding yields natural-image-like solutions.
    The entire inversion in Eq. 4 assumes gradient descent on this loss does not leave the natural image manifold; the conclusion explicitly notes no mechanism enforces this.
  • domain assumption FINER's bias initialization stratifies frequencies across layers, so layer-wise learning-rate scheduling produces coarse-to-fine generation.
    Invoked in Section 2.4 as the basis for frequency scheduling; inherited from FINER, not re-derived here.
  • domain assumption A single orthogonal Procrustes transformation computed from k nearest caption-image pairs aligns the local text submanifold to the image submanifold for the target prompt.
    Used in Eq. 3 to produce the target embedding et2i; assumes local linearity and orthogonality of the modality gap.
  • domain assumption Linear blending of CLIP image embeddings of nearby natural images yields a target close to the natural image manifold.
    The blending loss in Eq. 5b assumes softmax-weighted combinations of retrieved image embeddings are valid manifold anchors.
  • domain assumption Training INRs on blurred images with adversarial weight perturbations yields weights that are stable initialization points for inversion.
    AWP-based INR training in Section 2.2 and Supplementary 6.1 is asserted to stabilize optimization; no formal guarantee.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Implicit Inversion turns CLIP into a Decoder." pith.science (2026). https://pith.science/paper/FRYSE7P3

@misc{pith2026250523161,
  author       = {Pith},
  title        = {Pith review of: Implicit Inversion turns CLIP into a Decoder},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FRYSE7P3}},
  note         = {Machine review of arXiv:2505.23161}
}
read the original abstract

CLIP is a discriminative model trained to align images and text in a shared embedding space. Due to its multimodal structure, it serves as the backbone of many generative pipelines, where a decoder is trained to map from the shared space back to images. In this work, we show that image synthesis is nevertheless possible using CLIP alone -- without any decoder, training, or fine-tuning. Our approach optimizes a frequency-aware implicit neural representation that encourages coarse-to-fine generation by stratifying frequencies across network layers. To stabilize this inverse mapping, we introduce adversarially robust initialization, a lightweight Orthogonal Procrustes projection to align local text and image embeddings, and a blending loss that anchors outputs to natural image statistics. Without altering CLIP's weights, this framework unlocks capabilities such as text-to-image generation, style transfer, and image reconstruction. These findings suggest that discriminative models may hold untapped generative potential, hidden in plain sight.

Figures

Figures reproduced from arXiv: 2505.23161 by the authors.

Figure 1
Figure 1. Decoder-free text-to-image synthesis. CLIP−1 inverts CLIP’s image encoder using implicit neural representations, enabling text-to-image synthesis without any fine-tuning or dedicated generative decoder. All samples are generated with CLIP ViT-B/32 [26], with top rows showing generic scene prompts and bottom rows illustrating complex captions from MS-COCO [18]. Abstract CLIP is a discriminative model trained to align… view at source ↗
Figure 2
Figure 2. CLIP−1 text-to-image inversion pipeline. The image is initially represented using an Implicit Neural Representation (INR) fϕ(i, j), optimizing the INR weights to match an input text prompt. The starting point for the inversion is a robust INR trained with Adversarial Weight Perturbation (AWP). The optimization updates the INR layer-wise, so that the embedding of its rendering aligns with the input text prompt embedd… view at source ↗
Figure 3
Figure 3. Gaussian Scheduling. Each layer represents a frequency interval (b) for fϕ. The learning rate is centered on a specific layer and gradually shifted (c), decreas￾ing with a Gaussian attenuation across neighboring layers (a). Augmentations for stable optimization. Inspired by CLIP￾Draw [7] and CLIPAG [8], we apply image augmentations such as color shifts, scaling, and shearing, while optimizing. Each augmented image i… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison of prior pixel-based methods against different CLIP−1 configurations. 3.1 Generator-free text-to-image synthesis Setting. In text-to-image synthesis, the goal is to generate visually realistic images that are semantically aligned with a natural l…
Figure 5
Figure 5. Figure 5: Downstream Tasks. (a) Recreates the input image from its corresponding CLIP encoding. (b) Alters the input image based on a specified prompt. Prompt 1: «Snowy peaceful landscape» ; Prompt 2: «Torrential rainfall, lightning bolts». (c) Applies the visual style of a refe…
Figure 6
Figure 6. Figure 6: Quantitative ablation study. (a) Results on 1,000 MS-COCO captions. Lower FID is better; higher CLIPSIM and IS indicate better performance. (b) Samples for each case. Rows share the same prompt; columns show: – i. full model – ii. frequency scheduling – iii. AWP – iv. …
Figure 7
Figure 7. Figure 7: traces a prompt through 400 inversion steps to show how the most critical components influence the optimization. Two broad patterns emerge. (i) Frequency scheduling governs the refinement path: when it is present (top two rows) the image is generated in a coarse-to-fin…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 20 canonical work pages

  1. [1]

    Improving image generation with better captions.Computer Science

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions.Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023

    2023

  2. [2]

    Large scale gan training for high fidelity natural image synthesis

    Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. InInternational Conference on Learning Representations

    work page

  3. [3]

    Muse: Text-to-image generation via masked generative transformers

    Huiwen Chang, Han Zhang, Jarred Barber, Aaron Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Patrick Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. InInternational Conference on Machine Learning, pages 4055–4075. PMLR, 2023

    2023

  4. [4]

    Decoupled kullback-leibler divergence loss.arXiv preprint arXiv:2305.13948, 2023

    Jiequan Cui, Zhuotao Tian, Zhisheng Zhong, Xiaojuan Qi, Bei Yu, and Hanwang Zhang. Decoupled kullback-leibler divergence loss.arXiv preprint arXiv:2305.13948, 2023

    arXiv 2023

  5. [5]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. InInternational Conference on Machine Learning, pages 12606–12633. PMLR, 2024

    work page 2024

  6. [6]

    Direct ascent synthesis: Revealing hidden generative capabilities in discriminative models.arXiv preprint arXiv:2502.07753, 2025

    Stanislav Fort and Jonathan Whitaker. Direct ascent synthesis: Revealing hidden generative capabilities in discriminative models.arXiv preprint arXiv:2502.07753, 2025

    arXiv 2025

  7. [7]

    Kevin Frans, L. B. Soros, and Olaf Witkowski. Clipdraw: exploring text-to-drawing synthesis through language-image encoders. InProceedings of the 36th International Conference on Neural Information Processing Systems, Red Hook, NY , USA, 2022. Curran Associates Inc

    work page 2022

  8. [8]

    Clipag: Towards generator-free text-to-image generation, 2023

    Roy Ganz and Michael Elad. Clipag: Towards generator-free text-to-image generation, 2023

    work page 2023

Show all 40 references
  1. [9]

    Text-to-image generation via energy-based clip, 2024

    Roy Ganz and Michael Elad. Text-to-image generation via energy-based clip, 2024

    2024

  2. [10]

    Do perceptually aligned gradients imply robustness? In ICML, 2023

    Roy Ganz, Bahjat Kawar, and Michael Elad. Do perceptually aligned gradients imply robustness? In ICML, 2023

    2023

  3. [11]

    Clipscore: A reference-free evaluation metric for image captioning.arXiv preprint arXiv:2104.08718, 2021

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning.arXiv preprint arXiv:2104.08718, 2021

    2021 arXiv

  4. [12]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. InNeurIPS, 2017

    2017

  5. [13]

    Measuring forgetting of memorized training examples

    Matthew Jagielski, Om Thakkar, Florian Tramer, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Guha Thakurta, Nicolas Papernot, et al. Measuring forgetting of memorized training examples. InICLR, 2023

    2023

  6. [14]

    What do we learn from inverting clip models? InNeurips Safe Generative AI Workshop 2024

    Hamid Kazemi, Atoosa Chegini, Jonas Geiping, Soheil Feizi, and Tom Goldstein. What do we learn from inverting clip models? InNeurips Safe Generative AI Workshop 2024

    2024

  7. [15]

    Glow: Generative flow with invertible 1x1 convolutions.Advances in neural information processing systems, 31, 2018

    Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions.Advances in neural information processing systems, 31, 2018

    2018

  8. [16]

    Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning

    Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. InAdvances in Neural Information Processing Systems, pages 17612–17625. Curran Associates, Inc., 2022

    2022

  9. [17]

    Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.Advances in Neural Information Processing Systems, 35:17612–17625, 2022

    Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.Advances in Neural Information Processing Systems, 35:17612–17625, 2022. 10

    2022

  10. [18]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InECCV, 2014

    2014

  11. [19]

    Finer: Flexible spectral-bias tuning in implicit neural representation by variable-periodic activation functions

    Zhen Liu, Hao Zhu, Qi Zhang, Jingde Fu, Weibing Deng, Zhan Ma, Yanwen Guo, and Xun Cao. Finer: Flexible spectral-bias tuning in implicit neural representation by variable-periodic activation functions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Re...

    2024

  12. [20]

    Towards deep learning models resistant to adversarial attacks

    Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. InICLR, 2018

    2018

  13. [21]

    Latent space translation via semantic alignment.Advances in Neural Information Processing Systems, 36:55394–55414, 2023

    Valentino Maiorca, Luca Moschella, Antonio Norelli, Marco Fumero, Francesco Locatello, and Emanuele Rodolà. Latent space translation via semantic alignment.Advances in Neural Information Processing Systems, 36:55394–55414, 2023

    2023

  14. [22]

    Shedding more light on robust classifiers under the lens of energy-based models

    Mujtaba Hussain Mirza, Maria Rosaria Briglia, Senad Beadini, and Iacopo Masi. Shedding more light on robust classifiers under the lens of energy-based models. InEuropean Conference on Computer Vision, pages 451–468. Springer, 2024

    2024

  15. [23]

    Cross the gap: Exposing the intra-modal misalignment in clip via modality inversion.ICLR 2025, 2025

    Marco Mistretta, Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Andrew D Bagdanov. Cross the gap: Exposing the intra-modal misalignment in clip via modality inversion.ICLR 2025, 2025

    2025

  16. [24]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models

    Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. InInternational Conference on Machine Learning, page...

    2022

  17. [25]

    Unsupervised representation learning with deep convolutional generative adversarial networks.arXiv preprint arXiv:1511.06434, 2015

    Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks.arXiv preprint arXiv:1511.06434, 2015

    2015 arXiv

  18. [26]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  19. [27]

    Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 2022

    2022 arXiv

  20. [28]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  21. [29]

    Photorealistic text-to- image diffusion models with deep language understanding.Advances in neural information processing systems, 35:36479–36494, 2022

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- image diffusion models with deep language understanding.Advances in neural information...

    2022

  22. [30]

    Improved techniques for training gans.Advances in neural information processing systems, 29, 2016

    Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans.Advances in neural information processing systems, 29, 2016

    2016

  23. [31]

    Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis

    Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis. InInternational conference on machine learning, pages 30105–30118. PMLR, 2023

    2023

  24. [32]

    Laion-5b: An open large-scale dataset for training next generation image-text models.Advances in neural information processing systems, 2022

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models.Advances in neural informat...

    2022

  25. [33]

    Implicit neural representations with periodic activation functions.Advances in neural information processing systems, 33:7462–7473, 2020

    Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions.Advances in neural information processing systems, 33:7462–7473, 2020

    2020

  26. [34]

    Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T

    Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains, 2020

    2020

  27. [35]

    Galip: Generative adversarial clips for text-to-image synthesis, 2023

    Ming Tao, Bing-Kun Bao, Hao Tang, and Changsheng Xu. Galip: Generative adversarial clips for text-to-image synthesis, 2023. 11

    2023

  28. [36]

    Manifold alignment using procrustes analysis

    Chang Wang and Sridhar Mahadevan. Manifold alignment using procrustes analysis. InProceedings of the 25th International Conference on Machine Learning, page 1120–1127, New York, NY , USA, 2008. Association for Computing Machinery

    2008

  29. [37]

    Clip-gen: Language-free training of a text-to-image generator with clip.arXiv preprint arXiv:2203.00386, 2022

    Zihao Wang, Wei Liu, Qian He, Xinglong Wu, and Zili Yi. Clip-gen: Language-free training of a text-to-image generator with clip.arXiv preprint arXiv:2203.00386, 2022

    2022 arXiv

  30. [38]

    Adversarial weight perturbation helps robust generalization

    Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. Advances in neural information processing systems, 33:2958–2969, 2020

    2020

  31. [39]

    Xing, Laurent El Ghaoui, and Michael I

    Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. InICML, 2019

    2019

  32. [40]

    Connect, collapse, corrupt: Learning cross-modal tasks with uni-modal data.ICLR 2024, 2024

    Yuhui Zhang, Elaine Sui, and Serena Yeung-Levy. Connect, collapse, corrupt: Learning cross-modal tasks with uni-modal data.ICLR 2024, 2024. 6 Supplementary Material This supplementary document expands on key aspects of our work by providing additional technical details and ext...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.