Pith. sign in

REVIEW 5 major objections 5 minor 58 references

FLEX-CLIP: Feature-Level GEneration Network Enhanced CLIP for X-shot Cross-modal Retrieval

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FLEX-CLIP claims that generating synthetic target features and gating them with original CLIP features lets few-shot and zero-shot cross-modal retrieval beat the CLIP baseline by up to 7.9%.

desk verdict Plausible VAE-GAN + CLIP fusion for few-shot CMR, but the zero-shot headline is undercut by an internal contradiction about using target-domain data in stage 1. read the letter →

arxiv 2411.17454 v1 pith:SKVX3AOU submitted 2024-11-26 cs.CV cs.CL

Jingyou Xie , Jiayi Kuang , Zhenzhou Lin , Jiarui Ouyang , Zishuo Zhao , Ying Shen This is my paper · ORCID
classification cs.CVcs.CL
keywords cross-modalretrievalfew-shotlearningzero-shotCLIPfeaturegenerationVAE-GANgateresidualnetworkdataimbalance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the two main failures of CLIP-based few-shot cross-modal retrieval, target-domain feature degradation and extreme data imbalance, can both be fixed at the feature level. It proposes FLEX-CLIP, a two-stage method: first a composite VAE-GAN per modality generates pseudo target-domain samples from class embeddings, and second a gate residual network selectively fuses the original CLIP features with the projected features before retrieval. On four benchmark datasets across 0, 1, 3, and 5-shot settings, the method reports consistent gains over strong baselines and, notably, is the only model in its comparisons that surpasses the frozen CLIP features in zero-shot retrieval. A sympathetic reader would take this as evidence that feature-level generation plus selective fusion is a viable recipe for extending pretrained vision-language models to unseen classes.

What carries the argument

The central mechanism is the composite multimodal VAE-GAN plus the gate residual network. Each modality (image and text) has an encoder E, a generator G that serves as both GAN generator and VAE decoder, and a class-conditional discriminator D. The VAE loss makes G reconstruct real features from latent codes, the GAN loss makes G produce plausible pseudo features from class embeddings, and the same G is shared, so the generator learns the real feature distribution from source data and transfers that to the target classes. The gate residual network outputs a coefficient vector g from the concatenation of the original CLIP feature and the projected feature, and computes the final feature as g times the projected feature plus (1-g) times the original feature, allowing an adaptive per-dimension compromise between pretrained semantics and task-specific projection.

What would settle it

A concrete check is to train the VAE-GAN on source features, generate pseudo-features for the target classes, and compute the distribution distance (for example, Fréchet distance or maximum mean discrepancy) between generated and real target features in the CLIP feature space; if that distance does not correlate with the reported retrieval improvements, or if replacing the generated pseudo-features with random noise from the same class embeddings yields the same mAP, the central claim that generation is the cause of the gains would be falsified.

Watch

Extended reading notes

Core claim

The central claim is that FLEX-CLIP, by decoupling multimodal feature generation from common-space projection, solves the data imbalance and feature degradation problems that limit CLIP in x-shot cross-modal retrieval. In the generation stage, a composite VAE-GAN per modality uses the generator both as a GAN decoder (producing pseudo target samples from class attributes and noise) and as a VAE decoder (reconstructing real features from latent variables), so the generated distribution is constrained by both reconstruction and adversarial losses. In the projection stage, a gate network computes a per-dimension fusion coefficient between the raw CLIP feature and the projected feature, so the final common-space representation can retain useful pretrained semantics when the projection is unreliable. The paper reports that this two-stage design improves average mean average precision over state-of-the-art methods by 7% to 15% on four datasets, and in zero-shot settings outperforms the original CLIP features by margins from 2.31 to 7.90.

Load-bearing premise

The load-bearing premise is that a generative model trained only on source-domain features and class embeddings can produce target-domain pseudo-features whose distribution is close enough to the real target features that training the projection network on them improves retrieval, and this is especially fragile in the zero-shot case where no real target samples are seen at all.

Editorial extensions

If this is right

  • If pseudo-feature generation works as claimed, few-shot and zero-shot cross-modal retrieval no longer need to wait for large labeled target-domain data; the target class embeddings alone can synthesize training samples.
  • If the gate residual network works as claimed, fine-tuning CLIP for a new task does not have to forget the pretrained representations, since the gate can down-weight the projected features when they are harmful.
  • The two-stage separation of generation from projection implies that each stage can be improved or swapped independently, so better generative models or better projectors could be plugged in without retraining the whole pipeline.
  • The reported gains on Pascal Sentence and NUS-WIDE suggest the method is not dataset-specific, but the noise sensitivity observed on larger datasets indicates that the quality of the few real samples still matters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's ablation shows that removing the VAE-GAN still leaves the model above CLIP, suggesting that the gate residual network alone may be a lightweight fix for feature degradation; one extension would be to test the gate on other frozen encoders and other downstream tasks.
  • Because the generation stage operates on precomputed CLIP features, the approach could be transferred to other vision-language pretraining models or to other feature spaces, provided the class embeddings align with those features.
  • An indirect consequence is that the value of the generated pseudo-features depends on how well the VAE-GAN captures the source distribution; a testable extension is to measure the distribution distance between generated and real target features and correlate it with retrieval gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes FLEX-CLIP, a two-stage method for X-shot (zero- and few-shot) cross-modal retrieval. Stage 1 trains a composite VAE-GAN to generate pseudo image/text features from class embeddings and latent codes, aiming to alleviate data imbalance. Stage 2 trains projection networks plus a gate residual network that adaptively fuses original CLIP features with projected features, aimed at mitigating feature degradation. Experiments on Wikipedia, Pascal Sentence, NUS-WIDE, and NUS-WIDE-10K are reported with MAP in 0-, 1-, 3-, and 5-shot settings, claiming improvements over CLIP and other baselines. The authors do not provide code, data splits, random seeds, or variance estimates.

Significance. If the reported improvements are reproducible and the zero-shot protocol is sound, the idea of feature-level VAE-GAN generation combined with gated fusion of CLIP features is a plausible contribution to few-shot cross-modal retrieval. The paper's explicit goal of improving upon raw CLIP features is interesting, and the gated residual fusion is a simple, potentially useful mechanism. However, the central zero-shot claim is currently compromised by a protocol ambiguity and by the lack of distributional evidence for the generated features, and the experimental reporting is insufficient for assessing robustness. With the requested clarifications and additional analysis, the method could be a meaningful advance; as presented, the evidence is not yet convincing.

major comments (5)
  1. [Section IV-C vs. Section III-A] The zero-shot definition in Section III-A states that in the zero-shot scenario the model 'can only be trained by using the samples in the source domain Xs.' However, Section IV-C states that in the first phase, the multimodal feature generation model is 'trained with source domain data and a small amount of target domain data.' If that small amount includes real target-domain image/text features, the zero-shot results in Table III are not zero-shot because target information has been seen during training, making the comparison with the CLIP baseline unfair. If it refers only to target class embeddings (attributes), this must be stated explicitly, because the current wording is in direct conflict with the problem formulation.
  2. [Section III-D and Tables III-IV] The load-bearing premise that a VAE-GAN trained on source features and class embeddings can synthesize target-domain pseudo-features whose distribution is close enough to real target features to improve the downstream projection is not verified. The paper provides no quantitative evidence such as feature-space distance (e.g., FID or MMD) between generated and real target features, no visualization of generated versus real features, and no ablation that replaces pseudo-features with real target features (or varies the number of real target features used in stage 1). Without such evidence, the zero-shot improvements in Table III are not supported, and it remains unclear whether the pseudo-features are the cause of the gains.
  3. [Abstract and Contributions vs. Results] The abstract claims 'a 7%-15% improvement over state-of-the-art methods,' while the contributions section claims 'by up to 7.9%.' The 7.9% figure appears to be the best improvement over the CLIP baseline on a single dataset (Table III, NUS-WIDE), not over the best state-of-the-art method, and the 7%-15% range is not substantiated by any table. The paper should reconcile these numbers and state clearly which comparison (which baseline, which dataset, which setting) supports each claim.
  4. [Section IV-D and Implementation Details] The experimental protocol is under-specified and the reported improvements may be explained by an unfair comparison. The CLIP baseline is not fine-tuned in the reported experiments ('the 1,3,5-shot experiment results are same in CLIP'), whereas FLEX-CLIP trains projection networks and a gate network on the training set, meaning the comparison is between a frozen zero-shot CLIP and a model that has been trained on source (and possibly target) data. To make the comparison meaningful, the paper should either fine-tune the CLIP baseline under the same X-shot protocol or restrict claims of 'surpassing CLIP' to the specific setting where CLIP is not trained, and also report a CLIP baseline that uses the same trained projectors without the generation or gating components.
  5. [Tables III and IV, and Section IV-D1] The zero-shot results in Table III show FLEX-CLIP outperforming CLIP by substantial margins on NUS-WIDE-10K and NUS-WIDE (e.g., 0.579 vs. 0.503 average MAP on NUS-WIDE-10K). Given the protocol ambiguity in Section IV-C, it is critical to report the exact number of target-domain samples used in the first stage for each dataset and setting, and to release the data splits and random seeds so that the zero-shot condition can be verified. If the current numbers were obtained with target-domain real samples, the zero-shot claims should be retracted or re-run under the strict protocol of Section III-A.
minor comments (5)
  1. [Section IV-D1] The text references 'Table 1' and 'Table 2' when discussing the zero-shot and few-shot results; the corresponding tables are labeled Table III and Table IV, respectively.
  2. [Section IV-E1] The ablation results for the VAE-GAN network are said to be shown in 'Figure 3' at the beginning of Section IV-E1, but the referenced figure number appears to be incorrect; the paper should refer to the actual figure or table number containing those results.
  3. [Equation (2) and (4)] The notation in the GAN objectives is inconsistent: Eq. (2) uses Dv(ev, a; θDv, θGv), but ev appears to be generated by Gv, so the argument should reflect the generator parameters. Also, the gradient penalty term is written as if it is evaluated with respect to θDv, but the expectation over training examples is not specified clearly; standard WGAN-GP notation should be used for readability.
  4. [Section III-E2] The definition of the contrastive loss in Eq. (13)-(14) is ambiguous: the superscript of u∗i includes the modal indicator and the subscript i, but the summation in Eq. (13) is not fully specified. Clarifying the index sets and the exact form of P (u∗i_i) would improve reproducibility.
  5. [Table IV] The row 'Ours(1-shot)' contains results for 1-, 3-, and 5-shot; this should be relabeled to 'Ours' with separate shot rows for clarity, and the CLIP row should clearly indicate that the CLIP baseline is not trained in the few-shot setting (as the paper already notes with '-' for sample information).

Circularity Check

2 steps flagged · score 4.0 of 10

No derivation-level circularity; one ambiguous implementation sentence could make the zero-shot result a fitted-input prediction rather than a genuine zero-shot generalization.

  1. fitted input called prediction [Section IV-C (Implementation Details) vs. Section III-A (Problem Formulation); zero-shot results in Table III]
    "We conduct two training stages. In the first phase, the multimodal feature generation model is first trained with source domain data and a small amount of target domain data."

    Section III-A defines the zero-shot scenario as: 'the model can only be trained by using the samples in the source domain Xs and then tested on the target domain Xt.' Section IV-C states that the first training phase uses source data plus 'a small amount of target domain data.' If that phrase includes real target-domain image or text features, then Table III's zero-shot gains over CLIP (up to 7.90 mAP) are produced by a model already fitted to target-domain information, so the reported zero-shot 'prediction' reduces to a fitted-input result rather than a source-only generalization.

  2. other [Section III-E, Eq. (9)-(10), and Section IV-D1 claim of surpassing CLIP]
    "u v i = g v i × f v i + (1 − g v i ) × v i, ... FLEX-CLIP is the model that surpasses the original CLIP features."

    The gate residual output is a convex combination that explicitly includes the original CLIP feature v, so the model's hypothesis class contains the CLIP identity mapping (g=0). This makes it unsurprising that FLEX-CLIP can match or exceed CLIP, but it is not circular: the reported improvements are empirical test-set numbers, not consequences of the equation alone. The gate is trained with classification, consistency, and contrastive losses rather than being forced to reproduce CLIP, so the comparison remains an empirical claim.

full rationale

FLEX-CLIP is an empirical system: the headline results are measured MAP scores on held-out target queries, not quantities derived by definition from the model's inputs. The VAE-GAN generation objective (Eq. 5-6) and the projection losses (Eq. 11-15) do not assume the target result they are used to explain, and the paper cites no prior work by its own authors, so there is no self-citation chain or imported uniqueness theorem. The one circularity-adjacent defect is the implementation sentence saying the generation model is trained with 'a small amount of target domain data' after defining zero-shot as training only on source samples. Read literally, that makes the zero-shot prediction a fitted-input result and undermines the 7.9% margin over CLIP; read as target class embeddings only, the method is consistent with zero-shot CMR. Because the text is genuinely ambiguous and the few-shot (1/3/5-shot) experiments legitimately use target samples, the issue is partial rather than a complete collapse of the paper's contribution. Score 4 reflects this single potentially load-bearing ambiguity; if the phrase is clarified to mean only target class attributes, the appropriate score would be 0-1.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method relies on several hand-chosen hyperparameters and two strong domain assumptions (target-distributed pseudo-features and gating generalization). No new physical entities are introduced.

free parameters (5)
  • Loss weights α, β, γ
    Hyperparameters balancing classification, consistency, and contrastive losses in Eq. (15); no values or sensitivity analysis are reported.
  • Temperature τ
    Scale in contrastive loss Eq. (14); not specified in the paper.
  • Gradient penalty coefficient λ
    Used in WGAN-GP losses Eq. (2) and Eq. (4); chosen presumably as a standard value but not stated.
  • Number of generated pseudo-samples = 70, 30, 500, 300
    From Table II for Wikipedia, Pascal Sentence, NUS-WIDE, and NUS-WIDE-10K; the choice is arbitrary and affects training balance.
  • Learning rates and batch sizes = Table II
    Hand-selected per dataset; no sensitivity analysis is provided.
assumptions (4)
  • domain assumption Pretrained CLIP features are informative for source-to-target transfer.
    Invoked throughout, e.g., Section III-C; if false, the whole pipeline starts from weak features.
  • ad hoc to paper Generated pseudo-features from the VAE-GAN follow the target distribution.
    Core of the data imbalance solution (Section III-D); no independent evidence is provided.
  • domain assumption The gate residual network can select between original and projected features without overfitting.
    Assumes the gating function generalizes to unseen classes (Section III-E).
  • standard math Standard VAE reparameterization and WGAN-GP gradient penalty are reliable.
    Used without proof in Eq. 1-4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FLEX-CLIP: Feature-Level GEneration Network Enhanced CLIP for X-shot Cross-modal Retrieval." pith.science (2026). https://pith.science/paper/SKVX3AOU

@misc{pith2026241117454,
  author       = {Pith},
  title        = {Pith review of: FLEX-CLIP: Feature-Level GEneration Network Enhanced CLIP for X-shot Cross-modal Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SKVX3AOU}},
  note         = {Machine review of arXiv:2411.17454}
}
read the original abstract

Given a query from one modality, few-shot cross-modal retrieval (CMR) retrieves semantically similar instances in another modality with the target domain including classes that are disjoint from the source domain. Compared with classical few-shot CMR methods, vision-language pretraining methods like CLIP have shown great few-shot or zero-shot learning performance. However, they still suffer challenges due to (1) the feature degradation encountered in the target domain and (2) the extreme data imbalance. To tackle these issues, we propose FLEX-CLIP, a novel Feature-level Generation Network Enhanced CLIP. FLEX-CLIP includes two training stages. In multimodal feature generation, we propose a composite multimodal VAE-GAN network to capture real feature distribution patterns and generate pseudo samples based on CLIP features, addressing data imbalance. For common space projection, we develop a gate residual network to fuse CLIP features with projected features, reducing feature degradation in X-shot scenarios. Experimental results on four benchmark datasets show a 7%-15% improvement over state-of-the-art methods, with ablation studies demonstrating enhancement of CLIP features.

Figures

Figures reproduced from arXiv: 2411.17454 by the authors.

Figure 1
Figure 1. The illustration of the zero-shot cross-modal retrieval dataset setting. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An illustration of classical zero-shot CMR methods VLP-based method, and our method. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The whole architecture of our proposed model FLEX-CLIP. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The illustration of the dataset splitting setting. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: The mAP curve of the ablation experiment of the composite VAE-GAN network. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: The examples of top-5 retrieval results of our FLEX-CLIP and the [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 52 canonical work pages

  1. [1]

    A new approach to cross-modal multimedia retrieval,

    N. Rasiwasia, J. Costa Pereira, E. Coviello, G. Doyle, G. R. Lanck- riet, R. Levy, and N. Vasconcelos, “A new approach to cross-modal multimedia retrieval,” in Proceedings of the 18th ACM International Conference on Multimedia, MM ’10, (New York, NY , USA), p. 251–260, Association for Computing Machinery, 2010

    work page 2010

  2. [2]

    Joint feature selection and subspace learning for cross-modal retrieval,

    K. Wang, R. He, L. Wang, W. Wang, and T. Tan, “Joint feature selection and subspace learning for cross-modal retrieval,” IEEE transactions on pattern analysis and machine intelligence , vol. 38, no. 10, pp. 2010– 2023, 2015

    work page 2010

  3. [3]

    Deep adversarial metric learning for cross-modal retrieval,

    X. Xu, L. He, H. Lu, L. Gao, and Y . Ji, “Deep adversarial metric learning for cross-modal retrieval,” World Wide Web, vol. 22, pp. 657–672, 2019

    work page 2019

  4. [4]

    Cross-modal retrieval with cnn visual features: A new baseline,

    Y . Wei, Y . Zhao, C. Lu, S. Wei, L. Liu, Z. Zhu, and S. Yan, “Cross-modal retrieval with cnn visual features: A new baseline,” IEEE transactions on cybernetics, vol. 47, no. 2, pp. 449–460, 2016

    work page 2016

  5. [5]

    Correlated features synthesis and alignment for zero-shot cross-modal retrieval,

    X. Xu, K. Lin, H. Lu, L. Gao, and H. T. Shen, “Correlated features synthesis and alignment for zero-shot cross-modal retrieval,” in Pro- ceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval , pp. 1419–1428, 2020

    work page 2020

  6. [6]

    Ternary adversarial networks with self-supervision for zero-shot cross-modal retrieval,

    X. Xu, H. Lu, J. Song, Y . Yang, H. T. Shen, and X. Li, “Ternary adversarial networks with self-supervision for zero-shot cross-modal retrieval,” IEEE Transactions on Cybernetics , vol. 50, no. 6, pp. 2400– 2413, 2020

    work page 2020

  7. [7]

    Zero-shot cross-media embedding learning with dual adversarial distribution network,

    J. Chi and Y . Peng, “Zero-shot cross-media embedding learning with dual adversarial distribution network,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, no. 4, pp. 1173–1187, 2019

    work page 2019

  8. [8]

    Learning cross-aligned latent embeddings for zero-shot cross-modal retrieval,

    K. Lin, X. Xu, L. Gao, Z. Wang, and H. T. Shen, “Learning cross-aligned latent embeddings for zero-shot cross-modal retrieval,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 34, pp. 11515– 11522, 2020

    work page 2020

Show all 58 references
  1. [9]

    Devise: A deep visual-semantic embedding model,

    A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. Ranzato, and T. Mikolov, “Devise: A deep visual-semantic embedding model,” Advances in neural information processing systems , vol. 26, 2013

    2013

  2. [10]

    Label-embedding for attribute-based classification,

    Z. Akata, F. Perronnin, Z. Harchaoui, and C. Schmid, “Label-embedding for attribute-based classification,” in Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 819–826, 2013

    2013

  3. [11]

    Zero- shot cross-modal retrieval by assembling autoencoder and generative adversarial network,

    X. Xu, J. Tian, K. Lin, H. Lu, J. Shao, and H. T. Shen, “Zero- shot cross-modal retrieval by assembling autoencoder and generative adversarial network,” ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) , vol. 17, no. 1s, pp. 1–17, 2021

    2021

  4. [12]

    Multimodal disentanglement variational autoencoders for zero-shot cross-modal re- trieval,

    J. Tian, K. Wang, X. Xu, Z. Cao, F. Shen, and H. T. Shen, “Multimodal disentanglement variational autoencoders for zero-shot cross-modal re- trieval,” inProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval , pp. 960–969, 2022

    2022

  5. [13]

    Joint feature syn- thesis and embedding: Adversarial cross-modal retrieval revisited,

    X. Xu, K. Lin, Y . Yang, A. Hanjalic, and H. T. Shen, “Joint feature syn- thesis and embedding: Adversarial cross-modal retrieval revisited,”IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 44, no. 6, pp. 3030–3047, 2022

    2022

  6. [14]

    Dual adversarial networks for zero-shot cross-media retrieval.,

    J. Chi and Y . Peng, “Dual adversarial networks for zero-shot cross-media retrieval.,” in IJCAI, pp. 663–669, 2018

    2018

  7. [15]

    Contrastive embedding for gener- alized zero-shot learning,

    Z. Han, Z. Fu, S. Chen, and J. Yang, “Contrastive embedding for gener- alized zero-shot learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 2371–2381, 2021

    2021

  8. [16]

    Semantic-related feature generation for generalized zero-shot learning,

    P. Ma, R. Wu, and H. Lu, “Semantic-related feature generation for generalized zero-shot learning,” in 2022 IEEE International Conference on Multimedia and Expo (ICME) , pp. 1–6, 2022. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 11

    2022

  9. [17]

    F-vaegan-d2: A feature generating framework for any-shot learning,

    Y . Xian, S. Sharma, B. Schiele, and Z. Akata, “F-vaegan-d2: A feature generating framework for any-shot learning,” in2019 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) , pp. 10267– 10276, 2019

    2019

  10. [18]

    Triplet-based deep hashing network for cross-modal retrieval,

    C. Deng, Z. Chen, X. Liu, X. Gao, and D. Tao, “Triplet-based deep hashing network for cross-modal retrieval,” IEEE Transactions on Image Processing, vol. 27, no. 8, pp. 3893–3903, 2018

    2018

  11. [19]

    Matching images and text with multi-modal tensor fusion and re- ranking,

    T. Wang, X. Xu, Y . Yang, A. Hanjalic, H. T. Shen, and J. Song, “Matching images and text with multi-modal tensor fusion and re- ranking,” in Proceedings of the 27th ACM international conference on multimedia, pp. 12–20, 2019

    2019

  12. [20]

    Adversarial cross-modal retrieval,

    B. Wang, Y . Yang, X. Xu, A. Hanjalic, and H. T. Shen, “Adversarial cross-modal retrieval,” in Proceedings of the 25th ACM international conference on Multimedia , pp. 154–162, 2017

    2017

  13. [21]

    Clip and complementary methods,

    M. Hafner, M. Katsantoni, T. K ¨oster, J. Marks, J. Mukherjee, D. Staiger, J. Ule, and M. Zavolan, “Clip and complementary methods,” Nature Reviews Methods Primers , vol. 1, no. 1, p. 20, 2021

    2021

  14. [22]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning , pp. 8748–8763, PMLR, 2021

    2021

  15. [23]

    Hotelling, Relations Between Two Sets of Variates, pp

    H. Hotelling, Relations Between Two Sets of Variates, pp. 162–190. New York, NY: Springer New York, 1992

    1992

  16. [24]

    Learning cross-media joint representation with sparse and semisupervised regularization,

    X. Zhai, Y . Peng, and J. Xiao, “Learning cross-media joint representation with sparse and semisupervised regularization,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 24, no. 6, pp. 965–978, 2014

    2014

  17. [25]

    Mars: Learning modality-agnostic representation for scalable cross-media retrieval,

    Y . Wang and Y . Peng, “Mars: Learning modality-agnostic representation for scalable cross-media retrieval,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 7, pp. 4765–4777, 2022

    2022

  18. [26]

    Learning cross-modal retrieval with noisy labels,

    P. Hu, X. Peng, H. Zhu, L. Zhen, and J. Lin, “Learning cross-modal retrieval with noisy labels,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 5399–5409, 2021

    2021

  19. [27]

    Multimodal machine learning: A survey and taxonomy,

    T. Baltru ˇsaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,” IEEE transactions on pattern anal- ysis and machine intelligence , vol. 41, no. 2, pp. 423–443, 2018

    2018

  20. [28]

    Multimodal learning with transform- ers: A survey,

    P. Xu, X. Zhu, and D. A. Clifton, “Multimodal learning with transform- ers: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023

    2023

  21. [29]

    A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets,

    K. Bayoudh, R. Knani, F. Hamdaoui, and A. Mtibaa, “A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets,” The Visual Computer, vol. 38, no. 8, pp. 2939–2970, 2022

    2022

  22. [30]

    A survey on deep learning for multimodal data fusion,

    J. Gao, P. Li, Z. Chen, and J. Zhang, “A survey on deep learning for multimodal data fusion,” Neural Computation, vol. 32, no. 5, pp. 829– 864, 2020

    2020

  23. [31]

    Deep supervised cross-modal retrieval,

    L. Zhen, P. Hu, X. Wang, and D. Peng, “Deep supervised cross-modal retrieval,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 10394–10403, 2019

    2019

  24. [32]

    Probabilistic embeddings for cross-modal retrieval,

    S. Chun, S. J. Oh, R. S. De Rezende, Y . Kalantidis, and D. Larlus, “Probabilistic embeddings for cross-modal retrieval,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8415–8424, 2021

    2021

  25. [33]

    A review of generalized zero-shot learning meth- ods,

    F. Pourpanah, M. Abdar, Y . Luo, X. Zhou, R. Wang, C. P. Lim, X.-Z. Wang, and Q. J. Wu, “A review of generalized zero-shot learning meth- ods,” IEEE transactions on pattern analysis and machine intelligence , 2022

    2022

  26. [34]

    Robust fine-tuning of zero-shot models,

    M. Wortsman, G. Ilharco, J. W. Kim, M. Li, S. Kornblith, R. Roelofs, R. G. Lopes, H. Hajishirzi, A. Farhadi, H. Namkoong, et al. , “Robust fine-tuning of zero-shot models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 7959– 7971, 2022

    2022

  27. [35]

    f-vaegan-d2: A feature generating framework for any-shot learning,

    Y . Xian, S. Sharma, B. Schiele, and Z. Akata, “f-vaegan-d2: A feature generating framework for any-shot learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 10275–10284, 2019

    2019

  28. [36]

    Towards principled methods for training generative adversarial networks,

    M. Arjovsky and L. Bottou, “Towards principled methods for training generative adversarial networks,” in International Conference on Learn- ing Representations, 2016

    2016

  29. [37]

    Improved training of wasserstein gans,

    I. Gulrajani, F. Ahmed, M. Arjovsky, V . Dumoulin, and A. C. Courville, “Improved training of wasserstein gans,”Advances in neural information processing systems, vol. 30, 2017

    2017

  30. [38]

    Spectral normal- ization for generative adversarial networks,

    T. Miyato, T. Kataoka, M. Koyama, and Y . Yoshida, “Spectral normal- ization for generative adversarial networks,” in International Conference on Learning Representations , 2018

    2018

  31. [39]

    Collecting image annotations using amazon’s mechanical turk,

    C. Rashtchian, P. Young, M. Hodosh, and J. Hockenmaier, “Collecting image annotations using amazon’s mechanical turk,” in Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon’s Mechanical Turk , pp. 139–147, 2010

    2010

  32. [40]

    Nus-wide: a real-world web image database from national university of singapore,

    T.-S. Chua, J. Tang, R. Hong, H. Li, Z. Luo, and Y . Zheng, “Nus-wide: a real-world web image database from national university of singapore,” in Proceedings of the ACM international conference on image and video retrieval, pp. 1–9, 2009

    2009

  33. [41]

    Hcmsl: Hybrid cross- modal similarity learning for cross-modal retrieval,

    C. Zhang, J. Song, X. Zhu, L. Zhu, and S. Zhang, “Hcmsl: Hybrid cross- modal similarity learning for cross-modal retrieval,” ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), vol. 17, no. 1s, pp. 1–22, 2021

    2021

  34. [42]

    Cross modal retrieval with querybank normalisation,

    S.-V . Bogolin, I. Croitoru, H. Jin, Y . Liu, and S. Albanie, “Cross modal retrieval with querybank normalisation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 5194–5205, 2022

    2022

  35. [43]

    Comparative analysis on cross-modal information retrieval: A review,

    P. Kaur, H. S. Pannu, and A. K. Malhi, “Comparative analysis on cross-modal information retrieval: A review,” Computer Science Review, vol. 39, p. 100336, 2021

    2021

  36. [44]

    Zero-shot learning through cross-modal transfer,

    R. Socher, M. Ganjoo, C. D. Manning, and A. Ng, “Zero-shot learning through cross-modal transfer,” Advances in neural information process- ing systems, vol. 26, 2013

    2013

  37. [45]

    Research progress of zero-shot learning,

    X. Sun, J. Gu, and H. Sun, “Research progress of zero-shot learning,” Applied Intelligence, vol. 51, pp. 3600–3614, 2021

    2021

  38. [46]

    Generalized zero-shot learning with deep calibration network,

    S. Liu, M. Long, J. Wang, and M. I. Jordan, “Generalized zero-shot learning with deep calibration network,” Advances in neural information processing systems, vol. 31, 2018

    2018

  39. [47]

    Adaptive and generative zero-shot learning,

    Y .-Y . Chou, H.-T. Lin, and T.-L. Liu, “Adaptive and generative zero-shot learning,” in International conference on learning representations, 2020

    2020

  40. [48]

    Free: Feature refinement for generalized zero-shot learning,

    S. Chen, W. Wang, B. Xia, Q. Peng, X. You, F. Zheng, and L. Shao, “Free: Feature refinement for generalized zero-shot learning,” in Pro- ceedings of the IEEE/CVF international conference on computer vision , pp. 122–131, 2021

    2021

  41. [49]

    Attribute proto- type network for zero-shot learning,

    W. Xu, Y . Xian, J. Wang, B. Schiele, and Z. Akata, “Attribute proto- type network for zero-shot learning,” Advances in Neural Information Processing Systems, vol. 33, pp. 21969–21980, 2020

    2020

  42. [50]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 770–778, 2016

    2016

  43. [51]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. D. M.-W. C. Kenton and L. K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of NAACL-HLT, pp. 4171–4186, 2019

    2019

  44. [52]

    Cross-modal retrieval with correspondence autoencoder,

    F. Feng, X. Wang, and R. Li, “Cross-modal retrieval with correspondence autoencoder,” in Proceedings of the 22nd ACM International Conference on Multimedia, MM ’14, (New York, NY , USA), p. 7–16, Association for Computing Machinery, 2014

    2014

  45. [53]

    Retrieval-based knowledge augmented vision language pre-training,

    J. Rao, Z. Shan, L. Liu, Y . Zhou, and Y . Yang, “Retrieval-based knowledge augmented vision language pre-training,” in Proceedings of the 31st ACM International Conference on Multimedia , MM ’23, (New York, NY , USA), p. 5399–5409, Association for Computing Machinery, 2023

    2023

  46. [54]

    Unsupervised and pseudo-supervised vision-language alignment in visual dialog,

    F. Chen, D. Zhang, X. Chen, J. Shi, S. Xu, and B. XU, “Unsupervised and pseudo-supervised vision-language alignment in visual dialog,” in Proceedings of the 30th ACM International Conference on Multimedia , MM ’22, (New York, NY , USA), p. 4142–4153, Association for Com- putin...

    2022

  47. [55]

    Spaceclip: A vision-language pretraining framework with spatial reconstruction on text,

    B. Zou, C. Yang, C. Quan, and Y . Zhao, “Spaceclip: A vision-language pretraining framework with spatial reconstruction on text,” in Proceed- ings of the 31st ACM International Conference on Multimedia , MM ’23, (New York, NY , USA), p. 519–528, Association for Computing Machi...

    2023

  48. [56]

    Token embeddings alignment for cross-modal retrieval,

    C.-W. Xie, J. Wu, Y . Zheng, P. Pan, and X.-S. Hua, “Token embeddings alignment for cross-modal retrieval,” in Proceedings of the 30th ACM International Conference on Multimedia , MM ’22, (New York, NY , USA), p. 4555–4563, Association for Computing Machinery, 2022

    2022

  49. [57]

    Diachronic cross-modal embeddings,

    D. Semedo and J. Magalhaes, “Diachronic cross-modal embeddings,” in Proceedings of the 27th ACM International Conference on Multime- dia, MM ’19, (New York, NY , USA), p. 2061–2069, Association for Computing Machinery, 2019

    2019

  50. [58]

    Cross- modal retrieval with heterogeneous graph embedding,

    D. Chen, M. Wang, H. Chen, L. Wu, J. Qin, and W. Peng, “Cross- modal retrieval with heterogeneous graph embedding,” in Proceedings of the 30th ACM International Conference on Multimedia , MM ’22, (New York, NY , USA), p. 3291–3300, Association for Computing Machinery, 2022. Ji...

    2022

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.