Pith. sign in

REVIEW 3 major objections 6 minor 57 references

Rebalanced Vision-Language Retrieval Considering Structure-Aware Distillation

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read When text and image modalities are imbalanced, exact instance-level cross-modal matching is sub-optimal; preserving each modality's internal structure while aligning them yields better retrieval.

desk verdict Solid plug-and-play distillation recipe for cross-modal retrieval; the imbalance story is plausible but under-tested. read the letter →

arxiv 2412.10761 v1 pith:GGBYRRUU submitted 2024-12-14 cs.CV cs.AI

Yang Yang , Wenjuan Xi , Luping Zhou , Jinhui Tang This is my paper · ORCID
classification cs.CVcs.AI
keywords vision-languageretrievalmodalityimbalancestructure-awaredistillationcross-modalmatchingrelationalknowledgesingle-modalcontrastivelearningmulti-granularity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language retrieval models usually assume the two modalities can represent each other equally, but in practice text is often the stronger modality and image the weaker one. This paper argues that forcing exact instance-level cross-modal matching under that imbalance damages the latent structure of the stronger modality and hurts retrieval, not just single-modal search. To rebalance the learning, the authors add a multi-granularity distillation loss to the standard cross-modal matching loss: two single-modal teacher models transfer their intra-modal similarity structure to the cross-modal student, and the student is trained to keep its relational geometry close to a learned fusion of the two teacher structures. On four benchmarks the augmented model raises image-to-text and text-to-image Recall@1 by up to 11.4 points over its base model and also improves single-modal retrieval. The paper's core claim is that preserving each modality's internal structure while learning cross-modal consistency is the right way to handle imbalanced modalities.

What carries the argument

The machinery is a multi-granularity distillation module attached to the cross-modal student model. It consists of representation-level distillation losses ($\ell_{iic}$, $\ell_{ttc}$) that align each modality's [CLS] embeddings with the corresponding single-modal teacher, and a structure-aware distillation loss ($\ell_{sa}$) computed as the MAE between the student's cross-modal relational matrix $S_{IT}$ and a fused teacher matrix $S_O = \lambda S_I + (1-\lambda) S_T$, where $\lambda$ is learned. The teachers are trained with prototype-aware contrastive learning so that semantically similar instances are not treated as negatives. The total objective is $\mathcal{L} = \ell_{cr} + \ell_{md}$, with $\ell_{cr} = \ell_{itc} + \ell_{itm}$.

What would settle it

Train the same method on a dataset or synthetic modification where images are engineered to be the stronger modality (for example, high-resolution clean images paired with corrupted or repetitive captions). If the method still improves retrieval but the learned fusion weight $\lambda$ shifts toward the image teacher, the sufficiency-driven explanation is supported; if the structure damage persists regardless of which modality is stronger, or the method fails when the imbalance direction flips, the causal story is wrong.

Watch

Extended reading notes

Core claim

The paper's central claim is that 'ultimate cross-modal matching is generally sub-optimal for cross-modal retrieval when imbalanced modalities exist.' Using single-modal retrieval performance as a proxy for modality sufficiency, the authors identify text as the strong modality and images as the weak one across MS-COCO, FLICKR30K, and Vizwiz. They show that standard cross-modal training degrades the text model's text-to-text retrieval more than it improves the image model's image-to-image retrieval, and that distillation from the strong to the weak modality helps the weak modality while distillation from the weak to the strong hurts the strong one. Their proposed solution, X-VLM*+, keeps the cross-modal matching losses (ITC and ITM) but adds representation-level distillation and structure-aware distillation, so the learned common space preserves the intra-modal structure of both single-modal teachers. The experiments report consistent gains in cross-modal, single-modal, and mixed retrieval across four datasets, including Recall@1 gains of 6.2/11.4/7.5/6.2 for image-to-text and 3.1/5.1/7.9/6.5 for text-to-image over X-VLM.

Load-bearing premise

The load-bearing premise is that the measured single-modal retrieval gap between text and image reflects a real difference in modality sufficiency, and that this asymmetry, rather than optimizer dynamics or architecture capacity, is what causes cross-modal training to damage single-modal structure.

Editorial extensions

If this is right

  • Treating instance-level matching as the only objective is suboptimal when modalities are imbalanced; adding structure preservation raises cross-modal recall.
  • The extra distillation works as a plug-and-play module: adding it to SCAN, VSRN, ALBEF, and X-VLM improves their retrieval on the tested datasets.
  • Single-modal retrieval also improves, meaning the method partially reverses the degradation of the strong modality that cross-modal training usually causes.
  • The learned fusion coefficient $\lambda$ settles near 0.21 and 0.33, indicating that the text (strong) teacher contributes more structural guidance than the image (weak) teacher.
  • Applying the module during fine-tuning of a large pre-trained model improves single-modal retrieval while keeping cross-modal performance, so the idea is not limited to training from scratch.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the paper's causal story is to reverse the imbalance (for example, pair unusually clean, detailed images with noisy or repetitive captions) and check whether the learned $\lambda$ shifts toward the image teacher; the sufficiency-based diagnosis predicts it would.
  • The relational-consistency term acts as a structure regularizer on top of contrastive alignment, so the same multi-teacher distillation design could transfer to other imbalanced multi-modal retrieval settings, such as audio-text or video-text search.
  • Because the module is architecture-agnostic, it could be appended when fine-tuning large pre-trained vision-language models to prevent single-modal representation collapse, a use the paper demonstrates on X-VLM but does not claim as a general recipe.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper addresses vision-language retrieval under modality imbalance. It claims that exact/ultimate cross-modal matching is generally sub-optimal when the two modalities have different sufficiency, and it proposes a multi-granularity distillation loss composed of representation-level contrastive distillation (ℓiic, ℓttc) and structure-aware distillation (ℓsa) from two single-modal teachers, with a learnable fusion coefficient λ. The student model is X-VLM without the bounding-box module; experiments on MS-COCO (1K and 5K), FLICKR30K, and Vizwiz report cross-modal, single-modal, and mixed retrieval performance, together with ablations, parameter analyses, and generalization experiments on SCAN, VSRN, and ALBEF. The central reported result is that X-VLM*+ improves over X-VLM on I2T and T2I Recall@1 across all four test settings.

Significance. If the empirical results hold, the paper offers a practically useful plug-and-play training regularizer: it consistently improves cross-modal retrieval on four benchmarks and improves or maintains single-modal retrieval relative to most baselines, and the ablation shows that each loss term contributes. The generalization experiments to SCAN, VSRN, and ALBEF strengthen the claim that the module is not tied to a single architecture, and the comparison of MAE/MSE/WD as relational matching criteria is a useful practical detail. The paper does not contain a theoretical derivation or a controlled causal experiment, and the headline causal claim about modality imbalance is not established; the method may be effective for reasons other than 'rebalancing'. The significance is therefore currently that of a strong empirical method rather than a validated mechanism.

major comments (3)
  1. [Abstract and Section III-A] The statement that 'ultimate cross-modal matching is generally sub-optimal for cross-modal retrieval when imbalanced modalities exist' is a universal causal claim, but the support is observational: Table I shows that text encoders perform better than image encoders, Table II shows differences of only 0.1 NDCG points in the S2W/W2S experiments, and Fig. 1 shows structural degradation after cross-modal training. No experiment varies the degree or direction of modality imbalance, so the paper does not demonstrate that imbalance is the causal driver of the observed sub-optimality. Please add controlled experiments that manipulate modality sufficiency (for example, noise-corrupted images, truncated or paraphrased captions, or synthetic settings with reversed sufficiency) or explicitly restrict the claim to the studied text-as-strong/image-as-weak setting.
  2. [Section III-C and Section IV-F] The structure-aware target is SO = λSI + (1−λ)ST, and Section IV-F reports final learned values of λ = 0.21 on FLICKR30K and λ = 0.33 on Vizwiz. These values put more weight on the text teacher's structure ST, i.e., the 'strong' modality, which is not obviously a rebalancing of the weak image modality; one might instead expect a rebalancing method to emphasize SI. This discrepancy should be explained, and the paper should report ablations with fixed λ = 0 and λ = 1, including single-modal retrieval results, so the reader can see what the learned fusion is actually contributing.
  3. [Tables III, IV, and V] The principal empirical claims are reported as single point estimates. Improvements vary substantially across datasets (for example, I2T R@1 gains of 6.2, 11.4, 7.5, and 6.2 in Table III), and some single-modal differences are very small (for example, Table IV, FLICKR30K I2I NDCG@10: 64.1 vs. 64.3). Without variance estimates or significance tests, the reader cannot judge whether the smaller differences are reliable. Please report mean and standard deviation over at least three seeds for the main comparisons and the Table V ablations, or state explicitly that only single runs were performed.
minor comments (6)
  1. [Section IV-A] The text says experiments are conducted on 'three datasets' although the paper evaluates on four test settings (MS-COCO 1K, MS-COCO 5K, FLICKR30K, and Vizwiz); please clarify the counting.
  2. [Table II and Section III-A] The abbreviations S2W@Image and W2S@Text are used without expansion; please define them as strong-to-weak and weak-to-strong distillation in the text.
  3. [Fig. 2] The caption introduces 'ViT2BERT' without defining the architecture; please explain what this model is or remove the name if it is an earlier version of the student model.
  4. [References] References [9] and [12] appear to be the same paper (Diao et al., AAAI 2021, 'Similarity reasoning and filtration for image-text matching'); please remove the duplicate.
  5. [Section III-C] The teacher models are said to use 'unsupervised prototype-aware contrastive learning', but the mechanism is not described in the main text; please provide the objective or point the reader to the specific supplement section with the equations.
  6. [Section III-B, Eq. (2)] The two cosine similarities d(I,T) and d(T,I) are the same quantity up to argument order; please define the similarity function once and reuse it to avoid notational clutter.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central result is a new loss combination trained and evaluated on external benchmarks; self-citations are peripheral.

full rationale

The paper's derivation chain is not circular. It starts from an observational measurement of modality sufficiency (Table I), observes degradation of single-modal structure after cross-modal training (Fig. 1, Table II), then proposes an objective L = ℓcr + ℓmd with ℓmd = ℓiic + ℓttc + ℓsa, and finally evaluates on held-out splits of MS-COCO, Flickr30K, and Vizwiz against independent baselines. No fitted parameter is renamed as a prediction: the learnable λ is optimized end-to-end with the training objective, and the reported retrieval numbers are on test sets, so this is ordinary supervised training rather than a by-construction match to the evaluation metric. The teacher models are trained on the same datasets but without retrieval labels, and the student must still optimize the cross-modal matching losses ℓitc and ℓitm, so the distillation losses do not by themselves fix the cross-modal retrieval scores. The cited 'Proposition 2 in [16]' is an external result, and the self-citations that appear (e.g., [4], [18], [23], [53]-[55]) are contextual and not load-bearing for the central claim. The main weakness is that modality imbalance is never manipulated, so the causal interpretation that imbalance causes the degradation and that 'rebalancing' fixes it is underdetermined; this is an experimental-design limitation, not circular reasoning. No circular step can be exhibited from the paper's own equations or citations.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim depends on the domain assumption that single-modal retrieval performance proxies modality sufficiency, and on the ad hoc design choice that teacher similarity matrices define an optimal structure to transfer. No new physical entities are introduced; the learnable lambda and temperature are tuned parameters rather than invented entities.

free parameters (3)
  • fusion coefficient lambda = 0.21 (Flickr30K), 0.33 (Vizwiz) when learned
    learned scalar fusing teacher similarity matrices SI and ST; dataset-dependent, tuned to maximize retrieval performance (Section IV-F).
  • temperature tau = 0.1
    temperature in contrastive losses; selected after sweeping {0.1, 0.2, 0.5, 2, 3} on Flickr30K and Vizwiz (Section IV-F).
  • batch size J = 36
    batch size in contrastive losses; selected after sweeping {12, 24, 36, 48} (Section IV-F).
assumptions (3)
  • domain assumption Modality sufficiency is proportional to the performance of the optimal single-modal model (Proposition 2 of Wang and Zhou 2013).
    Used to classify text as strong and image as weak based on Table I; this bridges an abstract 'sufficiency' notion to an observable retrieval score.
  • ad hoc to paper The structure of intra-modal representations, as captured by pairwise similarity matrices of teacher models, is a desirable target for the cross-modal student.
    Underpins the structure-aware distillation loss l_sa; no proof that teacher structure is optimal for cross-modal retrieval, only empirical support.
  • domain assumption Mean Absolute Error between similarity matrices is an appropriate measure of geometric consistency.
    Chosen as the distillation objective; Table VI shows MAE outperforms KL but is comparable to MSE and Wasserstein distance, suggesting the specific metric is not critical.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rebalanced Vision-Language Retrieval Considering Structure-Aware Distillation." pith.science (2026). https://pith.science/paper/GGBYRRUU

@misc{pith2026241210761,
  author       = {Pith},
  title        = {Pith review of: Rebalanced Vision-Language Retrieval Considering Structure-Aware Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GGBYRRUU}},
  note         = {Machine review of arXiv:2412.10761}
}
read the original abstract

Vision-language retrieval aims to search for similar instances in one modality based on queries from another modality. The primary objective is to learn cross-modal matching representations in a latent common space. Actually, the assumption underlying cross-modal matching is modal balance, where each modality contains sufficient information to represent the others. However, noise interference and modality insufficiency often lead to modal imbalance, making it a common phenomenon in practice. The impact of imbalance on retrieval performance remains an open question. In this paper, we first demonstrate that ultimate cross-modal matching is generally sub-optimal for cross-modal retrieval when imbalanced modalities exist. The structure of instances in the common space is inherently influenced when facing imbalanced modalities, posing a challenge to cross-modal similarity measurement. To address this issue, we emphasize the importance of meaningful structure-preserved matching. Accordingly, we propose a simple yet effective method to rebalance cross-modal matching by learning structure-preserved matching representations. Specifically, we design a novel multi-granularity cross-modal matching that incorporates structure-aware distillation alongside the cross-modal matching loss. While the cross-modal matching loss constraints instance-level matching, the structure-aware distillation further regularizes the geometric consistency between learned matching representations and intra-modal representations through the developed relational matching. Extensive experiments on different datasets affirm the superior cross-modal retrieval performance of our approach, simultaneously enhancing single-modal retrieval capabilities compared to the baseline models.

Figures

Figures reproduced from arXiv: 2412.10761 by the authors.

Figure 1
Figure 1. The impact of imbalanced modalities on single-modal retrieval [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. T-SNE visualization of Best T2T, BLIP* T2T, ViT2BERT T2T on the FLICKR30K dataset, where BLIP* T2T uses the ViT/B and BERT as backbones. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 1
Figure 1. From Fig. 1, we observe that the text-to-text (T2T) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figures from the paper (5 more)
Figure 3
Figure 3. Figure 3: Illustration of our framework. Expanding on the framework of cross-modal matching, we incorporate a single-modal teacher network. Our multi [PITH_FULL_IMAGE:figures/full_fig_p004_3.png]
Figure 4
Figure 4. Figure 4: The results of I2IT@10 (a–d), T2IT@10 (e–h) of mixed retrieval task. The method with “+” sign, i.e., X-VLM*+, is our method. modal retrieval models as student models to validate the generalization capability of the multi-granularity distillation. Without any loss of ge…
Figure 5
Figure 5. Figure 5: Parameter analyses. We verify the influence of parameters our method under FLICKR30K and Vizwiz datasets. TABLE VI COMPARE MAE WITH MSE, KL, AND WD. EVALUATION CRITERIA ARE R@A AND NDCG@A. Methods FLICKR30K Vizwiz I2T T2I I2T T2I @1 @5 @10 @1 @5 @10 @1 @5 @10 @1 @5 @10…
Figure 6
Figure 6. Figure 6: Qualitative results of cross-modal retrieval. For each image query, we show the top-5 ranked sentences. For each sentence query, we show the top-3 ranked images, ranking from left to right (Best viewed in green). The examples are sampled from the FLICKR30K dataset. E. …
Figure 7
Figure 7. Figure 7: Qualitative results of mixed retrieval. For each query, we show the top-5 ranked instances, including images and sentences (Correct results viewed in green). The examples are sampled from the FLICKR30K dataset. respectively. This indicates that the language modality (i…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 54 canonical work pages

  1. [1]

    Vilbert: Pre- training task-agnostic visiolinguistic representations for vision-and-language tasks,

    J. Lu, D. Batra, D. Parikh, and S. Lee, “Vilbert: Pre- training task-agnostic visiolinguistic representations for vision-and-language tasks,” in NeurIPS, 2019, pp. 13– 23

    work page 2019

  2. [2]

    Explore instance similarity: An in- stance correlation based hashing method for multi-label cross-model retrieval,

    C. Huang, X. Luo, J. Zhang, Q. Liao, X. Wang, Z. L. Jiang, and S. Qi, “Explore instance similarity: An in- stance correlation based hashing method for multi-label cross-model retrieval,” IPM, vol. 57, no. 2, p. 102165, 2020

    work page 2020

  3. [3]

    Dynamic contrastive distillation for image-text retrieval,

    J. Rao, L. Ding, S. Qi, M. Fang, Y . Liu, L. Shen, and D. Tao, “Dynamic contrastive distillation for image-text retrieval,” TMM, vol. 25, pp. 8383–8395, 2023

    work page 2023

  4. [4]

    Covlr: Coordi- nating cross-modal consistency and intra-modal relations for vision-language retrieval,

    F. Wan, X. Wu, Z. Guan, and Y . Yang, “Covlr: Coordi- nating cross-modal consistency and intra-modal relations for vision-language retrieval,” in ICME, 2024, pp. 1–6

    work page 2024

  5. [5]

    Deep relation embedding for cross-modal retrieval,

    Y . Zhang, W. Zhou, M. Wang, Q. Tian, and H. Li, “Deep relation embedding for cross-modal retrieval,” TIP, vol. 30, pp. 617–627, 2021

    work page 2021

  6. [6]

    Stacked cross attention for image-text matching,

    K. Lee, X. Chen, G. Hua, H. Hu, and X. He, “Stacked cross attention for image-text matching,” in ECCV, 2018, pp. 212–228

    work page 2018

  7. [7]

    Cross- modal retrieval with partially mismatched pairs,

    P. Hu, Z. Huang, D. Peng, X. Wang, and X. Peng, “Cross- modal retrieval with partially mismatched pairs,” TPAMI, vol. 45, no. 8, pp. 9595–9610, 2023

    work page 2023

  8. [8]

    VSE++: improving visual-semantic embeddings with hard nega- tives,

    F. Faghri, D. J. Fleet, J. R. Kiros, and S. Fidler, “VSE++: improving visual-semantic embeddings with hard nega- tives,” in BMVC, 2018, p. 12

    work page 2018

Show all 57 references
  1. [10]

    Modality-specific cross- modal similarity measurement with recurrent attention network,

    Y . Peng, J. Qi, and Y . Yuan, “Modality-specific cross- modal similarity measurement with recurrent attention network,” TIP, vol. 27, no. 11, pp. 5585–5599, 2018

    2018

  2. [11]

    Show your faith: Cross-modal confidence-aware network for image- text matching,

    H. Zhang, Z. Mao, K. Zhang, and Y . Zhang, “Show your faith: Cross-modal confidence-aware network for image- text matching,” in AAAI, 2022, pp. 3262–3270

    2022

  3. [12]

    Similarity rea- soning and filtration for image-text matching,

    H. Diao, Y . Zhang, L. Ma, and H. Lu, “Similarity rea- soning and filtration for image-text matching,” in AAAI, 2021, pp. 1218–1226

    2021

  4. [13]

    Towards lightweight transformer IEEE TRANSACTIONS ON IMAGE PROCESSING, DECEMBER 2024 12 via group-wise transformation for vision-and-language tasks,

    G. Luo, Y . Zhou, X. Sun, Y . Wang, L. Cao, Y . Wu, F. Huang, and R. Ji, “Towards lightweight transformer IEEE TRANSACTIONS ON IMAGE PROCESSING, DECEMBER 2024 12 via group-wise transformation for vision-and-language tasks,” TIP, vol. 31, pp. 3386–3398, 2022

    2024

  5. [14]

    BLIP: boot- strapping language-image pre-training for unified vision- language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. C. H. Hoi, “BLIP: boot- strapping language-image pre-training for unified vision- language understanding and generation,” in ICML, 2022, pp. 12 888–12 900

    2022

  6. [15]

    Learning semantic relationship among instances for image-text matching,

    Z. Fu, Z. Mao, Y . Song, and Y . Zhang, “Learning semantic relationship among instances for image-text matching,” in CVPR, 2023, pp. 15 159–15 168

    2023

  7. [16]

    Co-training with insufficient views,

    W. Wang and Z. Zhou, “Co-training with insufficient views,” in ACML, 2013, pp. 467–482

    2013

  8. [17]

    Multi-modal mutual attention and iterative interaction for referring image segmentation,

    C. Liu, H. Ding, Y . Zhang, and X. Jiang, “Multi-modal mutual attention and iterative interaction for referring image segmentation,” TIP, vol. 32, pp. 3054–3065, 2023

    2023

  9. [18]

    Auxiliary information regularized machine for multiple modality feature learning,

    Y . Yang, H. Ye, D. Zhan, and Y . Jiang, “Auxiliary information regularized machine for multiple modality feature learning,” in IJCAI, 2015, pp. 1033–1039

    2015

  10. [19]

    Modality competition: What makes joint training of multi-modal network fail in deep learning? (provably),

    Y . Huang, J. Lin, C. Zhou, H. Yang, and L. Huang, “Modality competition: What makes joint training of multi-modal network fail in deep learning? (provably),” in ICML, vol. 162, 2022, pp. 9226–9259

    2022

  11. [20]

    BERT: pre-training of deep bidirectional transformers for lan- guage understanding,

    J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for lan- guage understanding,” in NAACL, 2019, pp. 4171–4186

    2019

  12. [21]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in ICCV, 2021, pp. 9992–10 002

    2021

  13. [22]

    Visualizing data using t-sne

    L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” JMLR, vol. 9, no. 11, 2008

    2008

  14. [23]

    Rethinking label-wise cross-modal retrieval from A semantic sharing perspective,

    Y . Yang, C. Zhang, Y . Xu, D. Yu, D. Zhan, and J. Yang, “Rethinking label-wise cross-modal retrieval from A semantic sharing perspective,” in IJCAI, 2021, pp. 3300– 3306

    2021

  15. [24]

    J. Yu, W. Zhang, Y . Lu, Z. Qin, Y . Hu, J. Tan, and Q. Wu, “Reasoning on the relation: Enhancing visual represen- tation for visual question answering and cross-modal retrieval: Enhancing visual representation for visual ques- tion answering and cross-modal retrieval,” TMM, v...

    2020

  16. [25]

    Joint feature synthesis and embedding: Adversarial cross-modal retrieval revisited,

    X. Xu, K. Lin, Y . Yang, A. Hanjalic, and H. T. Shen, “Joint feature synthesis and embedding: Adversarial cross-modal retrieval revisited,” TPAMI, vol. 44, no. 6, pp. 3030–3047, 2022

    2022

  17. [26]

    Ad- versarial graph convolutional network for cross-modal retrieval,

    X. Dong, L. Liu, L. Zhu, L. Nie, and H. Zhang, “Ad- versarial graph convolutional network for cross-modal retrieval,” TCSV, vol. 32, no. 3, pp. 1634–1645, 2022

    2022

  18. [27]

    Fine-grained visual textual alignment for cross-modal retrieval using transformer encoders,

    N. Messina, G. Amato, A. Esuli, F. Falchi, C. Gennaro, and S. Marchand-Maillet, “Fine-grained visual textual alignment for cross-modal retrieval using transformer encoders,” TOMCCAP, vol. 17, no. 4, pp. 128:1–128:23, 2021

    2021

  19. [28]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in ICML, 2021, pp. 8748–8763

    2021

  20. [29]

    Unsupervised contrastive cross-modal hashing,

    P. Hu, H. Zhu, J. Lin, D. Peng, Y . Zhao, and X. Peng, “Unsupervised contrastive cross-modal hashing,” TPAMI, vol. 45, no. 3, pp. 3877–3889, 2023

    2023

  21. [30]

    LXMERT: learning cross- modality encoder representations from transformers,

    H. Tan and M. Bansal, “LXMERT: learning cross- modality encoder representations from transformers,” in EMNLP, 2019, pp. 5099–5110

    2019

  22. [31]

    Align before fuse: Vision and language representation learning with momentum distillation,

    J. Li, R. R. Selvaraju, A. D. Gotmare, S. R. Joty, C. Xiong, and S. C. H. Hoi, “Align before fuse: Vision and language representation learning with momentum distillation,” in NeurIPS, 2021, pp. 9694–9705

    2021

  23. [32]

    What makes training multi-modal classification networks hard?

    W. Wang, D. Tran, and M. Feiszli, “What makes training multi-modal classification networks hard?” in CVPR, 2020, pp. 12 695–12 705

    2020

  24. [33]

    Trusted multi- view classification,

    Z. Han, C. Zhang, H. Fu, and J. T. Zhou, “Trusted multi- view classification,” in ICLR, 2021

    2021

  25. [34]

    Balanced multimodal learning via on-the-fly gradient modulation,

    X. Peng, Y . Wei, A. Deng, D. Wang, and D. Hu, “Balanced multimodal learning via on-the-fly gradient modulation,” in CVPR, 2022, pp. 8228–8237

    2022

  26. [35]

    Multi-grained vision lan- guage pre-training: Aligning texts with visual concepts,

    Y . Zeng, X. Zhang, and H. Li, “Multi-grained vision lan- guage pre-training: Aligning texts with visual concepts,” in ICML, 2022, pp. 25 994–26 009

    2022

  27. [36]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in CVPR, 2016, pp. 770– 778

    2016

  28. [37]

    Long short-term memory,

    S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735– 1780, 1997

    1997

  29. [38]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weis- senborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in ICLR, 2021

    2021

  30. [39]

    Relational knowledge distillation,

    W. Park, D. Kim, Y . Lu, and M. Cho, “Relational knowledge distillation,” in CVPR, 2019, pp. 3967–3976

    2019

  31. [40]

    Large-margin contrastive learning with distance polarization regularizer,

    S. Chen, G. Niu, C. Gong, J. Li, J. Yang, and M. Sugiyama, “Large-margin contrastive learning with distance polarization regularizer,” in ICML, 2021, pp. 1673–1683

    2021

  32. [41]

    DINO: DETR with improved denoising anchor boxes for end-to-end object detection,

    H. Zhang, F. Li, S. Liu, L. Zhang, H. Su, J. Zhu, L. M. Ni, and H. Shum, “DINO: DETR with improved denoising anchor boxes for end-to-end object detection,” in ICLR, 2023

    2023

  33. [42]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” JMLR, vol. 21, pp. 140:1–140:67, 2020

    2020

  34. [43]

    Microsoft coco: Common objects in context,

    T. Lin, M. Maire, S. J. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in ECCV, 2014, pp. 740–755

    2014

  35. [44]

    Deep visual-semantic alignments for generating image descriptions,

    A. Karpathy and F. Li, “Deep visual-semantic alignments for generating image descriptions,” in CVPR, 2015, pp. 3128–3137

    2015

  36. [45]

    The MIR flickr retrieval evaluation,

    M. J. Huiskes and M. S. Lew, “The MIR flickr retrieval evaluation,” in ACMMM, 2008, pp. 39–43

    2008

  37. [46]

    Captioning images taken by people who are blind,

    D. Gurari, Y . Zhao, M. Zhang, and N. Bhattacharya, “Captioning images taken by people who are blind,” in ECCV, 2020, pp. 417–434

    2020

  38. [47]

    IMRAM: iterative matching with recurrent attention memory for cross-modal image-text retrieval,

    H. Chen, G. Ding, X. Liu, Z. Lin, J. Liu, and J. Han, IEEE TRANSACTIONS ON IMAGE PROCESSING, DECEMBER 2024 13 “IMRAM: iterative matching with recurrent attention memory for cross-modal image-text retrieval,” in CVPR, 2020, pp. 12 652–12 660

    2024

  39. [48]

    Graph structured network for image-text matching,

    C. Liu, Z. Mao, T. Zhang, H. Xie, B. Wang, and Y . Zhang, “Graph structured network for image-text matching,” in CVPR, 2020, pp. 10 918–10 927

    2020

  40. [49]

    Visual semantic reasoning for image-text matching,

    K. Li, Y . Zhang, K. Li, Y . Li, and Y . Fu, “Visual semantic reasoning for image-text matching,” in ICCV, 2019, pp. 4654–4662

    2019

  41. [50]

    Negative- aware attention framework for image-text matching,

    K. Zhang, Z. Mao, Q. Wang, and Y . Zhang, “Negative- aware attention framework for image-text matching,” in CVPR, 2022, pp. 15 640–15 649

    2022

  42. [51]

    Cyclip: Cyclic contrastive language-image pretraining,

    S. Goel, H. Bansal, S. Bhatia, R. A. Rossi, V . Vinay, and A. Grover, “Cyclip: Cyclic contrastive language-image pretraining,” in NeurIPS, 2022, pp. 6704–6719

    2022

  43. [52]

    Improving multi-modal learning with uni- modal teachers,

    C. Du, T. Li, Y . Liu, Z. Wen, T. Hua, Y . Wang, and H. Zhao, “Improving multi-modal learning with uni- modal teachers,” CoRR, vol. abs/2106.11059, 2023

    2023 arXiv

  44. [53]

    Neighborhood discriminant hashing for large-scale image retrieval,

    J. Tang, Z. Li, M. Wang, and R. Zhao, “Neighborhood discriminant hashing for large-scale image retrieval,” TIP, vol. 24, no. 9, pp. 2827–2840, 2015

    2015

  45. [54]

    Learning dis- criminative cross-modality features for rgb-d saliency detection,

    F. Wang, J. Pan, S. Xu, and J. Tang, “Learning dis- criminative cross-modality features for rgb-d saliency detection,” TIP, vol. 31, pp. 1285–1297, 2022

    2022

  46. [55]

    Camera constraint-free view-based 3-d object retrieval,

    Y . Gao, J. Tang, R. Hong, S. Yan, Q. Dai, N. Zhang, and T. Chua, “Camera constraint-free view-based 3-d object retrieval,” TIP, vol. 21, no. 4, pp. 2269–2281, 2012

    2012

  47. [56]

    Picture it in your mind: generating high- level visual representations from textual descriptions,

    F. Carrara, A. Esuli, T. Fagni, F. Falchi, and A. M. Fern´andez, “Picture it in your mind: generating high- level visual representations from textual descriptions,” Information Retrieval Journal, vol. 21, no. 2-3, pp. 208– 229, 2018

    2018

  48. [57]

    Attention on attention for image captioning,

    L. Huang, W. Wang, J. Chen, and X. Wei, “Attention on attention for image captioning,” in ICCV, 2019, pp. 4633–4642

    2019

  49. [58]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in ICLR, 2019. Yang Yang (Member, IEEE) received a Ph.D. de- gree in computer science, from Nanjing University, China in 2019. In the same year, he became a faculty member at Nanjing University of Science an...

    2019

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.