Pith. sign in

REVIEW 3 major objections 4 minor 75 references

Adversarial Representation Learning for Text-to-Image Matching

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TIMAM claims an adversarial modality discriminator plus BERT word embeddings lifts text-to-image rank-1 accuracy by 2 to 5 points over prior methods.

desk verdict Useful empirical paper with a load-bearing sign error in its adversarial objective: as written, the update rule trains generators to help the discriminator, not fool it. read the letter →

arxiv 1908.10534 v1 pith:DD6Z6EKH submitted 2019-08-28 cs.CV

Nikolaos Sarafianos , Xiang Xu , Ioannis A. Kakadiaris This is my paper · ORCID
classification cs.CV
keywords text-to-imagematchingcross-modalretrievaladversarialrepresentationlearningBERTpersonsearchimage-textembeddingmodality-invariantfeaturesrank-1
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes TIMAM, a training scheme that learns a shared image-text embedding space by combining three objectives: a per-modality identification loss, a cross-modal projection matching loss, and an adversarial discriminator that tries to tell whether an embedding came from an image or from a sentence. The authors claim this yields state-of-the-art text-to-image retrieval on four public datasets, with absolute rank-1 gains of 2 to 5 points over the previous best methods, and that replacing the usual LSTM word embeddings with fine-tuned BERT embeddings is the largest single source of the gain. If the claims hold, the practical lesson is that better text-side representations and modality-invariant training can beat elaborate attention mechanisms on the core retrieval task while keeping the architecture simple.

What carries the argument

The load-bearing object is the modality discriminator, a two-layer fully-connected network (256 units, then a scalar) trained with a GAN loss to classify whether an embedding came from the visual branch $\varphi(V_i)$ or the textual branch $\tau(T_i)$, while both encoders try to fool it. Around it sit two other objectives: a norm-softmax identification loss that clusters embeddings by identity or class, and a cross-modal projection matching loss in which each image embedding is matched against normalized text embeddings by scalar projection and the KL divergence between the induced soft matching probabilities $p_{i,j}$ and the true label distribution $q_{i,j}$ is minimized in both directions. The text encoder is a pretrained BERT language model whose word embeddings feed a bidirectional LSTM; BERT is frozen first and then fine-tuned end-to-end.

What would settle it

Reimplement TIMAM from Algorithm 1 on CUHK-PEDES using the 11,003/1,000/1,000 identity split and cosine rank-1 evaluation, and check whether rank-1 reproduces 54.51% while the same code and protocol give the cited previous method its published 49.37%; if the gap shrinks or reverses, the central state-of-the-art claim fails. A second check is to drop only the discriminator from the objective and see whether rank-1 falls by the roughly 3 points the ablation attributes to it.

Watch

Extended reading notes

Core claim

TIMAM's central claim is that a small adversarial domain discriminator placed on the final image and text embeddings enforces modality invariance and thereby improves cross-modal matching. The full objective is $L = L_I + L_M + L_D$, where $L_I$ is a norm-softmax classification loss applied separately to each modality, $L_M$ is a bidirectional KL-divergence matching loss between soft scalar-projection matchings and true label matchings over each training batch, and $L_D$ is the discriminator's GAN loss. On CUHK-PEDES the paper reports rank-1 text-to-image accuracy of 54.51%, up from 49.37% for the previous best method; on Flickr30K text-to-image rank-1 is 42.6% against 41.5% for GXN; and on CUB and Flowers the AP@50 metrics rise to 70.3% and 73.7%. The ablations credit roughly 3 rank-1 points on CUHK-PEDES to the adversarial loss and roughly 4.8 points on Flickr30K to BERT, which indicates the text encoder is the largest single contributor. The paper also reports that image-to-text retrieval on Flickr30K remains below GXN (53.1 vs 56.8), so the claimed advantage is not uniform across both retrieval directions.

Load-bearing premise

The claim of 2 to 5 point improvements over prior work assumes TIMAM was tested under exactly the same identity splits, image preprocessing, and evaluation protocol as the published numbers it is compared against, with no released code to verify the match.

Editorial extensions

If this is right

  • The adversarial discriminator alone is credited with about 3 rank-1 points on CUHK-PEDES and 1.8 points on Flickr30K, so the same trick should transfer to other cross-modal retrieval tasks where one branch's embedding distribution lags behind the other's.
  • Because switching to BERT is credited with the largest single gain, roughly 4.8 points on Flickr30K, future text-to-image pipelines should treat a pretrained language model as a standard component rather than an optional upgrade.
  • The method reaches these numbers without attention blocks, part-of-speech tagging, or phrase-region alignment, which suggests those mechanisms are not necessary for strong text-to-image matching on single-object datasets.
  • On Flickr30K, where no identity labels exist, assigning a unique ID per image for the identification loss is claimed to work, so the recipe extends to datasets that provide only image-caption pairs.
  • The paper acknowledges that image-to-text retrieval on Flickr30K still trails GXN by 3.7 rank-1 points, so the claim is specifically that text-to-image matching improves, not that the model dominates both directions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to replace the discriminator with a gradient-reversal layer or a simpler domain-adversarial classifier; if the gain is truly modality invariance, those variants should recover most of TIMAM's roughly 3 points at lower training cost.
  • The BERT gain could come from pretrained language knowledge rather than from bidirectional context specifically; fine-tuning a comparable unidirectional or independently trained word encoder on the same data would separate the two explanations.
  • The headline numbers assume strict protocol parity with the cited baselines; before treating 54.51% as the new ceiling, a reader should confirm the CUHK-PEDES identity split, caption truncation length, and how multiple captions per image are used at test time, since no code is released.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes TIMAM, a text-to-image matching framework that combines three training objectives: norm-softmax identification losses for image and text embeddings, a cross-modal projection matching loss based on KL divergence, and an adversarial modality discriminator intended to learn modality-invariant representations. The text encoder uses BERT word embeddings followed by a bidirectional LSTM and a fully-connected layer. The authors report state-of-the-art results on CUHK-PEDES, Flickr30K, CUB, and Flowers, with claimed absolute rank-1 improvements from 2% to 5%, and they provide ablations on CUHK-PEDES and Flickr30K. The paper also includes qualitative results and a discussion of rejected alternatives such as conditional augmentation and text-to-image reconstruction.

Significance. If the claims hold, the paper would make two useful contributions: demonstrating that a pre-trained language model such as BERT can improve text-to-image matching, and showing that a modality discriminator can help learn shared embeddings. The experimental setup is straightforward, the loss definitions in Section 3 are mostly clear, and the ablation in Table 4 gives some evidence that each component contributes. However, the central contribution is undermined by an inconsistency between the stated adversarial objective and the training algorithm, and the state-of-the-art claim is overstated for at least one dataset and one retrieval direction. The results are also reported as single runs without code or error bars, which limits confidence in the exact SOTA numbers.

major comments (3)
  1. [Section 3.3, Eq. (4)-(5), Algorithm 1] The adversarial objective as written cannot learn modality-invariant representations. Eq. (4) defines LD as the standard discriminator classification loss, and Eq. (5) sets the total loss as L = LI + LM + LD. Algorithm 1 then says to update the network parameters using L. If the image and text encoders are updated by minimizing L, the term LD drives D(phi(V)) toward 1 and D(tau(T)) toward 0, i.e., toward correct modality classification, not toward fooling the discriminator. No gradient reversal layer, no separate generator loss, and no alternating update for the generators is described. This contradicts the text in Section 3.3, which states that the generators learn by fooling the discriminator, and it means the improvements attributed to ARL in Table 4 are not explained by the stated algorithm. The authors must specify the actual generator objective (e.g., maximizing LD for the generators, a non-saturating confusion loss, or a gradient reversal layer) and update Eq. (5) and Algorithm 1 accordingly.
  2. [Abstract and Section 4.1, Table 2] The abstract claims state-of-the-art cross-modal matching on four datasets with absolute improvements of 2% to 5% in rank-1 accuracy. This is not supported by Table 2 on Flickr30K: the text-to-image rank-1 gain over the previous best method is about 1.1 points (42.6 vs. 41.5 for GXN), and in image-to-text matching TIMAM is worse than several prior methods (53.1 vs. 56.8 for GXN, 55.5 for SCO, 55.0 for DAN). The state-of-the-art claim should be qualified per dataset and per retrieval direction, and the range of improvements should be stated accurately.
  3. [Section 4, Tables 1-3, Supplementary Implementation Details] The SOTA comparison rests on the assumption that TIMAM was evaluated under exactly the same data splits, preprocessing, and evaluation protocols as the published numbers it is compared against. The manuscript states that protocols were strictly followed and the supplement gives splits, but no code is released and all results are single runs without error bars. Small protocol differences, such as how multiple descriptions per image are used at test time or the exact identity split on CUHK-PEDES, could change rank-1 numbers by more than the reported margins. I recommend releasing code or a detailed protocol, and reporting results over multiple runs or seeds, so that the 2-5% improvements can be verified.
minor comments (4)
  1. [Eq. (3)] The expression is written as a sum of p log(p/q), which is KL(p||q), but the text says the loss is the KL divergence from the true matching distribution q_i to the predicted probability p_i. Please align the notation with the stated direction.
  2. [Table 4 and Section 4.2] The row ordering of Table 4 and the accompanying prose are ambiguous. The text says that introducing BERT increases accuracy to 52.97%, but as printed the row with BERT appears to be 51.3 and the row with ARL appears to be 52.9. Please reorder or relabel the rows so that the table matches the prose.
  3. [Section 3.4] The sentence 'to properly balance the training between GV, GT, and D, we followed several of the tricks discussed by Chintala et al. [9]' is vague; the actual tricks appear only in the supplement. It would help the reader if the main text at least listed the three tricks (separate mini-batches, label smoothing, and label flipping) or referred to the supplement more explicitly.
  4. [Section 4.1, CUB and Flowers datasets] The text says 'We tested TIMAM against all eight methods evaluated on these datasets,' but Table 3 reports fewer rows and the method list is not numbered. Please clarify which methods were selected and why the comparison is limited to eight methods.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: TIMAM's central claim is an empirical benchmark result; no reported number is generated by construction from its inputs.

full rationale

The central claim is an empirical state-of-the-art result measured on held-out test sets of four public datasets. The loss functions are explicitly defined: norm-softmax identification losses (Eq. 1), the CMPM cross-modal projection matching loss (Eq. 3, cited to Zhang and Lu [71]), and an adversarial discriminator loss (Eq. 4). The total training loss (Eq. 5) is a sum of these terms; no equation defines the reported rank-1 accuracies in terms of the method's inputs. The CMPM loss is borrowed from prior work and is independent external evidence, and BERT is a publicly available pre-trained model; neither is re-derived from the target result. The paper's self-citations ([45]-[47], [64]) concern curriculum learning and face recognition and are not load-bearing for the cross-modal matching claim. One non-circularity concern is that Algorithm 1 updates all network parameters by minimizing L = LD + ... , which as written would train the feature generators to be correctly classified by the discriminator rather than to fool it; this is an internal correctness/reproducibility issue, not a circular reduction of the predictions to the inputs. Similarly, the lack of released code is a reproducibility risk, not a circularity. The claimed improvements are therefore not forced by construction or by self-citation, and the circularity score is 0.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

This is an empirical deep-learning benchmark paper. The central claim depends on hand-selected hyperparameters, the borrowed CMPM loss, BERT's pretrained representations, and the untested hypothesis that modality-invariant embeddings improve matching. No new physical or conceptual entities are introduced.

free parameters (7)
  • Embedding dimension = 512
    Dimensionality of both visual and textual feature vectors; all losses and the discriminator operate in this space. Chosen by hand, no sensitivity analysis.
  • Bidirectional LSTM hidden size = 512
    Size of the LSTM that summarizes BERT word embeddings; hand-selected.
  • Learning rate schedule = 2e-4, decay by 10 on plateau; BERT fine-tuning at 2e-6 for 30 epochs
    Hand-tuned on validation; standard practice but arbitrary.
  • Batch size = 64
    Affects the cross-modal projection matching loss normalization and discriminator balance; no sweep reported.
  • GAN label smoothing = visual labels in [0.8,1.2], text labels in [0,0.3], 20% label flip
    Heuristics from Chintala et al. to stabilize adversarial training.
  • Max sentence length = 50 (CUHK-PEDES), 30 (Flickr30K, CUB, Flowers)
    Follows prior preprocessing; longer sentences are truncated, which may discard information.
  • Weight decay = 4e-4
    L2 regularization strength; chosen by hand.
assumptions (5)
  • domain assumption Norm-softmax cross-entropy identification loss provides good supervision for both modalities.
    Taken from face recognition literature (NormFace, SphereFace) and applied to text and image embeddings; not justified for this task beyond ablation.
  • domain assumption Cross-modal projection matching loss (CMPM) is an effective matching objective.
    Borrowed directly from Zhang and Lu [71]; the paper reuses it and justifies it by citing their experiments, not by re-derivation.
  • domain assumption BERT word embeddings, after fine-tuning, improve text features for matching.
    Relies on the NLP literature for BERT's quality; the paper's own ablation (Table 5) supports it empirically on Flickr30K.
  • ad hoc to paper Adversarial training encourages modality-invariant features that improve matching.
    This is the central hypothesis; the paper provides no theory, only the ablation in Table 4. It could in principle hurt by discarding modality-specific cues.
  • domain assumption The evaluation protocols and data splits match those of prior work.
    Comparisons are to previously published numbers; the paper claims to follow protocols but releases no code, so this is unverifiable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adversarial Representation Learning for Text-to-Image Matching." pith.science (2026). https://pith.science/paper/DD6Z6EKH

@misc{pith2026190810534,
  author       = {Pith},
  title        = {Pith review of: Adversarial Representation Learning for Text-to-Image Matching},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DD6Z6EKH}},
  note         = {Machine review of arXiv:1908.10534}
}
read the original abstract

For many computer vision applications such as image captioning, visual question answering, and person search, learning discriminative feature representations at both image and text level is an essential yet challenging problem. Its challenges originate from the large word variance in the text domain as well as the difficulty of accurately measuring the distance between the features of the two modalities. Most prior work focuses on the latter challenge, by introducing loss functions that help the network learn better feature representations but fail to account for the complexity of the textual input. With that in mind, we introduce TIMAM: a Text-Image Modality Adversarial Matching approach that learns modality-invariant feature representations using adversarial and cross-modal matching objectives. In addition, we demonstrate that BERT, a publicly-available language model that extracts word embeddings, can successfully be applied in the text-to-image matching domain. The proposed approach achieves state-of-the-art cross-modal matching performance on four widely-used publicly-available datasets resulting in absolute improvements ranging from 2% to 5% in terms of rank-1 accuracy.

Figures

Figures reproduced from arXiv: 1908.10534 by the authors.

Figure 1
Figure 1. We learn discriminative embeddings from the vi [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. TIMAM consists of three modules: (i) the feature extraction module which extracts textual and visual features [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The three learning objectives. Top-left: We learn [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Qualitative results on all datasets we tested our method. Given a textual/visual description as a query, we retrieve [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Two failure cases of the proposed approach. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Additional qualitative text-to-image retrieval results on the CUHK-PEDES (left) and Flickr30K (right) datasets. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 68 canonical work pages

  1. [1]

    Evaluation of output embeddings for fine- grained image classification

    Zeynep Akata, Scott Reed, Daniel Walter, Honglak Lee, and Bernt Schiele. Evaluation of output embeddings for fine- grained image classification. In Proc. Conference on Com- puter Vision and Pattern Recognition, Boston, MA, June 8- 10 2015. 6

    work page 2015

  2. [2]

    VQA: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In Proc. Conference on Computer Vision and Pattern Recognition, Boston, MA, June 8-10 2015. 5, 14

    work page 2015

  3. [3]

    Curriculum learning

    Yoshua Bengio, J ´erˆome Louradour, Ronan Collobert, and Ja- son Weston. Curriculum learning. In Proc. International Conference on Machine Learning, Montreal, Canada, 2009. 2

    work page 2009

  4. [4]

    Deep visual-semantic hashing for cross-modal retrieval

    Yue Cao, Mingsheng Long, Jianmin Wang, Qiang Yang, and Philip S Yu. Deep visual-semantic hashing for cross-modal retrieval. In Proc. International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, Aug. 13-17

    work page

  5. [5]

    Partial adversarial domain adaptation

    Zhangjie Cao, Lijia Ma, Mingsheng Long, and Jianmin Wang. Partial adversarial domain adaptation. In Proc. Eu- ropean Conference on Computer Vision, Munich, Germany, Sept. 8-14 2018. 4

    work page 2018

  6. [6]

    Improving deep visual representation for person re-identification by global and local image-language association

    Dapeng Chen, Hongsheng Li, Xihui Liu, Yantao Shen, Jing Shao, Zejian Yuan, and Xiaogang Wang. Improving deep visual representation for person re-identification by global and local image-language association. In Proc. European Conference on Computer Vision, Munich, Germany, Sept. 8- 14 2018. 1, 2, 3, 5, 6, 7, 12, 14

    work page 2018

  7. [7]

    Improving text-based person search by spatial matching and adaptive threshold

    Tianlang Chen, Chenliang Xu, and Jiebo Luo. Improving text-based person search by spatial matching and adaptive threshold. In Proc. Winter Conference on Applications of Computer Vision, Lake Tahoe, NV , Mar. 12-15 2018. 5, 14

    work page 2018

  8. [8]

    Beyond triplet loss: A deep quadruplet network for person re-identification

    Weihua Chen, Xiaotang Chen, Jianguo Zhang, and Kaiqi Huang. Beyond triplet loss: A deep quadruplet network for person re-identification. In Proc. Conference on Computer Vision and Pattern Recognition , Honolulu, HI, July 21-26

    work page

Show all 75 references
  1. [9]

    How to train a GAN? Tips and tricks to make GANs work

    Soumith Chintala, Emily Denton, Martin Arjovsky, and Michael Mathieu. How to train a GAN? Tips and tricks to make GANs work. github.com/soumith/ganhacks, 2016. 5, 14

    2016

  2. [10]

    BERT: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 2

    2018 arXiv

  3. [11]

    Class rectifi- cation hard mining for imbalanced deep learning

    Qi Dong, Shaogang Gong, and Xiatian Zhu. Class rectifi- cation hard mining for imbalanced deep learning. In Proc. International Conference on Computer Vision, Venice, Italy, Oct. 22-29 2017. 7

    2017

  4. [12]

    VSE++: Improved visual-semantic embeddings

    Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. VSE++: Improved visual-semantic embeddings. In Proc. British Machine Vision Conference , Newcastle, UK, Sep. 3-6 2018. 6, 15

    2018

  5. [13]

    Unsupervised domain adaptation by backpropagation

    Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Proc. International Con- ference on Machine Learning, Lille, France, July 6-11 2015. 4

    2015

  6. [14]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. InProc. Neural Information Processing Systems, Montral Canada, Dec. 8-13

  7. [15]

    Look, imagine and match: Improving textual-visual cross-modal retrieval with generative models

    Jiuxiang Gu, Jianfei Cai, Shafiq Joty, Li Niu, and Gang Wang. Look, imagine and match: Improving textual-visual cross-modal retrieval with generative models. In Proc. Con- ference on Computer Vision and Pattern Recognition , Salt Lake City, UT, June 18-22 2018. 2, 6, 14, 15

    2018

  8. [16]

    Unsupervised cross-modal retrieval through adversarial learning

    Li He, Xing Xu, Huimin Lu, Yang Yang, Fumin Shen, and Heng Tao Shen. Unsupervised cross-modal retrieval through adversarial learning. In Proc. International Conference on Multimedia and Expo, Hong Kong, July 10-14 2017. 2

    2017

  9. [17]

    Long short-term memory

    Sepp Hochreiter and J ¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. 3

    1997

  10. [18]

    Learning deep representation for imbalanced classifi- cation

    Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalanced classifi- cation. In Proc. Conference on Computer Vision and Pattern Recognition, Las Vegas, NV , June 26 - July 1 2016. 7

    2016

  11. [19]

    Bi-directional spatial-semantic attention networks for image-text matching

    Feiran Huang, Xiaoming Zhang, Zhonghua Zhao, and Zhou- jun Li. Bi-directional spatial-semantic attention networks for image-text matching. Transactions on Image Processing, 28(4):2008–2020, 2019. 2

    2008

  12. [20]

    Instance-aware im- age and sentence matching with selective multimodal LSTM

    Yan Huang, Wei Wang, and Liang Wang. Instance-aware im- age and sentence matching with selective multimodal LSTM. In Proc. Conference on Computer Vision and Pattern Recog- nition, Honolulu, HI, July 21-26 2017. 15

    2017

  13. [21]

    Learning semantic concepts and order for image and sen- tence matching

    Yan Huang, Qi Wu, Chunfeng Song, and Liang Wang. Learning semantic concepts and order for image and sen- tence matching. In Proc. Conference on Computer Vision and Pattern Recognition , Salt Lake City, UT, June 18-22

  14. [22]

    Cascade attention network for per- son search: Both image and text-image similarity selection

    Ya Jing, Chenyang Si, Junbo Wang, Wei Wang, Liang Wang, and Tieniu Tan. Cascade attention network for per- son search: Both image and text-image similarity selection. arXiv preprint arXiv:1809.08440, 2018. 2, 3, 5, 14

    2018 arXiv

  15. [23]

    Deep visual-semantic align- ments for generating image descriptions

    Andrej Karpathy and Li Fei-Fei. Deep visual-semantic align- ments for generating image descriptions. In Proc. Confer- ence on Computer Vision and Pattern Recognition , Boston, MA, June 8-10 2015. 12, 15

    2015

  16. [24]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 ,

    arXiv

  17. [25]

    Asso- ciating neural word embeddings with deep image represen- tations using fisher vectors

    Benjamin Klein, Guy Lev, Gil Sadeh, and Lior Wolf. Asso- ciating neural word embeddings with deep image represen- tations using fisher vectors. In Proc. Conference on Com- puter Vision and Pattern Recognition, Boston, MA, June 8- 10 2015. 6, 14, 15

    2015

  18. [26]

    Self-supervised adversarial hashing networks for cross-modal retrieval

    Chao Li, Cheng Deng, Ning Li, Wei Liu, Xinbo Gao, and Dacheng Tao. Self-supervised adversarial hashing networks for cross-modal retrieval. In Proc. Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, June 18- 22 2018. 2

    2018

  19. [27]

    Identity-aware textual-visual matching with la- tent co-attention

    Shuang Li, Tong Xiao, Hongsheng Li, Wei Yang, and Xiao- gang Wang. Identity-aware textual-visual matching with la- tent co-attention. In Proc. International Conference on Com- puter Vision, Venice, Italy, Oct. 22-29 2017. 1, 2, 3, 5, 6, 7, 14

    2017

  20. [28]

    Person search with natural lan- guage description

    Shuang Li, Tong Xiao, Hongsheng Li, Bolei Zhou, Dayu Yue, and Xiaogang Wang. Person search with natural lan- guage description. In Proc. Conference on Computer Vision and Pattern Recognition, Honolulu, HI, July 21-26 2017. 1, 2, 5, 12, 13, 14

    2017

  21. [29]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proc. European Conference on Computer Vision , Zurich, Switzerland, Sept. 6-12 2014. 2

    2014

  22. [30]

    Leveraging visual question an- swering for image-caption ranking

    Xiao Lin and Devi Parikh. Leveraging visual question an- swering for image-caption ranking. In Proc. European Con- ference on Computer Vision, Amsterdam, The Netherlands, Oct. 8-16 2016. 15

    2016

  23. [31]

    A neighbor-aware approach for image-text matching

    Chunxiao Liu, Zhendong Mao, Wenyu Zang, and Bin Wang. A neighbor-aware approach for image-text matching. In Proc. International Conference on Acoustics, Speech and Signal Processing, Brighton, United Kingdom, May 12-17

  24. [32]

    Sphereface: Deep hypersphere embed- ding for face recognition

    Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embed- ding for face recognition. In Proc. Conference on Computer Vision and Pattern Recognition , Honolulu, HI, July 21-26

  25. [33]

    Learning a recurrent residual fusion network for multimodal matching

    Yu Liu, Yanming Guo, Erwin M Bakker, and Michael S Lew. Learning a recurrent residual fusion network for multimodal matching. In Proc. International Conference on Computer Vision, Venice, Italy, Oct. 22-29 2017. 6, 7, 15

    2017

  26. [34]

    Mul- timodal convolutional neural networks for matching image and sentence

    Lin Ma, Zhengdong Lu, Lifeng Shang, and Hang Li. Mul- timodal convolutional neural networks for matching image and sentence. In Proc. International Conference on Com- puter Vision, Santiago, Chile, Dec. 13-16 2015. 15

    2015

  27. [35]

    Deep captioning with multimodal recurrent neural networks (m-RNN)

    Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, and Alan Yuille. Deep captioning with multimodal recurrent neural networks (m-RNN). In Proc. International Confer- ence on Learning Representations, San Diego, CA, May 7-9

  28. [36]

    Distributed representations of words and phrases and their compositionality

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Proc. Advances in Neural Information Processing Systems , Lake Tahoe, NV , Dec. 5-10 2013. 6

    2013

  29. [37]

    Learnable PINs: Cross-modal embeddings for person iden- tity

    Arsha Nagrani, Samuel Albanie, and Andrew Zisserman. Learnable PINs: Cross-modal embeddings for person iden- tity. In Proc. European Conference on Computer Vision, Mu- nich, Germany, Sept. 8-14 2018. 2

    2018

  30. [38]

    Dual attention networks for multimodal reasoning and matching

    Hyeonseob Nam, Jung-Woo Ha, and Jeonghee Kim. Dual attention networks for multimodal reasoning and matching. In Proc. Conference on Computer Vision and Pattern Recog- nition, Honolulu, HI, July 21-26 2017. 2, 6, 12, 15

    2017

  31. [39]

    Cm-gans: cross-modal genera- tive adversarial networks for common representation learn- ing

    Yuxin Peng and Jinwei Qi. Cm-gans: cross-modal genera- tive adversarial networks for common representation learn- ing. Transactions on Multimedia Computing, Communica- tions, and Applications, 15(1):22, 2019. 2

    2019

  32. [40]

    Deep contextualized word representations

    Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proc. North Amer- ican Chapter of the Association for Computational Linguis- tics, New Orleans, LA, June 1-6 2018. 2

    2018

  33. [41]

    Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazeb- nik. Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models. InProc. In- ternational Conference on Computer Vision, Santiago, C...

    2015

  34. [42]

    MirrorGAN: Learning text-to-image generation by re- description

    Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. MirrorGAN: Learning text-to-image generation by re- description. In Proc. Conference on Computer Vision and Pattern Recognition, Long Beach, CA, June 15-21 2019. 8

    2019

  35. [43]

    Improving language understanding with unsuper- vised learning

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding with unsuper- vised learning. Technical report, OpenAI, 2018. 2

    2018

  36. [44]

    Learning deep representations of fine-grained visual descrip- tions

    Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele. Learning deep representations of fine-grained visual descrip- tions. In Proc. Conference on Computer Vision and Pattern Recognition, Las Vegas, NV , June 26 - July 1 2016. 2, 5, 6, 12, 13, 14

    2016

  37. [45]

    Cur- riculum learning for multi-task classification of visual attributes

    Nikolaos Sarafianos, Theodore Giannakopoulos, Christophoros Nikou, and Ioannis A Kakadiaris. Cur- riculum learning for multi-task classification of visual attributes. In Proc. International Conference on Computer Vision Workshops, Venice, Italy, Oct. 22-29 2017. 2

    2017

  38. [46]

    Cur- riculum learning of visual attribute clusters for multi-task classification

    Nikolaos Sarafianos, Theodoros Giannakopoulos, Christophoros Nikou, and Ioannis A Kakadiaris. Cur- riculum learning of visual attribute clusters for multi-task classification. Pattern Recognition, 2018. 2

    2018

  39. [47]

    Deep imbalanced attribute classification using visual atten- tion aggregation

    Nikolaos Sarafianos, Xiang Xu, and Ioannis A Kakadiaris. Deep imbalanced attribute classification using visual atten- tion aggregation. In Proc. European Conference on Com- puter Vision, Munich, Germany, Sept. 8-14 2018. 3

    2018

  40. [48]

    Facenet: A unified embedding for face recognition and clus- tering

    Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clus- tering. In Proc. Conference on Computer Vision and Pattern Recognition, Boston, MA, June 8-10 2015. 7

    2015

  41. [49]

    Video google: A text retrieval approach to object matching in videos

    Josef Sivic and Andrew Zisserman. Video google: A text retrieval approach to object matching in videos. In Proc. International Conference on Computer Vision, Nice, France, Oct. 13-16 2003. 2

    2003

  42. [50]

    Improved deep metric learning with multi- class n-pair loss objective

    Kihyuk Sohn. Improved deep metric learning with multi- class n-pair loss objective. In Proc. Advances in Neural In- formation Processing Systems, Barcelona, Spain, Dec. 5-10

  43. [51]

    Polysemous visual- semantic embedding for cross-modal retrieval

    Yale Song and Mohammad Soleymani. Polysemous visual- semantic embedding for cross-modal retrieval. InProc. Con- ference on Computer Vision and Pattern Recognition, Long Beach, CA, June 16-20 2019. 2

    2019

  44. [52]

    In defense of the triplet loss for visual recognition

    Ahmed Taha, Yi-Ting Chen, Teruhisa Misu, and Larry Davis. In defense of the triplet loss for visual recognition. arXiv preprint arXiv:1901.08616, 2019. 7

    1901 arXiv

  45. [53]

    Simultaneous deep transfer across domains and tasks

    Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko. Simultaneous deep transfer across domains and tasks. In Proc. International Conference on Computer Vision , Santi- ago, Chile, Dec. 13-16 2015. 4

    2015

  46. [54]

    Learning deep embeddings with histogram loss

    Evgeniya Ustinova and Victor Lempitsky. Learning deep embeddings with histogram loss. In Proc. Advances in Neu- ral Information Processing Systems, Barcelona, Spain, Dec. 5-10 2016. 7

    2016

  47. [55]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. Neural In- formation Processing Systems , Long Beach, CA, Dec. 4-9

  48. [56]

    Show and tell: A neural image caption gen- erator

    Oriol Vinyals, Alexander Toshev, Samy Bengio, and Du- mitru Erhan. Show and tell: A neural image caption gen- erator. In Proc. Conference on Computer Vision and Pattern Recognition, Boston, MA, June 8-10 2015. 14

    2015

  49. [57]

    Adversarial cross-modal retrieval

    Bokun Wang, Yang Yang, Xing Xu, Alan Hanjalic, and Heng Tao Shen. Adversarial cross-modal retrieval. In Proc. ACM on Multimedia Conference, Mountain View, CA, Oct. 23-27 2017. 2

    2017

  50. [58]

    Normface: l 2 hypersphere embedding for face verifi- cation

    Feng Wang, Xiang Xiang, Jian Cheng, and Alan Loddon Yuille. Normface: l 2 hypersphere embedding for face verifi- cation. In Proc. ACM on Multimedia Conference, Mountain View, CA, Oct. 23-27 2017. 4

    2017

  51. [59]

    Learning deep structure-preserving image-text embeddings

    Liwei Wang, Yin Li, and Svetlana Lazebnik. Learning deep structure-preserving image-text embeddings. In Proc. Con- ference on Computer Vision and Pattern Recognition , Las Vegas, NV , June 26 - July 1 2016. 2, 15

    2016

  52. [60]

    Joint global and co-attentive represen- tation learning for image-sentence retrieval

    Shuhui Wang, Yangyu Chen, Junbao Zhuo, Qingming Huang, and Qi Tian. Joint global and co-attentive represen- tation learning for image-sentence retrieval. In Proc. ACM on Multimedia Conference, Seoul, South Korea, Oct. 22-26

  53. [61]

    Multilevel language and vision integration for text-to-clip re- trieval

    Huijuan Xu, Kun He, L Sigal, S Sclaroff, and K Saenko. Multilevel language and vision integration for text-to-clip re- trieval. In AAAI Conference on Artificial Intelligence, Hon- olulu, HI, Jan. 27 - Feb 1 2019. 2

    2019

  54. [62]

    Attngan: Fine- grained text to image generation with attentional generative adversarial networks

    Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine- grained text to image generation with attentional generative adversarial networks. In Proc. Conference on Computer Vi- sion and Pattern Recognition, Salt Lake City, UT, J...

  55. [63]

    Deep adversarial metric learning for cross-modal retrieval

    Xing Xu, Li He, Huimin Lu, Lianli Gao, and Yanli Ji. Deep adversarial metric learning for cross-modal retrieval. World Wide Web, 22(2):657–672, 2019. 2

    2019

  56. [64]

    Evaluation of a 3D-aided pose invariant 2D face recognition system

    Xiang Xu, Ha A Le, Pengfei Dou, Yuhang Wu, and Ioannis A Kakadiaris. Evaluation of a 3D-aided pose invariant 2D face recognition system. In Proc. International Joint Conference on Biometrics, Denver, CO, Oct. 1-4 2017. 4

    2017

  57. [65]

    d-SNE: Domain adap- tation using stochastic neighborhood embedding

    Xiang Xu, Xiong Zhou, Ragav Venkatesan, Gurumurthy Swaminathan, and Orchid Majumder. d-SNE: Domain adap- tation using stochastic neighborhood embedding. In Proc. Conference on Computer Vision and Pattern Recognition , Long Beach, CA, June 16-20 2019. 2

    2019

  58. [66]

    Deep correlation for matching images and text

    Fei Yan and Krystian Mikolajczyk. Deep correlation for matching images and text. In Proc. Conference on Com- puter Vision and Pattern Recognition, Boston, MA, June 8- 10 2015. 2, 15

    2015

  59. [67]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hocken- maier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions. Transactions of the Association for Computational Linguistics, 2:67–78, 2014. 2

    2014

  60. [68]

    Cross-modal and hierarchical modeling of video and text

    Bowen Zhang, Hexiang Hu, and Fei Sha. Cross-modal and hierarchical modeling of video and text. In Proc. European Conference on Computer Vision, Munich, Germany, Sept. 8- 14 2018. 2

    2018

  61. [69]

    Man: Moment alignment network for natu- ral language moment retrieval via iterative graph adjustment

    Da Zhang, Xiyang Dai, Xin Wang, Yuan-Fang Wang, and Larry S Davis. Man: Moment alignment network for natu- ral language moment retrieval via iterative graph adjustment. arXiv preprint arXiv:1812.00087, 2018. 2

    2018 arXiv

  62. [70]

    Stack- gan: Text to photo-realistic image synthesis with stacked generative adversarial networks

    Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiao- gang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stack- gan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proc. Conference on Computer Vision and Pattern Recognition , Honolulu,...

    2017

  63. [71]

    Deep cross-modal projection learning for image-text matching

    Ying Zhang and Huchuan Lu. Deep cross-modal projection learning for image-text matching. In Proc. European Con- ference on Computer Vision , Munich, Germany, Sept. 8-14

  64. [72]

    Dual-path convolutional image- text embedding with instance loss

    Zhedong Zheng, Liang Zheng, Michael Garrett, Yi Yang, and Yi-Dong Shen. Dual-path convolutional image- text embedding with instance loss. arXiv preprint arXiv:1711.05535, 2017. 3, 5, 14

    2017 arXiv

  65. [73]

    Simple baseline for visual question answering

    Bolei Zhou, Yuandong Tian, Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Simple baseline for visual question answering. arXiv preprint arXiv:1512.02167, 2015. 14

    2015 arXiv

  66. [74]

    R2GAN: Cross-modal recipe retrieval with generative adver- sarial network

    Bin Zhu, Chong-Wah Ngo, Jingjing Chen, and Yanbin Hao. R2GAN: Cross-modal recipe retrieval with generative adver- sarial network. In Proc. Conference on Computer Vision and Pattern Recognition, Long Beach, CA, June 16-20 2019. 2 Supplementary Material Discussion on Novelty Wha...

    2019

  67. [2018]

    4, 5, 6, 7, 12, 13, 14, 15

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.