REVIEW 4 major objections 4 minor 40 references
Large-scale Tag-based Font Retrieval with Generative Feature Learning
T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A staged recognition-retrieval model that learns font features by reconstructing synthetic glyphs in different characters achieves 28.08 mAP on single-tag queries, nearly doubling the best prior retrieval baseline.
desk verdict A solid, useful font-retrieval paper with a genuinely new dataset and a clever use of synthetic fonts, but the headline claim of state-of-the-art superiority is under-supported because the main evaluation uses noisy web tags and the human evaluation never compares against the baselines. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery has three parts. First, a generative feature learning module: because font glyphs can be rendered on demand, the network is given a standard-font glyph and must generate, from the hidden feature of an input glyph, a new glyph of the same font in a different character; a PatchGAN discriminator plus L1 reconstruction loss forces the feature to encode font design while discarding character shape. Second, an attentive feature selection module: a font classification head predicts a softmax distribution over font identities, and a fully-connected layer with sigmoid turns that distribution into a node-level attention map that re-weights the hidden feature before tag prediction, with training-time aggregation over several glyphs of the same font. Third, a retrieval model: a two-layer network maps the element-wise product of predicted tag probabilities and a binary query vector, through a power activation, to an affinity score, trained with a pairwise soft-ranking loss on multi-tag triplets.
What would settle it
A reader could take the released dataset, re-label a held-out subset of fonts with professional typographers using the same tag vocabulary, and rerun the MyFonts-test protocol on only those fonts; if the full model's margin over the best retrieval baseline collapses on the clean labels, the reported improvement was an artifact of community-label noise rather than of the generative feature and attention machinery.
Extended reading notes
Core claim
The paper's central claim is that tag-based font retrieval should be staged as recognition followed by ranking, and that the visual features for that pipeline should be made character-invariant by exploiting the synthetic nature of font images. On the MyFonts-test benchmark the full model achieves 28.08 mAP on single-tag queries and 16.74 mAP on multi-tag queries, against 15.33 and 7.52 for the best adapted retrieval baseline; on the AMT human-ranked set it selects the agreed ground-truth font in 47.50% of groups, up from 44.49% for the basic recognition model. The ablation shows each added component—generative feature learning, attentive feature selection, and the retrieval head—improves results monotonically. The paper takes this as evidence that the recognition-first design, the cross-character reconstruction objective, and the class-prediction attention map together capture font semantics that plain joint-embedding retrieval misses.
Load-bearing premise
The load-bearing premise is that the MyFonts community tag lists, after normalization, are accurate enough as ground truth that training and test metrics reflect real font semantics; if those labels are systematically noisy or skewed, the reported gains may not transfer to what users actually mean by a tag.
Editorial extensions
If this is right
- A recognition model trained only with multi-label cross-entropy already beats joint-embedding retrieval baselines on this task, so font-tag semantics may be better captured by per-tag classifiers than by a shared image-text embedding space.
- The generative cross-character reconstruction visibly removes character traces from features: retrieval improves with the GAN module even though the basic recognition model is already strong.
- The attention-from-font-class design lets the model emphasize different glyph features for different fonts, matching the observation that one visual dimension can tag some fonts and be irrelevant for others.
- The retrieval head addresses the tag-dominance problem for multi-tag queries, letting rare tags such as 'noisy' participate in ranking alongside frequent tags such as 'script'.
- Human-ranked accuracy on the AMT set rises with the full model, indicating that the learned ranking orders fonts more consistently with human preferences.
Reading between the lines
- The same render-and-reconstruct-across-a-nuisance-factor trick should transfer to other domains where images are synthetic or renderable, such as icons, CAD parts, signboards, or maps, where an algorithm can deliberately vary one factor to make features invariant to it.
- Because the MyFonts labels are community-contributed, the model likely learns tag patterns that include foundries' marketing language; a cleaner expert-annotated tag set would test whether the semantic ranking generalizes beyond the benchmark's label distribution.
- The attention-from-predicted-class idea could be applied to other fine-grained recognition tasks where the class distribution is informative about which visual attributes matter, such as bird or car attribute retrieval.
- A zero-shot extension is a natural next step: with tag embeddings or a font-tag ontology, the recognition head could produce probabilities for tags never seen in training, which the retrieval head could then rank.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper addresses large-scale tag-based font retrieval. It introduces a dataset of 18,815 fonts from MyFonts with 1,824 tags and a human-ranked AMT evaluation set. The proposed method trains a multi-label tag recognition model, enhances the visual features with a generative module that reconstructs glyph images of different characters, applies implicit attention based on font class predictions, and adds a relation-network-style retrieval model for multi-tag queries. Experiments on MyFonts-test report large mAP/nDCG gains over GNA-RNN, DeViSE, and RelationNet, and an ablation shows additive improvements. On the AMT-test the full model outperforms its own basic recognition version.
Significance. If the results hold, the paper makes a strong contribution: it is, to my knowledge, the first large-scale benchmark dataset for tag-based font retrieval, and the generative feature learning idea leverages the synthetic nature of font images in a principled way. The proposed method shows large, consistent gains over adapted baselines on the MyFonts-test set, and the ablation isolates the contribution of each module. The dataset release and the AMT evaluation protocol are valuable resources. However, the claim of 'significantly outperforms the state-of-the-art' rests on MyFonts-test, whose labels are acknowledged to be noisy and incomplete, while the only human-judgment comparison is against the paper's own basic model. These gaps reduce the strength of the headline claim.
major comments (4)
- [Section 3, Section 5.3, Table 1] The MyFonts-test ground truth is derived from the same web-sourced MyFonts tags that define the training labels in Eq. (1). The paper itself states (Section 3) that these tags are 'inevitably noisy and incomplete' and that 'some tags may be omitted by web users for a given font' (Section 5.3). Under the multi-label cross-entropy loss of Eq. (1), an omitted tag is treated as a negative even when the font visually matches, and the same incomplete labels are used as queries and relevance judgments on MyFonts-test. Consequently, the reported mAP advantage (28.08 vs 15.33 against RelationNet) may reflect, to an unknown degree, recovery of MyFonts foundry tagging conventions rather than true semantic font-tag relevance. The AMT-test set is the designed guard against this, but Table 2 compares only the full model with its own basic recognition version; none of the baselines (RelationNet, GNA-RNN, DeViSE) is evaluated on AMT-test. To support the abstract's claim of significant improvement over the state of the art, the authors should report AMT-test accuracy and average rank for the baselines, or otherwise justify why MyFonts-test is a sufficient criterion.
- [Table 1, Section 5.2, Eq. (7)] The paper reports no error bars, confidence intervals, or significance tests for any metric. The margins over the baselines are large, but the ablation increments are small (e.g., from 27.75 to 28.08 mAP when adding the generative module on top of attention, and from 16.06 to 16.74 mAP on multi-tag queries when adding the retrieval model). Without a variance estimate or a significance test, the claim that each component 'gets steady improvements' is not statistically supported. I request at least three runs with standard deviations, or bootstrap intervals, for the main configurations.
- [Section 4.4] The sentence 'It shows better performance in transforming probabilities than other common activation functions' is not supported by any experiment in the paper. Please provide an ablation comparing the power activation x->(x+epsilon)^alpha with alternatives (e.g., identity, sigmoid, ReLU) or remove this claim.
- [Section 3, Section 5.4] The AMT evaluation set is constructed by keeping only groups in which all three workers agree (1,661 groups out of the 9,000 generated). This unanimity filter likely selects easier groups, and the paper reports neither the number of discarded groups nor the distribution of agreements. Because the full model and the basic model are compared on the same retained groups, the internal comparison is fair, but this selection makes the absolute accuracy values hard to interpret, and any future baseline comparison on AMT-test should use the same retained groups. Please report the worker-agreement statistics and consider a supplementary evaluation on all groups with a majority-vote label.
minor comments (4)
- [Figure 5] Figure 5 includes a 'Shared Font Class FC Layer' that is not described in the text; clarify its role or remove it from the schematic.
- [Section 5.2] The hyper-parameter settings (lambda=10, beta=0.04, J=4, alpha=0.1, gamma=100) are given only for the final model; a sensitivity analysis of alpha and gamma would clarify the robustness of the retrieval model.
- [Supplementary Section 8.3] The attention module weights are initialized with a normal distribution of mu=0, sigma=5; this is an unusually large variance and may be a typo (possibly 0.5). Please verify.
- [Section 5.3] The paper states that the training data may be incomplete but provides no quantification of tag coverage or noise rates; an analysis of tag frequency and per-font tag counts would help readers assess the label quality.
Circularity Check
No significant circularity: the main results are evaluated on held-out test fonts and on an externally collected AMT human ranking set, and the core training objectives do not reduce to the measured quantities.
full rationale
The paper's central claims are supported by held-out evaluation, not by construction. The tag recognition model (Section 4.1, Eq. 1) is trained with cross-entropy on MyFonts tags, but MyFonts-test uses 1,877 held-out fonts and the AMT-test uses 1,661 human-ranked groups collected from Mechanical Turk, so the reported mAP and accuracy are not fitted to the test labels. The generative feature learning module (Section 4.2, Eqs. 2-5) uses a GAN and L1 reconstruction objective on raw glyph images, which is self-supervised with respect to tags and does not encode the tag labels into the feature. The attentive feature selection module (Section 4.3) is trained using font identity classification, not the tag labels being predicted, and the final retrieval model (Section 4.4, Eq. 7) is trained with a pairwise soft ranking loss over query-image triples and then evaluated against the baselines GNA-RNN, DeViSE, and RelationNet on MyFonts-test. No load-bearing step reduces to a fitted parameter renamed as a prediction, and no uniqueness claim or ansatz is imported from a self-citation. The acknowledged noise and incompleteness of MyFonts web tags is a criterion-validity concern about what is measured, not a circularity in the derivation chain. Author self-citations to DeepFont and related Adobe work are contextual comparisons rather than load-bearing support for the central claim.
Assumptions & free parameters
free parameters (5)
- lambda (L1 loss weight) =
10
- beta (GAN loss weight) =
0.04
- alpha (power activation exponent) =
0.1
- gamma (ranking loss scale) =
100
- J (number of glyphs for attention aggregation) =
4
assumptions (6)
- domain assumption MyFonts web tags, after preprocessing, provide valid ground truth for font-tag relations.
- domain assumption Font images are synthetic and controllable by the learning algorithm, enabling the generative reconstruction objective.
- ad hoc to paper The L1 and GAN reconstruction losses force the hidden feature to be font-specific and character-agnostic.
- ad hoc to paper The font class probability distribution is a useful conditioning signal for attention.
- domain assumption Rendering each font family using its regular version with random character strings represents the font's visual identity.
- domain assumption AMT majority agreement (3/3 workers) defines correct ranking for the human evaluation set.
invented entities (2)
-
Generative feature learning module
independent evidence
-
Attentive feature selection module
independent evidence
Cite this review
Pith. "Pith review of Large-scale Tag-based Font Retrieval with Generative Feature Learning." pith.science (2026). https://pith.science/paper/P2FGERC2
@misc{pith2026190902072,
author = {Pith},
title = {Pith review of: Large-scale Tag-based Font Retrieval with Generative Feature Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/P2FGERC2}},
note = {Machine review of arXiv:1909.02072}
}
read the original abstract
Font selection is one of the most important steps in a design workflow. Traditional methods rely on ordered lists which require significant domain knowledge and are often difficult to use even for trained professionals. In this paper, we address the problem of large-scale tag-based font retrieval which aims to bring semantics to the font selection process and enable people without expert knowledge to use fonts effectively. We collect a large-scale font tagging dataset of high-quality professional fonts. The dataset contains nearly 20,000 fonts, 2,000 tags, and hundreds of thousands of font-tag relations. We propose a novel generative feature learning algorithm that leverages the unique characteristics of fonts. The key idea is that font images are synthetic and can therefore be controlled by the learning algorithm. We design an integrated rendering and learning process so that the visual feature from one image can be used to reconstruct another image with different text. The resulting feature captures important font design details while is robust to nuisance factors such as text. We propose a novel attention mechanism to re-weight the visual feature for joint visual-text modeling. We combine the feature and the attention mechanism in a novel recognition-retrieval model. Experimental results show that our method significantly outperforms the state-of-the-art for the important problem of large-scale tag-based font retrieval.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[1]
Bottom-up and top-down attention for image captioning and vqa
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and vqa. arXiv preprint arXiv:1707.07998, 2017
arXiv 2017
-
[2]
Multi-content gan for few-shot font style transfer
Samaneh Azadi, Matthew Fisher, Vladimir Kim, Zhaowen Wang, Eli Shechtman, and Trevor Darrell. Multi-content gan for few-shot font style transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, volume 11, page 13, 2018
work page 2018
-
[3]
Large-scale visual font recognition
Guang Chen, Jianchao Yang, Hailin Jin, Jonathan Brandt, Eli Shechtman, Aseem Agarwala, and Tony X Han. Large-scale visual font recognition. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pages 3598–3605, 2014
work page 2014
-
[4]
Greedy function approximation: a gra- dient boosting machine
Jerome H Friedman. Greedy function approximation: a gra- dient boosting machine. Annals of statistics , pages 1189– 1232, 2001
work page 2001
-
[5]
Devise: A deep visual- semantic embedding model
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Tomas Mikolov, et al. Devise: A deep visual- semantic embedding model. In Advances in neural informa- tion processing systems, pages 2121–2129, 2013
work page 2013
-
[6]
Generative adversarial nets
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672–2680, 2014
2014
-
[7]
Look, imagine and match: Improving textual-visual cross-modal retrieval with generative models
Jiuxiang Gu, Jianfei Cai, Shafiq Joty, Li Niu, and Gang Wang. Look, imagine and match: Improving textual-visual cross-modal retrieval with generative models. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7181–7189, 2018
work page 2018
-
[8]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016
2016
Show all 40 references
-
[9]
Image-to-image translation with conditional adver- sarial networks
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adver- sarial networks. arXiv preprint, 2017
2017
-
[10]
Multimodal residual learning for visual qa
Jin-Hwa Kim, Sang-Woo Lee, Donghyun Kwak, Min-Oh Heo, Jeonghee Kim, Jung-Woo Ha, and Byoung-Tak Zhang. Multimodal residual learning for visual qa. In Advances in Neural Information Processing Systems , pages 361–369, 2016
2016
-
[11]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural net- works. In Advances in neural information processing sys- tems, 2012
2012
-
[12]
Learning to detect unseen object classes by between- class attribute transfer
Christoph H Lampert, Hannes Nickisch, and Stefan Harmel- ing. Learning to detect unseen object classes by between- class attribute transfer. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 951–958. IEEE, 2009
2009
-
[13]
Stacked cross attention for image-text matching
Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xi- aodong He. Stacked cross attention for image-text matching. arXiv preprint arXiv:1803.08024, 2018
2018 arXiv
-
[14]
Identity-aware textual-visual matching with latent co-attention
Shuang Li, Tong Xiao, Hongsheng Li, Wei Yang, and Xi- aogang Wang. Identity-aware textual-visual matching with latent co-attention. In Computer Vision (ICCV), 2017 IEEE International Conference on, pages 1908–1917. IEEE, 2017
2017
-
[15]
Person search with natural lan- guage description
Shuang Li, Tong Xiao, Hongsheng Li, Bolei Zhou, Dayu Yue, and Xiaogang Wang. Person search with natural lan- guage description. arXiv preprint arXiv:1702.05729, 2017
2017 arXiv
-
[16]
Multi-task adversarial network for disentangled feature learning
Yang Liu, Zhaowen Wang, Hailin Jin, and Ian Was- sell. Multi-task adversarial network for disentangled feature learning. In CVPR, 2018
2018
-
[17]
Syn- thetically supervised feature learning for scene text recog- nition
Yang Liu, Zhaowen Wang, Hailin Jin, and Ian Wassell. Syn- thetically supervised feature learning for scene text recog- nition. In The European Conference on Computer Vision (ECCV), September 2018
2018
-
[18]
Knowing when to look: Adaptive attention via a visual sen- tinel for image captioning
Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. Knowing when to look: Adaptive attention via a visual sen- tinel for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 6, page 2, 2017
2017
-
[19]
Da-gan: Instance-level image translation by deep attention generative adversarial networks
Shuang Ma, Jianlong Fu, Chang Wen Chen, and Tao Mei. Da-gan: Instance-level image translation by deep attention generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5657–5666, 2018
2018
-
[20]
Conditional generative adversarial nets
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014
2014 arXiv
-
[21]
Exploratory font selection using crowd- sourced attributes
Peter O’Donovan, J ¯anis L ¯ıbeks, Aseem Agarwala, and Aaron Hertzmann. Exploratory font selection using crowd- sourced attributes. ACM Transactions on Graphics (TOG) , 33(4):92, 2014
2014
-
[22]
Learning deep representations of fine-grained visual descrip- tions
Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele. Learning deep representations of fine-grained visual descrip- tions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 49–58, 2016
2016
-
[23]
Zero-shot learning through cross-modal transfer
Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. In Advances in neural information processing sys- tems, pages 935–943, 2013
2013
-
[24]
Disentangled rep- resentation learning gan for pose-invariant face recognition
Luan Tran, Xi Yin, and Xiaoming Liu. Disentangled rep- resentation learning gan for pose-invariant face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1415–1424, 2017
2017
-
[25]
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Re- port CNS-TR-2011-001, California Institute of Technology, 2011
2011
-
[26]
Adversarial cross-modal retrieval
Bokun Wang, Yang Yang, Xing Xu, Alan Hanjalic, and Heng Tao Shen. Adversarial cross-modal retrieval. In Pro- ceedings of the 2017 ACM on Multimedia Conference, pages 154–162. ACM, 2017
2017
-
[27]
Deepfont: Identify your font from an image
Zhangyang Wang, Jianchao Yang, Hailin Jin, Eli Shechtman, Aseem Agarwala, Jonathan Brandt, and Thomas S Huang. Deepfont: Identify your font from an image. In Proceedings of the 23rd ACM international conference on Multimedia , pages 451–459. ACM, 2015
2015
-
[28]
Zero- shot learning-the good, the bad and the ugly
Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero- shot learning-the good, the bad and the ugly. arXiv preprint arXiv:1703.04394, 2017
2017 arXiv
-
[29]
Learning to compare: Re- lation network for few-shot learning
Flood Sung Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Re- lation network for few-shot learning. 2018
2018
-
[30]
Stacked attention networks for image question answering
Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pages 21–29, 2016
2016
-
[31]
Image captioning with semantic attention
Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4651–4659, 2016
2016
-
[32]
Learning a deep embedding model for zero-shot learning
Li Zhang, Tao Xiang, Shaogang Gong, et al. Learning a deep embedding model for zero-shot learning. 2017
2017
-
[33]
Attention-aware deep adversarial hashing for cross-modal retrieval
Xi Zhang, Hanjiang Lai, and Jiashi Feng. Attention-aware deep adversarial hashing for cross-modal retrieval. In Eu- ropean Conference on Computer Vision , pages 614–629. Springer, 2018
2018
-
[34]
Separating style and content for generalized style transfer
Yexun Zhang, Ya Zhang, and Wenbin Cai. Separating style and content for generalized style transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, volume 1, 2018
2018
-
[35]
Dual-path convolutional image-text em- bedding
Zhedong Zheng, Liang Zheng, Michael Garrett, Yi Yang, and Yi-Dong Shen. Dual-path convolutional image-text em- bedding. arXiv preprint arXiv:1711.05535, 2017
2017 arXiv
-
[36]
Places: A 10 million image database for scene recognition
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6):1452–1464, 2018
2018
-
[37]
Unpaired image-to-image translation using cycle- consistent adversarial networks
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle- consistent adversarial networks. arXiv preprint, 2017
2017
-
[38]
Font recognition based on global texture analysis
Yong Zhu, Tieniu Tan, and Yunhong Wang. Font recognition based on global texture analysis. IEEE Transactions on pat- tern analysis and machine intelligence , 23(10):1192–1200, 2001
2001
-
[39]
Optical font recogni- tion using typographical features
Abdelwahab Zramdini and Rolf Ingold. Optical font recogni- tion using typographical features. IEEE Transactions on Pat- tern Analysis & Machine Intelligence, (8):877–882, 1998
1998
-
[40]
Large-scale Tag-based Font Retrieval with Generative Feature Learning
Supplementary Material In this document, we provide additional materials to sup- plement our paper “Large-scale Tag-based Font Retrieval with Generative Feature Learning”. In the first section, we provide more details about our collected font retrieval dataset. In the second se...
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.