REVIEW 4 major objections 5 minor 35 references
Cross-modal Zero-shot Hashing
T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read CZHash claims that a single joint objective—feature learning, category attribute spaces, and hash quantization—lets cross-modal hashing work for unseen categories with partially labeled data and differing label spaces.
desk verdict The problem setting is real and the high-level design is sensible, but the paper never defines a hash function for unseen queries, the gradients don't match the objective, and the reported gains are reversed on at least one benchmark configuration. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing objects are the composite similarity matrices and the unified objective. Within a modality, composite similarity supplements Euclidean feature similarity with Jaccard label similarity when both instances are labeled; across modalities, feature similarity is estimated by the average of paired within-modality similarities, and label overlap is added when both sides are labeled. This construction lets unlabeled data and partially overlapping label spaces steer deep feature learning. The unified objective ties this similarity-preserving feature learning to category-space learning, where each modality's deep features are reconstructed by a category representation matrix times a shared attribute matrix built from the union label space, and to hash-quantization learning, where category-space codes are linearly mapped to a shared binary code matrix; a sign-based update gives the closed-form code assignment at each alternating step.
What would settle it
Hold all training structure fixed but replace the Word2Vec category attribute vectors with random vectors of the same dimension (or with permuted vectors). If MAP for unseen-class queries does not drop to near chance, the zero-shot transfer is not actually mediated by the semantic attribute space the paper describes; if it does drop, the method's promise hinges on the quality of the category-name embedding.
Extended reading notes
Core claim
The discovery, stated on the paper's own terms, is that zero-shot cross-modal hashing remains feasible when labeled data are scarce and the modalities' label spaces only partially overlap, provided label information is folded into a composite similarity rather than used as a hard semantic embedding. The paper builds a composite similarity that uses Jaccard overlap of labels when labels exist and Euclidean feature similarity otherwise, with inter-modality similarity approximated by the average of paired within-modality similarities. It then minimizes a unified objective that simultaneously preserves this similarity in deep feature spaces, projects features into a category attribute space built from class-level Word2Vec attribute vectors, and quantizes the projected codes into shared binary codes. The optimization alternates between feature networks, category-space coefficients, hashing coefficients, and the binary code matrix. On NUS-WIDE, Wiki, and MIRFlickr, the paper reports consistently higher mean average precision than the compared deep and non-deep methods, including the cross-modal zero-shot baseline AgNet, with the largest gains in the semi-supervised and different-label-space scenarios.
Load-bearing premise
The zero-shot transfer rests on Word2Vec vectors of category names placing seen and unseen categories close enough that the learned mapping to hash codes generalizes; if those vectors misplace unseen classes, the hash codes for those classes are effectively arbitrary.
Editorial extensions
If this is right
- A deployed retrieval system could serve newly appeared categories without retraining the hash functions, provided it can obtain the new category's attribute vector (its name's embedding).
- Partially labeled multi-modal collections with mismatched label vocabularies become usable for hash-code learning: unlabeled instances contribute through feature similarity, and labels from either modality contribute even when the label appears in only one modality.
- Jointly optimizing feature learning, category-space learning, and quantization avoids the feature/quantization incompatibility the paper attributes to AgNet, and the reported MAP gains reflect this design.
- The closed-form binary-code update at each iteration keeps the alternating optimization tractable on large collections with minibatch SGD.
- The formulation is written for two modalities but the authors state it extends to m≥3 modalities, so multi-view zero-shot retrieval is a direct corollary of the setup.
Reading between the lines
- The paper fixes both modalities' attribute matrices to their union; an untested alternative is to weight each modality's contribution to the union, since a label present in only one modality could otherwise distort the other modality's learned category space.
- Because the attribute space is built from category-name embeddings rather than hand-engineered attributes, the method is portable to new vocabularies; this also implies the text-embedding model's quality sets a ceiling on zero-shot transfer, so swapping the embedding model should measurably change MAP.
- The composite similarity is computed on raw features before deep learning; replacing it with a self-supervised learned similarity on unlabeled data is a natural extension that could further reduce the need for labels.
- The sign-based code update connects CZHash to binary embedding methods; a testable extension is whether the learned codes also serve as features for zero-shot classification, not only retrieval.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CZHash, a cross-modal zero-shot hashing method that jointly learns deep feature representations, category attribute spaces, and hashing quantization in a unified objective (Eq. (7)). It introduces a composite similarity measure (Eqs. (2)-(3)) that combines label and feature similarities to handle labeled and unlabeled data, and it allows different label spaces across modalities. The authors report MAP improvements over several cross-modal and zero-shot hashing baselines on Nus-wide, Wiki, and Mirflickr under four scenarios (Tables IV-VII). The central problem, however, is that the paper never specifies how a query from an unseen category is encoded into a binary hash code at test time, and the optimization details in Section III-C.4 are internally inconsistent with the stated objective.
Significance. If the method were fully specified and reproducible, it would address a genuinely useful and under-studied setting: cross-modal zero-shot hashing with semi-supervised data and per-modality label spaces. The problem formulation is well motivated, and the experimental design covers several realistic scenarios. The paper also makes a reasonable attempt to leverage unlabeled data through the composite similarity. However, the significance assessment hinges entirely on whether a test-time hashing procedure exists; since the manuscript does not define one, the claimed contributions are not currently executable or verifiable.
major comments (4)
- [III-A and III-C.4] The paper never defines a test-time hash function for unseen queries. Section III-A promises two hashing functions H1 and H2, but they are never specified. In Eq. (7), the learned variables include per-instance matrices C(1) and C(2) for the n training instances, and the optimization in Section III-C.4 updates only these training variables. For a new query x from an unseen category, there is no rule to compute C(v) from the deep feature F(v), and no closed-form mapping from F(v) alone to a binary code is derived. The quantization term ||C(v)W(v) - B||^2 is defined only for the training instances. Thus the zero-shot retrieval procedure promised in the abstract and experiments is not defined, and the results in Tables IV-VII rely on an unstated evaluation protocol.
- [III-C.4, Eqs. (8)-(11)] The reported gradients are inconsistent with the objective in Eq. (7). For example, the gradient w.r.t. F(1) in Eq. (8) contains an unexplained 1/2 factor and a term involving S22, but the first term of Eq. (7) for the (1,1) and (1,2) pairs does not produce an S22 contribution in the derivative w.r.t. F(1). The derivative of ||F(1)-C(1)A(1)||^2 w.r.t. C(1) should be -2(F(1)-C(1)A(1))A(1)^T, whereas Eq. (9) has a 1/2 factor and no factor of 2 on the alpha term; the beta term is also missing a factor of 2. Moreover, Eq. (11) states B in {−1,1}^{n×c}, while Eq. (6) and Eq. (7) define B in R^{n×b}, where b is the code length and c is the number of categories; these are different dimensions. These inconsistencies mean that the optimization procedure as written is not a well-defined minimization of Eq. (7).
- [IV-B, Tables IV-VII] The experimental section does not specify how queries are encoded in the zero-shot scenarios. The text describes splitting categories into seen and unseen and testing on the unseen categories, but it never states whether the learned mapping is applied to the unseen query's feature vector, whether the attribute vector of the unseen category is used, or how the hash code is generated. The MAP definition in Eq. (13) does not clarify the protocol. Without a precise test-time inference rule, the reported performance numbers in Tables IV-VII cannot be reproduced or attributed to the model described in Section III. This is a load-bearing omission because the central claim of the paper is that CZHash can hash unseen categories.
- [III-C.2 and Eq. (4) vs. Eq. (7)] There is an inconsistency between the objective in Eq. (4) and the unified objective in Eq. (7). Eq. (4) contains three terms, with the cross-modal term weighted by 2, while Eq. (7) writes a double sum over v,v'=1,2, which would produce four terms (S11, S12, S21, S22) with equal weighting. If S12 is intended to be symmetric and counted twice, the relationship to Eq. (4) should be stated explicitly. Additionally, the claim in Section III-C.2 that A(1) and A(2) can be set to their union 'without loss of generality' assumes that categories absent from one modality do not distort that modality's learned category space; this assumption is not justified, and it affects the zero-shot transfer mechanism.
minor comments (5)
- [Table I] The table is badly garbled: the check/cross marks do not align with the column headers, and the row for 'PRDH' contains a placeholder '[?]'. This makes the comparison table difficult to interpret.
- [IV-B] The text introducing Tables VI-VII refers to a method 'DRCMH', but the method described and listed elsewhere is 'RDCMH'. Please correct the typo.
- [Throughout] There are several typos and inconsistent notations: 'beacuase' (Section IV-E), 'Mirfilcker' (Fig. 2), 'CZSHash' (Conclusion), 'ZCHash' (Section III-C.2), 'non-liner' (Section III-C.1), and the duplicated 'θ(1)_x and θ(1)_x' in Section III-C.1. These should be corrected.
- [Fig. 2] The axis labels in Figure 2 are garbled and overlapping; the figure should be redrawn with clear tick labels for alpha and beta.
- [III-C.4] The text says Eq. (7) is solved via ADMM, but the subsequent description uses SGD and back-propagation. The connection between these optimization strategies should be clarified, or the ADMM reference should be removed.
Circularity Check
No significant circularity: the joint objective fits external attributes and benchmark-based similarities, and no reported prediction reduces to a fitted input or a self-citation chain.
full rationale
The derivation chain in CZHash is self-contained against its inputs rather than circular. The composite similarities in Eqs. (1)-(3) are computed from input labels and features, but the optimized variables F, C, W, and B are not defined as those similarities; the hash matrix B is produced by the sign rule Eq. (12) from learned C and W, not taken directly from S or from the attribute matrices. The category attributes A(1), A(2) come from external Word2Vec vectors, and the experimental claims are evaluated against public benchmarks and independent baselines, so the central outperformance claim does not reduce to a fitted constant or to the model's own assumptions. The paper does cite RDCMH [11], whose authors overlap with the present paper, but RDCMH is used only as a related method and comparison baseline; CZHash outperforming RDCMH is an empirical result rather than a load-bearing premise. There is no imported uniqueness theorem, no ansatz smuggled in solely by self-citation, and no renaming of a known result as a new derivation. A genuine completeness concern exists: Section III-A promises hash functions H1 and H2, but the paper never defines their test-time form, and the optimization in Section III-C.4 only updates the training matrix B. That omission is a correctness or reproducibility issue, not a circularity, because nothing in the equations forces the reported MAP to equal an input by construction.
Assumptions & free parameters
free parameters (3)
- alpha (α) =
1
- beta (β) =
1
- d (deep feature and attribute dimension) =
500
assumptions (3)
- domain assumption Word2Vec attribute vectors encode sufficient semantic relationships between seen and unseen categories for zero-shot transfer.
- ad hoc to paper The attribute matrices A(1) and A(2) can be set to their union without loss of generality.
- ad hoc to paper The composite similarity in Eqs. (2)-(3) is a valid measure of semantic proximity for labeled/unlabeled and intra/inter-modal data.
Cite this review
Pith. "Pith review of Cross-modal Zero-shot Hashing." pith.science (2026). https://pith.science/paper/ATDCJ4OL
@misc{pith2026190807388,
author = {Pith},
title = {Pith review of: Cross-modal Zero-shot Hashing},
year = {2026},
howpublished = {\url{https://pith.science/paper/ATDCJ4OL}},
note = {Machine review of arXiv:1908.07388}
}
read the original abstract
Hashing has been widely studied for big data retrieval due to its low storage cost and fast query speed. Zero-shot hashing (ZSH) aims to learn a hashing model that is trained using only samples from seen categories, but can generalize well to samples of unseen categories. ZSH generally uses category attributes to seek a semantic embedding space to transfer knowledge from seen categories to unseen ones. As a result, it may perform poorly when labeled data are insufficient. ZSH methods are mainly designed for single-modality data, which prevents their application to the widely spread multi-modal data. On the other hand, existing cross-modal hashing solutions assume that all the modalities share the same category labels, while in practice the labels of different data modalities may be different. To address these issues, we propose a general Cross-modal Zero-shot Hashing (CZHash) solution to effectively leverage unlabeled and labeled multi-modality data with different label spaces. CZHash first quantifies the composite similarity between instances using label and feature information. It then defines an objective function to achieve deep feature learning compatible with the composite similarity preserving, category attribute space learning, and hashing coding function learning. CZHash further introduces an alternative optimization procedure to jointly optimize these learning objectives. Experiments on benchmark multi-modal datasets show that CZHash significantly outperforms related representative hashing approaches both on effectiveness and adaptability.
Figures
Reference graph
Works this paper leans on
-
[1]
Learning to hash for indexing big data – a survey,
J. Wang, W. Liu, S. Kumar, and S.-F. Chang, “Learning to hash for indexing big data – a survey,” Proceedings of the IEEE , vol. 104, no. 1, pp. 34–57, 2016
work page 2016
-
[2]
J. Wang, T. Zhang, N. Sebe, and H. T. Shen, “A survey on learning to hash,” TPAMI, vol. 40, no. 4, pp. 769–790, 2018
work page 2018
-
[3]
Online unsupervised multi-view feature selection,
W. Shao, L. He, C.-T. Lu, X. Wei, and S. Y . Philip, “Online unsupervised multi-view feature selection,” in ICDM, 2016, pp. 1203–1208
work page 2016
-
[4]
Multi-label cross-modal retrieval,
V . Ranjan, N. Rasiwasia, and C. V . Jawahar, “Multi-label cross-modal retrieval,” in ICCV, 2015, pp. 4094–4102
work page 2015
-
[5]
Gene function prediction based on gene ontology hierarchy preserving hashing,
Y . Zhao, G. Fu, J. Wang, M. Guo, and G. Yu, “Gene function prediction based on gene ontology hierarchy preserving hashing,” Genomics, vol. 111, pp. 334–342, 2019
work page 2019
-
[6]
Learning hash functions for cross-view similarity search,
S. Kumar and R. Udupa, “Learning hash functions for cross-view similarity search,” in IJCAI, 2011, pp. 1360–1365
work page 2011
-
[7]
Large-scale supervised multimodal hashing with semantic correlation maximization,
D. Zhang and W. J. Li, “Large-scale supervised multimodal hashing with semantic correlation maximization,” in AAAI, 2014, pp. 2177–2183
work page 2014
-
[8]
Latent semantic sparse hashing for cross-modal similarity search,
J. Zhou, G. Ding, and Y . Guo, “Latent semantic sparse hashing for cross-modal similarity search,” in ACM SIGIR, 2014, pp. 415–424
work page 2014
Show all 35 references
-
[9]
Collective matrix factorization hashing for multimodal data,
G. Ding, Y . Guo, and J. Zhou, “Collective matrix factorization hashing for multimodal data,” in CVPR, 2014, pp. 2083–2090
2014
-
[10]
Semi-supervised hashing for large-scale search
J. Wang, S. Kumar, and S. F. Chang, “Semi-supervised hashing for large-scale search.” TPAMI, vol. 34, no. 12, pp. 2393–2406, 2012
2012
-
[11]
Ranking- based deep cross-modal hashing,
X. Liu, G. Yu, C. Domeniconi, J. Wang, Y . Ren, and M. Guo, “Ranking- based deep cross-modal hashing,” arXiv preprint arXiv:1905.04450, 2019
1905 arXiv
-
[12]
Attribute-based classification for zero-shot visual object categorization,
C. H. Lampert, N. Hannes, and H. Stefan, “Attribute-based classification for zero-shot visual object categorization,” TPAMI, vol. 36, no. 3, pp. 453–465, 2014
2014
-
[13]
Learning multimodal latent attributes,
Y . Fu, T. M. Hospedales, T. Xiang, and S. Gong, “Learning multimodal latent attributes,” TPAMI, vol. 36, no. 2, pp. 303–316, 2014
2014
-
[14]
From zero-shot learning to conventional supervised classification: Unseen visual data synthesis,
Y . Long, L. Liu, L. Shao, F. Shen, G. Ding, and J. Han, “From zero-shot learning to conventional supervised classification: Unseen visual data synthesis,” in CVPR, 2017, pp. 6165–6174
2017
-
[15]
Zero-shot hashing via transferring supervised knowledge,
Y . Yang, Y . Luo, W. Chen, F. Shen, S. Jie, and H. T. Shen, “Zero-shot hashing via transferring supervised knowledge,” in ACM MM, 2016, pp. 1286–1295
2016
-
[16]
Scalable zero-shot learning via binary visual-semantic embeddings,
F. Shen, X. Zhou, J. Yu, Y . Yang, L. Liu, and H. T. Shen, “Scalable zero-shot learning via binary visual-semantic embeddings,” TIP, vol. 99, no. 1, pp. 1–10, 2019
2019
-
[17]
Attribute-guided network for cross-modal zero-shot hashing,
J. Zhong, Y . Sun, Y . Yu, Y . Pang, and J. Han, “Attribute-guided network for cross-modal zero-shot hashing,” TNNLS, vol. 99, no. 1, pp. 1–10, 2019
2019
-
[18]
Attribute hashing for zero-shot image retrieval,
Y . Xu, Y . Yang, F. Shen, X. Xu, Y . Zhou, and H. T. Shen, “Attribute hashing for zero-shot image retrieval,” in ICME, 2017, pp. 133–138
2017
-
[19]
Sitnet: discrete similarity transfer network for zero-shot hashing,
Y . Guo, G. Ding, J. Han, and Y . Gao, “Sitnet: discrete similarity transfer network for zero-shot hashing,” in IJCAI, 2017, pp. 1767–1773
2017
-
[20]
Transductive zero-shot hashing via coarse-to-fine similarity mining,
H. Lai, “Transductive zero-shot hashing via coarse-to-fine similarity mining,” in ICMR, 2018, pp. 196–203
2018
-
[21]
Data fusion through cross-modality metric learning using similarity-sensitive hashing,
M. M. Bronstein, A. M. Bronstein, F. Michel, and N. Paragios, “Data fusion through cross-modality metric learning using similarity-sensitive hashing,” in CVPR, 2010, pp. 3594–3601
2010
-
[22]
Quantized correlation hashing for fast cross-modal search,
B. Wu, Y . Qiang, W. S. Zheng, Y . Wang, and J. Wang, “Quantized correlation hashing for fast cross-modal search,” in AAAI, 2015, pp. 3946–3952
2015
-
[23]
Cross-view retrieval via probability- based semantics-preserving hashing,
Z. Lin, G. Ding, J. Han, and J. Wang, “Cross-view retrieval via probability- based semantics-preserving hashing,” IEEE Transactions on Cybernetics , vol. 47, no. 12, pp. 4342–4355, 2017
2017
-
[24]
Deep cross-modal hashing,
Q. Y . Jiang and W. J. Li, “Deep cross-modal hashing,” in CVPR, 2017, pp. 3270–3278
2017
-
[25]
Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly,
Y . Xian, C. H. Lampert, B. Schiele, and Z. Akata, “Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly,” TPAMI, vol. 99, no. 1, pp. 1–16, 2019
2019
-
[26]
A survey of zero-shot learning: Settings, methods, and applications,
W. Wang, V . W. Zheng, H. Yu, and C. Miao, “A survey of zero-shot learning: Settings, methods, and applications,” ACM TIST, vol. 10, no. 2, p. 13, 2019
2019
-
[27]
Semantics-preserving hashing for cross-view retrieval,
Z. Lin, G. Ding, M. Hu, and J. Wang, “Semantics-preserving hashing for cross-view retrieval,” in CVPR, 2015, pp. 3864–3872
2015
-
[28]
Correlation autoencoder hashing for supervised cross-modal search,
Y . Cao, M. Long, J. Wang, and H. Zhu, “Correlation autoencoder hashing for supervised cross-modal search,” in ACM ICMR, 2016, pp. 197–204
2016
-
[29]
Correlation hashing network for efficient cross-modal retrieval,
Y . Cao, M. Long, and J. Wang, “Correlation hashing network for efficient cross-modal retrieval,” in BMVC, 2017, pp. 1–12
2017
-
[30]
Multilabel dimensionality reduction via dependence maximization,
Y . Zhang and Z.-H. Zhou, “Multilabel dimensionality reduction via dependence maximization,” TKDD, vol. 4, no. 3, p. 14, 2010
2010
-
[31]
Multi-label sparse coding for automatic image annotation,
C. Wang, S. Yan, L. Zhang, and H.-J. Zhang, “Multi-label sparse coding for automatic image annotation,” in CVPR, 2009, pp. 1643–1650
2009
-
[32]
Imagenet classification with deep convolutional neural networks,
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in NeurIPS, 2012, pp. 1097– 1105
2012
-
[33]
Return of the devil in the details: Delving deep into convolutional nets,
K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman, “Return of the devil in the details: Delving deep into convolutional nets,” BMVC, vol. 5, no. 4, p. 14, 2014
2014
-
[34]
Distributed optimization and statistical learning via the alternating direction method of multipliers,
S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Foundations and Trends in Machine Learning , vol. 3, no. 1, pp. 1–122, 2011
2011
-
[35]
Distributed representations of words and phrases and their compositionality,
T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,”NeurIPS, vol. 26, pp. 3111–3119, 2013
2013
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.