REVIEW 5 major objections 5 minor 40 references
Cascaded Revision Network for Novel Object Captioning
T0 review · 5 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A caption-revising cascade lets an image captioning model name objects it never saw in training, reaching 64.08% average F1 on the eight held-out MSCOCO classes.
desk verdict A plausible cascade for novel object captioning with a real F1 gain, but the GloVe-based training/acceptance loop and a per-class overclaim need referee attention before the numbers are taken at face value. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the Cascaded Revision Network: a primary LSTM captioner, followed by a perplexity predictor that computes a confidence score $m_t=\sigma(W_m h_t+b_m)$ for each emitted word and flags words below a threshold $\tau_p$; a visual matching module that scores the hidden state $h_t$ against object-detector features and proposes object names; and a semantic matching module that gates each proposal by the cosine similarity of the flagged word and the detected object name in a pre-trained word-embedding space. The same similarity measure is used offline to build pseudo-object pairs, in which a known object is replaced by its most similar in-domain word so the captioner can practice the revise-and-match behaviour during training.
What would settle it
Retrain CRN on the held-out MSCOCO split using pseudo-object pairs chosen by word frequency rather than by embedding similarity, and at inference time replace the semantic gate with a frequency-matched random word; if the average F1 stays at 64.08%, embedding similarity is not the load-bearing mechanism.
Extended reading notes
Core claim
The central discovery is that the gap between a captioner's in-domain vocabulary and out-of-domain object names can be bridged by a caption-revise cascade instead of by training the captioner on the new objects. The authors claim that a perplexity predictor can identify which words in the initial caption are uncertain, a visual matching module can propose a detected object for each uncertain word, and a semantic matching module can reject proposals that would break the sentence's meaning. With these three stages cascaded after the primary captioner, CRN reports an average F1 of 64.08% on the eight held-out MSCOCO classes, which the paper states is 6.16 percentage points higher than the previous best of 57.92%. The paper also reports that the same approach scales when the detector is trained on a larger vocabulary, suggesting the mechanism itself is not tied to the eight test classes.
Load-bearing premise
The load-bearing premise is that two words with similar word-embedding vectors are genuinely interchangeable ways to describe the same pictured object, so swapping one for the other makes the caption better rather than merely different.
Editorial extensions
If this is right
- Novel object captioning becomes a post-editing problem: a base captioner trained on ordinary image-sentence pairs can acquire new object names by adding a detector and a word embedding, without paired captions for the new words.
- The cascade separates the question of which object is present from the question of where its name belongs in the sentence, so either stage can be upgraded independently.
- Because only low-confidence words are edited, known-object descriptions are mostly preserved, which is consistent with the paper's reported F1 scores on known objects.
- The reported scale-up to a larger detector vocabulary implies that the set of objects a captioning system can name is bounded mainly by the detector's vocabulary rather than by the captioning corpus.
Reading between the lines
- One consequence the paper leaves implicit is that the detector's recall sets an upper bound on the whole pipeline: if the detector never proposes a novel object, no later stage can insert its name, so gains should track improvements in rare-object detection.
- Since pseudo-object pairs are chosen by embedding similarity, the method inherits the biases of that embedding space; swapping in contextual word representations could change which words are flagged and accepted, and this is a testable variant the paper does not explore.
- The perplexity predictor could double as a novelty signal for human-in-the-loop systems: high-perplexity words mark places where the model itself suspects the caption is wrong, even when no detector-based replacement is available.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Cascaded Revision Network (CRN) for novel object captioning. CRN first generates a primary caption with an in-domain captioner, then uses a perplexity predictor to flag ambiguous words, a visual matching module to propose replacements from a pre-trained object detector, and a semantic matching module that uses GloVe cosine similarity to accept or reject the proposed replacements. The authors create pseudo-object training pairs by replacing in-domain object words with their GloVe nearest neighbors, and they report an average F1 of 64.08% on the eight held-out MSCOCO classes, which they state is 6.16% above the previous best of 57.92%, together with results on ImageNet and ablations.
Significance. The idea of decomposing novel-object captioning into a self-aware revision process with visual and semantic matching is interesting and practically motivated. The paper follows the standard held-out MSCOCO protocol, compares against several recent systems, and includes ablation studies that support the contribution of each module. If the reported gains are reproducible, the method is a useful step for describing images with objects unseen in caption training data. The average F1 improvement over DNOC is credible under the stated protocol. However, the paper's central claim is weakened by an incorrect statement about per-class results, an undefined dimension match in the core visual-matching equation, and the lack of independent validation of the GloVe-based semantic gate that is used both to construct training labels and to accept test-time replacements.
major comments (5)
- [IV-B, Table I] The paper states that 'the F1-scores of all novel objects surpass the best state-of-the-art result,' but Table I contradicts this. For example, CRN's microwave F1 is 53.76 versus 61.90 for NBT+G, racket is 62.02 versus 70.27 for LSTM-C*, suitcase is 57.69 versus 59.48 for DNOC, and zebra is 85.38 versus 92.03 for LSTM-C. Please correct the claim and discuss the classes where CRN is not the best.
- [III-B3, Eq. (11)] Equation (11), S_t = V_d h_t, requires the column dimension of V_d to equal the dimension of the captioner hidden state h_t, but the paper never specifies such a match. The captioner's LSTM hidden state is 1024-dimensional, while the object visual features are obtained by reusing the VGG network whose fc7 features are 4096-dimensional in the experimental setup. As written, the product is undefined. Please specify how the visual features are projected to the hidden-state space, or revise the equation to include the learned projection.
- [III-B1 and III-B4] The same GloVe cosine similarity is used for two purposes: to construct pseudo-object training pairs by pairing each object with its most similar in-domain word, and to accept or reject test-time replacements in the semantic matching module. This creates a closed semantic loop that is not independently validated. In particular, the qualitative example in Figure 4 shows that the gate accepts 'bus' as a replacement for 'railway' to produce 'busstation', which is a GloVe-similar but semantically wrong insertion. The METEOR score of 21.31 is also below several baselines in Table I. Please report precision and recall separately, add human evaluation or an independent compatibility measure, and discuss cases where the GloVe gate fails.
- [III-B3, Eq. (13)] Equation (13) is not well defined: the text above states that N_d is the number of target classes of the detector, but the loss definition says 'N_d is the number of detected objects at time step t,' and the term p(o_t|h_t) is not formally defined. Please clarify the notation and define the probability distribution used in the detection loss.
- [III-B2, Eq. (7)] Equation (7) defines h_t = w_h^T tanh(W_s x_t + W_z h_{t-1}), which produces a scalar if w_h is a weight vector, yet h_t is subsequently used as a vector in Equations (11) and (13) and in the rest of the model. Please correct the notation for the hidden state update, for example by removing w_h^T or by specifying that h_t is a vector-valued state.
minor comments (5)
- [III-B3] There is a typo: 'furher' should be 'further' in the description of the visual matching module.
- [IV-C, Table III] The row labels 'CRN I + II' and 'CRN w/o II' are easy to confuse; please clarify that 'CRN I + II' means the captioner plus perplexity predictor, while 'CRN w/o II' means the full model without the perplexity predictor.
- [IV-A] The threshold tau_p is described as 'learned by the model' but also set to 0.15 and tuned in Figure 5; please clarify whether the threshold is learned during training or selected on the validation set.
- [Figure 4] The generated caption 'a woman is looking at a busstation' should be discussed explicitly, since it illustrates a failure of the semantic matching module rather than a success; the current text presents the figure without highlighting this problematic case.
- [Various] Several minor language issues remain, such as 'Hanzhou Dianzi Univeristy' in the author biography and 'out-of MSCOCO' in Section IV-C; a careful proofread is recommended.
Circularity Check
No significant circularity: CRN's reported gains rest on an external held-out benchmark; the shared GloVe similarity is a consistent design choice, not a forced prediction.
full rationale
The paper's derivation is not circular in the sense defined by the analysis criteria. CRN uses a pre-trained GloVe embedding twice: once to construct pseudo-object training pairs (Section IV-A: each novel object is replaced by its most similar in-domain word under cosine similarity) and once in the semantic matching module (Section III-B.4: the detected object with the largest cosine similarity to an ambiguous caption word is selected for replacement). This is a consistent reuse of an external semantic resource, not a self-definitional reduction: the claimed outcome, an average F1 of 64.08% on the eight held-out MSCOCO classes, is measured against external ground-truth object labels, and the final caption word is additionally constrained by the primary captioner's output, the object detector's proposals, and the perplexity predictor's choices. The held-out classes were never used to fit the GloVe-based pseudo-object mapping, and the F1 evaluation is independent of the training signal. If GloVe similarity were a poor proxy for visual accuracy, the method's performance would suffer, but that is an empirical correctness risk, not a logical circularity. The only self-citation, DNOC [11], shares authors with the present paper, but it is cited for experimental setup and comparison, not as the load-bearing justification for CRN's central claim. No equation or fitted parameter is renamed as a prediction, and no uniqueness claim is imported from the authors' prior work. Therefore the derivation chain is self-contained against an external benchmark, and the circularity score is 0.
Assumptions & free parameters
free parameters (1)
- Perplexity threshold tau_p =
0.15
assumptions (4)
- domain assumption Pseudo-object pairs generated by GloVe nearest-neighbor substitution are a valid simulation of real novel-object caption ambiguity.
- ad hoc to paper GloVe cosine similarity is a valid criterion for accepting or rejecting test-time replacements in the semantic matching module.
- domain assumption The pretrained Faster R-CNN detector provides reliable object names for out-of-domain objects, including the held-out classes.
- ad hoc to paper Visual features from the detector ROI pooling and captioner hidden states are directly compatible via St = Vd ht without a learned projection.
invented entities (1)
-
Pseudo objects, GloVe-nearest words substituted for real object words in training captions
Cite this review
Pith. "Pith review of Cascaded Revision Network for Novel Object Captioning." pith.science (2026). https://pith.science/paper/TMZVLV7J
@misc{pith2026190802726,
author = {Pith},
title = {Pith review of: Cascaded Revision Network for Novel Object Captioning},
year = {2026},
howpublished = {\url{https://pith.science/paper/TMZVLV7J}},
note = {Machine review of arXiv:1908.02726}
}
read the original abstract
Image captioning, a challenging task where the machine automatically describes an image by sentences, has drawn significant attention in recent years. Despite the remarkable improvements of recent approaches, however, these methods are built upon a large set of training image-sentence pairs. The expensive labor efforts hence limit the captioning model to describe the wider world. In this paper, we present a novel network structure, Cascaded Revision Network, which aims at relieving the problem by equipping the model with out-of-domain knowledge. CRN first tries its best to describe an image using the existing vocabulary from in-domain knowledge. Due to the lack of out-of-domain knowledge, the caption may be inaccurate or include ambiguous words for the image with unknown (novel) objects. We propose to re-edit the primary captioning sentence by a series of cascaded operations. We introduce a perplexity predictor to find out which words are most likely to be inaccurate given the input image. Thereafter, we utilize external knowledge from a pre-trained object detection model and select more accurate words from detection results by the visual matching module. In the last step, we design a semantic matching module to ensure that the novel object is fit in the right position. By this novel cascaded captioning-revising mechanism, CRN can accurately describe images with unseen objects. We validate the proposed method with state-of-the-art performance on the held-out MSCOCO dataset as well as scale to ImageNet, demonstrating the effectiveness of this method.
Figures
Reference graph
Works this paper leans on
-
[1]
Show and tell: A neural image caption generator,
O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” in CVPR, 2015
2015
-
[2]
Densecap: Fully convolutional localization networks for dense captioning,
J. Johnson, A. Karpathy, and L. Fei-Fei, “Densecap: Fully convolutional localization networks for dense captioning,” in CVPR, 2016
work page 2016
-
[3]
Show, observe and tell: Attribute-driven attention model for image captioning,
H. Chen, G. Ding, Z. Lin, S. Zhao, and J. Han, “Show, observe and tell: Attribute-driven attention model for image captioning,” in IJCAI, 2018
work page 2018
-
[4]
Show and tell more: Topic- oriented multi-sentence image captioning,
Y . Mao, C. Zhou, X. Wang, and R. Li, “Show and tell more: Topic- oriented multi-sentence image captioning,” in IJCAI, 2018
work page 2018
-
[5]
Bottom-up and top-down attention for image captioning and VQA,
P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang, “Bottom-up and top-down attention for image captioning and VQA,” in CVPR, 2018
work page 2018
-
[6]
Video-based human behavior understanding: A survey,
P. V . K. Borges, N. Conci, and A. Cavallaro, “Video-based human behavior understanding: A survey,” IEEE Transactions on Circuits and Systems for Video Technology , 2013
work page 2013
-
[7]
R. V . H. M. Colque, C. Caetano, M. T. L. de Andrade, and W. R. Schwartz, “Histograms of optical flow orientation and magnitude and entropy to detect anomalous events in videos,” IEEE Transactions on Circuits and Systems for Video Technology , 2017
work page 2017
-
[8]
T-cnn: Tubelets with convolutional neural networks for object detection from videos,
K. Kang, H. Li, J. Yan, X. Zeng, B. Yang, T. Xiao, C. Zhang, Z. Wang, R. Wang, X. Wang, and W. Ouyang, “T-cnn: Tubelets with convolutional neural networks for object detection from videos,” IEEE Transactions on Circuits and Systems for Video Technology , 2018
work page 2018
Show all 40 references
-
[9]
Deep compositional captioning: Describing novel object categories without paired training data,
L. Anne Hendricks, S. Venugopalan, M. Rohrbach, R. Mooney, K. Saenko, and T. Darrell, “Deep compositional captioning: Describing novel object categories without paired training data,” in CVPR, 2016
2016
-
[10]
Neural baby talk,
J. Lu, J. Yang, D. Batra, and D. Parikh, “Neural baby talk,” in CVPR, 2018
2018
-
[11]
Decoupled novel object captioner,
Y . Wu, L. Zhu, L. Jiang, and Y . Yang, “Decoupled novel object captioner,” in ACM MM, 2018
2018
-
[12]
Faster R-CNN: towards real-time object detection with region proposal networks,
S. Ren, K. He, R. B. Girshick, and J. Sun, “Faster R-CNN: towards real-time object detection with region proposal networks,” IEEE Trans. Pattern Anal. Mach. Intell. , 2017
2017
-
[13]
ImageNet: A Large-Scale Hierarchical Image Database,
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” in CVPR, 2009
2009
-
[14]
I2T: image parsing to text description,
B. Z. Yao, X. Yang, L. Lin, M. W. Lee, and S. C. Zhu, “I2T: image parsing to text description,” IEEE, 2010
2010
-
[15]
Im2text: Describing images using 1 million captioned photographs,
V . Ordonez, G. Kulkarni, and T. L. Berg, “Im2text: Describing images using 1 million captioned photographs,” in NIPS, 2011
2011
-
[16]
Deep captioning with multimodal recurrent neural networks (m-rnn),
J. Mao, W. Xu, Y . Yang, J. Wang, Z. Huang, and A. Yuille, “Deep captioning with multimodal recurrent neural networks (m-rnn),” inICLR, 2015
2015
-
[17]
Deep visual-semantic alignments for generating image descriptions,
A. Karpathy and F. Li, “Deep visual-semantic alignments for generating image descriptions,” in CVPR, 2015
2015
-
[19]
Show, attend and tell: Neural image caption generation with visual attention,
K. Xu, J. Ba, R. Kiros, K. Cho, A. C. Courville, R. Salakhutdinov, R. S. Zemel, and Y . Bengio, “Show, attend and tell: Neural image caption generation with visual attention,” in ICML, 2015
2015
-
[20]
Learning like a child: Fast novel visual concept learning from sentence descriptions of images,
J. Mao, X. Wei, Y . Yang, J. Wang, Z. Huang, and A. L. Yuille, “Learning like a child: Fast novel visual concept learning from sentence descriptions of images,” in ICCV, 2015
2015
-
[21]
Image caption with global-local attention,
L. Li, S. Tang et al. , “Image caption with global-local attention,” in AAAI, 2017
2017
-
[22]
Text-guided attention model for image captioning,
J. Mun, M. Cho, and B. Han, “Text-guided attention model for image captioning,” in AAAI, 2017
2017
-
[23]
Learning to compose topic-aware mixture of experts for zero-shot video captioning,
X. Wang, J. Wu, D. Zhang, Y . Su, and W. Y . Wang, “Learning to compose topic-aware mixture of experts for zero-shot video captioning,” in AAAI, 2019
2019
-
[24]
Toward abnormal trajectory and event detection in video surveillance,
S. Cos ¸ar, G. Donatiello, V . Bogorny, C. Garate, L. O. Alvares, and F. Br´emond, “Toward abnormal trajectory and event detection in video surveillance,” IEEE Transactions on Circuits and Systems for Video Technology, 2017
2017
-
[25]
Cascade recurrent neural network for image caption generation,
J. Wu and H. Hu, “Cascade recurrent neural network for image caption generation,” Electronics Letters, 2017
2017
-
[26]
Very deep convolutional networks for large-scale image recognition,
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in ICLR, 2015
2015
-
[27]
Deep residual learning for image recognition,
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in CVPR, 2016
2016
-
[28]
Long short-term memory,
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, 1997
1997
-
[29]
Learning phrase representations using RNN encoder-decoder for statistical machine translation,
K. Cho, B. van Merrienboer et al. , “Learning phrase representations using RNN encoder-decoder for statistical machine translation,” in EMNLP, 2014
2014
-
[30]
Attribute-based classi- fication for zero-shot visual object categorization,
C. H. Lampert, H. Nickisch, and S. Harmeling, “Attribute-based classi- fication for zero-shot visual object categorization,” IEEE Trans. Pattern Anal. Mach. Intell. , 2014
2014
-
[31]
Zero-shot learning - A comprehensive evaluation of the good, the bad and the ugly,
Y . Xian, C. H. Lampert, B. Schiele, and Z. Akata, “Zero-shot learning - A comprehensive evaluation of the good, the bad and the ugly,” in CVPR, 2017
2017
-
[32]
Low-rank embedded ensemble semantic dictionary for zero-shot learning,
Z. Ding, M. Shao, and Y . Fu, “Low-rank embedded ensemble semantic dictionary for zero-shot learning,” in CVPR, 2017
2017
-
[33]
Devise: A deep visual- semantic embedding model,
A. Frome, G. S. Corrado, J. Shlens et al. , “Devise: A deep visual- semantic embedding model,” in NIPS, 2013
2013
-
[34]
Microsoft COCO: common objects in context,
T. Lin, M. Maire et al., “Microsoft COCO: common objects in context,” in CVPR, 2014
2014
-
[35]
Captioning images with diverse objects,
S. Venugopalan, L. A. Hendricks, M. Rohrbach, R. J. Mooney, T. Darrell, and K. Saenko, “Captioning images with diverse objects,” in CVPR, 2017
2017
-
[36]
Incorporating copying mechanism in image captioning for learning novel objects,
T. Yao, Y . Pan, Y . Li, and T. Mei, “Incorporating copying mechanism in image captioning for learning novel objects,” in CVPR, 2017
2017
-
[37]
Guided open vocabulary image captioning with constrained beam search,
P. Anderson, B. Fernando, M. Johnson, and S. Gould, “Guided open vocabulary image captioning with constrained beam search,” in EMNLP, 2017
2017
-
[38]
Speed/accuracy trade-offs for modern convolutional object detectors,
J. Huang, V . Rathod, C. Sun et al. , “Speed/accuracy trade-offs for modern convolutional object detectors,” in CVPR, 2016
2016
-
[39]
Long-term recurrent convolutional networks for visual recognition and description,
J. Donahue, L. A. Hendricks, S. Guadarrama et al., “Long-term recurrent convolutional networks for visual recognition and description,” inCVPR, 2015
2015
-
[40]
Visual genome: Connecting language and vision using crowdsourced dense image annotations,
R. Krishna, Y . Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz et al. , “Visual genome: Connecting language and vision using crowdsourced dense image annotations,” International Journal of Computer Vision , 2016. 9 Qianyu Feng received the M.S. degree in Shanghai Jiao Tong Uni...
2016
-
[2015]
student in University of Technology Sydney, Australia
He is currently a Ph.D. student in University of Technology Sydney, Australia. His research interests include language navigation and person re- identification. Hehe Fan received the M.S. degree in Huazhong University of Science and Technology, China, in 2015. He is currently a...
2015
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.