Pith. sign in

REVIEW 4 major objections 5 minor 34 references

Unpaired Cross-lingual Image Caption Generation with Self-Supervised Rewards

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Unpaired cross-lingual captioning is improved by self-supervised fluency and visual-relevancy rewards, with no paired target-language captions.

desk verdict A well-executed integration of fluency and visual-relevance rewards into RL for unpaired cross-lingual captioning, with real gains over baselines but a soft spot in the relevance reward's reliability on the target domain. read the letter →

arxiv 1908.05407 v1 pith:J2MZNDR5 submitted 2019-08-15 cs.CV cs.CL

Yuqing Song , Shizhe Chen , Yida Zhao , Qin Jin This is my paper · ORCID
classification cs.CVcs.CL
keywords cross-lingualimagecaptioningunpairedself-supervisedrewardreinforcementlearningvisual-semanticmatchingpivotlanguagefluency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tackles unpaired cross-lingual image captioning, where captions must be written in a target language without any human-annotated image–caption pairs in that language. It starts from the standard pivot approach—generate a caption in a pivot language, then machine-translate it into the target language—and argues that the remaining bottleneck is translation error, specifically disfluency and visual irrelevancy. The proposed self-supervised rewarding (SSR) model keeps the no-paired-data setup but trains the captioning model with reinforcement learning, using a fluency reward from target-language text and sentence- plus concept-level visual-relevancy rewards. If the paper is right, a language-pivoted pipeline can reach caption quality close to a supervised model trained on about four thousand paired captions, on both English and Chinese tasks, without collecting any target-language image–caption pairs.

What carries the argument

The load-bearing mechanism is the self-critical reinforcement learning loop, with three self-supervised reward functions replacing external caption-quality metrics. A fixed LSTM language model, pre-trained on mono-lingual target-language sentences, provides the fluency reward. A multi-level visual-semantic matching model (ML-VSE) provides the relevancy rewards: a sentence-level image–sentence scorer trained with a contrastive ranking loss and hard negative mining, and a concept-level image–concept scorer whose scores are normalized by concept prior probability to prevent frequent concepts from dominating. Both relevance scorers are trained on the same pseudo-translated image–caption pairs used to train the caption generator, which is why the rewards are called self-supervised.

What would settle it

Collect a small set of manually verified image–caption pairs in the target language, run the trained ML-VSE model on them, and measure retrieval recall: if true captions rank near chance while fluent translated captions score highly, the relevancy reward is not correcting visual errors but is inheriting the translator’s bias.

Watch

Extended reading notes

Core claim

The central claim is that translation errors, not model architecture, are the main weakness of pivot-based unpaired cross-lingual captioning, and that self-supervised rewards can repair them. Concretely, the paper shows that a captioning model trained on machine-translated pseudo pairs, then fine-tuned with reinforcement learning guided by three reward signals—an LSTM fluency score, an image–sentence relevance score, and an image–concept relevance score—outperforms the pseudo-pair baseline, the same baseline trained with CIDEr reward, and two two-stage pivot baselines. The experiments cover unpaired English captioning (Chinese as pivot) on MSCOCO and unpaired Chinese captioning (English as pivot) on AIC-ICC, and the paper reports the best BLEU, METEOR, and CIDEr scores among the compared methods, with human ratings on an English test set also favoring the SSR captions on fluency and visual relevancy.

Load-bearing premise

The relevancy reward model is trained on the same machine-translated captions that the captioning model learns from, so the whole improvement rests on the assumption that those pseudo pairs are accurate enough about which words belong with which image for the reward signal to correct errors rather than reinforce the translator’s mistakes.

Editorial extensions

If this is right

  • A useful captioning model for a new language can be built from mono-lingual text plus an existing pivot-language captioning system, without any target-language image–caption pairs.
  • Unpaired training with the proposed rewards reaches CIDEr comparable to a supervised model trained on roughly four thousand paired captions, indicating the gap to supervised captioning is not prohibitive.
  • The same reward scheme can be applied on top of any pivot-to-target translation service, because the rewards correct the translator’s errors instead of requiring the translator to be retrained or coupled to the captioning model.
  • Out-of-domain mono-lingual text still produces gains, so the fluency reward does not depend on having caption-style text in the target language.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same recipe could be tested on low-resource languages beyond English and Chinese, or on video captioning, whenever a pivot language and mono-lingual target text exist.
  • Because the relevance reward is learned from the same noisy pseudo pairs that train the captioner, a translator with a systematic visual bias (for example, always dropping people) could teach the reward model to penalize correct captions; filtering pseudo pairs or adding robust training before reinforcement learning would be a natural safeguard.
  • Swapping the LSTM language model for a larger pretrained language model is an obvious extension; whether fluency rewards then capture longer-range coherence, and whether that closes more of the gap to fully supervised captioning, remains an open empirical question.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper addresses unpaired cross-lingual image captioning, where no image–caption pairs exist in the target language. It uses a pivot language: an image-to-pivot captioning model and a pivot-to-target machine translation model produce pseudo image–target caption pairs, and a captioning model is trained on these noisy pairs. The authors propose a self-supervised rewarding (SSR) framework in reinforcement learning: a fluency reward from a target-language LSTM language model trained on mono-lingual text, and sentence-level and concept-level visual relevancy rewards from a multi-level visual-semantic embedding model (ML-VSE) trained on the same pseudo pairs. Experiments for unpaired English (using Chinese as pivot) and unpaired Chinese (using English as pivot) on MSCOCO and AIC-ICC report large gains over the implemented baselines, with ablations and human evaluation supporting the contribution.

Significance. If the reported gains are robust, the paper offers a practical method for extending image captioning to low-resource languages without paired data, using only mono-lingual corpora and an existing MT system. The reinforcement-learning formulation with self-supervised rewards is a reasonable extension of prior fluency-guided work, and the inclusion of both sentence-level and concept-level relevancy signals is a useful idea. The paper also provides ablations, an out-of-domain corpus study, a comparison to supervised captioning with limited data, and human evaluation, which are strengths. However, the central mechanism—the visual relevancy reward—is trained on the same noisy pseudo pairs that generate the training signal, and the paper's own retrieval evaluation shows weak performance on the target evaluation domain; without direct evidence that the reward is trustworthy, the claimed correction of visual irrelevancy remains uncertain. The absence of error bars/significance tests and the omission of pivot baselines for Chinese further limit confidence in the quantitative claims.

major comments (4)
  1. [§3.3, Eq. (6)–(10) and Table 4] The relevancy reward is produced by ML-VSE trained on the same pseudo image–caption pairs DT that supervise the captioning model, which creates a circularity risk: fluent-but-visually-wrong translations may be learned as correct visual-semantic associations and then be reinforced by the reward. The paper's own Table 4 shows a large drop in retrieval accuracy when moving from the training domain (AIC-ICC val: image-to-text R@1=52.8, text-to-image R@1=37.7) to the target evaluation domain (MSCOCO test: R@1=22.7 and 12.8, respectively), which is precisely the regime in which generated captions are judged. Please provide direct evidence that the reward is reliable on the target domain—for example, ranking accuracy on held-out target-domain captions, or an analysis comparing reward scores for correct versus hallucinated captions.
  2. [Table 3] The ablation results are not monotonic for Chinese: adding the concept-level reward r_crlv to r_flc + r_srlv decreases CIDEr from 18.5 to 18.3 (BLEU-4 improves from 12.1 to 12.3). The text states that combining fluency with both sentence- and concept-level rewards achieves additional gains on both languages, but this is not supported for Chinese CIDEr. Please report the variance across runs or significance tests and discuss this inconsistency, especially given the argument that concept-level rewards are needed.
  3. [Table 2] The Chinese unpaired captioning experiments omit the two-stage pivot baselines (2-Stage pivot Google and 2-Stage pivot joint) that are reported for the English task. As a result, the claim that SSR achieves the best performance among all compared methods is not fully demonstrated for Chinese, and the cross-language comparison is asymmetric. Please add these baselines for Chinese or explicitly justify their omission (e.g., incompatibility of the Chinese tokenizer or translation API).
  4. [§4.2–§4.3] No error bars, standard deviations, or significance tests are reported for any table, yet the abstract claims "significant performance improvement over state-of-the-art methods." The hyperparameters α, β, γ, λ are tuned on the validation set, which can inflate reported test performance. Please report results over multiple random seeds with standard deviations and, where feasible, paired significance tests (e.g., bootstrap or t-test) for the main comparisons.
minor comments (5)
  1. [Abstract] "descent image captioning models" should be "decent image captioning models."
  2. [§2.2] "Lanet al." is missing a space and should read "Lan et al."
  3. [§3.3] The sentence "we call this relevancy reward computed by the visual semantic matching model as 'self-supervised' reward" begins with a lowercase letter; please capitalize "We" and check the phrasing for grammatical completeness.
  4. [Eq. (10)] The concept-level reward r_crlv(w_j) is applied per token without a self-critical baseline, unlike the sentence-level rewards. Please clarify whether this is intended as a token-level reward and how its scale is calibrated relative to the sentence-level rewards in the joint loss of Eq. (2).
  5. [Table 6] The comparison with supervised captioning models would be clearer if it stated whether the supervised models use the same architecture (ResNet-101 encoder, single-layer LSTM decoder) and the same training schedule; currently only the number of pairs is varied.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: final caption metrics are evaluated externally, and reward models are training signals, not fitted predictions.

full rationale

The paper's derivation chain is empirical and externally validated. The final model is evaluated with BLEU/CIDEr/METEOR and human judgments against human reference captions on held-out MSCOCO and AIC-ICC test sets, not against the learned reward functions. The fluency reward (Eq. 4) comes from a language model trained on text-only corpora (Section 4.2), which is independent of the image-caption supervision. The relevancy reward (Eqs. 7, 9) uses ML-VSE trained on the same pseudo pairs DT that also supervise the caption generator (Eq. 1), so the reward signal can inherit translator bias; Table 4's low retrieval scores on MSCOCO test (R@1 22.7 image-to-text) and Table 3's mixed Chinese ablation (CIDEr 18.5 to 18.3 when adding r_crlv) quantify this risk. However, this is a training-signal validity concern, not a construction-level circularity: no equation makes the final caption output equal to the pseudo caption input, and the headline tests are external. Self-citations [4,5] appear only in related-work background and are not load-bearing.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method relies on several domain assumptions about the usability of machine-translated pseudo pairs and the validity of self-supervised rewards, but introduces no new physical or conceptual entities. The main free parameters are the four reward weighting hyperparameters tuned on validation.

free parameters (4)
  • alpha (fluency loss weight) = 0.05
    Chosen on the validation set according to loss scale and caption performance.
  • beta (relevancy loss weight) = 0.15
    Chosen on the validation set according to loss scale and caption performance.
  • gamma (relevancy loss weight) = 1.0
    Chosen on the validation set according to loss scale and caption performance.
  • lambda (concept prior penalty) = 0.5
    Chosen on the validation set; penalizes frequent concepts in the concept-level reward.
assumptions (4)
  • domain assumption The pivot-to-target machine translation model produces pseudo pairs that are mostly correct and informative enough to train both the captioning model and the visual-semantic matching model.
    The entire pipeline is initialized and rewarded based on these pseudo pairs; if the translator is too noisy, the rewards inherit the noise. This is the setup in Section 3.1 and 3.3.
  • domain assumption A language model trained on target-language text provides a valid fluency signal for generated captions.
    The fluency reward is the log probability from this LM, as defined in Eq. (4). The assumption is that higher LM probability correlates with human-perceived fluency in the captioning context.
  • ad hoc to paper Nouns and verbs in the pseudo captions capture the visual concepts needed for relevance.
    Concept extraction relies on POS tagging and assumes these two syntactic categories carry the semantic content relevant to images, as stated in Section 3.3.
  • domain assumption The contrastive ranking loss with hard negative mining learns a good joint embedding from noisy pseudo pairs.
    The VSE models are trained with Eq. (6) and Eq. (8) on pseudo pairs, and the reward quality depends on this joint embedding being meaningful despite the noise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unpaired Cross-lingual Image Caption Generation with Self-Supervised Rewards." pith.science (2026). https://pith.science/paper/J2MZNDR5

@misc{pith2026190805407,
  author       = {Pith},
  title        = {Pith review of: Unpaired Cross-lingual Image Caption Generation with Self-Supervised Rewards},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J2MZNDR5}},
  note         = {Machine review of arXiv:1908.05407}
}
read the original abstract

Generating image descriptions in different languages is essential to satisfy users worldwide. However, it is prohibitively expensive to collect large-scale paired image-caption dataset for every target language which is critical for training descent image captioning models. Previous works tackle the unpaired cross-lingual image captioning problem through a pivot language, which is with the help of paired image-caption data in the pivot language and pivot-to-target machine translation models. However, such language-pivoted approach suffers from inaccuracy brought by the pivot-to-target translation, including disfluency and visual irrelevancy errors. In this paper, we propose to generate cross-lingual image captions with self-supervised rewards in the reinforcement learning framework to alleviate these two types of errors. We employ self-supervision from mono-lingual corpus in the target language to provide fluency reward, and propose a multi-level visual semantic matching model to provide both sentence-level and concept-level visual relevancy rewards. We conduct extensive experiments for unpaired cross-lingual image captioning in both English and Chinese respectively on two widely used image caption corpora. The proposed approach achieves significant performance improvement over state-of-the-art methods.

Figures

Figures reproduced from arXiv: 1908.05407 by the authors.

Figure 1
Figure 1. Illustration of cross-lingual Chinese image caption [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the proposed SSR model framework, which consists of three components: a) the image captioning [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Top-10 predicted concepts for examples in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Examples of the English image captioning from the MSCOCO testing set, and Chinese image captioning from the [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 23 canonical work pages

  1. [1]

    Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering. In CVPR

    work page 2018

  2. [2]

    Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. In NIPS

    work page 2015

  3. [3]

    Ali Furkan Biten, Lluis Gomez, Marçal Rusiñol, and Dimosthenis Karatzas. 2019. Good News, Everyone! Context driven entity-aware captioning for news images. In CVPR

    work page 2019

  4. [4]

    Shizhe Chen, Qin Jin, and Jianlong Fu. 2019. From Words to Sentences: A Pro- gressive Learning Approach for Zero-resource Machine Translation with Visual Pivots. In IJCAI

    work page 2019

  5. [5]

    Shizhe Chen, Qin Jin, and Alexander Hauptmann. 2019. Unsupervised Bilingual Lexicon Induction from Mono-lingual Multimodal Data. In AAAI

    work page 2019

  6. [6]

    Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In EMNLP

    work page 2014

  7. [7]

    Michael Denkowski and Alon Lavie. 2014. Meteor Universal: Language Specific Translation Evaluation for Any Target Language. In Proceedings of the Ninth Workshop on Statistical Machine Translation. Association for Computational Lin- guistics, 376–380. https://doi.org/10.3115/v1/W14-3348

    doi:10.3115/v1/w14-3348 2014

  8. [8]

    Fleet, Jamie Ryan Kiros, and Sanja Fidler

    Fartash Faghri, David J. Fleet, Jamie Ryan Kiros, and Sanja Fidler. 2018. VSE++: Improving Visual-Semantic Embeddings with Hard Negatives. In spotlight pre- sentation at British Machine Vision Conference (BMVC) . https://arxiv.org/abs/ 1707.05612?context=cs.CV

    arXiv 2018

Show all 34 references
  1. [9]

    Platt, C

    Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh Srivastava, Li Deng, Piotr Dol- lÃąr, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C. Platt, C. Lawrence Zitnick, and Geoffrey Zweig. 2015. From Captions to Visual Concepts and Back. In CVPR

    2015

  2. [10]

    Yang Feng, Lin Ma, Wei Liu, and Jiebo Luo. 2018. Unsupervised Image Captioning. https://arxiv.org/abs/1811.10787

    2018 arXiv

  3. [11]

    Jiuxiang Gu, Shafiq Joty, Jianfei Cai, and Gang Wang. 2018. Unpaired Image Captioning by Language Pivoting. In Computer Vision – ECCV 2018 . Springer International Publishing, Cham, 519–535

    2018

  4. [12]

    Jiuxiang Gu, Gang Wang, Jianfei Cai, and Tsuhan Chen. 2017. An Empirical Study of Language CNN for Image Captioning. In ICCV. 10

    2017

  5. [13]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In CVPR

    2016

  6. [14]

    Julian Hitschler, Shigehiko Schamoni, and Stefan Riezler. 2016. Multimodal Pivots for Image Caption Translation. In ACL

    2016

  7. [15]

    Sepp Hochreiter and JÃijrgen Schmidhuber. 1997. Long Short-term Memory. Neural computation 9 (12 1997), 1735–80. https://doi.org/10.1162/neco.1997.9.8. 1735

    1997 doi

  8. [16]

    Xu Jia, Efstratios Gavves, Basura Fernando, and Tinne Tuytelaars. 2015. Guiding Long-Short Term Memory for Image Caption Generation. In ICCV

    2015

  9. [17]

    Yoon Kim. 2014. Convolutional Neural Networks for Sentence Classification. (2014). https://arxiv.org/abs/1408.5882

    2014 arXiv

  10. [18]

    Diederik Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimiza- tion. In ICLR

    2015

  11. [19]

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Li Fei-Fei. 2016. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations. ...

    2016 arXiv

  12. [20]

    Weiyu Lan, Xirong Li, and Jianfeng Dong. 2017. Fluency-Guided Cross-Lingual Image Captioning. In ACM Multimedia. ACM, 9. https://doi.org/10.1145/3123266. 3123366

    2017 doi

  13. [21]

    Lawrence Zitnick, and Piotr Dol- lÃąr

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dol- lÃąr. 2014. Microsoft COCO: Common Objects in Context. In ECCV

    2014

  14. [22]

    Daqing Liu, Zheng-Jun Zha, Hanwang Zhang, Yongdong Zhang, and Feng Wu

  15. [23]

    Ruotian Luo, Brian Price, Scott Cohen, and Gregory Shakhnarovich. 2018. Dis- criminability objective for training descriptive captions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2018

  16. [24]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL)

    2002

  17. [25]

    Rennie, Etienne Marcheret, and Youssef Mroueh et al

    Steven J. Rennie, Etienne Marcheret, and Youssef Mroueh et al. 2017. Self-Critical Sequence Training for Image Captioning. In CVPR

    2017

  18. [26]

    Satoshi Tsutsui and David Crandall. 2017. Using Artificial Tokens to Control Languages for Multilingual Image Caption Generation. (2017)

    2017

  19. [27]

    Lawrence Zitnick, and Devi Parikh

    Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based Image Description Evaluation. In CVPR

    2015

  20. [28]

    Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and Tell: A Neural Image Caption Generator. In CVPR. IEEE. https://doi.org/10. 1109/CVPR.2015.7298935

    2015

  21. [29]

    Jiahong Wu, He Zheng, Bo Zhao, Yixin Li, Baoming Yan, Rui Liang, Wenjia Wang, Shipei Zhou, Guosen Lin, Yanwei Fu, Yizhou Wang, and Yonggang Wang. 2017. AI Challenger : A Large-scale Dataset for Going Deeper in Image Understanding. CoRR abs/1711.06475 (2017)

    2017 arXiv

  22. [30]

    Jing Shao Dapeng Chen Xiaogang Wang Xihui Liu, Hongsheng Li. 2018. Show, Tell and Discriminate: Image Captioning by Self-retrieval with Partially Labeled Data. In ECCV

    2018

  23. [31]

    Courville, Ruslan Salakhutdinov, Richard S

    Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. abs/1502.03044 (2015). http://dblp.uni-trier.de/db/journals/corr/c...

    2015 arXiv

  24. [32]

    Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. 2016. Image Captioning with Semantic Attention. In CVPR

    2016

  25. [33]

    Wendong Zhang, Bingbing Ni, Yichao Yan, Jingwei Xu, and Xiaokang Yang. 2017. Depth Structure Preserving Scene Image Generation. In ACM Multimedia. ACM

    2017

  26. [2018]

    In ACM Multimedia

    Context-Aware Visual Policy Network for Sequence-Level Image Caption- ing. In ACM Multimedia. ACM, 9. https://doi.org/10.1145/3240508.3240632

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.