Pith. sign in

REVIEW 4 major objections 5 minor 49 references

A Semantics-Assisted Video Captioning Model Trained with Scheduled Sampling

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Three training fixes—semantic tag selection, scheduled sampling, and a length-modulated loss—lift video captioning to top reported scores on two benchmarks.

desk verdict A legitimate system paper with informative ablations, but the headline SOTA margin is mostly attributable to features and the semantic module, not the proposed scheduled sampling; the MSR-VTT result silently switches to argmax. read the letter →

arxiv 1909.00121 v3 pith:3QNO75EI submitted 2019-08-31 cs.CV cs.CL

Haoran Chen , Ke Lin , Alexander Maye , Jianming Li , Xiaolin Hu This is my paper · ORCID
classification cs.CVcs.CL
keywords videocaptioningscheduledsamplingsemanticdetectionnetworksentence-length-modulatedlossSCNLSTMmultinomialYouTube2TextMSR-VTT
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that three commonplace weaknesses in video captioning models—weak semantic features, the teacher-forcing gap between training and inference, and a loss that favors short captions—can each be fixed with a simple, modular change, and that together the fixes produce captions that score higher than previously published models on the YouTube2Text benchmark and higher than the previous best model on MSR-VTT on every metric. First, semantic features are produced by a multi-label classifier and selected by mean average precision, so the captioning decoder is fed tags that actually match the video. Second, scheduled sampling with random draws from the output distribution gradually turns training from teacher-guided to self-guided, smoothing the train/inference mismatch. Third, the log-likelihood loss is divided by sentence length raised to a power beta=0.7, countering the bias toward short captions without the redundancy seen at beta=1. The paper reports that its model outperforms all compared methods on all four metrics on YouTube2Text and beats ORG-TRL on all metrics on MSR-VTT.

What carries the argument

The load-bearing object is the SCN (Semantic Compositional Network) LSTM decoder, a variant of LSTM whose input, video feature, and hidden state are each modulated by element-wise multiplication with a semantic feature vector before entering the gates. Around it, three mechanisms carry the argument: a semantic detection network, a multi-label classifier trained on the K most frequent nouns, verbs, and adjectives, whose outputs are graded by mAP so that the best input feature set (ECO plus ResNeXt) and SDN scale are chosen; scheduled sampling with a multinomial draw from the decoder's output distribution, with probability $\epsilon$ growing per epoch, so training gradually stops using ground-truth words; and a loss divided by $L^\beta$ with $\beta=0.7$, which upweights long captions during training without beam search. The overall score normalizes BLEU-4, CIDEr, METEOR, and ROUGE-L by the best value in each column and averages them, and it is the criterion used for checkpoint selection.

What would settle it

Re-running the same three components under the exact feature extractors of ORG-TRL, or re-evaluating ORG-TRL with ECO and ResNeXt features and the same vocabulary, and finding that the reported margins shrink or reverse on either benchmark would falsify the central claim. Equivalently, training the model with $\beta=0$ and without scheduled sampling on the same features and showing that the gap over prior models disappears would indicate the three components are not the cause of the reported gains.

Watch

Extended reading notes

Core claim

The central claim is that a semantics-assisted SCN LSTM decoder, trained with scheduled multinomial sampling and a sentence-length-modulated loss, sets a new state of the art in video captioning. On YouTube2Text the model reports BLEU-4 62.4, CIDEr 109.7, METEOR 39.0, and ROUGE-L 77.0, with relative gains over ORG-TRL of 14.9%, 15.2%, 7.1%, and 4.2%; on MSR-VTT it reports 45.8, 53.2, 29.3, and 63.6, again above ORG-TRL on every metric. The authors attribute the result to a chain: using mAP to choose semantic inputs and SDN scale yields semantic features that measurably match video content; feeding those features through SCN lets the decoder condition each gate on the semantics; and the two training modifications let the model explore more of the state space while avoiding degenerate short outputs. They also claim the improvements are individually visible: higher-mAP semantic features improve captioning scores, and beta=0.7 is the length-normalization sweet spot between short captions and duplicated words.

Load-bearing premise

The result stands on the comparability of single-run test scores across papers that use different visual features, decoders, and training setups, together with the assumption that hyperparameters and the checkpoint chosen by the validation overall score transfer to the test set.

Editorial extensions

If this is right

  • Video captioning models do not need attention or reinforcement learning to beat the then-current state of the art; a standard SCN decoder with better semantic inputs and two training changes suffices on YouTube2Text and MSR-VTT.
  • Semantic feature quality, measured by mAP, transfers directly to caption quality: the model with the highest-mAP semantic features scores best, so evaluating and improving the semantic tagger is a reliable route to better captions.
  • Length normalization belongs in training, not only in decoding: dividing the loss by $L^\beta$ lengthens captions toward human length and avoids the repetition that appears at $\beta=1$.
  • Multinomial scheduled sampling acts as a regularizer and state-space explorer; it outperforms teacher forcing and arg-max sampling on YouTube2Text, and arg-max sampling on MSR-VTT, depending on dataset.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same mAP-guided semantic selection recipe could be applied to object detectors or scene-graph taggers, not just nouns, verbs, and adjectives, and would likely yield further captioning gains because the SDN input choice is currently made empirically rather than from a theory of what tag sets encode.
  • Editorial inference: because the overall score normalizes by the best number in each column, the reported large margin depends on which models are in the table; including more recent models would change the normalization and could shrink or shift the measured advantage.
  • Editorial inference: the claim that multinomial sampling explores the RNN state space more extensively is plausible but not directly measured; a testable extension is to track hidden-state diversity or effective sample size during scheduled sampling and correlate it with final caption scores.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes three modifications to a semantics-assisted video captioning model built on the SCN decoder: (i) a semantic detection network (SDN) whose input features and scale are selected using a mean-average-precision (mAP) criterion, (ii) a scheduled-sampling training strategy that gradually moves from teacher forcing to sampling words from the model's own output distribution, and (iii) a sentence-length-modulated loss function with a tunable exponent beta. The model is evaluated on YouTube2Text and MSR-VTT, where it is claimed to outperform all compared methods on YouTube2Text and to be competitive with the best previous method on MSR-VTT. Ablations in Section V examine the contribution of each component, while the final results are reported as single test-set numbers without error bars.

Significance. If the reported results were reproducible with proper controls, the paper would offer a useful practical recipe for video captioning: mAP-guided selection of semantic features, scheduled sampling, and length-modulated training are all simple and portable ideas, and the authors have released code. The paper also provides clear ablations showing that better semantic features (higher mAP) are associated with better captioning performance, and that the length-normalized loss mitigates the known tendency to generate short captions. However, the headline claims rest on single-run comparisons against baselines that use different visual features and decoding protocols, and on an inconsistency between the sampling procedure described in Algorithm 2 and the configuration that produced the MSR-VTT results. These issues need to be resolved before the contributions can be assessed reliably.

major comments (4)
  1. [§IV.D.1, Tables I and V] The claim in Section IV.D.1 that 'Our method outperforms all the other methods on all the metrics by a large margin' is not established by the presented evidence. The comparisons in Table I mix different visual feature extractors, decoding settings, and training procedures, so the margin over ORG-TRL (BLEU-4 54.3 vs 62.4, CIDEr 95.2 vs 109.7) may be largely due to the ResNeXt+ECO features and the SDN rather than the proposed training method. Table V shows that Teacher Forcing with the same visual features, same SDN, and length-modulated loss reaches BLEU-4=61.93 and CIDEr=108.56, only about 0.4 BLEU-4 and 1.1 CIDEr below the full scheduled-sampling model, and no standard deviations or multiple-seed results are reported anywhere. The authors should report variance over multiple runs and a controlled comparison in which the training strategy is the only difference.
  2. [§IV.D.2, Tables II and VI] The MSR-VTT row labeled 'Our model' in Table II (BLEU-4=45.8, CIDEr=53.2, METEOR=29.3, ROUGE-L=63.6) matches the argmax row in Table VI (45.83/53.16/29.28/63.64), not the multinomial sampling described in Algorithm 2 and Section III.B. Table VI itself shows that multinomial sampling on MSR-VTT is worse than argmax on all four metrics and worse than Teacher Forcing on CIDEr (51.77 vs 50.25; note Teacher Forcing CIDEr is 50.25, multinomial is 51.77, so multinomial is better than Teacher Forcing on CIDEr). The reported headline result is therefore a configuration selected between sampling strategies, not the method emphasized in the paper. The authors must state explicitly which training and inference protocol produced Table II, justify the selection procedure, and report both configurations consistently.
  3. [§IV.B and §IV.C, Eq. (16)] Hyperparameters and the final checkpoint are selected on the validation set using the overall score in Eq. (16), and only single runs are reported. Eq. (16) is not a standard benchmark metric: it normalizes each metric by the best value among the compared models, so the resulting score depends on the arbitrary set of models included and provides no calibration. Because the choice of beta=0.7, the epsilon schedule, the semantic tag vocabulary size, and the switch from multinomial to argmax on MSR-VTT are all determined through this nonstandard score, the reported test numbers are a selected configuration. The paper should provide sensitivity analyses for these choices and ideally multiple independent runs with standard deviations.
  4. [§V.A, Tables III and IV] The evidence that higher semantic mAP improves captioning performance consists of only three single-run data points per dataset. More importantly, the three mAP values come from SDNs trained on different input features (ResNeXt only, ECO only, or both, as shown in Figures 2 and 3), so the improvement in captioning metrics cannot be attributed solely to the 'adequate complexity' of the SDN or to the mAP-based selection criterion; it may reflect the type of visual features used to train the SDN. A controlled experiment that varies SDN capacity or semantic tag quality while keeping the visual features fixed would be needed to support the first contribution.
minor comments (5)
  1. [Algorithm 2] Algorithm 2 contains a redundant 't←t+1' inside the for loop 'for t = 1 to STEP', which should be removed for clarity.
  2. [§II.C] The paper cites Huszar's result on scheduled sampling but does not discuss its implication that scheduled sampling can be inconsistent with maximum-likelihood training; a brief response to this known criticism would be appropriate.
  3. [Abstract and Conclusion] The abstract says the model 'achieves better results' on YouTube2Text and is 'competitive' on MSR-VTT, while Section IV.D.1 says it outperforms all methods 'by a large margin' and the conclusion says it is 'superior' on YouTube2Text; these characterizations should be harmonized to avoid overclaiming.
  4. [Eq. (16)] The denominator values top1(·) used in Eq. (16) are never reported, so the reader cannot reproduce the overall scores in Tables I and II even though the metric values are listed.
  5. [Figure 4] The caption of Figure 4 lists a Greek-letter reference that appears to be a leftover artifact; also, the examples would be easier to evaluate if the corresponding metric values were included.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical benchmark results built from standard training modifications, not derivations that reduce to their own inputs.

full rationale

The paper makes three methodological proposals — mAP-based semantic feature selection, scheduled sampling, and a sentence-length-modulated loss — and evaluates them on YouTube2Text and MSR-VTT. None of these claims is derived from the evaluation metrics in a way that would make the comparison tautological. The custom overall score in Eq. (16) is a normalized average of standard BLEU-4, CIDEr, METEOR and ROUGE-L scores; choosing the model with the highest such score is ordinary model selection, not a constructed prediction. The semantic tags are produced by a separately trained multi-label classifier with ground-truth tag labels, and the mAP values in Tables III and IV are independent measures of that classifier's quality, not outputs of the captioning model. The scheduled sampling and length-modulated loss are standard training objectives whose effects are separately ablated in Tables V-VII; even though the MSR-VTT numbers in Table II coincide with the argmax row of Table VI rather than the multinomial algorithm emphasized in Section III.B, that is a reporting and attribution inconsistency about which configuration was used, not a circular reduction in which an input is renamed as a prediction. The decoder is the previously published SCN, cited as [2] from other authors, so no self-citation chain is load-bearing. All load-bearing evidence is external test-set performance against standard benchmarks, so the correct circularity finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No invented entities. The central result rests on standard supervised learning assumptions and on several unstated experimental choices (tag vocabulary size, epsilon schedule, beta) tuned on validation.

free parameters (4)
  • beta (length-modulation exponent) = 0.7
    Equation (13); selected on validation sets. Beta=0.7 balances caption length and redundancy; beta=0 is standard log loss and beta=1 over-lengthens.
  • epsilon schedule coefficient = 0.008 per epoch
    Section IV.C: sample probability epsilon = e_p * 0.008 at epoch e_p. Hand-set schedule for scheduled sampling; no sensitivity analysis.
  • semantic tag vocabulary size K = not reported
    Section III.A.1: 'manually select the K most common and meaningful words' from training and validation; K determines SDN output dimension and is not stated.
  • initial learning rates = 2e-4 (MSVD), 4e-4 (MSR-VTT)
    Section IV.C; per-dataset hyperparameter.
assumptions (4)
  • domain assumption ResNeXt and ECO pretrained visual features are sufficient and appropriate inputs for both semantic detection and caption decoding.
    Section IV.C uses ImageNet pretrained ResNeXt and Kinetics pretrained ECO without fine-tuning; the whole model depends on these features being informative.
  • domain assumption A fixed dictionary with <unk> substitution and <eos> token adequately represents the captioning output space.
    Section IV.A tokenizes training/validation captions and replaces rare words with <unk>.
  • domain assumption mAP over manually selected tags is a valid proxy for semantic feature quality and transfers to captioning quality.
    Section V.A uses mAP to choose input features and then assumes higher mAP semantic features improve downstream captions.
  • standard math Standard backpropagation with Adam and cross-entropy loss optimizes the model as expected.
    Section IV.C; no alternative optimization theory is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Semantics-Assisted Video Captioning Model Trained with Scheduled Sampling." pith.science (2026). https://pith.science/paper/3QNO75EI

@misc{pith2026190900121,
  author       = {Pith},
  title        = {Pith review of: A Semantics-Assisted Video Captioning Model Trained with Scheduled Sampling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3QNO75EI}},
  note         = {Machine review of arXiv:1909.00121}
}
read the original abstract

Given the features of a video, recurrent neural networks can be used to automatically generate a caption for the video. Existing methods for video captioning have at least three limitations. First, semantic information has been widely applied to boost the performance of video captioning models, but existing networks often fail to provide meaningful semantic features. Second, the Teacher Forcing algorithm is often utilized to optimize video captioning models, but during training and inference, different strategies are applied to guide word generation, leading to poor performance. Third, current video captioning models are prone to generate relatively short captions that express video contents inappropriately. Toward resolving these three problems, we suggest three corresponding improvements. First of all, we propose a metric to compare the quality of semantic features, and utilize appropriate features as input for a semantic detection network (SDN) with adequate complexity in order to generate meaningful semantic features for videos. Then, we apply a scheduled sampling strategy that gradually transfers the training phase from a teacher-guided manner toward a more self-teaching manner. Finally, the ordinary logarithm probability loss function is leveraged by sentence length so that the inclination of generating short sentences is alleviated. Our model achieves better results than previous models on the YouTube2Text dataset and is competitive with the previous best model on the MSR-VTT dataset.

Figures

Figures reproduced from arXiv: 1909.00121 by the authors.

Figure 1
Figure 1. Overall framework of our model. A 3D ConvNet, a 2D ConvNet and a Semantic Detection Net (SDN) constitute the encoder component of our model. [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. The quality of semantic features predicted with different sets of input [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. The quality of semantic features predicted with different sets of input [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Examples of machine-generated captions and human annotations(GT). [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 35 canonical work pages

  1. [1]

    Jointly modeling embedding and translation to bridge video and language,

    Y . Pan, T. Mei, T. Yao, H. Li, and Y . Rui, “Jointly modeling embedding and translation to bridge video and language,” in IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV , USA, June 27-30, 2016, 2016, pp. 4594–4602. SEMANTICS-ASSISTED VIDEO CAPTIONING 9 β = 0: a woman is mixing a bowl β = 0.7: a woman is mixing a bowl β = 1: a p...

    work page 2016

  2. [2]

    Semantic compositional networks for visual captioning,

    Z. Gan, C. Gan, X. He, Y . Pu, K. Tran, J. Gao, L. Carin, and L. Deng, “Semantic compositional networks for visual captioning,” in IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, July 21-26, 2017, 2017, pp. 1141–1150. [Online]. Available: https://doi.org/10.1109/CVPR.2017.127

    work page doi:10.1109/cvpr.2017.127 2017

  3. [3]

    Video captioning with attention-based lstm and semantic con- sistency,

    L. Gao, Z. Guo, H. Zhang, X. Xu, and H. T. Shen, “Video captioning with attention-based lstm and semantic con- sistency,” IEEE Transactions on Multimedia , vol. 19, no. 9, pp. 2045–2055, 2017

    work page 2017

  4. [4]

    Reinforced video caption- ing with entailment rewards,

    R. Pasunuru and M. Bansal, “Reinforced video caption- ing with entailment rewards,” in EMNLP, Copenhagen, Denmark, September 9-11, 2017 , 2017, pp. 979–985

    work page 2017

  5. [5]

    Sequence to sequence - video to text,

    S. Venugopalan, M. Rohrbach, J. Donahue, R. J. Mooney, T. Darrell, and K. Saenko, “Sequence to sequence - video to text,” in ICCV, Santiago, Chile, December 7-13, 2015, 2015, pp. 4534–4542

    work page 2015

  6. [6]

    Long-term recurrent convolutional networks for visual recognition and description,

    J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, T. Darrell, and K. Saenko, “Long-term recurrent convolutional networks for visual recognition and description,” in IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, June 7-12, 2015 , 2015, pp. 2625–2634

    work page 2015

  7. [7]

    Sched- uled sampling for sequence prediction with recurrent neural networks,

    S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer, “Sched- uled sampling for sequence prediction with recurrent neural networks,” in Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems, December 7-12, 2015, Montreal, Quebec, Canada , 2015, pp. 1171–1179

    work page 2015

  8. [8]

    Learning phrase representations using RNN encoder-decoder for statistical machine translation,

    K. Cho, B. van Merrienboer, C ¸ . G¨ulc ¸ehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y . Bengio, “Learning phrase representations using RNN encoder-decoder for statistical machine translation,” in EMNLP, October 25- 29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL , 2014, pp. 1724–1734

    work page 2014

Show all 49 references
  1. [9]

    Show and tell: A neural image caption generator,

    O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” in IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, June 7-12, 2015 , 2015, pp. 3156– 3164

    2015

  2. [10]

    Ex- plain images with multimodal recurrent neural networks,

    J. Mao, W. Xu, Y . Yang, J. Wang, and A. L. Yuille, “Ex- plain images with multimodal recurrent neural networks,” CoRR, vol. abs/1410.1090, 2014

    2014 arXiv

  3. [11]

    Neural machine translation by jointly learning to align and translate,

    D. Bahdanau, K. Cho, and Y . Bengio, “Neural machine translation by jointly learning to align and translate,” in 3rd International Conference on Learning Representations, San Diego, CA, USA, May 7-9, 2015 ,

    2015

  4. [12]

    Multiple object recognition with visual attention,

    J. Ba, V . Mnih, and K. Kavukcuoglu, “Multiple object recognition with visual attention,” in 3rd International Conference on Learning Representations, San Diego, CA, USA, May 7-9, 2015 , 2015. [Online]. Available: http://arxiv.org/abs/1412.7755

    2015 arXiv

  5. [13]

    Image captioning with semantic attention,

    Q. You, H. Jin, Z. Wang, C. Fang, and J. Luo, “Image captioning with semantic attention,” in IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV , USA, June 27-30, 2016 , 2016, pp. 4651–4659. [Online]. Available: https://doi.org/10.1109/CVPR.2016. 503

    2016 doi

  6. [14]

    Bottom-up and top-down attention for image captioning and visual question answering,

    P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, SEMANTICS-ASSISTED VIDEO CAPTIONING 10 S. Gould, and L. Zhang, “Bottom-up and top-down attention for image captioning and visual question answering,” in IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake ...

    2018

  7. [15]

    Self-critical sequence training for image captioning,

    S. J. Rennie, E. Marcheret, Y . Mroueh, J. Ross, and V . Goel, “Self-critical sequence training for image captioning,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 , 2017, pp. 1179–1195. [Online]. Available:...

    2017 doi

  8. [16]

    Exploring visual rela- tionship for image captioning,

    T. Yao, Y . Pan, Y . Li, and T. Mei, “Exploring visual rela- tionship for image captioning,” inComputer Vision - 15th European Conference, Munich, Germany, September 8- 14, 2018, Proceedings, Part XIV , 2018, pp. 711–727

    2018

  9. [17]

    Multimodal trans- former with multi-view visual representation for image captioning,

    J. Yu, J. Li, Z. Yu, and Q. Huang, “Multimodal trans- former with multi-view visual representation for image captioning,” IEEE Transactions on Circuits and Systems for Video Technology, pp. 1–1, 2019

    2019

  10. [18]

    Meshed-Memory Transformer for Image Captioning,

    M. Cornia, M. Stefanini, L. Baraldi, and R. Cucchiara, “Meshed-Memory Transformer for Image Captioning,” arXiv e-prints, Dec. 2019

    2019

  11. [19]

    Controllable video captioning with pos se- quence guidance based on gated fusion network,

    B. Wang, L. Ma, W. Zhang, W. Jiang, J. Wang, and W. Liu, “Controllable video captioning with pos se- quence guidance based on gated fusion network,” in The IEEE International Conference on Computer Vision (ICCV), October 2019

    2019

  12. [20]

    Memory-attended recurrent network for video captioning,

    W. Pei, J. Zhang, X. Wang, L. Ke, X. Shen, and Y .- W. Tai, “Memory-attended recurrent network for video captioning,” inThe IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019

    2019

  13. [21]

    Joint syntax representation learning and visual cue translation for video captioning,

    J. Hou, X. Wu, W. Zhao, J. Luo, and Y . Jia, “Joint syntax representation learning and visual cue translation for video captioning,” in The IEEE International Conference on Computer Vision (ICCV) , October 2019

    2019

  14. [22]

    Spatio-temporal dynamics and semantic attribute en- riched visual encoding for video captioning,

    N. Aafaq, N. Akhtar, W. Liu, S. Z. Gilani, and A. Mian, “Spatio-temporal dynamics and semantic attribute en- riched visual encoding for video captioning,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019

    2019

  15. [23]

    Syntax-aware action targeting for video captioning,

    Q. Zheng, C. Wang, and D. Tao, “Syntax-aware action targeting for video captioning,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020

    2020

  16. [24]

    Video paragraph captioning using hierarchical recurrent neural networks,

    H. Yu, J. Wang, Z. Huang, Y . Yang, and W. Xu, “Video paragraph captioning using hierarchical recurrent neural networks,” in IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV , USA, June 27-30, 2016, 2016, pp. 4584–4593

    2016

  17. [25]

    Show, attend and tell: Neural image caption generation with visual attention,

    K. Xu, J. Ba, R. Kiros, K. Cho, A. C. Courville, R. Salakhutdinov, R. S. Zemel, and Y . Bengio, “Show, attend and tell: Neural image caption generation with visual attention,” in Proceedings of the 32nd Interna- tional Conference on Machine Learning, Lille, France, 6-11 July 2...

    2015

  18. [26]

    Top-down visual saliency guided by captions,

    V . Ramanishka, A. Das, J. Zhang, and K. Saenko, “Top-down visual saliency guided by captions,” in IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, July 21-26, 2017 , 2017, pp. 3135–3144. [Online]. Available: https://doi. org/10.1109/CVPR.2017.334

    2017 doi

  19. [27]

    Less is more: Picking informative frames for video captioning,

    Y . Chen, S. Wang, W. Zhang, and Q. Huang, “Less is more: Picking informative frames for video captioning,” in Computer Vision - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XIII , 2018, pp. 367–384. [Online]. Available: https://doi.org/10...

    2018 doi

  20. [28]

    Watch, lis- ten, and describe: Globally and locally aligned cross- modal attentions for video captioning,

    X. Wang, Y . Wang, and W. Y . Wang, “Watch, lis- ten, and describe: Globally and locally aligned cross- modal attentions for video captioning,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Languag...

    2018

  21. [29]

    Multi-task video captioning with video and entailment generation,

    R. Pasunuru and M. Bansal, “Multi-task video captioning with video and entailment generation,” in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada, July 30, 2017 - August 4, Volume 1: Long Papers , 2017, pp. 1273–1283. ...

    2017

  22. [30]

    Videobert: A joint model for video and language representation learning,

    C. Sun, A. Myers, C. V ondrick, K. Murphy, and C. Schmid, “Videobert: A joint model for video and language representation learning,” in IEEE/CVF Inter- national Conference on Computer Vision, Seoul, Korea (South), October 27 - November 2, 2019 . IEEE, 2019, pp. 7463–7472

    2019

  23. [31]

    BERT: pre-training of deep bidirectional transformers for lan- guage understanding,

    J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for lan- guage understanding,” CoRR, vol. abs/1810.04805, 2018. [Online]. Available: http://arxiv.org/abs/1810.04805

    2018 arXiv

  24. [32]

    Learning to compose topic-aware mixture of experts for zero-shot video captioning,

    X. Wang, J. Wu, D. Zhang, Y . Su, and W. Y . Wang, “Learning to compose topic-aware mixture of experts for zero-shot video captioning,” in The Thirty-Third AAAI Conference on Artificial Intelligence, The Thirty- First Innovative Applications of Artificial Intelligence Conference...

    2019

  25. [33]

    Spatio-temporal graph for video captioning with knowledge distillation,

    B. Pan, H. Cai, D.-A. Huang, K.-H. Lee, A. Gaidon, E. Adeli, and J. C. Niebles, “Spatio-temporal graph for video captioning with knowledge distillation,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020

    2020

  26. [34]

    A learning algorithm for continually running fully recurrent neural networks,

    R. J. Williams and D. Zipser, “A learning algorithm for continually running fully recurrent neural networks,” Neural Computation, vol. 1, no. 2, pp. 270–280, 1989

    1989

  27. [35]

    How (not) to train your generative model: Scheduled sampling, likelihood, adversary?

    F. Huszar, “How (not) to train your generative model: Scheduled sampling, likelihood, adversary?” CoRR, vol. abs/1511.05101, 2015. SEMANTICS-ASSISTED VIDEO CAPTIONING 11

    2015 arXiv

  28. [36]

    Professor forcing: A new algorithm for training recurrent networks,

    A. Goyal, A. Lamb, Y . Zhang, S. Zhang, A. C. Courville, and Y . Bengio, “Professor forcing: A new algorithm for training recurrent networks,” in Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems, December 5- 10, 2...

    2016

  29. [37]

    Object relational graph with teacher-recommended learning for video captioning,

    Z. Zhang, Y . Shi, C. Yuan, B. Li, P. Wang, W. Hu, and Z.-J. Zha, “Object relational graph with teacher-recommended learning for video captioning,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020

    2020

  30. [38]

    Simple statistical gradient-following al- gorithms for connectionist reinforcement learning,

    R. J. Williams, “Simple statistical gradient-following al- gorithms for connectionist reinforcement learning,” Ma- chine Learning, vol. 8, no. 3, pp. 229–256, 1992

    1992

  31. [39]

    Finding structure in time,

    J. L. Elman, “Finding structure in time,” Cognitive Sci- ence, vol. 14, no. 2, pp. 179–211, 1990

    1990

  32. [40]

    Long short-term memory,

    S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735– 1780, 1997

    1997

  33. [41]

    Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation,

    Y . Wu, M. Schuster, Z. Chen, Q. V . Le, M. Norouzi, W. Macherey, M. Krikun, Y . Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, Ł. Kaiser, S. Gouws, Y . Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Ru...

    2016

  34. [42]

    Youtube2text: Recognizing and describing arbitrary ac- tivities using semantic hierarchies and zero-shot recog- nition,

    S. Guadarrama, N. Krishnamoorthy, G. Malkarnenkar, S. Venugopalan, R. J. Mooney, T. Darrell, and K. Saenko, “Youtube2text: Recognizing and describing arbitrary ac- tivities using semantic hierarchies and zero-shot recog- nition,” in IEEE International Conference on Computer Vi...

    2013

  35. [43]

    Collecting highly parallel data for paraphrase evaluation,

    D. L. Chen and W. B. Dolan, “Collecting highly parallel data for paraphrase evaluation,” in Proceedings of the 49th Annual Meeting of the Association for Computa- tional Linguistics (ACL-2011), Portland, OR, June 2011

    2011

  36. [44]

    MSR-VTT: A large video description dataset for bridging video and language,

    J. Xu, T. Mei, T. Yao, and Y . Rui, “MSR-VTT: A large video description dataset for bridging video and language,” in IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV , USA, June 27-30, 2016, 2016, pp. 5288–5296

    2016

  37. [45]

    Aggregated residual transformations for deep neural networks,

    S. Xie, R. B. Girshick, P. Doll ´ar, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, July 21-26, 2017, 2017, pp. 5987–5995. [Online]. Available: https://doi.org/...

    2017 doi

  38. [46]

    ECO: efficient convolutional network for online video understanding,

    M. Zolfaghari, K. Singh, and T. Brox, “ECO: efficient convolutional network for online video understanding,” in Computer Vision - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part II , 2018, pp. 713–730. [Online]. Available: https://doi.org/10.1...

    2018 doi

  39. [47]

    Sibnet: Sibling convolutional encoder for video captioning,

    S. Liu, Z. Ren, and J. Yuan, “Sibnet: Sibling convolutional encoder for video captioning,” in ACM Multimedia Conference on Multimedia Conference, Seoul, Republic of Korea, October 22-26, 2018 , 2018, pp. 1425–1434. [Online]. Available: https://doi.org/10. 1145/3240508.3240667

    2018

  40. [48]

    Multi-label classification: An overview,

    G. Tsoumakas and I. Katakis, “Multi-label classification: An overview,” International Journal of Data Warehous- ing and Mining , vol. 3, no. 3, pp. 1–13, 2007

    2007

  41. [2015]

    Available: http://arxiv.org/abs/1409.0473

    [Online]. Available: http://arxiv.org/abs/1409.0473

    arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.