Pith. sign in

REVIEW 4 major objections 4 minor 64 references

Controllable Video Captioning with POS Sequence Guidance Based on Gated Fusion Network

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Video captioning improves by first predicting the sentence's part-of-speech skeleton, then gating the decoder with it.

desk verdict A genuinely useful fusion mechanism and a sensible transfer of POS guidance to video, but the POS benefit and controllability claims are under-substantiated; worth a revision, not a rejection. read the letter →

arxiv 1908.10072 v1 pith:QGLE5D7G submitted 2019-08-27 cs.CV

Bairui Wang , Lin Ma , Wei Zhang , Wenhao Jiang , Jingwen Wang , Wei Liu This is my paper · ORCID
classification cs.CV
keywords videocaptioningpart-of-speechguidancecross-gatingfusionsyntaxcontrolsequencegenerationMSR-VTTMSVDgatednetwork
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a video captioning model can be improved by first predicting the part-of-speech (POS) sequence that the final caption should follow, and then using that predicted syntactic skeleton to guide word-by-word decoding. It proposes a gated fusion network with a cross-gating block that combines content and motion features of a video, rather than simply concatenating them. On MSR-VTT and MSVD, the model reports higher captioning scores than concatenation baselines and several prior systems, and it shows that manually editing the predicted POS tags changes the syntax of the generated sentence. If true, POS sequence guidance gives a simple handle for controlling caption structure and increasing output diversity without extra supervision.

What carries the argument

The load-bearing components are the cross-gating (CG) block and the POS-sequence-guided decoder. The CG block updates each feature stream using a gated, residual combination of the other stream: $\mathrm{Gating}(x,y)=\sigma(wx+b)y+y$, so content features are strengthened by motion information and vice versa before a fully connected layer fuses them. The POS sequence generator is an attention LSTM over the fused frame features that outputs a POS tag sequence; its final hidden state $\psi$ is the global syntactic representation. In the decoder, the word embedding of the previous word is gated against $\psi$ via the same gating function, so the global POS prior is adaptively incorporated at every step, and a second LSTM layer attends to the fused video features.

What would settle it

Train the model on MSR-VTT, then at inference replace the predicted POS sequence for each video with a randomly shuffled version and with a fixed template such as ART NOUN VERB ART NOUN; if CIDEr does not fall measurably below the model's no-POS baseline in either case, the POS guidance is not carrying the claimed causal load.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that predicting a global POS tag sequence from fused video features, then gating the decoder's word-embedding input with that global POS representation, improves video captioning accuracy and makes the caption's syntactic structure controllable. The central discovery is that syntax can act as a transferable prior: the fused video representation supports a small LSTM that generates tags like ART NOUN VERB VERB ART NOUN before any caption word is produced, and the last hidden state of that LSTM, $\psi$, is injected into the decoder through a gating operation. This lets the decoder know what kind of word to produce next, narrowing the search space, and it lets a user change a tag at inference time to force a different sentence shape with content still drawn from the video.

Load-bearing premise

The model assumes the video's fused features contain enough information to predict the global POS structure of the caption before any word is generated, and that this predicted POS signal is accurate enough to help rather than mislead the decoder.

Editorial extensions

If this is right

  • On the two benchmarks, the full model with self-critical reinforcement learning reaches state-of-the-art ROUGE-L and CIDEr on MSR-VTT and state-of-the-art scores on four metrics on MSVD, according to the paper's reported tables.
  • The ablation results indicate that cross-gating fusion alone beats simple concatenation as well as compact bilinear pooling and low-rank bilinear pooling baselines for combining content and motion features.
  • If the POS prior is as useful as claimed, the two-stage training procedure (POS generator first, then caption generator with the POS generator frozen) provides a stable syntactic prior that can be reused with different decoders.
  • A user can edit the predicted POS sequence at inference time to force syntactic changes, such as adding an adjective, producing a "there be" construction, or changing a quantity expression, while the decoder still draws content from the video.
  • Because the POS generator is a small additional LSTM on top of the fused representation, the added computational cost over a standard captioning model is modest, making the control mechanism practical for interactive use.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implicit testable extension: if the POS prior is truly causal, corrupting predicted POS tags (for example, randomly shuffling them) should degrade performance below the no-POS baseline; the paper does not report this negative control.
  • The controllability examples point toward user-specified caption style in interactive video description, such as requesting an adjective or a specific sentence frame; the paper only demonstrates a few hand-edited cases.
  • The global POS representation may transfer to related generation tasks like dense video captioning or visual storytelling, where controlling sentence structure across multiple sentences matters, though the paper does not test this.
  • The POS tagger used for ground-truth labeling was trained on written text, so the predicted syntax may match some video caption styles better than others; performance could vary by dataset register.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a video captioning model that fuses two or more video features (e.g., appearance and motion) via a cross-gating mechanism, and additionally introduces a part-of-speech (POS) sequence generator that predicts the POS tags of the caption to be generated from the fused video representation. The last hidden state of the POS generator is used as a global syntactic feature ψ that gates the word embeddings in a two-layer LSTM decoder. The authors also describe a controllability procedure in which a user modifies the predicted POS tags at inference to bias the syntactic structure of the generated caption. The model is evaluated on MSR-VTT and MSVD, with an additional ablation on ActivityNet 1.3 in the supplementary, and the authors report improvements over several baselines and, on some metrics, state-of-the-art results.

Significance. The cross-gating fusion component is simple and appears effective across ablations on two datasets (Table 3 and Supplementary Table 1), and the code is released, which are concrete strengths. If the POS guidance and controllability claims were properly validated, the idea of conditioning decoding on a global syntactic prior would be a useful contribution to video captioning. However, the current evidence does not isolate the POS information from added capacity, does not evaluate the quality of the POS predictions, and the qualitative controllability examples are internally inconsistent. The empirical claims are therefore not yet established to the standard required for publication.

major comments (4)
  1. [Section 3.4 (Training)] The training protocol computes the global POS representation ψ after the POS generator converges and before the video encoder and description generator are jointly trained. Since the video encoder is updated during the joint stage, the fused features X used by the decoder during training are computed from an encoder that differs from the one used to produce ψ, while at inference ψ is recomputed from the updated encoder. This train/test mismatch means the POS guidance is not optimized jointly with the decoder, and any observed gain from POS could be influenced by this inconsistency. The authors should either freeze the video encoder when computing ψ, jointly fine-tune the POS generator, or explicitly analyze the effect of this protocol choice.
  2. [Section 4.4, Table 3] The incremental gains of Ours over EncDec+CG are +0.1 to +0.8 CIDEr, +0.3 BLEU@4, +0.1 to +0.3 METEOR, and +0.2 to +0.6 ROUGE-L. These differences are not accompanied by error bars or significance tests and are of the same order as typical run-to-run variance. Moreover, adding the POS sequence generator and the gating mechanism increases the model capacity; no control (e.g., random POS tags, a fixed non-informative vector of the same dimension, or an auxiliary task on the same video features) is reported to isolate the contribution of POS information itself. The claim that POS guidance benefits captioning performance is therefore not established by the presented evidence.
  3. [Section 4.6, Figure 6] The controllability examples do not demonstrate that the generated caption respects the edited POS sequence. In the first example, the edited sequence ART ADJ NOUN VERB VERB ART NOUN is paired with the output "A man in a pink shirt is giving a presentation", whose POS tags are ART NOUN PREP ART ADJ NOUN AUX VERB ART NOUN. The second edited sequence NUM NOUN VERB VERB ART NOUN is paired with "Two teams are playing a game of rugby", tagged NUM NOUN AUX VERB ART NOUN PREP NOUN. Re-tagging the shown outputs does not reproduce the edited sequences, so the claim of syntactic control is not supported by the provided evidence. The authors should present a systematic evaluation of controllability, for example by measuring the proportion of generated tokens whose POS tags match the edited sequence, and show multiple examples with quantitative agreement.
  4. [Section 3.2, Eqs. (5)-(7)] The paper does not report any accuracy of the POS sequence generator, either on the training set or on a held-out set. Since ψ is the last hidden state of this generator and is the only carrier of POS information into the decoder (Eq. 8), the reader cannot assess whether the generator actually captures global syntactic structure. Reporting POS tag accuracy (and perhaps sequence accuracy) is necessary to support the claim that the model "well capture[s] the global syntactic structure" (Abstract). If the POS predictions are low-quality, the gating mechanism could be learning to discount them, in which case the captioning gains should not be attributed to POS guidance.
minor comments (4)
  1. [Introduction and throughout] There are several typographical errors: "approahces" in the Introduction, "excepted" for "expected" in Section 1, "motioned" for "mentioned" in Section 4.2, and "controllablity" in the Section 4.6 heading.
  2. [Eq. (7)] In the definition of the softmax weights, the denominator sums over k of exp(e_{t,i}) instead of exp(e_{t,k}); the index in the exponent should be the summation index. The same typo may need checking in Eq. (10).
  3. [Table 1 caption] The notation "Ours(C3D+M)" is confusing because "M" already denotes the motion features extracted by I3D, while C3D is also a spatiotemporal/motion-related feature; clarifying which feature acts as content and which as motion would improve readability.
  4. [Section 4.3] The comparisons with prior work mix models that use different feature sets; the authors acknowledge this, but it would be helpful to explicitly separate feature-matched comparisons from the rest, as is done in the discussion of MGSA(IR+C3D).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the POS-guidance claim rests on externally supervised training and held-out benchmark evaluation, not on a self-referential definition or a self-citation chain.

full rationale

The paper's central derivations are self-contained with respect to external supervision. The POS sequence generator is trained on ground-truth POS tags obtained by an external Stanford tagger using a standard negative log-likelihood objective (Eqs. 5, 11, 12), and the description generator is trained on human ground-truth captions (Eq. 13) plus self-critical CIDEr training against the benchmark metric. The reported results are computed on held-out test splits of MSR-VTT, MSVD, and ActivityNet using standard external metrics (BLEU, METEOR, ROUGE-L, CIDEr). No equation in the paper defines the predicted global POS representation psi in terms of the generated caption or its evaluation score, and no fitted parameter is renamed as a prediction: the ablation EncDec+CG vs. Ours adds the POS pathway as an extra learned component rather than re-fitting the evaluation quantity. The legitimate concerns raised by the skeptical analysis, such as missing POS accuracy numbers, the absence of oracle/random POS controls, unreplicated gains, and the two-stage training mismatch where psi is computed before the stage-2 encoder update, bear on empirical attribution and experimental rigor, not on circularity. Self-citations appear in the references and in mentions of prior captioning models, but they are not invoked to establish the POS-guidance mechanism or to forbid alternative explanations; no uniqueness theorem from the authors' prior work is load-bearing. Accordingly, the derivation chain does not reduce to its inputs, and the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on pretrained features, a POS tagger, standard benchmark metrics, and the RL training rule, all taken from prior work and listed as axioms above. All model parameters are learned from the training splits, and no hand-fitted constants or new physical entities are introduced.

assumptions (4)
  • domain assumption Pretrained CNN features (I3D motion, Inception-ResNet-v2 content, C3D) provide useful input representations for video captioning.
    Sec. 4.2: all experimental gains are measured on systems that start from these features.
  • domain assumption The Stanford POS tagger's labels on ground-truth sentences are a reliable supervisory signal for the POS generator.
    Sec. 4.2: the POS generator is supervised with these labels.
  • domain assumption BLEU, METEOR, ROUGE-L, and CIDEr scores on public splits capture meaningful caption quality.
    Sec. 4: the central empirical claim is based on these metrics.
  • standard math Self-critical sequence training supplies a valid gradient estimate for optimizing CIDEr.
    Supplementary Sec. 1.2, Eq. (7) in the supplementary material; this is borrowed from prior work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Controllable Video Captioning with POS Sequence Guidance Based on Gated Fusion Network." pith.science (2026). https://pith.science/paper/QGLE5D7G

@misc{pith2026190810072,
  author       = {Pith},
  title        = {Pith review of: Controllable Video Captioning with POS Sequence Guidance Based on Gated Fusion Network},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QGLE5D7G}},
  note         = {Machine review of arXiv:1908.10072}
}
read the original abstract

In this paper, we propose to guide the video caption generation with Part-of-Speech (POS) information, based on a gated fusion of multiple representations of input videos. We construct a novel gated fusion network, with one particularly designed cross-gating (CG) block, to effectively encode and fuse different types of representations, e.g., the motion and content features of an input video. One POS sequence generator relies on this fused representation to predict the global syntactic structure, which is thereafter leveraged to guide the video captioning generation and control the syntax of the generated sentence. Specifically, a gating strategy is proposed to dynamically and adaptively incorporate the global syntactic POS information into the decoder for generating each word. Experimental results on two benchmark datasets, namely MSR-VTT and MSVD, demonstrate that the proposed model can well exploit complementary information from multiple representations, resulting in improved performances. Moreover, the generated global POS information can well capture the global syntactic structure of the sentence, and thus be exploited to control the syntactic structure of the description. Such POS information not only boosts the video captioning performance but also improves the diversity of the generated captions. Our code is at: https://github.com/vsislab/Controllable_XGating.

Figures

Figures reproduced from arXiv: 1908.10072 by the authors.

Figure 1
Figure 1. The proposed model for video captioning consists of a [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The proposed model for video captioning consists of three components. The gated fusion network encodes and fuses multiple [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. An illustration of cross gating strategy in the proposed [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The architecture of the proposed description generator. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Visualization of some video captioning examples on the [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Visualization of some video captioning examples on the [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 58 canonical work pages

  1. [1]

    Meteor: An automatic metric for mt evaluation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the ACL workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization , 2005

    work page 2005

  2. [2]

    Quo vadis, action recognition? a new model and the kinetics dataset

    Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR , 2017

    work page 2017

  3. [3]

    Collecting highly parallel data for paraphrase evaluation

    David L Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In ACL , 2011

    work page 2011

  4. [4]

    Video captioning with guidance of multimodal latent topics

    Shizhe Chen, Jia Chen, Qin Jin, and Alexander Hauptmann. Video captioning with guidance of multimodal latent topics. In ACM MM , 2017

    work page 2017

  5. [5]

    Motion guided spatial attention for video captioning

    Shaoxiang Chen and Yu-Gang Jiang. Motion guided spatial attention for video captioning. In AAAI , 2019

    work page 2019

  6. [6]

    Microsoft coco captions: Data collection and evaluation server

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Doll \'a r, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv:1504.00325 , 2015

    arXiv 2015

  7. [7]

    Regularizing rnns for caption generation by reconstructing the past with the present

    Xinpeng Chen, Lin Ma, Wenhao Jiang, Jian Yao, and Wei Liu. Regularizing rnns for caption generation by reconstructing the past with the present. In CVPR , 2018

    work page 2018

  8. [8]

    Less is more: Picking informative frames for video captioning

    Yangyu Chen, Shuhui Wang, Weigang Zhang, and Qingming Huang. Less is more: Picking informative frames for video captioning. ECCV , 2018

    work page 2018

Show all 64 references
  1. [9]

    Embodied question answering

    Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answering. In CVPR , 2018

    2018

  2. [10]

    Diverse and controllable image captioning with part-of-speech guidance

    Aditya Deshpande, Jyoti Aneja, Liwei Wang, Alexander Schwing, and David A Forsyth. Diverse and controllable image captioning with part-of-speech guidance. CVPR , 2019

    2019

  3. [11]

    Long-term recurrent convolutional networks for visual recognition and description

    Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional networks for visual recognition and description. In CVPR , 2015

    2015

  4. [12]

    Unsupervised image captioning

    Yang Feng, Lin Ma, Wei Liu, and Jiebo Luo. Unsupervised image captioning. In CVPR , 2019

    2019

  5. [13]

    Multimodal compact bilinear pooling for visual question answering and visual grounding

    Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. In EMNLP , 2016

    2016

  6. [14]

    Semantic compositional networks for visual captioning

    Zhe Gan, Chuang Gan, Xiaodong He, Yunchen Pu, Kenneth Tran, Jianfeng Gao, Lawrence Carin, and Li Deng. Semantic compositional networks for visual captioning. In CVPR , 2017

    2017

  7. [15]

    Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition

    Sergio Guadarrama, Niveda Krishnamoorthy, Girish Malkarnenkar, Subhashini Venugopalan, Raymond Mooney, Trevor Darrell, and Kate Saenko. Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition. In ICCV , 2013

    2013

  8. [16]

    Image caption generation with part of speech guidance

    Xinwei He, Baoguang Shi, Xiang Bai, Gui-Song Xia, Zhaoxiang Zhang, and Weisheng Dong. Image caption generation with part of speech guidance. Pattern Recognition Letters , 119:229--237, 2019

    2019

  9. [17]

    Recurrent fusion network for image captioning

    Wenhao Jiang, Lin Ma, Yu-Gang Jiang, Wei Liu, and Tong Zhang. Recurrent fusion network for image captioning. In ECCV , 2018

    2018

  10. [18]

    Describing videos using multi-modal fusion

    Qin Jin, Jia Chen, Shizhe Chen, Yifan Xiong, and Alexander Hauptmann. Describing videos using multi-modal fusion. In ACM MM , 2016

    2016

  11. [19]

    The kinetics human action video dataset

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv:1705.06950 , 2017

    2017 arXiv

  12. [20]

    Hadamard product for low-rank bilinear pooling

    Jin-Hwa Kim, Kyoung-Woon On, Woosang Lim, Jeonghee Kim, Jung-Woo Ha, and Byoung-Tak Zhang. Hadamard product for low-rank bilinear pooling. ICLR , 2017

    2017

  13. [21]

    Natural language description of human activities from video images based on concept hierarchy of actions

    Atsuhiro Kojima, Takeshi Tamura, and Kunio Fukunaga. Natural language description of human activities from video images based on concept hierarchy of actions. International Journal of Computer Vision , 50(2):171--184, 2002

    2002

  14. [22]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out , 2004

    2004

  15. [23]

    Sibnet: Sibling convolutional encoder for video captioning

    Sheng Liu, Zhou Ren, and Junsong Yuan. Sibnet: Sibling convolutional encoder for video captioning. In ACM MM , 2018

    2018

  16. [24]

    Matching image and sentence with multi-faceted representations

    Lin Ma, Wenhao Jiang, Zequn Jie, Yu-Gang Jiang, and Wei Liu. Matching image and sentence with multi-faceted representations. IEEE Transactions on Circuits and Systems for Video Technology , 2019

    2019

  17. [25]

    Learning to answer questions from image using convolutional neural network

    Lin Ma, Zhengdong Lu, and Hang Li. Learning to answer questions from image using convolutional neural network. In AAAI , 2016

    2016

  18. [26]

    Multimodal convolutional neural networks for matching image and sentence

    Lin Ma, Zhengdong Lu, Lifeng Shang, and Hang Li. Multimodal convolutional neural networks for matching image and sentence. In ICCV , 2015

    2015

  19. [27]

    Jointly modeling embedding and translation to bridge video and language

    Yingwei Pan, Tao Mei, Ting Yao, Houqiang Li, and Yong Rui. Jointly modeling embedding and translation to bridge video and language. In CVPR , 2016

    2016

  20. [28]

    Video captioning with transferred semantic attributes

    Yingwei Pan, Ting Yao, Houqiang Li, and Tao Mei. Video captioning with transferred semantic attributes. In CVPR , 2017

    2017

  21. [29]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL , 2002

    2002

  22. [30]

    Tv-l1 optical flow estimation

    Javier S \'a nchez P \'e rez, Enric Meinhardt-Llopis, and Gabriele Facciolo. Tv-l1 optical flow estimation. Image Processing On Line , 2013:137--150, 2013

    2013

  23. [31]

    Multimodal video description

    Vasili Ramanishka, Abir Das, Dong Huk Park, Subhashini Venugopalan, Lisa Anne Hendricks, Marcus Rohrbach, and Kate Saenko. Multimodal video description. In ACM MM , 2016

    2016

  24. [32]

    Self-critical sequence training for image captioning

    Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In CVPR , 2017

    2017

  25. [33]

    Coherent multi-sentence video description with variable level of detail

    Anna Rohrbach, Marcus Rohrbach, Wei Qiu, Annemarie Friedrich, Manfred Pinkal, and Bernt Schiele. Coherent multi-sentence video description with variable level of detail. In GCPR , 2014

    2014

  26. [34]

    Translating video content to natural language descriptions

    Marcus Rohrbach, Wei Qiu, Ivan Titov, Stefan Thater, Manfred Pinkal, and Bernt Schiele. Translating video content to natural language descriptions. In ICCV , 2013

    2013

  27. [35]

    Imagenet large scale visual recognition challenge

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision , 115(3):211--252, 2015

    2015

  28. [36]

    Frame-and segment-level features and candidate pool evaluation for video caption generation

    Rakshith Shetty and Jorma Laaksonen. Frame-and segment-level features and candidate pool evaluation for video caption generation. In ACM MM , 2016

    2016

  29. [37]

    Quantization-based hashing: a general framework for scalable image and video retrieval

    Jingkuan Song, Lianli Gao, Li Liu, Xiaofeng Zhu, and Nicu Sebe. Quantization-based hashing: a general framework for scalable image and video retrieval. Pattern Recognition , 75:175--187, 2018

    2018

  30. [38]

    Inception-v4, inception-resnet and the impact of residual connections on learning

    Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In AAAI , 2017

    2017

  31. [39]

    Feature-rich part-of-speech tagging with a cyclic dependency network

    Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. Feature-rich part-of-speech tagging with a cyclic dependency network. In NAACL , 2003

    2003

  32. [40]

    Learning spatiotemporal features with 3d convolutional networks

    Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In ICCV , 2015

    2015

  33. [41]

    Cider: Consensus-based image description evaluation

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In CVPR , 2015

    2015

  34. [42]

    Sequence to sequence-video to text

    Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. Sequence to sequence-video to text. In ICCV , 2015

    2015

  35. [43]

    Translating videos to natural language using deep recurrent neural networks

    Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, and Kate Saenko. Translating videos to natural language using deep recurrent neural networks. NAACL , 2015

    2015

  36. [44]

    Hierarchical photo-scene encoder for album storytelling

    Bairui Wang, Lin Ma, Wei Zhang, Wenhao Jiang, and Feng Zhang. Hierarchical photo-scene encoder for album storytelling. In AAAI , 2019

    2019

  37. [45]

    Reconstruction network for video captioning

    Bairui Wang, Lin Ma, Wei Zhang, and Wei Liu. Reconstruction network for video captioning. In CVPR , 2018

    2018

  38. [46]

    Bidirectional attentive fusion with context gating for dense video captioning

    Jingwen Wang, Wenhao Jiang, Lin Ma, Wei Liu, and Yong Xu. Bidirectional attentive fusion with context gating for dense video captioning. In CVPR , 2018

    2018

  39. [47]

    M3: Multimodal memory modelling for video captioning

    Junbo Wang, Wei Wang, Yan Huang, Liang Wang, and Tieniu Tan. M3: Multimodal memory modelling for video captioning. In CVPR , 2018

    2018

  40. [48]

    A survey on learning to hash

    Jingdong Wang, Ting Zhang, Nicu Sebe, Jingkuang Song, and Heng Tao Shen. A survey on learning to hash. IEEE Transactions on Pattern Analysis and Machine Intelligence , 40(4):769--790, 2018

    2018

  41. [49]

    Interpretable video captioning via trajectory structured localization

    Xian Wu, Guanbin Li, Qingxing Cao, Qingge Ji, and Liang Lin. Interpretable video captioning via trajectory structured localization. In CVPR , 2018

    2018

  42. [50]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR , 2016

    2016

  43. [51]

    Learning multimodal attention lstm networks for video captioning

    Jun Xu, Ting Yao, Yongdong Zhang, and Tao Mei. Learning multimodal attention lstm networks for video captioning. In ACM MM , 2017

    2017

  44. [52]

    Jointly modeling deep video and compositional text to bridge vision and language in a unified framework

    Ran Xu, Caiming Xiong, Wei Chen, and Jason J Corso. Jointly modeling deep video and compositional text to bridge vision and language in a unified framework. In AAAI , 2015

    2015

  45. [53]

    Describing videos by exploiting temporal structure

    Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, and Aaron Courville. Describing videos by exploiting temporal structure. In ICCV , 2015

    2015

  46. [54]

    Video paragraph captioning using hierarchical recurrent neural networks

    Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, and Wei Xu. Video paragraph captioning using hierarchical recurrent neural networks. In CVPR , 2016

    2016

  47. [55]

    Adadelta: an adaptive learning rate method

    Matthew D Zeiler. Adadelta: an adaptive learning rate method. aarXiv:1212.5701 , 2012

    2012 arXiv

  48. [56]

    Reconstruct and represent video contents for captioning via reinforcement learning

    Wei Zhang, Bairui Wang, Lin Ma, and Wei Liu. Reconstruct and represent video contents for captioning via reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2019

    2019

  49. [57]

    B. G. Fabian Caba Heilbron, Victor Escorcia and J. C. Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 961--970, 2015

    2015

  50. [58]

    Pasunuru and M

    R. Pasunuru and M. Bansal. Reinforced video captioning with entailment rewards. arXiv preprint arXiv:1708.02300 , 2017

    2017 arXiv

  51. [59]

    Z. Ren, X. Wang, N. Zhang, X. Lv, and L.-J. Li. Deep reinforcement learning-based image captioning with embedding reward. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 290--298, 2017

    2017

  52. [60]

    S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel. Self-critical sequence training for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 7008--7024, 2017

    2017

  53. [61]

    R. S. Sutton, A. G. Barto, et al. Introduction to reinforcement learning , volume 135. MIT press Cambridge, 1998

    1998

  54. [62]

    Vedantam, C

    R. Vedantam, C. Lawrence Zitnick, and D. Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4566--4575, 2015

    2015

  55. [63]

    X. Wang, W. Chen, J. Wu, Y.-F. Wang, and W. Yang Wang. Video captioning via hierarchical reinforcement learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4213--4222, 2018

    2018

  56. [64]

    Zaremba and I

    W. Zaremba and I. Sutskever. Reinforcement learning neural turing machines-revised. arXiv preprint arXiv:1505.00521 , 2015

    2015 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.