RECORD
Controllable Video Captioning with an Exemplar Sentence
Computer Vision and Pattern Recognition · Submitted 2 Dec 2021
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Extensive experimental results demonstrate the effectiveness of our approach on generating syntax controllable and semantic preserved video captions.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
In this paper, we investigate a novel and challenging task, namely controllable video captioning with an exemplar sentence. Formally, given a video and a syntactically valid exemplar sentence, the task aims to generate one caption which not only describes the semantic contents of the video, but also follows the syntactic form of the given exemplar sentence. In order to tackle such an exemplar-based video captioning task, we propose a novel Syntax Modulated Caption Generator (SMCG) incorporated in an encoder-decoder-reconstructor architecture. The proposed SMCG takes video semantic representation as an input, and conditionally modulates the gates and cells of long short-term memory network with respect to the encoded syntactic information of the given exemplar sentence. Therefore, SMCG is able to control the states for word prediction and achieve the syntax customized caption generation. We conduct experiments by collecting auxiliary exemplar sentences for two public video captioning datasets. Extensive experimental results demonstrate the effectiveness of our approach on generating syntax controllable and semantic preserved video captions. By providing different exemplar sentences, our approach is capable of producing different captions with various syntactic structures, thus indicating a promising way to strengthen the diversity of video captioning.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | In this paper, we investigate a novel and challenging task, namely controllable video captioning with an exemplar sentence. | contribution 0.64 | 0.60 | 0.03 | asserted 0.81 |
| s1 | Formally, given a video and a syntactically valid exemplar sentence, the task aims to generate one caption which not only describes the semantic contents of the video, but also follows the syntactic form of the given exemplar sentence. | background 0.95 | 0.08 | 0.03 | asserted 0.98 |
| s2 | In order to tackle such an exemplar-based video captioning task, we propose a novel Syntax Modulated Caption Generator (SMCG) incorporated in an encoder-decoder-reconstructor architecture. | method 0.60 | 0.89 | 0.05 | asserted 1.00 |
| s3 | The proposed SMCG takes video semantic representation as an input, and conditionally modulates the gates and cells of long short-term memory network with respect to the encoded syntactic information of the given exemplar sentence. | method 1.00 | 0.76 | 0.04 | asserted 0.99 |
| s4 | Therefore, SMCG is able to control the states for word prediction and achieve the syntax customized caption generation. | method 0.54 | 0.72 | 0.14 | asserted 0.95 |
| s5 | We conduct experiments by collecting auxiliary exemplar sentences for two public video captioning datasets. | method 0.96 | 0.95 | 0.02 | asserted 0.99 |
| s6 | Extensive experimental results demonstrate the effectiveness of our approach on generating syntax controllable and semantic preserved video captions. | result 0.99 | 0.96 | 0.69 | asserted 1.00 |
| s7 | By providing different exemplar sentences, our approach is capable of producing different captions with various syntactic structures, thus indicating a promising way to strengthen the diversity of video captioning. | interpretation 0.95 | 0.87 | 0.39 | hedged possibility 0.85 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
In this paper, we investigate a novel and challenging task, namely controllable video captioning with an exemplar sentence. Formally, given a video and a syntactically valid exemplar sentence, the task aims to generate one caption which not only describes the semantic contents of the video, but also follows the syntactic form of the given exemplar sentence. In order to tackle such an exemplar-based video captioning task, we propose a novel Syntax Modulated Caption Generator (SMCG) incorporated in an encoder-decoder-reconstructor architecture. The proposed SMCG takes video semantic representation as an input, and conditionally modulates the gates and cells of long short-term memory network with respect to the encoded syntactic information of the given exemplar sentence. Therefore, SMCG is able to control the states for word prediction and achieve the syntax customized caption generation. We conduct experiments by collecting auxiliary exemplar sentences for two public video captioning datasets. Extensive experimental results demonstrate the effectiveness of our approach on generating syntax controllable and semantic preserved video captions. By providing different exemplar sentences, our approach is capable of producing different captions with various syntactic structures, thus indicating a promising way to strengthen the diversity of video captioning.
Human review
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.