RECORD
Write What You Want: Applying Text-to-video Retrieval to Audiovisual Archives
Multimedia · Submitted 9 Oct 2023
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim This work filled the application gap by examining such text-to-video retrieval methods from an implementation point of view and proposed and verified a classifier-enhanced workflow to allow better results when dealing with in-situ queries that might have been different from the training dataset.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Audiovisual (AV) archives, as an essential reservoir of our cultural assets, are suffering from the issue of accessibility. The complex nature of the medium itself made processing and interaction an open challenge still in the field of computer vision, multimodal learning, and human-computer interaction, as well as in culture and heritage. In recent years, with the raising of video retrieval tasks, methods in retrieving video content with natural language (text-to-video retrieval) gained quite some attention and have reached a performance level where real-world application is on the horizon. Appealing as it may sound, such methods focus on retrieving videos using plain visual-focused descriptions of what has happened in the video and finding videos such as instructions. It is too early to say such methods would be the new paradigms for accessing and encoding complex video content into high-dimensional data, but they are indeed innovative attempts and foundations to build future exploratory interfaces for AV archives (e.g. allow users to write stories and retrieve related snippets in the archive, or encoding video content at high-level for visualisation). This work filled the application gap by examining such text-to-video retrieval methods from an implementation point of view and proposed and verified a classifier-enhanced workflow to allow better results when dealing with in-situ queries that might have been different from the training dataset. Such a workflow is then applied to the real-world archive from T\'el\'evision Suisse Romande (RTS) to create a demo. At last, a human-centred evaluation is conducted to understand whether the text-to-video retrieval methods improve the overall experience of accessing AV archives.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Audiovisual (AV) archives, as an essential reservoir of our cultural assets, are suffering from the issue of accessibility. | background 1.00 | 0.05 | 0.02 | asserted 1.00 |
| s1 | The complex nature of the medium itself made processing and interaction an open challenge still in the field of computer vision, multimodal learning, and human-computer interaction, as well as in culture and heritage. | background 0.99 | 0.04 | 0.02 | asserted 1.00 |
| s2 | In recent years, with the raising of video retrieval tasks, methods in retrieving video content with natural language (text-to-video retrieval) gained quite some attention and have reached a performance level where real-world application is on the horizon. | background 1.00 | 0.04 | 0.04 | asserted 0.99 |
| s3 | Appealing as it may sound, such methods focus on retrieving videos using plain visual-focused descriptions of what has happened in the video and finding videos such as instructions. | background 0.46 | 0.07 | 0.04 | asserted 0.96 |
| s4 | It is too early to say such methods would be the new paradigms for accessing and encoding complex video content into high-dimensional data, but they are indeed innovative attempts and foundations to build future exploratory interfaces for AV archives (e.g. allow users to write stories and retrieve related snippets in the archive, or encoding video content at high-level for visualisation). | background 0.27 | 0.11 | 0.18 | hedged possibility 0.91 |
| s5 | This work filled the application gap by examining such text-to-video retrieval methods from an implementation point of view and proposed and verified a classifier-enhanced workflow to allow better results when dealing with in-situ queries that might have been different from the training dataset. | contribution 0.99 | 0.95 | 0.36 | asserted 1.00 |
| s6 | Such a workflow is then applied to the real-world archive from T\'el\'evision Suisse Romande (RTS) to create a demo. | method 0.59 | 0.92 | 0.05 | asserted 1.00 |
| s7 | At last, a human-centred evaluation is conducted to understand whether the text-to-video retrieval methods improve the overall experience of accessing AV archives. | method 0.65 | 0.94 | 0.13 | asserted 0.87 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Audiovisual (AV) archives, as an essential reservoir of our cultural assets, are suffering from the issue of accessibility. The complex nature of the medium itself made processing and interaction an open challenge still in the field of computer vision, multimodal learning, and human-computer interaction, as well as in culture and heritage. In recent years, with the raising of video retrieval tasks, methods in retrieving video content with natural language (text-to-video retrieval) gained quite some attention and have reached a performance level where real-world application is on the horizon. Appealing as it may sound, such methods focus on retrieving videos using plain visual-focused descriptions of what has happened in the video and finding videos such as instructions. It is too early to say such methods would be the new paradigms for accessing and encoding complex video content into high-dimensional data, but they are indeed innovative attempts and foundations to build future exploratory interfaces for AV archives (e.g. allow users to write stories and retrieve related snippets in the archive, or encoding video content at high-level for visualisation). This work filled the application gap by examining such text-to-video retrieval methods from an implementation point of view and proposed and verified a classifier-enhanced workflow to allow better results when dealing with in-situ queries that might have been different from the training dataset. Such a workflow is then applied to the real-world archive from T\'el\'evision Suisse Romande (RTS) to create a demo. At last, a human-centred evaluation is conducted to understand whether the text-to-video retrieval methods improve the overall experience of accessing AV archives.
Human review
How the literature uses it
What this paper does with the arXiv papers it cites
What it cites · 2 sentences without a confident label
Uses the method or tool of 2 papers
-
The entire MSR-VTT dataset is fed to the recently proposed Whisper [ref] model to obtain transcripts for each video clip, using the provided "small model" and following the implementation guides on hugging face[<https://huggingface.co/openai/whisper-small>].
Robust Speech Recognition via Large-Scale Weak Supervision · Workflow implementation details · 1.00
-
Then, the participants are asked to vote between the overall experiences of the text-to-video retrieval and the traditional search interface based on the acute-eval metric [ref] , which emphasises the four key aspects when evaluating the performance and improvements of novel methods and systems: Engagingness, Interestingness, Humanness, and Informativeness (relevance).
ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons · Stage II · 0.97
Cites as background 5 papers
-
Other works [ref] rely on pre-existing large language to image models to pretrain, and finetune the model using manually labelled text-video datasets.
CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment · Text-to-video retrieval · 0.84
-
Most recently, [ref] proposed a new model that utilises not only the automated generated weakly-paired text-video datasets but also manual-labelled text-image and standard text-video datasets.
MDMMT-2: Multidomain Multimodal Transformer for Video Retrieval, One More Step Towards Generalization · Text-to-video retrieval · 0.83
-
Others attempt to improve the performance by leveraging the text with its context [ref] , introducing Conditional Random Field (CRF) to understand the inter-tag dependencies [ref] .
Dialogue Act Classification with Context-Aware Self-Attention · Dialogue act recognizer · 0.86
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.