Pith. sign in

REVIEW 3 major objections 6 minor 75 references

Quantifying and Narrowing the Unknown: Interactive Text-to-Video Retrieval via Uncertainty Minimization

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Explicit uncertainty scores, not heuristics, should drive the questions an interactive video retriever asks.

desk verdict UMIVR is a clean integration of three training-free uncertainty metrics into interactive text-to-video retrieval, but the headline gains depend on a simulated user that is the same model doing the retrieval, so the real-user benefit is unproven. read the letter →

arxiv 2507.15504 v2 pith:CXTQT4VO submitted 2025-07-21 cs.CV

Bingqing Zhang , Zhuo Cao , Heming Du , Yang Li , Xue Li , Jiajun Liu , Sen Wang This is my paper
classification cs.CV
keywords text-to-videoretrievalinteractiveuncertaintyquantificationclarifyingquestionssemanticentropyJensen-Shannondivergenceframequalityassessmentqueryrefinement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that interactive text-to-video retrieval should decide what to ask the user by quantifying how uncertain the retrieval really is, rather than by fixed or ad-hoc question templates. It decomposes retrieval uncertainty into three measurable parts: ambiguous query wording, ambiguous mapping between query and candidate videos, and low-quality frames. It introduces training-free scores for the first two and a frame-quality sampler for the third, then uses them to generate clarifying questions and to stop interaction early. On the MSR-VTT-1k benchmark the method reports Recall@1 of 69.2% after ten rounds, surpassing both interactive and non-interactive baselines. A sympathetic reader would take the central insight to be that uncertainty measures can serve as a principled control loop for interactive retrieval.

What carries the argument

The load-bearing mechanism is the uncertainty-gated question-selection loop. A semantic-entropy score measures how spread a query's meaning is over caption clusters, a Jensen-Shannon score measures how flat the top-$k$ similarity distribution is, and a frame sampler chooses temporally diverse high-quality frames using a no-reference quality measure. These values decide which of three question-generation prompts the unified video-language model uses, and the same scores can halt interaction when both drop below thresholds. What carries the argument is that the scores are computed without training and can be tracked and reduced across interaction rounds.

What would settle it

Run the same interaction protocol with human users answering the same clarifying questions on MSR-VTT-1k queries and compare Recall@1 and Hit@1 across rounds. If human answers are noisier or less aligned with the retriever and the measured gains over non-interactive retrieval shrink substantially, the uncertainty-control claim would not survive. A second check would be to corrupt the offline caption corpus and verify whether the Text Ambiguity Score still routes questions correctly; if the gains hinge on perfect captions, the mechanism is less robust than claimed.

Watch

Extended reading notes

Core claim

The central claim is that interactive text-to-video retrieval can be steered by three explicit, training-free uncertainty measurements. The Text Ambiguity Score estimates semantic ambiguity by computing entropy over clusters of database captions similar to the query, normalized to $[0,1]$. The Mapping Uncertainty Score converts the top-$k$ similarity scores into a distribution and measures its Jensen-Shannon divergence from a one-hot certain distribution, normalizing by the maximum divergence. The Temporal Quality-based Frame Sampler scores frames with a no-reference quality measure, selects the best frame per temporal bin, and then uses embedding clustering to keep semantic diversity; it also works as a plug-in for existing retrievers. The retrieval loop computes the two scores each round, chooses one of three question-generation modes designed to target the dominant uncertainty, folds the simulated user's answer into the query, and repeats. On MSR-VTT-1k the paper reports Recall@1 of 69.2% after ten rounds, Hit@1 of 68.9% after three rounds, and consistent gains on three other video datasets, with the same pipeline transferring to interactive text-to-image retrieval.

Load-bearing premise

The evaluation assumes that simulated user answers generated by a video question-answering model are a good stand-in for real users' answers; the paper itself notes that this inevitably differs from real human interaction.

Editorial extensions

If this is right

  • Interactive video retrieval can be organized as an uncertainty-reduction loop: ask the question that targets the largest measured unknown, and gains follow without retraining the retriever.
  • Three rounds of interaction are enough to beat the strongest non-interactive method on MSR-VTT-1k on Hit@1, and ten rounds reach Recall@1 of 69.2%.
  • The frame sampler transfers: adding it to trained single-round video retrievers improves Recall@10 and lowers mean rank without fine-tuning.
  • The uncertainty scores can also act as an early-stopping signal, reducing the average interaction count to about three rounds at modest performance cost.
  • The same uncertainty-gated interaction extends to interactive text-to-image retrieval, suggesting that the mechanism is not tied to video-specific encoders.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is a human-in-the-loop study: the simulated user and the retriever share the same video-language model, so real users may answer less informatively and the measured advantage could shrink.
  • The three-way uncertainty decomposition could be reused as a diagnostic: high text-ambiguity suggests query rewriting, high mapping-uncertainty with low text-ambiguity suggests asking distinguishing questions, and persistent frame-quality problems suggest re-sampling or better capture.
  • One could train or prompt a model to compute the uncertainty scores internally rather than externally, a limitation the paper itself lists; if that coupling works, the same loop could generalise beyond predefined question levels.
  • The scores' thresholds are tuned per benchmark, so a testable extension is to predict them from query and video statistics, or to make the question-selection policy itself learned.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes UMIVR, an interactive text-to-video retrieval framework that explicitly quantifies three types of uncertainty: text ambiguity via a semantic-entropy Text Ambiguity Score (TAS, Section 3.1), mapping uncertainty via a Jensen-Shannon divergence Mapping Uncertainty Score (MUS, Section 3.2), and frame uncertainty via a Temporal Quality-based Frame Sampler (TQFS, Section 3.3). These scores drive an iterative pipeline (Section 3.4) in which a unified VideoLLaVA model generates one clarifying question per round, the user answers, and the query is refined. Experiments on MSR-VTT-1k, AVSD, MSVD, ActivityNet, and VisDial report consistent gains over interactive baselines, with a headline Recall@1 of 69.2% after 10 rounds on MSR-VTT-1k, and the paper also demonstrates TQFS as a plug-in for non-interactive retrieval methods.

Significance. If the results hold, UMIVR would be a useful step beyond heuristic question-generation for interactive retrieval: the uncertainty scores are training-free, interpretable, and the unified VideoLLaVA implementation substantially reduces GPU memory relative to ensemble baselines (Appendix D, Table 9). The paper also ships a plug-and-play frame sampler with reproducible gains on VideoCLIP and Xpool (Table 6), and it extends the framework to interactive text-to-image retrieval. However, the external validity of the headline interactive gains depends entirely on the simulated-user protocol, which uses the same VideoLLaVA backbone for answer generation, retrieval, captioning, and query refinement; this is the main load-bearing weakness.

major comments (3)
  1. [Section 3.4 (Step 3) and Appendix H] The evaluation protocol simulates user answers with VideoLLaVA prompted with the target video's frames (Appendix G, Table 17), while the same VideoLLaVA/LanguageBind backbone is used as the retriever, the captioner that builds the meta-information for Level-1 questions, and the query refiner. As a result, the refined query is constructed from text generated in the same model-specific semantic space as the target video embeddings, so the reported Recall@1 gains (e.g., 69.2% at round 10) may reflect model-model self-consistency rather than the general quality of uncertainty-gated clarifying questions. Appendix H itself concedes that the simulation "inevitably differs from real human interactions." The central claim that uncertainty-gated questions are a better foundation for interactive retrieval requires evidence that the user is not inside the retriever's echo chamber. Please add a decoupled-user experiment, for example answers generated by a different VideoQA or LLM backbone, paraphrased or noisy human-like answers, or a small human study, and report Recall@1, Hit@1, and BRI under that protocol.
  2. [Section 4.4, Table 5] The thresholds alpha (TAS) and beta (MUS) are selected by grid search and Table 5 reports performance on MSR-VTT-1k, but the manuscript does not state whether the reported numbers come from a held-out validation split or the test split, nor how the chosen values (0.5, 0.2) transfer to AVSD, MSVD, ActivityNet, and VisDial. Since these thresholds determine which of the three question-generation levels is triggered, tuning them on the evaluation set can inflate the interactive gains. Please specify the selection protocol and report results with a single fixed threshold across datasets, or with within-dataset validation splits.
  3. [Sections 3.1 and 3.3] TAS is described as being "transformed into the final TAS value in [0,1] through a normalization function" with possible "additional adjustments," but the exact normalization, the value of K for the top-K retrieved captions, and the number of clusters M are not specified. Similarly, TQFS uses K-means clustering without specifying K, the initialization, or how the cluster count is chosen per video. These unspecified components prevent reproduction and make it difficult to determine whether the reported behavior is driven by the proposed uncertainty metrics or by hidden normalization and hyperparameter choices. Please provide the exact formulas and hyperparameter values for all datasets.
minor comments (6)
  1. [Section 3, first paragraph] The sentence "the three types of uncertainty identified in Sec." ends with an incomplete reference; the section number is missing.
  2. [Table 6] The column header "MSR-VTT-1ka" appears to be a typo; it should be "MSR-VTT-1kA" for consistency with the main text.
  3. [Table 5] Several numbers in Table 5 are run together without spaces or separators (e.g., "91.694.8" and "57.168.973.9"); please fix the formatting for readability.
  4. [Appendix D, Table 10] The BRI column contains entries such as "1.100.89" and "0.690.63" with missing spacing; please reformat.
  5. [Abstract and Section 4.1] The code link is written as "avaliable athttps://github.com/bingqingzhang/umivr" without spacing or punctuation; please correct the typo.
  6. [Appendix H, last paragraph] The claim that information leakage is absent because retrieval and VideoQA are "stateless and executed through independent inference calls" does not address the shared-embedding-space self-consistency concern; I recommend rewording this paragraph to acknowledge the distinction between state leakage and representation sharing.

Circularity Check

1 steps flagged · score 4.0 of 10

Reported interactive gains are partly self-confirming because the simulated user, question generator, and retriever are the same VideoLLaVA model; the uncertainty metrics themselves are not fitted, so circularity is partial.

  1. other [Sec. 3.4 (Steps 3-4), Fig. 2(b); Appendix H; Table 17]
    "This interaction can be conducted with real users or approximated for evaluation purposes using simulated responses derived from VideoQA modules (Step 3). ... Fig. 2(b) summarizes this unified implementation, highlighting how UMIVR compactly realizes captioning, response simulation, clarifying question generation, and retrieval functionalities within a single model. ... our experiments rely on a simulated question-answering mechanism that mimics user responses ... it inevitably differs from real human interactions."

    The evaluation's 'user' is VideoLLaVA conditioned on the target video's frames (Table 17), and the retriever is the same VideoLLaVA/LanguageBind model. The simulated answer is therefore generated from the very visual representation that the retriever will match against; after the same model's query-refinement prompt folds that answer into the query, the refined text is almost guaranteed to lie close to the target video's embedding. Hence the measured R@1 improvement after interaction is partly a measure of model self-consistency rather than of whether uncertainty-gated questions produce better clarification for an independent user.

full rationale

The three uncertainty components have independent grounding: TAS is computed from semantic entropy over captions, MUS from JS divergence over retrieval-score distributions, and TQFS from NR-IQA scores plus clustering. None of these is fitted to the target rank labels, and only two global thresholds (alpha, beta) are tuned, so the central mechanism is not equivalent to its inputs by construction. The main circularity concern is the evaluation loop: the same VideoLLaVA model acts as captioner, question generator, simulated user, query refiner, and retriever, so the refined query is generated in the same embedding space used for retrieval. This makes the reported absolute gains and the claimed real-user foundation partially self-confirming, although the paper discloses the limitation in Appendix H. The grid-search of thresholds on the same benchmark is a test-set-tuning concern rather than a circular derivation, and no load-bearing self-citation chain was found. Overall, the uncertainty-aware question selection retains independent content, but the headline interactive-retrieval claim should be interpreted with the self-consistency caveat in mind.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The two fitted thresholds are the only explicit free parameters, but the pipeline also silently depends on unstated cluster counts and top-k choices. The main domain assumptions are that semantic entropy and JS divergence correspond to the named uncertainties, and that the same model can simulate a user faithfully.

free parameters (5)
  • TAS threshold alpha = 0.5
    Chosen by grid search on MSR-VTT-1k (Table 5); controls whether the system asks an open-ended question (Level-0) instead of a mapping-distinguishing question.
  • MUS threshold beta = 0.2
    Chosen by grid search on MSR-VTT-1k (Table 5); controls when mapping uncertainty triggers Level-1 questions.
  • Number of frames K in TQFS = 8 (VideoLLaVA input default)
    The unified model consumes 8 frames per video; this is inherited from the backbone rather than tuned, but it is a free choice for the sampler.
  • TAS cluster count M = not reported
    The semantic entropy definition in Section 3.1 depends on clustering captions into M groups, but M is never specified.
  • Top-k for MUS similarity scores = not reported
    Section 3.2 uses the top-k similarities without stating k; this affects the JS divergence and the resulting MUS values.
assumptions (4)
  • domain assumption Semantic entropy over clustered captions captures text ambiguity.
    Section 3.1 equates ambiguity with entropy over caption clusters; there is no validation that the clusters correspond to meanings rather than captioning noise.
  • domain assumption JS divergence of the similarity distribution to a one-hot vector measures mapping uncertainty.
    Section 3.2 assumes the ideal mapping is a single candidate (one-hot), which penalizes legitimate cases where multiple videos are equally correct.
  • domain assumption Simulated VideoQA answers approximate real user answers.
    Step 3 of Section 3.4 and Appendix H: all interactive results rely on answers generated by VideoLLaVA, not human users.
  • domain assumption The same VideoLLaVA model can reliably caption, ask questions, answer questions, and retrieve.
    The unified architecture in Section 3.4 assumes one backbone is competent at all four tasks; failures in any one would propagate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Quantifying and Narrowing the Unknown: Interactive Text-to-Video Retrieval via Uncertainty Minimization." pith.science (2026). https://pith.science/paper/CXTQT4VO

@misc{pith2026250715504,
  author       = {Pith},
  title        = {Pith review of: Quantifying and Narrowing the Unknown: Interactive Text-to-Video Retrieval via Uncertainty Minimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CXTQT4VO}},
  note         = {Machine review of arXiv:2507.15504}
}
read the original abstract

Despite recent advances, Text-to-video retrieval (TVR) is still hindered by multiple inherent uncertainties, such as ambiguous textual queries, indistinct text-video mappings, and low-quality video frames. Although interactive systems have emerged to address these challenges by refining user intent through clarifying questions, current methods typically rely on heuristic or ad-hoc strategies without explicitly quantifying these uncertainties, limiting their effectiveness. Motivated by this gap, we propose UMIVR, an Uncertainty-Minimizing Interactive Text-to-Video Retrieval framework that explicitly quantifies three critical uncertainties-text ambiguity, mapping uncertainty, and frame uncertainty-via principled, training-free metrics: semantic entropy-based Text Ambiguity Score (TAS), Jensen-Shannon divergence-based Mapping Uncertainty Score (MUS), and a Temporal Quality-based Frame Sampler (TQFS). By adaptively generating targeted clarifying questions guided by these uncertainty measures, UMIVR iteratively refines user queries, significantly reducing retrieval ambiguity. Extensive experiments on multiple benchmarks validate UMIVR's effectiveness, achieving notable gains in Recall@1 (69.2\% after 10 interactive rounds) on the MSR-VTT-1k dataset, thereby establishing an uncertainty-minimizing foundation for interactive TVR.

Figures

Figures reproduced from arXiv: 2507.15504 by the authors.

Figure 1
Figure 1. Illustration of uncertainty challenges in Text-to-Video [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. A unified implementation of UMIVR with Vide￾oLLaVA. (a) VideoLLaVA integrates LanguageBind and LLM into a unified architecture, enabling simultaneous handling of video-text tasks. (b) A summary illustrating how UMIVR lever￾ages the single, unified VideoLLaVA model to compactly realize all its core functionalities, significantly simplifying the system ar￾chitecture compared to prior ensemble or hybrid approaches. whe… view at source ↗
Figure 3
Figure 3. Pipeline of the UMIVR framework Videos are first preprocessed offline (Step 0) by TQFS for high-quality frame selection and captioning to generate meta-information. Given an initial user query, UMIVR quantifies textual and mapping uncertainties (TAS & MUS, Step 1), adaptively generates clarifying questions at different uncertainty levels (Level-0, 1 and 2, Step 2), and iteratively refines queries based on user respo… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Comparison of UMIVR with interactive baseline methods on MSR-VTT-1k across multiple interaction rounds. From left to right, we illustrate Recall@1, Hit@1, and BRI scores, respectively. UMIVR consistently outperforms competing interactive baselines by achieving higher R…
Figure 5
Figure 5. Figure 5: Case study of UMIVR’s interactive retrieval process. The examples illustrate how uncertainty-aware question genera￾tion progressively refines ambiguous queries, reducing the Text Ambiguity Score (TAS) and Mapping Uncertainty Score (MUS) while improving retrieval rank. …
Figure 6
Figure 6. Figure 6: Distribution of Text Ambiguity Score (TAS) across interaction rounds. The histograms illustrate the progressive reduction in TAS as the interactive retrieval process advances from Round 0 to Round 7. Initially, the majority of queries exhibit high ambiguity, with a str…
Figure 7
Figure 7. Figure 7: Mapping Uncertainty Score (MUS) across different Ground Truth (GT) ranks. The figure shows the relationship between MUS and the rank position of the ground truth video in the retrieval results. When GT is ranked 1, MUS is consistently low, indicating high confidence in…
Figure 8
Figure 8. Figure 8: Overview of the Reproduced IVR-Heuristic Pipeline Structure. The pipeline follows a structured, heuristic-driven interactive retrieval approach for text-to-video retrieval. It first categorizes videos into four types: human, cartoon, animal, and other, tailoring its qu…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

75 extracted references · 71 canonical work pages

  1. [1]

    Audio visual scene- aware dialog

    Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim K Marks, Chiori Hori, Peter Anderson, et al. Audio visual scene- aware dialog. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7558– 7567, 2019. 5

    work page 2019

  2. [2]

    Askvideos-videoclip: Language-grounded video embeddings

    AskVideos. Askvideos-videoclip: Language-grounded video embeddings. GitHub, 2024. 8

    work page 2024

  3. [3]

    Some explo- rations in initial interaction and beyond: Toward a devel- opmental theory of interpersonal communication.Human communication research, 1(2):99–112, 1974

    Charles R Berger and Richard J Calabrese. Some explo- rations in initial interaction and beyond: Toward a devel- opmental theory of interpersonal communication.Human communication research, 1(2):99–112, 1974. 12

    work page 1974

  4. [4]

    Ask&confirm: Active detail enriching for cross-modal retrieval with partial query.2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1815–1824, 2021

    Guanyu Cai, Jun Zhang, Xinyang Jiang, Yifei Gong, Lianghua He, Fufu Yu, Pai Peng, Xiaowei Guo, Feiyue Huang, and Xing Sun. Ask&confirm: Active detail enriching for cross-modal retrieval with partial query.2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1815–1824, 2021. 3

    work page 2021

  5. [5]

    Chen and William B

    David L. Chen and William B. Dolan. Collecting highly par- allel data for paraphrase evaluation. InProceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL-2011), Portland, OR, 2011. 5

    work page 2011

  6. [6]

    Semantics disentangling for generalized zero-shot learning

    Zhi Chen, Yadan Luo, Ruihong Qiu, Sen Wang, Zi Huang, Jingjing Li, and Zheng Zhang. Semantics disentangling for generalized zero-shot learning. InIEEE/CVF International Conference on Computer Vision (ICCV), 2021. 3

    work page 2021

  7. [7]

    Svip: Semantically contextualized visual patches for zero-shot learning

    Zhi Chen, Zecheng Zhao, Jingcai Guo, Jingjing Li, and Zi Huang. Svip: Semantically contextualized visual patches for zero-shot learning. InICCV2025, 2025. 3

    work page 2025

  8. [8]

    John Wiley & Sons, 1999

    Thomas M Cover.Elements of information theory. John Wiley & Sons, 1999. 4

    1999

Show all 75 references
  1. [9]

    Teachtext: Crossmodal generalized distillation for text-video retrieval.2021 IEEE/CVF International Conference on Com- puter Vision (ICCV), pages 11563–11573, 2021

    Ioana Croitoru, Simion-Vlad Bogolin, Yang Liu, Samuel Al- banie, Marius Leordeanu, Hailin Jin, and Andrew Zisserman. Teachtext: Crossmodal generalized distillation for text-video retrieval.2021 IEEE/CVF International Conference on Com- puter Vision (ICCV), pages 11563–11573, 2021. 2

    2021

  2. [10]

    Abhishek Das, Satwik Kottur, Jos ´e M. F. Moura, Stefan Lee, and Dhruv Batra. Learning cooperative visual dialog agents with deep reinforcement learning.2017 IEEE International Conference on Computer Vision (ICCV), pages 2970–2979,

    2017

  3. [11]

    Activitynet: A large-scale video bench- mark for human activity understanding

    Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles. Activitynet: A large-scale video bench- mark for human activity understanding. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 961–970, 2015. 5

    2015

  4. [12]

    UATVR: uncertainty-adaptive text-video retrieval

    Bo Fang, Wenhao Wu, Chang Liu, Yu Zhou, Yuxin Song, Weiping Wang, Xiangbo Shu, Xiangyang Ji, and Jingdong Wang. UATVR: uncertainty-adaptive text-video retrieval. InIEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 13677– 1...

    2023

  5. [13]

    Mask to reconstruct: Cooperative semantics comple- tion for video-text retrieval.Proceedings of the 31st ACM International Conference on Multimedia, 2023

    Han Fang, Zhifei Yang, Xianghao Zang, Chao Ban, and Hao Sun. Mask to reconstruct: Cooperative semantics comple- tion for video-text retrieval.Proceedings of the 31st ACM International Conference on Multimedia, 2023. 2

    2023

  6. [14]

    Detecting hallucinations in large language models using semantic entropy.Nat., 630(8017):625–630, 2024

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy.Nat., 630(8017):625–630, 2024. 3, 12

    2024

  7. [15]

    Multi-modal transformer for video retrieval

    Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV, pages 214–229. Springer, 2020. 1, 2

    2020

  8. [16]

    X-pool: Cross-modal language-video attention for text- video retrieval.2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4996–5005,

    Satya Krishna Gorti, No ¨el V ouitsis, Junwei Ma, Keyvan Golestan, Maksims V olkovs, Animesh Garg, and Guangwei Yu. X-pool: Cross-modal language-video attention for text- video retrieval.2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4996–5005,

    2022

  9. [17]

    Shubham Gupta, Peter W. J. Staar, and Christian de Sainte Marie. Clustering items from adaptively collected in- consistent feedback. InInternational Conference on Artifi- cial Intelligence and Statistics, 2024. 2

    2024

  10. [18]

    MERLIN: multimodal embedding re- finement via llm-based iterative navigation for text-video retrieval-rerank pipeline

    Donghoon Han, Eunhwan Park, Gisang Lee, Adam Lee, and Nojun Kwak. MERLIN: multimodal embedding re- finement via llm-based iterative navigation for text-video retrieval-rerank pipeline. InProceedings of the 2024 Con- ference on Empirical Methods in Natural Language Pro- cessing...

    2024

  11. [19]

    Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods.Mach

    Eyke H ¨ullermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods.Mach. Learn., 110(3):457–506,

  12. [20]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 2

    2024 arXiv

  13. [21]

    Tencent text-video retrieval: Hierarchi- cal cross-modal interactions with multi-level representations

    Jie Jiang, Shaobo Min, Weijie Kong, Hongfa Wang, Zhifeng Li, and Wei Liu. Tencent text-video retrieval: Hierarchi- cal cross-modal interactions with multi-level representations. IEEE Access, 2022. 6, 7

    2022

  14. [22]

    Semantic uncertainty: Linguistic invariances for uncertainty estima- tion in natural language generation

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estima- tion in natural language generation. InThe Eleventh Interna- tional Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview...

    2023

  15. [23]

    Interactive text-to-image retrieval with large language models: A plug-and-play approach

    Saehyung Lee, Sangwon Yu, Junsung Park, Jihun Yi, and Sungroh Yoon. Interactive text-to-image retrieval with large language models: A plug-and-play approach. InProceed- ings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), ...

    2024

  16. [24]

    Chatting makes perfect: Chat-based image retrieval

    Matan Levy, Rami Ben-Ari, Nir Darshan, and Dani Lischin- ski. Chatting makes perfect: Chat-based image retrieval. In Neural Information Processing Systems, 2023. 3, 8

    2023

  17. [25]

    BART: denoising sequence-to- sequence pre-training for natural language generation, trans- lation, and comprehension

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvinine- jad, Abdelrahman Mohamed, Omer Levy, Veselin Stoy- anov, and Luke Zettlemoyer. BART: denoising sequence-to- sequence pre-training for natural language generation, trans- lation, and comprehension. InProceedings of the 58t...

    2020

  18. [26]

    A survey on deep ac- tive learning: Recent advances and new frontiers.CoRR, abs/2405.00334, 2024

    Dongyuan Li, Zhen Wang, Yankai Chen, Renhe Jiang, Weip- ing Ding, and Manabu Okumura. A survey on deep ac- tive learning: Recent advances and new frontiers.CoRR, abs/2405.00334, 2024. 2

    2024 arXiv

  19. [27]

    Prototype-based aleatoric uncertainty quantifica- tion for cross-modal retrieval.ArXiv, abs/2309.17093, 2023

    Hao Li, Jingkuan Song, Lianli Gao, Xiaosu Zhu, and Heng- tao Shen. Prototype-based aleatoric uncertainty quantifica- tion for cross-modal retrieval.ArXiv, abs/2309.17093, 2023. 2

    2023 arXiv

  20. [28]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. InIn- ternational Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, pages 12...

    2022

  21. [29]

    Progressive spatio-temporal prototype matching for text- video retrieval

    Pandeng Li, Chen-Wei Xie, Liming Zhao, Hongtao Xie, Jiannan Ge, Yun Zheng, Deli Zhao, and Yongdong Zhang. Progressive spatio-temporal prototype matching for text- video retrieval. InIEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 20...

    2023

  22. [30]

    Simple baselines for interactive video retrieval with questions and answers

    Kaiqu Liang and Samuel Albanie. Simple baselines for interactive video retrieval with questions and answers. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 11057– 11067. IEEE, 2023. 2, 5, 7, 12, 13, 18

    2023

  23. [31]

    Video-llava: Learning united visual repre- sentation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual repre- sentation by alignment before projection. InProceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2024, Miami, FL, USA,...

    2024

  24. [32]

    Text-adaptive multiple visual prototype matching for video-text retrieval

    Chengzhi Lin, Ancong Wu, Junwei Liang, Jun Zhang, Wen- hang Ge, Wei-Shi Zheng, and Chunhua Shen. Text-adaptive multiple visual prototype matching for video-text retrieval. InAdvances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processin...

    2022

  25. [33]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceeding...

    2014

  26. [34]

    Use what you have: Video retrieval using represen- tations from collaborative experts

    Yang Liu, Samuel Albanie, Arsha Nagrani, and Andrew Zis- serman. Use what you have: Video retrieval using represen- tations from collaborative experts. In30th British Machine Vision Conference 2019, BMVC 2019, Cardiff, UK, Septem- ber 9-12, 2019, page 279. BMV A Press, 2019. 1, 2

    2019

  27. [35]

    Ts2-net: Token shift and selection transformer for text-video retrieval

    Yuqi Liu, Pengfei Xiong, Luhui Xu, Shengming Cao, and Qin Jin. Ts2-net: Token shift and selection transformer for text-video retrieval. InComputer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIV, pages 319–335. Spring...

    2022

  28. [36]

    Clip4clip: An empirical study of CLIP for end to end video clip retrieval and captioning

    Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of CLIP for end to end video clip retrieval and captioning. Neurocomputing, 508:293–304, 2022. 2, 4, 6

    2022

  29. [37]

    Dialogue-to-video retrieval

    Chenyang Lyu, Manh-Duy Nguyen, Van-Tu Ninh, Liting Zhou, Cathal Gurrin, and Jennifer Foster. Dialogue-to-video retrieval. InAdvances in Information Retrieval - 45th Eu- ropean Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2-6, 2023, Proceedings, Part I...

    2023

  30. [38]

    X-CLIP: end-to-end multi-grained con- trastive learning for video-text retrieval

    Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. X-CLIP: end-to-end multi-grained con- trastive learning for video-text retrieval. InMM ’22: The 30th ACM International Conference on Multimedia, Lisboa, Portugal, October 10 - 14, 2022, pages 638–647. ACM,

    2022

  31. [39]

    Learn- ing to retrieve videos by asking questions

    Avinash Madasu, Junier Oliva, and Gedas Bertasius. Learn- ing to retrieve videos by asking questions. InMM ’22: The 30th ACM International Conference on Multimedia, Lisboa, Portugal, October 10 - 14, 2022, pages 356–365. ACM,

    2022

  32. [40]

    Oliva, and Gedas Bertasius

    Avinash Madasu, Junier B. Oliva, and Gedas Bertasius. Learning to retrieve videos by asking questions.Proceed- ings of the 30th ACM International Conference on Multime- dia, 2022. 3

    2022

  33. [41]

    The jensen-shannon divergence

    Mar ´ıa Luisa Men ´endez, Julio Angel Pardo, Leandro Pardo, and Mar ´ıa del C Pardo. The jensen-shannon divergence. Journal of the Franklin Institute, 334(2):307–318, 1997. 3

    1997

  34. [42]

    Moorthy, and Alan C

    Anish Mittal, Anush K. Moorthy, and Alan C. Bovik. Blind/referenceless image spatial quality evaluator. InCon- ference Record of the Forty Fifth Asilomar Conference on Signals, Systems and Computers, ACSCC 2011, Pacific Grove, CA, USA, November 6-9, 2011, pages 723–727. IEEE, ...

    2011

  35. [43]

    ChatGPT.https://openai.com/blog/ chatgpt/, 2023

    OpenAI. ChatGPT.https://openai.com/blog/ chatgpt/, 2023. 3, 4

    2023

  36. [44]

    Hauptmann, Jo ˜ao F

    Mandela Patrick, Po-Yao Huang, Yuki Markus Asano, Flo- rian Metze, Alexander G. Hauptmann, Jo ˜ao F. Henriques, and Andrea Vedaldi. Support-set bottlenecks for video-text representation learning. In9th International Conference on Learning Representations, ICLR 2021, Virtual Ev...

    2021

  37. [45]

    Patro, Shivansh Patel, and Vinay P

    Badri N. Patro, Shivansh Patel, and Vinay P. Namboodiri. Granular multimodal attention networks for visual dialog. ArXiv, abs/1910.05728, 2019. 8

    1910 arXiv

  38. [46]

    John Wiley & Sons, 2010

    Maria MP Petrou and Costas Petrou.Image processing: the fundamentals. John Wiley & Sons, 2010. 4, 15

    2010

  39. [47]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, 10 Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning...

    2021

  40. [48]

    Rahmani, Xi Wang, Mohammad Aliannejadi, Mohammadmehdi Naghiaei, and Emine Yilmaz

    Hossein A. Rahmani, Xi Wang, Mohammad Aliannejadi, Mohammadmehdi Naghiaei, and Emine Yilmaz. Clarifying the path to user satisfaction: An investigation into clarifica- tion usefulness. InFindings of the Association for Compu- tational Linguistics: EACL 2024, St. Julian’s, Malt...

    2024

  41. [49]

    Huang, Michael Ortega-Binderberger, and Sharad Mehrotra

    Yong Rui, Thomas S. Huang, Michael Ortega-Binderberger, and Sharad Mehrotra. Relevance feedback: a power tool for interactive content-based image retrieval.IEEE Trans. Cir- cuits Syst. Video Technol., 8:644–655, 1998. 3

    1998

  42. [50]

    Bach, et al

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, et al. Multitask prompted training enables zero-shot task generalization, 2021. 2, 4, 13

    2021

  43. [51]

    Towards diverse paragraph captioning for untrimmed videos

    Yuqing Song, Shizhe Chen, and Qin Jin. Towards diverse paragraph captioning for untrimmed videos. InIEEE Con- ference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 11245–11254. Com- puter Vision Foundation / IEEE, 2021. 2

    2021

  44. [52]

    Towards efficient and effective text- to-video retrieval with coarse-to-fine visual representation learning

    Kaibin Tian, Yanhua Cheng, Yi Liu, Xinglin Hou, Quan Chen, and Han Li. Towards efficient and effective text- to-video retrieval with coarse-to-fine visual representation learning. InAAAI Conference on Artificial Intelligence,

  45. [53]

    Unified coarse-to-fine alignment for video-text retrieval

    Ziyang Wang, Yi-Lin Sung, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Unified coarse-to-fine alignment for video-text retrieval. InIEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 2804–2815. IEEE, 2023. 2, 6

    2023

  46. [54]

    Willhunter: Interac- tive image retrieval with multilevel relevance measurement

    Hong Wu, Hanqing Lu, and Songde Ma. Willhunter: Interac- tive image retrieval with multilevel relevance measurement. InInternational Conference on Pattern Recognition, 2004. 3

    2004

  47. [55]

    MSR-VTT: A large video description dataset for bridging video and lan- guage

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. MSR-VTT: A large video description dataset for bridging video and lan- guage. In2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV , USA, June 27-30, 2016, pages 5288–5296. IEEE Computer Society,

    2016

  48. [56]

    Clip-vip: Adapting pre- trained image-text model to video-language alignment

    Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Rui Song, Houqiang Li, and Jiebo Luo. Clip-vip: Adapting pre- trained image-text model to video-language alignment. InIn- ternational Conference on Learning Representations, 2022. 2, 4, 5, 6

    2022

  49. [57]

    Cross-modal and hierarchical modeling of video and text

    Bowen Zhang, Hexiang Hu, and Fei Sha. Cross-modal and hierarchical modeling of video and text. InEuropean Con- ference on Computer Vision, 2018. 5

    2018

  50. [58]

    Object detection difficulty: Suppressing over-aggregation for faster and better video object detection

    Bingqing Zhang, Sen Wang, Yifan Liu, Brano Kusy, Xue Li, and Jiajun Liu. Object detection difficulty: Suppressing over-aggregation for faster and better video object detection. InProceedings of the 31st ACM International Conference on Multimedia, pages 1768–1778, 2023. 4

    2023

  51. [59]

    Tokenbinder: Text-video retrieval with one-to-many alignment paradigm

    Bingqing Zhang, Zhuo Cao, Heming Du, Xin Yu, Xue Li, Jiajun Liu, and Sen Wang. Tokenbinder: Text-video retrieval with one-to-many alignment paradigm. InProceedings of the Winter Conference on Applications of Computer Vision (WACV), pages 4957–4967, 2025. 2

    2025

  52. [60]

    Cen- terclip: Token clustering for efficient text-video retrieval

    Shuai Zhao, Linchao Zhu, Xiaohan Wang, and Yi Yang. Cen- terclip: Token clustering for efficient text-video retrieval. InSIGIR ’22: The 45th International ACM SIGIR Confer- ence on Research and Development in Information Retrieval, Madrid, Spain, July 11 - 15, 2022, pages 970–...

    2022

  53. [61]

    Continual text-to-video retrieval with frame fusion and task-aware routing

    Zecheng Zhao, Zhi Chen, Zi Huang, Shazia Sadiq, and Tong Chen. Continual text-to-video retrieval with frame fusion and task-aware routing. InSIGIR 2025, 2025. 2

    2025

  54. [62]

    Are synthetic videos useful? a bench- mark for retrieval-centric evaluation of synthetic videos

    Zecheng Zhao, Selena Song, Tong Chen, Zhi Chen, Shazia Sadiq, and Yadan Luo. Are synthetic videos useful? a bench- mark for retrieval-centric evaluation of synthetic videos. arXiv preprint arXiv:2507.02316, 2025. 2

    2025 arXiv

  55. [63]

    Languagebind: Extending video-language pretraining to n- modality by language-based semantic alignment

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, Hongfa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zong- wei Li, Caiwan Zhang, Zhifeng Li, Wei Liu, and Li Yuan. Languagebind: Extending video-language pretraining to n- modality by language-based semantic alignment. InThe T...

    2024

  56. [64]

    Flow-guided feature aggregation for video object detec- tion

    Xizhou Zhu, Yujie Wang, Jifeng Dai, Lu Yuan, and Yichen Wei. Flow-guided feature aggregation for video object detec- tion. InIEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 408–

    2017

  57. [66]

    Explanation of UMIVR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

  58. [67]

    Reproduction of IVR Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

  59. [68]

    Analysis of Uncertainty Score Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

  60. [69]

    Discussion on Quantization Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

  61. [70]

    Early Stopping Strategy for Interactive Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

  62. [71]

    Impact of NR-IQA Methods in TQFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

  63. [72]

    Prompt Design for UMIVR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

  64. [73]

    Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 A. Explanation of UMIVR Uncertainty Reduction Theory (URT) [3], origin...

  65. [74]

    User Query:{cur text query}

    First you need to evaluate whether the user’s query includes sufficient visual details (such as characters, colors, objects, or locations). User Query:{cur text query}

  66. [75]

    Ask a question - If details are missing, generate one question to gather them. - If the query is already detailed, generate a clarifying question to further enrich the description (e.g., ’What other objects are present?’, ’What is the main color?’, or ’Where is the event takin...

  67. [417]

    4 11 Appendices Appendix Contents

    IEEE Computer Society, 2017. 4 11 Appendices Appendix Contents

    2017

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.