Pith. sign in

REVIEW 3 major objections 5 minor 68 references

MSAM: Multi-Semantic Adaptive Mining for Cross-Modal Drone Video-Text Retrieval

T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MSAM, a cross-modal retrieval model for drone video, represents each video and caption as a set of Gaussian semantic embeddings and pools frames with text guidance; the paper reports top results on two new drone video-text datasets.

desk verdict New task and two self-built datasets are the real contribution; the method is a plausible assembly, but the evaluation rests on random splits and unverified LLM captions, so the headline gains are not yet established. read the letter →

arxiv 2510.15470 v1 pith:K3LK3XTM submitted 2025-10-17 cs.CV cs.IR

Jinghao Huang , Yaxiong Chen , Ganchao Liu This is my paper
classification cs.CVcs.IR
keywords dronevideo-textretrievalcross-modalmulti-semanticlearningprobabilisticembeddingsdistribution-drivensemantictext-guidedvideopoolingaerialdatasets
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper identifies drone video-text retrieval (DVTR) as a distinct task whose overhead viewpoint creates repetitive scenes and ambiguous object configurations that ground-view video-text methods are not built for. To address it, the authors build two new datasets, USRD and UMCRD, with five captions per video, and propose MSAM. MSAM's core move is to replace single-vector video and text representations with k probabilistic embeddings, so one video can match many paraphrases of the same scene; a text-guided pooling mechanism (CIFFP) then suppresses frames unrelated to the query. In reported experiments, MSAM outperforms all 13 baselines on both datasets for both text-to-video and video-to-text retrieval, including text-to-video R@1 improvements from 48.0 to 49.5 on UMCRD and 28.6 to 29.9 on USRD.

What carries the argument

The load-bearing components are four. First, the adaptive semantic construction module maps text tokens and video frames to k mean/variance pairs, turning each sample into a Gaussian distribution. Second, the distribution-driven semantic learning term minimizes an asymmetric KL-like divergence between matched video and text distributions. Third, the diversity semantic term penalizes the Gram matrices of normalized embeddings for deviating from the identity matrix, keeping the k semantics decorrelated. Fourth, cross-modal interactive feature fusion pooling (CIFFP) computes frame-text attention, aggregates frames under text influence, applies a second text attention, and combines both branches

What would settle it

Check caption quality and split leakage directly: ask independent annotators to flag factual errors in a random sample of the GPT-generated captions, and compute nearest-frame similarity between training and test partitions to see whether frames from the same scene appear on both sides. Either finding would overturn the benchmark premise.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that drone-specific retrieval requires modelling semantic uncertainty rather than deterministic point matching. Each video and caption is converted into a set of probabilistic embeddings whose means and variances are aligned by a distribution-driven learning term, and an orthogonal diversity term keeps the k embeddings semantically distinct. The cross-modal interactive feature fusion pooling mechanism uses a two-stage attention over frame-text similarities to reconstruct video features, filtering out background frames. The paper reports that this combination beats every compared method on its two self-built datasets, and takes this as evidence t

Load-bearing premise

Everything hinges on the two new datasets honestly measuring retrieval: if the machine-generated captions contain systematic errors, or if frames from the same scene appear in both training and test sets, the reported gains overstate real-world performance.

Editorial extensions

If this is right

  • Dedicated drone retrieval mechanisms, rather than direct adaptation of ground-view models, can yield measurable gains on overhead footage.
  • Representing captions and videos as distributions rather than points helps when multiple videos share near-identical scenes yet have different valid descriptions.
  • Text-conditioned frame pooling can reduce the visual noise that average or self-attention pooling picks up in aerial video.
  • The two datasets give the community standard evaluation axes for a task that previously had none, assuming the promised code and data are released.
  • Each component contributes independently: CIFFP alone lifts the CLIP4Clip baseline on both datasets, and the semantic terms add further gains on the reported recall metrics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the random (non-scene-disjoint) split may let near-duplicate frames from the same flight appear in both training and test sets, so the reported margins could narrow when evaluated with geographically or flight-level disjoint splits.
  • Editorial inference: with LLM-generated captions, the benchmark inherits any systematic caption biases; human verification of a caption subsample would quantify this.
  • Editorial inference: the probabilistic-embedding mechanism is not drone-specific in principle, so the same distribution-alignment and diversity losses could transfer to other repetitive-scene retrieval problems, such as satellite or vehicle-mounted imagery.
  • Editorial inference: the visualization cases where a near-synonym query ranks a different video first (e.g., 'speedy' cars) suggest caption semantics, not visual features, may be the next bottleneck; richer captions could shift results.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces the drone video-text retrieval (DVTR) task, constructs two self-built datasets (USRD from ERA, UMCRD from YouTube plus manual filming), and proposes MSAM, consisting of a cross-modal interactive feature fusion pooling mechanism (CIFFP), probabilistic multi-semantic embeddings, a distribution-driven semantic loss, and a diversity term. The authors report improved retrieval over 13 baselines on both datasets, e.g., USRD text-to-video R@1 29.9 vs 28.6 for CLIP4Clip, and UMCRD text-to-video R@1 49.5 vs 48.0.

Significance. If the two datasets and their splits are valid benchmarks, the paper would provide useful resources and a reasonable baseline for a new task, and the extensive comparison against recent video-text retrieval methods is a strength. The method itself is largely a combination of established ideas: text-guided pooling (X-Pool) and probabilistic embeddings (PCME/UATVR), with an additional diversity regularizer. The main scientific value lies in the proposed DVTR evaluation setup and in whether the reported gains are reproducible on independent scene-disjoint splits. The manuscript ships no code or data yet and the equations for CIFFP are not implementable as printed, so at present the empirical claim is not independently verifiable.

major comments (3)
  1. [Section III-C, Eqs. (3)-(9)] The central CIFFP equations are not well-formed as printed. Eq. (3) contains a bare sum over D with no visible numerator; Eq. (4) multiplies a B×F×T attention score by a norm ||N_v||_F instead of the frame embeddings N_v; Eq. (5) produces a B×T×1 output from a D-dimensional sum; Eq. (9) writes the final similarity as a sum over D of a norm rather than as an explicit inner product. Since CIFFP is a key contribution and every subsequent objective uses S_vt, the method cannot be implemented or checked from the text. Rewrite all pooling equations with explicit tensors, summation axes, and output shapes.
  2. [Section IV-A and Tables II-III] The core empirical claim depends on unvalidated self-built benchmarks. The two datasets use random splits, captions are partly generated by GPT/Gemini, and there is no scene/event-level split, near-duplicate analysis, caption QA, or inter-annotator agreement. Because USRD clips come from ERA and UMCRD partly from YouTube, clips from the same scene or flight can easily appear in both training and test sets. The reported margins are small (Table II USRD t2v R@1 29.9 vs 28.6; Table III UMCRD 49.5 vs 48.0), and no standard deviations or multiple-seed results are reported. Without scene-disjoint splits and variance estimates, the claim that MSAM outperforms all baselines is not established.
  3. [Section IV-E, IV-F, IV-G, Fig. 5, Tables V-VII] Hyperparameters λ, k, and the frame count appear to be selected by inspecting test-set R@K, with no separate validation protocol or seed averaging. Fig. 5 reports λ=0.01 as the best value but λ=0.1 is used in the final model; Table V selects k=7 from test R@1 differences of about one point. This selection procedure can create optimistic results and makes the reported gains weak evidence for the method. Report hyperparameter selection on a held-out validation split (or averaged over multiple splits/seeds) and give mean ± std for the final configuration.
minor comments (5)
  1. [Section III-D, Eqs. (17)-(20)] T_sigma and V_sigma are D-dimensional, but Eq. (20) is written as a scalar expression; specify whether the loss is summed or averaged over dimensions. Also clarify whether F_mu and F_sigma are shared between modalities; Eq. (17) appears to feed both p_i_t and p_i_v into a single function for each output.
  2. [Section IV-A] The text first says captions were obtained using 'the GPT large model' and then says the Gemini API was used, citing ShareGPT4V. The inconsistency should be corrected and the prompt design / manual annotation instructions described in enough detail for replication.
  3. [Section IV-D, Table IV] The ablation discussion says adding L_dst slightly decreases R@1 and R@5 on USRD, but Table IV shows increases from 29.7/70.9 (++L_ddsl) to 29.9/71.2 (+++L_dst). On UMCRD, R@1 decreases from 49.8 to 49.5. Please reconcile the text with the table and explain the non-monotonic behavior.
  4. [Figures and typos] Fig. 5 lacks axis labels and a legend, and the caption 'Visual analysis of the differences parameter λ' is unclear. Typos include 'Comparism' in Fig. 1, 'presentes' in Section I, 'a orange-roofed' in Fig. 1, and the heading 'T otal Objective Term.'
  5. [General reproducibility] Give the exact CLIP checkpoint, training epochs, number of runs, and the evaluation protocol (e.g., whether metrics are over the full test set). The statement that code and datasets 'will be made publicly available' is not yet a release; please provide a reproducibility statement with the actual links or an ethics/data release plan.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: MSAM's loss and comparisons are not derived from their own outputs; self-built dataset concerns are benchmark-validity issues, not derivation circularity.

full rationale

MSAM's objective (Eq. 24, L = L_vtm + L_ddsl + lambda*L_dst) is built from a contrastive matching term (Eqs. 10-12), a distribution-alignment term (Eq. 20), and a diversity term (Eqs. 21-23). Each term operates on CLIP-extracted text/video features and is optimized with standard cross-entropy and KL-style losses; none of these quantities is defined in terms of, or fitted to, the test-set R@K it later reports. The reported gains are direct comparisons against 13 published baselines initialized from the same CLIP ViT-B/32 weights, so the ranking is not forced by construction. The paper does cite two of the authors' earlier drone-image retrieval papers ([14], [15]) in the introduction, but these citations only contextualize prior drone-video analysis and are not the load-bearing support for the MSAM claim. The strongest concern is the evaluation setting: USRD and UMCRD are self-built, captions are partly generated with the GPT/Gemini API, and splits are random, so external validity and possible scene-level leakage are unverified. That is a benchmark-integrity and correctness risk, not a circularity in the derivation chain: the ground-truth pairs and test queries are not produced by the model or by a parameter fitted to the target result, and no equation in the paper reduces a predicted retrieval score to its training input. Under the rubric requiring a specific reduction (Eq. X = Eq. Y by construction, or a fitted parameter renamed as a prediction), no circular step is present.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central empirical claim rests on the validity of two self-constructed, LLM-captioned datasets; the model also depends on CLIP's pretrained alignment transferring to overhead views and on a Gaussian probabilistic-embedding assumption (Eq. 19). Tuned hyperparameters λ, k, and frame count are selected on the same evaluation data, reducing the independence of the reported gains.

free parameters (3)
  • λ (trade-off for diversity term) = 0.1 (final); 0.01 best in Fig. 5
    Section IV-E: Fig. 5 shows λ=0.01 yields highest R@K, yet the final model uses λ=0.1; the choice is justified qualitatively, not by the plotted curve.
  • k (number of probabilistic embeddings) = 7
    Section IV-F/Table V: k=7 selected because t2v R@K stabilizes; k is tuned on the USRD evaluation data.
  • Number of sampled frames per video = 12
    Section IV-G/Tables VI-VII: 12 frames chosen as a plateau in R@K; 16-20 frames degrade UMCRD performance.
assumptions (3)
  • domain assumption CLIP ViT-B/32 pretrained text-image alignment transfers to overhead drone video frames.
    Section III-B encodes all frames and texts with CLIP; if the pretrained model cannot represent aerial scene structure, the entire pipeline is weakened. All comparisons share this assumption, but it is not validated.
  • ad hoc to paper Each drone video and text is modeled as k independent Gaussian distributions (Eq. 19).
    Eq. 19 defines p(z|t)=N(T_μ,T_σ^2) and p(z|v)=N(V_μ,V_σ^2); this probabilistic modeling assumption is introduced for the distribution-driven loss and is not derived from the data.
  • domain assumption Self-built, LLM-captioned datasets are a valid and leakage-free evaluation benchmark.
    Section IV-A uses random splits on self-built data with GPT/Gemini captions; no scene-level split, inter-annotator agreement, or caption-quality audit is provided. If false, reported R@K overstates generalization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MSAM: Multi-Semantic Adaptive Mining for Cross-Modal Drone Video-Text Retrieval." pith.science (2026). https://pith.science/paper/K3LK3XTM

@misc{pith2026251015470,
  author       = {Pith},
  title        = {Pith review of: MSAM: Multi-Semantic Adaptive Mining for Cross-Modal Drone Video-Text Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K3LK3XTM}},
  note         = {Machine review of arXiv:2510.15470}
}
read the original abstract

With the advancement of drone technology, the volume of video data increases rapidly, creating an urgent need for efficient semantic retrieval. We are the first to systematically propose and study the drone video-text retrieval (DVTR) task. Drone videos feature overhead perspectives, strong structural homogeneity, and diverse semantic expressions of target combinations, which challenge existing cross-modal methods designed for ground-level views in effectively modeling their characteristics. Therefore, dedicated retrieval mechanisms tailored for drone scenarios are necessary. To address this issue, we propose a novel approach called Multi-Semantic Adaptive Mining (MSAM). MSAM introduces a multi-semantic adaptive learning mechanism, which incorporates dynamic changes between frames and extracts rich semantic information from specific scene regions, thereby enhancing the deep understanding and reasoning of drone video content. This method relies on fine-grained interactions between words and drone video frames, integrating an adaptive semantic construction module, a distribution-driven semantic learning term and a diversity semantic term to deepen the interaction between text and drone video modalities and improve the robustness of feature representation. To reduce the interference of complex backgrounds in drone videos, we introduce a cross-modal interactive feature fusion pooling mechanism that focuses on feature extraction and matching in target regions, minimizing noise effects. Extensive experiments on two self-constructed drone video-text datasets show that MSAM outperforms other existing methods in the drone video-text retrieval task. The source code and dataset will be made publicly available.

Figures

Figures reproduced from arXiv: 2510.15470 by the authors.

Figure 1
Figure 1. Top row: Comparism of common video-text match￾ing and drone video-text matching. Bottom row: The video frames from the same scene in the drone dataset illustrate the strong structural homogeneity of drone videos, while the target combinations within a single video exhibit semantic diversity in their descriptions. streaming [9]. Furthermore, drones are not limited by weather conditions like cloud cover, granting them… view at source ↗
Figure 2
Figure 2. The overall overview of the MSAM framework is as follows. For a given drone video-text pair [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. This illustrates MSALM mechanism. The three dif [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Left column: visualization of the USRD drone video-text dataset and UMCRD drone video-text dataset. Right column: visualization word cloud of the (b) USRD dataset and (c) UMCRD dataset. where Ψ(x) denotes the attention mechanism [58]. LN (x) denotes LayerNorm and Ff (x…
Figure 5
Figure 5. Figure 5: Visual analysis of the differences parameter [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: The differences between k video and text embed￾dings generated by MSALM are visually analyzed using mean and variance. The legend illustrates their central tendency and variability. F. Analysis of Multi-Semantic Adaptive Learning Mechanism The MSALM increases the numbe…
Figure 7
Figure 7. Figure 7: The text-to-video retrieval results on the USRD (a-f) and UMCRD (e-h) datasets are presented. The query descriptions [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: The visualization displays the heatmap relationship between the query text and the intermediate frames of the [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

68 extracted references · 12 linked inside Pith

  1. [1]

    Overview and current status of remote sensing applications based on unmanned aerial vehicles (uavs),

    G. Pajares, “Overview and current status of remote sensing applications based on unmanned aerial vehicles (uavs),”Photogrammetric Engineer- ing & Remote Sensing, vol. 81, no. 4, pp. 281–329, 2015

    2015

  2. [2]

    Unmanned aerial systems for photogram- metry and remote sensing: A review - sciencedirect,

    I. Colomina and P. Molina, “Unmanned aerial systems for photogram- metry and remote sensing: A review - sciencedirect,”ISPRS Journal of Photogrammetry and Remote Sensing, vol. 92, no. 2, pp. 79–97, 2014

    2014

  3. [3]

    Uavs challenge to assess water stress for sustainable agriculture,

    J. Gago, C. Douthe, R. E. Coopman, P. P. Gallego, M. Ribas-Carbo, J. Flexas, J. Escalona, and H. Medrano, “Uavs challenge to assess water stress for sustainable agriculture,”Agricultural Water Management, vol. 153, pp. 9–19, 2015

    2015

  4. [4]

    Sustainable agriculture by increasing nitrogen fertilizer efficiency using low-resolution camera mounted on unmanned aerial vehicles

    Dong-Wook, Kim, Tae-Sun, Min, Yoonha, Renato, Rodrigues, Silva, Hae-Nam, and H. and, “Sustainable agriculture by increasing nitrogen fertilizer efficiency using low-resolution camera mounted on unmanned aerial vehicles.”International journal of environmental research and public health, vol. 16, no. 20, 2019

    2019

  5. [5]

    V . C. Moulianitis, G. Thanellas, N. Xanthopoulos, and N. A. Aspra- gathos,Evaluation of UAV Based Schemes for Forest Fire Monitoring, 2019

    2019

  6. [6]

    Visible-thermal UA V tracking: A large-scale benchmark and new baseline,

    P. Zhang, J. Zhao, D. Wang, H. Lu, and X. Ruan, “Visible-thermal UA V tracking: A large-scale benchmark and new baseline,” inComputer Vision and Pattern Recognition, 2022, pp. 8876–8885

    2022

  7. [7]

    High-resolution feature pyramid network for small object detection on drone view,

    Z. Chen, H. Ji, Y . Zhang, Z. Zhu, and Y . Li, “High-resolution feature pyramid network for small object detection on drone view,”IEEE Trans. Circuits Syst. Video Technol., vol. 34, no. 1, pp. 475–489, 2024

    2024

  8. [8]

    Earthnets: Empowering AI in earth observation,

    Z. Xiong, F. Zhang, Y . Wang, Y . Shi, and X. X. Zhu, “Earthnets: Empowering AI in earth observation,”CoRR, vol. abs/2210.04936, 2022

    arXiv 2022

Show all 68 references
  1. [9]

    Sdanet: Semantic- embedded density adaptive network for moving vehicle detection in satellite videos,

    J. Feng, Y . Liang, X. Zhang, J. Zhang, and L. Jiao, “Sdanet: Semantic- embedded density adaptive network for moving vehicle detection in satellite videos,”IEEE Trans. Image Process., vol. 32, pp. 1788–1801, 2023

    2023

  2. [10]

    Pareto refocusing for drone-view object detection,

    J. Leng, M. Mo, Y . Zhou, C. Gao, W. Li, and X. Gao, “Pareto refocusing for drone-view object detection,”IEEE Trans. Circuits Syst. Video Technol., vol. 33, no. 3, pp. 1320–1334, 2023

    2023

  3. [11]

    Temporal-spatial feature interac- tion network for multi-drone multi-object tracking,

    H. Wu, H. Sun, K. Ji, and G. Kuang, “Temporal-spatial feature interac- tion network for multi-drone multi-object tracking,”IEEE Trans. Circuits Syst. Video Technol., vol. 35, no. 2, pp. 1165–1179, 2025

    2025

  4. [12]

    Cross-drone transformer network for robust single object tracking,

    G. Chen, P. Zhu, B. Cao, X. Wang, and Q. Hu, “Cross-drone transformer network for robust single object tracking,”IEEE Trans. Circuits Syst. Video Technol., vol. 33, no. 9, pp. 4552–4563, 2023

    2023

  5. [13]

    Transformer- based spatio-temporal unsupervised traffic anomaly detection in aerial videos,

    T. M. Tran, D. C. Bui, T. V . Nguyen, and K. Nguyen, “Transformer- based spatio-temporal unsupervised traffic anomaly detection in aerial videos,”IEEE Trans. Circuits Syst. Video Technol., vol. 34, no. 9, pp. 8292–8309, 2024

    2024

  6. [14]

    Visual contextual semantic reasoning for cross-modal drone image-text retrieval,

    J. Huang, Y . Chen, S. Xiong, and X. Lu, “Visual contextual semantic reasoning for cross-modal drone image-text retrieval,”IEEE Trans. Geosci. Remote. Sens., vol. 62, pp. 1–12, 2024

    2024

  7. [15]

    Deep saliency smoothing hashing for drone image retrieval,

    Y . Chen, J. Huang, L. Mou, P. Jin, S. Xiong, and X. X. Zhu, “Deep saliency smoothing hashing for drone image retrieval,”IEEE Trans. Geosci. Remote. Sens., vol. 61, pp. 1–13, 2023

    2023

  8. [16]

    Uav-human: A large benchmark for human behavior understanding with unmanned aerial vehicles,

    T. Li, J. Liu, W. Zhang, Y . Ni, W. Wang, and Z. Li, “Uav-human: A large benchmark for human behavior understanding with unmanned aerial vehicles,” inComputer Vision and Pattern Recognition, 2021, pp. 16 266–16 275

    2021

  9. [17]

    Multi-modal transformer for video retrieval,

    V . Gabeur, C. Sun, K. Alahari, and C. Schmid, “Multi-modal transformer for video retrieval,” inEuropean Conference on Computer Vision, 2020, pp. 214–229

    2020

  10. [18]

    Towards video anomaly retrieval from video anomaly detection: New benchmarks and model,

    P. Wu, J. Liu, X. He, Y . Peng, P. Wang, and Y . Zhang, “Towards video anomaly retrieval from video anomaly detection: New benchmarks and model,”CoRR, vol. abs/2307.12545, 2023

    2023 arXiv

  11. [19]

    Multilevel semantic interaction alignment for video-text cross-modal retrieval,

    L. Chen, Z. Deng, L. Liu, and S. Yin, “Multilevel semantic interaction alignment for video-text cross-modal retrieval,”IEEE Trans. Circuits Syst. Video Technol., vol. 34, no. 7, pp. 6559–6575, 2024

    2024

  12. [20]

    Videoclip: Contrastive pre- training for zero-shot video-text understanding,

    H. Xu, G. Ghosh, P. Huang, D. Okhonko, A. Aghajanyan, F. Metze, L. Zettlemoyer, and C. Feichtenhofer, “Videoclip: Contrastive pre- training for zero-shot video-text understanding,” inEmpirical Methods in Natural Language Processing, 2021, pp. 6787–6800

    2021

  13. [21]

    T2VLAD: global-local sequence alignment for text-video retrieval,

    X. Wang, L. Zhu, and Y . Yang, “T2VLAD: global-local sequence alignment for text-video retrieval,” inComputer Vision and Pattern Recognition, 2021, pp. 5079–5088

    2021

  14. [22]

    Clip4clip: An empirical study of CLIP for end to end video clip retrieval,

    H. Luo, L. Ji, M. Zhong, Y . Chen, W. Lei, N. Duan, and T. Li, “Clip4clip: An empirical study of CLIP for end to end video clip retrieval,”CoRR, vol. abs/2104.08860, 2021

    2021 arXiv

  15. [23]

    Centerclip: Token clustering for efficient text-video retrieval,

    S. Zhao, L. Zhu, X. Wang, and Y . Yang, “Centerclip: Token clustering for efficient text-video retrieval,” inACM SIGIR Conference on Research and Development in Information Retrieval, 2022, pp. 970–981

    2022

  16. [24]

    X-pool: Cross-modal language-video attention for text-video retrieval,

    S. K. Gorti, N. V ouitsis, J. Ma, K. Golestan, M. V olkovs, A. Garg, and G. Yu, “X-pool: Cross-modal language-video attention for text-video retrieval,” inComputer Vision and Pattern Recognition, 2022, pp. 4996– 5005

    2022

  17. [25]

    X-CLIP: end-to- end multi-grained contrastive learning for video-text retrieval,

    Y . Ma, G. Xu, X. Sun, M. Yan, J. Zhang, and R. Ji, “X-CLIP: end-to- end multi-grained contrastive learning for video-text retrieval,” inACM International Conference on Multimedia, 2022, pp. 638–647

    2022

  18. [26]

    Less is more: Clipbert for video-and-language learning via sparse sampling,

    J. Lei, L. Li, L. Zhou, Z. Gan, T. L. Berg, M. Bansal, and J. Liu, “Less is more: Clipbert for video-and-language learning via sparse sampling,” inComputer Vision and Pattern Recognition, 2021, pp. 7331–7341

    2021

  19. [27]

    Frozen in time: A joint video and image encoder for end-to-end retrieval,

    M. Bain, A. Nagrani, G. Varol, and A. Zisserman, “Frozen in time: A joint video and image encoder for end-to-end retrieval,” inInternational Conference on Computer Vision, 2021, pp. 1708–1718

    2021

  20. [28]

    Align and tell: Boosting text-video retrieval with local alignment and fine-grained supervision,

    X. Wang, L. Zhu, Z. Zheng, M. Xu, and Y . Yang, “Align and tell: Boosting text-video retrieval with local alignment and fine-grained supervision,”IEEE Trans. Multim., vol. 25, pp. 6079–6089, 2023

    2023

  21. [29]

    Ts2-net: Token shift and selection transformer for text-video retrieval,

    Y . Liu, P. Xiong, L. Xu, S. Cao, and Q. Jin, “Ts2-net: Token shift and selection transformer for text-video retrieval,” inEuropean Conference on Computer Vision, 2022, pp. 319–335

    2022

  22. [30]

    K. P. Murphy,Machine learning - a probabilistic perspective. MIT Press, 2012

    2012

  23. [31]

    Modeling uncertainty with hedged instance embeddings,

    S. J. Oh, K. P. Murphy, J. Pan, J. Roth, F. Schroff, and A. C. Gallagher, “Modeling uncertainty with hedged instance embeddings,” inInternational Conference on Learning Representations, 2019

    2019

  24. [32]

    Probabilistic face embeddings,

    Y . Shi and A. K. Jain, “Probabilistic face embeddings,” inInternational Conference on Computer Vision, 2019, pp. 6901–6910

    2019

  25. [33]

    Data uncertainty learning in face recognition,

    J. Chang, Z. Lan, C. Cheng, and Y . Wei, “Data uncertainty learning in face recognition,” inComputer Vision and Pattern Recognition, 2020, pp. 5709–5718

    2020

  26. [34]

    View- invariant probabilistic embedding for human pose,

    J. J. Sun, J. Zhao, L. Chen, F. Schroff, H. Adam, and T. Liu, “View- invariant probabilistic embedding for human pose,” inEuropean Con- ference on Computer Vision, 2020, pp. 53–70. 14

    2020

  27. [35]

    Probabilistic embeddings for cross-modal retrieval,

    S. Chun, S. J. Oh, R. S. de Rezende, Y . Kalantidis, and D. Larlus, “Probabilistic embeddings for cross-modal retrieval,” inComputer Vision and Pattern Recognition, 2021, pp. 8415–8424

    2021

  28. [36]

    UATVR: uncertainty-adaptive text-video retrieval,

    B. Fang, W. Wu, C. Liu, Y . Zhou, Y . Song, W. Wang, X. Shu, X. Ji, and J. Wang, “UATVR: uncertainty-adaptive text-video retrieval,” in International Conference on Computer Vision, 2023, pp. 13 677–13 687

    2023

  29. [37]

    Fame-vil: Multi-tasking vision-language model for heterogeneous fashion tasks,

    X. Han, X. Zhu, L. Yu, L. Zhang, Y . Song, and T. Xiang, “Fame-vil: Multi-tasking vision-language model for heterogeneous fashion tasks,” CoRR, vol. abs/2303.02483, 2023

    2023 arXiv

  30. [38]

    Position-guided text prompt for vision-language pre-training,

    A. J. Wang, P. Zhou, M. Z. Shou, and S. Yan, “Position-guided text prompt for vision-language pre-training,”CoRR, vol. abs/2212.09737, 2022

    2022 arXiv

  31. [39]

    Multi-modal representation learning with text- driven soft masks,

    J. Park and B. Han, “Multi-modal representation learning with text- driven soft masks,”CoRR, vol. abs/2304.00719, 2023

    2023 arXiv

  32. [40]

    GALIP: generative adversarial clips for text-to-image synthesis,

    M. Tao, B. Bao, H. Tang, and C. Xu, “GALIP: generative adversarial clips for text-to-image synthesis,”CoRR, vol. abs/2301.12959, 2023

    2023 arXiv

  33. [41]

    MAGE: masked generative encoder to unify representation learning and image synthesis,

    T. Li, H. Chang, S. K. Mishra, H. Zhang, D. Katabi, and D. Krishnan, “MAGE: masked generative encoder to unify representation learning and image synthesis,”CoRR, vol. abs/2211.09117, 2022

    2022 arXiv

  34. [42]

    Towards accurate text-based image captioning with content diversity exploration,

    G. Xu, S. Niu, M. Tan, Y . Luo, Q. Du, and Q. Wu, “Towards accurate text-based image captioning with content diversity exploration,” in Computer Vision and Pattern Recognition, 2021, pp. 12 637–12 646

    2021

  35. [43]

    Show and tell: A neural image caption generator,

    O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” inComputer Vision and Pattern Recognition, 2015, pp. 3156–3164

    2015

  36. [44]

    Deep visual-semantic alignments for generating image descriptions,

    A. Karpathy and L. Fei-Fei, “Deep visual-semantic alignments for generating image descriptions,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 4, pp. 664–676, 2017

    2017

  37. [45]

    Visual semantic reasoning for image-text matching,

    K. Li, Y . Zhang, K. Li, Y . Li, and Y . Fu, “Visual semantic reasoning for image-text matching,” inInternational Conference on Computer Vision, 2019, pp. 4653–4661

    2019

  38. [46]

    Conditional prompt learning for vision-language models,

    K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Conditional prompt learning for vision-language models,” inComputer Vision and Pattern Recogni- tion, 2022, pp. 16 795–16 804

    2022

  39. [47]

    Bridging video-text retrieval with multiple choice questions,

    Y . Ge, Y . Ge, X. Liu, D. Li, Y . Shan, X. Qie, and P. Luo, “Bridging video-text retrieval with multiple choice questions,” inComputer Vision and Pattern Recognition, 2022, pp. 16 146–16 155

    2022

  40. [48]

    Visual abductive reasoning,

    C. Liang, W. Wang, T. Zhou, and Y . Yang, “Visual abductive reasoning,” inComputer Vision and Pattern Recognition, 2022, pp. 15 544–15 554

    2022

  41. [49]

    Membridge: Video-language pre-training with memory- augmented inter-modality bridge,

    J. Yang, X. Li, M. Zheng, Z. Wang, Y . Zhu, X. Guo, Y . Yuan, Z. Chai, and S. Jiang, “Membridge: Video-language pre-training with memory- augmented inter-modality bridge,”IEEE Trans. Image Process., vol. 32, pp. 4073–4087, 2023

    2023

  42. [50]

    A straightforward framework for video retrieval using CLIP,

    J. A. Portillo-Quintero, J. C. Ortiz-Bayliss, and H. Terashima-Mar ´ın, “A straightforward framework for video retrieval using CLIP,” inPattern Recognitio, vol. 12725, 2021, pp. 3–12

    2021

  43. [51]

    Hisa: Hierarchically semantic associating for video temporal grounding,

    Z. Xu, D. Chen, K. Wei, C. Deng, and H. Xue, “Hisa: Hierarchically semantic associating for video temporal grounding,”IEEE Trans. Image Process., vol. 31, pp. 5178–5188, 2022

    2022

  44. [52]

    Concept-aware video captioning: Describing videos with effective prior information,

    B. Yang, M. Cao, and Y . Zou, “Concept-aware video captioning: Describing videos with effective prior information,”IEEE Trans. Image Process., vol. 32, pp. 5366–5378, 2023

    2023

  45. [53]

    Hierarchical representation network with auxiliary tasks for video captioning and video question answering,

    L. Gao, Y . Lei, P. Zeng, J. Song, M. Wang, and H. T. Shen, “Hierarchical representation network with auxiliary tasks for video captioning and video question answering,”IEEE Trans. Image Process., vol. 31, pp. 202–215, 2022

    2022

  46. [54]

    Cross-attentional spatio-temporal semantic graph networks for video question answering,

    Y . Liu, X. Zhang, F. Huang, B. Zhang, and Z. Li, “Cross-attentional spatio-temporal semantic graph networks for video question answering,” IEEE Trans. Image Process., vol. 31, pp. 1684–1696, 2022

    2022

  47. [55]

    Adaptive spatio- temporal graph enhanced vision-language representation for video QA,

    W. Jin, Z. Zhao, X. Cao, J. Zhu, X. He, and Y . Zhuang, “Adaptive spatio- temporal graph enhanced vision-language representation for video QA,” IEEE Trans. Image Process., vol. 30, pp. 5477–5489, 2021

    2021

  48. [56]

    Exploring language hierarchy for video grounding,

    X. Ding, N. Wang, S. Zhang, Z. Huang, X. Li, M. Tang, T. Liu, and X. Gao, “Exploring language hierarchy for video grounding,”IEEE Trans. Image Process., vol. 31, pp. 4693–4706, 2022

    2022

  49. [57]

    Layer normalization,

    L. J. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,”CoRR, vol. abs/1607.06450, 2016

    2016 arXiv

  50. [58]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2017

    2017

  51. [59]

    ERA: A dataset and deep learning benchmark for event recognition in aerial videos,

    L. Mou, Y . Hua, P. Jin, and X. X. Zhu, “ERA: A dataset and deep learning benchmark for event recognition in aerial videos,”CoRR, vol. abs/2001.11394, 2020

    2001 arXiv

  52. [60]

    Sharegpt4v: Improving large multi-modal models with better captions,

    L. Chen, J. Li, X. Dong, P. Zhang, C. He, J. Wang, F. Zhao, and D. Lin, “Sharegpt4v: Improving large multi-modal models with better captions,” CoRR, vol. abs/2311.12793, 2023

    2023 arXiv

  53. [61]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” inInternational Conference on Learning Representations, 2019

    2019

  54. [62]

    SGDR: stochastic gradient descent with warm restarts,

    ——, “SGDR: stochastic gradient descent with warm restarts,” in International Conference on Learning Representations, 2017

    2017

  55. [63]

    Disentangled repre- sentation learning for text-video retrieval,

    Q. Wang, Y . Zhang, Y . Zheng, P. Pan, and X. Hua, “Disentangled repre- sentation learning for text-video retrieval,”CoRR, vol. abs/2203.07111, 2022

    2022 arXiv

  56. [64]

    Unified coarse-to-fine alignment for video-text retrieval,

    Z. Wang, Y . Sung, F. Cheng, G. Bertasius, and M. Bansal, “Unified coarse-to-fine alignment for video-text retrieval,” inInternational Con- ference on Computer Vision, 2023, pp. 2804–2815

    2023

  57. [65]

    Text is MASS: modeling as stochastic embedding for text- video retrieval,

    J. Wang, P. Wang, G. Sun, D. Liu, S. A. Dianat, R. Rao, M. Rabbani, and Z. Tao, “Text is MASS: modeling as stochastic embedding for text- video retrieval,” inComputer Vision and Pattern Recognition, 2024, pp. 16 551–16 560

    2024

  58. [66]

    DGL: dynamic global-local prompt tuning for text-video retrieval,

    X. Yang, L. Zhu, X. Wang, and Y . Yang, “DGL: dynamic global-local prompt tuning for text-video retrieval,” inAAAI Conference on Artificial Intelligence, 2024, pp. 6540–6548

    2024

  59. [67]

    Text-video retrieval with global-localsemantic consistent learning,

    H. Zhang, P. Zeng, L. Gao, J. Song, Y . Duan, X. Lyu, and H. T. Shen, “Text-video retrieval with global-localsemantic consistent learning,” IEEE Trans. Image Process., vol. 34, pp. 3463–3474, 2025

    2025

  60. [68]

    Tempme: Video temporal token merging for efficient text-video re- trieval,

    L. Shen, T. Hao, T. He, S. Zhao, Y . Zhang, P. Liu, Y . Bao, and G. Ding, “Tempme: Video temporal token merging for efficient text-video re- trieval,” inInternational Conference on Learning Representations, 2025

    2025

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.