Pith. sign in

REVIEW 3 major objections 7 minor 54 references

Shared prototypes lift text-video retrieval across four benchmarks

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 00:39 UTC pith:2AB5RH6J

load-bearing objection Solid incremental TVR paper with thorough ablations but an overstated abstract and an underspecified DPC-KNN backprop; deserves review, not desk reject. the 3 major comments →

arxiv 2608.00551 v1 pith:2AB5RH6J submitted 2026-08-01 cs.IR

PHA-Net: Prototype-based Hierarchical Alignment Network for Text-Video Retrieval

Xiaolun Jing , Kezhao Yin , Xinxing Yang , Genke Yang , Jian Chu This is my paper
classification cs.IR
keywords text-video retrievalhierarchical alignmentprototype learningmodality-shared prototypestoken mergingDPC-KNNprototype contrastive lossself-distillation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

PHA-Net sets out to fix a mismatch in text-video retrieval: a short query names a few concepts while a video carries far more visual detail. The paper's answer is a set of trainable, modality-shared prototypes that act as semantic bridges, letting one model align words-to-frames, phrases-to-clips, and sentence-to-video without stacking expensive cross-attention modules. On four standard benchmarks, PHA-Net reports higher recall sums than its hierarchical baseline, with the largest gains on ActivityNet (19.2 points) and MSR-VTT (8.8 points), and roughly one hour less of training time. The central bet is that the prototypes carry the cross-modal semantics, so the model can be trained end-to-end and still stay fast at inference.

Core claim

The paper claims that a single set of learnable prototypes shared by text and video can organize hierarchical alignment at three granularities: individual tokens (words and frames), local groups (phrases and clips), and the global sentence-video pair. Prototypes are concatenated with tokens and passed through a lightweight transformer; after clustering tokens into phrase/clip groups, the prototypes are repeated to act as attention queries that strengthen semantically strong tokens and suppress weak ones. A prototype contrastive loss keeps text and video prototypes distinct, and a self-distillation loss aligns the three levels' similarity distributions. The authors report that this combinatio

What carries the argument

Modality-shared learnable prototypes: random vectors initialized once, concatenated with both text and video token sequences, then refined by a shared lightweight transformer. At the local level, a prototype-supported token merge module uses DPC-KNN clustering to group tokens into phrase/clip groups, then treats the averaged prototypes as attention queries over the clustered tokens, so the prototypes decide which semantics survive the merge. The prototype contrastive loss and self-distillation tie the levels together during training; at inference the three similarity scores are simply weighted and summed.

Load-bearing premise

The central load-bearing premise is that the DPC-KNN clustering can be made end-to-end trainable through the described forward structural masking and backward attentional feature propagation; the paper asserts this without a formal gradient analysis or a citation, and if the clustering is not truly differentiable, or if training and test clusterings diverge, the reported retrieval gains lose their foundation.

What would settle it

Run the model with DPC-KNN replaced by a fixed, randomly initialized clustering with frozen masks and compare MSR-VTT SumR to the reported 414.1; if the gap is small, the clustering is not what carries the gain. More directly, compute gradients of the loss with respect to the DPC-KNN cluster assignments: if they are zero or the backward pass is not implemented, the end-to-end training claim is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the reported numbers hold, a retrieval model can align fine and coarse semantics with one shared prototype set instead of one cross-attention module per level, lowering inference cost.
  • The prototype contrastive loss gives a general recipe: keep shared prototypes semantically diverse, which should transfer to other cross-modal retrieval settings.
  • The 3:1 individual-to-local prototype ratio suggests a design rule: give lower levels more prototypes because words/frames carry more distinct attributes than merged phrases/clips.
  • With a stronger CLIP backbone and the DSL inference strategy, the same architecture scales: SumR rises to 423.4 with ViT-B/16 and 434.9 with DSL, so gains are not tied to one backbone.
  • The self-distillation term smooths differences among level similarities; its absence drops text-to-video R@5 by 2.1 points in the ablation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • [Editorial inference] The authors do not analyze the gradient path through DPC-KNN; if the clustering is effectively non-differentiable, the reported end-to-end gains could come from the surrounding attention and loss terms rather than from learning the clusters. A testable extension is to compare against a version with fixed random cluster assignments.
  • [Editorial inference] Because the prototype count is static (3 individual, 1 local), performance saturates and then drops as prototypes are added; the paper notes this and suggests dynamic prototypes as future work. One could expect content-adaptive prototype counts to help datasets with highly diverse captions like VATEX, where the gain is only 0.7 points.
  • [Editorial inference] The failure analysis shows the model misses quantity attributes ('number of girls') and smooths sharp scene transitions; extending the prototype set with attribute-specific prototypes or removing the 1D-Conv temporal smoothing might recover those cases.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. PHA-Net is a CLIP-based text-video retrieval model that adds modality-shared learnable prototypes at individual and local levels, a prototype-supported token merge module (1D convolutions, DPC-KNN clustering, and cross-attention with prototype queries), a prototype contrastive loss, and self-distillation between hierarchy levels. The training objective is a weighted combination of contrastive losses at three levels plus auxiliary losses. On MSR-VTT, ActivityNet, VATEX, and Charades, the paper reports higher SumR than the HBI baseline and most prior methods, and lower training time than competing hierarchical methods.

Significance. The contribution is potentially useful: the prototype bridge is a plausible way to reduce the cost of hierarchical cross-modal alignment while retaining multi-granularity matching. The paper evaluates on four standard benchmarks, includes extensive ablations (Tables 3-11), an efficiency comparison (Table 12), and qualitative analyses, and it provides a code link. I did not find circular evaluation: the losses do not encode the target retrieval metrics, and the results are on standard held-out splits. However, the paper's central novelty is not verifiable as described: end-to-end differentiability of the DPC-KNN clustering is asserted in one sentence, and the headline "significant" gains are not supported by uncertainty quantification. If the clustering mechanism is clarified or repositioned as a fixed preprocessing step, the work is likely a solid incremental advance over HBI.

major comments (3)
  1. [Sec. 4.3, Eqs. (5)-(7), Fig. 2] The only specification of end-to-end training for the token-merge module is the sentence: "To achieve end-to-end training, we adopt the forward structural masking and backward attentional feature propagation mechanisms. The former leverages DPC-KNN wrapped in gradient-free scopes to adaptively generate structure masks, whereas the latter performs fully differentiable tensor aggregations via continuous linear index-addition operations." No equations, pseudocode, citation, or gradient rule define these mechanisms. Equations (5)-(6) and the nearest-center assignment are non-differentiable piecewise-constant functions, so if the cluster mask is gradient-free, the encoders and prototypes never receive gradients through the clustering decisions; the module reduces to a fixed preprocessor plus trainable attention. This is load-bearing because the ablations in Tables 7, 9, 10, and 11 are present
  2. [Abstract; Tables 1-2; Sec. 4.2] The headline "significant improvements in the sum of all recalls on MSR-VTT (8.8%)" is an absolute SumR point increase (414.1 vs. 405.3), not a relative percentage (about 2.2%). The same conflation appears in Sec. 4.2 ("2.6%, 4.9%, and 2.2% in t2v RSum") and in the DSL comparisons. More importantly, no error bars, multiple seeds, or significance tests are reported. Several headline deltas are small: VATEX SumR 517.8 vs. 517.1; MSR-VTT ViT-B/16 SumR 423.4 vs. 422.6; and on VATEX the t2v R@5 is lower than HBI (90.2 vs. 90.4). Thus "significant" is not supported as a statistical claim. Please report mean ± std over at least three seeds and express all deltas consistently as absolute points or relative percentages.
  3. [Sec. 3.3, Eq. (8), Table 5] The prototype contrastive loss is introduced to "prevent the prototypes from focusing on the same region" and to ensure semantic diversity. However, Eq. (8) is a standard cross-instance InfoNCE: it pulls the i-th word prototype toward the i-th frame prototype and pushes it away from other prototype indices. It does not contain an intra-modality repulsion term, nor does it directly constrain the set of prototypes to be diverse. If diversity is the intended mechanism, the loss needs an explicit diversity term (e.g., a uniform or orthogonality penalty) or an argument for why the cross-modality InfoNCE yields diversity. As written, the loss name and the "diversity" interpretation do not match the equation.
minor comments (7)
  1. [Table 3] Header typo: "Tex-to-Video" should be "Text-to-Video".
  2. [References] Reference [43] appears in the bibliography but is not cited in the text.
  3. [Table 2] The multi-row/multi-column header for text-to-video and video-to-text is hard to parse; please separate the two retrieval directions more clearly.
  4. [Fig. 4] The axes and labels are not readable. State the dataset and metric used for the hyperparameter sweeps, and label the curves.
  5. [Sec. 4.2, results text] Phrases such as "0.4% and 0.9% performance gains on the R@1 metric" refer to absolute percentage-point differences; use "points" or "absolute R@1 gains" to avoid confusion.
  6. [Sec. 4.1, Implementation Details] It is stated that "We choose HBI [11] as the baseline model," but Table 2 reports HBI without a dagger and Table 1 reports HBI with a dagger. Clarify which numbers are the authors' reproduction and which are from the original paper.
  7. [Sec. 3.3] The description of computing the prototype similarity matrix S of shape N_p^I × N_p^I × B × B is under-specified: please state the exact tensor operations used to reduce this tensor to S^P.

Circularity Check

0 steps flagged

No circular derivation: benchmark gains are measured, not fitted; the load-bearing DPC-KNN trainability claim is unverified but not circular.

full rationale

PHA-Net's central claim is that prototype-guided hierarchical alignment improves SumR on four benchmarks. Nothing in the derivation defines the reported SumR into the model: the losses (Eqs. 2, 9, 14) and the inference score S_t,v = S_I + alpha*S_L + beta*S_G (Sec. 3.4) are standard cross-modal contrastive objectives on sampled train pairs, and the retrieval metrics are computed post hoc on held-out test sets. The prototype number, cluster number, K, and loss weights are selected by ablations (Tables 8-11, Fig. 4); that is benchmark tuning, not fitting the reported number into the loss. The only self-citation (TC-MGC [36]) appears as a related/comparison method, not as a premise for PHA-Net's mechanism, so it is not load-bearing. I therefore find no self-definitional, fitted-input, self-citation-chain, or renaming circularity. The one load-bearing assertion that is unsupported is the end-to-end trainability of the DPC-KNN token merge: 'To achieve end-to-end training, we adopt the forward structural masking and backward attentional feature propagation mechanisms...' (Sec. 4.3) is the sole specification, with no equations, gradient analysis, or reference. This is a genuine omitted proof and correctness risk, but it is not a circularity, because the claim is an implementation assertion rather than a derivation that reduces to its own output. The paper's own failure cases (Fig. 6) and note that static prototype numbers limit generalization are limitations, not circularity.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 1 invented entities

The central claim rests mainly on empirical architecture changes and a set of tuned hyperparameters. No new physical or formal entity with independent evidence is introduced; the prototypes are learned parameters whose behavior is only demonstrated in-house.

free parameters (6)
  • Individual-level prototype count N_p^I = 3
    Chosen via ablation sweep over {0,1,3,5,7,9}; best at 3, later rationalized with a pyramid-based mechanism.
  • Local-level prototype count N_p^L = 1
    Chosen via the same sweep; more prototypes degrade performance.
  • Loss weights alpha, beta, lambda, mu = 0.5, 0.1, 0.2, 0.2
    Tuned via sweeps in Fig. 4; temperature sigma fixed at 0.01.
  • Cluster number N_p (phrases/clips) = 6
    Tuned via Table 10; balances semantic coverage against redundancy.
  • Nearest neighbors K in DPC-KNN = 3
    Tuned via Table 11; sharp peak at K=3 with large drops at K=4 and K=5.
  • Frame count N_f = 12
    Standard setting; robustness across N_f shown on Charades, but 12 chosen for efficiency.
axioms (4)
  • ad hoc to paper The DPC-KNN clustering can be embedded in end-to-end training via forward structural masking and backward attentional feature propagation without gradient approximation errors.
    Asserted in the Cluster Algorithm paragraph of Section 4.3; no formal description or reference provided.
  • domain assumption Tokens with weak semantics are of little interest and suppressing them improves retrieval.
    Stated in the Introduction and Section 3.2; no evidence independent of the final metric.
  • domain assumption Pre-trained CLIP-ViT-B/32 features and the DRL token-wise interaction (Eq. 3) are suitable foundations.
    Relies on prior work [1] and [7] as standard components.
  • domain assumption The three levels (individual/local/global) are complementary and their contrastive losses can be combined linearly.
    Stated in Sections 3.2 and 4.3; supported only by ablation on one dataset.
invented entities (1)
  • Modality-shared learnable prototypes (individual: 3, local: 1) no independent evidence
    purpose: Bridge the modality gap and guide token merging at each level.
    New trainable parameters introduced by this paper; the only evidence is internal attention heatmaps and ablations, with no falsifiable external prediction.

pith-pipeline@v1.3.0-alltime-deepseek · 27678 in / 13963 out tokens · 145628 ms · 2026-08-05T00:39:45.505352+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of PHA-Net: Prototype-based Hierarchical Alignment Network for Text-Video Retrieval." pith.science (2026). https://pith.science/paper/2AB5RH6J

@misc{pith2026260800551,
  author       = {Pith},
  title        = {Pith review of: PHA-Net: Prototype-based Hierarchical Alignment Network for Text-Video Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2AB5RH6J}},
  note         = {Machine review of arXiv:2608.00551}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

With the emergence of large-scale image-text pre-training models, e.g., CLIP, text-video retrieval has experienced substantial advances in recent years. Existing best-performing methods involve aligning cross-modal semantics at individual, local, and global levels simultaneously, raising concerns about the intrinsic semantic mismatch between concise texts and rich videos. A canonical approach is to integrate multiple language-video attention modules into the hierarchical framework while this paradigm only optimizes visual representations with prohibitive computational costs. In this paper, we propose a new prototype-based hierarchical alignment network (PHA-Net) to align individual/local/global level representations across modalities. Concretely, we introduce multiple modality-shared prototypes as the bridge to efficiently optimize text and video representations for cross-modal alignment. Then, we argue that the imbalanced semantic distribution in clustered tokens may undermine retrieval performance, as tokens with weak semantics are of little interest. To reduce the impact of these tokens, a proposed prototype-supported token merge module is responsible for enhancing tokens with strong semantics and suppressing others with weak semantics via prototype semantics guidance. Moreover, we devise a prototype contrastive loss to encourage textual and visual prototypes to focus on different semantic information. The idea of this auxiliary loss is to ensure higher similarity between textual and visual prototypes from the same prototype than those from different prototypes. Extensive experiments on four benchmarks confirm the effectiveness of our PHA-Net, which achieves significant improvements in the sum of all recalls on MSR-VTT (8.8%), ActivityNet (19.2%), VATEX (0.7%), and Charades (4.9%). Code is available at https://github.com/JingXiaolun/PHA-Net.

Figures

Figures reproduced from arXiv: 2608.00551 by Genke Yang, Jian Chu, Kezhao Yin, Xiaolun Jing, Xinxing Yang.

Figure 1
Figure 1. Figure 1: Overview of the proposed PHA-Net. There are three main components in PHA-Net: (1) A set of modality-shared learnable prototypes for bridging the modality gap and facilitating hierarchical cross-modal alignments. (2) The prototype-supported token merge module for integrating prototype semantic guidance into the merging process. (3) The prototype contrastive loss for encouraging semantic diversity between te… view at source ↗
Figure 2
Figure 2. Figure 2: The prototype-supported token merge module. “1D￾Conv” means the one-dimensional convolutional layer. 𝑁 input tokens with 𝐷 channels are first clustered into 𝑀 clusters. Subse￾quently, 𝑃 prototype tokens in the inputs are averaged and repeated along the token dimension to get 𝑀 prototype tokens. Finally, we feed the obtained 𝑀 prototype tokens as queries 𝑄 and the merged tokens as keys 𝐾 and values 𝑉 into a… view at source ↗
Figure 3
Figure 3. Figure 3: Ablation study of sampling frames 𝑁𝑓 . Under different sampling frames settings, 𝑁𝑓 = {12, 15, 18, 21, 24}, we compare text-to￾video retrieval performance of PHA-Net with CLIP4Clip [4] and HBI [11] on the Charades [15] dataset. (a) Hyper-parameter (b) Hyper-parameter (c) Hyper-parameter (d) Hyper-parameter [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ablation study of hyper-parameters 𝛼 and 𝛽 in Eq. 9 as well as 𝜆 and 𝜇 in Eq. 14 [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Visualization of retrieval results of our PHA-Net and HBI on the MSR-VTT dataset. Given the text or video query, we provide the top-1 retrieved results of each method, with ground-truth and others in green and red. Note that the retrieval ranks of ground-truth in PHA-Net and HBI are shown under the ground-truth. Rank 1 a girl singing song on stage Rank 2 several young girls are singing on stage Rank 3 a wo… view at source ↗
Figure 6
Figure 6. Figure 6: Visualization of failure analysis on the MSR-VTT dataset. The top-1 retrieved results are not the given ground-truth, with ground-truth in green and others in red. learnable prototypes. From the video-to-text retrieval results in the bottom part, in the first and third examples, we find that the retrieved results of HBI are all overall descriptions about video queries but only PHA-Net captures static entit… view at source ↗
Figure 7
Figure 7. Figure 7: The t-SNE visualization of text and video features at individual and local levels. Left: the results of the method without prototypes. Middle: the results of the method with individual prototypes. Right: the results of the method with individual and local prototypes. (a) (b) (c) (d) (e) (f) (g) (h) Retrieval on MSR-VTT Retrieval on ActivityNet Retrieval on VATEX Retrieval on Charades Text-to-Video Retrieva… view at source ↗
Figure 8
Figure 8. Figure 8: CMC curves comparison between our method and other methods for both the text-to-video retrieval and video-to-text retrieval tasks. The curves from left to right correspond to MSR-VTT, ActivityNet, VATEX, and Charades, respectively. frames with similar semantics are tightly connected with the same prototype, and different prototypes focus on diverse semantic regions. In the top example, the words “as” and “… view at source ↗
Figure 9
Figure 9. Figure 9: Heatmap of attention weights between prototype and words as well as prototypes and frames. We take video9303 and video8446 in the MSR-VTT dataset as examples. Darker colors indicate higher similarity, while lighter colors indicate lower similarity. different letters are coming out and sounding out the way they sound w/o PCA w/ PCA some people video conferencing as they watch a movie children singing a song… view at source ↗
Figure 10
Figure 10. Figure 10: Visualization of the top-1 text-to-video retrieval re￾sults with and without the prototype-supported cross-attention (PCA) on the MSR-VTT dataset, with related words in blue. w/o PCA: a man and a woman are singing on the beach w/ PCA: a man and a woman are walking a dog on a beach w/o PCA: person playing a game w/ PCA: a video game character rides around on a motorcycle w/o PCA: it is the animation cartoo… view at source ↗
Figure 11
Figure 11. Figure 11: Visualization of the top-1 video-to-text retrieval re￾sults with and without the prototype-supported cross-attention (PCA) on the MSR-VTT dataset, with related words in blue. Prototype-supported Cross-attention. The compari￾son of text-to-video and video-to-text retrieval results with￾out and with the proposed prototype-supported cross-attention (PCA) is illustrated in [PITH_FULL_IMAGE:figures/full_fig_p… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 49 canonical work pages

  1. [1]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G.Sastry,A.Askell,P.Mishkin,J.Clark,etal.,Learningtransferable visual models from natural language supervision, in: International conference on machine learning, PMLR, 2021, pp. 8748–8763

    work page 2021

  2. [2]

    K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R.Zemel,Y.Bengio,Show,attendandtell:Neuralimagecaptiongen- erationwithvisualattention,in:Internationalconferenceonmachine learning, PMLR, 2015, pp. 2048–2057

    work page 2015

  3. [3]

    Antol, A

    S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, D. Parikh, Vqa: Visual question answering, in: Proceedings of the IEEE international conference on computer vision, 2015, pp. 2425– 2433

    work page 2015

  4. [4]

    H. Luo, L. Ji, M. Zhong, Y. Chen, W. Lei, N. Duan, T. Li, Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning, Neurocomputing 508 (2022) 293–304

    work page 2022

  5. [5]

    S. K. Gorti, N. Vouitsis, J. Ma, K. Golestan, M. Volkovs, A. Garg, G. Yu, X-pool: Cross-modal language-video attention for text-video retrieval, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 5006–5015

    work page 2022

  6. [6]

    X. Zou, C. Wu, L. Cheng, Z. Wang, Tokenflow: Rethinking fine- grained cross-modal alignment in vision-language retrieval, arXiv preprint arXiv:2209.13822 (2022)

    Pith/arXiv arXiv 2022

  7. [7]

    Q. Wang, Y. Zhang, Y. Zheng, P. Pan, X.-S. Hua, Disentan- gled representation learning for text-video retrieval, arXiv preprint arXiv:2203.07111 (2022)

    Pith/arXiv arXiv 2022

  8. [8]

    Y. Ma, G. Xu, X. Sun, M. Yan, J. Zhang, R. Ji, X-clip: End-to-end multi-grained contrastive learning for video-text retrieval, in: Pro- ceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 638–647

    work page 2022

  9. [9]

    Wang, Y.-L

    Z. Wang, Y.-L. Sung, F. Cheng, G. Bertasius, M. Bansal, Unified coarse-to-fine alignment for video-text retrieval, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 2816–2827

    work page 2023

  10. [10]

    Jiang, S

    J. Jiang, S. Min, W. Kong, H. Wang, Z. Li, W. Liu, Tencent text- videoretrieval:hierarchicalcross-modalinteractionswithmulti-level representations, IEEE Access (2022)

    work page 2022

  11. [11]

    P. Jin, J. Huang, P. Xiong, S. Tian, C. Liu, X. Ji, L. Yuan, J. Chen, Video-text as game players: Hierarchical banzhaf interac- tion for cross-modal representation learning, in: Proceedings of the IEEE/CVFConferenceonComputerVisionandPatternRecognition, 2023, pp. 2472–2482

    work page 2023

  12. [12]

    J. Xu, T. Mei, T. Yao, Y. Rui, Msr-vtt: A large video description dataset for bridging video and language, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 5288–5296

    work page 2016

  13. [13]

    Caba Heilbron, V

    F. Caba Heilbron, V. Escorcia, B. Ghanem, J. Carlos Niebles, Activi- tynet: A large-scale video benchmark for human activity understand- ing, in: Proceedings of the ieee conference on computer vision and pattern recognition, 2015, pp. 961–970

    work page 2015

  14. [14]

    X. Wang, J. Wu, J. Chen, L. Li, Y.-F. Wang, W. Y. Wang, Vatex: A large-scale, high-quality multilingual dataset for video-and-language research, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 4581–4591

    work page 2019

  15. [15]

    G.A.Sigurdsson,G.Varol,X.Wang,A.Farhadi,I.Laptev,A.Gupta, Hollywood in homes: Crowdsourcing data collection for activity understanding, in: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14, Springer, 2016, pp. 510–526

    work page 2016

  16. [16]

    W. Ma, X. Wu, S. Zhao, T. Zhou, D. Guo, L. Gu, Z. Cai, M. Wang, Fedsh: Towards privacy-preserving text-based person re- identification, IEEE Transactions on Multimedia 26 (2023) 5065– 5077

    work page 2023

  17. [17]

    R.Zeng,W.Ma,T.Zhou,Q.Wang,S.Wang,X.Mao,J.Liu,Towards mitigationoffalsenegativesintext-to-imagepersonre-identification, IEEE Transactions on Multimedia (2026)

    work page 2026

  18. [18]

    W. Ma, X. Wu, S. Chen, W. Liu, S. Zhao, Q. Wan, L. Gu, Tsgr2: Image-text matching via triple-level scene graph relation reasoning, Applied Soft Computing (2025) 114323

    work page 2025

  19. [19]

    Y.Xin,D.Yang,Y.Zou,Improvingtext-audioretrievalbytext-aware attention pooling and prior matrix revised loss, in: ICASSP 2023- 2023IEEEInternationalConferenceonAcoustics,SpeechandSignal Processing (ICASSP), IEEE, 2023, pp. 1–5

    work page 2023

  20. [20]

    Oncescu, J

    A.-M. Oncescu, J. F. Henriques, A. Zisserman, S. Albanie, A. S. Koepke, Asound approach: Using large languagemodels to generate audio descriptions for egocentric text-audio retrieval, in: ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2024, pp. 7300–7304

    work page 2024

  21. [21]

    X. Meng, C. Zhang, S. Xie, Y. Shi, X. Lu, Spatio-temporal semantic alignmentleveraginghumanstructuralpriorsfortext-to-videoperson retrieval, Information Sciences (2026) 123414

    work page 2026

  22. [22]

    Y. Liu, S. Albanie, A. Nagrani, A. Zisserman, Use what you have: Video retrieval using representations from collaborative experts, arXiv preprint arXiv:1907.13487 (2019)

    Pith/arXiv arXiv 1907

  23. [23]

    Gabeur, C

    V. Gabeur, C. Sun, K. Alahari, C. Schmid, Multi-modal transformer forvideoretrieval,in:ComputerVision–ECCV2020:16thEuropean Conference,Glasgow,UK,August23–28,2020,Proceedings,PartIV 16, Springer, 2020, pp. 214–229

    work page 2020

  24. [24]

    11915–11925

    S.Liu,H.Fan,S.Qian,Y.Chen,W.Ding,Z.Wang,Hit:Hierarchical transformer with momentum contrast for video-text retrieval, in: Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 11915–11925

    work page 2021

  25. [25]

    J. Lei, L. Li, L. Zhou, Z. Gan, T. L. Berg, M. Bansal, J. Liu, Less is more: Clipbert for video-and-language learning via sparse sampling, in:ProceedingsoftheIEEE/CVFconferenceoncomputervisionand pattern recognition, 2021, pp. 7331–7341

    work page 2021

  26. [26]

    1728–1738

    M.Bain,A.Nagrani,G.Varol,A.Zisserman,Frozenintime:Ajoint video and image encoder for end-to-end retrieval, in: Proceedings of theIEEE/CVFinternationalconferenceoncomputervision,2021,pp. 1728–1738

    work page 2021

  27. [27]

    Y. Ge, Y. Ge, X. Liu, D. Li, Y. Shan, X. Qie, P. Luo, Bridging video- text retrieval with multiple choice questions, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 16167–16176

    work page 2022

  28. [28]

    W. Ma, Q. Chen, T. Zhou, S. Zhao, Z. Cai, Using multimodal con- trastiveknowledgedistillationforvideo-textretrieval,IEEETransac- tions on Circuits and Systems for Video Technology 33 (10) (2023) 5486–5497

    work page 2023

  29. [29]

    S. Zhao, L. Zhu, X. Wang, Y. Yang, Centerclip: Token clustering for efficienttext-videoretrieval,in:Proceedingsofthe45thInternational ACM SIGIR Conference on Research and Development in Informa- tion Retrieval, 2022, pp. 970–981

    work page 2022

  30. [30]

    Y. Liu, P. Xiong, L. Xu, S. Cao, Q. Jin, Ts2-net: Token shift and se- lection transformer for text-video retrieval, in: European Conference on Computer Vision, Springer, 2022, pp. 319–335

    work page 2022

  31. [31]

    P. Jin, J. Huang, F. Liu, X. Wu, S. Ge, G. Song, D. Clifton, J. Chen, Expectation-maximization contrastive learning for compact video- and-language representations, Advances in neural information pro- cessing systems 35 (2022) 30291–30306

    work page 2022

  32. [32]

    B. Fang, W. Wu, C. Liu, Y. Zhou, Y. Song, W. Wang, X. Shu, X. Ji, J. Wang, Uatvr: Uncertainty-adaptive text-video retrieval, in: ProceedingsoftheIEEE/CVFInternationalConferenceonComputer Xiaolun Jing et al.: Preprint submitted to Elsevier Page 16 of 17 PHA-Net: Prototype-based Hierarchical Alignment Network for Text-Video Retrieval Vision, 2023, pp. 13723–13733

    work page 2023

  33. [33]

    P. Jin, H. Li, Z. Cheng, K. Li, X. Ji, C. Liu, L. Yuan, J. Chen, Diffusionret:Generativetext-videoretrievalwithdiffusionmodel,in: Proceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 2470–2481

    work page 2023

  34. [34]

    K. Tian, Y. Cheng, Y. Liu, X. Hou, Q. Chen, H. Li, Towards effi- cient and effective text-to-video retrieval with coarse-to-fine visual representation learning, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, 2024, pp. 5207–5214

    work page 2024

  35. [35]

    K. Tian, R. Zhao, Z. Xin, B. Lan, X. Li, Holistic features are almost sufficientfortext-to-videoretrieval,in:ProceedingsoftheIEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 17138–17147

    work page 2024

  36. [36]

    X. Jing, G. Yang, J. Chu, Tc-mgc: Text-conditioned multi-grained contrastive learning for text-video retrieval, Information Fusion (2025) 103151

    work page 2025

  37. [37]

    S. Chen, Y. Zhao, Q. Jin, Q. Wu, Fine-grained video-text retrieval with hierarchical graph reasoning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 10638–10647

    work page 2020

  38. [38]

    P. Wu, X. He, M. Tang, Y. Lv, J. Liu, Hanet: Hierarchical alignment networks for video-text retrieval, in: Proceedings of the 29th ACM international conference on Multimedia, 2021, pp. 3518–3527

    work page 2021

  39. [39]

    W. Ma, Q. Chen, F. Liu, T. Zhou, Z. Cai, Query-adaptive late fusion for hierarchical fine-grained video-text retrieval, IEEE Transactions onNeuralNetworksandLearningSystems35(5)(2022)7150–7161

    work page 2022

  40. [40]

    9197–9206

    K.Wang,J.H.Liew,Y.Zou,D.Zhou,J.Feng,Panet:Few-shotimage semantic segmentation with prototype alignment, in: proceedings of theIEEE/CVFinternationalconferenceoncomputervision,2019,pp. 9197–9206

    work page 2019

  41. [41]

    X. Yan, Z. Chen, A. Xu, X. Wang, X. Liang, L. Lin, Meta r- cnn: Towards general solver for instance-level low-shot learning, in: Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 9577–9586

    work page 2019

  42. [42]

    D. Wang, H. Lu, M.-H. Yang, Online object tracking with sparse prototypes,IEEEtransactionsonimageprocessing22(1)(2012)314– 325

    work page 2012

  43. [43]

    22118–22128

    H.Rao,C.Miao,Transg:Transformer-basedskeletongraphprototype contrastive learning with structure-trajectory prompted reconstruc- tion for person re-identification, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 22118–22128

    work page 2023

  44. [44]

    L. Meng, F. Zhang, X. Zhang, C. Xu, Prototype local–global align- ment network for image–text retrieval, International Journal of Mul- timedia Information Retrieval 11 (4) (2022) 525–538

    work page 2022

  45. [45]

    5210–5218

    P.Zeng,J.Zhu,J.Song,L.Gao,Progressivetree-structuredprototype networkforend-to-endimagecaptioning,in:Proceedingsofthe30th ACM international conference on multimedia, 2022, pp. 5210–5218

    work page 2022

  46. [46]

    C.Lin,A.Wu,J.Liang,J.Zhang,W.Ge,W.-S.Zheng,C.Shen,Text- adaptive multiple visual prototype matching for video-text retrieval, Advancesinneuralinformationprocessingsystems35(2022)38655– 38666

    work page 2022

  47. [47]

    4100–4110

    P.Li,C.-W.Xie,L.Zhao,H.Xie,J.Ge,Y.Zheng,D.Zhao,Y.Zhang, Progressive spatio-temporal prototype matching for text-video re- trieval,in:ProceedingsoftheIEEE/CVFInternationalConferenceon Computer Vision, 2023, pp. 4100–4110

    work page 2023

  48. [48]

    Y. Li, Y. Xin, J. Qian, Y. Dong, S2ca: Shared concept prototypes and concept-levelalignmentfortext–videoretrieval,Neurocomputing614 (2025) 128851

    work page 2025

  49. [49]

    M. Du, S. Ding, H. Jia, Study on density peaks clustering based on k-nearest neighbors and principal component analysis, Knowledge- Based Systems 99 (2016) 135–145

    work page 2016

  50. [50]

    Y. Yu, J. Kim, G. Kim, A joint sequence fusion model for video question answering and retrieval, in: Proceedings of the European conference on computer vision (ECCV), 2018, pp. 471–487

    work page 2018

  51. [51]

    Y.-B. Lin, J. Lei, M. Bansal, G. Bertasius, Eclipse: Efficient long- rangevideoretrievalusingsightandsound,in:EuropeanConference on Computer Vision, Springer, 2022, pp. 413–430

    work page 2022

  52. [52]

    Patrick, P.-Y

    M. Patrick, P.-Y. Huang, Y. Asano, F. Metze, A. Hauptmann, J. Hen- riques, A. Vedaldi, Support-set bottlenecks for video-text representa- tion learning, arXiv preprint arXiv:2010.02824 (2020)

    Pith/arXiv arXiv 2010

  53. [53]

    Cheng, H

    X. Cheng, H. Lin, X. Wu, F. Yang, D. Shen, Improving video-text retrieval by multi-stream corpus alignment and dual softmax loss, arXiv preprint arXiv:2109.04290 (2021)

    Pith/arXiv arXiv 2021

  54. [54]

    Van der Maaten, G

    L. Van der Maaten, G. Hinton, Visualizing data using t-sne., Journal of machine learning research 9 (11) (2008). Xiaolun Jing et al.: Preprint submitted to Elsevier Page 17 of 17

    work page 2008