Pith. sign in

REVIEW 3 major objections 4 minor 32 references

Image to Video Domain Adaptation Using Web Supervision

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a two-stage, appearance-first training procedure turns noisy search-engine images and videos into a video action classifier that reaches 74.9% top-1 on UCF-101 and 42.8% on Kinetics-400.

desk verdict A neat two-stage transfer design whose headline numbers rest on an unstated evaluation boundary that could inflate the reported gains. read the letter →

arxiv 1908.01449 v1 pith:WQOP3F47 submitted 2019-08-05 cs.CV

Andrew Kae , Yale Song This is my paper · ORCID
classification cs.CV
keywords webly-supervisedlearningvideoclassificationdomainadaptationattentionmechanismactionrecognitiontwo-stagetraining3DconvolutionalnetworksUCF-101
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a video action classifier can be trained using only noisy, search-engine-sourced images and videos, without manually labeled target videos. The recipe is to learn spatial/appearance features first from web images and video frames, then freeze those filters and learn temporal/motion features from web video chunks. Two auxiliary mechanisms carrying the argument are adversarial domain adaptation, which pulls web features toward unlabeled target-video features, and a target-relative attention module, which downweights web examples that look unlike target frames. On UCF-101 the full pipeline reaches 74.9% top-1, within 1.4 points of the leading webly-supervised method while using only two training stages, and on Kinetics-400 it reaches 42.8%.

What carries the argument

The load-bearing machinery is a combination of two losses optimized jointly with the classifier. A domain-adversarial loss uses a multi-layer discriminator with gradient reversal to make the encoder's embeddings indistinguishable across web images, web video frames, and unlabeled target frames. An attention loss computes, for each web example, a learned similarity $E(x_i) W E(x_k)^\top$ to target frames, keeps the top-$T$ scores per example, softmaxes them, and uses the resulting $\alpha_i$ weights to reweight the cross-entropy classification loss. The video stage starts from the learned spatial filters, fixes them, appends temporal filters initialized to zero with a residual connection, and applies the same attention and domain-adaptation losses to web video chunks versus target chunks.

What would settle it

Run a lower-threshold cosine search or exact-frame matching between the cleaned web images/videos and the official UCF-101 and Kinetics test clips; if even a small percentage of the claimed web-only training data is actually copied from the test set, removing those items would drop the 74.9% and 42.8% figures. A second check is to retrain with target test clips explicitly excluded from the unlabeled batches and confirm the accuracy is unchanged.

Watch

Extended reading notes

Core claim

The central claim is that separating appearance learning from motion learning makes webly-supervised video classification tractable: a 2D-CNN first learns domain-invariant, noise-filtered appearance features from web images and frames, and its spatial weights are transferred to a 3D-CNN whose temporal filters are initialized to zero and trained through a residual connection. The paper reports that both auxiliary components help on their own, and together they lift the image model to 72.6% and the video model to 74.9% top-1 on UCF-101, while a model initialized from ImageNet weights and trained directly on web videos reaches only 59.1%, supporting the two-stage ordering.

Load-bearing premise

The result depends on the cleaned web data truly having no duplicate or near-duplicate UCF-101/Kinetics clips, and on the unlabeled target frames used during training being separate from the clips used for the final accuracy measurement; if either is violated, the reported numbers rise.

Editorial extensions

If this is right

  • If the two-stage claim is right, obtaining a video classifier for a new action class may require only search-engine images and videos plus unlabeled target clips, not manually annotated target videos.
  • The appearance-first transfer means the same 2D image model can be reused as initialization for many video-model instantiations, lowering training cost.
  • Because attention and domain adaptation are trained jointly with classification, noisy web data can be filtered during training instead of in a separate preprocessing stage.
  • The reported numbers imply that the gap from webly-supervised to fully supervised action recognition on UCF-101 is roughly 23 points, a clear target for the next method.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the attention mechanism is compared against human relevance judgments on a noisy target domain, one could test whether the learned $\alpha$ weights really track which web images/frames contain the action.
  • An audit that checks the cleaned web data against UCF-101 and Kinetics at several cosine thresholds, and verifies that unlabeled target clips are disjoint from test clips, would settle how much of the reported gain depends on the deduplication step.
  • The same appearance-first with fixed spatial filters and zero-initialized temporal filters could be tried with stronger video backbones; if the gains persist, the contribution would be the training decomposition plus the two auxiliary losses rather than the specific backbone.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a two-stage, webly-supervised video classification pipeline. In the first stage, a 2D CNN image model is trained on web images and web video frames with a triplet architecture, a domain-adversarial loss, and a target-relative attention component. In the second stage, the spatial weights of this image model are transferred to a 3D CNN video model, whose temporal filters are trained on web video chunks while the spatial filters are kept fixed; the video model also includes domain adaptation and attention. The method is evaluated on UCF-101 and Kinetics-400, reporting 74.9% and 42.8% top-1 accuracy respectively, and is compared with prior webly-supervised and fully supervised approaches. The central claim is that separating appearance and temporal learning, together with the proposed attention and domain-adaptation components, yields competitive webly-supervised performance while simplifying training relative to multi-stage pipelines such as LeadExceed.

Significance. If the results hold, the paper makes a useful practical contribution: it shows that a two-stage image-to-video transfer procedure with integrated attention and domain adaptation can match, within about 1.4%, a more complex five-stage webly-supervised method on UCF-101, and it provides the first reported webly-supervised results on Kinetics-400. The ablation studies are informative and the two-stage design is supported by the large drop in accuracy when the video model is initialized from ImageNet instead of the web-trained image model. The paper is clearly written and the method is well motivated. However, the absolute gains from the two central components are small, no statistical significance is reported, and the evaluation protocol leaves a potentially load-bearing ambiguity about whether the unlabeled target videos used during training are disjoint from the videos used for scoring.

major comments (3)
  1. [Sections 3.3, 3.4, and 4.1]
  2. [Tables 1 and 2, Section 4.3]
  3. [Abstract, Section 5, and Table 3]
minor comments (4)
  1. [Section 3.4]
  2. [Tables 1 and 2]
  3. [Section 4.2]
  4. [Tables 3 and 4]

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the central result is an empirical comparison on external benchmarks, not a derivation from its own inputs.

full rationale

The paper's central claim—74.9% top-1 on UCF-101 and 42.8% on Kinetics-400 from webly-supervised data—is an empirical result evaluated against public benchmarks, not a quantity derived from a fitted parameter or from a self-citation. The DA loss (Eqs. 1-4) and attention weighting (Sec. 3.4) are trainable components whose contributions are measured by ablations in Tables 1 and 2; none of these ablations redefines the target metric as a training objective. The two-stage image-to-video design is motivated by external work (R(2+1)D, ResNet) and tested by an ImageNet-initialization comparison, so it is not assumed into existence by citation. The only self-citation is [22], used to justify the Hecate keyframe-extraction tool; this is a minor implementation detail and not load-bearing for the accuracy claim. The paper does leave a possible evaluation-boundary ambiguity: it does not state whether the unlabeled target frames used in DA/attention (Section 3.3-3.4) are disjoint from the UCF-101/Kinetics clips used for scoring, and Section 4.1's cosine-similarity cleanup filters web items rather than target items. That is a potential data-leakage/correctness concern, but it is not a circular derivation: no equation or fitted value in the paper reduces the reported accuracy to its own input by construction. Under the rule that circularity must be exhibited by a specific reduction, no such step exists here.

Assumptions & free parameters 5 free parameters · 7 assumptions · 0 invented entities

The method depends on a handful of empirical hyperparameters, including beta, tau, Top-T, and the overlap-detection threshold, whose values are mostly unreported. It also rests on standard domain assumptions about the cleanliness of curated videos, the transferability of spatial features, and the validity of transductive domain adaptation, plus the R(2+1)D factorization taken from prior work. No new physical or conceptual entities are introduced; attention and domain discriminators are learned model components, not independently evidenced postulates.

free parameters (5)
  • beta tradeoff parameter = not reported
    Balances classification and domain adaptation losses in Equations 6 and 8; chosen on a held-out validation set but no value or search range is given.
  • tau temperature in attention softmax = not reported
    Controls softmax sharpness in Section 3.4; chosen empirically, no value reported.
  • Top-T target scores in attention = not reported
    Number of highest target similarities summed per source sample in Section 3.4; authors say top T worked better but do not report T.
  • cosine similarity threshold = 0.9
    Threshold for removing web data overlapping UCF or Kinetics keyframes in Section 4.1; manually set and critical for the claimed evaluation cleanliness.
  • web data collection sizes = 800-900 images and 25-50 videos per class
    Amount of webly-supervised data per UCF-101 or Kinetics category in Section 4.1; affects final accuracy and is not independently justified.
assumptions (7)
  • domain assumption Webly-supervised labels and images carry enough semantic signal to learn appearance features.
    The whole method assumes noisy web search results are still usable for classification, Section 1.
  • domain assumption Curated target videos contain only the target concept and have little noise.
    Used to justify attention comparing source samples to target batches, Sections 1 and 3.4.
  • domain assumption Spatial appearance features transfer from 2D image models to 3D video models, and fixing them while learning temporal filters is beneficial.
    Core two-stage claim, Section 3.6; tested only empirically.
  • standard math R(2+1)D factorization into spatial then temporal convolutions preserves video modeling capacity.
    Borrowed from reference [25] and used as the block design in Figure 5; no new proof is given.
  • domain assumption Adversarial feature alignment across web image, web video, and target video domains improves target classification.
    Standard DA assumption used in Section 3.3, with empirical support only in the reported ablations.
  • domain assumption The cosine similarity cleanup at threshold 0.9 removes all overlapping web and target data.
    Evaluation cleanliness assumption in Section 4.1; not validated with manual inspection or a separate test.
  • domain assumption Transductive use of unlabeled target videos during training does not bias evaluation.
    The paper does not state whether target test clips are excluded from the unlabeled reference data, Sections 3.1 and 4.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Image to Video Domain Adaptation Using Web Supervision." pith.science (2026). https://pith.science/paper/WQOP3F47

@misc{pith2026190801449,
  author       = {Pith},
  title        = {Pith review of: Image to Video Domain Adaptation Using Web Supervision},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WQOP3F47}},
  note         = {Machine review of arXiv:1908.01449}
}
read the original abstract

Training deep neural networks typically requires large amounts of labeled data which may be scarce or expensive to obtain for a particular target domain. As an alternative, we can leverage webly-supervised data (i.e. results from a public search engine) which are relatively plentiful but may contain noisy results. In this work, we propose a novel two-stage approach to learn a video classifier using webly-supervised data. We argue that learning appearance features and then temporal features sequentially, rather than simultaneously, is an easier optimization for this task. We show this by first learning an image model from web images, which is used to initialize and train a video model. Our model applies domain adaptation to account for potential domain shift present between the source domain (webly-supervised data) and target domain and also accounts for noise by adding a novel attention component. We report results competitive with state-of-the-art for webly-supervised approaches on UCF-101 (while simplifying the training process) and also evaluate on Kinetics for comparison.

Figures

Figures reproduced from arXiv: 1908.01449 by the authors.

Figure 1
Figure 1. Given webly-supervised images and videos [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. T-SNE Plots. We randomly sampled from the web image (red points), web video (green points) and target video (blue points) (UCF-101 [23]) domains and show the T-SNE [28] plots of 4 actions: balance beam, long jump, surfing, and throw discus. The first row contains the T-SNE plot before domain adaptation using pre-trained RN-34 [11] and the second row shows the same actions after the network has been domain adapted. P… view at source ↗
Figure 3
Figure 3. Image Model. Triplet network with branches corresponding to web images, web video frames and target video frames. We add discriminators D to enforce domain invariance between the separate domains and add attention components A to downweight irrelevant examples. C cor￾responds to the classifiers and L corresponds to the losses. designed specifically for image to video domain adaptation and classification. 3. Model Ou… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Video Model. We use a Siamese model with branches corresponding to web video and curated video chunks. We initialize the spatial weights in the 3D-CNN and add an attention component A to reduce the noise from irrelevant shots or incorrect labels. C corresponds to the c…
Figure 5
Figure 5. Figure 5: Spatio-temporal Block. (a) the decomposition of the spatiotemporal block into a 2D spatial filter fol￾lowed by a 1D temporal filter (corresponds to R(2+1)D [25] model) (b) our modified block with an added residual con￾nection. The spatial weights are initialized from t…
Figure 6
Figure 6. Figure 6: Attention Weighting. For a web image batch, we show the weights (α in Equation 5) for each image and the category of the image (the weights sum to 1). Images with lower weight in the last row tend to be more cartoon-like or contain excessive text while images with high…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 31 canonical work pages

  1. [1]

    Bahdanau, K

    D. Bahdanau, K. Cho, and Y . Bengio. Neural machine trans- lation by jointly learning to align and translate. In ICLR,

    work page

  2. [2]

    Bousmalis, N

    K. Bousmalis, N. Silberman, D. Dohan, D. Erhan, and D. Kr- ishnan. Unsupervised pixel-level domain adaptation with generative adversarial networks. In Computer Vision and Pattern Recognition, 2017. 3

    work page 2017

  3. [3]

    Carreira and A

    J. Carreira and A. Zisserman. Quo vadis, action recognition? A new model and the kinetics dataset. CVPR, 2017. 3, 8

    work page 2017

  4. [4]

    J. Chen, Y . Cui, G. Ye, D. Liu, and S.-F. Chang. Event-driven semantic concept discovery by exploiting weakly tagged in- ternet images. In Proceedings of International Conference on Multimedia Retrieval, 2014. 2

    work page 2014

  5. [5]

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei- Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2009. 7

    work page 2009

  6. [6]

    Feichtenhofer, H

    C. Feichtenhofer, H. Fan, J. Malik, and K. He. Slowfast net- works for video recognition. CoRR, 2018. 8

    work page 2018

  7. [7]

    C. Gan, C. Sun, L. Duan, and B. Gong. Webly-supervised video recognition by mutually voting for relevant web im- ages and web video frames. In European Conference on Computer Vision (ECCV), 2016. 2, 5, 6, 8

    work page 2016

  8. [8]

    C. Gan, T. Yao, K. Yang, Y . Yang, and T. Mei. You lead, we exceed: Labor-free video concept learning by jointly exploit- ing web videos and images. In Computer Vision and Pattern Recognition (CVPR), 2016. 2, 6, 7, 8

    work page 2016

Show all 32 references
  1. [9]

    Ganin, E

    Y . Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V . Lempitsky. Domain- adversarial training of neural networks. Journal of Machine Learning Research, 17(59):1–35, 2016. 2, 3, 5

    2016

  2. [10]

    Goodfellow, J

    I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio. Gen- erative adversarial nets. In Advances in neural information processing systems, 2014. 2, 3, 4

    2014

  3. [11]

    K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, 2016. 2, 4, 6

    2016

  4. [12]

    Ioffe and C

    S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015. 6

    2015

  5. [13]

    W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, A. Nat- sev, M. Suleyman, and A. Zisserman. The kinetics human action video dataset. CoRR, abs/1705.06950, 2017. 6, 7, 8

    2017 arXiv

  6. [14]

    Lecun, L

    Y . Lecun, L. Bottou, Y . Bengio, and P. Haffner. Gradient- based learning applied to document recognition. In Proceed- ings of the IEEE, pages 2278–2324, 1998. 2

    1998

  7. [15]

    J. Li, Y . Wong, Q. Zhao, and M. S. Kankanhalli. Attention transfer from web images for video recognition. ACM Mul- timedia, 2017. 3, 8

    2017

  8. [16]

    Z. Luo, Y . Zou, J. Hoffman, and L. Fei-Fei. Label efficient learning of transferable representations across domains and tasks. In Conference on Neural Information Processing Sys- tems (NIPS), 2017. 3, 4

    2017

  9. [17]

    S. Ma, S. A. Bargal, J. Zhang, L. Sigal, and S. Sclaroff. Do less and achieve more: Training cnns for action recognition utilizing action images from the web. Pattern Recognition,

  10. [18]

    Paszke, S

    A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. De- Vito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Auto- matic differentiation in pytorch. In NIPS-W, 2017. 6

    2017

  11. [19]

    Z. Qiu, T. Yao, and T. Mei. Learning spatio-temporal repre- sentation with pseudo-3d residual networks. In ICCV, 2017. 3

    2017

  12. [20]

    Simonyan and A

    K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. In NIPS, 2014. 8

    2014

  13. [21]

    Singh, X

    B. Singh, X. Han, Z. Wu, V . I. Morariu, and L. S. Davis. Selecting relevant web trained concepts for automated event retrieval. In IEEE International Conference on Computer Vision (ICCV), 2015. 2

    2015

  14. [22]

    Y . Song, M. Redi, J. Vallmitjana, and A. Jaimes. To click or not to click: Automatic selection of beautiful thumbnails from videos. In CIKM, 2016. 6

    2016

  15. [23]

    Soomro, A

    K. Soomro, A. R. Zamir, and M. Shah. UCF101: A dataset of 101 human actions classes from videos in the wild.CRCV- TR-12-01, 2012. 2, 6, 7, 8

    2012

  16. [24]

    D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatiotemporal features with 3d convolutional net- works. In Proceedings of the 2015 IEEE lInternational Con- ference on Computer Vision (ICCV), 2015. 3, 8

    2015

  17. [25]

    D. Tran, H. Wang, L. Torresani, J. Ray, Y . LeCun, and M. Paluri. A closer look at spatiotemporal convolutions for action recognition. In CVPR, 2018. 1, 3, 5, 8

    2018

  18. [26]

    Tzeng, J

    E. Tzeng, J. Hoffman, T. Darrell, and K. Saenko. Simultane- ous deep transfer across domains and tasks. In ICCV, 2015. 3

    2015

  19. [27]

    Tzeng, J

    E. Tzeng, J. Hoffman, T. Darrell, and K. Saenko. Adversarial discriminative domain adaptation. In Computer Vision and Pattern Recognition (CVPR), 2017. 2, 3

    2017

  20. [28]

    van der Maaten and G

    L. van der Maaten and G. Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 2008. 2

    2008

  21. [29]

    L. Wang, Y . Xiong, D. Lin, and L. Van Gool. Untrimmednets for weakly supervised action recognition and detection. In CVPR, 2017. 2, 3

    2017

  22. [30]

    X. Wang, R. Girshick, A. Gupta, and K. He. Non-local neural networks. In CVPR, 2018. 8

    2018

  23. [31]

    Zhang, Z

    J. Zhang, Z. Ding, W. Li, and P. Ogunbona. Importance weighted adversarial nets for partial domain adaptation. In Computer Vision and Pattern Recognition (CVPR), 2018. 3, 5

    2018

  24. [32]

    Zhuang, L

    B. Zhuang, L. Liu, Y . Li, C. Shen, and I. D. Reid. Attend in groups: a weakly-supervised deep learning framework for learning from web data. In CVPR, 2017. 3, 5

    2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.