Pith. sign in

REVIEW 4 major objections 5 minor 31 references

Cross-Enhancement Transform Two-Stream 3D ConvNets for Action Recognition

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Cross-stream teaching lifts two-stream 3D action recognition

desk verdict The single-stream gains are intriguing, but a direct contradiction between §3 and §4.2 about which stream teaches which makes the headline results ambiguous. read the letter →

arxiv 1908.08916 v2 pith:AJ2JL2MX submitted 2019-08-19 cs.CV

Dong Cao , Lisha Xu , Dongdong Zhang This is my paper · ORCID
classification cs.CV
keywords actionrecognitiontwo-stream3DConvNetscross-enhancementknowledgedistillationfeaturetransferI3DopticalflowRGBstream
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that in a two-stream 3D convolutional network for action recognition, the stream that performs better on a given dataset can be used as a teacher to improve the weaker stream during training. Two bridge mappings between intermediate feature maps of the two streams are added, and the network minimizes the mean-squared error between these features along with the usual classification loss. The teaching direction is dataset-dependent: optical flow teaches RGB on HMDB-51 and UCF-101, while RGB teaches flow on Kinetics-400. The authors report that the enhanced stream combined with the teacher stream beats the standard two-stream I3D baseline by 0.2–0.3 percentage points on all three datasets, with much larger single-stream gains. The value of the claim is that a training-time-only feature transfer can adapt a two-stream architecture to the data distribution without changing the inference-time model.

What carries the argument

The central object is the cross-enhancement bridge: two parallel connections from one stream's feature maps to the other's, placed at an early 3D ConvNet layer and at the late network output. Each bridge contributes a squared-error term to the loss, so the total training loss is $\alpha \|F_1^{RGB} - F_1^{flow}\|_2^2 + \beta \|F_U^{RGB} - F_U^{flow}\|_2^2 + \gamma \,\mathrm{CrossEntropy}(P, y)$. The MSE terms force the weaker stream's features to resemble the stronger stream's at two depths, while the cross-entropy term keeps it discriminative. The direction of the bridges, which stream is the teacher, is chosen empirically per dataset.

What would settle it

Train the same two-stream model with the two bridge losses disabled (their weights set to zero) under identical hyperparameters; if the accuracy gain over the vanilla two-stream I3D baseline disappears, the bridge mechanism is doing the work, and if it persists, the gain comes from other training details. Also, repeat the UCF-101 evaluation across all three standard splits, since only split 1 is reported.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that cross-modal feature distillation between the two streams of an I3D-style network produces consistent, if modest, accuracy gains, and that the direction of distillation should be set by which stream is stronger on the target data. On HMDB-51 and UCF-101, where optical flow outperforms RGB, the flow stream is frozen and used as teacher; on Kinetics-400, where RGB is stronger due to camera motion, the roles reverse. The teacher's intermediate and late features are matched to the student's with two MSE losses, pushing motion or appearance information into the weaker stream. The final two-stream model, combining the enhanced student and the teacher, then exceeds the vanilla two-stream I3D accuracy on RGB+Flow.

Load-bearing premise

Matching the teacher's feature maps to the student's with mean-squared error is what actually carries the teacher's accuracy advantage into the student; if the matched features do not encode the task-relevant motion or appearance cues, the single-stream gains would not come from the proposed mechanism.

Editorial extensions

If this is right

  • If the claim holds, the same training-time feature-matching scheme should improve any two-stream architecture where one modality clearly outperforms the other on a given dataset.
  • The larger single-stream gains suggest that the weaker stream can absorb much of the teacher's signal, potentially allowing cheaper single-stream inference with less accuracy loss.
  • The dataset-dependent teaching direction indicates that the optimal transfer direction is a property of the data, not the modality, and can be chosen by validation accuracy.
  • The reported RGB+Flow improvements are small, so the method's clearest effect is on single-stream accuracy and on balancing the two streams.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural testable extension is to train the enhanced single stream alone and compare it against full two-stream inference, which would quantify how much of the flow stream's contribution can be distilled into RGB.
  • The validation-accuracy heuristic for choosing the teacher could be applied online during training, adapting the teaching direction per epoch or per video subset, rather than once per dataset.
  • Because the paper reports only a single split for UCF-101, the robustness of the 0.2–0.3 point RGB+Flow gain across splits remains open; a multi-split evaluation would settle whether the effect is systematic.
  • The same cross-stream MSE bridge idea could transfer to other paired modalities such as depth, infrared, or audio in action recognition or video understanding.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a cross-enhancement transform two-stream 3D ConvNet for action recognition. The core idea is to designate the better-performing stream (flow on HMDB-51/UCF-101, RGB on Kinetics-400, according to Section 3) as a teacher and use MSE-based feature-map matching to distill its knowledge into the weaker stream during training, after which the enhanced stream and the teacher are fused for inference. Experiments on HMDB-51, UCF-101, and Kinetics-400 report single-stream gains (e.g., RGB from 49.8 to 58.7 on HMDB-51) and small fused RGB+Flow gains of 0.2-0.3 percentage points over two-stream I3D (Table 3).

Significance. If the mechanism is confirmed, the paper would contribute a simple, dataset-adaptive way to transfer information between appearance and motion streams, building on prior knowledge-distillation work such as MARS. The explicit hypothesis that the better stream should teach the weaker stream, and that the direction should be chosen per dataset, is a reasonable and testable idea. The use of explicit MSE feature-matching losses (Eqs. 1 and 2) makes the proposal concrete. However, the current experimental evidence is too weak to establish the claim: the reported fused gains are on the order of random seed variation, model structure is selected on the same test sets used for final comparison, and the training/evaluation protocol is described inconsistently.

major comments (4)
  1. [§4.2, Table 3] The central experimental claim rests on results obtained after selecting among nine bridge structures on the same test sets used for the final comparison. Section 4.2 states that 'we train this series of models separately to obtain the optimal model structure' and then reports the test accuracy in Table 3. Because the same test partitions are used for both model selection and final evaluation, the reported improvements of 0.2-0.3 percentage points in RGB+Flow may simply reflect selection bias. The authors should instead select bridge positions on a separate validation set and report the resulting test accuracy, or provide repeated runs with different seeds so that a confidence interval can be estimated.
  2. [§3.1, §3.2, §4.2] There is a direct contradiction about the transfer direction. Sections 3.1 and 3.2 define the teacher as the better-performing stream: on HMDB-51 and UCF-101 the flow stream is better and teaches the RGB stream (Eq. 1), while on Kinetics-400 the RGB stream is better and teaches the flow stream (Eq. 2). However, Section 4.2 states the opposite: 'On HMDB-51 and UCF-101, the transmission is from the RGB stream to the flow stream.' This reverses the mechanism described in Sections 3.1 and 3.2 and is not consistent with the numerical pattern in Table 3, where the weaker stream is the one that improves. The authors must correct this and specify unambiguously which configuration produced each row of Table 3.
  3. [§3.1, §3.2] The test-phase protocols for the two proposed models are inconsistent. In Section 3.1 the multiple-enhanced-RGB model is used as-is at test time, whereas in Section 3.2 the bridge connections are removed and 'the complete two-stream model is retrained for inference.' This is a fundamental difference in the proposed method: if bridges are removed and the two-stream model is retrained, the reported gains may come from the retraining step rather than from cross-enhancement distillation. The paper should describe one unified protocol for both datasets and explain why the protocol differs between the two variants.
  4. [Table 3, §4.3] No error bars, number of runs, or random seeds are reported, so the claimed improvements cannot be distinguished from optimization noise. This is especially important because the fused RGB+Flow gains over two-stream I3D are only 0.2-0.3 percentage points, which is well within typical run-to-run variation for 3D ConvNets on these benchmarks. The authors should provide at least three independent runs with different seeds and report mean and standard deviation, or otherwise justify that the differences are statistically meaningful.
minor comments (5)
  1. [§4.2] The description of the nine bridge structures is vague: it says positions are chosen from 'front, medium and rear' of the 3D ConvNet, but it does not give the layer names or feature-map indices, making the experiments hard to reproduce.
  2. [Eqs. (1) and (2)] The loss weights α1, β1, γ1 and α2, β2, γ2 are introduced but their values are never reported. The paper should state the weight values used in the experiments, since the behavior of the distillation depends strongly on them.
  3. [Table 3] The row for '3D-Fused' shows '-' for the RGB and Flow columns. This should be explained; presumably 3D-Fused is a single-stream model that takes both RGB and flow as input, so only the RGB+Flow column applies.
  4. [§4.3] Implementation details are incomplete: no learning-rate schedule, number of epochs, or training-set augmentation strategy is given, and it is not stated how the frozen teacher stream is trained before the student stream.
  5. [References] Reference [9] is listed as 'Recurrent batch normalization' but the text refers to ConvNet+LSTM; the reference list entry does not match the citation context.

Circularity Check

2 steps flagged · score 6.0 of 10

Reported gains are selected maxima over nine bridge structures chosen on the same test sets, and the transfer direction claimed in §4.2 contradicts the proposed teacher mechanism.

  1. fitted input called prediction [Section 4.2 (Cross Enhancement Experiments), with Table 3]
    ""we first select a connection from ‘a’ to ‘i’ as the first bridge mapping according to the experimental results" (§3.1); "Finally, we train this series of models separately to obtain the optimal model structure. The test accuracy is shown in Table 3." (§4.2)"

    The bridge position and direction are not fixed in advance: the paper trains nine bridge configurations and selects the one with best test accuracy. Table 3 then reports the selected configuration's accuracy on the same test sets as evidence of effectiveness. The reported values are therefore maxima over configurations fitted to the evaluation labels, not predictions from a prespecified architecture. The fused gains over two-stream I3D (0.2-0.3 points, with single-stream RGB gains up to 8.9 points) are at the scale this selection bias can create, and no validation set is reported.

  2. other [Section 3.1 vs Section 4.2]
    ""Since the optical flow performance of the two-stream model is better on HMDB-51 and UCF-101, we first train the optical flow stream separately... we realize the transmission of the motion information extracted from optical flow stream to the RGB stream" (§3.1); "On HMDB-51 and UCF-101, the transmission is from the RGB stream to the flow stream." (§4.2)"

    The load-bearing premise is that the better stream teaches the weaker stream, with direction set per dataset. For HMDB-51 and UCF-101, §3.1 makes flow the teacher and RGB the student, which matches the large RGB improvement in Table 3. §4.2 states the opposite direction for the same datasets. If RGB-to-flow was used, the teacher is the weaker stream and the reported RGB gain is not explained by the proposed mechanism; if flow-to-RGB was used, §4.2 is wrong. Either way, the exact configuration behind Table 3 is ambiguous, so the claimed causal role of the MSE bridges is not established.

full rationale

The paper's core mechanism (MSE distillation between I3D streams, inspired by external MARS work) has independent content and does not rely on self-citation or a uniqueness theorem. However, the empirical claim that the proposed model improves over two-stream I3D is partially circular: the bridge structure is selected by training nine variants and picking the best according to the same test accuracies that Table 3 then reports as the model's result. This is a fitted input presented as a prediction, because the architecture itself is optimized on the evaluation set. The additional contradiction between §3.1 and §4.2 about transfer direction further prevents attribution of the gains to the described teacher-student mechanism. These issues compromise the headline comparison, though the underlying distillation idea is not definitionally circular; hence a score of 6 rather than higher.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim depends on several tunable choices (loss weights, bridge positions, transfer direction) and on the undocumented assumption that MSE feature matching between streams transfers task-relevant information. No new physical or mathematical entities are introduced.

free parameters (3)
  • Distillation loss weights alpha and beta (Eq. 1 and Eq. 2) = not reported
    The balance between feature-matching MSE terms and cross entropy is set by weights alpha1, beta1, gamma1 and alpha2, beta2, gamma2; no values or tuning procedure are given.
  • Bridge connection positions = not reported; selected from front, medium, rear positions
    Section 4.2 states that models with different bridge positions were trained and the best one was selected on the benchmark.
  • Transfer direction per dataset = flow to RGB on HMDB-51 and UCF-101; RGB to flow on Kinetics-400
    The direction is chosen based on the known accuracy ranking of the two I3D streams on each dataset, rather than predicted by the model.
assumptions (3)
  • domain assumption The stronger stream has features worth transferring to the weaker stream.
    The method assumes that feature maps of the higher-accuracy stream contain task-relevant information that the weaker stream lacks; this is not demonstrated independently but is the basis of Eqs. 1 and 2.
  • domain assumption MSE feature matching is an effective way to transfer knowledge between appearance and motion streams.
    The losses L_UH1, L_UH2, L_K1, and L_K2 all rely on this assumption; no ablation or analysis is provided to show MSE matching is the cause of any gain.
  • domain assumption Per-dataset stream rankings from I3D baselines are stable and justify the transfer direction.
    Section 3.1 and 3.2 set the teacher based on the authors' observation of I3D performance on each dataset, assuming this ranking holds for the new training procedure.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cross-Enhancement Transform Two-Stream 3D ConvNets for Action Recognition." pith.science (2026). https://pith.science/paper/AJ2JL2MX

@misc{pith2026190808916,
  author       = {Pith},
  title        = {Pith review of: Cross-Enhancement Transform Two-Stream 3D ConvNets for Action Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AJ2JL2MX}},
  note         = {Machine review of arXiv:1908.08916}
}
read the original abstract

Action recognition is an important research topic in computer vision. It is the basic work for visual understanding and has been applied in many fields. Since human actions can vary in different environments, it is difficult to infer actions in completely different states with a same structural model. For this case, we propose a Cross-Enhancement Transform Two-Stream 3D ConvNets algorithm, which considers the action distribution characteristics on the specific dataset. As a teaching model, stream with better performance in both streams is expected to assist in training another stream. In this way, the enhanced-trained stream and teacher stream are combined to infer actions. We implement experiments on the video datasets UCF-101, HMDB-51, and Kinetics-400, and the results confirm the effectiveness of our algorithm.

Figures

Figures reproduced from arXiv: 1908.08916 by the authors.

Figure 2
Figure 2. Multiple-Enhanced-Flow Two Stream 3D ConvNets for Kinetics-400. Training phase Firstly, the optical flow stream does not participate in training while the RGB stream is trained. We froze the RGB stream and fix the trained parameters for following operation after the RGB stream finish training. Next, similar to multiple-enhanced-RGB two stream 3D ConvNets, we establish two bridge mappings from RGB stream to optical f… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 28 canonical work pages

  1. [1]

    Joao, and Z

    C. Joao, and Z. Andrew, 2017 Quo vadis, action recognition? a new model and the kinetics dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR

    work page 2017

  2. [2]

    Crasto, P

    N. Crasto, P. Weinzaepfel, K. Alahari,, and C. Schmid, 2019 MARS: Motion- augmented RGB stream for action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR

    work page 2019

  3. [3]

    Karpathy, G

    A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and F .F. Li. ,2014 Large -scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, CVPR

    work page 2014

  4. [4]

    Laptev, M

    I. Laptev, M. Marszalek, C. Schmid, and B. Rozenfeld, 2008 Learning realistic human actions f rom movies. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, CVPR

    work page 2008

  5. [5]

    J. C. Niebles, H. Wang, and F. F Li, 2008 Unsupervised learning of human action categories using spatial-temporal words. In International journal of computer vision, 79(3), 299–318

    work page 2008

  6. [6]

    Wang, and C

    H. Wang, and C. Schmid, 2013 Action recognition with improved trajectories. In IEEE International Conference on Computer Vision , ICCV

    work page 2013

  7. [7]

    Donahue, L.A

    J. Donahue, L.A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell, 2015 Long -term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR

    work page 2015

  8. [8]

    J. Y. H. Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga, and G. Toderici, 2015 Beyond short snippets: Deep networks for video classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR

    work page 2015

Show all 31 references
  1. [9]

    Cooijmans, N

    T. Cooijmans, N. Ballas, C. Laurent, Ç. Gü lç ehre, and A. Courville, 2016 Recurrent batch normalization. arXiv preprint arXiv:1603.09025

    2016 arXiv

  2. [10]

    S. Ji, W. Xu, M. Yang, and K. Yu, 2013 3D convolutional neural networks for human action recognition. In IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(1), 221-231

    2013

  3. [11]

    W. T. Graham, R. Fergus, Y. Lecun, and C. Bregler, 2010 Convolutional Learning of Spatio -temporal Features. In Proceedings of European Conference on Computer Vision, ECCV

    2010

  4. [12]

    D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, 2015 Learning Spatiotemporal Features with 3D Co nvolutional Networks. In 2015 IEEE International Conference on Computer Vision, ICCV

    2015

  5. [13]

    Varol, I

    G. Varol, I. Laptev, and C. Schmid, 2017 Long-term temporal convolutions for action recognition. In IEEE Transactions on Pattern Analys is and Machine Intelligence

    2017

  6. [14]

    Simonyan, and A

    K. Simonyan, and A. Zisserman, 2014 Two-stream convolutional networks for action recognition in videos. In Advances in Neural Information Processing Systems

    2014

  7. [15]

    Feichtenhofer, A

    C. Feichtenhofer, A. Pinz, and A. Zisserman, 2016 Convolutional Two - Stream Network Fusion for Video Action Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition , CVPR

    2016

  8. [16]

    Y. Tang, L. Ma, and L. Zhou, 201 9 Hallucinating optical flow features for video classification. In International Joint Conferences on Artificial Intelligence, IJCAI

  9. [17]

    Simonyan, and A

    K. Simonyan, and A. Zisserman, 2014 Two-stream convolutional networks for a ction recognition in videos. In Advances in Neu ral Information Processing Systems, NIPS

    2014

  10. [18]

    Y. Zhu, Z. Lan, S. Newsam, and A.G. Hauptmann, 2018 Hidden two -stream convolutional netw orks for action recognition. In Asian Conference on Computer Vision ACCV

    2018

  11. [19]

    H. Wang, A. Klä ser, C. Schmid, C. Liu, 2011 Action recogn ition by dense trajectories. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR

    2011

  12. [20]

    H. Wang, C. Schmid, 2013 Action recognition with improv ed trajectories. In IEEE International Conference on Computer Vision , ICCV

    2013

  13. [21]

    C. Zach, T. Pock, and H. Bischof, 2007 A duality based approach for realtime TV-L1 optical flow. In Deutsche Arbeitsgemeinschaft fü r Mustererkennung, DAGM

    2007

  14. [22]

    Dosovitskiy, P

    A. Dosovitskiy, P. Fischer, E. Ilg, P. Hä usser, C. Hazırbäs, V. Golkov, P. V. D. Smagt, D. Cremers, and T. Brox, 2015 Flownet: Learning optical flow with convolutional networks. In IEEE International Conference on Computer Vision, ICCV

    2015

  15. [23]

    E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox, 2017 Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR

    2017

  16. [24]

    Colin, D

    L. Colin, D. F. Michael, V. René, R. Austin, and D. H. Gregory, 2017 Temporal convolutional networks for action segmentation and detection . In IEEE Conference on Computer Vision and Pattern Recognition, CVPR

    2017

  17. [25]

    L. Wang, Y. Xiong, Z . Wang, Y. Qiao, D. Lin, X. Tang, and V. Gool, 2016 Temporal segment networks: towards good practices for deep action recognition. In Proceedings of European Conference on Computer Vision, ECCV

    2016

  18. [26]

    M. Xu, A. Sharghi, X. Chen, and D. Crandall, 2018 fully-coupled two-stream spatiotemporal networks for extremely low resolution action recognition. arXiv preprint arXiv: 1801.03983

    2018 arXiv

  19. [27]

    Lopez-Paz, L

    D. Lopez-Paz, L. Bottou, B. Scholkopf, and V. Vapnik, 2016 Unifying distillation and privileged information. In International Conference on Learning Representations, ICLR

    2016

  20. [28]

    Kuehne, H

    H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre, 2011 HMDB: A large video database for human motion recognition. In IEEE International Conference on Computer Vision, ICCV

    2011

  21. [29]

    Soomro, A.R

    K. Soomro, A.R. Zamir, and M. Shah, 2012 UCF101: A dataset of 101 human actions classes fromvideos in the wild. arXiv preprint arXiv:1212.0402

    2012 arXiv

  22. [30]

    W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, M. Suleyman, and A. Zisserman, 2017 The kinetics human action video dataset. arXiv preprint arXiv:1705.06950

    2017 arXiv

  23. [31]

    Purwanto, R

    D. Purwanto, R. R. A. Pramono, Y. T. Chen, and W. H. Fang, 2019 E xtreme low resolution action recognition with spatial -temporal multi -head self - attention and knowledge distillation. In IEEE International Conference on Computer Vision Workshop

    2019

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.