Bypass Enhancement RGB Stream Model for Pedestrian Action Recognition of Autonomous Vehicles

Dong Cao; Lisha Xu

REVIEW 4 major objections 5 minor 32 references

Bypass Enhancement RGB Stream Model for Pedestrian Action Recognition of Autonomous Vehicles

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A Bypass Enhancement RGB Stream model distills optical-flow knowledge into an RGB-only network, keeping action-recognition accuracy within about one point of a two-stream baseline while removing optical-flow computation at inference.

desk verdict A sensible MARS variant whose central real-time claim is unmeasured and whose accuracy tables conflict; incremental and not ready to cite. read the letter →

arxiv 1908.05674 v2 pith:F4S74YJZ submitted 2019-08-15 cs.CV cs.LGeess.IV

Dong Cao , Lisha Xu This is my paper · ORCID

classification cs.CVcs.LGeess.IV

keywords PedestrianActionRecognitionAutonomousDrivingBypassEnhancementKnowledgeDistillationOpticalFlowRGBStreamVideoReal-timeInference

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Pedestrian action recognition in autonomous driving normally pays a heavy runtime price for motion information: optical flow is accurate but expensive to compute. This paper proposes a Bypass Enhancement RGB Stream (BERS) model that trains a flow-based teacher branch alongside an RGB student branch, then discards the flow branch at inference. The claim is that the RGB-only model keeps accuracy essentially unchanged, reaching 68.7% on Kinetics and 95.5% on UCF101, within 0.9 and 0.1 points of the two-stream MARS+RGB baseline, while avoiding all optical-flow computation at run time. That matters because a cheaper inference stream is directly relevant to the latency and safety budget of an autonomous vehicle.

What carries the argument

The mechanism is a two-phase distillation architecture. In the training phase, a bottom optical-flow branch computes TV-L1 flow from RGB frames and a 3D ResNeXt101 extracts $\mathrm{Feature}_1$; this branch is trained first and then frozen. In the second phase, an upper RGB branch is trained with the combined loss $L = L_a + \lambda\|\mathrm{Feature}_1 - \mathrm{Feature}_2\|$, where $L_a$ is cross-entropy over the concatenated outputs of three bypass branches tapped at different depths: a small residual network before average pooling, another branch before $\mathrm{Feature}_2$, and $\mathrm{Feature}_2$ itself. The bypass branches are the load-bearing addition over the MARS baseline, giving the RGB stream multiple granularities of motion-influenced features. At inference, only the upper model runs, so no optical-flow computation is needed.

What would settle it

Measure end-to-end inference latency or FLOPs for BERS versus MARS and MARS+RGB on the same GPU and video clips; if the RGB-only BERS model is not faster than the two-stream baseline, the real-time claim fails even though accuracy may hold. A second check would ablate the three bypass branches and see whether accuracy falls back to MARS-level.

Watch

Extended reading notes

Core claim

The paper argues that motion information from optical flow can be distilled into an RGB-only stream without paying the two-stream inference cost, provided the RGB stream is given multiple bypass branches and a two-part loss. The model first trains a TV-L1 optical-flow branch built on 3D ResNeXt101 to produce a feature vector $\mathrm{Feature}_1$; then it trains the RGB branch with cross-entropy on three pooled residual-branch outputs plus a distillation term $\lambda\|\mathrm{Feature}_1 - \mathrm{Feature}_2\|$. At inference only the RGB branch runs. On Kinetics validation the model scores 68.7%, which is 3.5 points above the MARS RGB-only baseline and 0.9 points below MARS+RGB; on UCF101 split 1 it scores 95.5%, 0.9 points above MARS and 0.1 points below MARS+RGB. The paper reads these numbers as evidence that the bypass branches recover most of the accuracy lost when optical flow is removed.

Load-bearing premise

The load-bearing premise is that the three bypass branches active during inference add negligible computational cost, so dropping optical flow still yields a real-time speedup; the paper reports no latency measurements to verify this.

Editorial extensions

If this is right

Autonomous driving systems can run pedestrian action recognition on RGB frames alone, eliminating the optical-flow computation that two-stream methods require at inference.
On Kinetics validation, BERS scores 68.7%, 3.5 points above the MARS RGB-only stream and only 0.9 points below MARS+RGB; on UCF101 split 1 it scores 95.5%, 0.9 points above MARS and 0.1 points below MARS+RGB.
For static actions such as making sushi, eating cake, and reading newspaper, the RGB-only model improves over MARS by roughly 10 to 12 percentage points, suggesting it handles motion-static cases better.
The expensive optical-flow stage can be performed offline during training, and the deployed model is a single RGB stream that can be fine-tuned from Kinetics to UCF101.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the bypass branches are indeed cheap at inference, the same training-time distillation plus multi-branch pooling recipe could be transferred to stronger two-stream teachers, potentially closing the remaining gap to MARS+RGB while keeping inference RGB-only.
The large gains on static actions suggest the flow-distilled RGB stream learns a motion prior that suppresses spurious dynamic cues; one testable prediction is that its feature maps should show less activation on background motion than a plain RGB stream.
The paper reports no timing data, so the real-time advantage remains an open empirical question; a straightforward benchmark of frames-per-second on embedded vehicle hardware would settle it.
An ablation removing the three bypass branches one at a time would separate the contribution of distillation from that of multi-branch pooling, since the paper's design changes both relative to MARS.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing: this is a MARS variant with a three-branch RGB stream, and the paper's advertised real-time improvement is not measured anywhere. The tables also contradict each other: OUR is 95.5% on UCF101 split 1 in Table 1 and 97.2% on the average of 3 splits in Table 3, a gap the text never explains. The stress-test note holds up on reading.

What it does well: it targets a legitimate practical problem—optical flow is expensive at inference—and the idea of distilling flow features into an RGB-only model, already in MARS, is sensible. The specific twist here is three auxiliary branches from different ResNeXt blocks pooled and concatenated before classification, plus a weighted feature-matching loss between the RGB student and the pre-trained flow teacher. That is a real, if incremental, architectural variation. The paper reports numbers on Kinetics-400 and UCF101 and compares with several baselines. The related work is cited appropriately, including MARS.

The soft spots are substantial. No runtime measurement exists: no latency, no FLOPs, no parameter count. The architecture adds three active branches at inference, so relative speed versus MARS is unknown and could be worse. The abstract's 'accuracy does not decrease' is contradicted by the paper's own Table 1, where the model is 0.9% lower than MARS+RGB on Kinetics and 0.1% lower on UCF101-1. There are no ablations isolating the bypass contribution, no lambda sensitivity, no error bars, and Table 2 cherry-picks three static-action classes without a full comparison. The UCF101 discrepancy is large enough to suggest a possible training/testing protocol change or a typo; either way, the paper doesn't address it.

Would I cite this? Not in the current form. It's a plausible engineering idea that needs proper timing and ablation experiments to be convincing. The right reader is someone compiling a survey of distillation-based RGB-only action recognition, but they'd be better served by the original MARS paper.

Recommendation: if this lands on my desk as an editor, I'd desk-reject it. The central claim is unsupported and the accuracy reporting is internally inconsistent, so it is not acceptable as-is.

Referee Report

4 major / 5 minor

Summary. The paper proposes the Bypass Enhancement RGB Stream (BERS) model for video action recognition, targeting autonomous-driving pedestrian behaviour. BERS trains an RGB stream with a combination of a classification loss on three bypass branches and a distillation loss that aligns the RGB feature with an optical-flow teacher (TV-L1 flow plus 3D ResNeXt101), following the MARS framework. At inference, the flow branch is removed and only the RGB stream with its three bypass branches is deployed. The authors report accuracy on Kinetics400 and UCF101 and claim a significant improvement in real-time performance with no accuracy loss.

Significance. The idea of distilling motion information into a single RGB stream is practically relevant for latency-sensitive applications, and the paper builds directly on the MARS baseline by adding a multi-branch bypass structure. The accuracy comparisons on two standard benchmarks, even if not state-of-the-art, are descriptive. However, the paper's central contribution — a real-time gain without accuracy loss — is not substantiated by any runtime measurement, and the contribution of the bypass branches is not isolated by ablation. If these deficiencies were addressed, the work could be a useful incremental improvement over MARS, but in its current form the evidence does not support the advertised claims.

major comments (4)

[Section 4, Tables 1 and 3] The UCF101 accuracy is reported as 95.5% (split 1) in Table 1 and 97.2% (average of 3 splits) in Table 3 without explicitly flagging this protocol change in the results text. Although the dataset section mentions averaging over 3 splits, the two numbers are used interchangeably in the comparisons, and the 1.7-point gap is large. More importantly, the headline claim that accuracy does not decrease is ambiguous: relative to MARS+RGB, BERS is 0.9% lower on Kinetics and 0.1% lower on UCF101-1 in Table 1, so the claim holds only if the baseline is MARS, not the two-stream RGB+flow combination. The authors must state the intended baseline and consistently use one evaluation protocol.
[Section 3 and Section 4] No runtime measurement, latency figure, FLOP count, or parameter count is reported anywhere, although the abstract and conclusion assert that real-time performance is significantly improved. The architecture in Figure 1 keeps the three bypass branches and a fully connected layer active in inference mode (Eq. (1)), so the computational cost of BERS relative to MARS, which also uses only the RGB stream at inference, is unknown and could even be higher. The paper should provide hardware-specific inference time and complexity measurements for BERS and for the baselines it claims to beat.
[Section 3, Eq. (1)] The contribution of the three bypass branches is not ablated. No experiment removes the branches, replaces them with a single feature map, or varies the number of branches. Therefore the reported accuracy gains over MARS (3.5% on Kinetics and 0.9% on UCF101-1 in Table 1) cannot be causally attributed to the bypass mechanism; they may be due to other training differences, such as the distillation weight lambda, the number of training iterations, or fine-tuning details. An ablation study is required to support the central architectural claim.
[Section 4, Table 2] The static-action comparison is made on only three classes, with very low absolute accuracies (e.g., 14.1% for 'Eating cake'), no other method except MARS, no standard deviation, and no indication of the number of test videos per class. The statement that the model 'standout' on static actions is not supported by this table. This table should either be expanded to the full set of static classes with proper statistics or removed from the paper.

minor comments (5)

[Section 3, Eq. (2)] The distillation weight lambda is chosen by grid search, but its value is never reported; include the selected value and a sensitivity analysis.
[Section 1] 'HDMI51' is a typo for 'HMDB51'.
[Figure 1] The figure is low-resolution and the branch labels, pooling names, and loss-equation references are difficult to read; a higher-quality figure with a legend is needed.
[Throughout] 'COV' should be 'CONV', and 'Avg poll' in Eq. (1) should be 'Avg pool'.
[Table 3] Comparing against I3D (98.0% on UCF101) is not sufficient to justify the claim of state-of-the-art performance; more recent methods should be included or the claim mitigated.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: an empirical benchmark paper whose accuracy claims are tested on external datasets; real-time claim lacks latency evidence but is not circular.

full rationale

This paper reports benchmark comparisons on Kinetics400 and UCF101 using an external baseline (MARS, [28]) and standard backbones (ResNeXt101). The training objective L = La + lambda||Feature1 - Feature2|| is a distillation loss that encourages the RGB branch to mimic the optical-flow branch; this is a transfer mechanism, not a definitional rewriting of the reported accuracy numbers. Lambda is tuned by grid search and results are given on the validation set, which is a statistical overfitting/selection concern, but no fitted quantity is relabeled as a prediction. The architecture adds active bypass branches during inference, yet no latency measurements support the 'real-time' claim; that is an evidentiary gap, not circularity. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via prior work by the same authors. The central accuracy comparisons are independent external benchmark results, so the derivation chain is not circular.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities are introduced; the bypass branches are a network architecture component rather than a postulated entity. The main free parameter is the distillation weight lambda, which is tuned by an unspecified grid search.

free parameters (1)

lambda (distillation weight) = not reported (grid-searched)
Eq. (2) weights the L2 feature-matching loss between the flow branch and the RGB branch; the value is chosen by grid search and not disclosed, so the reported results depend on an unstated tuning choice.

assumptions (4)

domain assumption Knowledge distillation via L2 feature matching transfers useful motion information from the optical flow branch to the RGB branch.
Invoked in Eq. (2) and Section 3 (Training phase); no validation shows that the distilled features are what drive the accuracy gain.
domain assumption TV-L1 optical flow computed during training is a sufficient motion representation for pedestrian actions.
Section 3 states TV-L1 is used to obtain optical flow frames; the paper does not compare with other flow estimators or analyze sensitivity.
ad hoc to paper The three bypass branches and the fully connected layer in Eq. (1) improve accuracy without prohibitive inference cost.
This is the paper's own design premise; no ablation removes the branches or measures their runtime, so the assumption is unsupported.
domain assumption Baseline numbers taken from prior work (e.g., MARS in [28]) are comparable under the same evaluation protocols.
Tables 1 and 3 mix split-1 and 3-split averaging, and some baseline numbers are quoted from other papers; the comparability is not established.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Bypass Enhancement RGB Stream Model for Pedestrian Action Recognition of Autonomous Vehicles." pith.science (2026). https://pith.science/paper/F4S74YJZ

@misc{pith2026190805674,
  author       = {Pith},
  title        = {Pith review of: Bypass Enhancement RGB Stream Model for Pedestrian Action Recognition of Autonomous Vehicles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F4S74YJZ}},
  note         = {Machine review of arXiv:1908.05674}
}

read the original abstract

Pedestrian action recognition and intention prediction is one of the core issues in the field of autonomous driving. In this research field, action recognition is one of the key technologies. A large number of scholars have done a lot of work to im-prove the accuracy of the algorithm for the task. However, there are relatively few studies and improvements in the computational complexity of algorithms and sys-tem real-time. In the autonomous driving application scenario, the real-time per-formance and ultra-low latency of the algorithm are extremely important evalua-tion indicators, which are directly related to the availability and safety of the au-tonomous driving system. To this end, we construct a bypass enhanced RGB flow model, which combines the previous two-branch algorithm to extract RGB feature information and optical flow feature information respectively. In the train-ing phase, the two branches are merged by distillation method, and the bypass enhancement is combined in the inference phase to ensure accuracy. The real-time behavior of the behavior recognition algorithm is significantly improved on the premise that the accuracy does not decrease. Experiments confirm the superiority and effectiveness of our algorithm.

Figures

Figures reproduced from arXiv: 1908.05674 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 30 canonical work pages

[1]

In: NIPS (2014)

Simonyan, K., and Zisserman, A.: Two -stream convolutional networks for action recogni- tion in videos. In: NIPS (2014)

work page 2014
[2]

Yu.: 3D convolutional neural networks for human action recognition

Ji, S., Xu, W., Yang, M., and K. Yu.: 3D convolutional neural networks for human action recognition. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(1), 221–231 (2013)

work page 2013
[3]

A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., and Darrell, T.: Long-term recurrent convolutional networks for visual recognition and de- scription

Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., and Darrell, T.: Long-term recurrent convolutional networks for visual recognition and de- scription. In: CVPR ( 2015)

work page 2015
[4]

In: CVPR (2017)

Carreira, J., and Zisserman, A.: Quo vadis, action recognition? A new model and the kinetics dataset. In: CVPR (2017)

work page 2017
[5]

R., and Shah, M.: UCF101: A dataset of 101 human actions classe s from videos in the wild

Soomro, K., Zamir, A. R., and Shah, M.: UCF101: A dataset of 101 human actions classe s from videos in the wild. Computer Science (2012)

work page 2012
[6]

In: ICCV (2011)

Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., and Serre, T.: HMDB: a large video data- base for human motion recognition. In: ICCV (2011). 8

work page 2011
[7]

In: CVPR (2011)

Wang, H., Klä ser, A., Schmid, C., Liu, C.: Action recognition by dense trajectories. In: CVPR (2011)

work page 2011
[8]

In: ICCV (2013)

Wang, H., Schmid, C.: Action recognition with improved trajectories. In: ICCV (2013)

work page 2013

Show all 32 references

[9]

In: DAGM (2007)

Zach, C., Pock, T., and Bischof, H.: A duality based approach for realtime TV -L1 optical flow. In: DAGM (2007)

2007
[10]

v.d., Cremers, D., and Brox, T.: Flownet: Learning optical flow with convolutional networks

Dosovitskiy, A., Fischer, P., Ilg, E., Häusser, P., Hazırba¸s, C., Golkov, V., Smagt, P. v.d., Cremers, D., and Brox, T.: Flownet: Learning optical flow with convolutional networks. In: ICCV (2015)

2015
[11]

and Brox, T.: Flo wnet 2.0: Evo- lution of optical flow estimation with deep networks

Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A. and Brox, T.: Flo wnet 2.0: Evo- lution of optical flow estimation with deep networks. In: CVPR (2017)

2017
[12]

arXiv:1712.00636 (2017)

Wu, C.Y., Zaheer, M., Hu, H., Manmatha, R., Smola, A.J., and Krhenbhl, P.: Compressed Video Action Recognition. arXiv:1712.00636 (2017)

2017 arXiv
[13]

M.: Motion from Color

Golland, P., Bruckstein, A. M.: Motion from Color. In: Computer Vision and Image Under- standing (1997)

1997
[14]

In: International Journal of Information & Computation Technology (2014)

Singla, N.: Motion detection based on frame difference method. In: International Journal of Information & Computation Technology (2014)

2014
[15]

In: CVPR (2018)

Gao, R., Xiong, B., and Grauman, K.: Im2Flow: Motio n Hallucination from Static Images for Action Recognition. In: CVPR (2018)

2018
[16]

In: ACCV (2018)

Zhu, Y., Lan, Z., Newsam, S., and Hauptmann, A.G.: Hidden two-stream convolutional net- works for action recognition. In: ACCV (2018)

2018
[17]

In: IJCAI (2019)

Tang, Y., Ma, L., and Zhou, L.: Hallucinating optical flow features for video classification. In: IJCAI (2019)

2019
[18]

In: ACM SIGKDD (2006)

Bucila, C., Caruana, R., and Niculescu-Mizil, A.: Model Compression. In: ACM SIGKDD (2006)

2006
[19]

In: NIPS workshop (2015)

Hinton, G., Vinyals, Oriol., and Dean, J.: Distilling the knowledge in a neural network. In: NIPS workshop (2015)

2015
[20]

Computer Science, 14(7), 38-39 (2015)

Papamakarios, G.: Distilling Model Knowledge. Computer Science, 14(7), 38-39 (2015)

2015
[21]

JMIR.org, 16(1), 2023-2049 (2015)

Vapnik, V., Izmailov, R.: Learning Using Privileged Information: Similarity Control and Knowledge Transfer. JMIR.org, 16(1), 2023-2049 (2015)

2015
[22]

Computer Science (2015)

Lopez-Paz, D., Bottou, L., Schö lkopf, B., Vapnik, V.: Unifying distillation and privileged information. Computer Science (2015)

2015
[23]

H., Song, B

Lee, S., Kim, D. H., Song, B. C.: Self-supervised knowledge distillation using singular value decomposition. In: ECCV (2018)

2018
[24]

In: 2018 ACM Multimedia Con- ference on Multimedia Conference

Liu, Y., Sheng, L., Shao, J., Yan, J., Xiang, S., Pan, C.: Multi-label image classification via knowledge distillation from weakly-supervised detection. In: 2018 ACM Multimedia Con- ference on Multimedia Conference. ACM (2018)

2018
[25]

arXiv preprint arXiv:1907.02226

Lee, S., and Song, B.: Graph-based knowledge distillation by multi-head attention network. arXiv preprint arXiv:1907.02226. (2019) 9

2019 arXiv
[26]

In: CVPR (2019)

Liu, Y., Chen, K., Liu, C., Qin, Z., Luo, Z., and Wang, J.: Structured knowledge distillation for semantic segmentation. In: CVPR (2019)

2019
[27]

In: CVPR (2019)

Park, W., Kim, D., Lu, Y., and Cho, M: Relational knowledge distillation. In: CVPR (2019)

2019
[28]

In: CVPR (2019)

Crasto, N., Weinzaepfel, P., Alahari, K., and Schmid, C.: MARS: Motion-augmented RGB stream for action recognition. In: CVPR (2019)

2019
[29]

arXiv preprint arXiv:1705.06950 (2017)

Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., et al.: The kinetics human action video dataset. arXiv preprint arXiv:1705.06950 (2017)

2017 arXiv
[30]

R., and Shah, M.: UCF101: a dataset of 101 human actions classes from videos in the wild

Soomro, K., Zamir, A. R., and Shah, M.: UCF101: a dataset of 101 human actions classes from videos in the wild. Computer Science (2012)

2012
[31]

In: CVPR (2018)

Hara, Kensho., Kataoka, H., and Satoh, Y: Can spatiotemporal 3D CNNs retrace the history of 2D CNNs and ImageNet. In: CVPR (2018)

2018
[32]

In: CVPR (2017)

Xie, S., Girshick, R., Dollá r, Piotr, Tu, Z., and He, K.: Aggregated residual transformations for deep neural networks. In: CVPR (2017)

2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

In: NIPS (2014)

Simonyan, K., and Zisserman, A.: Two -stream convolutional networks for action recogni- tion in videos. In: NIPS (2014)

work page 2014

[2] [2]

Yu.: 3D convolutional neural networks for human action recognition

Ji, S., Xu, W., Yang, M., and K. Yu.: 3D convolutional neural networks for human action recognition. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(1), 221–231 (2013)

work page 2013

[3] [3]

A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., and Darrell, T.: Long-term recurrent convolutional networks for visual recognition and de- scription

Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., and Darrell, T.: Long-term recurrent convolutional networks for visual recognition and de- scription. In: CVPR ( 2015)

work page 2015

[4] [4]

In: CVPR (2017)

Carreira, J., and Zisserman, A.: Quo vadis, action recognition? A new model and the kinetics dataset. In: CVPR (2017)

work page 2017

[5] [5]

R., and Shah, M.: UCF101: A dataset of 101 human actions classe s from videos in the wild

Soomro, K., Zamir, A. R., and Shah, M.: UCF101: A dataset of 101 human actions classe s from videos in the wild. Computer Science (2012)

work page 2012

[6] [6]

In: ICCV (2011)

Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., and Serre, T.: HMDB: a large video data- base for human motion recognition. In: ICCV (2011). 8

work page 2011

[7] [7]

In: CVPR (2011)

Wang, H., Klä ser, A., Schmid, C., Liu, C.: Action recognition by dense trajectories. In: CVPR (2011)

work page 2011

[8] [8]

In: ICCV (2013)

Wang, H., Schmid, C.: Action recognition with improved trajectories. In: ICCV (2013)

work page 2013