RECORD
Advancing Compressed Video Action Recognition through Progressive Knowledge Distillation
Computer Vision and Pattern Recognition · Submitted 2 Jul 2024
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Our experiments demonstrate the effectiveness of training the ICs with PKD compared to standard cross-entropy-based training, showing IC accuracy improvements of up to 5.87% and 11.42% on the UCF-101 and HMDB-51 datasets, respectively.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Compressed video action recognition classifies video samples by leveraging the different modalities in compressed videos, namely motion vectors, residuals, and intra-frames. For this purpose, three neural networks are deployed, each dedicated to processing one modality. Our observations indicate that the network processing intra-frames tend to converge to a flatter minimum than the network processing residuals, which in turn converges to a flatter minimum than the motion vector network. This hierarchy in convergence motivates our strategy for knowledge transfer among modalities to achieve flatter minima, which are generally associated with better generalization. With this insight, we propose Progressive Knowledge Distillation (PKD), a technique that incrementally transfers knowledge across the modalities. This method involves attaching early exits (Internal Classifiers - ICs) to the three networks. PKD distills knowledge starting from the motion vector network, followed by the residual, and finally, the intra-frame network, sequentially improving IC accuracy. Further, we propose the Weighted Inference with Scaled Ensemble (WISE), which combines outputs from the ICs using learned weights, boosting accuracy during inference. Our experiments demonstrate the effectiveness of training the ICs with PKD compared to standard cross-entropy-based training, showing IC accuracy improvements of up to 5.87% and 11.42% on the UCF-101 and HMDB-51 datasets, respectively. Additionally, WISE improves accuracy by up to 4.28% and 9.30% on UCF-101 and HMDB-51, respectively.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Compressed video action recognition classifies video samples by leveraging the different modalities in compressed videos, namely motion vectors, residuals, and intra-frames. | background 1.00 | 0.07 | 0.02 | asserted 1.00 |
| s1 | For this purpose, three neural networks are deployed, each dedicated to processing one modality. | method 0.70 | 0.35 | 0.02 | asserted 1.00 |
| s2 | Our observations indicate that the network processing intra-frames tend to converge to a flatter minimum than the network processing residuals, which in turn converges to a flatter minimum than the motion vector network. | result 0.99 | 0.91 | 0.06 | asserted 0.50 |
| s3 | This hierarchy in convergence motivates our strategy for knowledge transfer among modalities to achieve flatter minima, which are generally associated with better generalization. | interpretation 0.98 | 0.65 | 0.10 | asserted 0.86 |
| s4 | With this insight, we propose Progressive Knowledge Distillation (PKD), a technique that incrementally transfers knowledge across the modalities. | contribution 0.88 | 0.70 | 0.11 | asserted 0.99 |
| s5 | This method involves attaching early exits (Internal Classifiers - ICs) to the three networks. | method 0.97 | 0.77 | 0.03 | asserted 0.99 |
| s6 | PKD distills knowledge starting from the motion vector network, followed by the residual, and finally, the intra-frame network, sequentially improving IC accuracy. | method 0.99 | 0.84 | 0.08 | asserted 1.00 |
| s7 | Further, we propose the Weighted Inference with Scaled Ensemble (WISE), which combines outputs from the ICs using learned weights, boosting accuracy during inference. | method 0.53 | 0.89 | 0.11 | asserted 1.00 |
| s8 | Our experiments demonstrate the effectiveness of training the ICs with PKD compared to standard cross-entropy-based training, showing IC accuracy improvements of up to 5.87% and 11.42% on the UCF-101 and HMDB-51 datasets, respectively. | result 1.00 | 0.97 | 0.27 | asserted 1.00 |
| s9 | Additionally, WISE improves accuracy by up to 4.28% and 9.30% on UCF-101 and HMDB-51, respectively. | result 1.00 | 0.96 | 0.16 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Compressed video action recognition classifies video samples by leveraging the different modalities in compressed videos, namely motion vectors, residuals, and intra-frames. For this purpose, three neural networks are deployed, each dedicated to processing one modality. Our observations indicate that the network processing intra-frames tend to converge to a flatter minimum than the network processing residuals, which in turn converges to a flatter minimum than the motion vector network. This hierarchy in convergence motivates our strategy for knowledge transfer among modalities to achieve flatter minima, which are generally associated with better generalization. With this insight, we propose Progressive Knowledge Distillation (PKD), a technique that incrementally transfers knowledge across the modalities. This method involves attaching early exits (Internal Classifiers - ICs) to the three networks. PKD distills knowledge starting from the motion vector network, followed by the residual, and finally, the intra-frame network, sequentially improving IC accuracy. Further, we propose the Weighted Inference with Scaled Ensemble (WISE), which combines outputs from the ICs using learned weights, boosting accuracy during inference. Our experiments demonstrate the effectiveness of training the ICs with PKD compared to standard cross-entropy-based training, showing IC accuracy improvements of up to 5.87% and 11.42% on the UCF-101 and HMDB-51 datasets, respectively. Additionally, WISE improves accuracy by up to 4.28% and 9.30% on UCF-101 and HMDB-51, respectively.
Human review
How the literature uses it
1 arXiv paper cite this in 3 sentences
Papers citing it · 2 sentences without a confident label
Compares with it 1 paper
-
We compare to existing methods in CD [ref] and find that our framework leverages complementary dense-spatial sparse-temporal learning effectively to consistently outperform them by up to $14%$ and achieves new state-of-the-art performance at $19×$ lower inference costs.
Towards Scalable Modeling of Compressed Videos for Efficient Action Recognition · Quantitative Results · 1.00
What it cites · 6 sentences without a confident label
Uses the method or tool of 2 papers
-
Architecture and Backbone We used two types of backbone networks as illustrated in Figure [ref]: 1) CoViAR [ref] , and 2) TEAM-Net [ref] .
TEAM-Net: Multi-modal Learning for Video Action Recognition with Partial Decoding · Datasets and Implementation Details · 0.91
-
Further optimization was carried out on UCF-101 and HMDB-51, employing mini-batch training and the Adam optimizer [ref] with a weight decay of 0.0001, an epsilon value of 0.001, an initial learning rate of 0.003 for the I-frame input, a learning rate of 0.01 for the MV input, and a learning rate of 0.005 for the R input.
Adam: A Method for Stochastic Optimization · Datasets and Implementation Details · 1.00
Uses data or values from 1 paper
-
Datasets We evaluated our method on two action recognition datasets: UCF-101 [ref] and HMDB-51 [ref] .
UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild · Datasets and Implementation Details · 0.89
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.