Would Mega-scale Datasets Further Enhance Spatiotemporal 3D CNNs?

Hirokatsu Kataoka; Kensho Hara; Tenga Wakamiya; Yutaka Satoh

RECORD

Would Mega-scale Datasets Further Enhance Spatiotemporal 3D CNNs?

Hirokatsu Kataoka, Tenga Wakamiya, Kensho Hara, Yutaka Satoh

Computer Vision and Pattern Recognition · Submitted 10 Apr 2020

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim Therefore, in the present paper, we conduct exploration study in order to improve spatiotemporal 3D CNNs as follows: (i) Recently proposed large-scale video datasets help improve spatiotemporal 3D CNNs in terms of video classification accuracy.

how it is stated · asserted 0.94 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID2004.04968 (version 1)

Pith IDZOI7HSNW, Pith's permanent record of this paper and its review

CitationHirokatsu Kataoka, Kensho Hara, Tenga Wakamiya, Yutaka Satoh. "Would Mega-scale Datasets Further Enhance Spatiotemporal 3D CNNs?." arXiv:2004.04968 (2020). https://pith.science/paper/ZOI7HSNW

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 9,176 input tokens.

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.29 · human participants 0.08 · controlled comparison 0.87 · randomized assignment 0.02 · states quantitative result 0.98 · compares with prior results 0.93 · mentions code or data availability 0.03.

The abstract, sentence by sentence

arXiv listing abstract v1. Results and the main claim are underlined; show every role.

How can we collect and use a video dataset to further improve spatiotemporal 3D Convolutional Neural Networks (3D CNNs)? In order to positively answer this open question in video recognition, we have conducted an exploration study using a couple of large-scale video datasets and 3D CNNs. In the early era of deep neural networks, 2D CNNs have been better than 3D CNNs in the context of video recognition. Recent studies revealed that 3D CNNs can outperform 2D CNNs trained on a large-scale video dataset. However, we heavily rely on architecture exploration instead of dataset consideration. Therefore, in the present paper, we conduct exploration study in order to improve spatiotemporal 3D CNNs as follows: (i) Recently proposed large-scale video datasets help improve spatiotemporal 3D CNNs in terms of video classification accuracy. We reveal that a carefully annotated dataset (e.g., Kinetics-700) effectively pre-trains a video representation for a video classification task. (ii) We confirm the relationships between #category/#instance and video classification accuracy. The results show that #category should initially be fixed, and then #instance is increased on a video dataset in case of dataset construction. (iii) In order to practically extend a video dataset, we simply concatenate publicly available datasets, such as Kinetics-700 and Moments in Time (MiT) datasets. Compared with Kinetics-700 pre-training, we further enhance spatiotemporal 3D CNNs with the merged dataset, e.g., +0.9, +3.4, and +1.1 on UCF-101, HMDB-51, and ActivityNet datasets, respectively, in terms of fine-tuning. (iv) In terms of recognition architecture, the Kinetics-700 and merged dataset pre-trained models increase the recognition performance to 200 layers with the Residual Network (ResNet), while the Kinetics-400 pre-trained model cannot successfully optimize the 200-layer architecture.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	How can we collect and use a video dataset to further improve spatiotemporal 3D Convolutional Neural Networks (3D CNNs)?	gap 0.50	0.04	0.02	unclear 0.97
s1	In order to positively answer this open question in video recognition, we have conducted an exploration study using a couple of large-scale video datasets and 3D CNNs.	contribution 0.83	0.93	0.05	asserted 0.91
s2	In the early era of deep neural networks, 2D CNNs have been better than 3D CNNs in the context of video recognition.	background 1.00	0.04	0.02	asserted 1.00
s3	Recent studies revealed that 3D CNNs can outperform 2D CNNs trained on a large-scale video dataset.	background 1.00	0.05	0.06	asserted 0.94
s4	However, we heavily rely on architecture exploration instead of dataset consideration.	gap 0.96	0.16	0.03	asserted 1.00
s5	Therefore, in the present paper, we conduct exploration study in order to improve spatiotemporal 3D CNNs as follows: (i) Recently proposed large-scale video datasets help improve spatiotemporal 3D CNNs in terms of video classification accuracy.	contribution 0.93	0.68	0.47	asserted 0.94
s6	We reveal that a carefully annotated dataset (e.g., Kinetics-700) effectively pre-trains a video representation for a video classification task.	result 0.94	0.85	0.16	asserted 1.00
s7	(ii) We confirm the relationships between #category/#instance and video classification accuracy.	contribution 0.90	0.90	0.10	asserted 0.97
s8	The results show that #category should initially be fixed, and then #instance is increased on a video dataset in case of dataset construction.	result 0.61	0.90	0.15	asserted 0.97
s9	(iii) In order to practically extend a video dataset, we simply concatenate publicly available datasets, such as Kinetics-700 and Moments in Time (MiT) datasets.	method 0.93	0.86	0.08	asserted 0.91
s10	Compared with Kinetics-700 pre-training, we further enhance spatiotemporal 3D CNNs with the merged dataset, e.g., +0.9, +3.4, and +1.1 on UCF-101, HMDB-51, and ActivityNet datasets, respectively, in terms of fine-tuning.	result 1.00	0.95	0.29	asserted 1.00
s11	(iv) In terms of recognition architecture, the Kinetics-700 and merged dataset pre-trained models increase the recognition performance to 200 layers with the Residual Network (ResNet), while the Kinetics-400 pre-trained model cannot successfully optimize the 200-layer architecture.	result 0.99	0.92	0.16	asserted 1.00

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 0.98.

What was examined, and what was not

sources read

arXiv listing abstract v1

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

How can we collect and use a video dataset to further improve spatiotemporal 3D Convolutional Neural Networks (3D CNNs)? In order to positively answer this open question in video recognition, we have conducted an exploration study using a couple of large-scale video datasets and 3D CNNs. In the early era of deep neural networks, 2D CNNs have been better than 3D CNNs in the context of video recognition. Recent studies revealed that 3D CNNs can outperform 2D CNNs trained on a large-scale video dataset. However, we heavily rely on architecture exploration instead of dataset consideration. Therefore, in the present paper, we conduct exploration study in order to improve spatiotemporal 3D CNNs as follows: (i) Recently proposed large-scale video datasets help improve spatiotemporal 3D CNNs in terms of video classification accuracy. We reveal that a carefully annotated dataset (e.g., Kinetics-700) effectively pre-trains a video representation for a video classification task. (ii) We confirm the relationships between #category/#instance and video classification accuracy. The results show that #category should initially be fixed, and then #instance is increased on a video dataset in case of dataset construction. (iii) In order to practically extend a video dataset, we simply concatenate publicly available datasets, such as Kinetics-700 and Moments in Time (MiT) datasets. Compared with Kinetics-700 pre-training, we further enhance spatiotemporal 3D CNNs with the merged dataset, e.g., +0.9, +3.4, and +1.1 on UCF-101, HMDB-51, and ActivityNet datasets, respectively, in terms of fine-tuning. (iv) In terms of recognition architecture, the Kinetics-700 and merged dataset pre-trained models increase the recognition performance to 200 layers with the Residual Network (ResNet), while the Kinetics-400 pre-trained model cannot successfully optimize the 200-layer architecture.

Human review

How Pith checks papers → · Full audit of this paper →

References

How the literature uses it

20 arXiv papers cite this in 34 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers citing it · 16 sentences without a confident label

Uses its method or tool 6 papers

We used 3D-ResNet [ref] , a simple yet powerful framework for video action recognition, to recognize traffic accidents in videos.

Traffic Incident Database with Multiple Labels Including Various Perspective Environmental Information · Implementation details · 0.99
Given class-agnostic bounding boxes from a video frame, we extract RoI align [ref] features from each; * An inflated 3D Resnet50 (R3D) [ref] , pre-trained on [ref] ; * R3D, same as (iii), with a Transformer encoder to enable multimodal fusion with cross-attention.

Revisiting spatio-temporal layouts for compositional action recognition · Appearance branch and multimodal fusion · 0.97
We use R(2+1)D-152 [ref] , R(2+1)D-50 [ref] , R3D-18 [ref] , Shufflenet-V1 [ref] , Shufflenet-V2 [ref] as the different clip-level classifiers.

Skimming and Scanning for Untrimmed Video Action Recognition · Experiments · 1.00

Compares with it 5 papers

For this pain estimation dataset, we evaluate our proposed QAFE-Net against recent and SOTA pain estimation methods [ref] , as well as Former-DFER [ref] , and R3D-18 [ref] and R3D-34 [ref] baselines, again applied with a MLP as a regression head (see <Ref>).

QAFE-Net: Quality Assessment of Facial Expressions with Landmark Heatmaps · Results on UNBC-McMaster Dataset · 1.00
Compared with ResNet3D [ref] and ResNet2+1D [ref] on the same feature size, SSTCN is more effective on recognizing sign glosses using skeleton features.

Sign Language Recognition via Skeleton-Aware Multi-Model Ensemble · Ablation Study on SSTCN and 3DCNN · 1.00
Our MTNet$_S$ performs similarly to both r3d-101 [ref] and r(2+1)d-50 [ref] , while not being pre-trained on another dataset nor including their large number of parameters.

Efficient Modelling Across Time of Human Actions and Interactions · Results on HACS · 0.99

Cites it as background 5 papers

Mainstream clip-level VAR techniques [ref] is consisted of three steps. * Divide the long video into a set of short fixed-length video clips without overlapping with each other. * Apply clip-level classifier to all the clips to get their classes.

Skimming and Scanning for Untrimmed Video Action Recognition · Related Work · 0.85
However, conventional 3D ConvNets e.g. 3D-ResNet [ref] , (2+1)D-ResNet [ref] , I3D [ref] , 3D-ResNeXt [ref] typically accept square-shaped video frames as input.

Unsupervised Action Localization Crop in Video Retargeting for 3D ConvNets · Introduction · 0.87
Action recognition methods are mostly 3D CNN based [ref] .

Revisiting spatio-temporal layouts for compositional action recognition · Related work · 0.97

What it cites · 11 sentences without a confident label

Uses the method or tool of 2 papers

When we train a multi-label dataset on Mini-HVU, we assign a combined loss function with cross entropy and softmax loss by referring to Diba et al. [ref] .

Large Scale Holistic Video Understanding · Implementation details · 1.00
Moreover, 10-crop augmentation and multi-scale sizing are randomly selected in mini-batch training, where we refer to the settings of Wang et al. [ref] .

Towards Good Practices for Very Deep Two-Stream ConvNets · Implementation details · 1.00

Uses data or values from 4 papers

Basically, we conducted pre-training on Kinetics-700 [ref] , MiT [ref] , and STAIR Action (STAIR) [ref] , in addition to fine-tuning then UCF-101 [ref] , HMDB-51 [ref] , and ActivityNet [ref] .

YouTube-8M: A Large-Scale Video Classification Benchmark · Introduction · 0.94
Basically, we conducted pre-training on Kinetics-700 [ref] , MiT [ref] , and STAIR Action (STAIR) [ref] , in addition to fine-tuning then UCF-101 [ref] , HMDB-51 [ref] , and ActivityNet [ref] .

STAIR Actions: A Video Dataset of Everyday Home Actions · Introduction · 0.94
Basically, we conducted pre-training on Kinetics-700 [ref] , MiT [ref] , and STAIR Action (STAIR) [ref] , in addition to fine-tuning then UCF-101 [ref] , HMDB-51 [ref] , and ActivityNet [ref] .

A Short Note on the Kinetics-700 Human Action Dataset · Introduction · 0.94

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.