Stacked Spatio-Temporal Graph Convolutional Networks for Action Segmentation

Ajay Divakaran; Larry S. Davis; Pallabi Ghosh; Yi Yao

RECORD

Stacked Spatio-Temporal Graph Convolutional Networks for Action Segmentation

Pallabi Ghosh, Yi Yao, Larry S. Davis, Ajay Divakaran

Computer Vision and Pattern Recognition · Submitted 26 Nov 2018

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim Our Stacked-STGCN in general achieves 4.0% performance improvement over the best reported results in F1 score on CAD120 and 1.3% in mAP on Charades using VGG features.

how it is stated · asserted 1.00 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID1811.10575 (version 6)

Pith ID4EWFK2P6, Pith's permanent record of this paper and its review

CitationAjay Divakaran, Larry S. Davis, Pallabi Ghosh, Yi Yao. "Stacked Spatio-Temporal Graph Convolutional Networks for Action Segmentation." arXiv:1811.10575 (2018). https://pith.science/paper/4EWFK2P6

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 5,189 input tokens.

What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.10 · human participants 0.16 · controlled comparison 0.49 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.97 · mentions code or data availability 0.03.

The abstract, sentence by sentence

arXiv listing abstract v6. Results and the main claim are underlined; show every role.

We propose novel Stacked Spatio-Temporal Graph Convolutional Networks (Stacked-STGCN) for action segmentation, i.e., predicting and localizing a sequence of actions over long videos. We extend the Spatio-Temporal Graph Convolutional Network (STGCN) originally proposed for skeleton-based action recognition to enable nodes with different characteristics (e.g., scene, actor, object, action, etc.), feature descriptors with varied lengths, and arbitrary temporal edge connections to account for large graph deformation commonly associated with complex activities. We further introduce the stacked hourglass architecture to STGCN to leverage the advantages of an encoder-decoder design for improved generalization performance and localization accuracy. We explore various descriptors such as frame-level VGG, segment-level I3D, RCNN-based object, etc. as node descriptors to enable action segmentation based on joint inference over comprehensive contextual information. We show results on CAD120 (which provides pre-computed node features and edge weights for fair performance comparison across algorithms) as well as a more complex real-world activity dataset, Charades. Our Stacked-STGCN in general achieves 4.0% performance improvement over the best reported results in F1 score on CAD120 and 1.3% in mAP on Charades using VGG features.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	We propose novel Stacked Spatio-Temporal Graph Convolutional Networks (Stacked-STGCN) for action segmentation, i.e., predicting and localizing a sequence of actions over long videos.	contribution 1.00	0.84	0.12	asserted 0.96
s1	We extend the Spatio-Temporal Graph Convolutional Network (STGCN) originally proposed for skeleton-based action recognition to enable nodes with different characteristics (e.g., scene, actor, object, action, etc.), feature descriptors with varied lengths, and arbitrary temporal edge connections to account for large graph deformation commonly associated with complex activities.	contribution 0.64	0.84	0.04	asserted 0.99
s2	We further introduce the stacked hourglass architecture to STGCN to leverage the advantages of an encoder-decoder design for improved generalization performance and localization accuracy.	contribution 0.53	0.89	0.05	asserted 0.96
s3	We explore various descriptors such as frame-level VGG, segment-level I3D, RCNN-based object, etc. as node descriptors to enable action segmentation based on joint inference over comprehensive contextual information.	method 0.76	0.86	0.03	asserted 0.84
s4	We show results on CAD120 (which provides pre-computed node features and edge weights for fair performance comparison across algorithms) as well as a more complex real-world activity dataset, Charades.	method 0.78	0.81	0.04	asserted 0.99
s5	Our Stacked-STGCN in general achieves 4.0% performance improvement over the best reported results in F1 score on CAD120 and 1.3% in mAP on Charades using VGG features.	result 1.00	0.96	0.72	asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.86.

What was examined, and what was not

sources read

arXiv listing abstract v6

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

We propose novel Stacked Spatio-Temporal Graph Convolutional Networks (Stacked-STGCN) for action segmentation, i.e., predicting and localizing a sequence of actions over long videos. We extend the Spatio-Temporal Graph Convolutional Network (STGCN) originally proposed for skeleton-based action recognition to enable nodes with different characteristics (e.g., scene, actor, object, action, etc.), feature descriptors with varied lengths, and arbitrary temporal edge connections to account for large graph deformation commonly associated with complex activities. We further introduce the stacked hourglass architecture to STGCN to leverage the advantages of an encoder-decoder design for improved generalization performance and localization accuracy. We explore various descriptors such as frame-level VGG, segment-level I3D, RCNN-based object, etc. as node descriptors to enable action segmentation based on joint inference over comprehensive contextual information. We show results on CAD120 (which provides pre-computed node features and edge weights for fair performance comparison across algorithms) as well as a more complex real-world activity dataset, Charades. Our Stacked-STGCN in general achieves 4.0% performance improvement over the best reported results in F1 score on CAD120 and 1.3% in mAP on Charades using VGG features.

Human review

How Pith checks papers → · Full audit of this paper →

References

How the literature uses it

5 arXiv papers cite this in 13 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers citing it · 5 sentences without a confident label

Contests it 1 paper

The problem with merging two branches through feature concatenation [ref] or pooling [ref] , or adding scene features as a separate graph node [ref] is that videos (and even frames in the same video) contain a variable number of objects, and this makes the learned features very noisy.

Spatio-Temporal Graph for Video Captioning with Knowledge Distillation · Object-Aware Knowledge Distillation · contest 0.81

Uses its method or tool 1 paper

Similar to previous work [ref] , we also directly model the frame sequence through a separate scene branch.

Spatio-Temporal Graph for Video Captioning with Knowledge Distillation · Scene Branch · 0.84

Compares with it 1 paper

While [ref] only use human pose features, [ref] also leverage additional contextual cues such as object features, functional relationships, etc.

Representation Learning for Dynamic Graphs: A Survey · Spatiotemporal Graphs · 1.00

Cites it as background 4 papers

Ghosh et al. [ref] recently developed a graph-based spatiotemporal CNN to exploit environmental cues for better segmentation.

Toward Ergonomic Risk Prediction via Segmentation of Indoor Object Manipulation Actions Using Spatiotemporal Convolutional Networks · Introduction · 0.91
In computer vision, graph convolutions have gained popularity for capturing relationships between objects spatially and temporally for video object understanding, as well as capturing spatio-temporal dynamics for action understanding [ref] .

Action Graphs: Weakly-supervised Action Localization with Graph Convolution Networks · Related work · 0.82
The resulting representation is then used to perform various down-stream tasks, such as action classification [ref] , action localization [ref] , relation prediction [ref] , and gaze prediction [ref] .

Spatio-Temporal Graph for Video Captioning with Knowledge Distillation · Related Work · 0.94

What it cites · 1 sentence without a confident label

Uses the method or tool of 1 paper

Further, we used the GNN-based situation recognition technique [ref] trained on the ImSitu dataset [ref] to generate the verb feature for the actor nodes.

Situation Recognition with Graph Neural Networks · Charades · 0.99

Cites as background 10 papers

Recently, thanks to the rapid development in GNNs, graph-based representation becomes a popular option for action recognition, for instance skeleton-based activity recognition using STGCN [ref] and Graph Edge Convolution Networks [ref] .

Graph Edge Convolutional Neural Networks for Skeleton Based Action Recognition · Action Recognition · 0.85
Later, a number of variations of TCNs were studied [ref] .

TricorNet: A Hybrid Temporal Convolutional and Recurrent Network for Video Action Segmentation · Action Segmentation · 0.87
In [ref] , GCN is applied to space-time graphs extracted from the whole video segment to output an accumulative descriptor, which is later combined with the aggregated frame-level features to generate action predictions.

Videos as Space-Time Region Graphs · Action Recognition · 0.83

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.