Learning Object-Action Relations from Bimanual Human Demonstration Using Graph Networks

Christian R. G. Dreher; Mirko W\"achter; Tamim Asfour

REVIEW 3 major objections 4 minor 32 references

Learning Object-Action Relations from Bimanual Human Demonstration Using Graph Networks

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A graph network trained on scene graphs of symbolic spatial relations can classify, frame by frame, the action of each hand in bimanual demonstrations without prior temporal segmentation.

desk verdict A useful bimanual action dataset and a clean per-hand graph-network baseline, but the subject-generalization numbers are likely optimistic because the YOLO detector was trained on frames from every subject. read the letter →

arxiv 1908.08391 v2 pith:IWDSGRQO submitted 2019-08-22 cs.RO

Christian R. G. Dreher , Mirko Wächter , Tamim Asfour This is my paper · ORCID

classification cs.RO

keywords bimanualactionrecognitionscenegraphsgraphnetworksspatialrelationsRGB-Ddatasetprogrammingbydemonstrationper-handclassificationsegmentation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a bimanual manipulation action cannot always be captured by one action label, and that each hand should get its own label. It presents a system that turns raw RGB-D video into scene graphs — nodes for objects and hands, edges for symbolic spatial relations such as above, contact, or inside — and trains a graph network to predict, frame by frame, the action of each hand. On a new dataset of 540 kitchen and workshop recordings with per-hand ground truth, the classifier places the true action in its top 3 predictions with macro F1 of 0.86, without any prior temporal action segmentation. The point of the claim is that a robot observing a human teacher could segment and recognize bimanual actions directly from object–hand relations, which is the granularity programming by demonstration needs.

What carries the argument

The load-bearing object is the scene graph, a graph whose nodes are detected object instances plus the two hands and whose edges carry one-hot encodings of 15 spatial relations (contact, above, below, left, right, front, behind, inside, surround, moving together, halting together, fixed moving together, getting close, moving apart, stable) plus a temporal edge connecting one object instance across frames. Ten consecutive frame graphs are concatenated, preserving spatial edges within a frame and adding temporal edges across frames. The classifier is an encode-process-decode graph network with ten processing steps, trained on the right-hand action and on mirrored graphs for the left hand; the output probability distribution over 14 action classes is stored in the graph's global attribute.

What would settle it

Take a held-out set of bimanual recordings containing hammering, sawing, and pouring; replace the pipeline's depth-cluster 3D boxes with boxes from manual annotation or segmentation masks, recompute the scene graphs, and retrain and evaluate the same classifier. If top-3 macro F1 does not improve over 0.86, the bounding-box bottleneck claim is wrong; if it improves substantially, the central mechanism depends on relation quality.

Watch

Extended reading notes

Core claim

The central claim is that object-action relations can be learned from symbolic scene graphs alone: a graph-network classifier fed only object classes, hand classes, and spatial relations between them can identify the true executed action of each hand within its top 3 predictions on a frame-by-frame basis, reaching a macro F1 of 0.86, and a macro F1 of 0.63 for the single top prediction. The paper further claims that per-hand labeling is necessary because bimanual motion contains simultaneous distinct actions, and that mirroring the scene graph lets one model trained on the right hand also classify the left hand. Ablations support the representation choice: keeping only contact relations lowers the macro F1 to 0.46, replacing symbolic relations with raw centroid coordinates lowers it to 0.31, and removing temporal links lowers it to 0.60.

Load-bearing premise

The load-bearing premise is that the automatically estimated 3D bounding boxes are accurate enough that the extracted spatial relations match reality; the paper reports that this fails for thin objects like hammers and saws and for bottles, and attributes the largest prediction errors to wrong relations from misplaced 3D boxes.

Editorial extensions

If this is right

Bimanual demonstrations can be segmented frame-wise without a separate temporal segmentation stage, since each frame receives per-hand action candidates.
The graph representation is invariant to the number and order of object instances, so the classifier generalizes across scenes with different object counts.
Mirroring lets one network serve both hands, effectively doubling the training data without extra annotations.
Symbolic spatial relations carry more usable signal than raw 3D centroids, since replacing them with centroids drops macro F1 from 0.63 to 0.31.
A short temporal window of about 333 ms already improves classification, and longer sequence information is a plausible next step.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the 3D bounding boxes were replaced by segmentation-based object extents, the paper's own error analysis predicts that confusions among pour, saw, hammer, and place would shrink and top-1 performance would approach top-3 performance.
Adding the teacher's head and the table as graph nodes would give the classifier reference points that could disambiguate drink from lift and place, directly addressing the confusions the paper reports.
Because the classifier operates on symbolic relations, the same trained graph network could be transferred to another robot's observations if the object detector and depth source change, without retraining the relation model.
A testable extension is to feed the classifier scene graphs built from ground-truth object poses instead of estimated boxes; the gap in F1 would quantify how much of the remaining error is perceptual rather than representational.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Worth a look if you work on programming by demonstration or action recognition. The thing that's actually new is the dataset: 540 RGB-D recordings of six subjects doing nine kitchen/workshop tasks, with per-hand frame-wise action labels. That alone is a solid resource. The method is a sensible pipeline — YOLO + OpenPose to get objects and hands, depth clustering for 3D boxes, then 15 symbolic spatial relations built into a scene graph, and an encode-process-decode graph network that predicts one action per hand from a temporal stack of ten graphs. The per-hand formulation is a genuine gap and they handle it cleanly by mirroring the scene graph for the left hand.

The reported top-3 macro F1 of 0.86 on leave-one-subject-out is fine as a headline, but the top-1 macro F1 of 0.63 is the more honest number. The ablation study is a real plus: removing spatial relations drops F1 to 0.31, contact-only to 0.46, and no temporal concatenation to 0.60. That tells you the representation is doing the work.

The soft spots, in order. First, and nobody seems to have spotted this: the YOLO detector is trained on 5413 frames sampled from every recording, including all subjects. The evaluation section only describes the train/test split for the classifier; it never says YOLO is retrained per fold. So for each held-out subject, the detector has already seen frames from that subject's videos. That means the scene graphs fed to the graph network aren't produced under genuinely unseen-subject conditions. I don't think this sinks the paper — the core architecture is still interesting — but it pulls the 0.86 down toward "the detector already knows this person." The authors should state the split explicitly or retrain YOLO per fold.

Second, the ground truth is a single annotator. That is common for a dataset of this size but should be acknowledged as a limitation, and inter-annotator agreement would be cheap to add.

Third, no error bars or per-fold numbers. With six subjects, one subject can move the macro average by a lot. Fourth, no comparison to any baseline, which they justify by saying there is no bimanual baseline. But they could compare to single-label variants or to a CRF on the same features. The absence weakens the "graph networks are useful here" claim.

Overall: the dataset is worth citing, the method is a reasonable first attempt, and the paper is honest about the 3D-box problems. For peer review, I'd send it out rather than desk-reject. I'd ask for the YOLO split issue to be resolved, per-fold results, and ideally a second annotator or at least a paragraph on why that's not feasible.

Referee Report

3 major / 4 minor

Summary. The paper presents a system for frame-wise classification of bimanual actions from RGB-D video. The system extracts 2D object and hand detections (YOLO, OpenPose), converts them to 3D bounding boxes via depth clustering, computes symbolic spatial relations, and builds a scene graph. A graph-network classifier is trained on these scene graphs to predict the action of each hand per frame, using temporal concatenation of 10 frames and a mirroring strategy for the left hand. The authors contribute a new dataset of 6 subjects performing 9 bimanual tasks (540 recordings, about 2h18m) with per-hand action labels, and they evaluate with leave-one-subject-out cross-validation. The headline result is a macro F1 of 0.86 when a prediction is considered correct if the ground-truth action is in the top 3 predictions. An ablation study shows that removing spatial relations, contact relations, or temporal concatenation degrades performance.

Significance. If the reported result holds, the paper makes a useful empirical contribution: a full pipeline from raw RGB-D to per-hand action labels without a prior temporal segmentation, using a graph network to handle variable-size scenes. The new bimanual dataset with per-hand labels is a valuable resource for the community, and the ablation study gives insight into which features matter. The graph-network architecture is well matched to the symbolic scene-graph representation. However, the central subject-generalization claim is threatened by a potential data-leakage issue in the YOLO training set, and the evaluation lacks variance reporting. These issues need to be addressed before the result can be considered reliable.

major comments (3)

[III-A and IV] The YOLO object detector is trained on 5413 frames sampled at about 10 random frames per recording (Section III-A), and Section IV states that 'for each involved training process the dataset was split into a training and a testing set' without specifying whether this split applies to YOLO. If YOLO's training frames include frames from the held-out test subject in each leave-one-subject-out fold, then the scene graphs for the test fold are produced by a detector that has already seen that subject's data. This would inflate the reported top-3 macro F1 of 0.86 relative to a genuinely unseen subject, because more accurate 2D detections lead to more correct 3D bounding boxes and spatial relations, which are exactly the classifier's inputs. Please clarify whether YOLO was retrained per fold with test-subject frames excluded; if not, re-run the evaluation with a subject-disjoint YOLO training set or otherwise quantify the impact of this overlap.
[IV, Table I] The paper reports only aggregate macro F1 scores across all six leave-one-subject-out folds, with no per-fold breakdown, variance, or confidence intervals. With only six subjects, the 0.86 top-3 macro F1 could be dominated by one or two folds. Please report the per-fold scores and the mean/standard deviation or a confidence interval for the macro F1, for both top-1 and top-3 evaluations.
[III-A] The ground-truth action labels and the 5413 object bounding-box labels were produced by a single annotator, and no inter-annotator agreement is reported. This limits the reliability of the ground truth and the detector training labels. The paper should at least acknowledge this limitation explicitly and, if possible, provide a measure of annotation consistency on a subset.

minor comments (4)

[III-C, Figure 4] The label 'MLP input size 256 neurons' in Figure 4 is ambiguous; it likely means each MLP has two hidden layers of 256 neurons. Please clarify the notation.
[IV, Table I] The table headers abbreviate 'Precision' and 'Weighted' as 'Precis.' and 'Weigh.'; consider using the full words for clarity.
[Abstract and IV] The abstract emphasizes the top-3 macro F1 of 0.86, while the top-1 macro F1 is only 0.63. Although the top-3 metric is transparently defined, the paper should more prominently discuss the practical interpretation of top-3 accuracy for the intended use in programming by demonstration.
[III-A] The sentence 'Wächter and Asfour [9] used a similar detail of labeling in which the hand approaches an object and, after using it, retreats' is slightly unclear; consider rephrasing to say that their labeling includes approach and retreat phases.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central F1 result is an empirical cross-validation measurement, not a quantity derived from or defined by its own inputs.

full rationale

The paper's central claim is a measured action-classification macro F1 of 0.86 under leave-one-subject-out cross-validation. The graph-network classifier is trained on scene graphs from training subjects and evaluated on a held-out subject, so the reported top-3 accuracy is an empirical result rather than a quantity that reduces to the training input by construction. I checked the derivation chain: scene graphs are constructed from YOLO detections, OpenPose hands, depth-based 3D bounding boxes, and the spatial-relation set of Ziaeetabar et al.; the probability distribution over action classes is the output of an encode-process-decode graph network. No equation in the paper defines the predicted action in terms of the ground-truth label or fits a parameter to the test labels and then reports that fit as a prediction. The self-citations (Wächter and Asfour [9] for labeling granularity, Mandery et al. [10] for contact-based segmentation, Zöllner et al. [11] for bimanual programming by demonstration) are contextual and are not load-bearing for the claimed result. The adoption of graph networks is from the external Battaglia et al. reference, and the spatial-relation vocabulary is explicitly modified from external work, not smuggled in as an unverified self-citation. The only notable concern is that YOLO is trained on 5413 frames sampled from the recordings while the paper does not explicitly state that the sampling respects the leave-one-subject-out folds, which could affect whether the reported number represents a truly unseen subject. That is a data-hygiene and generalization concern, not a circularity: the F1 remains a measured quantity rather than a self-defined one. Therefore no specific circular step can be quoted, and the appropriate score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several empirical and structural assumptions: the sufficiency of the chosen spatial relations, the reliability of 3D bounding box estimation, the correctness of a single annotator's labels, and the reliability of the off-the-shelf detectors. The graph network hyperparameters and the YOLO detector are fitted components that are not independently justified. No new physical entities are introduced.

free parameters (5)

temporal concatenation window = 10 frames
The number of scene graphs concatenated for each prediction was chosen after 'evaluating multiple test series' (Section III-C). The central claim depends on this window.
MLP hidden size = 256 neurons
The multilayer perceptrons in the graph network blocks use 2 layers with 256 neurons, determined empirically (Section III-C).
number of processing steps = 10
The core of the graph network performs 10 processing steps, determined empirically (Section III-C).
Gaussian filter parameter = 3 sigma = 250 ms
The smoothing filter for 3D bounding boxes was parameterized with 3 sigma equal to 250 ms (Section III-B).
YOLO object detector = trained on 5413 frames from the same dataset
The object detector is trained on frames sampled from the same 540 recordings that are used for the action classification evaluation. This is a fitted component whose accuracy directly affects the scene graphs and the reported F1.

assumptions (4)

domain assumption The 15 spatial relations from Ziaeetabar et al. (2018), without exception conditions, are sufficient to discriminate the 14 action classes when combined with object classes.
The classifier only receives these relations and object classes as input. The paper's confusions (e.g., drink vs. hold, place vs. saw) suggest this assumption is partially violated.
domain assumption The largest depth cluster inside a 2D bounding box belongs to the detected object.
Stage 2 of the pipeline (Section III-B) uses this assumption to estimate 3D bounding boxes. The paper states this fails for thin objects like hammers and saws.
domain assumption A single annotator's per-hand action labels are correct and consistent.
Section III-A states that one annotator manually labeled the entire dataset. No inter-annotator agreement is reported, so label noise is an unquantified factor.
domain assumption YOLO and OpenPose detections from the robot's viewpoint are reliable enough to construct informative scene graphs.
The whole pipeline depends on these detectors. The paper acknowledges occasional misclassifications and noisy depth images (Section III-C).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Learning Object-Action Relations from Bimanual Human Demonstration Using Graph Networks." pith.science (2026). https://pith.science/paper/IWDSGRQO

@misc{pith2026190808391,
  author       = {Pith},
  title        = {Pith review of: Learning Object-Action Relations from Bimanual Human Demonstration Using Graph Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IWDSGRQO}},
  note         = {Machine review of arXiv:1908.08391}
}

read the original abstract

Recognizing human actions is a vital task for a humanoid robot, especially in domains like programming by demonstration. Previous approaches on action recognition primarily focused on the overall prevalent action being executed, but we argue that bimanual human motion cannot always be described sufficiently with a single action label. We present a system for frame-wise action classification and segmentation in bimanual human demonstrations. The system extracts symbolic spatial object relations from raw RGB-D video data captured from the robot's point of view in order to build graph-based scene representations. To learn object-action relations, a graph network classifier is trained using these representations together with ground truth action labels to predict the action executed by each hand. We evaluated the proposed classifier on a new RGB-D video dataset showing daily action sequences focusing on bimanual manipulation actions. It consists of 6 subjects performing 9 tasks with 10 repetitions each, which leads to 540 video recordings with 2 hours and 18 minutes total playtime and per-hand ground truth action labels for each frame. We show that the classifier is able to reliably identify (action classification macro F1-score of 0.86) the true executed action of each hand within its top 3 predictions on a frame-by-frame basis without prior temporal action segmentation.

Figures

Figures reproduced from arXiv: 1908.08391 by the authors.

**Figure 2.** Exemplary recordings from our proposed dataset. First row: Preparing breakfast cereals by cutting and pouring a banana into a bowl, followed by [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Schematic of the 3-stage processing pipeline. Input: An [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Architecture of our action classifier, an encode-process-decode graph network with 10 processing steps. The input is a scene graph [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Example of how a temporal concatenation of scene graphs (left) is [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: Qualitative evaluation by visualizing the top prediction of the classifier for the right hand ( [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 7.** Figure 7: Accumulative classification correctness over all folds depicted as normalized confusion matrix for the top prediction (left), and where a classification [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 29 canonical work pages

[1]

Robot Programming by Demonstration,

A. Billard, S. Calinon, R. Dillmann, and S. Schaal, “Robot Programming by Demonstration,” in Handbook of Robotics , Springer, 2008, pp. 1371– 1394

work page 2008
[2]

Anticipating Human Activities Using Object Affordances for Reactive Robotic Response,

H. S. Koppula and A. Saxena, “Anticipating Human Activities Using Object Affordances for Reactive Robotic Response,” Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , vol. 38, no. 1, pp. 14–29, 2016

work page 2016
[3]

Recognition and Prediction of Manipulation Actions Using Enriched Semantic Event Chains,

F. Ziaeetabar, T. Kulvicius, M. Tamosiunaite, and F. Wörgötter, “Recognition and Prediction of Manipulation Actions Using Enriched Semantic Event Chains,” Robotics and Autonomous Systems (RAS) , vol. 110, pp. 173–188, 2018

work page 2018
[4]

3D Convolutional Neural Networks for Human Action Recognition,

S. Ji, W. Xu, M. Yang, and K. Yu, “3D Convolutional Neural Networks for Human Action Recognition,” Transactions on Pattern Analysis and Machine Intelligence (TPAMI), vol. 35, no. 1, pp. 221–231, 2013

work page 2013
[5]

Visual Object-Action Recognition: Inferring Object Affordances from Human Demonstration,

H. Kjellström, J. Romero, and D. Kragi ´c, “Visual Object-Action Recognition: Inferring Object Affordances from Human Demonstration,” Computer Vision and Image Understanding , vol. 115, no. 1, pp. 81–90, 2011

work page 2011
[6]

Action MACH: A Spatio- Temporal Maximum Average Correlation Height Filter for Action Recognition,

M. D. Rodriguez, J. Ahmed, and M. Shah, “Action MACH: A Spatio- Temporal Maximum Average Correlation Height Filter for Action Recognition,” in Conference on Computer Vision and Pattern Recognition (CVPR), Anchorage, AK, USA: IEEE, 2008, pp. 1–8

work page 2008
[7]

Actions as Space-Time Shapes,

M. Blank, L. Gorelick, E. Shechtman, M. Irani, and R. Basri, “Actions as Space-Time Shapes,” in International Conference on Computer Vision (ICCV), vol. 2, Beijing, China: IEEE, 2005, pp. 1395–1402

work page 2005
[8]

Learning the Semantics of Object-Action Relations by Observation,

E. E. Aksoy, A. Abramov, J. Dörr, K. Ning, B. Dellen, and F. Wörgötter, “Learning the Semantics of Object-Action Relations by Observation,” International Journal of Robotics Research (IJRR) , vol. 30, no. 10, pp. 1229–1249, 2011

work page 2011

Show all 32 references

[9]

Hierarchical Segmentation of Manipulation Actions Based on Object Relations and Motion Characteristics,

M. Wächter and T. Asfour, “Hierarchical Segmentation of Manipulation Actions Based on Object Relations and Motion Characteristics,” in International Conference on Advanced Robotics (ICAR) , Istanbul, Turkey: IEEE, 2015, pp. 549–556

2015
[10]

Analyzing Whole- Body Pose Transitions in Multi-Contact Motions,

C. Mandery, J. Borràs, M. Jöchner, and T. Asfour, “Analyzing Whole- Body Pose Transitions in Multi-Contact Motions,” in International Conference on Humanoid Robots (Humanoids) , Seoul, South Korea: IEEE, 2015, pp. 1020–1027

2015
[11]

Programming by Demonstration: Dual-Arm Manipulation Tasks for Humanoid Robots,

R. Zöllner, T. Asfour, and R. Dillmann, “Programming by Demonstration: Dual-Arm Manipulation Tasks for Humanoid Robots,” in International Conference on Intelligent Robots and Systems (IROS) , vol. 1, Sendai, Japan: IEEE, 2004, pp. 479–484

2004
[12]

Relational Inductive Biases, Deep Learning, and Graph Networks,

P. W. Battaglia, J. B. Hamrick, V . Bapst, A. Sanchez-Gonzalez, V . Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, C. Gulcehre, F. Song, A. Ballard, J. Gilmer, G. Dahl, A. Vaswani, K. Allen, C. Nash, V . Langston, C. Dyer, N. Heess, D. Wierstra, P. ...

2018 arXiv
[13]

The Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities,

H. Kühne, A. Arslan, and T. Serre, “The Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities,” in Conference on Computer Vision and Pattern Recognition (CVPR) , Columbus, OH, USA: IEEE, 2014, pp. 780–787

2014
[14]

Scaling Egocentric Vision: The EPIC-KITCHENS Dataset,

D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray, “Scaling Egocentric Vision: The EPIC-KITCHENS Dataset,” in European Conference on Computer Vision (ECCV) , Munich, Germany: Springer, 2018, pp. 753–771

2018
[15]

Towards Automatic Learning of Procedures from Web Instructional Videos,

L. Zhou, C. Xu, and J. J. Corso, “Towards Automatic Learning of Procedures from Web Instructional Videos,” in AAAI CONFERENCE on Artiﬁcial Intelligence , New Orleans, LA, USA: AAAI Press, 2018, pp. 7590–7598

2018
[16]

Watch-n-Patch: Unsupervised Understanding of Actions and Relations,

C. Wu, J. Zhang, S. Savarese, and A. Saxena, “Watch-n-Patch: Unsupervised Understanding of Actions and Relations,” in Conference on Computer Vision and Pattern Recognition (CVPR) , Boston, MA, USA: IEEE, 2015, pp. 4362–4370

2015
[17]

Human Activity Detection from RGBD Images,

J. Sung, C. Ponce, B. Selman, and A. Saxena, “Human Activity Detection from RGBD Images,” in AAAI CONFERENCE on Artiﬁcial Intelligence Workshops, San Francisco, CA, USA: AAAI Press, 2011

2011
[18]

Mining Actionlet Ensemble for Action Recognition with Depth Cameras,

J. Wang, Z. Liu, Y . Wu, and J. Yuan, “Mining Actionlet Ensemble for Action Recognition with Depth Cameras,” in Conference on Computer Vision and Pattern Recognition (CVPR) , Providence, RI, USA: IEEE, 2012, pp. 1290–1297

2012
[19]

View Invariant Human Action Recognition Using Histograms of 3D Joints,

L. Xia, C. -C. Chen, and J. K. Aggarwal, “View Invariant Human Action Recognition Using Histograms of 3D Joints,” in Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , Providence, RI, USA: IEEE, 2012, pp. 20–27

2012
[20]

Learning Human Activities and Object Affordances from RGB-D Videos,

H. S. Koppula, R. Gupta, and A. Saxena, “Learning Human Activities and Object Affordances from RGB-D Videos,” International Journal of Robotics Research (IJRR) , vol. 32, no. 8, pp. 951–970, 2013

2013
[21]

Model-Free Incremental Learning of the Semantics of Manipulation Actions,

E. E. Aksoy, M. Tamosiunaite, and F. Wörgötter, “Model-Free Incremental Learning of the Semantics of Manipulation Actions,” Robotics and Autonomous Systems (RAS) , vol. 71, pp. 118–133, 2015

2015
[22]

The TUM Kitchen Data Set of Everyday Manipulation Activities for Motion Tracking and Action Recognition,

M. Tenorth, J. Bandouch, and M. Beetz, “The TUM Kitchen Data Set of Everyday Manipulation Activities for Motion Tracking and Action Recognition,” in International Conference on Computer Vision (ICCV) Workshops, Kyoto, Japan: IEEE, 2009, pp. 1089–1096

2009
[23]

A Survey on Vision-Based Human Action Recognition,

R. Poppe, “A Survey on Vision-Based Human Action Recognition,” Image and Vision Computing , vol. 28, no. 6, pp. 976–990, 2010

2010
[24]

A Survey of Vision-Based Methods for Action Representation, Segmentation and Recognition,

D. Weinland, R. Ronfard, and E. Boyer, “A Survey of Vision-Based Methods for Action Representation, Segmentation and Recognition,” Computer Vision and Image Understanding , vol. 115, no. 2, pp. 224–241, 2011

2011
[25]

A Survey of Video Datasets for Human Action and Activity Recognition,

J. M. Chaquet, E. J. Carmona, and A. Fernández-Caballero, “A Survey of Video Datasets for Human Action and Activity Recognition,” Computer Vision and Image Understanding , vol. 117, no. 6, pp. 633–659, 2013

2013
[26]

RGB-D-Based Action Recognition Datasets: A Survey,

J. Zhang, W. Li, P. O. Ogunbona, P. Wang, and C. Tang, “RGB-D-Based Action Recognition Datasets: A Survey,” Pattern Recognition, vol. 60, pp. 86–105, 2016

2016
[27]

Going Deeper into Action Recognition: A Survey,

S. Herath, M. Harandi, and F. Porikli, “Going Deeper into Action Recognition: A Survey,” Image and Vision Computing , vol. 60, pp. 4–21, 2017

2017
[28]

The Robot Software Framework ArmarX,

N. Vahrenkamp, M. Wächter, M. Kröhnert, K. Welke, and T. Asfour, “The Robot Software Framework ArmarX,” It – Information Technology , vol. 57, no. 2, pp. 99–111, 2015

2015
[29]

YOLOv3: An Incremental Improvement,

J. Redmon and A. Farhadi, “YOLOv3: An Incremental Improvement,”
[30]

OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Afﬁnity Fields,

Z. Cao, G. Hidalgo, T. Simon, S. -E. Wei, and Y . Sheikh, “OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Afﬁnity Fields,”
[32]

arXiv: 1812.08008 [cs]

arXiv
[2018]

arXiv: 1804.02767 [cs]

arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Robot Programming by Demonstration,

A. Billard, S. Calinon, R. Dillmann, and S. Schaal, “Robot Programming by Demonstration,” in Handbook of Robotics , Springer, 2008, pp. 1371– 1394

work page 2008

[2] [2]

Anticipating Human Activities Using Object Affordances for Reactive Robotic Response,

H. S. Koppula and A. Saxena, “Anticipating Human Activities Using Object Affordances for Reactive Robotic Response,” Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , vol. 38, no. 1, pp. 14–29, 2016

work page 2016

[3] [3]

Recognition and Prediction of Manipulation Actions Using Enriched Semantic Event Chains,

F. Ziaeetabar, T. Kulvicius, M. Tamosiunaite, and F. Wörgötter, “Recognition and Prediction of Manipulation Actions Using Enriched Semantic Event Chains,” Robotics and Autonomous Systems (RAS) , vol. 110, pp. 173–188, 2018

work page 2018

[4] [4]

3D Convolutional Neural Networks for Human Action Recognition,

S. Ji, W. Xu, M. Yang, and K. Yu, “3D Convolutional Neural Networks for Human Action Recognition,” Transactions on Pattern Analysis and Machine Intelligence (TPAMI), vol. 35, no. 1, pp. 221–231, 2013

work page 2013

[5] [5]

Visual Object-Action Recognition: Inferring Object Affordances from Human Demonstration,

H. Kjellström, J. Romero, and D. Kragi ´c, “Visual Object-Action Recognition: Inferring Object Affordances from Human Demonstration,” Computer Vision and Image Understanding , vol. 115, no. 1, pp. 81–90, 2011

work page 2011

[6] [6]

Action MACH: A Spatio- Temporal Maximum Average Correlation Height Filter for Action Recognition,

M. D. Rodriguez, J. Ahmed, and M. Shah, “Action MACH: A Spatio- Temporal Maximum Average Correlation Height Filter for Action Recognition,” in Conference on Computer Vision and Pattern Recognition (CVPR), Anchorage, AK, USA: IEEE, 2008, pp. 1–8

work page 2008

[7] [7]

Actions as Space-Time Shapes,

M. Blank, L. Gorelick, E. Shechtman, M. Irani, and R. Basri, “Actions as Space-Time Shapes,” in International Conference on Computer Vision (ICCV), vol. 2, Beijing, China: IEEE, 2005, pp. 1395–1402

work page 2005

[8] [8]

Learning the Semantics of Object-Action Relations by Observation,

E. E. Aksoy, A. Abramov, J. Dörr, K. Ning, B. Dellen, and F. Wörgötter, “Learning the Semantics of Object-Action Relations by Observation,” International Journal of Robotics Research (IJRR) , vol. 30, no. 10, pp. 1229–1249, 2011

work page 2011