Pith. sign in

REVIEW 4 major objections 4 minor 134 references

InterAct claims to be the largest 3D human-object interaction benchmark to date, and shows a unified model reaching state-of-the-art on six generation tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 18:51 UTC pith:NQWJHMY3

load-bearing objection The consolidated benchmark is a real contribution, but the correction-quality gains are partly measured by its own objectives and the hand-contact indicator inherits the inaccuracies it is meant to fix. the 4 major comments →

arxiv 2509.09555 v1 pith:NQWJHMY3 submitted 2025-09-11 cs.CV

InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation

Sirui Xu , Dongting Li , Yucheng Zhang , Xiyan Xu , Qi Long , Ziyin Wang , Yunzhi Lu , Shuchang Dong
show 4 more authors
Hezi Jiang Akshat Gupta Yu-Xiong Wang Liang-Yan Gui
This is my paper
classification cs.CV
keywords 3D human-object interactionHOI benchmarkmotion generationtext-to-interactioncontact correctiondata augmentationmulti-task diffusionmarker-based representation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the main bottleneck in 3D human-object interaction generation is data, not models, and presents InterAct as the largest such benchmark: 21.81 hours of standardized, text-annotated motion from seven existing datasets, expanded to 30.70 hours through a contact-preserving augmentation. It claims that a unified optimization pipeline removes common MoCap artifacts such as penetration, floating, and faulty hand poses, and that a single multi-task diffusion model trained on this data achieves state-of-the-art results across six tasks, including text-to-interaction and physics-based imitation. If true, the work gives the field a common data resource, a correction recipe, and a baseline, making interaction generation research more reproducible and more capable of handling new objects.

Core claim

On its own terms, the paper's central claim is that heterogeneous, artifact-laden human-object interaction recordings can be turned into a large, clean, richly annotated resource by (1) unifying human representation with surface markers, (2) correcting contacts and hand poses through gradient-based optimization, and (3) applying the principle of contact invariance to synthesize new training sequences at no extra MoCap cost. The paper then claims that a multi-task transformer-diffusion model, given this resource, outperforms existing approaches on all six benchmark tasks, and that corrected data improve downstream imitation success from 84.4% to 90.7%.

What carries the argument

The method rests on four pieces: a marker-based human representation chosen because contact happens on the body surface, not at joints; a contact indicator ci that gates where the optimizer enforces hand-object contact; contact-invariance augmentation, which displaces the object and re-optimizes the body so that close vertex pairs stay close (the Ealign distance-matrix loss); and a multi-task output eta giving each marker's vector to the object surface, which lets one diffusion model handle text-, action- and object-conditioned generation jointly.

Load-bearing premise

The correction stage decides where to enforce contact using a contact indicator computed from the hand-object distances in the very data it is correcting; the paper itself notes (Limitations, Sec. E) that large distances may be classified as no-contact and left uncorrected, so the pipeline's success depends on the indicator labeling contact correctly despite the source data being faulty.

What would settle it

Re-capture a subset of corrected sequences with a high-fidelity hand tracking system (e.g., magnetic or glove-based) and measure whether the corrected hand poses are actually closer to this ground truth than the raw poses; if not, or if regions the indicator labeled non-contact are revealed to be in genuine contact, the correction claim is falsified. A simpler check: train the same generation model on raw versus corrected data; if raw-trained FID or contact metrics match or beat corrected-trained ones, the correction step is not delivering its claimed benefit.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A single transformer-diffusion model, trained once on InterAct-X, covers five kinematic generation tasks plus imitation, replacing task-specific pipelines.
  • Physics-based imitation policies trained on corrected data succeed 90.7% of the time versus 84.4% on raw data, so higher-quality inputs translate into better control.
  • Training the interaction prediction model on three copies of the dataset lowers global per-marker error from 0.106 m to 0.091 m, suggesting the benchmark supports scaling-law-style improvements.
  • Marker representations reduce penetration artifacts compared with joint- or SMPL-based representations, giving other HOI methods a simple drop-in representation.
  • The benchmark's 217 objects and six tasks establish a common evaluation ground for future HOI generation work.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The contact-invariance augmentation principle is stated for single human-object pairs, but the same distance-matrix alignment could be applied to multi-object scenes or human-human-object interactions, as long as the contact graph is preserved.
  • The geometric contact indicator could be replaced by a learned contact prior trained on a small, high-accuracy hand tracking dataset; if that prior corrected mislabeled contacts, the pipeline might fix the very failures it currently leaves untouched.
  • Because the interaction-aware text encoder is trained with contrastive learning on 30 hours of paired text-motion data, it may transfer to video-based HOI retrieval or captioning tasks that currently lack paired 3D data.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces InterAct, a large-scale 3D human-object interaction (HOI) benchmark built by consolidating seven existing datasets (21.81 h of base data, expanded to 30.70 h via optimization-based augmentation), with unified marker-based human representations and detailed text/action annotations. It also proposes a unified multi-task transformer/diffusion framework covering six HOI generation tasks (text-to-interaction, action-to-interaction, object-to-human, human-to-object, interaction prediction, and interaction imitation). A three-stage optimization pipeline is used for full-body correction, hand correction, and interaction augmentation. Experiments report improved contact/penetration statistics, state-of-the-art results on language- and action-conditioned tasks, scaling benefits for interaction prediction, and an improved physics-based imitation success rate.

Significance. If the data-quality and augmentation claims hold, InterAct would be a substantial community resource: it standardizes heterogeneous HOI data, supplies text annotations at scale, defines six benchmarking tasks, and releases data and models. The marker-based representation and the multi-task formulation with contact prediction are useful methodological contributions. The strongest independent evidence is the physics-based imitation success gain (90.7% vs. 84.4% on corrected vs. raw data), which suggests the correction helps global control. However, the central data-quality claim rests partly on metrics that coincide with the optimization objectives, so the benchmark's foundational value needs additional independent validation.

major comments (4)
  1. [Sec. 5.1, Table 2] The two quantitative quality metrics reported in Table 2—penetration depth and contact ratio—are exactly the quantities minimized and maximized by the optimization losses E_pene and E_cont (Sec. 3.2, Supplementary Eqs. 1 and 4). Reporting improvements on these metrics after optimization is partly a check that the optimizer did what it was asked, not independent evidence of physical correctness. The user study uses only 30 tuples and only the BEHAVE and OMOMO subsets, so it is too small to establish per-contact localization across the seven datasets. I recommend independent validation: manual contact labels on a held-out subset, comparison against a separate capture modality, or a downstream metric not used in the correction objective.
  2. [Sec. 3.2, Supplementary Eq. (4)] The contact indicator c_i is a piecewise function of min_j d_j[i], the hand-object distance computed from the very poses being corrected. If the initial reconstruction places a truly touching hand more than epsilon_2 = 0.10 m away, c_i = 0 and no contact-promotion gradient is applied; conversely, a noisy near-but-not-touching hand can be pushed into the object. The paper's own limitation section (Sec. E) concedes that large distances may be identified as no contact and remain uncorrected. Since the motivation for correction is precisely that source hand poses are inaccurate (BEHAVE, OMOMO), this circularity weakens the central data-quality claim. Please validate the indicator against an independent source, e.g., manual contact annotation or a separate hand tracker.
  3. [Sec. 5.2, Tables 3 and 4] The FID and MM-Dist feature encoder used for evaluation is trained on InterAct corrected data and then used to score models trained on the same corrected data. This is acceptable for ranking design variants within the paper, but it cannot fully support the absolute 'state-of-the-art' claim or the data-quality claim, because the encoder may have learned the specific correction statistics and artifacts of InterAct. Please report results with a fixed feature extractor not trained on the corrected data, or at least demonstrate that the relative FID ranking is stable across different feature extractors.
  4. [Sec. 3.2, Interaction Augmentation] The 'contact invariance' principle is an assumed axiom: displaced object trajectories with re-optimized human motion are asserted to preserve interaction semantics. The filtering step removes some low-quality augmentations, but no metric directly demonstrates semantic equivalence for the retained augmented data. The physical imitation success gain is useful but is reported on only four sequences and does not isolate augmentation quality. Please provide an ablation that trains on corrected data with and without augmentation, or a per-sample semantic-preservation evaluation (automatic or human).
minor comments (4)
  1. [Table 2] The table formatting is very hard to read: the check marks and numeric columns are run together (e.g., '× ×0.017 0.048 22.3'). Please reformat with clear column separators so each row unambiguously shows the configuration and corresponding metric values.
  2. [Supplementary Sec. B.2] The sentence 'We only select sequences that contain rigid objects' appears twice verbatim in the Chairs processing paragraph. Please remove the duplicate.
  3. [Supplementary Sec. C.6] Typo: 'Addtional Implementation Details' should be 'Additional Implementation Details'.
  4. [Abstract and Sec. 1] The claim of being 'the most extensive 3D HOI benchmark' is based on 21.81 h of base data, with 30.70 h including synthetic augmentation. Please state the base versus augmented split clearly in the abstract or contribution list to avoid overstating the captured data volume.

Circularity Check

2 steps flagged

The benchmark's central data-quality claim is partly self-referential: the correction pipeline optimizes the same penetration and contact quantities used as quality metrics, and its contact indicator is computed from the very hand-object distances the correction is meant to fix.

specific steps
  1. self definitional [Sec. 3.2 (Hand Correction) and Supplementary Eq. 4 (B.3); Sec. 5.1 metrics]
    "Econt = Σ_i c_i Σ_j d_j[i] ... The contact indicator ci, inferred from ground truth data, is a function based on hand-object distance minj dj[i], which we provide details in supplementary. ... ci = 1 if minj dj[i] ≤ ε; 0 if minj dj[i] > ε2 ... where minj dj[i] refers to hand-object chamfer distance"

    The contact indicator that decides where contact is promoted is computed from the same hand-object chamfer distances that the correction stage is supposed to repair. If a truly contacting hand is reconstructed more than ε2=10cm away, c_i=0 and the loss never pulls it into contact; the paper's own Sec. E concedes this. The Contact Ratio metric used in Table 2 counts vertices whose distance to the object is under a threshold, so minimizing Econt for the c_i>0 regions directly increases the reported metric. The claimed correction gain is therefore partly the optimization objective itself, not an independent measurement of physical fidelity.

  2. self definitional [Sec. 5.1 (metrics) vs Supplementary Eq. 1 and Eq. 5 (Full-Body Correction)]
    "Penetration refers to the intersection depth – maximum of negative sign distances from human vertices to the object's surface – average across the sequence. ... Epene = −Σ_i Σ_do min(sdf_i(v_o^i[k]),0). ... E = λpeneEpene + λsmoothEsmooth + λrecErec"

    The primary quality metric used to demonstrate correction success is the same function that the optimizer minimizes. Full-body correction takes E with a penetration term Epene, and Table 2 then reports 'Pene (m)' as evidence that the correction improves quality. Reducing the optimization objective reduces the reported penetration metric by construction (modulo reconstruction and smoothness tradeoffs). Thus the quantitative comparison does not independently validate the physical accuracy of the corrected data; the evaluation criterion is built into the correction objective.

full rationale

InterAct's scale claim and the multi-task generation results are not circular: they are measured against external baselines (OMOMO, HOI-Diff, InterDiff, PhysHOI) with standard metrics, and the physics-based imitation success (90.7% vs 84.4%) plus the user study provide partial independent support. However, the paper's core second contribution—that the optimization framework enhances data quality—rests on Table 2, whose two quantitative metrics (penetration and contact ratio) are, by construction, the same quantities minimized and maximized by the correction losses. The contact indicator in Eq. 4 also uses the erroneous hand-object distances as its own ground truth, and Sec. E admits that large-distance artifacts are therefore uncorrected. This is not a full 8-10 circularity because there are non-self-referential signals (imitation, user preference), but the foundational data-quality evidence is partly self-validating, so a score of 6 is appropriate.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The central claims rest on hand-chosen thresholds and weights, on the assumption that marker correspondences transfer across body models within 1 cm, on contact invariance as a validity principle for augmentation, and on the reliability of the contact indicator computed from the very data being corrected. No new physical entities are introduced.

free parameters (5)
  • contact distance threshold epsilon = 0.02 m
    Used to define contact indicator c_i in hand correction; chosen following [4], but central to which contact is enforced.
  • non-contact threshold epsilon_2 = 0.10 m
    Used in contact indicator linear interpolation; hand-chosen.
  • beta (regularization weight in augmentation) = 5
    Weights the non-interaction mask in augmentation regularization.
  • guidance weights tau1, tau2 = 0.1, 0.2
    Control gradient update magnitude in classifier guidance; chosen by hand.
  • object motion loss weight = 0.9 relative to human loss
    Chosen for text-to-interaction training; any value changes balance between human and object.
axioms (4)
  • domain assumption Marker correspondences between SMPL-H and SMPL-X are consistent within 1 cm across poses.
    The unified marker representation (Sec 3.1) depends on this; the paper states it was 'extensively evaluated' but provides no data.
  • ad hoc to paper Contact invariance: displaced object trajectories with re-optimized human motion preserve interaction semantics.
    Introduced in Sec 3.2 to justify augmentation; no independent evidence that semantics are preserved.
  • domain assumption The optimization framework can recover natural hand poses even when the source hand poses are inaccurate.
    Hand correction (Sec 3.2) relies on the contact indicator computed from the same inaccurate hand poses.
  • domain assumption GPT-4 generated text and action labels are correct and aligned with the motion.
    The annotation pipeline (Sec 3.1) uses GPT-4 for rewriting and labeling; no quantitative evaluation of label accuracy.

pith-pipeline@v1.3.0-alltime-deepseek · 22597 in / 10821 out tokens · 100354 ms · 2026-08-04T18:51:58.744557+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation." pith.science (2026). https://pith.science/paper/NQWJHMY3

@misc{pith2026250909555,
  author       = {Pith},
  title        = {Pith review of: InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NQWJHMY3}},
  note         = {Machine review of arXiv:2509.09555}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

While large-scale human motion capture datasets have advanced human motion generation, modeling and generating dynamic 3D human-object interactions (HOIs) remain challenging due to dataset limitations. Existing datasets often lack extensive, high-quality motion and annotation and exhibit artifacts such as contact penetration, floating, and incorrect hand motions. To address these issues, we introduce InterAct, a large-scale 3D HOI benchmark featuring dataset and methodological advancements. First, we consolidate and standardize 21.81 hours of HOI data from diverse sources, enriching it with detailed textual annotations. Second, we propose a unified optimization framework to enhance data quality by reducing artifacts and correcting hand motions. Leveraging the principle of contact invariance, we maintain human-object relationships while introducing motion variations, expanding the dataset to 30.70 hours. Third, we define six benchmarking tasks and develop a unified HOI generative modeling perspective, achieving state-of-the-art performance. Extensive experiments validate the utility of our dataset as a foundational resource for advancing 3D human-object interaction generation. To support continued research in this area, the dataset is publicly available at https://github.com/wzyabcas/InterAct, and will be actively maintained.

Figures

Figures reproduced from arXiv: 2509.09555 by Akshat Gupta, Dongting Li, Hezi Jiang, Liang-Yan Gui, Qi Long, Shuchang Dong, Sirui Xu, Xiyan Xu, Yucheng Zhang, Yunzhi Lu, Yu-Xiong Wang, Ziyin Wang.

Figure 1
Figure 1. Figure 1: An overview of InterAct, our large-scale 3D human-object interaction (HOI) benchmark, covering six HOI generation tasks. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: (a) Our data processing pipeline consolidating data, [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Marker-based representation for human. icantly surpasses existing datasets, as summarized in [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative evaluation of interaction correction (bottom) on the OMOMO [38] dataset shows hand recovery compared to the ground truth interaction (top). Zoom in to see details of the hand recovery. Dataset Correction Augmentation Pene (m)↓ Cont Ratio User Study (%) BEHAVE [4] × × 0.017 0.048 22.3 ✓ × 0.016 0.071 39.7 ✓ ✓ 0.016 0.069 38.0 OMOMO [38] × × 0.009 0.071 23.9 ✓ × 0.007 0.131 39.4 ✓ ✓ 0.011 0.137 3… view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative evaluation of interaction augmentation (bottom) shows high-quality synthetic data varied from original (top). HOI-Aware Object Enc. HOI-Aware Text Enc. Contact Generation Contact Guidance R-Precision↑ FID↓ MM Dist↓ Multimodality↑ Diversity→ Top 1 Top 2 Top 3 Ground Truth 0.852±0.000 0.966±0.001 0.989±0.001 0.000±0.000 2.810±0.002 - 11.489±0.011 ✗ ✗ ✗ ✗ 0.733±0.007 0.909±0.002 0.957±0.002 3.192±… view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative results demonstrate the successful imitation [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

134 extracted references · 35 linked inside Pith

  1. [1]

    http:// mocap.cs.cmu.edu/

    CMU graphics lab motion capture database. http:// mocap.cs.cmu.edu/. 2

  2. [2]

    Easymocap - make human motion capture easier. Github,

  3. [3]

    Pmp: Learning to physically interact with environments using part-wise motion priors

    Jinseok Bae, Jungdam Won, Donggeun Lim, Cheol-Hui Min, and Young Min Kim. Pmp: Learning to physically interact with environments using part-wise motion priors. In SIGGRAPH, 2023. 3, 5

    2023

  4. [4]

    BEHA VE: Dataset and method for tracking human object interactions

    Bharat Lal Bhatnagar, Xianghui Xie, Ilya Petrov, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. BEHA VE: Dataset and method for tracking human object interactions. InCVPR, 2022. 2, 3, 4, 6, 7, 1

    2022

  5. [5]

    Bedlam: A synthetic dataset of bodies exhibiting detailed lifelike animated motion

    Michael J Black, Priyanka Patel, Joachim Tesch, and Jinlong Yang. Bedlam: A synthetic dataset of bodies exhibiting detailed lifelike animated motion. InCVPR, 2023. 5

    2023

  6. [6]

    Physically plausible full- body hand-object interaction synthesis.arXiv preprint arXiv:2309.07907, 2023

    Jona Braun, Sammy Christen, Muhammed Kocabas, Emre Aksan, and Otmar Hilliges. Physically plausible full- body hand-object interaction synthesis.arXiv preprint arXiv:2309.07907, 2023. 3

    Pith/arXiv arXiv 2023

  7. [7]

    Lan- guage models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. InNeurIPS, 2020. 3

    2020

  8. [8]

    Long-term human motion prediction with scene context

    Zhe Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh V o, and Jitendra Malik. Long-term human motion prediction with scene context. InECCV, 2020. 2

    2020

  9. [9]

    Text2hoi: Text-guided 3d motion generation for hand- object interaction

    Junuk Cha, Jihyeon Kim, Jae Shin Yoon, and Seungryul Baek. Text2hoi: Text-guided 3d motion generation for hand- object interaction. InCVPR, 2024. 3

    2024

  10. [10]

    Learning to sit: Synthesizing human-chair interactions via hierarchical control

    Yu-Wei Chao, Jimei Yang, Weifeng Chen, and Jia Deng. Learning to sit: Synthesizing human-chair interactions via hierarchical control. InAAAI, 2021. 3

    2021

  11. [11]

    Dexycb: A benchmark for capturing hand grasping of objects

    Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al. Dexycb: A benchmark for capturing hand grasping of objects. InCVPR,

  12. [12]

    Allava: Harnessing gpt4v-synthesized data for a lite vision-language model

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.11684, 2024. 5

    Pith/arXiv arXiv 2024

  13. [13]

    Diffh2o: Diffusion-based synthesis of hand- object interactions from textual descriptions.arXiv preprint arXiv:2403.17827, 2024

    Sammy Christen, Shreyas Hampali, Fadime Sener, Edoardo Remelli, Tomas Hodan, Eric Sauser, Shugao Ma, and Bu- gra Tekin. Diffh2o: Diffusion-based synthesis of hand- object interactions from textual descriptions.arXiv preprint arXiv:2403.17827, 2024. 3

    Pith/arXiv arXiv 2024

  14. [14]

    Context-aware human motion prediction

    Enric Corona, Albert Pumarola, Guillem Alenya, and Francesc Moreno-Noguer. Context-aware human motion prediction. InCVPR, 2020. 3

    2020

  15. [15]

    AnySkill: Learning open- vocabulary physical skill for interactive agents

    Jieming Cui, Tengyu Liu, Nian Liu, Yaodong Yang, Yixin Zhu, and Siyuan Huang. AnySkill: Learning open- vocabulary physical skill for interactive agents. InCVPR,

  16. [16]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. InNeurIPS, 2021. 6, 7

    2021

  17. [17]

    Exploiting cyclic symmetry in convolutional neural net- works

    Sander Dieleman, Jeffrey De Fauw, and Koray Kavukcuoglu. Exploiting cyclic symmetry in convolutional neural net- works. InICML, 2016. 5

    2016

  18. [18]

    CG-HOI: Contact-guided 3d human-object interaction generation

    Christian Diller and Angela Dai. CG-HOI: Contact-guided 3d human-object interaction generation. InCVPR, 2024. 3, 5, 6

    2024

  19. [19]

    Scaling laws of synthetic images for model training

    Lijie Fan, Kaifeng Chen, Dilip Krishnan, Dina Katabi, Phillip Isola, and Yonglong Tian. Scaling laws of synthetic images for model training... for now. InCVPR, 2024. 5

    2024

  20. [20]

    Black, and Otmar Hilliges

    Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J. Black, and Otmar Hilliges. ARCTIC: A dataset for dexterous bimanual hand- object manipulation. InCVPR, 2023. 3

    2023

  21. [21]

    IMoS: Intent-driven full-body motion synthesis for human-object interactions

    Anindita Ghosh, Rishabh Dabral, Vladislav Golyanik, Chris- tian Theobalt, and Philipp Slusallek. IMoS: Intent-driven full-body motion synthesis for human-object interactions. arXiv preprint arXiv:2212.07555, 2022. 3

    Pith/arXiv arXiv 2022

  22. [22]

    Generating diverse and natural 3d human motions from text

    Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. InCVPR, 2022. 1, 4, 6

    2022

  23. [23]

    Honnotate: A method for 3d annotation of hand and object poses

    Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vin- cent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. InCVPR, 2020. 2

    2020

  24. [24]

    Resolving 3d human pose ambiguities with 3d scene constraints

    Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael J Black. Resolving 3d human pose ambiguities with 3d scene constraints. InICCV, 2019. 2

    2019

  25. [25]

    Stochastic scene- aware motion prediction

    Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael Black. Stochastic scene- aware motion prediction. InICCV, 2021. 2

    2021

  26. [26]

    Synthesizing physi- cal character-scene interactions

    Mohamed Hassan, Yunrong Guo, Tingwu Wang, Michael Black, Sanja Fidler, and Xue Bin Peng. Synthesizing physi- cal character-scene interactions. InSIGGRAPH, 2023. 3

    2023

  27. [27]

    Composi- tional 3d human-object neural animation.arXiv preprint arXiv:2304.14070, 2023

    Zhi Hou, Baosheng Yu, and Dacheng Tao. Composi- tional 3d human-object neural animation.arXiv preprint arXiv:2304.14070, 2023. 3

    Pith/arXiv arXiv 2023

  28. [28]

    Black, and Dim- itrios Tzionas

    Yinghao Huang, Omid Taheri, Michael J. Black, and Dim- itrios Tzionas. InterCap: Joint markerless 3D tracking of humans and objects in interaction. InGCPR, 2022. 2, 3, 1, 4

    2022

  29. [29]

    CHAIRS: Towards full-body articulated human-object interaction

    Nan Jiang, Tengyu Liu, Zhexuan Cao, Jieming Cui, Yixin Chen, He Wang, Yixin Zhu, and Siyuan Huang. CHAIRS: Towards full-body articulated human-object interaction. In ICCV, 2023. 2, 3, 1, 4

    2023

  30. [30]

    Scaling up dynamic human-scene interaction mod- eling

    Nan Jiang, Zhiyuan Zhang, Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, and Siyuan Huang. Scaling up dynamic human-scene interaction mod- eling. InCVPR, 2024. 3

    2024

  31. [31]

    From skin to skeleton: Towards biomechanically accurate 3d digital humans.ACM Transactions on Graphics (TOG), 42(6):1–12,

    Marilyn Keller, Keenon Werling, Soyong Shin, Scott Delp, Sergi Pujades, C Karen Liu, and Michael J Black. From skin to skeleton: Towards biomechanically accurate 3d digital humans.ACM Transactions on Graphics (TOG), 42(6):1–12,

  32. [32]

    ParaHome: Parameterizing everyday home activities to- wards 3d generative modeling of human-object interactions

    Jeonghwan Kim, Jisoo Kim, Jeonghyeon Na, and Hanbyul Joo. ParaHome: Parameterizing everyday home activities to- wards 3d generative modeling of human-object interactions. arXiv preprint arXiv:2401.10232, 2024. 3

    Pith/arXiv arXiv 2024

  33. [33]

    NCHO: Unsupervised learning for neural 3d composition of humans and objects

    Taeksoo Kim, Shunsuke Saito, and Hanbyul Joo. NCHO: Unsupervised learning for neural 3d composition of humans and objects. InICCV, 2023. 3

    2023

  34. [34]

    The kit bimanual manipulation dataset

    Franziska Krebs, Andre Meixner, Isabel Patzer, and Tamim Asfour. The kit bimanual manipulation dataset. InHu- manoids, 2021. 3

    2021

  35. [35]

    NIFTY: Neural object interaction fields for guided human motion synthesis.arXiv preprint arXiv:2307.07511, 2023

    Nilesh Kulkarni, Davis Rempe, Kyle Genova, Abhijit Kundu, Justin Johnson, David Fouhey, and Leonidas Guibas. NIFTY: Neural object interaction fields for guided human motion synthesis.arXiv preprint arXiv:2307.07511, 2023

    Pith/arXiv arXiv 2023

  36. [36]

    Locomotion-Action- Manipulation: Synthesizing human-scene interactions in complex 3d environments

    Jiye Lee and Hanbyul Joo. Locomotion-Action- Manipulation: Synthesizing human-scene interactions in complex 3d environments. InICCV, 2023. 3

    2023

  37. [37]

    Controllable human-object in- teraction synthesis.arXiv preprint arXiv:2312.03913, 2023

    Jiaman Li, Alexander Clegg, Roozbeh Mottaghi, Jiajun Wu, Xavier Puig, and C Karen Liu. Controllable human-object in- teraction synthesis.arXiv preprint arXiv:2312.03913, 2023. 3, 5, 6

    Pith/arXiv arXiv 2023

  38. [38]

    Object motion guided human motion synthesis.ACM Transactions on Graphics (TOG), 42(6):1–11, 2023

    Jiaman Li, Jiajun Wu, and C Karen Liu. Object motion guided human motion synthesis.ACM Transactions on Graphics (TOG), 42(6):1–11, 2023. 2, 3, 4, 5, 6, 7, 8

    2023

  39. [39]

    Task-oriented human-object interactions genera- tion with implicit neural representations.arXiv preprint arXiv:2303.13129, 2023

    Quanzhou Li, Jingbo Wang, Chen Change Loy, and Bo Dai. Task-oriented human-object interactions genera- tion with implicit neural representations.arXiv preprint arXiv:2303.13129, 2023. 3

    Pith/arXiv arXiv 2023

  40. [40]

    InterGen: Diffusion-based multi-human motion generation under complex interactions.arXiv preprint arXiv:2304.05684, 2023

    Han Liang, Wenqian Zhang, Wenxuan Li, Jingyi Yu, and Lan Xu. InterGen: Diffusion-based multi-human motion generation under complex interactions.arXiv preprint arXiv:2304.05684, 2023. 2

    Pith/arXiv arXiv 2023

  41. [41]

    Motion-X: A large- scale 3d expressive whole-body human motion dataset

    Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. Motion-X: A large- scale 3d expressive whole-body human motion dataset. In NeurIPS, 2023. 1

    2023

  42. [42]

    Learning basketball drib- bling skills using trajectory optimization and deep reinforce- ment learning.ACM Transactions on Graphics (TOG), 37 (4):1–14, 2018

    Libin Liu and Jessica Hodgins. Learning basketball drib- bling skills using trajectory optimization and deep reinforce- ment learning.ACM Transactions on Graphics (TOG), 37 (4):1–14, 2018. 3

    2018

  43. [43]

    Contactgen: Generative contact modeling for grasp generation

    Shaowei Liu, Yang Zhou, Jimei Yang, Saurabh Gupta, and Shenlong Wang. Contactgen: Generative contact modeling for grasp generation. InICCV, 2023. 3

    2023

  44. [44]

    Hoi4d: A 4d egocentric dataset for category-level human- object interaction

    Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category-level human- object interaction. InCVPR, 2022. 2

    2022

  45. [45]

    SMPL: A skinned multi- person linear model.ACM transactions on graphics, 2015

    Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. SMPL: A skinned multi- person linear model.ACM transactions on graphics, 2015. 3, 5

    2015

  46. [46]

    Hu- mantomato: Text-aligned whole-body motion generation

    Shunlin Lu, Ling-Hao Chen, Ailing Zeng, Jing Lin, Ruimao Zhang, Lei Zhang, and Heung-Yeung Shum. Hu- mantomato: Text-aligned whole-body motion generation. arxiv:2310.12978, 2023. 6

    Pith/arXiv arXiv 2023

  47. [47]

    Smplolympics: Sports environ- ments for physically simulated humanoids.arXiv preprint arXiv:2407.00187, 2024

    Zhengyi Luo, Jiashun Wang, Kangni Liu, Haotian Zhang, Chen Tessler, Jingbo Wang, Ye Yuan, Jinkun Cao, Zihui Lin, Fengyi Wang, et al. Smplolympics: Sports environ- ments for physically simulated humanoids.arXiv preprint arXiv:2407.00187, 2024. 3

    Pith/arXiv arXiv 2024

  48. [48]

    Himo: A new benchmark for full-body human interacting with multiple objects

    Xintao Lv, Liang Xu, Yichao Yan, Xin Jin, Congsheng Xu, Shuwen Wu, Yifan Liu, Lincheng Li, Mengxiao Bi, Wenjun Zeng, et al. Himo: A new benchmark for full-body human interacting with multiple objects. InECCV, 2025. 2

    2025

  49. [49]

    Diff-ip2d: Diffusion-based hand-object interaction predic- tion on egocentric videos.arXiv preprint arXiv:2405.04370,

    Junyi Ma, Jingyi Xu, Xieyuanli Chen, and Hesheng Wang. Diff-ip2d: Diffusion-based hand-object interaction predic- tion on egocentric videos.arXiv preprint arXiv:2405.04370,

    arXiv

  50. [50]

    AMASS: Archive of motion capture as surface shapes

    Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Ger- ard Pons-Moll, and Michael J Black. AMASS: Archive of motion capture as surface shapes. InICCV, 2019. 1

    2019

  51. [51]

    Isaac gym: High performance gpu-based physics simulation for robot learning.arXiv preprint arXiv:2108.10470, 2021

    Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning.arXiv preprint arXiv:2108.10470, 2021. 8

    Pith/arXiv arXiv 2021

  52. [52]

    The kit whole-body human motion database

    Christian Mandery, ¨Omer Terlemez, Martin Do, Nikolaus Vahrenkamp, and Tamim Asfour. The kit whole-body human motion database. InICAR, 2015. 3

    2015

  53. [53]

    Unifying representations and large-scale whole-body motion databases for studying human motion.IEEE Transactions on Robotics, 32(4):796– 809, 2016

    Christian Mandery, ¨Omer Terlemez, Martin Do, Nikolaus Vahrenkamp, and Tamim Asfour. Unifying representations and large-scale whole-body motion databases for studying human motion.IEEE Transactions on Robotics, 32(4):796– 809, 2016. 3

    2016

  54. [54]

    Single-shot multi-person 3D pose estimation from monocular RGB

    Dushyant Mehta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Srinath Sridhar, Gerard Pons-Moll, and Christian Theobalt. Single-shot multi-person 3D pose estimation from monocular RGB. In3DV, 2018. 2

    2018

  55. [55]

    Catch & carry: reusable neu- ral controllers for vision-guided whole-body tasks.ACM Transactions on Graphics (TOG), 39(4):39–1, 2020

    Josh Merel, Saran Tunyasuvunakool, Arun Ahuja, Yuval Tassa, Leonard Hasenclever, Vu Pham, Tom Erez, Greg Wayne, and Nicolas Heess. Catch & carry: reusable neu- ral controllers for vision-guided whole-body tasks.ACM Transactions on Graphics (TOG), 39(4):39–1, 2020. 3

    2020

  56. [56]

    Interhand2

    Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. Interhand2. 6m: A dataset and baseline for 3d interacting hand pose estimation from a single rgb image. InECCV, 2020. 2

    2020

  57. [57]

    Dataset diffusion: Diffusion-based synthetic data generation for pixel-level semantic segmentation

    Quang Nguyen, Truong Vu, Anh Tran, and Khoi Nguyen. Dataset diffusion: Diffusion-based synthetic data generation for pixel-level semantic segmentation. InNeurIPS, 2024. 5

    2024

  58. [58]

    Assemblyhands: Towards ego- centric activity understanding via 3d hand pose estimation

    Takehiko Ohkawa, Kun He, Fadime Sener, Tomas Hodan, Luan Tran, and Cem Keskin. Assemblyhands: Towards ego- centric activity understanding via 3d hand pose estimation. InCVPR, 2023. 2

    2023

  59. [59]

    Repre- sentation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018. 6

    Pith/arXiv arXiv 2018

  60. [60]

    OpenAI. ChatGPT. https://chat.openai.com/ ,

  61. [61]

    Synthesizing physically plausible human motions in 3d scenes.arXiv preprint arXiv:2308.09036, 2023

    Liang Pan, Jingbo Wang, Buzhen Huang, Junyu Zhang, Haofan Wang, Xu Tang, and Yangang Wang. Synthesizing physically plausible human motions in 3d scenes.arXiv preprint arXiv:2308.09036, 2023. 3

    Pith/arXiv arXiv 2023

  62. [62]

    Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from a single image. InCVPR, 2019. 3, 4, 1, 2

    2019

  63. [63]

    HOI-Diff: Text-driven syn- thesis of 3d human-object interactions using diffusion mod- els.arXiv preprint arXiv:2312.06553, 2023

    Xiaogang Peng, Yiming Xie, Zizhao Wu, Varun Jampani, Deqing Sun, and Huaizu Jiang. HOI-Diff: Text-driven syn- thesis of 3d human-object interactions using diffusion mod- els.arXiv preprint arXiv:2312.06553, 2023. 2, 3, 5, 6, 7

    Pith/arXiv arXiv 2023

  64. [64]

    Object pop-up: Can we infer 3d objects and their poses from human interactions alone? InCVPR, 2023

    Ilya A Petrov, Riccardo Marin, Julian Chibane, and Gerard Pons-Moll. Object pop-up: Can we infer 3d objects and their poses from human interactions alone? InCVPR, 2023. 3

    2023

  65. [65]

    Black, and G¨ul Varol

    Mathis Petrovich, Michael J. Black, and G¨ul Varol. TEMOS: Generating diverse human motions from textual descriptions. InECCV, 2022. 4

    2022

  66. [66]

    TMR: Text-to-motion retrieval using contrastive 3d human motion synthesis

    Mathis Petrovich, Michael J Black, and G ¨ul Varol. TMR: Text-to-motion retrieval using contrastive 3d human motion synthesis. InICCV, 2023. 6

    2023

  67. [67]

    The kit motion-language dataset.Big data, 4(4):236–252,

    Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit motion-language dataset.Big data, 4(4):236–252,

  68. [68]

    Efficient learning on point clouds with basis point sets

    Sergey Prokudin, Christoph Lassner, and Javier Romero. Efficient learning on point clouds with basis point sets. In ICCV, 2019. 5, 7

    2019

  69. [69]

    Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra Quiros-Ramirez, and Michael J

    Abhinanda R. Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra Quiros-Ramirez, and Michael J. Black. BABEL: Bodies, action and behavior with english labels. InCVPR, 2021. 1

    2021

  70. [70]

    Pointnet++: Deep hierarchical feature learning on point sets in a metric space

    Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. InNeurIPS, 2017. 7

    2017

  71. [71]

    Action-conditioned gen- eration of bimanual object manipulation sequences

    Haziq Razali and Yiannis Demiris. Action-conditioned gen- eration of bimanual object manipulation sequences. InAAAI,

  72. [72]

    Sentence-bert: Sentence embeddings using siamese bert-networks.arXiv preprint arXiv:1908.10084, 2019

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks.arXiv preprint arXiv:1908.10084, 2019. 6

    Pith/arXiv arXiv 1908

  73. [73]

    Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies together.ACM Transactions on Graphics, 36(6), 2017. 3, 4, 1, 2

    2017

  74. [74]

    Hoianimator: Generating text-prompt human-object anima- tions using novel perceptive diffusion models

    Wenfeng Song, Xinyu Zhang, Shuai Li, Yang Gao, Aimin Hao, Xia Hou, Chenglizhao Chen, Ning Li, and Hong Qin. Hoianimator: Generating text-prompt human-object anima- tions using novel perceptive diffusion models. InCVPR,

  75. [75]

    Neural state machine for character-scene interactions.ACM Trans

    Sebastian Starke, He Zhang, Taku Komura, and Jun Saito. Neural state machine for character-scene interactions.ACM Trans. Graph., 38(6):209–1, 2019. 3

    2019

  76. [76]

    Local motion phases for learning multi-contact char- acter movements.ACM Transactions on Graphics (TOG), 39(4):54–1, 2020

    Sebastian Starke, Yiwei Zhao, Taku Komura, and Kazi Za- man. Local motion phases for learning multi-contact char- acter movements.ACM Transactions on Graphics (TOG), 39(4):54–1, 2020. 3

    2020

  77. [77]

    GRAB: A dataset of whole-body human grasping of objects

    Omid Taheri, Nima Ghorbani, Michael J Black, and Dim- itrios Tzionas. GRAB: A dataset of whole-body human grasping of objects. InECCV, 2020. 2, 3, 1, 4

    2020

  78. [78]

    GOAL: Generating 4d whole-body motion for hand-object grasping

    Omid Taheri, Vasileios Choutas, Michael J Black, and Dim- itrios Tzionas. GOAL: Generating 4d whole-body motion for hand-object grasping. InCVPR, 2022. 3

    2022

  79. [79]

    Grip: Generating interaction poses using spatial cues and latent consistency

    Omid Taheri, Yi Zhou, Dimitrios Tzionas, Yang Zhou, Duygu Ceylan, Soren Pirk, and Michael J Black. Grip: Generating interaction poses using spatial cues and latent consistency. In3DV, 2024. 4

    2024

  80. [80]

    Flex: Full-body grasping without full-body grasps

    Purva Tendulkar, D´ıdac Sur´ıs, and Carl V ondrick. Flex: Full-body grasping without full-body grasps. InICCV, 2023. 2

    2023

Showing first 80 references.