Towards Consistent Long-Term Pose Generation

Filippos Bellos; Jason Corso; Yayuan Li

REVIEW 3 major objections 6 minor 37 references

Towards Consistent Long-Term Pose Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Pose sequences can be generated in one forward pass from a single image and a text prompt, avoiding the error accumulation that plagues autoregressive and quantized pose generators.

desk verdict A clean non-autoregressive idea for long-term pose generation, undermined by an evaluation that does not rule out split leakage and pseudo-label bias. read the letter →

arxiv 2507.18382 v1 pith:463FAMPA submitted 2025-07-24 cs.CV

Yayuan Li , Filippos Bellos , Jason Corso This is my paper · ORCID

classification cs.CV

keywords posegenerationlong-termforecastingvision-languageconditioningplaceholdertokensrelativemovementpredictioncontinuouscoordinatespacetraining-inferenceconsistencyerroraccumulation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that long-term pose generation fails because existing methods either quantize continuous coordinates into discrete tokens (losing information) or generate frames autoregressively (training and inference inputs differ, so errors accumulate). It proposes a one-stage Transformer that predicts the whole future pose sequence in a single forward pass, using placeholder tokens for every future frame so that training and inference see identical inputs. Instead of absolute coordinates, the model predicts the relative movement of each joint from an initial detected pose, and a loss on pairwise joint distances and directions keeps the body structure coherent. On two datasets (human body and hand actions), the method reports consistently lower error and higher keypoint accuracy than five baselines, with the gap growing at longer horizons. If the claim holds, it means long-horizon pose forecasting can be made drift-resistant without extra scene data or two-stage pipelines.

What carries the argument

The load-bearing mechanism is the placeholder-token input matrix $X_{\mathrm{ours}}$: a $T \times 2N$ block whose first row holds the initial pose coordinates and whose remaining rows are filled with a shared, information-free token $[\mathrm{PRD}]$, so positional encodings alone distinguish timestamps. Because the decoder sees identical inputs during training and inference and predicts all future poses simultaneously with full (non-masked) self-attention, there is no input-distribution shift and no cascade of previously predicted frames. Two further components carry the argument: relative movement prediction, which models each predicted pose as a displacement from the detected initial pose and thereby retains spatial context, and the relative pose representation loss, which supervises pairwise Euclidean distances and unit direction vectors between adjacent joints alongside MSE.

What would settle it

Re-train and re-evaluate the method on a subset of Penn Action and F-PHAB frames with manual keypoint annotations while keeping the same 90/10 video-grouped split; if ADE/FDE rise substantially or flatten the gap versus baselines, the claimed long-term consistency partly reflects detector bias or temporal leakage. A second check: measure whether mean error per timestamp stays flat for the model's own predictions when trained with the placeholder input; a rising curve would refute the claim that train/inference consistency alone removes drift.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the train/inference distribution mismatch—not insufficient capacity—is the main cause of long-horizon pose drift. Because autoregressive training feeds ground-truth poses as inputs while inference feeds its own predictions, errors compound; because quantization-based methods first map poses to discrete codes, precision is lost before generation begins. The paper's architecture sidesteps both: a Transformer decoder receives the detected initial pose $P_0$ and a matrix of placeholder tokens $[\mathrm{PRD}]$ for every future timestamp, and non-masked self-attention produces all future poses $\hat{P}_1 \dots \hat{P}_T$ at once. With the same input structure in both training and inference, the distribution shift disappears. Predictions are made as displacements relative to the initial pose, preserving spatial context, and a loss combining pairwise joint distances, directions, and MSE enforces structural coherence.

Load-bearing premise

The results depend on the automated pose detector's pseudo-annotations being faithful ground truth, and on the 90/10 split not placing frames from the same video on both sides; if either fails, the reported long-horizon advantage may be inflated.

Editorial extensions

If this is right

Forecasting error stays roughly flat as the horizon grows from 1 to 45 frames, instead of rising, so long-horizon pose generation becomes practical for animation and task guidance.
The single-forward design removes the autoregressive inference loop, cutting generation cost to one decoder pass per sequence.
Operating directly in continuous coordinates avoids the information bottleneck of VQ-VAE and quantization stages, so reconstruction quality no longer caps prediction accuracy.
The same architecture transfers across body and hand targets with only a change of keypoint count, suggesting the mechanism is target-agnostic.
Language-only and image-only inputs are combined through frozen multimodal features, so the method needs no 3D scene data, broadening applicability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct test of the paper's mechanism: an autoregressive baseline trained to predict all tokens in one pass with the same placeholder trick should show a similar plateau in long-horizon error, isolating the placeholder input as the source of the gain rather than the relative-loss or multimodal fusion.
If the placeholder-token trick generalizes, it could be applied to other continuous spatiotemporal regression tasks—such as object trajectory forecasting or audio waveform generation—where autoregressive drift is the dominant failure mode.
The paper's pseudo-annotation assumption could be checked by re-evaluating on a small human-annotated subset; if the relative-movement targets are detector-aligned rather than anatomically aligned, the reported error might understate real pose error.
Because the 90/10 split's grouping is unspecified, a video-level split experiment would clarify whether part of the long-horizon gain comes from temporal leakage between training and test clips.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper's core idea is genuinely neat: instead of autoregressive next-token prediction, feed the decoder the initial pose plus one shared placeholder token for every future frame, and let it produce the whole sequence in a single forward pass in continuous coordinates. That removes the train/inference input mismatch that makes autoregressive pose forecasting drift, and the relative-movement loss (pairwise distances and directions) is a known but sensible way to keep joint structure intact. The writing is clear and the method is simple enough to reproduce.

What is actually new is the specific combination: one-stage, non-autoregressive, continuous-coordinate pose generation conditioned on a single frame and short text label. The closest prior work is [24] on non-autoregressive image generation without vector quantization; the authors cite it. The placeholder-token mechanism is essentially parallel decoding, but applying it to image-and-text-conditioned pose forecasting is a fair contribution.

The soft spots are serious. First, the 90/10 split is not described as video-grouped. The data are video streams; if the split is at frame level, near-duplicate sequences from the same video appear in both train and test, which would flatten the error-vs-horizon curve—the paper's central evidence—and would also explain why the nearest-neighbor baseline scores ADE 0.09. This needs to be fixed and re-run. Second, all pose ground truth is Mediapipe pseudo-labels. That is acceptable for training, but using the same detector output as evaluation ground truth means the metrics may measure fit to the detector, not to real pose. Third, Table 4 lists 'Ours' with ADE 0.0169 on Penn Action while Table 1 gives ADE 0.0578 for the same method and dataset. That is a factor-of-three discrepancy, not a minor typo; the protocols must be reconciled. Fourth, there are no error bars or repeated-seed results, and the loss weights alpha, beta, theta are not reported. Individually these are minor, but together with the leakage issue they leave the headline claim unverified.

The citation pattern looks reasonable, and the authors are transparent that the SOTA comparison is not entirely fair.

Verdict: this paper should go to peer review—the idea deserves referee time—but the authors need to rerun on a video-grouped split, reconcile Table 4 with Table 1, and validate against at least a subset of human-annotated poses before the empirical claims can be trusted.

Referee Report

3 major / 6 minor

Summary. The paper proposes a one-stage vision-language-conditioned pose generation model that predicts T future 2D poses in a single forward pass from one RGB image and a short text description. The key ideas are (i) a unified placeholder token [PRD] in the decoder input, so that training and inference use the same input structure and autoregressive exposure bias is avoided, and (ii) a relative pose representation loss combining pairwise distance and direction terms with an MSE term. Experiments on Penn Action and F-PHAB compare the method against nearest-neighbor, LSTM, Transformer, and VQ-VAE-plus-Transformer baselines, with additional hardness analysis, ablations, and a comparison to prior single-modality generation work. The central claim is that the placeholder-token design keeps prediction error roughly flat as the forecasting horizon grows, unlike autoregressive baselines that drift.

Significance. If the empirical results hold, the core architectural idea is a legitimate and simple way to align training and inference for continuous pose regression: predicting all future poses from placeholder tokens in one forward pass removes the main source of error accumulation in autoregressive pose generation. The relative-displacement formulation is also a sensible inductive bias, and the paper makes a useful conceptual point that quantization is not necessary for long-horizon pose prediction. However, the significance of the paper as submitted is undermined by evaluation-protocol problems: the train/test split is not specified as video-grouped, the ground truth appears to be Mediapipe pseudo-annotations rather than native annotations, loss weights and other free parameters are unreported, and Table 4 reports numbers that are inconsistent with Table 1 for the same method. The method itself is coherent and not circular; the weakness is in the validation, which needs to be repaired before the stated claims can be accepted.

major comments (3)

[Sec. 4.1 and Fig. 4] The 90/10 split is not stated to group frames by video or clip. If the split is performed at the frame/sample level, adjacent frames from the same video can appear in both training and test sets, making long-term forecasting artificially easy and flattening the error-versus-horizon curve in Figure 4. This would also explain the strong nearest-neighbor baseline NNP (ADE 0.0901 in Table 1), since a near-duplicate training sequence may be retrievable for each test sequence. Please specify the split granularity, re-run the experiments with a video-grouped split, and report results over multiple split seeds with error bars; without this, the central claim that the placeholder design avoids accumulated error cannot be validated from the text.
[Sec. 4.1 and Sec. 4.3] The evaluation appears to use Mediapipe-generated pseudo-annotations as both training supervision and ground truth for all metrics. The paper states 'For datasets without pose annotations, we use Mediapipe to generate pseudo-annotations,' but both Penn Action and F-PHAB are standard benchmarks with native pose annotations (2D joint annotations and 3D hand pose annotations, respectively). If pseudo-labels replace these native annotations, detector bias and noise propagate into the relative-movement targets, the distance and direction losses, and the reported ADE/FDE/PCK/RMSE values, so the reported superiority may reflect fitting the detector rather than true pose. Please state explicitly which annotation source is used for each dataset and split, evaluate on native annotations where possible, and quantify the detector's error against them.
[Sec. 4.5, Table 4] Table 4 reports 'Ours' on Penn Action with ADE 0.0169, FDE 0.0170, PCK 0.860, and RMSE 0.012, whereas Table 1 reports ADE 0.0578, FDE 0.0766, PCK 0.8179, and RMSE 0.0350 for the same method and dataset. No protocol difference is described, so the two tables are mutually inconsistent. In addition, the comparison with PHD [36] mixes a 3D video-prediction task with the 2D single-image setting, and only the PCK column is available for that method. Please reconcile the numbers or remove the SOTA comparison, and provide the precise evaluation protocol used for each row in Table 4.

minor comments (6)

[Sec. 1] The phrase 'due to accumulated performance' should read 'due to accumulated error'.
[Sec. 3.1] There is a missing period after 'relies only on a single RGB image' before 'We parameterize each pose'.
[Sec. 4] The word 'effeciency' in 'ablation study to demonstrate the effeciency of our designs' should be 'effectiveness'.
[Fig. 4] The y-axis label is missing; please provide labeled axes and units so the reader can interpret the error-versus-horizon curves.
[Sec. 4.4] For the nearest-neighbor baselines, please state whether retrieval is performed only within the training split and how the retrieval set is affected by the train/test split protocol.
[Sec. 4.2 and Eqs. (8), (11)] The values of alpha, beta, theta, and the normalization factor sigma are not reported; please report the chosen values and, if possible, a small sensitivity analysis.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the one-forward placeholder decoder and relative pose loss are self-contained; remaining concerns are benchmark-validity issues, not derivation circularity.

full rationale

The paper's derivation chain is self-contained: the model is a single-pass Transformer decoder (Eq. 1) that maps the detected initial pose P0 and frozen BLIP vision-language features to T future poses in one forward pass, and training and inference use the same placeholder-token input structure (Eq. 3), so the claimed training-inference consistency is a direct architectural property rather than a result imported from the evaluation. The relative pose loss (Eqs. 4-11) is a standard structural auxiliary loss on pairwise distances and directions plus MSE, with loss weights as ordinary hyperparameters; it does not encode the benchmark metrics and is not fitted to the model's own predictions. There are no self-citations to prior work by this authorship, and no fitted parameter is relabeled as a prediction. The concerns a reader might raise—Mediapipe pseudo-annotations serving as both training supervision and evaluation ground truth, the unstated whether the 90/10 split is grouped by video, and the admittedly unfair SOTA comparison in Table 4—are external-validity or benchmark-construction issues, not circular steps: the model's outputs are not defined in terms of its own outputs, and the central one-forward mechanism is independent of those evaluation choices.

Assumptions & free parameters 6 free parameters · 4 assumptions · 1 invented entities

The method's central claim rests on the detected initial pose, pseudo-labeled ground truth, a split that may leak temporal context, short text labels, and the placeholder-token design. These are domain assumptions rather than formal axioms. Several loss weights are free parameters with unreported values.

free parameters (6)

alpha (distance loss weight) = not reported
Equation (8) combines distance and direction losses with weights alpha and beta, but no values or tuning procedure are given.
beta (direction loss weight) = not reported
Equation (8) uses beta for the direction loss; the value is not reported.
theta (MSE loss weight) = not reported
Equation (11) adds an MSE term scaled by theta; the value is not reported.
sigma normalization factor = 0.8
Section 4.2 sets sigma to 0.8 without justification.
PCK threshold delta = 0.05 for human, 0.15 for hands
Section 4.3 sets two thresholds by target type; the choice is conventional but target-specific.
forecasting horizon T = 45
Training and evaluation use 45 timestamps; results depend on this horizon choice.

assumptions (4)

domain assumption Mediapipe pose detection provides accurate initial pose P0 and pseudo-labels for training and evaluation.
Section 3.2 uses P0 from a pre-trained pose detector and Section 4.1 uses Mediapipe for datasets without pose annotations. If detection is poor, the relative-movement targets and evaluation metrics are unreliable.
domain assumption A random 90/10 split prevents temporal leakage between training and test sequences.
Section 4.1 states only the split ratio; no mention of grouping by video, so adjacent frames from the same video may appear in both training and test sets.
domain assumption Short text descriptions such as 'golf_swing' provide enough semantic signal to drive pose generation.
The method relies on short labels; no ablation removes the text branch or analyzes text ambiguity, so the actual contribution of language is not demonstrated.
domain assumption Transformer positional encoding alone can make identical placeholder rows produce distinct pose outputs.
Section 3.2 states positional encoding is the sole distinguishable information; this is an architectural assumption not tested by removing positional encodings.

invented entities (1)

Unified placeholder token [PRD]
purpose: A constant input vector marking future timestamps in the decoder, enabling one-forward parallel prediction.
The token is a paper-specific construct; its benefit is shown only through the paper's own ablations, and no independent or formal evidence is given.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Towards Consistent Long-Term Pose Generation." pith.science (2026). https://pith.science/paper/463FAMPA

@misc{pith2026250718382,
  author       = {Pith},
  title        = {Pith review of: Towards Consistent Long-Term Pose Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/463FAMPA}},
  note         = {Machine review of arXiv:2507.18382}
}

read the original abstract

Current approaches to pose generation rely heavily on intermediate representations, either through two-stage pipelines with quantization or autoregressive models that accumulate errors during inference. This fundamental limitation leads to degraded performance, particularly in long-term pose generation where maintaining temporal coherence is crucial. We propose a novel one-stage architecture that directly generates poses in continuous coordinate space from minimal context - a single RGB image and text description - while maintaining consistent distributions between training and inference. Our key innovation is eliminating the need for intermediate representations or token-based generation by operating directly on pose coordinates through a relative movement prediction mechanism that preserves spatial relationships, and a unified placeholder token approach that enables single-forward generation with identical behavior during training and inference. Through extensive experiments on Penn Action and First-Person Hand Action Benchmark (F-PHAB) datasets, we demonstrate that our approach significantly outperforms existing quantization-based and autoregressive methods, especially in long-term generation scenarios.

Figures

Figures reproduced from arXiv: 2507.18382 by the authors.

**Figure 2.** Examples about the issue of long term forecasting from [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Overview of our proposed method. Given a single RGB image [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Performance across generation timestamps. Our method consistently outperforms baseline methods across most of the times [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Qualitative results of our proposed method. Red poses are annotations, and blue poses are predictions from our method. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 21 canonical work pages

[24]

Autoregressive image generation without vec- tor quantization

Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vec- tor quantization. arXiv preprint arXiv:2406.11838, 2024. 2

arXiv 2024
[36]

Predicting 3d human dynamics from video

Jason Y Zhang, Panna Felsen, Angjoo Kanazawa, and Jiten- dra Malik. Predicting 3d human dynamics from video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7114–7123, 2019. 8

work page 2019
[1]

Lan- guage2pose: Natural language grounded pose forecasting

Chaitanya Ahuja and Louis-Philippe Morency. Lan- guage2pose: Natural language grounded pose forecasting. 2019 International Conference on 3D Vision (3DV) , pages 719–728, 2019. 1

2019
[2]

The pitfalls of next-token prediction, 2024

Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction, 2024. 1, 2

work page 2024
[3]

Language models are few-shot learners

Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. 3

arXiv 2005
[4]

Realtime multi-person 2d pose estimation using part affinity fields

Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7291–7299, 2017. 4

work page 2017
[5]

Long-term Human Motion Prediction with Scene Context

Zhe Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh V o, and Jitendra Malik. Long-term human motion pre- diction with scene context. ArXiv, abs/2007.03672, 2020. 1

work page Pith review arXiv 2007
[6]

Forecasting human dynamics from static images

Yu-Wei Chao, Jimei Yang, Brian Price, Scott Cohen, and Jia Deng. Forecasting human dynamics from static images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 548–556, 2017. 2

work page 2017

Show all 37 references

[7]

Humanmac: Masked motion completion for human motion prediction

Ling-Hao Chen, Jiawei Zhang, Yewen Li, Yiren Pang, Xi- aobo Xia, and Tongliang Liu. Humanmac: Masked motion completion for human motion prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 9544–9555, 2023. 2

2023
[8]

Action-agnostic human pose fore- casting

Hsu-kuang Chiu, Ehsan Adeli, Borui Wang, De-An Huang, and Juan Carlos Niebles. Action-agnostic human pose fore- casting. In 2019 IEEE winter conference on applications of computer vision (WACV), pages 1423–1432. IEEE, 2019. 1

2019
[9]

Genhowto: Learning to generate actions and state trans- formations from instructional videos

Dima Damen, Michael Wray, Ivan Laptev, Josef Sivic, et al. Genhowto: Learning to generate actions and state trans- formations from instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6561–6571, 2024. 2

2024
[10]

Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Za¨ıd Harchaoui, and Yejin Choi

Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jian, Bill Yuchen Lin, Peter West, Chandra Bhagavat- ula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Za¨ıd Harchaoui, and Yejin Choi. Faith and fate: Limits of transfor...

2023 arXiv
[11]

Yao Feng, Jing Lin, Sai Kumar Dwivedi, Yu Sun, Priyanka Patel, and Michael J. Black. ChatPose: Chatting about 3d human pose. In CVPR, 2024. 2, 4

2024
[12]

Recurrent network models for human dynam- ics

Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Ji- tendra Malik. Recurrent network models for human dynam- ics. In Proceedings of the IEEE international conference on computer vision, pages 4346–4354, 2015. 2

2015
[13]

Future pose pre- diction from 3d human skeleton sequence with surrounding situation

Tomohiro Fujita and Yasutomo Kawanishi. Future pose pre- diction from 3d human skeleton sequence with surrounding situation. Sensors, 23(2):876, 2023. 2

2023
[14]

First-person hand action bench- mark with rgb-d videos and 3d hand pose annotations

Guillermo Garcia-Hernando, Shanxin Yuan, Seungryul Baek, and Tae-Kyun Kim. First-person hand action bench- mark with rgb-d videos and 3d hand pose annotations. In Proceedings of Computer Vision and Pattern Recognition (CVPR), 2018. 2, 5

2018
[15]

Tm2d: Bimodality driven 3d dance generation via music-text integration

Kehong Gong, Dongze Lian, Heng Chang, Chuan Guo, Zi- hang Jiang, Xinxin Zuo, Michael Bi Mi, and Xinchao Wang. Tm2d: Bimodality driven 3d dance generation via music-text integration. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9942–9952,
[16]

Ac- tion2motion: Conditioned generation of 3d human motions

Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Ac- tion2motion: Conditioned generation of 3d human motions. In Proceedings of the 28th ACM International Conference on Multimedia, page 2021–2029, New York, NY , USA, 2020. Assoc...

2021
[17]

Tm2t: Stochastic and tokenized modeling for the reciprocal genera- tion of 3d human motions and texts

Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for the reciprocal genera- tion of 3d human motions and texts. InEuropean Conference on Computer Vision, pages 580–597. Springer, 2022. 1, 2, 3, 6, 8

2022
[18]

Momask: Generative masked model- ing of 3d human motions

Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Generative masked model- ing of 3d human motions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1900–1910, 2024. 2

1900
[19]

Long short-term memory

Sepp Hochreiter and J ¨urgen Schmidhuber. Long short-term memory. Neural Comput., 9(8):1735–1780, 1997. 6

1997
[20]

Diffusion- based generation, optimization, and planning in 3d scenes

Siyuan Huang, Zan Wang, Puhao Li, Baoxiong Jia, Tengyu Liu, Yixin Zhu, Wei Liang, and Song-Chun Zhu. Diffusion- based generation, optimization, and planning in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16750–16761, 2023. 2

2023
[21]

Lego: Learning egocentric action frame generation via visual instruction tuning

Bolin Lai, Xiaoliang Dai, Lawrence Chen, Guan Pang, James M Rehg, and Miao Liu. Lego: Learning egocentric action frame generation via visual instruction tuning. arXiv preprint arXiv:2312.03849, 2023. 2, 4

2023 arXiv
[22]

Dancing to music

Hsin-Ying Lee, Xiaodong Yang, Ming-Yu Liu, Ting-Chun Wang, Yu-Ding Lu, Ming-Hsuan Yang, and Jan Kautz. Dancing to music. ArXiv, abs/1911.02001, 2019. 1

1911 arXiv
[23]

Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Interna- tional conference on machine learning, pages 12888–12900. PMLR, 2022. 4

2022
[25]

Medi- apipe: A framework for building perception pipelines

Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris Mc- Clanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo- Ling Chang, Ming Guang Yong, Juhyun Lee, et al. Medi- apipe: A framework for building perception pipelines. arXiv preprint arXiv:1906.08172, 2019. 3

1906 arXiv
[26]

Contact-aware human motion generation from textual de- scriptions

Sihan Ma, Qiong Cao, Jing Zhang, and Dacheng Tao. Contact-aware human motion generation from textual de- scriptions. arXiv preprint arXiv:2403.15709, 2024. 1, 2

2024 arXiv
[27]

Action- conditioned 3d human motion synthesis with transformer vae

Mathis Petrovich, Michael J Black, and G ¨ul Varol. Action- conditioned 3d human motion synthesis with transformer vae. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 10985–10995, 2021. 1, 2 9

2021
[28]

What matters in learning from of- fline human demonstrations for robot manipulation

Jackson Spencer, Sanjiban Choudhury, Matthew Barnes, Christopher Dellin, et al. What matters in learning from of- fline human demonstrations for robot manipulation. In Con- ference on Robot Learning, 2022. 2

2022
[29]

Bigram-based nat- ural language model and statistical motion symbol model for scalable language of humanoid robots

Wataru Takano and Yoshihiko Nakamura. Bigram-based nat- ural language model and statistical motion symbol model for scalable language of humanoid robots. 2012 IEEE Interna- tional Conference on Robotics and Automation, pages 1232– 1237, 2012. 2

2012
[30]

Statistical mu- tual conversion between whole body motion primitives and linguistic sentences for human motions

Wataru Takano and Yoshihiko Nakamura. Statistical mu- tual conversion between whole body motion primitives and linguistic sentences for human motions. The International Journal of Robotics Research, 34:1314 – 1328, 2015. 2

2015
[31]

Human motion diffu- sion model

Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffu- sion model. In The Eleventh International Conference on Learning Representations, 2023. 2

2023
[32]

Attention is all you need

A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 6

2017
[33]

Synthesizing long-term 3d human motion and in- teraction in 3d scenes

Jiashun Wang, Huazhe Xu, Jingwei Xu, Sifei Liu, and Xiao- long Wang. Synthesizing long-term 3d human motion and in- teraction in 3d scenes. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 9401–9411, 2021. 1, 2

2021
[34]

Humanise: Language-conditioned hu- man motion generation in 3d scenes

Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. Humanise: Language-conditioned hu- man motion generation in 3d scenes. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 2

2022
[35]

Move as you say, interact as you can: Language-guided human motion generation with scene af- fordance

Zan Wang, Yixin Chen, Baoxiong Jia, Puhao Li, Jinlu Zhang, Jingze Zhang, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. Move as you say, interact as you can: Language-guided human motion generation with scene af- fordance. In Proceedings of the IEEE/CVF Conference on Comp...

2024
[37]

From actemes to action: A strongly-supervised representa- tion for detailed action understanding

Weiyu Zhang, Menglong Zhu, and Konstantinos G Derpanis. From actemes to action: A strongly-supervised representa- tion for detailed action understanding. In Proceedings of the IEEE international conference on computer vision, pages 2248–2255, 2013. 2, 5 10

2013

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [24]

Autoregressive image generation without vec- tor quantization

Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vec- tor quantization. arXiv preprint arXiv:2406.11838, 2024. 2

arXiv 2024

[2] [36]

Predicting 3d human dynamics from video

Jason Y Zhang, Panna Felsen, Angjoo Kanazawa, and Jiten- dra Malik. Predicting 3d human dynamics from video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7114–7123, 2019. 8

work page 2019

[3] [1]

Lan- guage2pose: Natural language grounded pose forecasting

Chaitanya Ahuja and Louis-Philippe Morency. Lan- guage2pose: Natural language grounded pose forecasting. 2019 International Conference on 3D Vision (3DV) , pages 719–728, 2019. 1

2019

[4] [2]

The pitfalls of next-token prediction, 2024

Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction, 2024. 1, 2

work page 2024

[5] [3]

Language models are few-shot learners

Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. 3

arXiv 2005

[6] [4]

Realtime multi-person 2d pose estimation using part affinity fields

Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7291–7299, 2017. 4

work page 2017

[7] [5]

Long-term Human Motion Prediction with Scene Context

Zhe Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh V o, and Jitendra Malik. Long-term human motion pre- diction with scene context. ArXiv, abs/2007.03672, 2020. 1

work page Pith review arXiv 2007

[8] [6]

Forecasting human dynamics from static images

Yu-Wei Chao, Jimei Yang, Brian Price, Scott Cohen, and Jia Deng. Forecasting human dynamics from static images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 548–556, 2017. 2

work page 2017