Pith. sign in

REVIEW 3 major objections 4 minor 72 references

StreamTalk: Streaming Co-Speech Gesture Generation with Key-Pose Anchoring

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read StreamTalk claims that a single retrieved key pose at each clip's tail, used as a destination anchor with flow-matching refinement, suppresses long-horizon drift in streaming co-speech gesture generation while running in real time at 76…

desk verdict A genuinely useful closed-loop recipe for streaming gesture generation, but the headline SOTA FGD comparison is confounded by test-time injection of real training poses; the core idea still deserves refereeing. read the letter →

arxiv 2608.01643 v1 pith:BBFDFPBY submitted 2026-08-03 cs.CV cs.AIcs.GR

Xiangyue Zhang , Jianfang Li , Jiaxu Zhang , Kaixing Yang , Steven Hoi This is my paper · ORCID
classification cs.CVcs.AIcs.GR
keywords co-speechgesturegenerationstreamingmotionsynthesisclosed-loopkey-poseanchoringretrieval-basedfeedbackflowmatchingdriftsuppressionBEAT2
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

StreamTalk addresses the drift problem in streaming co-speech gesture generation, where per-clip errors accumulate over minute-scale sequences because each model has no information about where its motion should arrive. The paper argues that drift is a direction problem, not a local-quality problem: modern diffusion backbones already make convincing short clips, but an open-loop generator can wander anywhere that locally satisfies the audio. Its solution is a closed-loop generate-retrieve-refine cycle that supplies a single plausible tail pose, retrieved from a speaker-specific motion database, as a destination anchor for each clip. The authors report state-of-the-art FGD on BEAT2, significantly suppressed long-horizon drift relative to open-loop baselines, and real-time inference at 76 FPS.

What carries the argument

The load-bearing object is the Streaming Pose-Guided Generation (SPG) cycle, a generate-retrieve-refine loop at each 60-frame clip boundary. Generate produces a coarse clip by integrating a flow-matching model; Retrieve matches the tail pose in forward-kinematics joint space against a per-speaker database and replaces the tail frame with the nearest candidate; Refine re-integrates from the cached midpoint with the new anchor before forwarding tail frames to the next window. Two supporting mechanisms make the anchor usable: Stochastic Anchor Masking (SAM), which randomly masks pose and translation frames during training so the model learns to inpaint motion from sparse boundary conditions, and a part-aware DiT with separate hand, body, and translation branches, which lets SPG supply pose-only anchors without disturbing translation. The refinement pass costs roughly half the initial pass because it starts from the cached midpoint, which is how the closed loop stays real-time.

What would settle it

Run StreamTalk on a long speech segment whose ground-truth gesture ends in a pose far from every database entry, such as an instructed novel arm raise, and measure whether the retrieved nearest anchor forces an unnatural jerk or discontinuity at the refined tail in joint-trajectory error against ground truth. If no such degradation appears when the anchor is semantically mismatched, the paper's claim that retrieval quality drives drift suppression would be weakened; if degradation appears, the anchor's plausibility is confirmed as load-bearing.

Watch

Extended reading notes

Core claim

The central claim is that open-loop streaming co-speech gesture generation drifts because of the absence of forward constraints, and that closing the loop with a single key-pose anchor per clip is sufficient to restore long-horizon stability. StreamTalk implements this by first generating a coarse clip conditioned on past context, retrieving the nearest plausible tail pose from a per-speaker database using forward-kinematics joint-space distance, then re-integrating the flow-matching ODE from the cached midpoint with the anchor as an updated prompt. Training with Stochastic Anchor Masking prepares the model to inpaint complete motion from exactly these sparse boundary conditions, and a part-aware DiT keeps hand, body, and translation streams separate so that retrieved pose anchors can be injected without borrowing context-dependent global translation. The paper reports that this design achieves the best FGD among compared methods on BEAT2 in both single-speaker and all-speakers settings, keeps FGD flat across roughly 1,800 frames where open-loop baselines drift upward, and runs at 76 FPS on a V100.

Load-bearing premise

The load-bearing premise is that a nearest-neighbor pose pulled from the speaker's training-set database is a valid destination for the current clip, even though that pose was recorded under different speech, rhythm, and context; if the retrieved anchor is semantically wrong, the refinement step enforces a wrong waypoint.

Editorial extensions

If this is right

  • A closed-loop, periodic correction at clip boundaries is enough to keep minute-scale generated motion near the natural pose distribution; open-loop overlap extension or longer context windows address only local smoothness.
  • Retrieval quality, not anchor density, is the controlling factor: a single well-chosen tail pose suppresses drift, and adding more anchors per clip degrades FGD.
  • The closed loop reduces physical implausibility: self-intersecting frames drop by roughly an order of magnitude because retrieved anchors come from anatomically valid training data.
  • The pipeline is real-time at 76 FPS on a V100 with the default database and stays above real-time (36 FPS) even with a 20x larger database, so the correction cycle is practical for live avatars.
  • Because SPG operates on the output trajectory rather than on any single backbone, the same closed-loop correction can be applied on top of either VQ-VAE or diffusion streaming generators.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to apply the same generate-retrieve-refine pattern to other autoregressive motion tasks, such as music-driven dance or text-to-motion, wherever a database of plausible continuation states exists.
  • The paper retrieves pose only, not translation, because global trajectory is context-dependent; I infer that pure root-position drift remains uncorrected, and a context-conditioned translation anchor would be a direct next test.
  • The nearest-neighbor search in joint space is exact; I infer that replacing it with an approximate index would let the database scale well beyond the tested 20x size without changing the generation pipeline, at some cost in anchor quality.
  • Because the paper ties drift suppression to a single forward waypoint, I infer that any source of a trustworthy destination, such as a learned trajectory prior or a motion critic, could substitute for retrieval; the essential ingredient is the forward constraint, not the database itself.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes StreamTalk, a streaming co-speech gesture generation method that replaces open-loop clip-by-clip generation with a generate–retrieve–refine cycle. At inference (Sec. 3.3, Algorithm 1), a coarse 60-frame clip is generated by a flow-matching DiT, its tail pose is matched against a per-speaker database of training-set poses, the nearest candidate below threshold θp is used as a hard tail anchor, and the clip is refined from the cached midpoint. Stochastic Anchor Masking (Sec. 3.4) trains the model to inpaint from sparse boundary conditions, and a part-aware DiT separates hands, body, and translation streams. On BEAT2 the method reports state-of-the-art FGD (0.383 in the 1-speaker setting, 0.293 in the all-speakers setting), stable sliding-window FGD over roughly 60 seconds, and real-time performance at 76 FPS.

Significance. If the central claims survive a controlled test, the work has a useful message: a single forward destination anchor can suppress drift in streaming motion generation, and retrieval from a speaker-specific library can be an effective practical component. The paper includes several internally consistent ablations—Table 2(c) shows that random anchors do not reproduce the gain, Table 2(a) shows that SAM and SPG together improve over the base model, and Table 3(a) validates the single-tail-key design—and the efficiency analysis is concrete. The main caveat is that the headline comparisons do not isolate the effect of injecting real training poses at inference from the effect of closing the loop, so the state-of-the-art and drift-suppression claims are currently over-stated.

major comments (3)
  1. [Sec. 3.3, Algorithm 1, Sec. 4.1] The inference-time database is built from the training partition, and SPG replaces each clip's tail frame with a retrieved real training pose before refinement. This means StreamTalk's test-time outputs are periodically conditioned on real frames from the training distribution, a resource that none of the open-loop baselines in Table 1 or Figure 7(a) can access. This confound alone could explain part of the FGD gain and the flattened drift curve, because the output distribution is repeatedly pulled toward ground-truth poses. The argument in Section 4.1 that speaker-ID conditioning makes the comparison fair is not equivalent: speaker ID tells the model a style, not a concrete valid pose at the current time. Table 2(c) actually strengthens the concern, since random anchors fail to reproduce the gain, showing that the real-pose database is doing work. Please add a controlled experiment that isolates the closed-loop mechanism from the data-source effect, e.g., (i) SPG with anchors retrieved from a held-out validation/test database, or (ii) SPG with anchors generated by an unconditional model, or (iii) an existing open-loop baseline wrapped in the same SPG loop. Without one of these, the claims of state-of-the-art motion quality and significant long-horizon drift suppression are not established.
  2. [Sec. 4.2, Quantitative Comparison] The paper claims that StreamTalk's closed-loop correction benefits both diffusion and VQ-VAE paradigms because it operates on the output trajectory rather than any specific backbone, but no diffusion or VQ-VAE baseline is ever run inside the SPG loop. The ablations in Tables 2 and 3 only vary StreamTalk's own components, so the backbone-agnosticism claim is unsupported. A minimal test would be to apply the same generate–retrieve–refine wrapper to one diffusion baseline (e.g., GestureLSM or DiffSHEG) and one VQ-VAE baseline (e.g., EMAGE or SemTalk) and report FGD and sliding-window drift.
  3. [Fig. 7(a), Sec. 4.2] The long-horizon sliding-window FGD comparison is the primary evidence for drift suppression, but it inherits the same confound as the static FGD: every clip is re-anchored to a real training-set pose. In addition, the evaluation protocol is underspecified: the window size and stride for the sliding-window FGD are not stated, and it is unclear whether the long test sequences come from the test split while the database contains only training frames. Please specify the protocol and, in the controlled experiment requested above, report sliding-window FGD for the loop-closure condition and the real-pose-injection condition separately.
minor comments (4)
  1. [Sec. 3.3, Retrieve] The acceptance threshold θp is never given a value, and the behavior when no candidate falls below θp is not specified. Because retrieval failure changes the effective anchor rate, please report the threshold value and the fallback rule.
  2. [Fig. 7(a)] The sliding-window FGD protocol should state the window length, the stride, and the number of test sequences used, so that the drift claim is reproducible.
  3. [Table 3(c)] The self-intersection counts should describe how intersection frames are detected and whether the same detection procedure is applied to all baselines; as written, the comparison is difficult to interpret.
  4. [Sec. 4.1 vs. Table 3] The 1-speaker setting is called Speaker 'Scott' in Section 4.1 but Speaker-2 in Table 3; please reconcile the naming.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the anchor/refinement mechanism is empirically isolated by ablations showing random anchors and direct anchor insertion degrade FGD, so the central claim is not equivalent to its inputs.

full rationale

The paper's central derivation is that supplying a single retrieved key pose at each clip tail and refining the clip with flow-matching reduces drift. This is not circular. The retrieved pose is a conditioning input rather than a hidden copy of the output: Algorithm 1 re-integrates from a cached midpoint with the pose as a prompt, and the final frame is not set equal to the anchor. The training-set database is an external resource, but the ablations show that the resource alone does not explain the result. Table 2(c) reports Random anchor FGD 0.673, Retrieved anchor 0.503, Retrieved + Linear refinement 0.471, and Retrieved + our refinement 0.383, while Table 2(a) gives StreamTalk (base) 0.478; therefore naively injecting a real training pose actually worsens FGD, and the improvement appears only when the retrieved anchor is coupled with corrective re-integration. Random anchors also fail, so the benefit is not simply 'any real pose from the training set.' The 'drift is a direction problem' conclusion is supported by the tail vs middle vs random placement ablation (Table 3a), which is an empirical test rather than a definitional restatement. No load-bearing self-citation or imported uniqueness theorem is present: self-citations appear as baselines (SemTalk, EchoMask) or implementation details (clip length, FK loss), not as premises that force the result. The use of a training-set motion database at inference is a potential benchmark-resource confound relative to open-loop baselines, but that is an evaluation-fairness concern, not a circular derivation, because the reported FGD is not mathematically forced by the database as shown by the random-anchor and no-refinement controls.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim of drift suppression via a single destination anchor imports several assumptions: that local pose can be transplanted across clips after removing translation, that a single nearest pose from training is a valid target for the current speech, that the refinement pass preserves audio conditioning, and that the BEAT2 FGD comparison is fair despite the test-time injection of training poses. No explicit free parameter is fitted to data, but the retrieval threshold, loss weights, and streaming protocol are hand-chosen, and the threshold value is not reported.

free parameters (5)
  • Retrieval acceptance threshold theta_p = not reported
    SPG only replaces the tail with a retrieved pose whose FK joint-space distance is below theta_p; the threshold controls how often anchors are accepted, and no value or sensitivity study is given.
  • Loss weights lambda_fk and lambda_prompt = lambda_fk = 1, lambda_prompt = 0.1
    Chosen by hand for the training objective; they affect motion quality but are not central to the conceptual claim.
  • Number of key poses per clip n = 1
    The paper argues one anchor is best, and ablation Table 3a supports this, but it is still a hand-set design hyperparameter.
  • Flow-matching steps N = 10
    Denoising and ODE steps at inference; affects speed and quality.
  • Clip length and prompt overlap = 60 frames, 8-frame overlap
    Streaming protocol constants inherited from prior work; they set the granularity of the feedback loop.
assumptions (5)
  • domain assumption Local joint rotation poses are trajectory-invariant and can be transplanted across clips without the associated translation.
    Section 3.3 states translation is context-dependent and deliberately excluded from retrieval, assuming hand and arm configurations from one recording remain valid in another.
  • domain assumption A single retrieved tail pose remains semantically and rhythmically compatible with the current speech after refinement.
    SPG replaces the tail before refinement; if the nearest database pose clashes with the utterance's meaning or beat, the anchor would pull motion away from audio-conditioned content. The paper does not measure semantic fidelity beyond beat consistency.
  • domain assumption Flow-matching re-integration from a cached midpoint under a changed prompt recovers a valid motion and does not undo the audio conditioning.
    Algorithm 1 Refine restarts from stage floor(N/2) with an edited prompt; this assumes the ODE trajectory can be corrected without recomputing from the start.
  • domain assumption FGD on BEAT2 is a fair measure of drift and quality even though test-time outputs contain real training-set frames at clip tails.
    The database is built from the training partition; the paper does not quantify how much of its FGD gain comes from copying real poses into generated clips.
  • standard math SMPL-X forward kinematics and 6D rotation representation correctly map joint rotations to joint positions.
    Relied on for the FK loss and for joint-space retrieval; no new mathematical claim is made.

how reviews work

0 comments
Cite this review

Pith. "Pith review of StreamTalk: Streaming Co-Speech Gesture Generation with Key-Pose Anchoring." pith.science (2026). https://pith.science/paper/BBFDFPBY

@misc{pith2026260801643,
  author       = {Pith},
  title        = {Pith review of: StreamTalk: Streaming Co-Speech Gesture Generation with Key-Pose Anchoring},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BBFDFPBY}},
  note         = {Machine review of arXiv:2608.01643}
}
read the original abstract

Real-time co-speech gesture generation must produce 3D motion clip by clip as speech arrives. Existing streaming methods are open-loop: each clip depends on past context, but the model cannot check or correct its trajectory. Small errors therefore accumulate and cause drift over long sequences. We observe that this failure is mainly caused by the lack of a forward constraint rather than poor short-clip quality. A plausible key pose at the end of each clip provides a destination anchor that limits drift. Based on this observation, we propose StreamTalk, a closed-loop framework with a periodic generate-retrieve-refine cycle. Streaming Pose-Guided Generation first predicts a coarse clip, retrieves a plausible tail pose from a speaker-specific motion database, and refines the clip using this pose before continuing to the next window. During training, Stochastic Anchor Masking randomly masks pose and translation frames, teaching the model to recover complete motion from sparse boundary conditions. A part-aware DiT separates hand, body, and translation streams to reduce interference between global displacement and local articulation. On BEAT2, StreamTalk achieves state-of-the-art FGD, reduces long-horizon drift relative to open-loop baselines, and runs in real time at 76 FPS. Project page: https://xiangyue-zhang.github.io/StreamTalk/.

Figures

Figures reproduced from arXiv: 2608.01643 by the authors.

Figure 1
Figure 1. (a) Existing open-loop streaming methods generate each clip conditioned only on past context. Without forward constraints, small errors accumulate across clips, causing distributional drift—motion that progressively departs from the natural pose distribution. (b) StreamTalk introduces closed-loop streaming generation: our Stream￾ing Pose-Guided Generation (SPG) module retrieves a plausible key pose from a motion dat… view at source ↗
Figure 2
Figure 2. Architecture of StreamTalk. (a) Training. SAM independently masks ran￾dom pose and translation frames from ground-truth motion to form sparse Pose and Trans prompts. The noised sequence Xt is split into hands Ht, body Bt, and transla￾tion Tt and processed by a Part-aware DiT with three branches; outputs are fused to predict Xˆ 1. This teaches the model to inpaint complete motion from partial bound￾ary conditions. (b… view at source ↗
Figure 3
Figure 3. Open-loop vs. closed-loop inference. (a) Open-loop errors accumulate into drift. (b) SPG closes the loop with a retrieved destination anchor. Algorithm 1: Closed-Loop Inference with SPG Input: Audio a, pose prompt Pprompt, step size N, condition r Output: Generated motion x1 x0 ∼ N (0, I); xt ← x0; h ← 1/N; stages ← [x0] 1. Generate (forward integration): for i = 0 to N do x1 ← fθ(xt, i·h, Pprompt, a, r) xt ← x0 (1−… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: User study results. StreamTalk ranks first across all four perceptual cri￾teria [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Qualitative comparison on BEAT2 [22]. Each column shows gestures aligned with the same speech segment. StreamTalk (bottom) produces smoother, more expressive, and rhythm-aligned gestures with consistent arm spacing, while baselines exhibit motion freezing, desynchroniz…
Figure 6
Figure 6. Figure 6: Effect of key pose count. Using 8 or 4 anchors introduces visible jitter (red), while a single key pose produces smoother motion. Long-Horizon Stability. The closed-loop advantage becomes more pro￾nounced over longer sequences [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Long-horizon stability and refinement analysis. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: SPG visual analysis. (a) Without SPG, motion shows spatial drift and abrupt corrections; with SPG, trajectories remain smooth. (b) Self-intersection artifacts are dramatically reduced by SPG. Beyond distributional metrics, we examine physical plausibility via self￾inte…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 52 canonical work pages

  1. [1]

    ACM Transactions on Graphics (TOG)42(4), 1–20 (2023)

    Alexanderson,S.,Nagy,R.,Beskow,J.,Henter,G.E.:Listen,denoise,action!audio- driven motion synthesis with diffusion models. ACM Transactions on Graphics (TOG)42(4), 1–20 (2023)

    work page 2023

  2. [2]

    ACM Transactions on Graphics (TOG)42(4), 1–18 (2023)

    Ao, T., Zhang, Z., Liu, L.: Gesturediffuclip: Gesture diffusion model with clip latents. ACM Transactions on Graphics (TOG)42(4), 1–18 (2023)

    work page 2023

  3. [3]

    In: 2022 International Conference on 3D Vision (3DV)

    Athanasiou, N., Petrovich, M., Black, M.J., Varol, G.: Teach: Temporal action composition for 3d humans. In: 2022 International Conference on 3D Vision (3DV). pp. 414–423. IEEE (2022)

    2022

  4. [4]

    In: Proceedings of the 32nd ACM International Conference on Multimedia

    Chen, B., Li, Y., Ding, Y.X., Shao, T., Zhou, K.: Enabling synergistic full-body control in prompt-based co-speech motion generation. In: Proceedings of the 32nd ACM International Conference on Multimedia. pp. 6774–6783 (2024)

    work page 2024

  5. [5]

    arXiv preprint arXiv:2601.21904 (2026)

    Chen, H., Lyu, G., Xu, C., Yan, J., Yang, X., Deng, C.: Beyond global align- ment: Fine-grained motion-language retrieval via pyramidal shapley-taylor learn- ing. arXiv preprint arXiv:2601.21904 (2026)

    work page arXiv 2026

  6. [6]

    In: Proceedings of the 33rd ACM International Conference on Multimedia

    Chen, H., Xu, C., Yan, J., Deng, C.: Astf: Motion style transfer via adaptive statistics fusor. In: Proceedings of the 33rd ACM International Conference on Multimedia. pp. 5557–5566 (2025)

    work page 2025

  7. [7]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Chen, J., Liu, Y., Wang, J., Zeng, A., Li, Y., Chen, Q.: Diffsheg: A diffusion-based approach for real-time speech-driven holistic 3d expression and gesture generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7352–7361 (2024) 16 X. Zhang et al

    work page 2024

  8. [8]

    IEEE Journal of Selected Topics in Signal Processing16(6), 1505–1518 (2022)

    Chen,S.,Wang,C.,Chen,Z.,Wu,Y.,Liu,S.,Chen,Z.,Li,J.,Kanda,N.,Yoshioka, T., Xiao, X., et al.: Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing16(6), 1505–1518 (2022)

    work page 2022

Show all 72 references
  1. [9]

    Cheng, S., Zhang, J., Song, Q., Liu, S., Guo, Z., Zhang, X., Zhang, C., Li, X., Tu, Z.: Unison: Harmonizing motion, speech, and sound for human-centric audio-video generation (2026),https://arxiv.org/abs/2605.08729

    2026 arXiv

  2. [10]

    IEEE Transactions on Circuits and Systems for Video Technology36(4), 4550–4564 (2025).https: //doi.org/10.1109/TCSVT.2025.3632359

    Cheng, S., Zhang, J., Liu, Y., Xiao, A., Tu, Z.: Owlsight: A robust illumination adaptation framework for dark video human action recognition. IEEE Transactions on Circuits and Systems for Video Technology36(4), 4550–4564 (2025).https: //doi.org/10.1109/TCSVT.2025.3632359

    2025

  3. [11]

    arXiv preprint arXiv:2503.13229 (2025)

    Cheng, Y., Huang, S.: Hologest: Decoupled diffusion and motion priors for gen- erating holisticly expressive co-speech gestures. arXiv preprint arXiv:2503.13229 (2025)

    2025 arXiv

  4. [12]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Chhatre, K., Athanasiou, N., Becherini, G., Peters, C., Black, M.J., Bolkart, T., etal.:Emotionalspeech-driven3dbodyanimationviadisentangledlatentdiffusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1942–1953 (2024)

    2024

  5. [13]

    In: Proceedings of the 21st ACM International Conference on Intelligent Virtual Agents

    Habibie, I., Xu, W., Mehta, D., Liu, L., Seidel, H.P., Pons-Moll, G., Elgharib, M., Theobalt, C.: Learning speech-driven 3d conversational gestures from video. In: Proceedings of the 21st ACM International Conference on Intelligent Virtual Agents. pp. 101–108 (2021)

    2021

  6. [14]

    ACM SIGGRAPH Computer Graphics21(4), 215–224 (1987)

    Isaacs, P.M., Cohen, M.F.: Controlling dynamic simulation with kinematic con- straints. ACM SIGGRAPH Computer Graphics21(4), 215–224 (1987)

    1987

  7. [15]

    In: Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition

    Lee, D., Kim, C., Kim, S., Cho, M., Han, W.S.: Autoregressive image generation using residual quantization. In: Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition. pp. 11523–11532 (2022)

    2022

  8. [16]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Li, J., Kang, D., Pei, W., Zhe, X., Zhang, Y., He, Z., Bao, L.: Audio2gestures: Generating diverse gestures from speech audio with conditional variational autoen- coders. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11293–11302 (2021)

    2021

  9. [17]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Li, R., Zhang, Y., Zhang, Y., Zhang, H., Guo, J., Zhang, Y., Liu, Y., Li, X.: Lodge: A coarse to fine diffusion network for long dance generation guided by the characteristic dance primitives. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognitio...

    2024

  10. [18]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Li, R., Yang, S., Ross, D.A., Kanazawa, A.: Ai choreographer: Music conditioned 3d dance generation with aist++. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 13401–13412 (2021)

    2021

  11. [19]

    arXiv preprint arXiv:1707.05363 (2017)

    Li, Z., Zhou, Y., Xiao, S., He, C., Huang, Z., Li, H.: Auto-conditioned recur- rent networks for extended complex human motion synthesis. arXiv preprint arXiv:1707.05363 (2017)

    2017 arXiv

  12. [20]

    arXiv preprint arXiv:2210.02747 (2022)

    Lipman, Y., Chen, R.T., Ben-Hamu, H., Nickel, M., Le, M.: Flow matching for generative modeling. arXiv preprint arXiv:2210.02747 (2022)

    2022 arXiv

  13. [21]

    Liu, H., Iwamoto, N., Zhu, Z., Li, Z., Zhou, Y., Bozkurt, E., Zheng, B.: Disco: Disentangled implicit content and rhythm learning for diverse co-speech gestures synthesis.In:Proceedingsofthe30thACMinternationalconferenceonmultimedia. pp. 3764–3773 (2022)

    2022

  14. [22]

    In: Proceedings of the StreamTalk 17 IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Liu, H., Zhu, Z., Becherini, G., Peng, Y., Su, M., Zhou, Y., Zhe, X., Iwamoto, N., Zheng, B., Black, M.J.: Emage: Towards unified holistic co-speech gesture generation via expressive masked audio gesture modeling. In: Proceedings of the StreamTalk 17 IEEE/CVF Conference on Com...

    2024

  15. [23]

    In: European conference on computer vision

    Liu, H., Zhu, Z., Iwamoto, N., Peng, Y., Li, Z., Zhou, Y., Bozkurt, E., Zheng, B.: Beat: A large-scale semantic and emotional multi-modal dataset for conversa- tional gestures synthesis. In: European conference on computer vision. pp. 612–630. Springer (2022)

    2022

  16. [24]

    In: Proceed- ings of the IEEE/CVF International Conference on Computer Vision

    Liu, L., Ghaleb, E., Ozyurek, A., Yumak, Z.: Semges: Semantics-aware co-speech gesture generation using semantic coherence and relevance learning. In: Proceed- ings of the IEEE/CVF International Conference on Computer Vision. pp. 13963– 13973 (2025)

    2025

  17. [25]

    arXiv preprint arXiv:2501.18898 (2025)

    Liu, P., Song, L., Huang, J., Liu, H., Xu, C.: Gesturelsm: Latent shortcut based co-speech gesture generation with spatial-temporal modeling. arXiv preprint arXiv:2501.18898 (2025)

    2025 arXiv

  18. [26]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Liu, Y., Cao, Q., Wen, Y., Jiang, H., Ding, C.: Towards variable and coordinated holistic co-speech motion generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1566–1576 (2024)

    2024

  19. [27]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Mughal, M.H., Dabral, R., Scholman, M.C., Demberg, V., Theobalt, C.: Retrieving semantics from the deep: an rag solution for gesture synthesis. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 16578–16588 (2025)

    2025

  20. [28]

    In: IEEE Conference on Computer Vision and Pattern Recognition (2024)

    Ng, E., Romero, J., Bagautdinov, T., Bai, S., Darrell, T., Kanazawa, A., Richard, A.: From audio to photoreal embodiment: Synthesizing humans in conversations. In: IEEE Conference on Computer Vision and Pattern Recognition (2024)

    2024

  21. [29]

    arXiv preprint arXiv:2602.18432 (2026)

    Ng, E., Zhang, S., Chen, Z., Zollhoefer, M., Richard, A.: Sarah: Spatially aware real-time agentic humans. arXiv preprint arXiv:2602.18432 (2026)

    2026

  22. [30]

    IEEE Transactions on Visualization and Computer Graphics (2024)

    Pan, Y., Tan, S., Cheng, S., Lin, Q., Zeng, Z., Mitchell, K.: Expressive talking avatars. IEEE Transactions on Visualization and Computer Graphics (2024)

    2024

  23. [31]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Pavlakos, G., Choutas, V., Ghorbani, N., Bolkart, T., Osman, A.A., Tzionas, D., Black, M.J.: Expressive body capture: 3d hands, face, and body from a single im- age. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10975–10985 (2019)

    2019

  24. [32]

    In: Proceedings of the AAAI conference on artificial intelligence

    Perez, E., Strub, F., De Vries, H., Dumoulin, V., Courville, A.: Film: Visual rea- soning with a general conditioning layer. In: Proceedings of the AAAI conference on artificial intelligence. vol. 32 (2018)

    2018

  25. [33]

    arXiv preprint arXiv:2303.01418 (2023)

    Shafir, Y., Tevet, G., Kapon, R., Bermano, A.H.: Human motion diffusion as a generative prior. arXiv preprint arXiv:2303.01418 (2023)

    2023 arXiv

  26. [34]

    Song, Q., He, Y., Zhang, Y., Cheng, S., He, Z., Guo, Z., Zhang, C., Li, X., Jiang, C.: Interactiveavatar: Real-time streaming video generation for consistent and intent- aware avatars (2026),https://arxiv.org/abs/2606.22905

    2026 arXiv

  27. [35]

    Advances in Neural Information Processing Systems38, 10947–10969 (2026)

    Tang, X., Peng, Z., Hu, Y., He, J., Liu, H., et al.: Megadance: Mixture-of-experts architecture for genre-aware 3d dance generation. Advances in Neural Information Processing Systems38, 10947–10969 (2026)

    2026

  28. [36]

    Advances in neural information processing systems30(2017)

    Van Den Oord, A., Vinyals, O., et al.: Neural discrete representation learning. Advances in neural information processing systems30(2017)

    2017

  29. [37]

    arXiv preprint arXiv:2209.13204 (2022)

    Wang, W., Zhe, X., Ke, Q., Kang, D., Li, T., Chen, R., Bao, L.: Neural marionette: A transformer-based multi-action human motion synthesis system. arXiv preprint arXiv:2209.13204 (2022)

    2022 arXiv

  30. [38]

    IEEE Transactions on Image Processing34, 7079–7092 (2025) 18 X

    Wang, X., Kang, Z., Mu, Y.: Text-controlled motion mamba: Text-instructed tem- poral grounding of human motion. IEEE Transactions on Image Processing34, 7079–7092 (2025) 18 X. Zhang et al

    2025

  31. [39]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Wang, X., Xu, K., Li, F., Sheng, C., Yu, J., Mu, Y.: Generating attribute-aware human motions from textual prompt. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 40, pp. 10216–10224 (2026)

    2026

  32. [40]

    In: 2023 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR)

    Wang, X., Xu, X., Mu, Y.: Neural koopman pooling: Control-inspired temporal dy- namics encoding for skeleton-based action recognition. In: 2023 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR). pp. 10597–10607. IEEE (2023)

    2023

  33. [41]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Xiao, A., Cheng, S., Xu, Y., Ren, Y., Chen, H., Yokoya, N.: Geommbench and geommagent: Towardexpert-levelmultimodal intelligence ingeoscienceand remote sensing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 34843–34853 (2026)

    2026

  34. [42]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Xiao, L., Lu, S., Pi, H., Fan, K., Pan, L., Zhou, Y., Feng, Z., Zhou, X., Peng, S., Wang, J.: Motionstreamer: Streaming motion generation via diffusion-based autoregressive model in causal latent space. In: Proceedings of the IEEE/CVF International Conference on Computer Visio...

    2025

  35. [43]

    Advances in Neural Information Processing Systems37, 20055–20080 (2024)

    Xu, Z., Lin, Y., Han, H., Yang, S., Li, R., Zhang, Y., Li, X.: Mambatalk: Efficient holistic gesture synthesis with selective state space models. Advances in Neural Information Processing Systems37, 20055–20080 (2024)

    2024

  36. [44]

    In: Proceedings of the 2024 International Conference on Multimedia Retrieval

    Yang, K., Tang, X., Diao, R., Liu, H., He, J., Fan, Z.: Codancers: Music-driven coherent group dance generation with choreographic unit. In: Proceedings of the 2024 International Conference on Multimedia Retrieval. pp. 675–683 (2024)

    2024

  37. [45]

    arXiv preprint arXiv:2511.21029 (2025)

    Yang, K., Tang, X., Peng, Z., Zhang, X., Wang, P., He, J., Liu, H.: Flower- dance: Meanflow for efficient and refined 3d dance generation. arXiv preprint arXiv:2511.21029 (2025)

    2025 arXiv

  38. [46]

    In: Proceedings of the 33rd ACM International Conference on Multimedia

    Yang, K., Tang, X., Wu, H., Qin, B., Liu, H., He, J., Fan, Z.: Cohedancers: Enhanc- ing interactive group dance generation through music-driven coherence decompo- sition. In: Proceedings of the 33rd ACM International Conference on Multimedia. pp. 6663–6671 (2025)

    2025

  39. [47]

    In: Proceedings of the 2024 International Conference on Multimedia Retrieval

    Yang, K., Zhou, X., Tang, X., Diao, R., Liu, H., He, J., Fan, Z.: Beatdance: A beat- based model-agnostic contrastive learning framework for music-dance retrieval. In: Proceedings of the 2024 International Conference on Multimedia Retrieval. pp. 11–19 (2024)

    2024

  40. [48]

    arXiv preprint arXiv:2606.30019 (2026)

    Yang, K., Zhu, J., Tang, X., Peng, Z., Zhang, X., Chen, C., Wang, P., Wu, J., Chu, X., Liu, H., et al.: Omnidance: Multimodal driven dance video generation with large-scale internet data. arXiv preprint arXiv:2606.30019 (2026)

    2026 arXiv

  41. [49]

    In: Proceedings of the Special Interest Group on Com- puter Graphics andInteractive TechniquesConference Conference Papers.pp

    Yang, K., Zhu, J., Tang, X., Peng, Z., Zhang, X., Wang, P., Wu, J., Chu, X., Liu, H., He, J.: Mace-dance: Motion-appearance cascaded experts for music-driven dance video generation. In: Proceedings of the Special Interest Group on Com- puter Graphics andInteractive TechniquesC...

    2026

  42. [50]

    arXiv e-prints pp

    Yang, S., Wu, Z., Li, M., Zhang, Z., Hao, L., Bao, W., Cheng, M., Xiao, L.: Dif- fusestylegesture: Stylized audio-driven co-speech gesture generation with diffusion models. arXiv e-prints pp. arXiv–2305 (2023)

    2023

  43. [51]

    In: Proceedings of the 31st ACM International Conference on Multimedia

    Yang, Z., Su, B., Wen, J.R.: Synthesizing long-term human motions with diffu- sion models via coherent sampling. In: Proceedings of the 31st ACM International Conference on Multimedia. pp. 3954–3964 (2023)

    2023

  44. [52]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Yi, H., Liang, H., Liu, Y., Cao, Q., Wen, Y., Bolkart, T., Tao, D., Black, M.J.: Gen- erating holistic 3d human motion from speech. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 469–480 (2023) StreamTalk 19

    2023

  45. [53]

    In: The Thirty-ninth Annual Con- ference on Neural Information Processing Systems (2025)

    Yin, Z., Tsui, Y.H., Hui, P.: Pyramotion: Attentional pyramid-structured motion integration for co-speech 3d gesture synthesis. In: The Thirty-ninth Annual Con- ference on Neural Information Processing Systems (2025)

    2025

  46. [54]

    ACM Transactions on Graphics (TOG)39(6), 1–16 (2020)

    Yoon, Y., Cha, B., Lee, J.H., Jang, M., Lee, J., Kim, J., Lee, G.: Speech gesture generation from the trimodal context of text, audio, and speaker identity. ACM Transactions on Graphics (TOG)39(6), 1–16 (2020)

    2020

  47. [55]

    IEEE Transactions on Visualization and Computer Graphics30(10), 6984–6996 (2024)

    Zhang, F., Wang, Z., Lyu, X., Zhao, S., Li, M., Geng, W., Ji, N., Du, H., Gao, F., Wu, H., et al.: Speech-driven personalized gesture synthetics: Harnessing auto- matic fuzzy feature inference. IEEE Transactions on Visualization and Computer Graphics30(10), 6984–6996 (2024)

    2024

  48. [56]

    In: International Conference on Learning Representations (2024)

    Zhang, J., Huang, S., Tu, Z., Chen, X., Zhan, X., Yu, G., Shan, Y.: Tapmo: Shape- aware motion generation of skeleton-free characters. In: International Conference on Learning Representations (2024)

    2024

  49. [57]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Zhang, J., Weng, J., Kang, D., Zhao, F., Huang, S., Zhe, X., Bao, L., Shan, Y., Wang, J., Tu, Z.: Skinned motion retargeting with residual perception of motion se- mantics and geometry. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. ...

    2023

  50. [58]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Zhang, M., Guo, X., Pan, L., Cai, Z., Hong, F., Li, H., Yang, L., Liu, Z.: Re- modiffuse: Retrieval-augmented motion diffusion model. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 364–373 (2023)

    2023

  51. [59]

    arXiv preprint arXiv:2605.06064 (2026)

    Zhang, X., Cai, Y., Li, K., Yang, K., Zhou, Y., Li, Z., Chu, X., Zhang, J., Liu, H.: Personagesture: Single-reference co-speech gesture personalization for unseen speakers. arXiv preprint arXiv:2605.06064 (2026)

    2026 arXiv

  52. [60]

    IEEE Transactions on Circuits and Systems for Video Technology35(10), 10410–10422 (2025)

    Zhang, X., Jia, Y., Zhang, J., Yang, Y., Tu, Z.: Robust 2d skeleton action recogni- tion via decoupling and distilling 3d latent features. IEEE Transactions on Circuits and Systems for Video Technology35(10), 10410–10422 (2025)

    2025

  53. [61]

    arXiv preprint arXiv:2511.10076 (2025)

    Zhang, X., Li, J., Ren, J., Zhang, J.: Mitigating error accumulation in co-speech motion generation via global rotation diffusion and multi-level constraints. arXiv preprint arXiv:2511.10076 (2025)

    2025

  54. [62]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Zhang, X., Li, J., Zhang, J., Dang, Z., Ren, J., Bo, L., Tu, Z.: Semtalk: Holistic co-speech motion generation with frame-level semantic emphasis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 13761–13771 (2025)

    2025

  55. [63]

    In: Pro- ceedings of the 33rd ACM International Conference on Multimedia

    Zhang, X., Li, J., Zhang, J., Ren, J., Bo, L., Tu, Z.: Echomask: Speech-queried attention-based mask modeling for holistic co-speech motion generation. In: Pro- ceedings of the 33rd ACM International Conference on Multimedia. pp. 10827– 10836 (2025)

    2025

  56. [64]

    ACMTransactionson Graphics (TOG)43(4), 1–17 (2024)

    Zhang, Z., Ao, T., Zhang, Y., Gao, Q., Lin, C., Chen, B., Liu, L.: Semantic gesticu- lator: Semantics-aware co-speechgesture synthesis. ACMTransactionson Graphics (TOG)43(4), 1–17 (2024)

    2024

  57. [65]

    arXiv preprint arXiv:2407.10061 (2024)

    Zhang, Z., Liu, A., Chen, Q., Chen, F., Reid, I., Hartley, R., Zhuang, B., Tang, H.: Infinimotion: Mamba boosts memory in transformer for arbitrary long motion generation. arXiv preprint arXiv:2407.10061 (2024)

    2024 arXiv

  58. [66]

    IEEE Transactions on Mul- timedia (2025)

    Zhang, Z., Cai, W., Liu, Q., Wang, Y.: Skeletonx: Data-efficient skeleton-based ac- tion recognition via cross-sample feature aggregation. IEEE Transactions on Mul- timedia (2025)

    2025

  59. [67]

    arXiv preprint arXiv:2605.27055 (2026)

    Zhang, Z., Cui, Y., Liu, Q., Wang, Y.: Semantic-aware motion encoding for topology-agnostic character animation. arXiv preprint arXiv:2605.27055 (2026)

    2026 arXiv

  60. [68]

    In: Proceedings of the 33rd ACM International Conference on Multimedia

    Zhang, Z., Kong, B., Liu, Q., Wang, Y.: Towards robust and controllable text- to-motion via masked autoregressive diffusion. In: Proceedings of the 33rd ACM International Conference on Multimedia. pp. 9326–9335 (2025) 20 X. Zhang et al

    2025

  61. [69]

    In: ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing

    Zhang, Z., Zhou, H., Liu, Q., Wang, Y.: Skeletonmix: A mixup-based data aug- mentation framework for skeleton-based action recognition. In: ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing. pp. 1–5. IEEE (2025)

    2025

  62. [70]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Zhi, Y., Cun, X., Chen, X., Shen, X., Guo, W., Huang, S., Gao, S.: Livelyspeaker: Towards semantic-aware co-speech gesture generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 20807–20817 (2023)

    2023

  63. [71]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Zhou, Y., Barnes, C., Lu, J., Yang, J., Li, H.: On the continuity of rotation rep- resentations in neural networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5745–5753 (2019)

    2019

  64. [72]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Zhuo, W., Ma, F., Fan, H.: Infinidreamer: Arbitrarily long human motion genera- tion via segment score distillation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 14688–14698 (2025)

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.