Pith. sign in

REVIEW 3 major objections 7 minor 56 references

Multi-Modal Motion Retrieval by Learning a Fine-Grained Joint Embedding Space

T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that motion retrieval improves when text, video, audio, and motion are aligned at the token level in a shared embedding space, reporting state-of-the-art gains of 10.16% in text-to-motion R@10 and 25.43% in…

desk verdict First audio-to-motion retrieval setup with useful synthetic datasets, but the ablation evidence for the central mechanism is internally inconsistent and the audio evaluation is partly circular; worth refereeing after fixes. read the letter →

arxiv 2507.23188 v1 pith:NSA27YZL submitted 2025-07-31 cs.CV

Shiyao Yu , Zi-An Wang , Kangning Yin , Zheng Tian , Mingyuan Zhang , Weixin Si , Shihao Zou This is my paper
classification cs.CV
keywords motionretrievalmulti-modalfine-grainedcontrastivelearningjointembeddingspacesequence-levelalignmentaudio-to-motionbody-partencodingtext-to-motion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that motion retrieval is better served by aligning text, video, audio, and motion as sequences of tokens than by compressing each input into a single global vector. The authors add audio as a fourth retrieval modality for the first time, and they report state-of-the-art results on HumanML3D and KIT-ML, including a 10.16% gain in text-to-motion R@10 and a 25.43% gain in video-to-motion R@1 over prior multi-modal methods. The claimed mechanism is a fine-grained contrastive loss in which every token from one modality is scored against its best matching token in the other modality, plus a motion encoder that decomposes poses into eight body parts. If the framework is right, fine-grained alignment and body-part decomposition are what drive the improvement, and spoken audio becomes a practical query modality alongside text and video.

What carries the argument

The load-bearing object is the sequence-level similarity function $h(e_x, e_y)$ of Eq. 8: for every token in modality $x$ it takes the maximum dot product with tokens in modality $y$, does the same in the reverse direction, and combines the two weighted sums, with per-token weights learned by a softmax linear layer. This replaces the global-token similarity used by prior work and is what lets phrases like "turns around" match the few motion frames that actually contain the turn. Around it sit four encoders: a body-part motion encoder that splits each pose into eight parts before temporal and spatial transformer stages; DistilBERT plus a transformer for text; CLIP's image encoder plus a temporal transformer for video; and WavLM plus a memory-retrieval attention module that turns variable-length speech into compact tokens. All six modality pairs are aligned with bidirectional KL contrastive losses, and a reconstruction loss masks and recovers motion tokens with help from the other modalities.

What would settle it

Record a large set of real human speakers describing motions from the HumanML3D test set, run audio-to-motion retrieval with the released model, and compare against the synthetic-audio results; if real-speech R@1 falls toward chance while text-motion R@1 stays at roughly 9.4, the generalization claim fails. Independently, an audit that measures whether ChatGPT rewrites change the executed motion (for example, counting how often "fold arms" becomes "place hands on shoulders") would falsify the assumption that the oral dataset inherits the original text-motion labels.

Watch

Extended reading notes

Core claim

The central claim, stated the way the paper argues it, is that sequence-level maximum-similarity alignment creates a joint embedding space precise enough for cross-modal motion retrieval, and that this precision is what produces the reported gains. Instead of pooling a text, video, or audio input into one token, the paper keeps the token stream and computes, for each token in one modality, its maximum dot-product similarity to tokens in the other modality, then averages these weighted maxima (Eq. 8). The motion encoder supports this by decomposing each pose into eight body parts, encoding them with alternating temporal and spatial transformers, and reshaping the result into fine-grained motion tokens. Audio is encoded with WavLM followed by a memory-retrieval attention module that compresses variable-length speech into fixed-length token sequences. The paper reports that this four-modal model beats prior two- and three-modal methods on text-motion and video-motion retrieval on both datasets, and that ablations removing body-part decomposition or replacing max-token alignment with global alignment substantially lower performance.

Load-bearing premise

The load-bearing premise is that the synthetic audio inherits the text-motion ground truth: speech synthesized from ChatGPT-rewritten text is assumed to preserve motion-relevant meaning closely enough that audio-motion pairs are valid training signal, and the only check is a two-speaker qualitative test.

Editorial extensions

If this is right

  • On HumanML3D, the four-modal model improves text-to-motion R@10 from 33.67 to 43.83 and motion-to-text R@10 from 36.55 to 43.74 compared with the prior three-modal method (Tab. II).
  • Video-to-motion retrieval jumps from 39.35 to 64.78 R@1 on HumanML3D, and the four-modal version beats the three-modal version, showing audio contributes beyond its own retrieval task (Tab. II).
  • Audio-to-motion retrieval reaches 53.20 R@10 on HumanML3D, making audio a practical alternative query modality with performance comparable to text-motion retrieval (Tab. II).
  • Ablations attribute most of the gain to body-part decomposition and to max-based sequence-level alignment; removing the alignment losses collapses R@1 to near zero (Tab. V).
  • Models trained on conversational "Oral" audio instructions substantially outperform models trained on original-style speech, indicating that speaking style matters for audio-motion retrieval (Tab. IV).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If weighted max-token alignment, rather than the larger model or extra data, is the causal driver, then transplanting the same similarity function into existing global-pooled retrieval models for image-text, video-text, or audio-text should produce similar gains; this is directly testable.
  • The synthetic-audio evaluation leaves open how well the results transfer to real speech: the paper's real-speech evidence is two speakers shown in two figures, so a multi-speaker, multi-accent, noisy-audio benchmark would settle whether audio-to-motion retrieval truly works in the wild.
  • The ChatGPT rewrites sometimes change the described motion (for example, "folding arms" becomes "placing hands on shoulders then by the side"), so the oral dataset may not share the original text-motion ground truth exactly; retrieval performance on it partly measures the model's ability to exploit the rewrite distribution.
  • A four-modal model may also gain from having more contrastive pairs and a reconstruction regularizer, so ablations that keep token length and model capacity fixed while varying modality count would separate alignment benefit from data or capacity effects.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes a four-modality motion retrieval framework that aligns text, audio, video, and motion in a joint embedding space. The main technical ingredients are a sequence-level contrastive similarity based on token-level maximum scores (Eq. 8), a body-part decomposition for the motion encoder, a memory-retrieval-based audio compression module built on WavLM, and an auxiliary motion reconstruction loss. Because no audio-motion retrieval dataset exists, the authors augment HumanML3D and KIT-ML with synthetic audio generated by Tortoise from both original text annotations and ChatGPT-rewritten conversational versions. The paper reports state-of-the-art results on text-to-motion and video-to-motion retrieval, plus audio-to-motion results on the new synthetic benchmarks, and claims that the 4-modal model outperforms its 3-modal counterpart.

Significance. If the results hold, the paper would make several useful contributions: it introduces audio as a retrieval modality for human motion, demonstrates that token-level max alignment can improve over global alignment, and provides two augmented datasets with synthetic audio instructions. The body-partition ablation (Table V: 43.83 vs. 37.51 R@10) and the alignment-loss ablation (Table V: collapse to near-chance) are clean and support those specific components. The paper also includes a failure-case analysis, efficiency measurements, and a qualitative real-audio generalization test, which are valuable. However, the significance is currently limited by unresolved internal inconsistencies in the ablation tables and by the fact that the audio benchmark is built from TTS renderings of the very text annotations that define the retrieval ground truth.

major comments (3)
  1. [Tab. V vs. Tab. VIII/VI] Table V reports 'Ours w/o Sequence-level Alignment' as 9.15 R@1 / 42.82 R@10 for text-to-motion on HumanML3D, and Section IV.C defines this condition as replacing sequence-level alignment with global-token-based alignment. Table VIII's 'Global Only' row and Table VI's 1-token row both report 6.37 R@1 / 33.67 R@10 for what appears to be the same configuration. Two different numbers cannot both describe the same ablation under the same data and modality settings. Because this row is the key evidence for the paper's central claim that sequence-level max alignment drives the gains, the correct value must be established before the claim can be accepted.
  2. [Tab. VIII] The row 'Global + Sequence-level Alignment (Max)' reports 9.70 R@1 / 44.29 R@10 for text-to-motion, while the row 'Sequence-level Alignment (Max)' reports 8.78 / 40.69, which exactly matches Ours(3-modal) in Table II. This indicates that Table VIII is reporting 3-modal variants. If so, a 3-modal variant exceeds the final 4-modal model (9.41 / 43.83 in Table V); if Table VIII is instead reporting 4-modal variants, then the final model is not the best configuration found in the paper. Either way, the manuscript's claim that the 4-modal framework significantly outperforms its 3-modal counterpart is not currently supported and must be reconciled with this table.
  3. [Section III.C and Tables II-IV] The audio-motion benchmark is constructed by synthesizing TTS audio from the text annotations that define the retrieval ground truth, with ChatGPT-rewritten variants. Under this construction, audio-to-motion retrieval is effectively text-to-motion retrieval passed through a TTS bottleneck, so the audio retrieval numbers in Tables II-IV cannot by themselves establish that the model understands spoken instructions as an independent modality. The only real-audio evidence is qualitative (Figs. 10-11, two speakers, with one noisy condition). A quantitative real-audio evaluation, or at minimum a validation that the synthetic audio preserves motion-relevant semantics (e.g., ASR round-trip consistency or human judgment on semantic equivalence), is needed to support the headline claim that audio is a usable fourth retrieval modality.
minor comments (7)
  1. [Abstract and Introduction] The phrase 'an 10.16%' should be 'a 10.16%' or preferably '10.16 percentage points' to avoid ambiguity between absolute and relative improvement.
  2. [Eq. (8)] The notation h(e_x, e_y) is used for the sequence-level similarity function, but Eqs. (6)-(7) apply h to token-level arguments; please align the notation to distinguish token-pair similarities from sequence-level similarities.
  3. [References] Reference [55] is cited for the AdamW optimizer, but the citation points to Oord et al.'s contrastive predictive coding paper; the correct reference is Loshchilov and Hutter, 'Decoupled Weight Decay Regularization,' ICLR 2019.
  4. [Table II caption] The caption contains the typo 'PERVIOUS'; it should read 'PREVIOUS'.
  5. [Section IV.C] The sentence 'as shown in, Table V, Table VII, and Table VIII' contains a stray comma after 'in'.
  6. [Implementation Details vs. Efficiency Analysis] Implementation Details states that all experiments use eight NVIDIA A6000 GPUs, while the Efficiency Analysis states that training the 4-modality model uses four NVIDIA A6000 GPUs; please clarify which configuration was used.
  7. [Fig. 4 caption] The caption says 'derived from the augmented Oral Datasets,' but the distribution appears to concern the synthesized audio generally; please clarify whether the histogram covers the Original dataset, the Oral dataset, or both.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the text/video retrieval claims are externally benchmarked, and the audio benchmark, though self-constructed from TTS, is an external-validity caveat rather than a definitional reduction.

full rationale

Walking the claimed derivation chain, the core text-to-motion and video-to-motion results are measured on the standard HumanML3D and KIT-ML splits against external baselines (Tabs. II-III), so those claims do not reduce to the paper's own inputs. The sequence-level similarity in Eq. (8) is a definite computation whose effect is ablated, and the body-part encoder is an architectural choice; neither is defined in terms of the reported metric. The audio-motion benchmark is self-constructed: Section III.C synthesizes audio from the same text annotations that define the text-motion ground truth, so the quantitative audio-retrieval numbers in Tabs. II-IV are best read as text retrieval passed through a TTS/paraphrase bottleneck, and the only real-audio check (Sec. IV.D, Figs. 10-11) is qualitative with two speakers. This is a real external-validity limitation, but the paper explicitly frames audio as an 'alternative semantic representation' rather than an independent information source, and there is no train/test leak, fitted-parameter-renamed-as-prediction, or load-bearing self-citation. The self-citation of the authors' prior LA VIMO [23] is normal incremental-work citation. Separately, the ablation evidence is internally inconsistent: 'Global Only' appears as 42.82 R@10 in Tab. V but 33.67 in Tabs. VI/VIII, and 'Global + Sequence-level (Max)' in Tab. VIII (44.29 R@10) exceeds the reported final 'Ours' (43.83). That is a serious correctness/reproducibility problem for the mechanism claim, but it is not a circular-reasoning step under the rubric. Overall circularity score 2 reflects the minor self-citation and the self-built audio caveat, not a formal circular derivation.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The central claims rest on the standard HumanML3D/KIT-ML splits plus the paper's own design choices: the reconstruction weight, evaluation threshold, token lengths, learnable temperature, and the unstated audio memory-token count are hand-set or fitted, and the audio benchmark assumes TTS fidelity to the text annotations. The max-similarity alignment objective and body-part decomposition are justified only by in-house ablations.

free parameters (5)
  • reconstruction loss weight lambda_recon = 0.1
    Set by hand; balances L_align and L_recon in Eq. 12; no sensitivity analysis reported (Section IV, Implementation Details).
  • evaluation similarity threshold = 0.80
    Hand-chosen cutoff for the 'All with threshold' protocol; directly determines the reported numbers in protocol (b) of Tabs. II-III (Section IV, Evaluation Metrics).
  • contrastive temperature tau = learnable, value not reported
    Learnable temperature in Eqs. 6-7; fitted during training with the final value omitted from the paper.
  • input token lengths (text 32, motion 192, frame cap 196) = 32 tokens text / 192 tokens motion at 20 FPS
    Chosen by hand; Tab. VI shows retrieval accuracy grows with token length, so the reported numbers depend on this choice.
  • audio memory token count (compressed length L_a) = not reported
    Number of learnable memory tokens in the audio module (Eq. 3, Fig. 5) is unspecified, though it sets the audio feature length.
assumptions (5)
  • domain assumption The standard HumanML3D and KIT-ML train/test splits used by prior work are fair and leakage-free.
    All comparisons inherit these splits (Section IV, Datasets); a split artifact would invalidate the SOTA claims.
  • ad hoc to paper Tortoise-synthesized audio from ChatGPT-rewritten text preserves motion-relevant semantics, making synthetic audio a valid proxy for real spoken instructions.
    Load-bearing for every audio result; tested only qualitatively with two speakers (Section III.C, Section IV.D, Figs. 10-11).
  • domain assumption Max-over-token-pairs similarity (Eq. 8) is the right fine-grained alignment objective.
    Justified only by in-house ablations (Tab. VIII); mean-based aggregation performs far worse, a result the paper does not explain.
  • domain assumption Body-part decomposition improves motion representation because humans recognize motion from relative body-part movement.
    Stated in Section III.A.1; supported only by the internal ablation in Tab. V.
  • domain assumption Pretrained CLIP (video frames) and WavLM (audio) features provide adequate semantic grounding for retrieval.
    The encoders are fixed or adapted from pretrained models; any bias in these features transfers to the results (Sections III.A.3-4).
invented entities (1)
  • Learnable memory tokens (key/value pairs) in the audio compression module
    purpose: Compress variable-length WavLM audio features into a fixed-length representation for the joint embedding (Eq. 3, Fig. 5).
    A trainable architectural component validated only on the authors' self-generated audio benchmark; no released code or external benchmark confirms its contribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Modal Motion Retrieval by Learning a Fine-Grained Joint Embedding Space." pith.science (2026). https://pith.science/paper/NSA27YZL

@misc{pith2026250723188,
  author       = {Pith},
  title        = {Pith review of: Multi-Modal Motion Retrieval by Learning a Fine-Grained Joint Embedding Space},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NSA27YZL}},
  note         = {Machine review of arXiv:2507.23188}
}
read the original abstract

Motion retrieval is crucial for motion acquisition, offering superior precision, realism, controllability, and editability compared to motion generation. Existing approaches leverage contrastive learning to construct a unified embedding space for motion retrieval from text or visual modality. However, these methods lack a more intuitive and user-friendly interaction mode and often overlook the sequential representation of most modalities for improved retrieval performance. To address these limitations, we propose a framework that aligns four modalities -- text, audio, video, and motion -- within a fine-grained joint embedding space, incorporating audio for the first time in motion retrieval to enhance user immersion and convenience. This fine-grained space is achieved through a sequence-level contrastive learning approach, which captures critical details across modalities for better alignment. To evaluate our framework, we augment existing text-motion datasets with synthetic but diverse audio recordings, creating two multi-modal motion retrieval datasets. Experimental results demonstrate superior performance over state-of-the-art methods across multiple sub-tasks, including an 10.16% improvement in R@10 for text-to-motion retrieval and a 25.43% improvement in R@1 for video-to-motion retrieval on the HumanML3D dataset. Furthermore, our results show that our 4-modal framework significantly outperforms its 3-modal counterpart, underscoring the potential of multi-modal motion retrieval for advancing motion acquisition.

Figures

Figures reproduced from arXiv: 2507.23188 by the authors.

Figure 1
Figure 1. Overview of Our Work. Our framework encodes text, video, or audio descriptions and computes their similarity within a shared joint embedding space, ranking candidate motions based on similarity scores to retrieve the most relevant motion. Recent advancements in generative modeling have intro￾duced various approaches for synthesizing motion sequences based on multi-modal conditions, such as action [5]–[8], text [9]–[… view at source ↗
Figure 2
Figure 2. Global contrastive learning (Left) computes similarity between two modalities using global representations, where motion and text data are compressed into a single token for cross-modal alignment. In contrast, sequence-level contrastive learning (Right) aligns individual tokens with their most relevant counterparts, enabling the model to focus on key frames in the motion sequence and important keywords in the text d… view at source ↗
Figure 3
Figure 3. Pipeline of Our Work. The text, audio, and video inputs are encoded using pre-trained models to extract feature tokens. Meanwhile, the motion encoder segments human motion by body parts and processes them through a transformer, effectively integrating both body part and temporal information. This design ensures that the alignment process captures both spatial and temporal dependencies. Once inputs from different mod… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4 [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Overview of the Audio Processing. WavLM [25] extracts audio features, which are then processed through a memory-retrieval-based module designed to standardize input audio conditions of significantly varying lengths. By converting all audio signals into a uniform length…
Figure 6
Figure 6. Figure 6: Dataset Augmentation with Audio Modality. The text data from the KIT-ML [9] and HumanML3D [10] datasets are processed using the text-to￾speech model Tortoise [26] to generate audio signals with randomly assigned speaker identities, forming Original Dataset. Additionall…
Figure 7
Figure 7. Figure 7: Qualitative Comparison of Text-to-Motion Retrieval on the HumanML3D Dataset. We compare our results with LAVIMO [23]. In the first row, our model accurately retrieves the motion of kicking the right leg, while the comparison model retrieves a motion involving the left …
Figure 8
Figure 8. Figure 8: Qualitative Comparison of Motion Retrieval Using Text or Audio. The textual descriptions and audio instructions convey the same meanings. Our approach achieves comparable performance with either modality, highlighting the effectiveness of audio signals as a semantic re…
Figure 9
Figure 9. Figure 9: Qualitative Comparison of Models Trained on Original and Oral Dataset. Models trained on the Original dataset struggle to retrieve the correct motions using spoken-style audio signals in Oral dataset, highlighting the importance of our proposed dataset for more convers…
Figure 10
Figure 10. Figure 10: Motion Retrieval Using Real Audio Signals. We collect real audio recordings from two different subjects, each speaking the same content, and use them for motion retrieval. The results demonstrate that our model, trained on synthetic audio, effectively generalizes to r…
Figure 11
Figure 11. Figure 11: Real-world audio- to-motion retrieval results. Our model, trained on synthetic audio, generalizes well to some real-world and noisy audios from unseen speakers, showing robust retrieval performance across varied real-world conditions. A person stands still, then shift…
Figure 12
Figure 12. Figure 12: Failure Case Analysis. The retrieval fails when using a long text sequence for retrieval, likely due to the motion length limit set in our model. However, when we split the sentence into two shorter ones, the retrieved motions are correct, where post-processing can be…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 48 canonical work pages

  1. [1]

    Dual stream relation learning network for image-text retrieval,

    D. Wu, H. Li, C. Gu, L. Guo, and H. Liu, “Dual stream relation learning network for image-text retrieval,” TMM, 2025

    work page 2025

  2. [2]

    One-shot human motion transfer via occlusion-robust flow prediction and neural texturing,

    Y . Ji, C. Zheng, and T.-J. Cham, “One-shot human motion transfer via occlusion-robust flow prediction and neural texturing,” TMM, 2025

    work page 2025

  3. [3]

    Ta2v: Text-audio guided video generation,

    M. Zhao, W. Wang, T. Chen, R. Zhang, and R. Li, “Ta2v: Text-audio guided video generation,” TMM, 2024

    work page 2024

  4. [4]

    Cross-modal quantization for co-speech gesture generation,

    Z. Wang, W. Zhang, L. Ye, D. Zeng, and T. Mei, “Cross-modal quantization for co-speech gesture generation,” TMM, 2024

    work page 2024

  5. [5]

    Generative adversarial graph convolutional networks for human action synthesis,

    B. Degardin, J. Neves, V . Lopes, J. Brito, E. Yaghoubi, and H. Proenc ¸a, “Generative adversarial graph convolutional networks for human action synthesis,” in WACV, 2022

    work page 2022

  6. [6]

    Action-conditioned 3d human motion synthesis with transformer vae,

    M. Petrovich, M. J. Black, and G. Varol, “Action-conditioned 3d human motion synthesis with transformer vae,” in ICCV, 2021

    work page 2021

  7. [7]

    Multiact: Long-term 3d human motion generation from multiple action labels,

    T. Lee, G. Moon, and K. M. Lee, “Multiact: Long-term 3d human motion generation from multiple action labels,” in AAAI, 2023

    work page 2023

  8. [8]

    Executing your commands via motion diffusion in latent space,

    X. Chen, B. Jiang, W. Liu, Z. Huang, B. Fu, T. Chen, and G. Yu, “Executing your commands via motion diffusion in latent space,” in CVPR, 2023

    work page 2023

Show all 56 references
  1. [9]

    The kit motion-language dataset,

    M. Plappert, C. Mandery, and T. Asfour, “The kit motion-language dataset,” Big data , 2016

    2016

  2. [10]

    Generating diverse and natural 3d human motions from text,

    C. Guo, S. Zou, X. Zuo, S. Wang, W. Ji, X. Li, and L. Cheng, “Generating diverse and natural 3d human motions from text,” in CVPR, 2022

    2022

  3. [11]

    Human motion diffusion model,

    G. Tevet, S. Raab, B. Gordon, Y . Shafir, D. Cohen-or, and A. H. Bermano, “Human motion diffusion model,” in ICLR, 2023

    2023

  4. [12]

    Generating human motion from textual descriptions with discrete representations,

    J. Zhang, Y . Zhang, X. Cun, Y . Zhang, H. Zhao, H. Lu, X. Shen, and Y . Shan, “Generating human motion from textual descriptions with discrete representations,” in CVPR, 2023

    2023

  5. [13]

    Groupdancer: Music to multi-people dance synthesis with style collaboration,

    Z. Wang, J. Jia, H. Wu, J. Xing, J. Cai, F. Meng, G. Chen, and Y . Wang, “Groupdancer: Music to multi-people dance synthesis with style collaboration,” in ACM MM, 2022

    2022

  6. [14]

    Music- driven group choreography,

    N. Le, T. Pham, T. Do, E. Tjiputra, Q. D. Tran, and A. Nguyen, “Music- driven group choreography,” in CVPR, 2023

    2023

  7. [15]

    Edge: Editable dance generation from music,

    J. Tseng, R. Castellon, and K. Liu, “Edge: Editable dance generation from music,” in CVPR, 2023

    2023

  8. [16]

    Pc-dance: Posture- controllable music-driven dance synthesis,

    J. Gao, J. Pu, H. Zhang, Y . Shan, and W.-S. Zheng, “Pc-dance: Posture- controllable music-driven dance synthesis,” in ACM MM, 2022

    2022

  9. [17]

    Couch: Towards controllable human-chair interactions,

    X. Zhang, B. L. Bhatnagar, S. Starke, V . Guzov, and G. Pons-Moll, “Couch: Towards controllable human-chair interactions,” in ECCV, 2022

    2022

  10. [18]

    Goal: Generating 4d whole-body motion for hand-object grasping,

    O. Taheri, V . Choutas, M. J. Black, and D. Tzionas, “Goal: Generating 4d whole-body motion for hand-object grasping,” in CVPR, 2022

    2022

  11. [19]

    Human motion generation: A survey,

    W. Zhu, X. Ma, D. Ro, H. Ci, J. Zhang, J. Shi, F. Gao, Q. Tian, and Y . Wang, “Human motion generation: A survey,” TPAMI, 2023

    2023

  12. [20]

    Phase-functioned neural networks for character control,

    D. Holden, T. Komura, and J. Saito, “Phase-functioned neural networks for character control,” ACM TOG, 2017

    2017

  13. [21]

    Learned motion matching,

    D. Holden, O. Kanoun, M. Perepichka, and T. Popa, “Learned motion matching,” ACM TOG, 2020

    2020

  14. [22]

    Tmr: Text-to-motion retrieval using contrastive 3d human motion synthesis,

    M. Petrovich, M. J. Black, and G. Varol, “Tmr: Text-to-motion retrieval using contrastive 3d human motion synthesis,” in ICCV, 2023

    2023

  15. [23]

    Tri-modal motion retrieval by learning a joint embedding space,

    K. Yin, S. Zou, Y . Ge, and Z. Tian, “Tri-modal motion retrieval by learning a joint embedding space,” in CVPR, 2024

    2024

  16. [24]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  17. [25]

    Wavlm: Large-scale self-supervised pre- training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao et al. , “Wavlm: Large-scale self-supervised pre- training for full stack speech processing,” JSTSP, 2022

    2022

  18. [26]

    Better speech synthesis through scaling,

    J. Betker, “Better speech synthesis through scaling,” arXiv preprint arXiv:2305.07243, 2023

    2023 arXiv

  19. [27]

    Motionclip: Exposing human motion generation to clip space,

    G. Tevet, B. Gordon, A. Hertz, A. H. Bermano, and D. Cohen-Or, “Motionclip: Exposing human motion generation to clip space,” in ECCV, 2022

    2022

  20. [28]

    Auto-encoding variational bayes,

    D. P. Kingma, “Auto-encoding variational bayes,” ICLR, 2014

    2014

  21. [29]

    Temos: Generating diverse human motions from textual descriptions,

    M. Petrovich, M. J. Black, and G. Varol, “Temos: Generating diverse human motions from textual descriptions,” in ECCV, 2022

    2022

  22. [30]

    Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts,

    C. Guo, X. Zuo, S. Wang, and L. Cheng, “Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts,” in ECCV, 2022

    2022

  23. [31]

    Neural discrete representation learning,

    A. Van Den Oord, O. Vinyals et al. , “Neural discrete representation learning,” NeurIPS, 2017

    2017

  24. [32]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in ICML, 2022

    2022

  25. [33]

    Coca: Contrastive captioners are image-text foundation models,

    J. Yu, Z. Wang, V . Vasudevan, L. Yeung, M. Seyedhosseini, and Y . Wu, “Coca: Contrastive captioners are image-text foundation models,”TMLR, 2022

    2022

  26. [34]

    Global meets local: Dual activation hashing network for large-scale fine-grained image retrieval,

    X. Jiang, H. Tang, and Z. Li, “Global meets local: Dual activation hashing network for large-scale fine-grained image retrieval,” TKDE, 2024

    2024

  27. [35]

    Dvf: Advancing robust and accurate fine-grained image retrieval with retrieval guidelines,

    X. Jiang, H. Tang, R. Yan, J. Tang, and Z. Li, “Dvf: Advancing robust and accurate fine-grained image retrieval with retrieval guidelines,” in ACM MM, 2024

    2024

  28. [36]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in ACL, 2019

    2019

  29. [37]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,

    V . Sanh, “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,” NeurIPS, 2019

    2019

  30. [38]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of machine learning research, 2020

    2020

  31. [39]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  32. [40]

    Audiolm: a language modeling approach to audio generation,

    Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Shar- ifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi et al., “Audiolm: a language modeling approach to audio generation,” TASLP, 2023

    2023

  33. [41]

    Robust speech recognition via large-scale weak super- vision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak super- vision,” in ICML, 2023

    2023

  34. [42]

    Superb: Speech processing universal performance benchmark,

    S.-w. Yang, P.-H. Chi, Y .-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y . Y . Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin et al., “Superb: Speech processing universal performance benchmark,” arXiv preprint arXiv:2105.01051 , 2021

    2021 arXiv

  35. [43]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, “An image is worth 16x16 words: Transformers for image recognition at scale,” ICLR, 2020

    2020

  36. [44]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al. , “Learning transferable visual models from natural language supervision,” in ICML, 2021

    2021

  37. [45]

    Videopoet: A large language model for zero-shot video generation,

    D. Kondratyuk, L. Yu, X. Gu, J. Lezama, J. Huang, G. Schindler, R. Hornung, V . Birodkar, J. Yan, M.-C. Chiu et al., “Videopoet: A large language model for zero-shot video generation,” ICML, 2024

    2024

  38. [46]

    Label independent memory for semi-supervised few-shot video classification,

    L. Zhu and Y . Yang, “Label independent memory for semi-supervised few-shot video classification,” TPAMI, 2020

    2020

  39. [47]

    Memory-enhanced transformer for representation learning on temporal heterogeneous graphs,

    L. Li, L. Duan, J. Wang, C. He, Z. Chen, G. Xie, S. Deng, and Z. Luo, “Memory-enhanced transformer for representation learning on temporal heterogeneous graphs,” Data Science and Engineering , 2023

    2023

  40. [48]

    An efficient memory module for graph few-shot class-incremental learning,

    D. Li, A. Zhang, J. Gao, and B. Qi, “An efficient memory module for graph few-shot class-incremental learning,” NeurIPS, 2024

    2024

  41. [49]

    Imagebind: One embedding space to bind them all,

    R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V . Alwala, A. Joulin, and I. Misra, “Imagebind: One embedding space to bind them all,” in CVPR, 2023

    2023

  42. [50]

    Grounded language-image pre- training,

    L. H. Li, P. Zhang, H. Zhang, J. Yang, C. Li, Y . Zhong, L. Wang, L. Yuan, L. Zhang, J.-N. Hwang et al., “Grounded language-image pre- training,” in CVPR, 2022

    2022

  43. [51]

    Actionclip: A new paradigm for video action recognition,

    M. Wang, J. Xing, and Y . Liu, “Actionclip: A new paradigm for video action recognition,” arXiv preprint arXiv:2109.08472 , 2021

    2021 arXiv

  44. [52]

    Delving into multimodal prompting for fine-grained visual classifica- tion,

    X. Jiang, H. Tang, J. Gao, Z. Wang, D. Guan, Y . Liang, and Z. Liu, “Delving into multimodal prompting for fine-grained visual classifica- tion,” in AAAI, 2024

    2024

  45. [53]

    Amass: Archive of motion capture as surface shapes,

    N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black, “Amass: Archive of motion capture as surface shapes,” in ICCV, 2019. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 15

    2019

  46. [54]

    Action2motion: Conditioned generation of 3d human motions,

    C. Guo, X. Zuo, S. Wang, S. Zou, Q. Sun, A. Deng, M. Gong, and L. Cheng, “Action2motion: Conditioned generation of 3d human motions,” in ACM MM, 2020

    2020

  47. [55]

    Representation learning with contrastive predictive coding,

    A. v. d. Oord, Y . Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748 , 2018

    2018 arXiv

  48. [56]

    Randaugment: Practical automated data augmentation with a reduced search space,

    E. D. Cubuk, B. Zoph, J. Shlens, and Q. V . Le, “Randaugment: Practical automated data augmentation with a reduced search space,” in CVPR workshops, 2020

    2020

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.