ELGAR: Expressive Cello Performance Motion Generation for Audio Rendition

Chao Tan; Chongwu Wang; Feng Yu; Qionghai Dai; Tao Yu; Xiaobing Li; Yi Shi; Yitong Jin; Yuan Wang; Zhiping Qiu

REVIEW 4 major objections 6 minor 31 references

ELGAR: Expressive Cello Performance Motion Generation for Audio Rendition

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ELGAR generates whole-body cello performance motion with fine-grained hand and bow detail directly from raw audio.

desk verdict A genuinely new task and a useful dataset, but the evaluation is too circular and too under-powered to back the 'directly from audio' claim as it stands. read the letter →

arxiv 2505.04203 v2 pith:7YVMCFWM submitted 2025-05-07 cs.GR cs.SDeess.AS

Zhiping Qiu , Yitong Jin , Yuan Wang , Yi Shi , Chongwu Wang , Chao Tan , Xiaobing Li , Feng Yu

show 2 more authors

Tao Yu Qionghai Dai

This is my paper · ORCID

classification cs.GRcs.SDeess.AS

keywords motiongenerationcelloperformanceaudio-conditioneddiffusionmodelshand-objectinteractioninversekinematicscapturedatasetanimation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims to open a task no earlier system handles end-to-end: turning a raw cello recording into a full-body animated performance—torso, both hands, and the bow—without MIDI, sheet music, or a physics simulator. Its central argument is that earlier audio-to-motion work failed because it modeled body pose while ignoring the performer-instrument interaction, so ELGAR adds two training losses, HICL and BICL, that explicitly pull the note-playing fingertip and the bow onto the strings the audio implies. To support this, the authors normalize the SPD motion-capture corpus into a new benchmark, SPD-GEN (about 7000 seconds of whole-body cello motion on a shared body and cello), and propose three string-specific evaluation metrics: finger-contact distance, bow-string distance, and bowing score. A sympathetic reader should care because, if the claim holds, expressive instrumental animation becomes a one-input problem, which changes how films, games, music education, and interactive art can be produced.

What carries the argument

The load-bearing object is the Interactive Contact Loss, split into a Hand Interactive Contact Loss (HICL) and a Bow Interactive Contact Loss (BICL). Given the fundamental frequency of the audio, it computes a theoretical contact position on the cello fingerboard and identifies the activating string; HICL penalizes the distance between the generated note-playing fingertip and that contact position, BICL penalizes the distance between the generated bow and the activating string, and indicator functions switch the losses on only while a pitch is sounding. Around that core sits a diffusion transformer with DiT adaLN-Zero blocks, cross-attention to frozen Jukebox audio features, classifier-free guidance, and geometric losses (position, foot contact, rotation velocity, position velocity) for physical plausibility. The supporting dataset object is SPD-GEN, which normalizes all performers to one SMPL-X body and one shared cello with a restored arched bridge, using Kabsch alignment and two-stage inverse kinematics, so that audio-to-motion learning sees consistent geometry.

What would settle it

Take a passage where the same pitch recurs on different strings and measure finger-contact and bow-string distance using the string choice fixed by the audio estimator alone, without the nearest-trigger selection. If the generated motions' error jumps while the paper's reported metric stays small, the correct conclusion is that ELGAR reproduces contact geometry only under a lenient, self-selected target; a human listener or a spectrogram check of whether the generated bow direction fits the recorded timbre would settle which string the audio truly implies.

Watch

Extended reading notes

Core claim

ELGAR's claim, stated in the authors' terms, is that it is the first solution for generating whole-body instrument performance motion directly from audio, with the fine-grained hand gestures and intricate hand-string and bow-string interactions that prior work left out. The model represents a performance as a 309-dimensional motion vector (SMPL-X body and hand rotations plus a bow direction) and trains a diffusion transformer, conditioned by a frozen Jukebox audio encoder, to denoise that vector into clean motion. The two contact losses are the distinctive step: HICL supervises the left-hand note-playing finger to land on the audio-derived contact position while the other fingers keep a natural clearance from it, and BICL supervises the bow to reach the activating string with both endpoints respecting that string. The ablation results are reported as evidence for the core claim: adding BICL to HICL cuts average bow-string distance from 23.98 mm to 5.40 mm and lifts bowing F1-score from about 0.43 to 0.47, while the authors argue that general distribution metrics like FID are not appropriate here because the contact losses deliberately push generated motions away from the dataset distribution.

Load-bearing premise

The load-bearing premise is that the audio-derived contact position reliably identifies which finger stops which string, although the same pitch can be produced on more than one string; the evaluation softens this further by choosing the trigger position closest to the generated finger, so a motion that plays the wrong string can still score well.

Editorial extensions

If this is right

An animator could generate a full-body cellist, hands and bow included, directly from any audio clip, with no MIDI transcription or physics simulation in the loop.
SPD-GEN gives the research community a normalized benchmark for 3D instrument performance motion generation, filling a gap the paper identifies in existing datasets.
The ablations show the interaction losses are the decisive component for contact accuracy: adding BICL reduces bow-string distance from 23.98 mm to 5.40 mm.
The new metrics allow future string-performance generators to be judged by whether the fingers and bow actually meet the right places on the instrument, instead of only by distributional similarity to training data.
The overlapping-window long-form sampling strategy extends the method from five-second clips to full pieces, which is what practical use in animation demands.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the same pitch can be played on more than one string, the evaluation's choice of the trigger position nearest the generated finger means a wrong string choice can still earn a small finger-contact error; a sharper test would fix the intended string from the audio first and then measure whether the generated hand and bow are on that string.
If contact supervision is the key, the same HICL/BICL recipe should transfer to the rest of the violin family, where the physics of vibrating string length is identical and only the fingerboard geometry and bow angles change.
The paper's own listed simplifications—binary finger-string pressure, a static cello, occasional bow-string lift-off, and small dataset size—suggest the next measurable step is force or pressure sensing and instrument-body tracking rather than more pose data alone.
A musician-listener study could test whether the generated motions read as musical: the current metrics measure physical contact distances, not whether the timing and bow direction feel like a real interpretation.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is real work—new task framing, a cleaned-up dataset, and losses that improve contact plausibility—but the headline numbers don't yet prove that the motions are actually driven by the audio. The paper deserves a serious referee, and it will need a different evaluation design to support its central claim.

What's genuinely new: ELGAR is the first system I know of that generates whole-body (torso + hands + bow) cello performance motion straight from raw audio, with no MIDI or physics sim. The SPD-GEN dataset—normalized to one body and one cello, with an arched bridge—is a useful contribution on its own. HICL and BICL are sensible domain losses, and the qualitative videos in the supplementary material show noticeably cleaner hand-string and bow-string interaction than the no-ICL variant. The paper is also honest about its limits: static cello, binary finger pressure, occasional bow-string separation.

The soft spots are in the evaluation. FCD and BSD are not independent of the training objectives: FCD is essentially the first term of HICL (Eq. 6), and BSD is the first term of BICL (Eq. 7). An ablation table that shows "adding HICL/BICL improves FCD/BSD" is, to a significant degree, a restatement of the loss design. The less directly optimized metrics—bowing F1 (0.4721 with ICL) and cosine similarity (0.7515)—are more informative, but there are no error bars or significance tests, and no baselines at all, neither prior partial-body methods nor a simple pose-only model. The stress-test concern holds: nothing in the experiments tests whether the audio condition actually controls the output. A model that ignores audio and emits dataset-typical cello gestures could plausibly achieve similar FCD/BSD numbers because the string/finger ambiguity is resolved by picking the trigger position nearest the generated finger. To support "directly from audio," the authors need a conditioning control: shuffled audio, no-audio, or a different piece of music as input, reported across all metrics. The discussion acknowledges some artifacts, but the central claim lives or dies on the quantitative evidence.

Who should read it: anyone working on audio-driven animation or human-object interaction generation. It's a useful system paper and the dataset is worth borrowing, but treat the evaluation as a proof-of-concept, not a validated result. If I were the editor, I would send it to review—there is enough new substance here—but I would push hard for the conditioning baseline and for reporting variance before accepting.

Referee Report

4 major / 6 minor

Summary. The paper proposes ELGAR, a diffusion-transformer framework that generates whole-body cello performance motion from raw audio using Jukebox audio features. The motion representation includes body and hand rotations plus a bow direction vector, and the framework uses classifier-free guidance. Two domain-specific losses are introduced: Hand Interactive Contact Loss (HICL) and Bow Interactive Contact Loss (BICL), which encourage contact between the left-hand fingers and the audio-derived string positions and between the bow and the active string. The authors contribute SPD-GEN, a normalized motion-capture dataset, and new evaluation metrics for string performance: finger-contact distance, bow-string distance, and bowing F1/cosine similarity. Experiments are presented as ablations over the interaction losses.

Significance. If the central claims were fully supported, ELGAR would be the first audio-only, full-body model for instrument performance motion, and the SPD-GEN dataset plus the contact-aware losses would be useful community assets. The authors are transparent about limitations (static cello, binary finger-string pressure, occasional bow-string contact loss) and promise code and data release. The main weakness is evidentiary: the headline metrics largely mirror the training losses, and no experiment isolates the audio-conditioning behavior, so the strongest claims currently outrun the evaluation.

major comments (4)

[Section 4.2, Table 1, Eqs. (6)-(7)] The two headline metrics are essentially re-statements of the training losses. Finger-contact distance is exactly the note-playing-finger term in HICL (the first term of Eq. 6 with indicator I_f0), and bow-string distance is exactly the first term of BICL (Eq. 7). The reported improvements therefore mostly verify that the optimizer minimized the losses, not that the generated motions are more correct in an independent sense. I recommend adding distribution-level metrics (e.g., FID on pose features), penetration/collision counts between the hand or bow and the cello mesh, and a perceptual study, or at least reporting metrics computed under a protocol that is not directly optimized.
[Section 3.4 and Section 4.2] The experiments do not test whether the audio condition causally controls generation. All test samples use the same audio that produced the ground-truth motion, the CFG scale w in Eq. (4) is not swept, and no unconditional or mismatched-audio condition is evaluated. A model that ignores audio and returns dataset-typical cello motions could plausibly achieve similar FCD and BSD values because the evaluation selects the trigger position closest to the generated finger. Please add an explicit control experiment: generate with mismatched audio (e.g., swapped between test pieces) and with the condition dropped or masked, and show clear degradation on all four metrics; also report a small sweep over w.
[Section 4.2] The ambiguity-resolution rule, choosing the trigger position closest to the generated note-playing finger, makes FCD and BSD lenient and can conceal wrong-string or wrong-position playing. For example, a generated finger near an alternative string position for the same pitch is scored as correct even if it does not match the performer's actual string choice. Please report, in addition, the per-frame accuracy of the selected string/finger against the ground-truth annotation, or evaluate FCD and BSD with the ground-truth trigger positions fixed, so the metrics penalize actual fingering errors.
[Table 1 and Section 1] No baselines, error bars, or significance tests are reported. The numbers in Table 1 are single values without variance or test-set size, and the 'state-of-the-art' claim in the abstract is not supported by any comparison to existing audio-to-motion or performance-motion methods. At minimum, compare with a deterministic regression baseline and an audio-agnostic diffusion variant, and report mean plus or minus standard deviation over multiple seeds or test splits.

minor comments (6)

[Section 3.1] The text says the bow frog is anchored between the middle finger, ring finger, and thumb of the left hand, but the bow is held in the right hand (Section 3.3.2 and Fig. 2). Please correct this typo, since the hand assignment is essential for reproducing the motion representation.
[Equation (3)] The simple loss is written with a norm rather than a squared norm; if the squared L2 loss is intended, as in the cited works, please fix the notation.
[Section 4.2] The terms 'bowing score' and 'bowing F1-score' are used interchangeably; define the exact quantity and clarify whether the reported BCS is the cosine similarity only.
[Section 4.1] The loss weights in Eq. (8) and the CFG scale w in Eq. (4) are not reported; please include their values, as they are needed to reproduce the ablations.
[Table 1] The number of test segments and the split procedure (e.g., whether overlapping 5-second slices are deduplicated) should be stated.
[References] The reference list contains duplicate entries (Karunratanakul et al. 2023a/b and Li et al. 2025a/b refer to the same papers); please consolidate.

Circularity Check

2 steps flagged · score 6.0 of 10

FCD and BSD are the HICL and BICL loss terms re-measured, so the ablation validates those losses by construction; the diffusion pipeline itself retains independent content.

self definitional [Section 3.3.2 (Eq. 6) vs Section 4.2 (finger-contact distance; Table 1)]
"L_hand = 1_note || d̂_cp ⊙ I_f0 ||^2_2 + 1_others || (d̂_cp − d_cp) ⊙ I_f0 ||^2_2 (6) ... The finger-contact distance examines the deviation between the tip of the left-hand note-playing finger and the trigger position on the cello for the current pitch."

HICL's first summand is exactly the predicted fingertip-to-trigger distance for the note-playing finger, masked by the pitch indicator. FCD measures the same distance, with the additional leniency that the trigger position is chosen as the one closest to the generated finger. Therefore the Table 1 FCD improvement when HICL is added (18.64 to 14.56) is the training loss descending, not an independent test of musical correctness. The reported validation of HICL is partly definitional.
self definitional [Section 3.3.2 (Eq. 7) vs Section 4.2 (bow-string distance; Table 1)]
"L_bow = || d̂_ls,lb ⊙ I_f0 ||^2_2 + || (d̂_p,ls − d_p,ls) ⊙ I_f0 ||^2_2 (7) ... The bow-string distance reflects the deviation between the bow and the string to be struck, which is uniquely identified once the aforementioned trigger position is determined."

BICL's first summand is precisely the predicted bow-to-playing-string distance, and BSD reports that same distance. The Table 1 improvement from 23.98 to 5.40 after adding BICL is therefore the loss objective itself being re-measured, not an independent demonstration that the generated bowing follows the music. Because BSD inherits the lenient 'closest trigger' selection from FCD, it is even more permissive than a fixed ground-truth comparison.

full rationale

The core generative pipeline is not circular: ELGAR is a conditional diffusion model using frozen Jukebox audio features through cross-attention and CFG, trained on SPD-GEN, and its headline contribution is an empirical capability claim rather than a derivation from an equation. However, two of the four headline metrics are constructed as the same geometric quantities that the proposed losses minimize. Eq. (6) minimizes predicted fingertip-to-trigger distance, and Section 4.2's finger-contact distance is that same deviation; Eq. (7) minimizes predicted bow-to-string distance, and bow-string distance is that same deviation. Thus the large FCD and BSD gains in Table 1 after adding HICL and BICL largely restate loss minimization. The remaining metrics, bowing F1 and cosine similarity, are not direct loss terms and provide some independent signal, which prevents the whole paper from being a purely circular exercise. The paper's stronger claim that motions are generated 'directly from audio' also lacks an audio-conditioning control, but that is an evidence gap rather than a definitional circularity. Self-citation of [Jin et al. 2024a] for SPD data and contact positions is legitimate provenance, though it amplifies the self-referential character of FCD and BSD because the same prior pipeline supplies both the contact-loss supervision targets and the metric reference positions.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the reliability of audio-derived contact positions (self-cited SPD processing), on the bow and hand contact modeling choices, and on the validity of the SMPL-X IK representation. No new physical entities are postulated. Loss weights and guidance scale are free hyperparameters not reported.

free parameters (3)

Loss weights lambda_simple, lambda_foot, lambda_pos, lambda_rotvel, lambda_posvel, lambda_hand, lambda_bow = not reported
The overall loss (Eq. 8) is a weighted sum; values are not given in the paper and must be tuned by hand.
Classifier-free guidance scale w = not reported
Used in sampling (Eq. 4); not specified in implementation details.
Bowing F1 tolerance delta = 3 frames
Set to 0.1s following Kao and Su 2020; affects the Bowing F1 metric.

assumptions (5)

domain assumption Audio-derived contact positions from the SPD pipeline reliably identify the note-playing finger and the activating string for each pitch.
Used as training targets in HICL (Eq. 6) and BICL (Eq. 7), and as ground truth for FCD and BSD metrics (Sec. 4.2). Same pitch can be played on multiple strings, but the pipeline assigns a unique contact position.
domain assumption The bow model with frog anchored between the middle, ring, and thumb PIP/DIP joints and a unit direction vector captures the real bow geometry.
Establishes the bow endpoints used in BICL and the bow-string distances (Fig. 2, Sec. 3.1).
domain assumption SMPL-X two-stage IK fitting from sparse keypoints yields a valid whole-body motion representation, including wrist and hand rotations.
The motion representation r in R^306 is obtained through this IK process (Sec. 3.1); errors in hand pose propagate to all losses and evaluations.
ad hoc to paper A static cello with a restored arched bridge and fixed placement is a sufficient proxy for the real instrument across all performances.
The paper aligns all cellos to one shared instrument and assumes the instrument does not move during performance (Sec. 3.1 and Sec. 5).
domain assumption Jukebox audio features are robust enough to condition generated performance motions.
The frozen Jukebox encoder is the only audio condition (Sec. 3.4); if these features are insufficient, the audio-to-motion mapping fails.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ELGAR: Expressive Cello Performance Motion Generation for Audio Rendition." pith.science (2026). https://pith.science/paper/7YVMCFWM

@misc{pith2026250504203,
  author       = {Pith},
  title        = {Pith review of: ELGAR: Expressive Cello Performance Motion Generation for Audio Rendition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7YVMCFWM}},
  note         = {Machine review of arXiv:2505.04203}
}

read the original abstract

The art of instrument performance stands as a vivid manifestation of human creativity and emotion. Nonetheless, generating instrument performance motions is a highly challenging task, as it requires not only capturing intricate movements but also reconstructing the complex dynamics of the performer-instrument interaction. While existing works primarily focus on modeling partial body motions, we propose Expressive ceLlo performance motion Generation for Audio Rendition (ELGAR), a state-of-the-art diffusion-based framework for whole-body fine-grained instrument performance motion generation solely from audio. To emphasize the interactive nature of the instrument performance, we introduce Hand Interactive Contact Loss (HICL) and Bow Interactive Contact Loss (BICL), which effectively guarantee the authenticity of the interplay. Moreover, to better evaluate whether the generated motions align with the semantic context of the music audio, we design novel metrics specifically for string instrument performance motion generation, including finger-contact distance, bow-string distance, and bowing score. Extensive evaluations and ablation studies are conducted to validate the efficacy of the proposed methods. In addition, we put forward a motion generation dataset SPD-GEN, collated and normalized from the MoCap dataset SPD. As demonstrated, ELGAR has shown great potential in generating instrument performance motions with complicated and fast interactions, which will promote further development in areas such as animation, music education, interactive art creation, etc.

Figures

Figures reproduced from arXiv: 2505.04203 by the authors.

**Figure 1.** ELGAR is capable of generating cello performance motion with precise details and complicated interactions solely from audio [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

**Figure 2.** Top: We position the starting point of the bow (frog) at the midpoint between the PIP and DIP joints of the middle finger, ring finger, and thumb (highlighted in red). Bottom: As shown in (b), SPD-GEN reconstructs the arched cello bridge, unlike the flat bridge in SPD, closely matching the actual instrument illustrated in (c). This enables the performer to play the two middle strings without unintended contact with … view at source ↗

**Figure 3.** Given performance audio, ELGAR employs DiT blocks with adaLN-Zero to denoise the performance motions from [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: A variety of sample motions generated by ELGAR, shown from the holistic view and the performer’s view, to reveal their diversity and richness. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: In this figure, we present a comparative demonstration of the bow and left hand motions before and after the introduction of the [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 19 canonical work pages

[2]

ACM Transactions on Graphics (TOG) 42, 4 (2023), 1–20

Listen, denoise, action! audio-driven motion synthesis with diffusion models. ACM Transactions on Graphics (TOG) 42, 4 (2023), 1–20. Rodrigo Castellon, Chris Donahue, and Percy Liang

work page 2023
[4]

IEEE Transactions on Visualization and Computer Graphics 29, 2 (2021), 1400–1414

A music-driven deep generative adversarial model for guzheng playing animation. IEEE Transactions on Visualization and Computer Graphics 29, 2 (2021), 1400–1414. Setareh Cohan, Guy Tevet, Daniele Reda, Xue Bin Peng, and Michiel van de Panne

work page 2021
[7]

Advances in neural information processing systems 34 (2021), 8780–8794

Diffusion models beat gans on image synthesis. Advances in neural information processing systems 34 (2021), 8780–8794. Christian Diller and Angela Dai

work page 2021
[10]

ACM Trans- actions on Graphics (TOG) 43, 6 (2024), 1–17

Geometry-Aware Retargeting for Two-Skinned Characters Interaction. ACM Trans- actions on Graphics (TOG) 43, 6 (2024), 1–17. Peng Jin, Yang Wu, Yanbo Fan, Zhongqian Sun, Wei Yang, and Li Yuan. 2024b. Act as you wish: Fine-grained control of motion diffusion model with hierarchical semantic graphs. Advances in Neural Information Processing Systems 36 (2024)...

work page 2024
[16]

Inter- national Journal of Computer Vision (2024), 1–21

Intergen: Diffusion-based multi-human motion generation under complex interactions. Inter- national Journal of Computer Vision (2024), 1–21. Jun-Wei Liu, Hung-Yi Lin, Yu-Fen Huang, Hsuan-Kai Kao, and Li Su

work page 2024
[17]

In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Body movement generation for expressive violin performance applying neural networks. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 3787–3791. Evonne Ng, Javier Romero, Timur Bagautdinov, Shaojie Bai, Trevor Darrell, Angjoo Kanazawa, and Alexander Richard

work page 2020
[18]

arXiv preprint arXiv:2112.10741 (2021)

Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 (2021). Alexander Quinn Nichol and Prafulla Dhariwal

arXiv 2021
[20]

arXiv preprint arXiv:2204.06125 1, 2 (2022),

Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1, 2 (2022),

arXiv 2022

Show all 31 references

[21]

Advances in neural information processing systems 32 (2019)

Generating diverse high- fidelity images with vq-vae-2. Advances in neural information processing systems 32 (2019). Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al

2019
[22]

Advances in neural information processing systems 35 (2022), 36479–36494

Photorealistic text-to-image diffusion models with deep language under- standing. Advances in neural information processing systems 35 (2022), 36479–36494. Eli Shlizerman, Lucio Dery, Hayden Schoen, and Ira Kemelmacher-Shlizerman

2022
[23]

IEEE Transactions on Pattern Analysis and Machine Intelligence (2023)

Bailando++: 3d dance gpt with choreographic memory. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023). Jiaming Song, Chenlin Meng, and Stefano Ermon

2023
[24]

arXiv preprint arXiv:2010.02502 (2020)

Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020). Mikihiro Tanaka and Kent Fujiwara

2020 arXiv
[25]

arXiv preprint arXiv:2209.14916 (2022)

Human Motion Diffusion Model. arXiv preprint arXiv:2209.14916 (2022). Jonathan Tseng, Rodrigo Castellon, and Karen Liu

2022 arXiv
[26]

In SIGGRAPH Asia 2024 Conference Papers

FürElise: Capturing and Physically Synthesizing Hand Motion of Piano Performance. In SIGGRAPH Asia 2024 Conference Papers . 1–11. Megan Wei, Michael Freeman, Chris Donahue, and Chen Sun

2024
[27]

Yiming Xie, Varun Jampani, Lei Zhong, Deqing Sun, and Huaizu Jiang

Do Music Genera- tion Models Encode Music Theory? arXiv preprint arXiv:2410.00872 (2024). Yiming Xie, Varun Jampani, Lei Zhong, Deqing Sun, and Huaizu Jiang

2024 arXiv
[28]

arXiv preprint arXiv:2310.08580 (2023)

Omni- control: Control any joint at any time for human motion generation. arXiv preprint arXiv:2310.08580 (2023). Pei Xu and Ruocheng Wang

2023 arXiv
[29]

In SIGGRAPH Asia 2024 Conference Papers

Synchronize Dual Hands for Physics-Based Dexter- ous Guitar Playing. In SIGGRAPH Asia 2024 Conference Papers . 1–11. Siwei Zhang, Bharat Lal Bhatnagar, Yuanlu Xu, Alexander Winkler, Petr Kadlecek, Siyu Tang, and Federica Bogo

2024
[30]

In ACM SIGGRAPH 2023 Conference Proceedings

Simulation and retargeting of complex multi-character interactions. In ACM SIGGRAPH 2023 Conference Proceedings . 1–11. Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li

2023
[37]

Yitong Jin, Zhiping Qiu, Yi Shi, Shuangpeng Sun, Chongwu Wang, Donghao Pan, Jiachen Zhao, Zhenghao Liang, Yuan Wang, Xiaobing Li, et al

Wiley Online Library, 311–320. Yitong Jin, Zhiping Qiu, Yi Shi, Shuangpeng Sun, Chongwu Wang, Donghao Pan, Jiachen Zhao, Zhenghao Liang, Yuan Wang, Xiaobing Li, et al. 2024a. Audio Matters Too! Enhancing Markerless Motion Capture with Audio Signals for String Performance Captu...

2024
[224]

Jiaman Li, Alexander Clegg, Roozbeh Mottaghi, Jiajun Wu, Xavier Puig, and C Karen Liu. 2025a. Controllable human-object interaction synthesis. In European Conference on Computer Vision. Springer, 54–72. SIGGRAPH Conference Papers ’25, August 10–14, 2025, Vancouver, BC, Canada....

2025
[1976]

Acta Crystallographica Section A: Crystal Physics, Diffraction, Theoretical and General Crystallography 32, 5 (1976), 922–923

A solution for the best rotation to relate two sets of vectors. Acta Crystallographica Section A: Crystal Physics, Diffraction, Theoretical and General Crystallography 32, 5 (1976), 922–923. Hsuan-Kai Kao and Li Su

1976
[2003]

InProceed- ings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation

Handrix: animating the human hand. InProceed- ings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation . 110–119. Kehong Gong, Dongze Lian, Heng Chang, Chuan Guo, Zihang Jiang, Xinxin Zuo, Michael Bi Mi, and Xinchao Wang

2003
[2013]

Computer Animation and Virtual Worlds 24, 5 (2013), 445–457

A system for automatic animation of piano performances. Computer Animation and Virtual Worlds 24, 5 (2013), 445–457. SIGGRAPH Conference Papers ’25, August 10–14, 2025, Vancouver, BC, Canada

2013
[2016]

A computational approach to studying interdependence in string quartet performance. (2016). Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black

2016
[2018]

In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Crepe: A convo- lutional representation for pitch estimation. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 161–165. Hanyang Kong, Kehong Gong, Dongze Lian, Michael Bi Mi, and Xinchao Wang

2018
[2019]

arXiv preprint arXiv:1812.04948 (2019)

A Style-Based Generator Architecture for Generative Adversarial Networks. arXiv preprint arXiv:1812.04948 (2019). Korrawe Karunratanakul, Konpat Preechakul, Supasorn Suwajanakorn, and Siyu Tang. 2023a. Guided motion diffusion for controllable human motion synthesis. In Pro- ce...

2019 arXiv
[2020]

arXiv preprint arXiv:2005.00341 (2020)

Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341 (2020). Prafulla Dhariwal and Alexander Nichol

2020 arXiv
[2021]

arXiv preprint arXiv:2107.05677 (2021)

Codified audio language modeling learns useful representations for music information retrieval. arXiv preprint arXiv:2107.05677 (2021). Junuk Cha, Jihyeon Kim, Jae Shin Yoon, and Seungryul Baek

2021 arXiv
[2022]

arXiv preprint arXiv:2207.12598 (2022)

Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 (2022). Inseo Jang, Soojin Choi, Seokhyeon Hong, Chaelin Kim, and Junyong Noh

2022 arXiv
[2023]

arXiv preprint arXiv:2303.08774 (2023)

Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023). Simon Alexanderson, Rajmund Nagy, Jonas Beskow, and Gustav Eje Henter

2023 arXiv
[2024]

arXiv preprint arXiv:2405.11126 (2024)

Flexible Motion In-betweening with Diffusion Models. arXiv preprint arXiv:2405.11126 (2024). Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever

2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [2]

ACM Transactions on Graphics (TOG) 42, 4 (2023), 1–20

Listen, denoise, action! audio-driven motion synthesis with diffusion models. ACM Transactions on Graphics (TOG) 42, 4 (2023), 1–20. Rodrigo Castellon, Chris Donahue, and Percy Liang

work page 2023

[2] [4]

IEEE Transactions on Visualization and Computer Graphics 29, 2 (2021), 1400–1414

A music-driven deep generative adversarial model for guzheng playing animation. IEEE Transactions on Visualization and Computer Graphics 29, 2 (2021), 1400–1414. Setareh Cohan, Guy Tevet, Daniele Reda, Xue Bin Peng, and Michiel van de Panne

work page 2021

[3] [7]

Advances in neural information processing systems 34 (2021), 8780–8794

Diffusion models beat gans on image synthesis. Advances in neural information processing systems 34 (2021), 8780–8794. Christian Diller and Angela Dai

work page 2021

[4] [10]

ACM Trans- actions on Graphics (TOG) 43, 6 (2024), 1–17

Geometry-Aware Retargeting for Two-Skinned Characters Interaction. ACM Trans- actions on Graphics (TOG) 43, 6 (2024), 1–17. Peng Jin, Yang Wu, Yanbo Fan, Zhongqian Sun, Wei Yang, and Li Yuan. 2024b. Act as you wish: Fine-grained control of motion diffusion model with hierarchical semantic graphs. Advances in Neural Information Processing Systems 36 (2024)...

work page 2024

[5] [16]

Inter- national Journal of Computer Vision (2024), 1–21

Intergen: Diffusion-based multi-human motion generation under complex interactions. Inter- national Journal of Computer Vision (2024), 1–21. Jun-Wei Liu, Hung-Yi Lin, Yu-Fen Huang, Hsuan-Kai Kao, and Li Su

work page 2024

[6] [17]

In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Body movement generation for expressive violin performance applying neural networks. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 3787–3791. Evonne Ng, Javier Romero, Timur Bagautdinov, Shaojie Bai, Trevor Darrell, Angjoo Kanazawa, and Alexander Richard

work page 2020

[7] [18]

arXiv preprint arXiv:2112.10741 (2021)

Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 (2021). Alexander Quinn Nichol and Prafulla Dhariwal

arXiv 2021

[8] [20]

arXiv preprint arXiv:2204.06125 1, 2 (2022),

Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1, 2 (2022),

arXiv 2022