{"id":"7aa3c03f-cec4-4cc3-a962-c75929252d76","arxiv_id":"2509.02278","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Think2Sing uses LLM-generated, time-aligned motion subtitles and a motion-intensity proxy to guide diffusion-based 3D head animation from singing audio and lyrics.","lead":"A new AI system generates 3D singing-head animations by first asking a large language model to write time-stamped \"motion subtitles\" for each facial region, then using those subtitles to guide a diffusion model. It also builds a 37-hour singing dataset with synchronized facial motion and acoustic annotations, and claims it produces more expressive and emotionally faithful animations than prior audio-driven methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (5) per-clip min-max normalization discards absolute motion amplitude; Intensity2Motion (Eq. 6) cannot recover it, so the claimed control of emotional intensity is not actually supported by the architecture.","rationale":"The reader's weakest assumption was the validity of automatically generated labels from EMOCA v2 and hand-set annotation thresholds. That is a legitimate external-validity concern, but it is not the most load-bearing issue: even if the labels were perfect, the proposed motion intensity proxy would still be unable to represent absolute motion amplitude because Eq. (5) is affine-invariant under positive scaling. The central claim of 'emotional fidelity' and 'motion intensity prediction' depends on the proxy carrying intensity information, yet the normalization removes exactly that information, and Eq. (6) provides no path to recover it. This is an internal, analytical issue that can be checked without external 3D capture or manual annotation. It does not necessarily invalidate the whole method—the normalized pattern still carries useful temporal shape, and fϕ can learn a dataset-average amplitude—but it does undermine the specific claim that the method can modulate emotional intensity and produce expressive magnitude variation. Since the reader already gave a CONDITIONAL verdict, this concern sharpens the condition rather than moving to accept or reject: the authors should validate amplitude recovery or revise the representation. I therefore keep the verdict unchanged but add this test as a required condition.","tokens_in":30941,"tokens_out":7710,"duration_ms":105229,"concrete_test":"Amplitude-recovery probe: on 100 held-out test clips, compute the ground-truth per-clip RMS amplitude of the FLAME expression/jaw parameters (or landmark displacements). Run the trained Think2Sing pipeline to generate animations, compute the same RMS amplitude on the outputs, and report the Pearson correlation and mean absolute ratio between generated and GT amplitudes. If the correlation is near zero (or outputs cluster near the training-set mean), Eq. (5) has removed the scale that Eq. (6) cannot restore. A companion ablation: retrain with a per-clip scale channel appended to I (e.g., the max displacement) and compare FVE/SND; material improvement would confirm the bottleneck.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Think2Sing's central claim to 'emotional fidelity' rests on the motion intensity proxy. In Eq. (5), each landmark's displacement is min-max normalized over the temporal window [1,T], making the proxy invariant to any positive rescaling of the underlying motion: I(a·ΔL)=I(ΔL) for any a>0. The intensity sequence therefore encodes only the normalized shape of motion, not its magnitude. Eq. (6) then maps I directly to FLAME coefficients and jaw pose through fϕ, with no audio, subtitle, or scale input. A single shared fϕ cannot invert the discarded scale: for a given normalized pattern, it must output a fixed (or training-set-mean) amplitude. Consequently, a louder, higher-pitch, or more emotionally intense passage cannot produce larger facial displacements unless the normalized pattern happens to correlate with amplitude in the training distribution. This is an internal representational limitation, independent of the separate concern about EMOCA label validity: even with perfectly accurate ground-truth FLAME parameters, the architecture cannot represent absolute motion intensity. The paper explicitly motivates the proxy as capturing 'dynamic behavior' and 'precise, region-specific control,' but per-clip normalization removes exactly the quantity that would make intensity controllable. The reported FVE/FID improvements may partly reflect the smoother, lower-variance outputs induced by this bottleneck rather than genuine expressive superiority.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Think2Sing, a diffusion-based framework for singing-driven 3D head animation. It uses an LLM (Gemini 2.5 Flash) with a singing chain-of-thought and acoustic-guided retrieval (Sing-CoT + AGRA) to generate SRT-style, timestamped motion subtitles from lyrics and audio; these subtitles condition four region-specific mappers that predict a normalized motion-intensity proxy and neck pose, which are then converted to FLAME parameters through an Intensity2Motion predictor. A new 37-hour dataset, SingMoSub, is introduced with FLAME parameters, acoustic descriptors, and automatically annotated motion subtitles. Quantitative and qualitative experiments on SingMoSub claim large improvements over several talking-head baselines in lip synchronization, realism, expressiveness, and music alignment, plus a subtitle-based editing interface.","tokens_in":31325,"tokens_out":5388,"duration_ms":67416,"significance":"The central idea is timely and potentially valuable: using structured, timestamped natural-language motion descriptions as an interpretable intermediate representation for expressive facial animation is a genuine departure from direct audio-to-motion mapping. The dataset with region-wise motion subtitles is a useful asset for the community, and the LLM prompting/reasoning pipeline (AGRA + Sing-CoT) is reusable beyond this paper. The work is not circular: the system is trained to predict ground-truth FLAME from audio/subtitles and evaluated on held-out songs. However, the current evidence does not fully support the strength of the claims. The evaluation lacks error bars and significance tests, several metrics are self-defined, baseline training is modified, and the intensity proxy discards absolute motion amplitude, undermining the emotional-intensity-control claim. With these issues addressed, the contribution could be significant; in its present form the claims outrun the evidence.","major_comments":[{"comment":"The per-clip min-max normalization in Eq. (5) makes the motion intensity invariant to positive rescaling of landmark displacement: I(a·ΔL)=I(ΔL) for any a>0. Eq. (6) then maps only this normalized intensity to FLAME coefficients and jaw pose through fϕ, with no audio, subtitle, or scale input. The architecture therefore cannot represent absolute motion amplitude; a louder, higher-pitch, or more emotionally intense passage cannot produce larger facial displacements unless the normalized pattern happens to correlate with amplitude in the training set. This directly weakens the paper's 'emotional fidelity' and 'precise motion intensity control' claims. Please either add a scale channel to the proxy (e.g., per-region RMS displacement or a separate amplitude estimate) or provide a controlled experiment where volume/intensity is varied while the normalized pattern is held fixed, demonstrating","section":"§4.2.1, Eq. (5) and Eq. (6)"},{"comment":"All quantitative results are single-run means with no error bars, confidence intervals, or significance tests, despite the abstract's claim of 'significantly outperforms'. Some headline gaps are large (SND 4.89 vs. 21.19), but several key comparisons are close (e.g., Table 3: Ours vs. GT Sub in FIDfm is 4.8187 vs. 4.7649; LVE 8.3036 vs. 7.8513). The user study also lacks statistical detail. Please report at least three training seeds with standard deviations and appropriate paired significance tests, and describe the user-study protocol more fully.","section":"§5, Tables 2 and 3"},{"comment":"The baseline comparison protocol is modified in a way that may advantage the proposed method. Vertex-based methods are trained with neck pose zeroed, and FLAME variants have altered output heads. Moreover, the table states that BA is not utilized for methods without neck pose modules, making head-motion alignment comparisons incomplete. These changes are not standard for the cited baselines and could explain part of the performance gap. Please provide an additional comparison under the baselines' original training settings (or a controlled subset where neck motion is minimal) and report the hyperparameter/budget used for each baseline.","section":"§5.2, Table 2"},{"comment":"The self-defined metrics need validation. Freeze Rate uses a per-sequence mean ground-truth vertex displacement as the 'minimal movement' threshold, which penalizes low-amplitude but natural motions; SND is a sum of two FID values and is not a standard metric. Since these metrics are used to support the main superiority claim, please provide sensitivity analyses to the threshold choices, report correlations with human judgments, and compare against established motion-quality metrics where possible.","section":"§5.1.2"},{"comment":"The validation of LLM-generated subtitles is performed by the same LLM (Gemini 2.5 Flash), and Table 4 reports only LLM pass rates. This self-assessment does not establish subtitle correctness. In Table 3, GT Sub outperforms the full method on LVE, FVE, and FIDfm; the claim that the AGRA+Sing-CoT subtitles are 'comparable' to ground truth is not statistically supported. Please add human evaluation of subtitle correctness and measure the downstream animation metrics when conditioning on generated versus manually curated subtitles, with error bars.","section":"§4.1.2 Step 3 and Table 4"},{"comment":"The ground-truth FLAME parameters are obtained from EMOCA v2 monocular reconstruction, and the motion-subtitle thresholds (AU intensity difference 0.25/0.5, eyelid distances 9.5 mm and 4–6 mm, neck rotation 10 degrees) are hand-set without independent validation. Since training and evaluation share the same label source, systematic reconstruction bias could inflate the reported LVE/FVE/FID improvements. Please validate a subset of the dataset with independent 3D capture or manual annotation, and report threshold sensitivity analyses.","section":"§3.1 and §5.1.2"}],"minor_comments":[{"comment":"Typo: 'the its annotations' should be 'its annotations'.","section":"§2.2"},{"comment":"Typo: 'preceise' should be 'precise'; 'title right/left' in Fig. 1 should be 'tilt right/left'; 'suffer form' should be 'suffer from'.","section":"§3.1 and supplementary"},{"comment":"The text says 'We use Adam [65]', but reference [65] is the Adan optimizer paper. Please correct the citation or the optimizer name.","section":"§5.1.1"},{"comment":"The main text and supplementary both contain large pipeline figures; the repeated prompt screenshots in the supplementary should be compressed or reduced. Also, metric notation such as FIDfm, FIDΔfm, and SND should be defined in one place for readers.","section":"Figures 1 and 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good fit for TVCG in terms of topic and novelty. I do not see circularity in the core audio-to-motion training, but the intensity-scale issue is a real architectural limitation that must be addressed before the emotional-fidelity claim can stand. The evaluation needs statistical grounding, and the baseline modifications should be justified more carefully. If the authors can add a scale-aware intensity representation and strengthen the evaluation, the paper would likely be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper introduces a genuinely new intermediate representation for singing-driven head animation — SRT-style motion subtitles with timestamps and region-specific descriptions — and backs it with a 37-hour dataset, SingMoSub, the first to include such annotations. That is a real contribution, and the ablation (GT Sub vs. Lyrics Sub vs. Ours) suggests the subtitles carry useful signal. Second, the headline claim about \"emotional fidelity\" and \"motion intensity\" control does not survive contact with the equations. Eq. (5) min-max normalizes each landmark's displacement over the temporal window, which makes the intensity proxy invariant to absolute amplitude. Eq. (6) then maps that normalized intensity directly to FLAME parameters with no scale or audio input, so the network cannot represent whether a passage is sung louder or with more energy. A louder performance cannot produce larger facial displacements unless the normalized pattern happens to correlate with amplitude in the training data. The paper's own vocabulary of \"strongly\" vs. \"gently\" lives only in the subtitle text; the intensity proxy has thrown away exactly the quantity the text claims to control.\n\nThe engineering is otherwise sound: the diffusion setup, the Semantic Modulation layer, the AGRA retrieval, and the Sing-CoT loop are all coherent, and the small user study points in the right direction. The dataset, if released, will be useful to the subfield. But verification is thin. No code or data are available yet. The quantitative tables have no error bars or significance tests, and several metrics (Freeze Rate, SND) are self-defined. Baselines are modified — neck pose is zeroed for vertex-based methods — which makes the comparison generous to the proposed method in ways that are hard to audit. The FLAME labels come from EMOCA v2 on in-the-wild video and are treated as ground truth without external validation; that matters for both training and evaluation. The LLM-based subtitle validation is also self-referential: the LLM scores its own outputs, so the 96% third-round success rate does not tell us much about downstream motion quality.\n\nThis is a solid systems paper for a narrow audience. People working on audio-driven facial animation or singing avatars should read it for the dataset and the subtitle idea. It deserves a serious referee, but the referee should demand artifact release, error bars, and either a fix or an explicit caveat about the intensity-scale limitation. I would accept it for review, not for publication as-is.","headline":"A genuinely new subtitle-based intermediate representation plus a large dataset, but the intensity proxy discards absolute amplitude and the evaluation is too thin to back the headline claims.","tokens_in":31794,"tokens_out":3274,"would_cite":false,"duration_ms":41399,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Singing avatars gain expression from LLM-written motion subtitles","keywords":["singing-driven head animation","3D facial animation","motion subtitles","large language models","diffusion model","motion intensity proxy","multimodal dataset","LLM reasoning"],"falsifier":"Capture a small set of singing performances with marker-based 3D motion capture, retrain Think2Sing on the ground-truth FLAME parameters, and check whether the reported LVE/FVE/FID gains over audio-only baselines persist; if the gains vanish, they depend on the monocular label bias.","tokens_in":30878,"feed_emoji":"🎤","tokens_out":5590,"duration_ms":59615,"temperature":0.7,"pith_summary":"This paper claims that the expressive gap in singing-driven 3D head animation can be closed by inserting a structured, human-readable intermediate representation between audio and motion: motion subtitles—SRT-style lines with precise timestamps and region-specific descriptions of eyebrow, eye, mouth, and neck movement. Think2Sing uses a large language model to generate these subtitles from time-aligned lyrics and low-level acoustic descriptors, then conditions a diffusion-based motion generator on them. The generator predicts per-region motion intensities rather than raw vertex offsets or FLAME coefficients, which the authors argue decomposes the difficult audio-to-motion mapping into tractable subtasks and enables fine-grained control. If the claim holds, singing avatar animation becomes more emotionally faithful, more controllable, and—with subtitle generation done offline—runs in real time. The paper also introduces the first multimodal singing dataset (SingMoSub) with synchronized video, acoustic descriptors, and motion subtitles.","feed_headline":"Singing avatars gain expression from LLM-written motion subtitles","feed_subtitle":"Timestamped, region-wise motion descriptions guide a diffusion model to beat audio-only baselines on realism and emotion.","key_machinery":"Motion subtitles are SRT-style text lines of the form '<start> → <end>: <region> <description>', automatically inferred by an LLM through Sing-CoT with acoustic-guided retrieval (AGRA). The motion intensity proxy is a per-landmark normalized amplitude (range [-1, 1]) computed from FLAME landmark displacements relative to neutral, restricted to eyebrows, eyes, and mouth; it carries the argument by converting the mapping problem into intensity prediction per region, which the diffusion model's region-specific mappers supervise with reconstruction, velocity, and acceleration losses. An Intensity2Motion predictor converts the predicted intensities into FLAME expression and jaw parameters.","core_discovery":"Think2Sing's central claim is that an LLM-assisted pipeline can produce 3D head animation for singing that is more realistic, expressive, and emotionally faithful than existing speech-driven and singing-driven methods. The paper attributes the gains to three coordinated choices: (1) motion subtitles, generated by a Singing Chain-of-Thought (Sing-CoT) process with acoustic-guided retrieval augmentation, which give the model explicit, timestamped, region-wise motion instructions; (2) a motion intensity proxy—normalized displacement amplitudes of selected eyebrow, eye, and mouth landmarks—which serves as a spatially disentangled intermediate target easier to learn than dense vertex offsets or c","pith_inferences":["Because the subtitle generator is a general-purpose LLM, its choices encode that LLM's prior about which lyrics trigger which facial motions; swapping or fine-tuning the LLM would plausibly change the animation style, a testable way to modulate expressiveness.","The same subtitle-plus-intensity-proxy recipe should transfer to speech-driven characters or embodied conversational agents where semantic alignment matters; the AGRA retrieval step could be adapted to any aligned text-acoustic input.","The hand-set annotation thresholds (AU difference 0.25/0.5, eyelid openness 4–9.5 mm, neck rotation 10°) may not be optimal across identities; per-singer calibration or learned thresholds could improve the dataset and the upper bound on quality.","The reported gains are aggregated metrics; a controlled user study that isolates lip-sync accuracy, upper-face emotion, and beat alignment against audio-only baselines would clarify which component (subtitles vs intensity proxy vs diffusion) drives human preference."],"forward_implications":["Offline LLM subtitle generation plus real-time (200+ FPS) motion inference makes expressive singing avatars practical for interactive use.","Users can edit animations by editing the motion-subtitle text, not by tweaking low-level parameters.","The SingMoSub benchmark (37 hours, 1.79% silence rate) provides a common testbed for future singing-driven head animation work.","The motion intensity proxy removes the need for dense vertex supervision, shrinking model size (22.14M parameters) relative to most baselines.","The framework works without pre-defined emotion labels or manual motion annotations, since subtitles are inferred from lyrics and acoustics."],"supporting_citations":[{"why":"The largest existing singing 3D head dataset (SingingHead) that SingMoSub is compared against and whose limitations motivate the new dataset.","marker":"[2]"},{"why":"MMHead, the most relevant prior work using textual motion descriptions for 3D facial animation; serves as both a baseline and the closest comparison for subtitle-style annotations.","marker":"[44]"},{"why":"FLAME, the parametric head model used as the target representation for all generated 3D motion in the paper.","marker":"[47]"},{"why":"EMOCA v2, the monocular 3D face reconstruction model used to extract FLAME parameters from in-the-wild singing videos, supplying the ground truth labels for the dataset.","marker":"[48]"},{"why":"Retrieval-augmented generation concept that AGRA builds on to ground LLM subtitle generation with external reference examples.","marker":"[55]"},{"why":"Chain-of-thought prompting that Sing-CoT adapts into a stepwise emotion-to-subtitle reasoning scheme.","marker":"[56]"},{"why":"Whisper ASR, used to produce time-aligned lyrics and audio segments that serve as the input to AGRA and Sing-CoT.","marker":"[58]"},{"why":"DDPM diffusion formulation that the motion generator follows for forward noising and reverse denoising.","marker":"[59]"},{"why":"CLIP text encoder, used to embed motion subtitle text into conditioning features for the region-specific mappers.","marker":"[60]"}],"fun_headline_variants":["LLM-written motion subtitles make singing avatars expressive","Singing heads gain emotion from LLM-orchestrated motion cues","Think2Sing: LLM motion subtitles improve singing avatar animation","Structured LLM subtitles drive realistic singing head motion"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The pipeline treats FLAME parameters extracted from monocular singing videos by EMOCA v2, and the subtitles derived from hand-set thresholds on those parameters, as ground truth for both training and evaluation; if these labels carry systematic error, the reported quality gains may reflect label bias rather than genuinely better animation.","fun_headline_variants_meta":{"raw":{"variants":["LLM-written motion subtitles make singing avatars expressive","Singing heads gain emotion from LLM-orchestrated motion cues","Think2Sing: LLM motion subtitles improve singing avatar animation","Structured LLM subtitles drive realistic singing head motion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000319,"raw_usage":{"total_tokens":1650,"prompt_tokens":768,"completion_tokens":882,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":811}},"tokens_in":512,"tokens_out":882,"duration_ms":9763,"temperature":1.0,"reasoning_tokens":811,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T11:41:47.191279+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Capture a small set of singing performances with marker-based 3D motion capture, retrain Think2Sing on the ground-truth FLAME parameters, and check whether the reported LVE/FVE/FID gains over audio-only baselines persist; if the gains vanish, they depend on the monocular label bias.","supporting_citations":[{"cited_title":"SingingHead: A Large-scale 4D Dataset for Singing Head Animation","cited_arxiv_id":"2312.04369","evidence_quote":"The largest existing singing 3D head dataset (SingingHead) that SingMoSub is compared against and whose limitations motivate the new dataset."},{"cited_title":"Mmhead: Towards fine-grained multi-modal 3d facial animation,","cited_arxiv_id":null,"evidence_quote":"MMHead, the most relevant prior work using textual motion descriptions for 3D facial animation; serves as both a baseline and the closest comparison for subtitle-style annotations."},{"cited_title":"Emoca: Emotion driven monocular face capture and animation,","cited_arxiv_id":null,"evidence_quote":"EMOCA v2, the monocular 3D face reconstruction model used to extract FLAME parameters from in-the-wild singing videos, supplying the ground truth labels for the dataset."},{"cited_title":"Retrieval- augmented generation for knowledge-intensive nlp tasks,","cited_arxiv_id":null,"evidence_quote":"Retrieval-augmented generation concept that AGRA builds on to ground LLM subtitle generation with external reference examples."},{"cited_title":"Chain-of-thought prompting elicits reasoning in large language models,","cited_arxiv_id":null,"evidence_quote":"Chain-of-thought prompting that Sing-CoT adapts into a stepwise emotion-to-subtitle reasoning scheme."},{"cited_title":"Robust speech recognition via large-scale weak su- pervision,","cited_arxiv_id":null,"evidence_quote":"Whisper ASR, used to produce time-aligned lyrics and audio segments that serve as the input to AGRA and Sing-CoT."},{"cited_title":"Denoising diffusion probabilistic models,","cited_arxiv_id":null,"evidence_quote":"DDPM diffusion formulation that the motion generator follows for forward noising and reverse denoising."},{"cited_title":"Learning transferable visual models from natural language supervision,","cited_arxiv_id":null,"evidence_quote":"CLIP text encoder, used to embed motion subtitle text into conditioning features for the region-specific mappers."}],"review_version":1}