{"id":"6b54eedd-8d7c-4a3f-8a54-8b70fb934c9a","arxiv_id":"2504.19189","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Sketch2Anim aligns 2D sketch keyposes and joint trajectories with 3D embeddings and uses a trajectory ControlNet plus keypose adapter to generate 3D motion clips from storyboards.","lead":"This paper introduces Sketch2Anim, a system that converts 2D sketch storyboards, composed of keypose drawings, joint trajectory strokes, and action words, into 3D character animations using a diffusion-based motion generator. It matters because it offers animators a direct path from rough sketches to editable 3D motion, replacing a manual, trial-and-error workflow.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"End-to-end claim rests on synthetic projections plus a few qualitative real examples; no quantitative real-storyboard test is provided, so the direct 2D-to-3D claim is not yet established.","rationale":"The reader's weakest assumption is essentially the same one I would isolate: the system is only quantitatively validated on synthetic projections of HumanML3D, with real storyboards shown qualitatively. This is the place where the central claim is least secure because the entire point is to handle user-drawn storyboards. The paper's own Sec. 7.4 limitation admits the upstream joint detector may fail, and no metric reports how often it succeeds on real drawings. The synthetic augmentation covers a limited span of variation (yaw +/- 45 degrees, pitch 0-30 degrees, Gaussian joint noise, body-part scaling), and all generated 2D inputs are by construction the projections of plausible 3D motions, so the difficult cases - drawings that are not near any in-distribution projection, occlusions, stylized proportions, inconsistent camera framing - are not tested. I also considered the inference-guidance issue, which is real (Table A3 shows roughly 45% improvement from guidance), but it is less fatal to the central claim because inference guidance is part of the proposed system; the more serious gap is lack of quantitative evidence on real target inputs. A controlled drawing-from-displayed-3D benchmark would directly test whether the synthetic results transfer. If the errors remain comparable, the central claim is supported; if they degrade substantially, the paper should be re-scoped to synthetic-style sketches or the system needs additional robustness modules. The user study and released code are genuine supporting evidence, but they do not replace a quantitative real-sketch evaluation.","tokens_in":28384,"tokens_out":11265,"duration_ms":123812,"concrete_test":"Build a real-storyboard benchmark: take 50-100 HumanML3D motions, render each as a 3D character from a fixed 3/4 orthographic view, ask several users to draw a corresponding stick-figure keypose and trajectory (with the action word supplied), then run the full Sketch2Anim pipeline and measure (a) Sketch2Pose joint-detection success rate, (b) MPJPE-3D of the generated keypose against the displayed 3D pose, and (c) 3D trajectory error against the displayed path. If the end-to-end MPJPE-3D is more than roughly 2x the synthetic test value (Table 1: 0.0478) or if detection fails on more than 10% of drawings, the central end-to-end claim should be re-scoped.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that a user can draw a storyboard and get a high-quality 3D animation directly. The load-bearing assumption is that the synthetic 2D conditions used in training and evaluation are representative of what users actually draw (Sec. 7 Dataset; Suppl. B). The quantitative benchmark is constructed by orthographically projecting HumanML3D motions, selecting a TMR-scored keypose, and adding only camera-view, joint-noise, and body-proportion perturbations. This makes every test 2D signal exactly realizable by an in-distribution 3D motion, which is precisely the favorable case. Real storyboard sketches appear only in qualitative results (Figs. 1, 6, 11, 12), and the user-study protocol does not state that the tested stimuli are real user drawings. Additionally, the pipeline depends on Sketch2Pose to convert raw strokes into the 22-joint keypose; Sec. 7.4 explicitly concedes Sketch2Pose 'might fail if the drawing deviates too much from a reasonable human character.' Since no real-sketch success rate or error distribution is reported, the end-to-end claim 'successfully transfers storyboards into high-quality 3D motions' is not quantitatively supported for the actual target inputs. The synthetic numbers may be entirely correct for projected HumanML3D motions and still overstate performance on real storyboards.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Sketch2Anim, a system for generating 3D human motion clips from 2D storyboard frames that contain a stick-figure keypose, optional joint-trajectory strokes, and an action word. The architecture has two modules: a multi-conditional latent diffusion motion generator, which combines a pre-trained motion diffusion model with a trajectory ControlNet and a trajectory-aware keypose adapter; and a neural mapper that aligns 2D and 3D keypose/trajectory embeddings so that 2D sketch conditions can be fed into the generator at inference time. The authors evaluate on the HumanML3D dataset with synthetic projection-based benchmarks, compare against three baselines, run ablations of the generator design and loss terms, and report a pairwise user study. They also demonstrate qualitative results on real hand-drawn storyboards and show a 3D editing application. The paper claims to be the first approach to adapt a motion diffusion model to generate 3D animations directly from 2D storyboards.","tokens_in":28711,"tokens_out":4390,"duration_ms":45276,"significance":"If the claims hold, the system is a practical contribution to sketch-based animation: it reduces manual keyframing by letting users specify a keypose, trajectory strokes, and an action word. The trajectory-aware keypose adapter is a reasonable design for combining multiple conditions in a pre-trained motion diffusion model, and the paper provides a substantial set of ablations (network module variants, loss terms, inference-guidance order), a user study, and releases code, data, and trained models. The central technical claims are, however, undermined by an apparent sign error in the alignment loss (Eq. 10), by the fact that the headline trajectory-control metric is the direct objective of a test-time L-BFGS optimization, and by the absence of a quantitative evaluation on real user sketches. These issues do not necessarily invalidate the system, but they must be addressed before the paper's broader claims can be accepted.","major_comments":[{"comment":"The alignment loss is written as Lmatch = -1/B * sum ||s3D - s2D||^2 and then included as a positive term in Lalign (Eq. 13), which is minimized during training. This drives the paired embeddings apart rather than enforcing them to be close, contradicting the sentence immediately before Eq. (10) that says the goal is to make paired embeddings 'as close as possible.' Please correct the sign (or clarify if the loss is instead maximized) and state explicitly which form was actually used in training, since this is the core mechanism of the 2D-3D neural mapper.","section":"Sec. 5, Eq. (10)"},{"comment":"The sharp improvement in Avg. Err.-2D and Avg. Err.-3D reported in Table 1 comes largely from a test-time L-BFGS inference guidance whose objective G(z_t, t, v, Tr_2D) is exactly the projected-trajectory error used as the evaluation metric. Table A3 shows that without inference guidance, Avg. Err.-2D is 0.158 (Average) and 0.159 (Cross), whereas with second-order guidance it drops to 0.0867 and 0.0792. The main comparison should report both guided and unguided numbers in the main text, and the text should be clear that the trajectory-accuracy numbers reflect a post hoc optimizer, not the learned controller alone. As written, the claim that the model 'effectively adheres to both keypose and joint trajectory constraints' is overstated for the unguided model.","section":"Sec. 6 and Suppl. C, Eqs. (15)-(16) and Table A3"},{"comment":"The quantitative benchmark is constructed by orthographically projecting HumanML3D motions, selecting a TMR-scored keypose, and applying camera, joint-noise, and body-proportion augmentations. This means every test 2D signal is, by construction, a valid projection of an in-distribution 3D motion. Real user sketches appear only in qualitative results (Figs. 1, 6, 11, 12), and while the paper shows robust Sketch2Pose detection in those examples, Sec. 7.4 explicitly concedes that Sketch2Pose 'might fail if the drawing deviates too much from a reasonable human character.' No real-sketch success rate, joint-detection error distribution, or end-to-end quantitative result is reported. To support the abstract's claim that the system 'successfully transfers storyboards into high-quality 3D motions,' the authors should provide a quantitative evaluation on authentic user drawings, or alternatively narrow the claim to inputs that pass the current Sketch2Pose front end.","section":"Sec. 7, Dataset processing, and Sec. 7.4 Limitations"},{"comment":"All quantitative results are single-run point estimates with no error bars, confidence intervals, or significance tests. Given that diffusion sampling is stochastic and that several comparative differences are modest (e.g., Cross-setting FID 0.577 for Ours vs. 0.738 for Lift-and-Control, R-precision 0.815 vs. 0.796 in Table A2), the strength of the cross-method and cross-ablation conclusions is not statistically established. The authors should report multiple random seeds (or similar) together with variance, at least for the headline comparisons.","section":"Tables 1-3 and Fig. 10"}],"minor_comments":[{"comment":"Eq. (8) uses the mask notation m'_i in both numerator and denominator, but the text defines only m_i. Please align the notation.","section":"Sec. 4.3, Eq. (8)"},{"comment":"The subsection title contains a typo: 'Trajetory-aware Keypose Adapter' should be 'Trajectory-aware Keypose Adapter.'","section":"Sec. 4.2, title"},{"comment":"The symbol Z(r') appears in Eq. (4) but Z is not defined in the main text at that point; a brief definition (zero-initialized linear layer) is given for the ControlNet in Sec. 4.1 but should be restated or referenced here for the adapter.","section":"Sec. 4.2, Eq. (4)"},{"comment":"The 'Ours' row in Table A3 differs from the 'Ours' row in Table 1 because inference guidance is disabled in A3. This is stated in the supplementary text, but it would help readers if the main-text caption also noted that Table 1 includes inference guidance.","section":"Table A3 and Table 1"},{"comment":"For the Motion Retrieval baseline, the FID is computed on retrieved real motions rather than on generated samples, which makes it not directly comparable to the generative FID numbers of the other methods. Please clarify this in the comparison description.","section":"Sec. 7.1, Motion Retrieval baseline"},{"comment":"The phrase 'we invent a neural mapper' is unusual for a research paper; consider 'we propose' or 'we design' instead.","section":"Abstract and Sec. 1"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue in the manuscript is the sign of Eq. (10): as written, minimizing Lalign would repel paired 2D/3D embeddings. I suspect this is a typographical error in the paper rather than in the implemented code, but the authors must confirm the implementation and correct the equation, because the neural mapper is a core contribution. Additionally, given that the trajectory-control metric is optimized at test time by L-BFGS, the unguided performance should be reported prominently in the main text. The paper's scope and topic are a good fit for ACM TOG; the system is potentially useful if these issues are resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What to know: this paper does something new and mostly works. It is the first to take the full combination of 2D sketch keyposes, joint trajectory strokes, and action words and produce 3D motion. The architecture is well motivated: instead of fusing two ControlNets, they add a keypose adapter that refines the trajectory ControlNet's residual features, and they align 2D and 3D conditions in a shared embedding space so training can use precise 3D surrogates while inference consumes 2D sketches. The ablations genuinely support these choices over single and double ControlNet alternatives, and the code, data, and models are promised. The user study is a step beyond what this subfield usually ships.\n\nThe soft spots are mostly in the evaluation, and the paper is fairly honest about them. The quantitative benchmark is built by orthographically projecting HumanML3D motions, selecting a TMR-scored keypose, and adding camera, joint-noise, and body-proportion perturbations. That makes every test 2D signal exactly realizable by an in-distribution 3D motion, which is the favorable case for the approach. Real storyboard sketches appear only in qualitative figures and in the user study; there is no reported success rate or error distribution for Sketch2Pose on real drawings. The authors explicitly concede in Sec. 7.4 that Sketch2Pose may fail on sketches that deviate from a reasonable human figure. So the end-to-end claim \"successfully transfers storyboards\" is not quantitatively pinned down for actual target inputs.\n\nA second concern: the trajectory control metric is also the objective of the L-BFGS inference guidance applied at test time (Eq. 15, Sec. C), so the headline Avg. Err. numbers partly reflect optimizer convergence. That said, Table A3 shows the method without inference guidance still beats all baselines on trajectory error, so the comparison is not hollow. Still, all tables are single-run point estimates with no error bars or significance tests, which is common in this literature but worth flagging. The user study reports 58 participants; motion-realism preference over retrieval is only 69%, which reads as an honest result.\n\nOverall, the technical core is sound and the limitations are stated rather than hidden. The paper is a good TOG-style systems contribution. I would send it to review, and I would ask the authors for a real-sketch quantitative evaluation (even a modest one) and for error bars or significance tests on the main comparisons, plus a clearer separation of learned control from inference-guidance optimization. For someone working on sketch-based animation or conditional motion synthesis, this is worth a close read.","headline":"Solid systems paper: the 2D-3D embedding mapper and keypose adapter are real contributions, but the end-to-end storyboard claim rests mainly on qualitative evidence because the quantitative evaluation uses synthetic projections.","tokens_in":29218,"tokens_out":2142,"would_cite":true,"duration_ms":23266,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims a first: a sketch storyboard — stick-figure keypose, traced joint trajectories, and an action word — can drive a motion-diffusion model to generate a 3D animation clip directly, with no manual keyframing and no 3D…","keywords":["sketch-based animation","storyboard","motion diffusion model","conditional motion synthesis","keypose control","trajectory control","2D-3D embedding alignment","motion editing"],"falsifier":"Run the full pipeline on a corpus of authentic hand-drawn storyboards spanning varied styles and drawing quality, and measure the failure rate of Sketch2Pose joint detection and the resulting motion quality against the Table 1 numbers; the end-to-end claim fails if detection errors on genuine cartoons are frequent or if quality drops substantially relative to the synthetic-projection evaluation.","tokens_in":1754,"feed_emoji":"🎬","tokens_out":2104,"duration_ms":97142,"temperature":0.7,"pith_summary":"Sketch2Anim claims to be the first system that transfers a 2D sketch storyboard into 3D animation by treating the storyboard as three conditioning signals: a stick-figure keypose, joint-trajectory strokes, and an action word. The payoff, if the claim holds, is that animators can prototype motion by drawing rather than by manually posing joints in 3D software through trial and error. The paper's core trick is to avoid lifting 2D drawings into 3D altogether: it trains a motion generator on precise 3D keyposes and trajectories, then trains a neural mapper that aligns 2D sketch embeddings with those 3D conditions in a shared embedding space, so inference can consume 2D drawings directly. Experiments on synthetic projections of a motion dataset and on real user sketches, plus a perceptual study, back the claim.","feed_headline":"Draw a stick figure, get a 3D animation","feed_subtitle":"Storyboard artists point and draw; the diffusion model generates the motion — no manual keyframing.","key_machinery":"The load-bearing machinery is the 2D-3D neural mapper plus the two-module conditioning stack. The neural mapper is a pair of 2D encoders trained to push 2D keyposes and trajectories into the same embedding space as the frozen 3D encoders, using a matching loss, a CLIP-style contrastive loss, and a noise-reconstruction loss that couples alignment to generation. The conditioning stack pairs a trajectory ControlNet — a trainable copy of the diffusion model's transformer encoder whose residual features steer the frozen denoiser — with a trajectory-aware keypose adapter, a second trainable copy that consumes the ControlNet's residuals plus a grounded action embedding $\\mathbf{a}' = \\mathbf{a} + E^{3D}_k(K^{3D})$ and outputs corrective residuals. This division lets the trajectory control carry global dynamics while the adapter pins the local pose at the keypose timestep, which the ablation shows beats both a single fused ControlNet and two parallel ControlNets.","core_discovery":"On its own terms, the paper establishes a conditional motion-generation recipe: a pre-trained latent motion diffusion model, frozen, is steered by two lightweight trainable modules — a trajectory ControlNet that injects joint-trajectory residuals, and a trajectory-aware keypose adapter that refines those residuals with the keypose condition. Because the keypose is treated as a grounding of the action word (its embedding is added to the text embedding), the adapter adds local pose constraints without disturbing the trajectory's global dynamics. A second training stage aligns 2D and 3D keypose and trajectory encoders in a shared embedding space using a matching loss, a noise-reconstruction loss, and a contrastive loss, so that at inference the user's 2D sketch lands exactly where the 3D conditions used in training do. The result is the claimed first direct 2D-storyboard-to-3D-motion transfer, with reported trajectory errors roughly halved and FID roughly five times better than training the generator directly on 2D conditions.","pith_inferences":["The train-on-3D-surrogates, infer-on-2D pattern is a general recipe: any diffusion generator with well-defined 3D conditions could gain a sketch interface the same way, from scene layout to character-object interaction, without retraining the base model.","The reported asymmetry — keyposes converge nearly ten times faster than trajectories — suggests a design principle that likely extends beyond this paper: when fusing controls with very different learning speeds, layer the slower, static condition as a refinement of the faster, dynamic one's residuals.","Because the quantitative evaluation uses synthetic orthographic projections rather than authentic cartoons, feeding real hand-drawn storyboards through the mapper during alignment training, using detection confidence as a per-joint weight, is a plausible route to closing the domain gap the authors acknowledge.","Retrieval nearly tied generation on perceived realism in the user study, so a hybrid that retrieves a close motion and then regenerates it under the sketch constraints might combine retrieval's naturalness with the generator's controllability."],"forward_implications":["An animator can go from sketch to motion clip in about half a second of generation time: a 40-frame clip conditioned on detected 2D joints, traced trajectories, and the action word, with no manual keyframing.","Generated clips become directly editable in 3D: dragging keypose or trajectory points produces an updated motion, because the generator consumes these conditions natively.","Separating trajectory dynamics from keypose statics — rather than fusing both into one control signal — is what the ablations attribute the control gains to.","Storyboard frames compose into a single coherent animation through inversion-based blending, making the pipeline operate on whole storyboards, not just isolated frames.","On the reported benchmarks, 2D-3D embedding alignment beats direct 2D-conditioned training by roughly fivefold in FID and halves trajectory error, and beats explicit 2D-to-3D lifting on every reported metric."],"supporting_citations":[{"why":"Supplies the pre-trained latent motion diffusion model that Sketch2Anim freezes and conditions, the backbone of the whole generator.","marker":"[Chen et al. 2023]"},{"why":"Sketch2Pose provides the 2D joint detection that turns raw storyboard strokes into the keypose input; the paper's claimed robustness rests on it.","marker":"[Brodt and Bessmeltsev 2022]"},{"why":"OmniControl supplies the trajectory-ControlNet conditioning pattern and the Average/Cross evaluation protocol the paper adopts.","marker":"[Xie et al. 2024]"},{"why":"HumanML3D provides the training motions, the redundant motion representation, and the text-motion vocabulary.","marker":"[Guo et al. 2022a]"},{"why":"CLIP embeds the action word into the conditioning space and inspires the contrastive loss used to align 2D and 3D embeddings.","marker":"[Radford et al. 2021]"},{"why":"ControlNet contributes the frozen-copy, zero-initialized-residual design used by both the trajectory ControlNet and the keypose adapter.","marker":"[Zhang et al. 2023a]"},{"why":"TMR is the retrieval baseline the method must beat and is also used to select representative keyposes when building the synthetic dataset.","marker":"[Petrovich et al. 2023]"}],"fun_headline_variants":["Sketch to 3D animation: direct transfer via neural mapper","Storyboard sketches become 3D animation directly","2D sketch to 3D animation: direct neural mapping","From storyboard sketch to 3D motion in one step"],"cache_read_input_tokens":31360,"weakest_assumption_plain":"The load-bearing premise is that Sketch2Pose will reliably find the 2D joints in whatever the user draws, and that orthographically projected motion-capture data with camera, joint, and body-proportion perturbations adequately stands in for real storyboard drawings — the authors themselves concede that joint detection \"might fail if the drawing deviates too much from a reasonable human character.\"","fun_headline_variants_meta":{"raw":{"variants":["Sketch to 3D animation: direct transfer via neural mapper","Storyboard sketches become 3D animation directly","2D sketch to 3D animation: direct neural mapping","From storyboard sketch to 3D motion in one step"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000991,"raw_usage":{"total_tokens":4238,"prompt_tokens":1020,"completion_tokens":3218,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":3149}},"tokens_in":636,"tokens_out":3218,"duration_ms":19925,"temperature":1.0,"reasoning_tokens":3149,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:58:58.071506+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full pipeline on a corpus of authentic hand-drawn storyboards spanning varied styles and drawing quality, and measure the failure rate of Sketch2Pose joint detection and the resulting motion quality against the Table 1 numbers; the end-to-end claim fails if detection errors on genuine cartoons are frequent or if quality drops substantially relative to the synthetic-projection evaluation.","supporting_citations":[],"review_version":1}