{"id":"27bc2257-9a78-4cde-98b9-15a311c79532","arxiv_id":"2412.15646","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"By training appearance and motion LoRAs on separate layers and distilling from their single-LoRA teachers for 30 steps, the combined model produces videos with a custom subject and a custom motion.","lead":"CustomTTT is a method for generating a video that uses the motion from one reference clip and the look of a small set of reference images. It trains lightweight adapters on different parts of a video diffusion model, then applies a short test-time update to remove the artifacts that appear when the adapters are combined.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TTT objective is asymmetric: Eq. 4's appearance loss enforces the solo appearance teacher's arbitrary motion, while Eq. 6's temporal loss cancels appearance; no loss-wise ablation or gradient analysis demonstrates decoupled combination.","rationale":"The reader correctly identifies teacher fidelity as a risk. I sharpen it: the TTT losses are not merely trusting the teachers, they are asymmetric. Eq. 4 matches the full noise of the appearance teacher's latent, including that teacher's motion, while Eq. 6 uses the anchor transform to cancel appearance but no analogous transform cancels motion in Eq. 4. Consequently, even if both solo LoRAs are individually perfect, the two targets can conflict, and the 30-step optimization has no provable way to resolve the conflict. The absence of separate Lap/Ltp ablations and the undefined β and anchor in Eq. 5 compound this: the objective is not fully specified. The paper does provide qualitative layer-selection ablations and a small user study supporting the headline claim, so I do not reject it; however, the mechanism behind the central innovation is under-determined. My proposed check would settle whether the conflict is real. The reader's conditional verdict stands.","tokens_in":11079,"tokens_out":8566,"duration_ms":73603,"concrete_test":"Run three TTT variants on a fixed subset of the 13-object/18-motion dataset: (1) Lap only, (2) Ltp only, (3) both, plus the existing w/o-TTT baseline. Measure Motion similarity and Appearance similarity with the metrics of Table 1. Additionally, at TTT initialization, compute the Frobenius norms of ∂Lap/∂ΔW_t^{2,5} and ∂Ltp/∂ΔW_s^{2,6}; if the cross-gradients exceed, say, 50% of the corresponding diagonal gradients, the losses are not decoupled. If Lap-only lowers motion similarity below w/o TTT, the appearance loss entrenches the teacher's arbitrary motion and the central claim needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The weakest point is not teacher fidelity alone but the construction of the TTT objective in Sec. 3.4. Lap (Eq. 4) trains the combined model to predict noise added to a latent z_s^f generated by the solo appearance model with a related prompt c'. This target latent carries the appearance teacher's own motion (whatever the base model and c' produce), and Lap has no term to cancel or down-weight that motion. In contrast, Ltp (Eq. 6) explicitly removes appearance via the anchor transform φ, but the symmetry is not completed: nothing in Lap removes the appearance teacher's motion. Because z_s^f and z_t^f are generated independently, the two targets are generally incompatible (e.g., a panda sitting vs. a man lifting), so the two loss terms pull the combined LoRA weights in conflicting directions. The paper reports only the combined TTT result and 'w/o TTT'; it does not ablate Lap and Ltp separately, does not report the gradient of Lap w.r.t. ΔW_t^{2,5} or of Ltp w.r.t. ΔW_s^{2,6}, and gives no argument that the 30-step optimization resolves the conflict rather than merely finding a compromise. If Lap alone already degrades motion similarity relative to w/o TTT, the claimed joint customization is a fragile balance, not a principled decoupling.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes CustomTTT, a method for jointly customizing appearance and motion in a pre-trained text-to-video diffusion model (AnimateDiff). It first identifies spatial layers 2 and 6 and temporal layers 2 and 5 as important for appearance and motion via prompt-embedding replacement experiments, trains separate LoRA adapters on those layers, and then combines them with a test-time training stage. In this stage, reference latents generated by the solo appearance and motion LoRA models are used as distillation targets for the combined model, with an appearance preservation loss (Eq. 4) and a temporal preservation loss (Eq. 6). Experiments compare against DreamVideo, MotionDirector, DiffDirector, and full-LoRA baselines, reporting improvements in CLIP-T, CLIP-I, temporal consistency, and user-study scores.","tokens_in":11349,"tokens_out":6054,"duration_ms":49410,"significance":"If the results hold, the paper is a useful empirical contribution: it shows that layer-selective LoRA placement plus a short self-distillation refinement can reduce artifacts in multi-concept video customization, and it does so with fewer trainable parameters than the compared baselines. The code link and the ablation studies on layer placement, reference-latent step f, and number of training steps are positive elements, and the central idea is plausible and easy to reproduce. The contribution is empirical rather than theoretical; its significance depends on how robust the reported gains are, which is not yet established by the paper's statistics.","major_comments":[{"comment":"The TTT objective is asymmetric, and the paper does not ablate its two terms. Eq. (4) trains the combined model against a latent z_f^s sampled from the solo appearance LoRA, whose motion content is whatever that teacher happens to produce for prompt c'; Eq. (6) removes the appearance influence from the motion target via the anchor transform, but nothing in Eq. (4) removes or down-weights the appearance teacher's motion. Since z_f^s and z_f^t are generated independently, the two loss terms can pull the combined LoRA weights in conflicting directions. The reported experiments only compare 'with TTT' against 'w/o TTT' (Figs. 8 and 9); no separate L_ap-only or L_tp-only run, no gradient analysis, and no convergence study is provided. A minimal fix is to ablate each loss separately and to report gradient norms of L_ap with respect to ΔW_t^{2,5} and of L_tp with respect to ΔW_s^{2,6} to demonstrate decoupling.","section":"Sec. 3.4, Eqs. (4) and (6)"},{"comment":"The layer-importance analysis is qualitative and under-supported. The conclusion that spatial layers 2 and 6 control appearance and temporal layers 2 and 5 control motion rests on a handful of prompt-injection videos; no quantitative criterion (e.g., CLIP-I/CLIP-T or motion similarity under systematic layer combinations) is reported, and the temporal selection is a heuristic ('neighbor temporal layers at i=2,5') because the architecture lacks some modules. The passage 'We conduct many experiments... in the supplementary material' points to material not present in this arXiv version. Please either provide the supplement or quantify the layer-sweep results in the main paper.","section":"Sec. 3.2 and Figs. 4-5"},{"comment":"The main quantitative evidence is not statistically supported. Table 1 reports only point estimates for CLIP-T, CLIP-I, and temporal consistency, with no error bars, no number of evaluation videos per cell, and no significance tests; the user study uses 11 participants and 10 generated videos, and the table lists scores without variance or inter-rater statistics. Because the reported improvements over DiffDirector and Full LoRA are modest for some metrics (e.g., temporal consistency 0.971 vs. 0.978), the claim of consistent superiority needs repeated runs or per-item analysis. Please report means and standard deviations over seeds and a paired significance test for the user study.","section":"Table 1 and Sec. 4.2"},{"comment":"Teacher fidelity is not evaluated. The distillation targets are latents generated by the solo appearance and motion LoRA models; if these teachers already contain artifacts, missing attributes, or prompt leakage, the TTT phase will entrench those errors in the combined model. The paper's own limitation statement in Sec. 4.4 acknowledges failures when appearance and motion references differ greatly, but it does not test whether the solo teachers are the source of the failure. Please add an evaluation of the solo-LoRA outputs (e.g., whether each teacher preserves the intended attribute and avoids unintended motion/appearance leakage) before using them as references.","section":"Sec. 3.4 and Sec. 4.4"}],"minor_comments":[{"comment":"The caption contains the typo 'A teedy bear'; it should read 'A teddy bear'.","section":"Fig. 8 caption"},{"comment":"The phrase 'we invert 11 participants' should be 'we invite 11 participants' or 'we recruit 11 participants'.","section":"Sec. 4.1"},{"comment":"The column grouping is confusing because the text says 'We employ three metrics' but the table lists seven columns. The caption should explicitly separate objective metrics (CLIP-T, CLIP-I, Temporal consistency) from user-study metrics (Motion similarity, Appearance similarity, Prompt alignment, Video quality).","section":"Table 1"},{"comment":"β_anchor is described only as 'the selected frame as the anchor frame'; its value, how it is selected, and how Eq. (5) removes appearance are not specified. Please define β and β_anchor concretely.","section":"Eq. (5)"},{"comment":"The notation 'DDIM (x, ϵ(c′; ΔW_s^{2,6}))' introduces x without definition; state that x is the initial noise/latent and how the f-step DDIM sampling is run.","section":"Sec. 3.4"},{"comment":"The claim that AnimateDiff has no temporal layer at i=4 and no cross-attention at i=3,5 needs a short architecture description or a diagram reference so the layer indexing is self-contained.","section":"Sec. 3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the method is plausible, but the evidence for the central claims is thinner than the abstract suggests. The main revision should add quantitative layer sweeps, loss ablations, and statistics; without these, the claims of state-of-the-art performance are not fully supported. I see no grounds for rejection, but the revision needs to be substantive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CustomTTT is a practical recipe for joint appearance and motion customization in T2V diffusion, built on two solid findings: in AnimateDiff, appearance is mostly controlled by spatial layers 2 and 6, motion by temporal layers 2 and 5; and a short test-time distillation from the individual LoRA teachers cleans up merge artifacts. The reported gains over DreamVideo, MotionDirector, and DiffDirector are consistent across metrics, and the method uses fewer trainable parameters. That's a genuinely useful contribution.\n\nWhat's good: the layer-prompt injection analysis in Sec. 3.2 is clean and the ablation in Fig. 7 confirms the selected layers work. The TTT idea is well motivated: merge then adapt. The paper is honest about limitations (large appearance-motion gaps, small objects), and the user study adds a useful human-check, even if small.\n\nMain soft spot is the TTT objective's asymmetry. Eq. 4 (appearance preservation) matches the combined model to the solo appearance teacher's latent, which carries whatever motion the base model and prompt c' produce; there's no term to down-weight that motion. Eq. 6 (temporal preservation) removes appearance via the anchor transform, but the symmetry isn't completed. Because the two targets come from independently sampled latents, the losses can conflict. The paper never ablates L_ap and L_tp separately, nor examines gradients of each loss with respect to the other LoRA. The 30-step optimization might find a good compromise, but 'w/o TTT' is the only ablated baseline. A loss-wise ablation is needed to support the decoupling claim.\n\nSecondary: the layer selection is qualitative. Fig. 4-5 are convincing but few examples; there's no quantitative criterion, and the motion analysis found i=4 but training uses temporal layers 2 and 5 because no temporal module at i=4. Reasonable, but needs a sentence. Table 1 lacks error bars; the user study is 11 participants, 10 videos; the code link is unverifiable.\n\nTeacher fidelity is a real but mild concern; the paper's own limitation section covers it.\n\nSend to review. Ask for the loss ablation, error bars, and code/checkpoints. With those, the paper is a solid conference contribution.","headline":"A practical layer-specific LoRA recipe with a test-time distillation step that works empirically, but the TTT objective's asymmetry needs a loss-wise ablation before the decoupling claim is fully supported.","tokens_in":11917,"tokens_out":3625,"would_cite":false,"duration_ms":32238,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A short test-time training pass lets a pre-trained video diffusion model adopt a new subject's appearance and a reference video's motion at once, without the artifacts that come from merging separately trained adapters.","keywords":["text-to-video generation","diffusion models","LoRA customization","test-time training","appearance and motion disentanglement","layer importance analysis","model merging"],"falsifier":"Generate a solo appearance reference and a solo motion reference for a case where the subject and motion are very different, inspect the reference latents for artifacts or prompt leakage, and compare the combined model trained with those latents against a version trained with clean ground-truth references; if deliberately corrupting the teacher latents does not degrade the combined output, the distillation is not actually carrying the appearance and motion information.","tokens_in":10852,"feed_emoji":"🎬","tokens_out":6007,"duration_ms":45454,"temperature":0.7,"pith_summary":"The paper proposes a way to put two customized concepts into one text-to-video model without artifacts: train one low-rank adapter for appearance and another for motion, place them only on the layers that control each factor, and then run a brief test-time training step that uses the individually trained models as teachers. The author argues that a pre-trained video diffusion model can therefore generate videos combining a specific subject from a few images with a specific motion from a reference video. The central empirical claim is that this beats merging all LoRA weights or previous methods such as DreamVideo and MotionDirector in both automatic metrics and user studies.","feed_headline":"30 test-time steps merge a look and a motion into one video model","feed_subtitle":"CustomTTT pins appearance to spatial layers 2 and 6, motion to temporal 2 and 5, then distills both into one video.","key_machinery":"The load-bearing mechanism is layer-selective LoRA placement combined with teacher-distillation test-time training. A LoRA is a low-rank weight update written as $W = W_0 + \\triangle W = W_0 + AB$; here the appearance adapter is attached to spatial layers 2 and 6, and the motion adapter to temporal layers 2 and 5, chosen by a prompt-replacement experiment on AnimateDiff. After combination, the model is fine-tuned for 30 steps with two losses: appearance preservation compares the combined model's noise prediction to the noise that produced a DDIM reference latent from the solo appearance model, and temporal preservation compares a frame-anchored relative motion term $\\phi(\\epsilon_i) = \\sqrt{\\beta^2+1}\\,\\epsilon_i - \\beta\\,\\text{anchor}$ against the solo motion model's reference. This lets the combined model inherit each single concept without full weight merging.","core_discovery":"CustomTTT claims that in a text-to-video diffusion U-Net, appearance is controlled mainly by the spatial cross-attention layers at indexes 2 and 6, and motion by the temporal attention layers at indexes 2 and 5. Training the appearance LoRA only on spatial layers 2 and 6 and the motion LoRA only on temporal layers 2 and 5 decouples the two concepts. Directly inserting both LoRAs still leaves artifacts, so the paper adds a test-time training stage: generate reference latents from each solo LoRA model with DDIM, then update the combined LoRAs for 30 steps against an appearance preservation loss (Eq. 4) and a temporal preservation loss (Eq. 6). The resulting 12.12M-parameter adaptation achieves the reported best scores on text-video alignment, appearance similarity, motion similarity, temporal consistency, and user rankings.","pith_inferences":["The layer-importance findings may transfer to other U-Net-based video diffusion models with similar spatial and temporal module order, but that transfer is not tested in the paper; a future experiment could repeat the prompt-injection test on another backbone.","The distillation targets are assumed faithful: if a solo adapter's reference latents already contain artifacts or prompt leakage, the test-time training will entrench those errors in the combined model, and this teacher fidelity is not evaluated independently.","The temporal preservation loss, being appearance-anchored, could serve as a general regularizer for motion consistency in other video editing tasks beyond customization.","Part of the reported gains over baselines may come from the AnimateDiff backbone and dataset construction; re-running on a common backbone and fixed prompt set would isolate the contribution of test-time training itself."],"forward_implications":["Users can combine any few-image subject with any reference-video motion in a single base model using only 12.12M trainable parameters, fewer than the compared methods use.","The layer analysis suggests that prompt-conditioned video generation has a sparse control structure, with a couple of spatial layers driving appearance and a couple of temporal layers driving motion, which could guide where future adapters are inserted.","Teacher distillation from solo LoRA models offers a general recipe for merging any pair of individually trained adapters, not only appearance and motion adapters.","The 30-step test-time training stage is short enough for single-GPU use, making multi-concept customization practical in a typical research setting."],"supporting_citations":[{"why":"Supplies the LoRA low-rank adaptation method used for all appearance and motion adapters.","marker":"Hu et al. 2021"},{"why":"Provides the AnimateDiff base text-to-video model used for the layer analysis, training, and evaluation.","marker":"Guo et al. 2023"},{"why":"B-LoRA prompt replacement technique inspires the layer-importance analysis for appearance and motion.","marker":"Frenkel et al. 2025"},{"why":"DDIM sampling is used to generate the reference latents during test-time training and during inference.","marker":"Song, Meng, and Ermon 2021"},{"why":"MotionDirector is a main baseline and its temporal debiased loss inspires the temporal preservation loss.","marker":"Zhao et al. 2025"},{"why":"DreamVideo is the pioneer joint appearance-and-motion customization baseline that this work compares against.","marker":"Wei et al. 2024"},{"why":"Dreambooth supplies the appearance reference objects used in the evaluation dataset.","marker":"Ruiz et al. 2023"},{"why":"CustomDiffusion supplies additional appearance objects and prior multi-concept customization context.","marker":"Kumari et al. 2023"}],"fun_headline_variants":["30 test-time steps merge look and motion into one video","Layer-pinned LoRAs plus test-time training fuse appearance and motion","CustomTTT blends two LoRA concepts with 30-step test-time tuning","Fix multi-LoRA artifacts: train the combined model at test time","Appearance and motion decoupled by layer choice, then TTT combines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the assumption that the individually trained appearance and motion adapters produce clean, faithful reference videos; if those references already contain artifacts or wrong attributes, the test-time training will permanently copy those errors into the combined model.","fun_headline_variants_meta":{"raw":{"variants":["30 test-time steps merge look and motion into one video","Layer-pinned LoRAs plus test-time training fuse appearance and motion","CustomTTT blends two LoRA concepts with 30-step test-time tuning","Fix multi-LoRA artifacts: train the combined model at test time","Appearance and motion decoupled by layer choice, then TTT combines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000209,"raw_usage":{"total_tokens":1398,"prompt_tokens":927,"completion_tokens":471,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":377}},"tokens_in":543,"tokens_out":471,"duration_ms":4589,"temperature":1.0,"reasoning_tokens":377,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:13:51.049413+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate a solo appearance reference and a solo motion reference for a case where the subject and motion are very different, inspect the reference latents for artifacts or prompt leakage, and compare the combined model trained with those latents against a version trained with clean ground-truth references; if deliberately corrupting the teacher latents does not degrade the combined output, the distillation is not actually carrying the appearance and motion information.","supporting_citations":[],"review_version":1}