{"id":"6a7c8f4e-52bf-417e-9273-ba6fa101b5ca","arxiv_id":"2412.12093","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"CAP4D combines a morphable multi-view diffusion model with 3D Gaussian splatting to build animatable 4D head avatars from 1 to 100 reference images, claiming state-of-the-art results.","lead":"CAP4D is a system that builds a moving 3D portrait avatar from one to 100 photos of a person and animates it in real time. It uses a diffusion model conditioned on a 3D face model to invent consistent new views, then fits a 3D Gaussian avatar to those views.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never states that the nine Nersemble self-reenactment test subjects were excluded from MMDM training; if they were not, the single- and few-image gains may reflect memorization rather than generalization.","rationale":"The reader's weakest assumption is precisely the load-bearing concern: the train/test split between the MMDM training data and the Nersemble self-reenactment evaluation is not stated, and the paper's own description of training on Nersemble makes subject overlap highly plausible. This threatens the central generalization claim directly, because the MMDM is the generative engine whose output drives the 4D avatar reconstruction; if it has memorized the test subjects, the single- and few-image SOTA results in Table 1 are not evidence of generalization. The concern is concrete and testable by inspecting dataset metadata or rerunning on a disjoint split. I agree with the reader that this warrants a conditional verdict. I do not see a separate, equally load-bearing flaw in the method itself: the pipeline is clearly described, the ablations are informative, and the stochastic I/O conditioning idea is well motivated. The lack of released code and checkpoints is a reproducibility limitation but not a correctness attack on the central claim. Missing error bars are secondary and should be addressed in revision. Therefore the reader's CONDITIONAL verdict is appropriate and unchanged; the authors must clarify the split and, if needed, re-evaluate on held-out subjects.","tokens_in":24368,"tokens_out":3349,"duration_ms":32337,"concrete_test":"Request from the authors the subject-level IDs of the nine Nersemble test sequences and the complete MMDM training subject list, and verify that there is zero overlap. If overlap exists, rerun the single- and 10-reference-image self-reenactment evaluation on Nersemble subjects that were excluded from training (or on held-out subjects from another multi-view dataset such as Ava-256) and report Table 1 again. If CAP4D's margins over GAGAvatar and other baselines collapse or become statistically insignificant, the state-of-the-art claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—state-of-the-art 4D avatar reconstruction from one to 100 images—rests on the self-reenactment results in Table 1. Section 4 trains the MMDM on a pool that explicitly includes Nersemble (along with VFHQ, MEAD, Ava-256), and Section 5.1 evaluates on nine Nersemble sequences, holding out only 4 of 16 camera viewpoints. No subject-level train/test split is stated anywhere in the main text or supplement. Since training samples reference and target images uniformly across views and frames within a sequence (Supp. D), the 12 non-held-out viewpoints of those same nine subjects are almost certainly present in training unless the authors deliberately excluded them. The MMDM could then memorize subject-specific appearance, and the generated images used to fit the 4D avatar would be near-ground-truth for the held-out viewpoints. This would inflate CAP4D's single- and few-image metrics precisely where it claims to beat generalizable baselines. The baselines are not trained on Nersemble subjects, so the comparison would be unfair. The cross-reenactment experiment is less affected because identities come from FFHQ, but the headline generality claim is carried by the self-reenactment numbers. Missing error bars make it impossible to assess whether the reported margins are significant, compounding the risk. This is not an accusation of misconduct; it is a missing experimental detail that must be resolved before the central claim can be evaluated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CAP4D, a two-stage pipeline for reconstructing animatable 4D portrait avatars from one to hundreds of reference images. In the first stage, a morphable multi-view diffusion model (MMDM), initialized from Stable Diffusion 2.1 and conditioned on FLAME-based pose, expression, view-direction, and mask maps, generates many novel views with controlled expressions using a stochastic input/output conditioning procedure that alternates reference and generated image subsets across diffusion timesteps. In the second stage, the generated and reference images are used to fit a real-time 4D avatar based on GaussianAvatars, augmented with a UV-space deformation U-Net and an LPIPS loss. The method is evaluated on self-reenactment (Nersemble, with 1/10/100 reference images) and cross-reenactment (FFHQ references driven by VFHQ/Nersemble videos), reporting improved PSNR, LPIPS, CSIM, JOD, and human-preference results over several baselines, along with ablations of the main components.","tokens_in":24536,"tokens_out":7654,"duration_ms":61095,"significance":"If the empirical claims hold, CAP4D would be a valuable unified solution bridging single-image and multi-view avatar reconstruction, with a practical real-time rendering avatar and a generation stage that scales from one to hundreds of references. The stochastic I/O conditioning idea is interesting and the ablations are reasonably broad, covering conditioning signals, sampling strategy, reconstruction losses, and the number of generated views. The paper also presents failure cases and a brief ethics statement, which is commendable. However, the central generalization claim depends on the self-reenactment evaluation, and that evaluation has a potential training/test overlap with the MMDM's training corpus, as well as internal table inconsistencies that currently prevent a confident assessment of the method's actual performance.","major_comments":[{"comment":"The paper never states that the nine Nersemble sequences used for self-reenactment evaluation in §5.1 were excluded from the MMDM training corpus described in §4. Supp. D says that during training, 'we randomly select R reference images and G target images from all views and frames within a sequence with equal probability,' and the training pool explicitly includes Nersemble. If the evaluation sequences were among those training sequences, the single- and few-image results in Table 1 would reflect subject memorization rather than generalization, and the comparison would be unfair because the single-view baselines (Voodoo3D, GAGAvatar, Real3D, Portrait4D-v2) are not trained on Nersemble subjects. Please state explicitly whether a subject-level train/test split was enforced; if it was not, the evaluation needs to be rerun on held-out subjects before the central claim can be assessed.","section":"§4, §5.1, Supp. D"},{"comment":"The ablation study in §5.3 claims that all ablations in Table 3 use 10 reference images, but the '4D rep.' rows report PSNR 21.69, LPIPS 0.311, CSIM 0.633, JOD 5.67, which exactly match the single-reference CAP4D row in Table 1 (21.69, 0.311, 0.633, 5.672). In addition, the MMDM-only row in Table 1 at 10 references reports CSIM 0.804, while the corresponding 'sampling/Ours' row in Table 3 reports 0.779. These inconsistencies make it impossible to know which setting the 4D-representation ablations actually used; please correct the tables or state the setting used for each ablation.","section":"§5.3, Tables 1 and 3"},{"comment":"Algorithm 1 is internally inconsistent as printed. The first instruction in the t-loop shuffles generated latents into (Z'gen,t, C'gen), but the batch selection immediately overwrites these variables using the original, unshuffled (Zgen,t, Cgen)[iG'+1 : (i+1)G']. Moreover, the DDIM update inside the loop writes to Z'gen,t-1 for the current batch only, and there is no statement that reassembles Zgen,t-1 before the next outer iteration. Since the stochastic I/O conditioning is a central contribution and no code is provided, this pseudocode needs to be corrected so that the sampling procedure is reproducible.","section":"Algorithm 1"},{"comment":"Table 1 reports only point estimates over the nine Nersemble sequences, with no error bars, per-sequence results, or significance tests. Some reported margins are modest (e.g., single-reference PSNR 21.69 vs. GAGAvatar's 20.78; 100-reference PSNR 23.30 vs. FlashAvatar's 22.87), so the statement that CAP4D 'significantly outperforms every baseline' is not yet supported. Please provide variance estimates or per-subject results, particularly for the single- and few-image settings where the train/test overlap concern is most acute.","section":"§5.1, Table 1"}],"minor_comments":[{"comment":"Table 2 caption states the user study had 23 participants, while §5.2 and Supp. E.3 state 24; please correct the inconsistency.","section":"Table 2 caption, §5.2, Supp. E.3"},{"comment":"The caption of Table 1 describes 'single-image (left) and multi-image (right)' results, but the table has three blocks (single, 10, 100 reference images); please update the caption.","section":"Table 1 caption"},{"comment":"The Discussion (§6) says generation takes 'up to 8 hours,' while §4 reports 840-image generation taking ~4 hours on 4xRTX6000 GPUs; please clarify whether the 8-hour figure includes 4D avatar reconstruction.","section":"§6 vs §4"},{"comment":"The caption of Fig. S9 contains the typo 'Quanitity' for 'Quantity'.","section":"Fig. S9 caption"},{"comment":"The expression database used for sampling is built from the Nersemble dataset; please clarify that only expression parameters are used, not identity or appearance information, to avoid any appearance leakage through the sampling step.","section":"Supp. D"}],"recommendation":"major_revision","confidential_remarks":"The Table 3 versus Table 1 discrepancy (the '4D rep.' rows matching single-reference results exactly) looks like a copy-paste error, but it must be fixed before publication. The more serious issue is the missing subject-level split for the Nersemble evaluation; if the authors cannot confirm that the test subjects were excluded from MMDM training, the empirical core of the paper is substantially weakened. The pseudocode in Algorithm 1 also needs correction for reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the thing you should know: this is a well-built system with a real contribution, but the central self-reenactment numbers are on shaky ground until the authors clarify the Nersemble train/test split. They train the MMDM on Nersemble (among other datasets) and then evaluate on nine Nersemble sequences, holding out only 4 of 16 viewpoints per sequence. The paper never says the nine test subjects were excluded from training. Since training samples views and frames uniformly within a sequence, the other 12 viewpoints of those same subjects are almost certainly in the training set unless deliberately removed. That would let the diffusion model memorize subject appearance, and the generated 'novel views' would be near-ground-truth. The baselines are not trained on those subjects, so the comparison would be unfair. This is a missing experimental detail, not evidence of misconduct, but it is load-bearing: the single- and few-image SOTA claims rest on this table.\n\nWhat is genuinely new: the stochastic I/O conditioning procedure is a clean solution to the variable-reference-count problem, and the morphable conditioning stack (pose maps, expression deformation maps, view direction maps) plus the UV-remeshed GaussianAvatar with a deformation U-Net is solid engineering. The ablations are thorough, the user study is properly set up with chi-square tests, and the failure cases are honestly reported. The paper is well written and the method description is detailed enough to reproduce without code, though code/checkpoints would help.\n\nSofter spots: no error bars on any of the Table 1 metrics, so 'significantly outperforms' is unsupported. That is minor relative to the split issue but still worth fixing. The cross-reenactment experiment is less affected because the identities come from FFHQ, but it is also the weaker experiment—CSIM is mixed and the additional metrics are comparable to baselines. The user study carries the cross-reenactment claim.\n\nWho this is for: people working on single-image or few-shot avatar reconstruction, and anyone building multi-view diffusion pipelines with variable reference counts. It deserves a serious referee. If the split is confirmed clean, this is a conditional accept with minor revisions; if the split is leaked, the central claim needs rethinking. I'd ask the authors to state the split explicitly, add error bars or statistical tests, and release at least the MMDM checkpoints. As it stands, the paper is not ready to be taken at face value.","headline":"Strong engineering and a genuinely useful stochastic conditioning idea, but the headline self-reenactment claims currently hinge on an unstated train/test split that a referee must resolve.","tokens_in":25207,"tokens_out":2877,"would_cite":false,"duration_ms":24627,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CAP4D proposes a single morphable multi-view diffusion pipeline that reconstructs photoreal, animatable 4D portrait avatars from anywhere from one to one hundred reference images and renders them in real time.","keywords":["4D avatar reconstruction","multi-view diffusion model","3D Gaussian splatting","3D morphable model","portrait reenactment","novel view synthesis","stochastic conditioning","real-time rendering"],"falsifier":"Check the nine Nersemble evaluation sequence IDs against the training corpus and rerun the self-reenactment comparison with the model retrained on a strict subject-level holdout; if the single- and few-image advantages over the baselines disappear, the claimed generalization is not real.","tokens_in":24016,"feed_emoji":"🎭","tokens_out":6920,"duration_ms":60581,"temperature":0.7,"pith_summary":"This paper claims that one pipeline can turn any number of reference photos of a face—one, ten, or a hundred—into a single animatable 3D avatar that moves and renders in real time. The key idea is to condition a diffusion model not only on the input photos but on the estimated 3D shape, expression, and camera direction of every photo, so it can generate hundreds of new views with new expressions that stay consistent with the subject's identity. Those generated views are then distilled into a 3D Gaussian avatar whose expression-driven deformations are learned by a small network, giving a representation that can be driven by another person's video or by speech. If the central claim holds, content creators and visual effects pipelines could use the same method whether they have a single selfie or a studio capture rig.","feed_headline":"One pipeline turns 1 to 100 photos into animatable 4D avatars","feed_subtitle":"Morphable diffusion generates expression-controlled views; a Gaussian avatar animates them in real time.","key_machinery":"The morphable multi-view diffusion model (MMDM) is the central object: a latent diffusion model initialized from Stable Diffusion 2.1, with 3D attention across images and cross-attention removed, conditioned on six channels of per-image geometry: 3D pose maps, expression deformation maps, view-direction maps, and masks. It is what turns an arbitrary set of reference images into a large, expression-diverse set of self-consistent views. The second load-bearing mechanism is stochastic I/O conditioning: at each DDIM timestep, reference and generated latents are randomly shuffled and processed in batches, so all images participate jointly in denoising even though the network can only see a few at a time. Finally, the 4D avatar represents the subject as 3D Gaussians attached to a FLAME head mesh with a U-Net predicting expression-dependent UV deformations, which makes the distilled avatar animatable and real-time renderable.","core_discovery":"CAP4D's central discovery is that the gap between single-image and multi-view avatar reconstruction can be bridged by treating novel-view generation as a multi-view diffusion problem with 3D morphable-model conditioning, then distilling the generated images into an animatable Gaussian representation. The diffusion model takes up to four reference images at a time and generates images for target viewpoints, poses, and expressions specified by a FLAME model; a stochastic input-output conditioning loop shuffles and resamples reference and generated images at every diffusion timestep, so the same joint denoising process can absorb one or many reference images and emit hundreds of consistent novel views. These views are then used to optimize a 3D Gaussian splat avatar attached to a remeshed FLAME head, with expression-dependent corrective deformations predicted by a U-Net. The result is claimed to outperform prior methods on self- and cross-reenactment, to sharpen as more reference images are added, and to remain animatable and renderable in real time.","pith_inferences":["The stochastic I/O conditioning loop is not tied to heads: any multi-view diffusion model with a small context window could use the same shuffle-and-resample trick to generate hundreds of mutually consistent views for static scenes, objects, or full bodies.","Because the final avatar is only as expressive as the FLAME parameters that drive it, the method's animation range for hair, glasses, tongue, and jaw details is a likely ceiling; moving beyond mesh-bound Gaussians would be the natural next step.","The observation that the diffusion model plateaus while the final avatar keeps improving with hundreds of references suggests the generate-and-reconstruct decomposition is the part that scales, so future work should focus on cheaper generation rather than bigger diffusion models.","A decisive experiment not reported in the paper is a strict subject-level train/test split on the Nersemble data; the reported single- and few-image gains would be much more convincing if they survive that split."],"forward_implications":["A single captured image becomes a fully animatable 4D avatar, closing much of the fidelity gap with multi-view studio methods.","Adding more reference images improves the reconstructed avatar, and the same pipeline scales from one to hundreds without reconfiguration.","The avatar can be driven by another person's video or by speech, and the rendered output remains temporally consistent.","Because the MMDM conditions on 3D morphable-model parameters, avatars can be edited in 2D, such as with makeup or relighting, and then reanimated as 4D.","Single- and few-image self-reenactment is claimed to outperform prior single-view and multi-view baselines on photometric fidelity, identity preservation, and temporal consistency."],"supporting_citations":[{"why":"Supplies the closest predecessor: a morphable multi-view diffusion model for single-image 3D avatar reconstruction whose fixed-view limitation CAP4D lifts.","marker":"[18]"},{"why":"Provides the generate-and-reconstruct paradigm and the multi-view diffusion architecture that CAP4D adapts.","marker":"[31]"},{"why":"Provides the 3D Gaussian splatting avatar representation on which the second stage is built.","marker":"[69]"},{"why":"FLAME supplies the 3D morphable model whose shape, pose, and expression parameters drive both generation and animation.","marker":"[53]"},{"why":"Stable Diffusion 2.1 latent diffusion model is the base architecture and pretrained autoencoder for the MMDM.","marker":"[72]"},{"why":"Off-the-shelf multi-view head tracker estimates the 3DMM parameters and camera calibration used in conditioning.","marker":"[83]"},{"why":"DDIM sampling procedure is extended by the stochastic I/O conditioning loop.","marker":"[80]"},{"why":"Nersemble provides the multi-view training and evaluation data.","marker":"[50]"}],"fun_headline_variants":["CAP4D: From 1 to 100 images to animatable 4D avatars","Morphable diffusion builds animatable 4D avatars from any photo count","CAP4D: Single shot to 100-view real-time 4D avatars","Bridge the gap: CAP4D blends single-image and multi-view avatar craft","Animatable 4D avatars from 1 to 100 photos via morphable diffusion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the nine Nersemble subjects used for testing were not part of the 6,317 subjects the diffusion model trained on; the paper evaluates on Nersemble after training on Nersemble and reports only a camera-view holdout, not a subject-level split, so its single- and few-image gains could in part reflect memorized identities.","fun_headline_variants_meta":{"raw":{"variants":["CAP4D: From 1 to 100 images to animatable 4D avatars","Morphable diffusion builds animatable 4D avatars from any photo count","CAP4D: Single shot to 100-view real-time 4D avatars","Bridge the gap: CAP4D blends single-image and multi-view avatar craft","Animatable 4D avatars from 1 to 100 photos via morphable diffusion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000492,"raw_usage":{"total_tokens":2428,"prompt_tokens":966,"completion_tokens":1462,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":1351}},"tokens_in":582,"tokens_out":1462,"duration_ms":9649,"temperature":1.0,"reasoning_tokens":1351,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:17:00.513588+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check the nine Nersemble evaluation sequence IDs against the training corpus and rerun the self-reenactment comparison with the model retrained on a strict subject-level holdout; if the single- and few-image advantages over the baselines disappear, the claimed generalization is not real.","supporting_citations":[{"cited_title":"GaussianAvatars: Photorealistic head avatars with rigged 3D Gaussians","cited_arxiv_id":null,"evidence_quote":"Provides the 3D Gaussian splatting avatar representation on which the second stage is built."},{"cited_title":"Learning a model of facial shape and ex- pression from 4D scans","cited_arxiv_id":null,"evidence_quote":"FLAME supplies the 3D morphable model whose shape, pose, and expression parameters drive both generation and animation."},{"cited_title":"3D face tracking from 2D video through iterative dense uv to image flow","cited_arxiv_id":null,"evidence_quote":"Off-the-shelf multi-view head tracker estimates the 3DMM parameters and camera calibration used in conditioning."},{"cited_title":"Denois- ing diffusion implicit models","cited_arxiv_id":null,"evidence_quote":"DDIM sampling procedure is extended by the stochastic I/O conditioning loop."},{"cited_title":"Nersemble: Multi-view ra- diance field reconstruction of human heads","cited_arxiv_id":null,"evidence_quote":"Nersemble provides the multi-view training and evaluation data."}],"review_version":1}