{"id":"c6ebe788-5e64-47d7-8767-51867c5f8d94","arxiv_id":"2504.19165","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"IM-Portrait generates 3D-aware talking head videos by directly diffusing Multiplane Images, trained on monocular video without multi-view data.","lead":"A new system generates 3D-ready videos of a talking person from a single photo plus facial expression signals, using an AI diffusion model that directly outputs layered images for novel viewpoints. It is designed for VR and immersive viewing because the two eye views can be computed from the same generated representation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The pseudo-ground-truth bootstrap in Sec. 3.3 can imprint the model's own color-branch errors into the geometry supervision, so the claimed 3D consistency is not yet separated from self-bias.","rationale":"The reader identified the same load-bearing weakness: the self-bootstrapping pseudo ground truth in Sec. 3.3, combined with late-stage noise sampling, is the least secure condition for the paper's central claim that 3D shape is learned from monocular videos. I find no additional flaw that would push the verdict to rejection. The concern is a real risk because the paper's only quantitative 3D evaluations (Fig. 5 disparity maps and Table D side-view FID) do not measure agreement with true multi-view geometry, and the supplementary explicitly acknowledges color-tone drift in the pseudo ground truth. However, it is not a demonstrated failure: the model may still learn useful geometry from the target-view loss, the mask and disparity regularizers, and the 3DMM-conditioned alpha planes. The conditional verdict remains appropriate. The proposed multi-view ablation would directly settle whether the pseudo-GT signal is biased or benign, and would also address the reviewer's request for stronger validation of the geometry branch. Independent support in the paper is limited by the absence of code and checkpoints, but that is an availability issue rather than a correctness flaw in the argument.","tokens_in":17537,"tokens_out":6169,"duration_ms":72316,"concrete_test":"Acquire a small multi-view or synthetic talking-head set with known camera poses and ground-truth geometry, and compare three variants: (i) the published pseudo-GT training, (ii) the same training with the reference-camera pseudo ground truth replaced by the real reference-camera frame from the multi-view capture, and (iii) a target-camera-only variant with the reference branch disabled. Measure held-out novel-view error directly (L1, SSIM, and disparity error against the true geometry), not only FID. If variant (i) shows markedly worse 3D error than (ii) or does not improve over (iii), the pseudo-GT bootstrap is injecting bias and the central 3D-consistency claim is weakened; if (i) and (ii) are close, the bootstrap concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing condition is that the reference-camera pseudo ground truth \\hat{O}^{mpi}_k in Sec. 3.3 is a reliable proxy for a real reference-camera observation, so that the target-frame loss L_side (Eq. 7) supervises true geometry rather than the model's own sampling bias. Because \\hat{O}^{mpi}_k is produced by the same network being trained, any systematic error in the color branch enters the geometry branch's training target. The supplementary material (Sec. F) explicitly admits 'drifting in the color tone' in the pseudo ground truth. Late-stage noise sampling (t sampled from [T/2,T]) only hides the distribution mismatch at high noise levels; it does not remove the bias at the smallest t actually used, and L_side is selected with probability 0.2, meaning the geometry branch is repeatedly trained against renderings generated by the model itself. The reported 3D evidence—stereo disparity maps in Fig. 5 and side-view FID in Table D—is consistent with plausible geometry, but it is also consistent with the model having learned a smooth depth prior keyed to 3DMM UV maps, with the pseudo-GT enforcing appearance rather than true shape. No held-out multi-view or known-geometry experiment is reported, so the central claim that genuine 3D shape is learned from monocular video, rather than imprinted by the bootstrap, is not yet separated from this confound.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes IM-Portrait, a video diffusion model that generates talking-head videos as Multiplane Image (MPI) videos in a single denoising pass, conditioned on a reference identity image and 3DMM-based expression/pose controls. The key architectural claim is that the model outputs an MPI sequence directly, so novel views (stereo pairs, perspective shifts) can be rendered by MPI compositing without a separate 3D reconstruction stage, NeRF fitting, or Gaussian splatting. To train from monocular in-the-wild videos, the authors introduce an alternating reference-target training scheme: when MPIs are constructed in the target camera, the target frame supervises the frontal view (Eq. 6); when MPIs are constructed in the reference camera, the model bootstraps its own pseudo ground truth and supervises a side view (Eq. 7), which is the only explicit multi-view geometric supervision. Experiments compare against 2D talking-head methods, a NeRF-based baseline, and a 2D-diffusion-plus-depth baseline, reporting competitive image/video metrics, the best FVD on several datasets, efficient rendering, and qualitative stereo/perspective results.","tokens_in":17823,"tokens_out":3967,"duration_ms":44543,"significance":"If validated, this is a useful step toward 3D-aware generative video: directly producing a renderable 3D representation from a single diffusion process, without multi-view training data or post-hoc reconstruction, is a clean and potentially practical formulation for VR/AR talking-head applications. The paper reports broad experiments across multiple datasets, several baselines, an ablation of the residual image, an ablation of late-stage noise sampling, and a comparison against a NeRF-based 3D baseline, and it ships a large body of qualitative evidence including side views and disparity maps. The rendering speed advantage (109 FPS at 512x512) is concrete and well aligned with the stated application. However, the load-bearing geometric claim rests on a self-supervised bootstrap whose validity is not independently established, and the evaluation does not measure geometric accuracy against ground-truth novel views or known geometry, so the 3D-consistency claim currently remains plausible but not fully verified.","major_comments":[{"comment":"The only multi-view supervision for geometry is L_side in Eq. (7), where the side-view ground truth is the image O^mpi_k rendered from the MPIs generated by the model itself in the reference camera. This is a self-training bootstrap, not independent supervision. The authors acknowledge in Supp. Sec. F that the pseudo ground truth images exhibit \"drifting in the color tone,\" and the late-stage noise sampling t in [T/2,T] only down-weights, rather than removes, the mismatch at the smallest t that is actually used. Because the same network produces both the color and geometry branches, systematic color-branch errors can enter the geometry branch's training target, so the reported 3D consistency may partly reflect the model's internal bias rather than true shape. The paper should provide a concrete test that separates these: for example, a held-out multiview or known-geometry experiment (e.g., render MPIs from a known mesh and compare disparity or depth error), or an ablation in which L_side is supervised by real multiview frames instead of pseudo ground truth on a small dataset, quantifying how much geometry quality degrades.","section":"Sec. 3.3, Eq. (7), Fig. 3; Supp. Sec. F"},{"comment":"The reported novel-view evaluation does not measure geometric accuracy against ground truth. The stereo disparity maps in Fig. 5 are estimated from the model's own rendered images with RAFT-Stereo, so they can look plausible even if the depth is a smooth prior rather than correct geometry. The side-view FID in Table D compares distributions of generated images, not the correctness of the rendered geometry. The paper claims parity with the NeRF-based Portrait4D-v2 on 3D effects, but no metric on depth, disparity, or novel-view PSNR/SSIM against ground-truth images of a known head model or a multiview dataset is reported. I ask the authors to add a quantitative geometry metric (e.g., disparity error on a synthetic or lightstage-like sequence, or novel-view PSNR/SSIM against a held-out multidataset) and to report such a metric also for the ablations in Sec. 4.3, where the claim that the residual image improves geometry is currently supported mainly by the qualitative Fig. 6.","section":"Sec. 4.2, Sec. G, Table D, Fig. 5"},{"comment":"Several quantitative claims are made without error bars or significance tests. For example, in Table 1 the HDTF LPIPS difference between Ours (0.118) and X-Portrait (0.119) is within any reasonable run-to-run variation, and in Table A the FID differences between Ours and X-Portrait on VFHQ (33.10 vs. 26.22) are opposite in sign to the HDTF ranking. The central claim 'best FVD overall' should be supported by variance estimates across evaluation subsets or multiple training runs, and the authors should state how many evaluation clips and frames each metric uses, especially for the self-collected dataset (50 identities, 32 frames each). This is a standard reproducibility requirement and would strengthen the robustness of the performance comparisons.","section":"Table 1, Table A, Supp. Sec. C"}],"minor_comments":[{"comment":"Please fix typos and notation inconsistencies: 'pesudo' (Sec. 3.3), 'specifiv' (Supp. Sec. A), 'X-xportrait' (Sec. 4.2), and the inconsistent naming 'Talkinghead1kh' (main paper) vs 'Talkinghead1Kh' (elsewhere). Also, the parameter 'r' for near/far plane placement in Supp. Sec. A should be defined precisely (distance from MPI frontal camera to which 3DMM head joint?).","section":"Throughout"},{"comment":"The equation environment for Eq. (7) is not typeset correctly (the L_side definition appears inline as 'Lside θ = EOref,Ompi,Cmpi,t,Oside,ϕside[∥Oside− ˆOside t−1∥2]'), and it is not consistent with the LaTeX formatting of Eq. (6). Please ensure both equations are rendered with the same style.","section":"Sec. 3.3"},{"comment":"Since the method is trained on a self-collected 35k-video dataset that is not released, and the code is not released, the paper should include an explicit reproducibility statement describing the data collection and any plans to release code or pretrained models.","section":"Sec. 4 and Supp. Sec. A"},{"comment":"The description of the top and bottom branches in the figure caption is helpful, but the caption should clarify which branch corresponds to Eq. (6) and which to Eq. (7) in the main text to make the connection immediate.","section":"Fig. 3"}],"recommendation":"major_revision","confidential_remarks":"The paper has an industrially polished presentation and a broad evaluation, but the central 3D claim depends on the bootstrapped pseudo-GT mechanism in Sec. 3.3. The supplementary material's own admission of color-tone drift in the pseudo GT, combined with the absence of any ground-truth geometric evaluation, makes the '3D consistency' claim vulnerable to a self-bias confound. I recommend requesting the authors to add an independent geometric validation before acceptance, even if only on a small held-out multiview or synthetic dataset."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the core idea—training a video diffusion model to emit Multiplane Image video directly, in one denoising pass, from monocular in-the-wild videos—is a genuine step forward. It removes the per-subject optimization and post-hoc 3D lifting stages that dominate the 3D avatar literature, and the 109 FPS render speed at 512^2 makes it practically interesting for VR. The reference-target alternating training with bootstrapping is the kind of trick that only sounds obvious after someone does it.\n\nCredit where due: the experiments are broad, covering four datasets, baselines across three method families, ablations, side-view FID, stereo and perspective rendering. Best FVD on both HDTF and Talkinghead1kh is a real result, not a cherry-picked metric. The side-view FID (18.12 vs 27.83 for Portrait4d-v2) is the strongest evidence that the MPI carries actual geometry. The residual-plane ablation is a thoughtful control: it shows the residual image is not cosmetic but contributes to geometric quality.\n\nSoft spots. The pseudo-GT bootstrap in Sec. 3.3 is the right thing to worry about. The supplementary Sec. F admits color-tone drift in the pseudo GT, and the late-stage noise sampling (t from [T/2, T]) does not remove the bias, it just hides it behind noise. That said, the circularity is not total: the disparity loss, supervised by 3DMM mesh disparity, gives the geometry branch an external anchor. So the model is not learning pure 3D from monocular video; it is learning to refine a 3DMM prior. The paper should state this more prominently. The bigger runtime issues are missing code/data and no error bars. Metrics are reported once, without standard deviation, so it is hard to know whether the FVD gap over the runner-up is real or noise. The FID on Talkinghead1kh is markedly worse than X-Portrait and Follow-your-emoji (27.83 vs 19.86 and 19.37); the authors call the overall result comparable, which is fair, but 'comparable or superior' leans on FVD.\n\nWho this is for: people building one-shot talking heads for VR/AR, and anyone interested in diffusion models emitting layered 3D representations rather than flat images. It deserves a serious referee. I would send it out, but ask for code/checkpoints, error bars, and at least one held-out multi-view validation of the geometry branch—for example, rendered stereo pairs compared against real stereo, or a synthetic head with known geometry. The central idea holds up; the validation needs tightening.","headline":"Single-pass MPI video diffusion for talking heads is a real advance and the paper deserves a serious referee, but the bootstrapped geometry supervision is under-validated and the missing code/data makes the quantitative claims hard to trust.","tokens_in":18366,"tokens_out":2564,"would_cite":true,"duration_ms":27456,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a video diffusion model trained only on monocular clips can generate talking-head videos as Multiplane Images, so novel views render in one pass without reconstruction.","keywords":["talking head synthesis","video diffusion models","multiplane images","monocular video training","3D-aware generation","novel view synthesis","face reenactment","VR stereo rendering"],"falsifier":"Using synchronized multi-camera footage of a subject, render side views from the MPIs at increasing baselines and compare MPI-derived disparity with ground-truth disparity; if disparity error grows sharply as the baseline increases while frontal quality stays high, the reference-camera bootstrap is encoding appearance bias rather than true geometry. A cheaper check on public data is to compare the MPI disparity in rigid facial regions such as the forehead and nose with the fitted parametric-model depth at matched head poses and see whether disagreement falls inside the render tolerance.","tokens_in":1872,"feed_emoji":"🎭","tokens_out":2427,"duration_ms":113515,"temperature":0.7,"pith_summary":"The paper tries to establish that a single video diffusion model, trained only on monocular in-the-wild videos, can generate talking-head videos that are already 3D: the output is a Multiplane Image (MPI) video, so stereo pairs and perspective shifts are rendered by one differentiable compositing pass instead of a separate NeRF or Gaussian reconstruction. This matters because existing diffusion-based talking-head methods are 2D and need extra 3D lifting, while 3D avatar methods do not inherit diffusion's image quality. The paper's load-bearing solution is reference-target alternating training, in which the model sometimes builds the MPI in the target camera (supervised by the true frame) and sometimes in the reference camera (supervised by its own bootstrapped pseudo ground truth under late-stage noise), letting one network learn sharp appearance and view-consistent geometry at once. The authors report competitive image and video quality against 2D diffusion baselines and novel-view rendering comparable to a NeRF-based one-shot avatar baseline, at substantially faster render speed.","feed_headline":"Diffusion draws 3D-ready talking heads in one pass","feed_subtitle":"A single denoising run outputs Multiplane Images, so VR views need no reconstruction step.","key_machinery":"The load-bearing object is the MPI video, defined as a sequence of D RGBA planes parallel to a frontal camera, equally spaced in disparity, where rendering from any nearby camera is a homography warp plus $\\alpha$ blend. The argument is carried by a video diffusion U-Net that outputs an MPI video in the reverse process: the network predicts MPI frames, a differentiable renderer turns them into the estimated clean frontal video, and that rendered video plugs back into the denoising update. The color branch predicts frontal and residual RGB, the geometry branch predicts per-plane $\\alpha$, and zero-convolution lets information flow from color to geometry so the appearance branch can bootstrap the shape branch. Training alternates two MPI camera placements, the target camera (ground-truth images, 80% of iterations) and the reference camera (pseudo ground truth from the bootstrapped model, 20% of iterations), with late-stage noise sampling $t \\in [T/2, T]$ suppressing the pseudo image's flaws. This mechanism is what converts monocular head-pose variation into weak multi-view supervision, and it is also what makes the whole pipeline a single denoising run.","core_discovery":"The paper's central claim is that 3D-aware talking-head video can be generated as Multiplane Image (MPI) video in a single denoising pass from monocular training data. Each generated frame is an MPI: a stack of RGBA planes parallel to a chosen frontal camera; warping those planes by homographies and alpha-compositing them yields the image in any nearby camera. The model is conditioned on one identity image and expression and head-pose controls from a parametric head model, and its network has a color branch (predicting frontal and residual RGB) and a geometry branch (predicting per-plane alpha), coupled by zero-convolution. Training on ordinary monocular videos alternates between building the MPI in the target camera, where the ground-truth frame supervises sharp appearance, and in the reference camera, where the target frame serves as a side-view ground truth for geometry; the reference-camera branch is fed a pseudo ground truth generated by the bootstrapped model itself, with noise sampled only from the late half of the diffusion schedule to hide its imperfections. The result is that novel-view rendering, including stereo pairs and camera push-ins, is a direct render call on the denoised MPI video rather than a separate reconstruction or optimization step.","pith_inferences":["If the central claim transfers beyond faces, the alternating-camera bootstrap is a general recipe: any monocular video with known relative camera pose between frames could train a single-pass MPI diffusion model for objects, hands, or bodies; this is an inference, not a paper claim.","The paper's own limitation statement about excessive viewpoint changes implies the practical promise is modest-baseline VR; a natural next test would be the same denoising-with-rendering loop on a 3D Gaussian or mesh representation to see whether the view range is bounded by MPI or by the training scheme.","A tension the paper does not resolve is that the pseudo ground truth used to supervise geometry is generated by the same appearance model whose errors it is supposed to correct; in practice this makes the claimed 3D consistency an upper bound on the bootstrap's structural accuracy, and a small multi-view validation set would expose how tight that bound is."],"forward_implications":["Novel stereo rendering becomes a byproduct: after one denoising run, binocular or shifted-camera views are produced by alpha-compositing the MPI planes, with no per-view sampling or 3D reconstruction.","Training needs only ordinary monocular talking-head videos; the alternating reference and target camera scheme supplies weak multi-view supervision from head-pose variation, so the method can scale to large in-the-wild collections.","At inference the paper reports roughly 109 FPS at 512x512 for novel-view rendering of its MPI pipeline, about an order of magnitude faster than the NeRF-based Portrait4D-v2 baseline, while still producing visually comparable stereo disparity.","Long videos are stitched autoregressively: each clip feeds its last generated frame to the next clip as a first-frame condition, with a scheduled guidance scale that prevents error accumulation across clips.","Because MPI planes can be composited by a differentiable renderer, the same network can be trained end-to-end without ground-truth multi-view video, which is the main practical enabler of the method."],"supporting_citations":[{"why":"Supplies the denoising diffusion forward and reverse process and the noise schedule used in the paper's equations.","marker":"[25]"},{"why":"Source of the diffusion-with-forward-model idea, inserting a differentiable renderer into the denoising loop, and of the frontal-view and side-view two-loss supervision scheme.","marker":"[37]"},{"why":"Defines the MPI representation, the frontal-plus-residual RGB compositing, and the disparity and edge-aware smoothing losses used in training.","marker":"[39]"},{"why":"Provides the Space-Time U-Net video diffusion backbone and the first-frame-conditioned clip chaining for long video generation.","marker":"[5]"},{"why":"The NeRF-based one-shot talking-head baseline whose novel-view quality and render speed are compared.","marker":"[13]"},{"why":"A 2D video diffusion talking-head baseline used for image and video quality comparisons on HDTF and TalkingHead1kh.","marker":"[31]"},{"why":"A second 2D video diffusion baseline used for quality and controllability comparisons.","marker":"[46]"},{"why":"Source of the MPI near and far plane placement and rendering settings used in training and inference.","marker":"[53]"},{"why":"Off-the-shelf stereo matcher used to visualize disparity maps in the binocular stereo evaluation.","marker":"[29]"},{"why":"Depth predictor used to build the 2D-diffusion-plus-depth baseline for side-view rendering comparison.","marker":"[47]"}],"fun_headline_variants":["Diffusion one-shot: 3D heads, no rebuild","Single denoising pass yields 3D talking heads","3D MPI video from one diffusion pass","No reconstruction: diffusion draws 3D heads","One pass diffusion for 3D-aware talking heads"],"cache_read_input_tokens":20480,"weakest_assumption_plain":"The 3D training signal is the model's own rendered image in the reference camera, so if that bootstrapped image is structurally wrong the learned shape inherits the same error, and the authors concede in the supplement that these pseudo images can drift in color tone.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion one-shot: 3D heads, no rebuild","Single denoising pass yields 3D talking heads","3D MPI video from one diffusion pass","No reconstruction: diffusion draws 3D heads","One pass diffusion for 3D-aware talking heads"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001266,"raw_usage":{"total_tokens":5198,"prompt_tokens":979,"completion_tokens":4219,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":4143}},"tokens_in":595,"tokens_out":4219,"duration_ms":28834,"temperature":1.0,"reasoning_tokens":4143,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:59:06.670695+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Using synchronized multi-camera footage of a subject, render side views from the MPIs at increasing baselines and compare MPI-derived disparity with ground-truth disparity; if disparity error grows sharply as the baseline increases while frontal quality stays high, the reference-camera bootstrap is encoding appearance bias rather than true geometry. A cheaper check on public data is to compare the MPI disparity in rigid facial regions such as the forehead and nose with the fitted parametric-model depth at matched head poses and see whether disagreement falls inside the render tolerance.","supporting_citations":[{"cited_title":"Diffusion with forward models: Solv- ing stochastic inverse problems without direct supervision","cited_arxiv_id":null,"evidence_quote":"Source of the diffusion-with-forward-model idea, inserting a differentiable renderer into the denoising loop, and of the frontal-view and side-view two-loss supervision scheme."},{"cited_title":"Single-view view synthe- sis with multiplane images","cited_arxiv_id":null,"evidence_quote":"Defines the MPI representation, the frontal-plus-residual RGB compositing, and the disparity and edge-aware smoothing losses used in training."},{"cited_title":"Portrait4d-v2: Pseudo multi-view data creates better 4d head synthesizer","cited_arxiv_id":null,"evidence_quote":"The NeRF-based one-shot talking-head baseline whose novel-view quality and render speed are compared."},{"cited_title":"X-portrait: Expressive portrait anima- tion with hierarchical motion attention","cited_arxiv_id":null,"evidence_quote":"A second 2D video diffusion baseline used for quality and controllability comparisons."},{"cited_title":"Generative multi- plane images: Making a 2d gan 3d-aware","cited_arxiv_id":null,"evidence_quote":"Source of the MPI near and far plane placement and rendering settings used in training and inference."},{"cited_title":"Raft-stereo: Multilevel recurrent field transforms for stereo matching","cited_arxiv_id":null,"evidence_quote":"Off-the-shelf stereo matcher used to visualize disparity maps in the binocular stereo evaluation."}],"review_version":1}