{"id":"cc0be456-cefa-4599-a298-d92c9e907a43","arxiv_id":"2508.12336","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A GAN-based pipeline with dense landmark guidance and SynergyNet removes HMDs from monocular RGB video and regresses 3D facial geometry from the inpainted frames.","lead":"This paper builds a system that removes head-mounted displays in videos and reconstructs a 3D face model using one camera and a single reference image. It combines a video inpainting network with a 3D morphable model and tests how few facial landmarks are needed.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's 3D geometry metrics are circular: GT meshes come from the same SynergyNet pipeline that reconstructs from inpainted frames, so Chamfer/RMS/Hausdorff measure round-trip consistency, not absolute face geometry accuracy.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing gap: Table 2's ground truth is generated by the same SynergyNet pipeline used for prediction, so the geometry numbers cannot establish absolute accuracy. I agree, and the paper strengthens this concern by fine-tuning SynergyNet during Stage 2 and by using DenseLMLoss to align SynergyNet landmarks with MediaPipe landmarks from the same original frames, making the metrics partially optimized at training time. The RGB inpainting results are internally consistent and show real improvements over baselines, so the appearance part of the claim is reasonably supported. The 3D geometry claim, however, rests entirely on a circular evaluation, and this is not acknowledged in the paper's own limitations. A scan-based or multi-view re-evaluation would settle whether the geometry claim holds; until then, the CONDITIONAL verdict is appropriate. The separate issue of GT landmarks used at test time is secondary and already partly acknowledged in Section 5.","tokens_in":18762,"tokens_out":6582,"duration_ms":73017,"concrete_test":"Evaluate on a subset of test subjects with true 3D ground truth, e.g., multi-view stereo or structured-light face scans captured with and without an HMD. Recompute the three metrics from Table 2 by comparing SynergyNet meshes from both inpainted and original unoccluded frames directly against the scans, and additionally compute the same metrics for a public single-image reconstruction method not used in training (e.g., DECA or 3DDFA-V2). If SynergyNet's reconstructions from unoccluded frames deviate from the scans by margins comparable to or larger than the reported Table 2 differences, the geometry claim should be re-scoped to inpainting-to-SynergyNet consistency.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative support for the 3D geometry claim is Table 2. Sections 3.4 and 4.1 state that GT face meshes are generated by applying SynergyNet to the original, unoccluded frames. The predicted meshes are produced by SynergyNet from inpainted frames, and Section 4.1 adds that SynergyNet is fine-tuned in Stage 2 jointly with the generator. The reported Chamfer, RMS, and Hausdorff distances therefore measure how consistently the inpainted and original frames pass through the same (and partly fine-tuned) 3DMM regression backbone, not fidelity to true face geometry. The circularity is compounded by DenseLMLoss (Eq. 2), which trains the generator so that SynergyNet landmarks on inpainted frames match MediaPipe landmarks extracted from the same original frames; the geometry metrics are partially optimized at training time. Consequently, Table 2 supports only the statement that inpainted frames preserve the 3DMM signature of the originals. The RGB metrics in Table 1 support the inpainting claim, but the 'accurate 3D face reconstruction' part of the central claim lacks absolute geometric validation. The paper's Section 5 limitations acknowledge real-world landmark and mask issues, but they do not flag this self-referential geometry evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a geometry-aware framework that jointly removes head-mounted display (HMD) occlusions from monocular RGB video and reconstructs 3D face geometry. The pipeline uses an EVI-HRnet-based GAN for video inpainting, a dense landmark regression module, and a SynergyNet backbone that regresses 3DMM parameters from the inpainted frames. Training combines adversarial, FER, style, VGG, reconstruction, and dense-landmark Huber losses, with a two-stage protocol that fine-tunes SynergyNet jointly with the generator. Experiments on FaceForensics with synthetic HMD masks compare RGB inpainting performance against EVI-HRnet, CombCN, and LGTSM, and an ablation study varies the number of facial landmarks from 10 to 216 to measure both RGB quality and 3D mesh distances. The paper also includes a qualitative real-world demonstration with a Meta Quest Pro and a webcam.","tokens_in":19065,"tokens_out":4665,"duration_ms":46040,"significance":"The RGB inpainting contribution is evaluated carefully against relevant baselines on a standard dataset, and the consistent improvements in FID, MSE, LPIPS, SSIM, and PSNR are a credible strength. The systematic ablation over landmark density is also useful, and the promise of public code and output videos would aid reproducibility. However, the central claim of accurate 3D face reconstruction is not yet supported: the 3D ground truth is generated by the same SynergyNet module used for prediction, and the geometry metrics therefore measure round-trip consistency rather than absolute accuracy. The paper would be valuable if the 3D component were validated against independent ground truth or if the claims were appropriately restricted.","major_comments":[{"comment":"The 3D geometry evaluation is circular. The GT meshes are produced by applying SynergyNet to the original unoccluded frames (Section 3.4 and Section 4.1), while the predicted meshes are produced by the same SynergyNet backbone, fine-tuned jointly with the generator in Stage 2, from the inpainted frames. The Chamfer, RMS, and Hausdorff distances therefore measure how consistently the inpainted frames reproduce SynergyNet's own 3DMM output, not how accurately the recovered geometry matches true face shape. I ask for independent validation: for example, multi-view or depth-sensor ground truth, or at minimum a cross-check with a different single-image 3D face reconstruction method applied to both inpainted and original frames, together with a report of SynergyNet's own error against independent GT on the original frames. Without this, Table 2 cannot support the abstract's claim of accurate 3D face reconstruction.","section":"§3.4, §4.1, §4.2, Table 2"},{"comment":"The geometry metrics are partially optimized at training time. DenseLMLoss (Eq. 2) trains the model so that SynergyNet landmarks extracted from inpainted frames match MediaPipe landmarks from the original unoccluded frames, and in Stage 2 SynergyNet itself is fine-tuned jointly with the generator. Consequently, the reported mesh distances in Table 2 are not an independent measure of geometric fidelity; they reflect agreement with the specific 3DMM/landmark signature used as supervision. Please either evaluate on a held-out source of 3D ground truth or clearly restrict the claim to consistency with SynergyNet reconstruction rather than absolute geometric accuracy.","section":"§3.4, Eq. (2), §4.1"},{"comment":"The reported numbers for the 10-landmark configuration are internally inconsistent: Table 2 lists an Average Chamfer Distance of 2.754, while the text states 2.934. Since these values are load-bearing for the landmark-density conclusion, please correct the discrepancy and ensure that all quantitative claims in the text match the corresponding tables exactly.","section":"§4.2, Table 2"}],"minor_comments":[{"comment":"There is a typo in the paragraph on the EVI-HRnet baseline: 'EVI-HRnnet' should read 'EVI-HRnet'.","section":"§2"},{"comment":"The second-best values in the Ours (68 LM) row are not visible in the rendered table. Please fix the table formatting so that every reported value is readable.","section":"§4.1, Table 1"},{"comment":"The Huber loss definition appears garbled in the manuscript. Please typeset Eq. (1) properly and state the numerical value of the threshold delta used during training.","section":"§3.4, Eq. (1)"},{"comment":"The six loss weights in Eq. (4) are listed without any sensitivity analysis. If all weights are equal to unity, this should be stated explicitly; otherwise a brief justification of the chosen values would help reproducibility.","section":"§3.4, Eq. (4)"},{"comment":"For the proposed method the reference frame is masked outside the HMD region, while for baseline methods the first frame is left fully unmasked. Please clarify whether this input difference affects the fairness of the comparison in Table 1.","section":"§3.2, §4.1"},{"comment":"The real-world comparison in Figure 6 uses different capture setups for each method, and the caption notes that identical input images are not available. The text should clearly separate this illustrative demonstration from the benchmark evaluation so that readers do not interpret Figure 6 as a quantitative comparison.","section":"§5, Figure 6"},{"comment":"The qualitative demonstration says the reconstructed 3D meshes are textured using UV maps from FFHQ-UV, not by the proposed method. Please state explicitly in the main text which parts of the rendered results are generated by the framework and which are borrowed from external data.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"The circularity of the 3D evaluation is the main blocking issue and was already identified in the reader's report. The manuscript itself acknowledges limitations about synthetic masks and landmark accuracy in Section 5, but it does not mention that the geometry ground truth comes from the same module used for prediction. The revision should either add independent 3D ground truth or substantially soften the 3D accuracy claims. The numerical discrepancy between Table 2 and the text should also be corrected carefully."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline here is that the paper does a decent job on the inpainting side and a questionable job on the 3D geometry side. The main new thing is combining EVI-HRnet-style video inpainting with SynergyNet 3DMM regression and testing how landmark density affects both outputs. That combination is new relative to the cited work, and the landmark-density ablation is genuinely useful: the degradation from 216 to 10 landmarks is smooth, which supports the practical story about consumer HMDs.\n\nThe RGB evaluation is the strongest part. On FaceForensics with synthetic HMD masks, the method beats EVI-HRnet, CombCN, and LGTSM on all five metrics, and the gain is consistent across landmark counts. The comparison looks fair: same masks, same test split, standard metrics. The authors also acknowledge that the dataset is mostly professional newscasters, so generalization to casual settings is untested.\n\nThe soft spot is exactly where your stress-test note lands. Table 2 is circular. The GT meshes come from applying SynergyNet to unoccluded frames, and the predicted meshes come from the same SynergyNet backbone, fine-tuned in Stage 2. So the Chamfer, RMS, and Hausdorff distances measure round-trip consistency, not geometric accuracy. The DenseLMLoss makes it even more self-referential, because the generator is trained to make SynergyNet landmarks match MediaPipe landmarks from the unoccluded frames. Nothing in the paper provides independent 3D ground truth, so the 'accurate 3D face reconstruction' claim is unsupported in its current form. That's a load-bearing flaw for the title and abstract, though it doesn't taint the RGB results.\n\nThere is also a minor numeric inconsistency: the text says 10-LM Chamfer distance is 2.934, but Table 2 reports 2.754. The limitations section correctly mentions synthetic masks, HMD shadows, and real-world landmark detection, but it never flags the circular geometry evaluation, which should have been the first thing on the list.\n\nWho gets value from this? Anyone working on HMD removal or monocular face reconstruction in XR will find the landmark-density ablation and the full pipeline description useful. The paper does not demonstrate absolute geometric accuracy, so it should be framed as a system for removing HMDs and preserving the face's 3DMM signature, not as a validated 3D reconstruction method.\n\nI would send it to peer review, but as a major revision. The RGB comparisons are solid enough to deserve refereeing, and the landmark analysis is worth publishing. The authors need to either bring in independent geometry ground truth or explicitly scope the claims to inpainting consistency and 3DMM preservation. Fix the typo, and the paper becomes a reasonable contribution to a niche but active area.","headline":"Solid RGB inpainting benchmark, but the 3D geometry evaluation is circular—GT meshes come from the same SynergyNet backbone used for prediction.","tokens_in":19596,"tokens_out":2938,"would_cite":false,"duration_ms":29357,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a monocular RGB video of a headset-wearing user, guided by facial landmarks and a single clean reference frame, can be inpainted to remove the headset and regressed into a complete 3D face model, preserving identity…","keywords":["extended reality","head-mounted display removal","video inpainting","3D face reconstruction","3D Morphable Model","facial landmarks","generative adversarial networks","social XR"],"falsifier":"An independent 3D capture, such as a depth camera or multi-view scan, of the same people with and without the headset would settle the geometry claim: if the recovered meshes show large errors against that scan while round-trip scores look good, the claimed reconstruction of true 3D face shape is refuted. A second check is to drive the pipeline with landmarks from real headset eye trackers rather than landmarks detected on unobstructed frames; if inpainting quality collapses, the practical claim fails.","tokens_in":18576,"feed_emoji":"🕶️","tokens_out":7825,"duration_ms":77750,"temperature":0.7,"pith_summary":"Head-mounted displays hide the eyes and eyebrows, breaking the eye contact and expression cues that make video teleconferencing feel social. This paper claims that the occlusion can be erased from ordinary RGB video and the user's full 3D face geometry recovered at the same time, using only one front-facing camera, a single reference frame taken without the headset, and facial landmarks as geometric guidance. The mechanism is a loop: landmarks steer a GAN-based video inpainter, the inpainted frames are fed to a 3D Morphable Model regressor, and refined landmarks from that regressor supply a dense landmark loss that makes appearance restoration and mesh recovery pull in the same direction. On a large public benchmark of news presenter videos, the pipeline outperforms existing video inpainting baselines on image-quality metrics, and the reported geometry errors decrease as landmark density rises from 10 to 216. If the claim holds, social XR teleconferencing could restore gaze and expression without infrared cameras, depth sensors, or per-user calibration.","feed_headline":"Headset occlusions removed from video, 3D face rebuilt in one pass","feed_subtitle":"A single reference photo plus facial landmarks lets one RGB camera restore hidden eyes and recover full face geometry.","key_machinery":"The load-bearing object is a landmark-to-3DMM feedback loop. Dense 3D facial landmarks (up to 216 points in the occluded region) are fed into the inpainting network alongside a masked RGB sequence and one occlusion-free reference frame; a gated temporal shift module with self-attention fills the missing upper face; a convolutional network refines landmarks on the inpainted frames by injecting 3D Morphable Model semantics; a SynergyNet-style backbone regresses 3DMM identity, expression, and pose parameters from the frames and refined landmarks, yielding a mesh with per-vertex $(x,y,z)$ coordinates; and a Huber-based dense landmark loss compares predicted landmarks against ground-truth landmarks to optimize the whole chain. Depth is never estimated by a separate network; it comes out of the 3DMM parameters.","core_discovery":"The paper's central claim is that headset removal and 3D face reconstruction belong in one geometry-aware pipeline rather than two separate stages. Dense facial landmarks extracted from unobstructed frames, or available from headset eye trackers, provide spatial structure that guides the video inpainter; the inpainted RGB frames are then passed to a 3D Morphable Model based regressor that produces a face mesh; and the regressor's refinement of landmarks supplies a dense Huber loss that supervises both stages. In the authors' evaluation, the full pipeline using 216 landmarks achieves the best scores on image quality metrics and beats the inpainting-only baselines, while the same framework with only 10 eyelid landmarks remains close, showing graceful degradation. The recovered meshes, textured for visualization, are claimed to be photorealistic and identity-preserving.","pith_inferences":["The 3D geometry metrics are self-referential, so the 'photorealistic 3D face geometry' claim should be read as round-trip fidelity to the chosen reconstruction backbone until independent 3D ground truth is provided.","Because the reference frame comes from the same video sequence, the framework may depend on similar pose, lighting, and expression between reference and target; the paper's acknowledged shadowing and large-head-movement limitations suggest this dependence is real.","A natural next experiment is to remove the landmark channel entirely and drive the inpainter with gaze and eyelid estimates from headset sensors; if quality is preserved, the geometry-aware loop could run with near-zero external inputs.","Synthetic headset masks do not reproduce shadows, reflections, or skin deformation under the device, so the reported gains may shrink on real headset footage; a real-world capture benchmark with varied lighting is the missing test."],"forward_implications":["Video calls in XR can restore eye contact and eyebrow and expression cues from a single webcam stream plus one clean reference photo, without infrared cameras, depth sensors, or per-user calibration.","Because the output includes a 3D mesh, remote participants could view the user from novel angles rather than only seeing the fixed frontal video.","The gradual degradation from 216 to 10 landmarks means consumer headsets with sparse eye-tracking landmark data may still produce usable avatars.","Geometry-aware supervision improves the inpainted video itself, so appearance-only inpainting baselines are not sufficient for high-quality headset removal.","The same landmark-to-3DMM loop should transfer to other fixed upper-face occlusions, such as surgical masks or safety goggles, without changing the pipeline's logic."],"supporting_citations":[{"why":"The inpainting backbone the framework builds on; supplies the generator, temporal shift, and discriminator architecture.","marker":"[11]"},{"why":"Supplies the learnable gated temporal shift mechanism for temporally consistent video inpainting within the backbone.","marker":"[12]"},{"why":"Supplies the 3DMM regression backbone and refined-landmark loop that produce the face mesh from inpainted RGB frames.","marker":"[13]"},{"why":"Supplies dense 3D facial landmark detection used for input guidance and for the dense landmark loss.","marker":"[14]"},{"why":"A video inpainting baseline used in the quantitative image-quality comparisons.","marker":"[48]"},{"why":"The public video dataset used for training and testing with simulated headset masks.","marker":"[49]"}],"fun_headline_variants":["Single RGB camera erases headset, rebuilds face in 3D","Geometry-aware inpainting removes HMD, reconstructs face","One-pass headset removal with joint 3D face reconstruction","Landmark-guided inpainting erases headset, rebuilds face mesh","Single-view removal of headset yields full 3D face geometry"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation's ground-truth 3D meshes are produced by the same reconstruction network that later processes the inpainted frames, so the geometric error scores measure round-trip agreement with that network's own outputs rather than absolute face shape.","fun_headline_variants_meta":{"raw":{"variants":["Single RGB camera erases headset, rebuilds face in 3D","Geometry-aware inpainting removes HMD, reconstructs face","One-pass headset removal with joint 3D face reconstruction","Landmark-guided inpainting erases headset, rebuilds face mesh","Single-view removal of headset yields full 3D face geometry"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001028,"raw_usage":{"total_tokens":4336,"prompt_tokens":953,"completion_tokens":3383,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":3290}},"tokens_in":569,"tokens_out":3383,"duration_ms":23229,"temperature":1.0,"reasoning_tokens":3290,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:22:34.321244+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An independent 3D capture, such as a depth camera or multi-view scan, of the same people with and without the headset would settle the geometry claim: if the recovered meshes show large errors against that scan while round-trip scores look good, the claimed reconstruction of true 3D face shape is refuted. A second check is to drive the pipeline with landmarks from real headset eye trackers rather than landmarks detected on unobstructed frames; if inpainting quality collapses, the practical claim fails.","supporting_citations":[{"cited_title":"Ghorbani Lohesara, K","cited_arxiv_id":null,"evidence_quote":"The inpainting backbone the framework builds on; supplies the generator, temporal shift, and discriminator architecture."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 3DMM regression backbone and refined-landmark loop that produce the face mesh from inpainted RGB frames."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies dense 3D facial landmark detection used for input guidance and for the dense landmark loss."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A video inpainting baseline used in the quantitative image-quality comparisons."}],"review_version":2}