REVIEW 3 major objections 5 minor 31 references
Face reenactment can finally transfer tongue motion from one person to another by anchoring diffusion synthesis to driving-frame geometry.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-31 19:42 UTC pith:PNQKXHNO
load-bearing objection Real gap, working pipeline, but the abstract’s “>2× on every tongue metric” does not match Table 1, and Full scores drift between tables. the 3 major comments →
TongueReenact: Geometry-Anchored Tongue Synthesis for Face Reenactment
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Cross-identity tongue dynamics can be transferred in face reenactment by decoupling geometric alignment from generative fill: a Gaussian-splatting stage driven by FLAME parameters yields a source-identity, tongue-absent render whose mouth region is then synthesized by a latent masked diffusion model that is recomposed against the reference latent at every denoising step, conditioned on both source geometry and the driving tongue crop, with the active region set by a bootstrapped tongue mask that is adaptively dilated.
What carries the argument
Geometry-anchored latent masked diffusion: at every DDIM step the denoised latent is forced back to the reenacted reference outside a driving-frame tongue mask (adaptively dilated), while dual guidance encoders inject source facial structure and driving tongue appearance so synthesis is spatially locked rather than free hallucination.
Load-bearing premise
That a vision-language model fine-tuned on only about a hundred expert side-by-side labels can stand in for human judgment of tongue quality on a thousand more frames.
What would settle it
Run the same VLM protocol (or real expert raters) on held-out pairs where an independent tongue segmenter and human labels disagree with the paper’s full model, or swap in a different reenactment backbone and check whether Tongue IoU, area similarity, and best-tongue rates still more than double the listed baselines.
If this is right
- Reenacted talking heads can show visible, driving-matched tongue articulation instead of empty or blurred mouths during speech and extreme expressions.
- Any geometry-faithful reenactment backbone can be paired with the same masked diffusion stage without redesigning the whole avatar pipeline.
- Tongue-specific metrics (presence, IoU, area similarity, regional LPIPS) become usable benchmarks for oral interior fidelity beyond FID-style global scores.
- A bootstrapped segmenter removes the need for large curated tongue masks before training reenactment systems on in-the-wild video.
Where Pith is reading between the lines
- The same geometry-anchor-plus-masked-diffusion pattern could extend to other transient oral structures (teeth gaps, saliva, extreme jaw interiors) that parametric meshes leave empty.
- If the VLM judge is validated against a larger multi-annotator panel, automated oral-region QA could become standard in portrait-animation benchmarks.
- Failure modes will concentrate where the driving tongue is barely visible or the reenactment backbone misplaces the mouth, because the mask and reference both inherit that geometry.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes TongueReenact, claimed as the first cross-identity tongue-dynamics transfer framework for face reenactment. A foundation-model-assisted bootstrapping pipeline (SAM + face parsing + iterative human refinement) trains a dedicated BiSeNet* tongue segmenter for in-the-wild frames. A two-stage system then (i) produces a tongue-absent, geometry-grounded render via FLAME-conditioned Gaussian splatting and (ii) synthesizes tongue appearance with a latent masked diffusion model that recomposes the reference latent outside an adaptively dilated driving tongue mask at every DDIM step, dual-conditioned on source geometry and driving mouth appearance. Experiments on VFHQ compare against five reenactment baselines on four tongue-specific metrics, with ablations, temporal-consistency checks, mouth-masked ArcFace identity checks, and a Qwen3-VL perceptual protocol fine-tuned on 100 expert frames.
Significance. If the technical claims hold, the work fills a genuine and previously ignored gap: existing parametric, warping, and diffusion reenactment systems leave the oral interior unconstrained or empty, which matters for speech and extreme expression. The bootstrapped segmenter, geometry-anchored masked diffusion with per-step recomposition, and adaptive dilation are concrete, reusable components, and the diffusion stage is explicitly designed to couple to other geometry-faithful backbones. Table 1 shows consistent gains over strong baselines on tongue IoU and area similarity; ablations move in the expected direction; temporal and masked-identity checks address two natural failure modes. These are real contributions for face animation and avatar pipelines, even after the overstated multiplicative headline is corrected.
major comments (3)
- [Abstract; §5.3; Table 1] Abstract and §5.3 claim “improvements of more than two times over all baselines on every tongue-specific metric.” Table 1 does not support that wording. Relative to the best baseline per row: LPIPS-tongue improves only 0.2591→0.2306 (~1.12×, lower better); Tongue Presence 0.7367→0.7891 (~1.07×); Tongue IoU 0.1918→0.3582 (~1.87×); only Tongue Area Similarity is clearly >2× (0.1330→0.4338). “Every” and “more than two times” are jointly false on the reported SOTA table. The abstract and corresponding discussion must be rewritten to match the actual ratios (or report absolute deltas with clear baselines).
- [Table 1 vs Table 2; §5.2–5.4] The same “Full (Ours)” system is scored differently in Table 1 vs Table 2 with no disclosed subset, seed, or protocol change (LPIPS 0.2306 vs 0.2089; Presence 0.7891 vs 0.8851; IoU 0.3582 vs 0.3956; Area 0.4338 vs 0.4779). Tongue Presence/IoU/Area (and the tongue crop for LPIPS) depend on automatic masks—almost certainly BiSeNet*, which also drives training and inference (§4.1–4.2). Unexplained drift undermines the stability of the primary quantitative claim. Please lock a single evaluation protocol (split, mask source, post-processing), report Full under that protocol in both tables, and state whether metrics use BiSeNet* or an independent tongue detector.
- [§5.6; Table 4; Supplementary §2] The VLM protocol (Table 4; Supp. §2) fine-tunes Qwen3-VL on 100 expert frames that compare the authors’ Full model side-by-side against their own ablations, then applies it to 1,000 further frames, yielding 99.9–100% “best tongue/overall.” This is a useful scaled preference check but is not an independent human study: the labeled set and forced four-way comparison can bias the model toward Full, and there is no held-out human agreement on the 1,000-frame pool or comparison against external baselines (e.g., X-NeMo). Either (i) report inter-rater / VLM–human agreement on a held-out expert set and include SOTA baselines in the VLM ballot, or (ii) clearly demote Table 4 to an ablation-preference diagnostic and not “replicat[ing] expert annotation at scale” as primary perceptual evidence.
minor comments (5)
- [§5 / Figures] Figure numbering jumps from Fig. 3 to Fig. 5 in the main text; Fig. 4 appears missing or mislabeled.
- [Algorithm 1; §4.1] Algorithm 1 Phase 1 line “V_ext += GaS(f, FLAME(t))” is underspecified: it is unclear how reenacted frames enter the segmentation bootstrap and whether this couples segmenter training to the reenactment backbone.
- [§4.1–4.2; §5.1] Stratification thresholds τ1, τ2 and dilation hyperparameters ρ, r_min are named but not given numeric values in the main text (only ρ=0.3, r_min=5 at inference in §5.1). A short hyperparameter table would aid reproducibility.
- [§2.4] Related work on oral/tongue modeling in speech animation and medical tongue segmentation is thin beyond TongueSAM; a brief pointer to talking-head / viseme work would better situate the “first framework” claim.
- [§5.4] Identity-preservation paragraph in §5.4 reports ArcFace gaps but no table; adding a small numeric row (full vs mouth-masked) would make the claim easier to verify.
Circularity Check
No load-bearing derivation circularity: tongue synthesis is conditioned on external driving appearance and source geometry. Only mild self-reference in the VLM judge trained on the authors’ own full-vs-ablation labels then used to reconfirm superiority.
specific steps
-
fitted input called prediction
[§5.6 Table 4; Supplementary §2.1–2.3]
"We therefore fine-tune Qwen3-VL [26] on 100 expert-annotated frames to replicate human judgement across four criteria: tongue quality (0–5 scale), tongue positioning (binary), best tongue, and best overall reenactment. The fine-tuned model is applied to the remaining 1,000 frames... As shown in Table 4, the full model achieves a mean tongue quality rating of 3.07, is selected as best tongue in 99.9% of frames and best overall in 100% of frames... confirming perceptual superiority across all ablation variants."
The perceptual “confirmation” is produced by a judge fitted on expert labels that already compare the same Full model to the same ablations, then scored on further frames of the same variant set. Near-ceiling Best Tongue/Overall rates are therefore largely the VLM generalizing that small labeled preference, not an independent external measurement of the method. This is mild and confined to the VLM protocol; it does not force Table 1 baseline metrics or the synthesis equations.
full rationale
TongueReenact is a systems/ML paper, not a first-principles derivation. The two-stage pipeline takes external inputs (source identity pixels, driving FLAME parameters, driving-frame tongue crop/mask) and produces a reenacted frame; success is measured against independent published baselines (GPAvatar, Portrait4D, LivePortrait, X-NeMo, X-Portrait) on held-out VFHQ pairs. Nothing in Eqs. (1)–(11) or Algorithm 1 defines the reported metrics in terms of a fitted scalar that is then renamed a prediction. Bootstrapped BiSeNet* is trained on SAM/face-parser pseudo-labels with human refinement, then used at inference—an ordinary train-then-deploy loop, not X-defined-as-Y. Self-citations are absent as load-bearing uniqueness claims; backbones (GAGAvatar, FLAME, SD, SAM) are external. The sole mild circularity-adjacent pattern is evaluation: a VLM is LoRA-tuned on 100 expert side-by-side labels of Full vs the authors’ own ablations, then applied to 1,000 further frames to report 99.9–100% “best tongue/overall” for Full (Table 4). That amplifies a small expert preference study rather than independently discovering superiority, but it does not make the method’s outputs equal their inputs by construction. Overstated “>2× on every metric” wording is a correctness/reporting issue, not circularity. Score 2 reflects one non-load-bearing evaluation self-reference only.
Axiom & Free-Parameter Ledger
free parameters (5)
- Adaptive dilation ratio ρ and r_min =
ρ=0.3, r_min=5 pixels
- Tongue-area stratification thresholds τ1, τ2 =
Uniform partition of observed mask-area range (exact numeric cutoffs not stated)
- DDIM steps and diffusion training hyperparameters =
T_train=1000; 20 DDIM steps at inference; lr=1e-5
- Human-refined seed and iterative pseudo-label set D^(k)
- VLM LoRA fine-tune on 100 expert frames =
100 annotated frames; 80/20 split
axioms (5)
- domain assumption FLAME parameters capture driving pose/expression well enough that a FLAME-conditioned Gaussian render localizes the mouth for tongue inpainting.
- ad hoc to paper Recomposing the reference latent outside the (dilated) tongue mask at every DDIM step preserves source identity while allowing tongue generation inside the mask.
- domain assumption SAM + face-parser prompts plus stratified human refinement yield pseudo-labels adequate to train an in-the-wild tongue segmenter.
- domain assumption Standard latent diffusion / VAE scaling and frozen CLIP identity conditioning behave as in prior SD image-variation setups.
- domain assumption Tongue IoU/area/presence and LPIPS in the tongue region are valid proxies for transfer fidelity when paired with perceptual checks.
invented entities (3)
-
BiSeNet* bootstrapped tongue segmenter for in-the-wild reenactment
independent evidence
-
Geometry-anchored latent masked diffusion with adaptive mask dilation
no independent evidence
-
VLM-based tongue quality evaluation protocol (fine-tuned Qwen3-VL)
no independent evidence
read the original abstract
Modern face reenactment systems achieve impressive pose and expression transfer using geometry-driven representations. However, they largely ignore tongue dynamics, leading to anatomically inconsistent mouth interiors during speech and expressive motions. We introduce the first framework for cross-identity tongue dynamics transfer in face reenactment. We propose a foundation-model-assisted bootstrapping pipeline that produces a dedicated tongue segmentation model for in-the-wild reenactment without curated annotations. We further introduce a spatially constrained latent masked diffusion model for realistic tongue synthesis, with adaptive mask dilation for seamless mouth boundary transitions. Extensive experiments demonstrate improvements of more than two times over all baselines on every tongue-specific metric. We additionally propose a VLM-based evaluation protocol that replicates expert annotation at scale, confirming perceptual superiority across all ablation variants.
Figures
Reference graph
Works this paper leans on
-
[1]
In: Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp
Blanz, V., Vetter, T.: A morphable model for the synthesis of 3d faces. In: Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp. 157–164 (2023)
2023
-
[2]
In: 2023 IEEE international conference on bioinfor- matics and biomedicine (BIBM)
Cao, S., Wu, Q., Ma, L.: Tonguesam: An universal tongue segmentation model based on sam with zero-shot. In: 2023 IEEE international conference on bioinfor- matics and biomedicine (BIBM). pp. 4520–4526. IEEE (2023)
2023
-
[3]
Advances in Neural Information Processing Systems37, 57642–57670 (2024)
Chu, X., Harada, T.: Generalizable and animatable gaussian head avatar. Advances in Neural Information Processing Systems37, 57642–57670 (2024)
2024
-
[4]
arXiv preprint arXiv:2401.10215 (2024)
Chu, X., Li, Y., Zeng, A., Yang, T., Lin, L., Liu, Y., Harada, T.: Gpavatar: Gener- alizable and precise head avatar from image (s). arXiv preprint arXiv:2401.10215 (2024)
Pith/arXiv arXiv 2024
-
[5]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Deng, J., Guo, J., Xue, N., Zafeiriou, S.: Arcface: Additive angular margin loss for deep face recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4690–4699 (2019) TongueReenact: Tongue Synthesis for Face Reenactment 15
2019
-
[6]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Deng, Y., Wang, D., Ren, X., Chen, X., Wang, B.: Portrait4d: Learning one-shot 4d head avatar synthesis using synthetic data. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7119–7130 (2024)
2024
-
[7]
In: European Conference on Computer Vision
Dhamo, H., Nie, Y., Moreau, A., Song, J., Shaw, R., Zhou, Y., Pérez-Pellitero, E.: Headgas: Real-time animatable head avatars via 3d gaussian splatting. In: European Conference on Computer Vision. pp. 459–476. Springer (2024)
2024
-
[8]
arXiv preprint arXiv:2407.03168 (2024)
Guo, J., Zhang, D., Liu, X., Zhong, Z., Zhang, Y., Wan, P., Zhang, D.: Liveportrait: Efficient portrait animation with stitching and retargeting control. arXiv preprint arXiv:2407.03168 (2024)
Pith/arXiv arXiv 2024
-
[9]
Iclr1(2), 3 (2022)
Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. Iclr1(2), 3 (2022)
2022
-
[10]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Hu, L.: Animate anyone: Consistent and controllable image-to-video synthesis for character animation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8153–8163 (2024)
2024
-
[11]
In: European Conference on Computer Vision
Ju, X., Liu, X., Wang, X., Bian, Y., Shan, Y., Xu, Q.: Brushnet: A plug-and- play image inpainting model with decomposed dual-branch diffusion. In: European Conference on Computer Vision. pp. 150–168. Springer (2024)
2024
-
[12]
ACM Trans
Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G., et al.: 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph.42(4), 139–1 (2023)
2023
-
[13]
In: Proceedings of the IEEE/CVF international conference on computer vision
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4015–4026 (2023)
2023
-
[14]
ACM Transactions on Graphics (TOG)42(4), 1–14 (2023)
Kirschstein, T., Qian, S., Giebenhain, S., Walter, T., Nießner, M.: Nersemble: Multi-view radiance field reconstruction of human heads. ACM Transactions on Graphics (TOG)42(4), 1–14 (2023)
2023
-
[15]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Lee, C.H., Liu, Z., Wu, L., Luo, P.: Maskgan: Towards diverse and interactive facial image manipulation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5549–5558 (2020)
2020
-
[16]
ACM Trans
Li, T., Bolkart, T., Black, M.J., Li, H., Romero, J.: Learning a model of facial shape and expression from 4d scans. ACM Trans. Graph.36(6), 194–1 (2017)
2017
-
[17]
In: Proceedings of the IEEE/CVF winter conference on applications of computer vision
Lin,S.,Liu,B.,Li,J.,Yang,X.:Commondiffusionnoiseschedulesandsamplesteps are flawed. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. pp. 5404–5411 (2024)
2024
-
[18]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., Van Gool, L.: Re- paint: Inpainting using denoising diffusion probabilistic models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11461– 11471 (2022)
2022
-
[19]
Qian, S., Kirschstein, T., Schoneveld, L., Davoli, D., Giebenhain, S., Nießner, M.: Gaussianavatars:Photorealisticheadavatarswithrigged3dgaussians.In:Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20299–20309 (2024)
2024
-
[20]
In: International conference on machine learning
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)
2021
-
[21]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)
2022
-
[22]
In: Proceedings of the IEEE conference on com- puter vision and pattern recognition
Shrivastava, A., Gupta, A., Girshick, R.: Training region-based object detectors with online hard example mining. In: Proceedings of the IEEE conference on com- puter vision and pattern recognition. pp. 761–769 (2016) 16 M. W. Khan et al
2016
-
[23]
arXiv preprint arXiv:2010.02502 (2020)
Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020)
Pith/arXiv arXiv 2010
-
[24]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Xie, L., Wang, X., Zhang, H., Dong, C., Shan, Y.: Vfhq: A high-quality dataset and benchmark for video face super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 657–666 (2022)
2022
-
[25]
In: ACM SIGGRAPH 2024 conference papers
Xie, Y., Xu, H., Song, G., Wang, C., Shi, Y., Luo, L.: X-portrait: Expressive portrait animation with hierarchical motion attention. In: ACM SIGGRAPH 2024 conference papers. pp. 1–11 (2024)
2024
-
[26]
arXiv preprint arXiv:2505.09388 (2025)
Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al.: Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025)
Pith/arXiv arXiv 2025
-
[27]
In: Proceedings of the European conference on computer vision (ECCV)
Yu, C., Wang, J., Peng, C., Gao, C., Yu, G., Sang, N.: Bisenet: Bilateral segmenta- tion network for real-time semantic segmentation. In: Proceedings of the European conference on computer vision (ECCV). pp. 325–341 (2018)
2018
-
[28]
In: Proceedings of the IEEE conference on computer vision and pattern recognition
Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 586–595 (2018)
2018
-
[29]
arXiv preprint arXiv:2507.23143 (2025)
Zhao, X., Xu, H., Song, G., Xie, Y., Zhang, C., Li, X., Luo, L., Suo, J., Liu, Y.: X-nemo: Expressive neural motion reenactment via disentangled latent attention. arXiv preprint arXiv:2507.23143 (2025)
Pith/arXiv arXiv 2025
-
[30]
In: European Conference on Computer Vision
Zhou, Z., Ma, F., Fan, H., Yang, Z., Yang, Y.: Headstudio: Text to animatable head avatars with 3d gaussian splatting. In: European Conference on Computer Vision. pp. 145–163. Springer (2024)
2024
-
[31]
In: European Conference on Computer Vision
Zhu, S., Chen, J.L., Dai, Z., Dong, Z., Xu, Y., Cao, X., Yao, Y., Zhu, H., Zhu, S.: Champ: Controllable and consistent human image animation with 3d parametric guidance. In: European Conference on Computer Vision. pp. 145–162. Springer (2024) TongueReenact: Tongue Synthesis for Face Reenactment 17 Supplementary Material for TongueReenact: Geometry-Anchore...
arXiv 2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.