{"id":"6210dbd6-3218-4207-a793-de7761706b4d","arxiv_id":"2412.11058","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SHMT is a self-supervised latent-diffusion makeup transfer method that decouples face content and makeup and reconstructs the input, avoiding pseudo-paired training data.","lead":"This paper introduces SHMT, a diffusion-based method that transfers makeup from a reference face to a source face without needing paired before-and-after images. It separates a face into shape, texture, and makeup components, then retrains a latent diffusion model to recombine them for controllable makeup styles.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported FID/CLS advantage may reflect reference-identity leakage rather than makeup transfer; no identity-controlled metric is reported.","rationale":"The reader's weakest assumption identified the decoupling claim in Sec. 3.3 as load-bearing. I agree that this is the core risk, but I would sharpen it: even if random distortion preserves makeup, the self-supervised reconstruction objective with the same identity's distorted image as Im means the network can learn that 'makeup' includes photometric identity information. The appendix's own hierarchical analysis says SHMT-h4 must distill high-frequency information from Im; at test time that information is the reference face's texture. The evaluation then compounds the risk because FID and CLS are reference-similarity metrics that reward exactly this leakage. Key-sim is the only content-preservation metric, and h4 is slightly worse than Stable-Makeup on it, so the current evidence cannot separate 'better makeup transfer' from 'better reference-appearance copying.' This is a missing-controls problem rather than an internal inconsistency, so a conditional acceptance is appropriate if the identity audit is added. If the audit shows leakage, the verdict should move to reject; if it shows no leakage, the original conditional evidence is substantially strengthened.","tokens_in":13877,"tokens_out":8560,"duration_ms":85195,"concrete_test":"On the same 1000 test pairs used for Table 1, compute ArcFace (or equivalent face-recognition) cosine similarities between each transferred result and its source and between each result and its reference, for SHMT-h0, SHMT-h4, Stable-Makeup, and at least one GAN baseline. Also compute the same similarities for the original source-reference pairs as a reference ceiling. If SHMT-h4 outputs are, on average, closer to the reference identity than to the source identity, or substantially closer to the reference identity than Stable-Makeup outputs while Key-sim is comparable, then the FID/CLS advantage is confounded by identity leakage and the SOTA claim does not stand. If output-source ArcFace similarity remains high and output-reference similarity is low, the concern does not land.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"SHMT's SOTA claim depends on two unverified links: (i) the distorted foreground Im in Sec. 3.3 is a makeup representation that retains makeup but not identity/content, and (ii) the Table 1 metrics (FID to reference images, DINO CLS cosine to references) measure makeup fidelity. During self-supervised training, Im is a distorted version of the same image the model must reconstruct, so nothing forces the network to separate makeup from that face's skin albedo, illumination, pores, or wrinkles. At inference with a different reference, SHMT-h4 deliberately transfers high-frequency information from Im (Sec. 3.3; Appendix A), so reference-specific facial texture can be copied instead of makeup applied. Because FID and CLS reward global appearance similarity to the reference, and Key-sim for SHMT-h4 is actually slightly below Stable-Makeup (e.g., 0.953 vs 0.973 on MT in Table 1), the highest FID/CLS may be an artifact of reference-identity copying. No identity-similarity metric is reported for any method, so the central claim is not yet distinguished from a face-appearance-swap effect.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SHMT, a self-supervised makeup transfer method built on latent diffusion models. Following a decoupling-and-reconstruction paradigm, it extracts a content representation (3D face shape plus hierarchical texture details from a Laplacian pyramid) and a makeup representation (a spatially distorted foreground), then reconstructs the original image during training. At inference, the makeup representation comes from the reference image, and an Iterative Dual Alignment (IDA) module refines the injected condition at each denoising step. Experiments on MT, Wild-MT, and LADN report favorable FID, CLS, and Key-sim values against seven baselines, with ablations for the texture level and IDA. The code is publicly available.","tokens_in":14145,"tokens_out":3602,"duration_ms":31762,"significance":"If the central claims hold, the self-supervised paradigm is a relevant alternative to pseudo-paired training data for makeup transfer, and the hierarchical texture control is a useful capability that prior GAN-based methods lack. The paper provides a reasonable amount of ablative evidence (e.g., Table 3 shows IDA improves FID from 32.42 to 27.01 on LADN) and releases code, which supports reproducibility. However, the significance is tempered by the fact that the reported quantitative advantages depend on a non-standard FID usage and the absence of identity-controlled metrics, as detailed below.","major_comments":[{"comment":"The central claim that SHMT-h4 achieves the highest FID and CLS may be confounded by reference-identity leakage. FID is computed between transferred results and reference images, and CLS is the cosine similarity of DINO CLS tokens between the same pairs. Since SHMT-h4 intentionally transfers high-frequency texture details from the reference foreground (Sec. 3.3 and Appendix A), the improved FID/CLS could reflect copying the reference face's identity, geometry, and skin texture rather than makeup fidelity per se. No identity-similarity metric between source and result (e.g., face-recognition cosine distance) is reported for any method, so the paper does not currently distinguish high-fidelity makeup transfer from a face-appearance-swap effect.","section":"Sec. 4.2, Table 1"},{"comment":"The load-bearing premise that random cropping, rotation, and elastic distortion of the foreground 'destroy content information' while 'retaining makeup information' is asserted but not demonstrated. No quantitative analysis shows that Im loses identity/geometry but preserves makeup attributes. During self-supervised training, Im is a distorted version of the same image the model must reconstruct, so the optimization does not force the network to separate makeup from skin albedo, illumination, pores, wrinkles, or identity-bearing facial texture. Without a direct evaluation of Im's content and makeup retention (e.g., face-recognition scores on Im and makeup-attribute classification), the decoupling claim is not established.","section":"Sec. 3.3, 'Makeup Representation'"},{"comment":"The use of FID 'between reference images and transferred results' is non-standard and potentially misleading. FID is defined for two distributions; in this paper the reference set and the generated set are paired, and a low FID rewards any resemblance to the reference distribution, including identity and background transfer. The paper should justify this choice, and ideally supplement it with makeup-specific metrics (e.g., per-component makeup color statistics or a makeup-attribute classifier) to isolate makeup fidelity from global appearance matching.","section":"Sec. 4.2, Evaluation Metrics"},{"comment":"No error bars, confidence intervals, or significance tests are reported for any metric. The metrics are computed on a random selection of 1000 source-reference pairs, and FID estimates are known to have nontrivial sampling variance. For instance, the CLS advantage of SHMT-h4 over Stable-Makeup on MT is 0.715 vs 0.682, but without variance estimates it is unclear whether such differences are stable across random subsets. Adding repeated evaluation with different seeds or bootstrapped intervals would strengthen the quantitative claims.","section":"Sec. 4.2, Table 1"}],"minor_comments":[{"comment":"The baseline list says 'PSAGN [17]' but the cited work is PSGAN; please correct the typo.","section":"Sec. 4.1, Baselines"},{"comment":"The notation fc(i) and fm(j) is not fully defined; please specify that i and j index spatial positions in the content and makeup feature maps, respectively.","section":"Sec. 3.4, Eq. (2)"},{"comment":"The plot of w versus timestep t lacks axis labels and a legend for the two curves; adding these would improve readability.","section":"Sec. 4.3, Figure 8(c)"},{"comment":"The statement that the autoencoder with downsampling factor 4 'preserves texture details better' is not backed by a comparison; either provide a supporting ablation or soften the claim.","section":"Sec. 3.3, Implementation Details"},{"comment":"The limitation regarding face parsing mislabeling forehead makeup as hair is clearly illustrated, but a short sentence on possible mitigations (e.g., retraining the parser on makeup images) would make the limitation more constructive.","section":"Appendix D, Limitations"}],"recommendation":"major_revision","confidential_remarks":"The paper's self-citation pattern is not unusual for a NeurIPS-style submission, and the reader's concern about circularity is not well-founded: the model does not optimize FID/CLS directly, and hyperparameters like tau and pyramid level are hand-set. The real issue is the evaluation protocol. If the authors add an identity-preservation metric (e.g., face-recognition cosine similarity between source and result), a direct test of whether Im retains identity, and confidence intervals for the reported metrics, the central claim could become convincing. These additions are feasible within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Jonathan,\n\nTwo things to know about SHMT. First, the self-supervised decoupling-and-reconstruction idea is genuine: they train without pseudo-pairs by reconstructing the input from a distorted foreground (makeup) plus 3D shape and hierarchical texture details (content). That's a clean departure from the pseudo-ground-truth line. The Laplacian-pyramid detail control and the IDA time-varying alignment module are sensible add-ons, and the code is out. Second, the headline \"SOTA\" numbers are not solid: the FID and CLS are computed against the reference images, which rewards copying the reference's facial texture. SHMT-h4's Key-sim is actually below Stable-Makeup (0.953 vs 0.973 on MT), consistent with the model transferring high-frequency identity details rather than makeup. No identity-similarity metric is reported, so the central claim is not yet distinguished from face-appearance swap.\n\nThe paper does several things well. The ablations support each component: dropping IDA hurts FID/CLS, and the h0-to-h4 texture sweep shows a clear trade-off. Qualitative results, especially on complex makeup, look genuinely better than the GAN baselines. The limitations section honestly acknowledges the face-parsing failure mode.\n\nThe soft spots are the usual ones but they matter here. There are no error bars or significance tests on any metric; the 1000-pair selection is not described with repeatability in mind. The h4 variant is selected post-hoc from the same table that reports its victory. The user study uses 8 participants and no statistics. More fundamentally, the self-supervised training gives no guarantee that the distorted foreground encodes makeup rather than skin albedo, pores, or wrinkles. The paper's own Appendix A shows that as you go from h0 to h4, the model increasingly transfers high-frequency source texture from the reference; that's the leakage vector. I'd want to see an identity metric (e.g., cosine similarity of a face-recognition embedding between source and result) across all methods, and perhaps a perceptual study that isolates makeup fidelity while holding identity.\n\nWho is this for? Researchers in face editing and generative-model evaluation. If the identity-leakage issue is addressed, the self-supervised paradigm is worth building on. As is, it deserves a serious referee—the ideas are novel and the evidence is suggestive—but not publication without a strengthened evaluation. I'd send it to review with a request for an identity-controlled experiment and error bars.\n\nBest,\n[Your name]","headline":"Self-supervised training without pseudo-pairs is a real departure, but the SOTA claim is undercut by metrics that reward copying reference identity; no identity metric is reported.","tokens_in":14670,"tokens_out":3076,"would_cite":true,"duration_ms":27077,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SHMT transfers makeup with a self-supervised latent diffusion model, removing pseudo-paired data and beating seven baselines on realism and makeup fidelity.","keywords":["makeup transfer","latent diffusion models","self-supervised learning","Laplacian pyramid","spatial alignment","face image editing","decoupling-and-reconstruction","Iterative Dual Alignment"],"falsifier":"Compare the makeup representation extracted from an original foreground with the representation from its randomly distorted version on faces with strongly asymmetric makeup: if the distorted representation no longer encodes which eye carried the heavy shadow, or if the reconstruction drops that asymmetry, the self-supervised premise fails.","tokens_in":13697,"feed_emoji":"💄","tokens_out":8165,"duration_ms":68128,"temperature":0.7,"pith_summary":"SHMT is a makeup-transfer method that claims to remove the need for pseudo-paired training data, which previous methods synthesize by histogram matching or geometric warping and which the authors argue misleads the model. The method trains a latent diffusion model to reconstruct a face from separated content and makeup representations, then transfers makeup at inference by swapping in a reference image's makeup representation. To handle the fact that simple and elaborate makeup styles affect the face differently, it decomposes grayscale source texture with a Laplacian pyramid and lets the user choose a fine or coarse detail level. The paper reports that this self-supervised approach surpasses seven baselines, including a diffusion-based rival, on FID, CLS (makeup fidelity), and Key-sim (content preservation), and that the choice of texture level trades content preservation against makeup fidelity.","feed_headline":"Makeup transfer skips fake pairs, beats seven baselines","feed_subtitle":"Choosing fine or coarse texture decides whether freckles stay or heavy makeup comes in, with no pseudo-paired training data.","key_machinery":"The load-bearing mechanism is a 'decoupling-and-reconstruction' pipeline built on a latent diffusion model. A pretrained face parser separates background from foreground; the makeup representation is produced by randomly cropping, rotating, and elastically distorting the foreground, on the premise that these transformations destroy identity and geometry while preserving cosmetic color and texture. The content representation is the combination of a 3D face shape from a face-reconstruction model and one level of a Laplacian pyramid, a multi-scale decomposition of an image into successively coarser high-frequency bands, computed on the grayscale foreground. Fine levels (h0) retain source details such as freckles and eyelashes, while coarse levels (h4) force the model to pull high-frequency makeup from the reference. At each denoising step, the Iterative Dual Alignment module computes a softmax correlation between content and makeup features, deforms the makeup latent with it, then blends in a second alignment computed between the noisy intermediate image and the makeup representation, with the blend weight predicted by a small MLP from the timestep; the mixed condition is injected into the U-Net encoder and the diffusion model is trained with the standard noise-prediction loss.","core_discovery":"On the paper's own terms, SHMT establishes that high-fidelity makeup transfer can be learned entirely from single-image reconstruction rather than from synthesized source-reference pairs. The authors report that the variant SHMT-h4, which injects the coarsest Laplacian texture detail into the content representation, achieves the best FID and CLS values on the MT, Wild-MT, and LADN test sets, while SHMT-h0 achieves the best Key-sim, quantifying an explicit tradeoff between transferring the reference's makeup and preserving the source's content. An additional Iterative Dual Alignment module dynamically re-aligns the makeup feature against the noisy intermediate image at every denoising step, correcting the semantic alignment errors caused by the domain gap between content and makeup representations. The paper also demonstrates flexible controls: staggering SHMT-h0 and SHMT-h4 across denoising timesteps produces seamless interpolation, and mixing aligned makeup feature maps yields global or local makeup interpolation.","pith_inferences":["If random distortion truly cancels content while keeping makeup, the same decoupling recipe should transfer to other localized appearance edits without paired data, such as hair color changes, tattoos, or temporary age effects.","The Laplacian level index acts as a continuous texture-transfer dial; a natural extension would be a single model conditioned on the level rather than one separately trained model per level.","The failure mode the paper shows, face parsing labeling forehead makeup as hair, sets a ceiling on worst-case quality, suggesting that a makeup-aware segmentation module or learned foreground mask would improve robustness.","IDA's timestep-dependent weighting learns when to trust the noisy intermediate image as an anchor; that principle could generalize to any diffusion conditioning task where the condition and the target come from different domains."],"forward_implications":["Choosing texture level h0 preserves source details like freckles, eyelashes, and subtle expressions, while h4 reproduces heavy or high-frequency reference makeup; staggering the two models over denoising timesteps gives a seamless interpolation between those behaviors.","The aligned makeup feature maps support global makeup interpolation between two reference faces and local interpolation restricted to lip or eye regions, with the source image serving as the neutral endpoint.","Because IDA re-aligns against the noisy intermediate image, the model is robust to large differences in pose, expression, age, and gender between source and reference, and it generalizes to sketch and anime images outside its training distribution.","The reported results imply that pseudo-paired data, however carefully synthesized, is not a necessary ingredient for makeup transfer; a reconstruction-only training objective can beat methods that invest in better pseudo-pair generation.","Skin tone can be preserved by interpolating the makeup feature within the face region toward the source's own feature, with the interpolation weight at zero leaving the source skin unchanged."],"supporting_citations":[{"why":"Supplies the latent diffusion backbone, the U-Net denoiser, and the noise-prediction training objective that SHMT adapts.","marker":"[31]"},{"why":"Provides the MT dataset and the histogram-matching pseudo-pairing baseline the method sets out to replace.","marker":"[21]"},{"why":"Defines the Wild-MT dataset and the pose/expression robustness goal, and serves as a GAN-based baseline.","marker":"[17]"},{"why":"Supplies the 3D face shape reconstruction used as the face-shape half of the content representation.","marker":"[13]"},{"why":"Defines the Laplacian pyramid decomposition used to build the hierarchical texture details.","marker":"[2]"},{"why":"Provides the face-parsing model that separates the foreground face from the background for reconstruction.","marker":"[49]"},{"why":"Provides the spatial attention mechanism that computes the correlation matrix for semantically deforming the makeup latent.","marker":"[40]"},{"why":"Establishes the CLS and Key-sim metrics used to measure makeup fidelity and content preservation.","marker":"[39]"},{"why":"Is the diffusion-based makeup transfer baseline with pseudo-paired data that SHMT compares against.","marker":"[52]"},{"why":"Provides the LADN dataset of complex makeup styles and a local adversarial baseline.","marker":"[11]"}],"fun_headline_variants":["Self-supervised makeup transfer without fake pairs via latent diffusion","Hierarchical texture control in makeup transfer from single images","Latent diffusion makeup transfer with iterative alignment and no pseudo pairs","Makeup transfer via Laplacian texture hierarchy and dual alignment","Self-supervised hierarchical makeup transfer with iterative alignment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Randomly cropping, rotating, and elastically warping a face erases its identity and geometry while leaving its makeup intact, and the pretrained face-parsing and 3D-reconstruction models used to separate content are accurate enough not to break the reconstruction.","fun_headline_variants_meta":{"raw":{"variants":["Self-supervised makeup transfer without fake pairs via latent diffusion","Hierarchical texture control in makeup transfer from single images","Latent diffusion makeup transfer with iterative alignment and no pseudo pairs","Makeup transfer via Laplacian texture hierarchy and dual alignment","Self-supervised hierarchical makeup transfer with iterative alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000736,"raw_usage":{"total_tokens":3298,"prompt_tokens":964,"completion_tokens":2334,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":2254}},"tokens_in":580,"tokens_out":2334,"duration_ms":14796,"temperature":1.0,"reasoning_tokens":2254,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:20:09.523000+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the makeup representation extracted from an original foreground with the representation from its randomly distorted version on faces with strongly asymmetric makeup: if the distorted representation no longer encodes which eye carried the heavy shadow, or if the reconstruction drops that asymmetry, the self-supervised premise fails.","supporting_citations":[{"cited_title":"Beautygan: Instance-level facial makeup transfer with deep generative adversarial network","cited_arxiv_id":null,"evidence_quote":"Provides the MT dataset and the histogram-matching pseudo-pairing baseline the method sets out to replace."},{"cited_title":"Psgan: Pose and expression robust spatial-aware gan for customizable makeup transfer","cited_arxiv_id":null,"evidence_quote":"Defines the Wild-MT dataset and the pose/expression robustness goal, and serves as a GAN-based baseline."},{"cited_title":"Towards fast, accurate and stable 3d dense face alignment","cited_arxiv_id":null,"evidence_quote":"Supplies the 3D face shape reconstruction used as the face-shape half of the content representation."},{"cited_title":"The laplacian pyramid as a compact image code","cited_arxiv_id":null,"evidence_quote":"Defines the Laplacian pyramid decomposition used to build the hierarchical texture details."},{"cited_title":"Bisenet: Bilateral segmentation network for real-time semantic segmentation","cited_arxiv_id":null,"evidence_quote":"Provides the face-parsing model that separates the foreground face from the background for reconstruction."},{"cited_title":"Ladn: Local adversarial disentangling network for facial makeup and de-makeup","cited_arxiv_id":null,"evidence_quote":"Provides the LADN dataset of complex makeup styles and a local adversarial baseline."}],"review_version":1}