{"id":"89f4a0a6-a376-4062-a234-56980c3a4194","arxiv_id":"2412.14371","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"SEREP learns a semantic facial expression code from unpaired 3D scans and a semi-supervised image encoder, reporting lower 3D expression error than DECA, EMICA, and SMIRK on the new MultiREX benchmark.","lead":"Monocular face capture usually treats expressions as vertex displacements, which fails when transferring a smile or wink to a different face. This paper instead learns a semantic expression code from 3D scans without paired data, and reports lower 3D expression error than prior methods on a new benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The MultiREX state-of-the-art claim is vulnerable to identity overlap between the unnamed semantic-model training data and the 8 benchmark identities; no identity-disjoint split is reported.","rationale":"The reader's weakest assumption focused on whether the learned z_exp is genuinely shared across identities, enforced only by cycle consistency and the delta loss. My concern is a more immediate confound: even if the code is well-disentangled, the quantitative evidence for generalization is compromised when the benchmark identities appear in the semantic model's training data. This is a related but distinct failure of the same underlying assumption that the representation transfers to unseen identities. The concern is concrete because the training corpus is unnamed, the dynamic training set has 10 subjects while MultiREX has 8 identities, and no identity-disjoint split is stated anywhere in the main text or appendices. I am not claiming the overlap definitely exists; I am claiming that the paper as written does not rule it out, and the test would settle it. The reader already flagged the unnamed 3D training data and possible overlap in the rationale, but did not make it the primary weakest assumption, so agreement is partial. Since the issue is addressable by retraining/reporting rather than a fundamental contradiction, the verdict remains CONDITIONAL/UNCHANGED rather than moving to ACCEPT or REJECT.","tokens_in":15270,"tokens_out":5938,"duration_ms":56732,"concrete_test":"Retrain the full SEREP pipeline (semantic model + synthetic data generation + capture model) with all MultiREX/Multiface identities excluded from the 865 neutral scans and the 10 dynamic sequences, then re-run Tables 2 and 3. Report the overlap between the semantic-model training subjects and the MultiREX subjects before and after exclusion. If the MultiREX average error remains at 2.19 mm (or within noise) and still beats EMICA and DECA, the semantic-generalization claim survives; if the advantage shrinks or disappears, the reported SOTA result is contaminated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim (SEREP beats DECA/EMICA/SMIRK on MultiREX, Tab. 2-3, and preserves identity better in Tab. 1) rests on the assumption that the capture model generalizes to unseen identities. The semantic model is trained on neutral scans from 865 individuals plus dynamic expressive sequences from 10 subjects (Sec. 4.1), but the source dataset is never named. MultiREX is built from 8 Multiface identities (Sec. 5.1). If these 8 identities overlap with the 10 dynamic subjects or the 865 neutral scans, then the semantic encoder and decoder have seen the exact identity geometry, and the synthetic captions generated in Sec. 3.2 are produced from those same identities. The capture model can then exploit identity-specific memory: Eid(N) and Dmesh have memorized how to deform those meshes, so the 2.19 mm average error is not evidence of semantic expression generalization. The paper provides no identity-disjoint train/test split or overlap analysis. This is directly relevant because the method's stated advantage over 3DMM baselines is precisely its identity-conditioned representation. The cycle and delta losses (Eq. 1) are soft regularizers and cannot by themselves rule out memorization, especially when the evaluation identities coincide with synthetic-data identities. Absent this check, the headline outperformance claim is not supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SEREP, a three-stage pipeline for monocular facial expression capture and retargeting. First, a mesh-based autoencoder learns a 64-dimensional expression code and an identity code from unpaired 3D expressive scans, using reconstruction, cycle-consistency, edge, eye-closure, and delta losses (Eq. 1). Second, this model is used to generate synthetic rendered training images with known expression codes (Sec. 3.2). Third, an image encoder is trained on synthetic and real images with a landmark loss and a gradient-reversal domain-adversarial loss to regress the expression code (Eq. 2). The paper also introduces MultiREX, a new 3D geometric benchmark derived from Multiface with eight identities in five viewpoints, and reports that SEREP achieves lower per-vertex error than DECA, EMICA, and SMIRK on this benchmark (Tables 2 and 3) and higher identity-preservation CSIM than FLAME in retargeting (Table 1).","tokens_in":15454,"tokens_out":7668,"duration_ms":66298,"significance":"If the reported results are reliable, the paper makes three useful contributions: a nonlinear, identity-conditioned expression representation that could reduce identity leakage compared to linear 3DMMs; a low-cost synthetic-data generation and domain-adaptation recipe that avoids hand-crafted assets; and a publicly released geometric benchmark for expression capture. The benchmark release and the reproducibility-oriented materials (code, assets, conversion scripts) are valuable to the community. However, the support for the central claims currently depends on two unverified assumptions: that the evaluation identities are disjoint from the identity data used to train the semantic model, and that the predicted 13k-vertex mesh can be fairly evaluated after a conversion that is not described. The paper does not ship machine-checked proofs, but it does provide code and a benchmark, which are strengths.","major_comments":[{"comment":"The MultiREX evaluation protocol uses meshes in the FLAME/Multiface topology, while Dmesh outputs meshes in the 13k-vertex topology used to train the semantic model (Sec. 4.1). No step is described for converting SEREP's predicted meshes to FLAME topology before applying the per-subject FLAME-to-Multiface transform. Without this conversion, the per-vertex errors in Tables 2 and 3 are not reproducible, and the comparison to FLAME-based baselines (which directly output FLAME meshes) is not well-defined. Please specify the conversion procedure (e.g., nearest-neighbor mapping, learned regression, or direct evaluation in SEREP topology after mapping ground truth) and report its effect on the metrics.","section":"Sec. 5.1 and 5.3"},{"comment":"The dataset used to train the semantic model is never named. The model is trained on 865 neutral scans and 10 dynamic subjects (Sec. 4.1), and MultiREX is built from 8 Multiface identities (Sec. 5.1). If the 10 dynamic subjects include any of the 8 MultiREX identities, or if any of the 865 neutral scans correspond to those identities, then the semantic encoder/decoder and the synthetic data generation (Sec. 3.2) have already seen the evaluation identity geometry. In that case, the capture model could exploit identity-specific memory rather than a generic semantic expression code. Please name the dataset, report whether the MultiREX identities are disjoint from all identities used in the semantic model and in synthetic data generation, and, if they are not disjoint, re-run the MultiREX experiment with held-out identities.","section":"Sec. 4.1 and 5.1"},{"comment":"The reconstruction experiment optimizes expression codes per test mesh to minimize Lrec. This measures the representational capacity of the decoder, not the ability of Eexp to produce codes for new identities. The sentence \"our expression model is better able to capture the geometric deformations caused by expressions on unseen subjects\" conflates fitting power with generalization. Please clarify the claim, and if generalization is intended, evaluate with a protocol that does not optimize over the test mesh (e.g., infer codes with Eexp, or use a per-subject calibration expression to factor out identity-specific fitting gains).","section":"Sec. 4.2"},{"comment":"The claim that z_exp encodes the same semantic expression across identities is an assumption. The cycle-consistency loss and delta loss are soft regularizers; they do not by themselves guarantee that the code space is identity-invariant in the intended way. Please provide a direct test: on the 56-subject/19-expression test set, check whether the optimized codes for the same expression index cluster together across identities (e.g., via nearest-neighbor retrieval or a classifier trained on z_exp to predict identity and expression), and report the degree of identity dependence in the codes. This is load-bearing for the retargeting claims.","section":"Sec. 3.1, Eq. (1)"},{"comment":"The in-the-wild robustness claim is supported only by qualitative examples (Fig. 8 and Sec. 5.4) and by the acknowledged trade-off that the full model (with domain loss Ld) underperforms the variant without Ld on MultiREX (Table 2: Ours 2.19 mm vs. Ours (no Ld) 2.10 mm). To substantiate the statement that Ld is necessary for generalization to in-the-wild conditions, please provide a quantitative in-the-wild evaluation, for example landmark alignment error against a detector on a standard test set such as WFLW's test split, or an expression classification metric. Without such a metric, the benefit of the proposed full model over its ablated variant is not measured.","section":"Sec. 5.3 and 5.4"}],"minor_comments":[{"comment":"Typo: \"sythetic\" should be \"synthetic\" in the description of the domain classifier.","section":"Sec. 3.3"},{"comment":"The text reports 61,600 retargeted meshes, but with 56 subjects and 19 fixed expressions (Sec. 4.1) the number should be 58,520 (56 x 55 x 19). Please clarify whether the neutral expression is also included or whether another count is used.","section":"Sec. 4.3"},{"comment":"The caption states bold results are statistically significant via a Wilcoxon test, but no values are bolded in the table as presented in the provided text. Please ensure the final PDF marks significance consistently, and define the number of paired samples used for the test.","section":"Table 1"},{"comment":"The expression encoder is trained with synthetic images that include ground truth codes, but the decoder Dmesh is not fine-tuned after synthetic data generation. A short discussion of how decoder imperfections affect the synthetic captions would help readers assess the self-consistency of the pipeline.","section":"Sec. 5.2"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the potential identity overlap between the unnamed semantic-model training data and the MultiREX evaluation identities. This is a correctness risk for the headline outperformance claim, and it cannot be checked from the paper as written. I would ask the editor to require the authors to name the dataset and provide an identity-disjoint evaluation. The missing topology conversion description is also a reproducibility blocker. The paper has solid elements (new benchmark, code release, a novel training recipe), so I see this as fixable within a revision rather than a reject."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real step in a useful direction. The unpaired semantic expression basis is genuinely different from paired-data models like NPHM and from FACS-based rigs, and the synthetic-data semi-supervised capture scheme with gradient reversal is a reasonable way to get a monocular predictor without rendering-heavy synthetic data. The MultiREX benchmark is the most valuable piece: a public geometric benchmark for expression capture on Multiface with region-based alignment, multiple viewpoints, and released evaluation code. The paper's own ablations are sensible – the delta loss preventing decoder collapse and the identity conditioning for CSIM both show up in the numbers – and the authors are honest that the domain loss hurts the lightstage numbers but helps in-the-wild generalization. Real-time numbers are a nice extra.\n\nThe soft spot is the identity overlap question, and it is load-bearing. The semantic model is trained on 865 neutral scans plus dynamic sequences from 10 subjects, but the source dataset is never named. MultiREX is built from 8 Multiface identities. If those identities are among the 10 dynamic subjects, or even among the 865 neutral scans, the synthetic renders used to train the capture model come from exactly the identities used in the benchmark, and the reported 2.19 mm average could reflect identity-specific memory rather than semantic generalization. The paper reports no identity-disjoint split or overlap analysis. The cycle and delta losses are soft regularizers; they do not by themselves rule this out. This is not a fundamental contradiction – the idea could still be right – but the headline 'outperforms SOTA' claim is not fully supported as written.\n\nOther concerns are minor. The in-the-wild evaluation is qualitative, which the authors acknowledge and is hard to avoid without a geometric benchmark in the wild. The topology conversion between the model's mesh space and the MultiREX evaluation topology is actually described in Sec 5.1 (per-subject sparse conversion matrix from FLAME to Multiface), so that specific worry is overblown. The reconstruction evaluation in Sec 4.2 is a fitting procedure, standard for expression models, and the MultiREX evaluation is on real GT meshes, so that part is not circular. What is missing is method code – only the benchmark assets are released – but that is a reproducibility gap, not a correctness flaw.\n\nWho should read this: anyone working on monocular face capture, retargeting, or expression benchmarks. It deserves a serious referee. I would send it out, but with a clear request: name the training dataset, check overlap with the 8 MultiREX identities, and report results on an identity-disjoint held-out set. If the numbers survive that, the paper is a solid contribution. If not, the benchmark still stands on its own.","headline":"Serious and mostly well-executed advance in semantic facial expression representation; the unaddressed identity-overlap risk in the MultiREX evaluation is the one thing that should decide whether the SOTA claim stands.","tokens_in":16120,"tokens_out":3671,"would_cite":true,"duration_ms":30436,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that representing facial expressions semantically, rather than as vertex displacements, makes monocular capture more accurate across views and preserves identity during retargeting.","keywords":["semantic expression representation","facial performance capture","expression retargeting","3D morphable models","monocular face reconstruction","domain adaptation","synthetic data","MultiREX benchmark"],"falsifier":"Gather a held-out set of identities who each perform the same instructed expression (for example, a closed-mouth smile) in front of a camera, run SEREP on those frames, and compare the distances between their expression codes with distances between codes for different expressions. If the same-expression codes across unseen identities are not substantially closer to each other than to codes of other expressions, the shared-semantic-code assumption is falsified.","tokens_in":14958,"feed_emoji":"🎭","tokens_out":7617,"duration_ms":62772,"temperature":0.7,"pith_summary":"The paper targets a known flaw of 3D morphable models: their expression coefficients produce identical vertex displacements on any face, so the same expression on different identities gets different codes and identity leaks into retargeted results. SEREP replaces this with a semantic expression representation, a latent code trained to mean the same expression regardless of identity, decoded into identity-specific geometry by a network conditioned on the target face. The authors learn this space from unpaired 3D scans with no FACS, emotion, or paired-expression labels, then use it to generate simple synthetic renders that train a monocular capture model together with real in-the-wild images through domain-adversarial learning. On the new MultiREX benchmark, SEREP reports a lower average per-vertex error (2.19 mm) than EMICA (2.36), DECA (2.59), and SMIRK (2.74), and higher identity-preservation during retargeting than FLAME (CSIM 0.791 vs. 0.766). The paper also releases MultiREX as a public benchmark for geometric expression evaluation.","feed_headline":"Semantic expression codes beat vertex shifts in face capture","feed_subtitle":"One code carries the same smile to any face, and SEREP posts lower error on the new MultiREX benchmark.","key_machinery":"The load-bearing object is a semantic expression space learned by a mesh autoencoding triplet: an identity encoder $E_{id}$, an expression encoder $E_{exp}$, and a shared decoder $D_{mesh}$. The expression code $z_{exp}$ is aligned across identities by a cycle-consistency loss $L_{cycle} = \\|z_{exp} - E_{exp}(D_{mesh}(E_{id}(N_t), z_{exp}))\\|^2$, which requires retargeting the same code to another neutral mesh to yield a mesh from which the same code is recovered; a delta loss $L_{delta} = \\|(\\hat{E}_t - N_t) - (E_s - N_s)\\|^2$ keeps the decoder from collapsing toward neutral. For image capture, a ConvNeXt-B encoder predicts the expression code from a single image, with a landmark head shared across real and synthetic images and a gradient-reversal domain classifier pulling the two feature distributions together. The same decoded mesh is used for both reconstruction and retargeting, so the representation does the work of separating what is said from who says it.","core_discovery":"The paper's central claim is that expression should be encoded as a semantic code shared across identities rather than as additive vertex displacements. Starting from unpaired 3D expressive meshes, SEREP learns encoders that map any mesh to an identity code and an expression code, and a decoder that produces identity-specific deformations from the pair. The cycle-consistency and delta losses make the same expression code recover the same semantic expression on a different neutral face. From this representation the method renders simple synthetic images with known expression codes and trains a monocular encoder with a landmark head and a gradient-reversal domain classifier, so the model can capture expressions from real in-the-wild images. On the introduced MultiREX benchmark SEREP reports average per-vertex error of 2.19 mm, lower than EMICA's 2.36, DECA's 2.59, and SMIRK's 2.74, and retargeting CSIM of 0.791 versus FLAME's 0.766.","pith_inferences":["Beyond the paper: the same semantic code should make video capture temporally smoother, because the representation is viewpoint-independent; one could test this by measuring frame-to-frame code jitter on MultiREX sequences.","Beyond the paper: because the code is learned without labels, its axes may not match FACS action units or emotion categories; probing which dimensions drive which visible motions would test how semantically interpretable the space is.","Beyond the paper: the domain-adversarial loss creates an explicit trade-off between lightstage accuracy and in-the-wild robustness; sweeping the gradient-reversal scale on MultiREX would quantify that trade-off.","Beyond the paper: the synthetic pipeline is deliberately minimal (random texture, environment map, no hair or clothing), so the method suggests a recipe for generating training data from any semantic face model; adding more diverse synthetic identities should improve capture further."],"forward_implications":["Expression capture training can rely on cheap synthetic renders plus unlabeled real images, avoiding expensive photo-real synthetic datasets and differentiable-rendering supervision.","Retargeting a captured performance to a new character should better preserve the target actor's identity, because the code is semantic rather than additive vertex offsets.","Side-view and profile capture should be more stable, since the expression is represented as a whole in 3D space rather than reconstructed from image-space cues that disappear at steep angles.","Larger unpaired 3D expression corpora become usable for training, since subjects no longer need to perform identical posed expressions for paired supervision.","MultiREX gives the field a geometric, region-based, multi-view protocol for comparing expression capture methods directly in 3D."],"supporting_citations":[{"why":"FLAME, the linear 3DMM whose additive expression basis SEREP is designed to replace; provides the main baseline for reconstruction and retargeting comparison.","marker":"[31]"},{"why":"Multiface dataset, the source of the monocular videos and ground-truth meshes on which the MultiREX benchmark is built.","marker":"[47]"},{"why":"EMICA, one of the main state-of-the-art expression-capture baselines compared on MultiREX and in-the-wild retargeting.","marker":"[10]"},{"why":"DECA, a widely used in-the-wild face reconstruction baseline whose per-vertex error is compared on MultiREX.","marker":"[16]"},{"why":"SMIRK, a recent neural-synthesis expression capture method used as a baseline on MultiREX and in the wild.","marker":"[36]"},{"why":"Cycle consistency loss, the mechanism that aligns expression codes across identities without paired data.","marker":"[51]"},{"why":"Domain-adversarial training by gradient reversal, used to bridge the synthetic-to-real gap in the capture model.","marker":"[20]"},{"why":"REALY benchmark, whose region-based rigid-alignment evaluation protocol MultiREX adapts.","marker":"[5]"},{"why":"ConvNeXt-B backbone used as the image encoder of the expression capture model.","marker":"[33]"}],"fun_headline_variants":["Semantic expression codes unify face capture and retargeting","SEREP posts lowest error on MultiREX with semantic codes","Semantic codes, not vertex shifts: SEREP improves face capture","Same expression code transfers smile to any avatar","SEREP captures expressions semantically, beats state-of-the-art"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim depends on a single learned expression code genuinely meaning the same expression on every identity, with the cycle-consistency and delta losses enforcing that shared semantics without paired data or labels; if the code only aligns on training identities but drifts on unseen faces, the capture and retargeting improvements are not guaranteed.","fun_headline_variants_meta":{"raw":{"variants":["Semantic expression codes unify face capture and retargeting","SEREP posts lowest error on MultiREX with semantic codes","Semantic codes, not vertex shifts: SEREP improves face capture","Same expression code transfers smile to any avatar","SEREP captures expressions semantically, beats state-of-the-art"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000838,"raw_usage":{"total_tokens":3627,"prompt_tokens":889,"completion_tokens":2738,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":2654}},"tokens_in":505,"tokens_out":2738,"duration_ms":18557,"temperature":1.0,"reasoning_tokens":2654,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:18:22.374574+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Gather a held-out set of identities who each perform the same instructed expression (for example, a closed-mouth smile) in front of a camera, run SEREP on those frames, and compare the distances between their expression codes with distances between codes for different expressions. If the same-expression codes across unseen identities are not substantially closer to each other than to codes of other expressions, the shared-semantic-code assumption is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FLAME, the linear 3DMM whose additive expression basis SEREP is designed to replace; provides the main baseline for reconstruction and retargeting comparison."},{"cited_title":"Multiface: A dataset for neural face rendering","cited_arxiv_id":null,"evidence_quote":"Multiface dataset, the source of the monocular videos and ground-truth meshes on which the MultiREX benchmark is built."},{"cited_title":"Emotional speech- driven animation with content-emotion disentanglement","cited_arxiv_id":null,"evidence_quote":"EMICA, one of the main state-of-the-art expression-capture baselines compared on MultiREX and in-the-wild retargeting."},{"cited_title":"Black, and Timo Bolkart","cited_arxiv_id":null,"evidence_quote":"DECA, a widely used in-the-wild face reconstruction baseline whose per-vertex error is compared on MultiREX."},{"cited_title":"3d facial expressions through analysis-by- neural-synthesis","cited_arxiv_id":null,"evidence_quote":"SMIRK, a recent neural-synthesis expression capture method used as a baseline on MultiREX and in the wild."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Cycle consistency loss, the mechanism that aligns expression codes across identities without paired data."},{"cited_title":"Unsupervised domain adaptation by backpropagation","cited_arxiv_id":null,"evidence_quote":"Domain-adversarial training by gradient reversal, used to bridge the synthetic-to-real gap in the capture model."},{"cited_title":"Realy: Rethinking the evaluation of 3d face reconstruction","cited_arxiv_id":null,"evidence_quote":"REALY benchmark, whose region-based rigid-alignment evaluation protocol MultiREX adapts."}],"review_version":1}