{"id":"a3eac6a8-7431-464e-9ca7-2a0eda8ba5dc","arxiv_id":"2505.01932","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"OT-Talk trains a HuBERT-transformer-ChebNet model with a sliced Wasserstein distance loss on mesh-as-probability-measure representations and reports the best lip and face errors on VOCASET and Multiface.","lead":"This paper introduces a speech-driven 3D talking head animation system that uses optimal transport (sliced Wasserstein distance) as a training loss to make predicted lip and face motions smoother and more accurate. It reports state-of-the-art results on two audio-mesh datasets and a small user study.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Superiority claim rests on an unverifiable comparison: Multiface baselines are not shown to be trained to convergence, and the DTW_lip metric in Tables 1–2 is never defined.","rationale":"The reader's CONDITIONAL verdict is appropriate, and my read does not move it. The method is plausible: ChebNet mesh encoding, Hubert audio features, a transformer temporal model, and a sliced Wasserstein loss are standard, coherent building blocks, and the ablation study gives some internal evidence that each component contributes. However, the central sentence in Section 4.1 is an empirical superiority claim over three published baselines. That claim depends on three things the paper does not supply: evidence that the Multiface baselines were trained properly, a definition of the DTW_lip metric, and error bars or significance tests. The qualitative figures and the 20-volunteer user study are suggestive but not sufficient, especially without reported variance or statistical testing. These are addressable conditions rather than demonstrated flaws, so the correct response is not to reject but to require release of code, trained models, evaluation scripts, and baseline training details before the superiority claim can be accepted.","tokens_in":14063,"tokens_out":11604,"duration_ms":127356,"concrete_test":"Run a reproducibility audit: obtain the authors' released code and evaluation scripts, retrain the three baselines on Multiface with the paper's exact split using three random seeds and validation-based early stopping, and recompute all five metrics using a released definition of DTW_lip. If the published baseline numbers are not reproduced, or if OT-Talk no longer wins every metric with non-overlapping confidence intervals, the central superiority claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 states \"Our method achieves the best score on all metrics for both datasets.\" For that claim to hold, the comparison protocol must be fair and the metrics must be well-defined. On Multiface, the paper only says it \"adhere[s] to the official implementations for both training and testing\"; it reports no training curves, epoch counts, hyperparameter settings, seeds, or validation-based checkpointing for VOCA, FaceFormer, or CodeTalker. If those baselines were undertrained or run with settings unsuited to Multiface's 30 FPS data and different mesh topology, their reported errors would be inflated and OT-Talk's margins would be an artifact. On VOCASET, official pretrained weights are used, but all baselines are evaluated under a nonstandard \"select the best-performing identity\" protocol, and it is unclear whether the same identity choice is used for every metric and in the user study. Independently, the temporal-alignment metric DTW_lip in Tables 1 and 2 is never defined anywhere in the paper, so the claim of better temporal alignment cannot be checked or reproduced. No error bars or significance tests are reported, making it unknown whether margins such as E_head_mean 0.87 vs. 1.00 mm on VOCASET are meaningful.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript introduces OT-Talk, an audio-driven 3D talking-head animation method whose main components are a HuBERT audio front-end, a transformer temporal encoder, a ChebNet mesh encoder/decoder, and an auxiliary sliced Wasserstein distance (SWD) loss computed on varifold-encoded mesh measures. The authors claim that this is the first use of optimal transport for talking-head animation and that it yields smoother and better-synchronized motion. Experiments on VOCASET and Multiface compare with VOCA, FaceFormer, and CodeTalker, reporting per-frame lip/face/head errors, a temporal-alignment metric DTW_lip, ablations, qualitative side-by-sides, and a 20-participant user study. The central quantitative claim is stated in Section 4.1: 'Our method achieves the best score on all metrics for both datasets.'","tokens_in":14284,"tokens_out":7040,"duration_ms":72827,"significance":"If the evaluation is accepted, the contribution is a sensible and reasonably novel engineering combination: SWD as a training regularizer on mesh measures is computationally attractive, the ChebNet backbone is a natural fit for fixed-topology meshes, and the ablation study gives evidence that each component helps. The paper also provides enough implementation detail (loss terms, hyperparameters, architecture sizes) to reproduce the method itself. However, the strength of the claimed superiority is currently not established because several evaluation choices are underspecified or undefined, and no uncertainty quantification is provided. The theoretical novelty is moderate: the OT formulation is largely a loss-term swap, and the varifold-plus-SWD construction needs clarification. There is no circularity problem: the SWD is an additional training objective, and the reported metrics are computed independently of that loss.","major_comments":[{"comment":"DTW_lip is reported in Tables 1 and 2 but is never defined anywhere in the manuscript. It is used to support the temporal-alignment part of the central claim, so the reader cannot check what is being measured, what the reference/query alignment is, or why the values are on the order of 600-900 (labeled mm). Please define the metric precisely (input trajectories, distance measure, warping constraints, normalization, units) and state the implementation used.","section":"Section 4.1, Tables 1-2"},{"comment":"The fairness of the comparison is not fully documented. For Multiface, the paper only states that the authors adhere to the official implementations; no training epochs, hyperparameters, seeds, or validation-based checkpointing are given for VOCA, FaceFormer, or CodeTalker on this dataset. For VOCASET, pre-trained weights are used but the protocol selects the best-performing identity condition among all identities; the paper should state whether the same identity choice is used for every metric and in the qualitative/user-study comparisons, and should justify why this protocol is not biased. Without these details, the claimed margins (e.g., 0.87 mm versus 1.00 mm head error on VOCASET) cannot be attributed to the method rather than to comparison logistics.","section":"Section 4.1, Baseline Methods"},{"comment":"No error bars, repeated runs, or significance tests are reported. Several margins are small relative to plausible run-to-run variation (e.g., E_lip_mean 2.81 versus 2.93 mm and E_head_mean 0.87 versus 1.00 mm on VOCASET), and the ablation rows are single numbers. Please report standard deviations over test subjects or repeated training runs, and give a significance test or bootstrap confidence intervals for the main comparisons.","section":"Tables 1-2"},{"comment":"The varifold construction places the mesh measure on R^3 x S^2 (position plus oriented normal), but the sliced Wasserstein distance is defined only for measures on R^d and the projection directions satisfy theta in S^{d-1}; the ambient dimension d is never specified. Clarify whether the SWD is computed on the six-dimensional (position, normal) vectors, and if so justify this choice; if only vertex positions are projected, explain what role the varifold/normal component plays. As written, the central OT step is ambiguous.","section":"Section 3.2, Eqs. (4)-(7)"},{"comment":"The abstract and Section 4.1 say the method outperforms state-of-the-art techniques, but the comparison set omits several recent methods discussed in Section 2, especially ScanTalk, which is designed for unregistered scans and multiple mesh topologies, as well as the diffusion-based FaceDiffuser and DiffPoseTalk. Either add at least one of these baselines or qualify the claim as applying to the three compared methods.","section":"Section 4, Baseline set"}],"minor_comments":[{"comment":"The model name is written inconsistently as 'Hubert Large' and 'Huber model'; it should be 'HuBERT Large' throughout.","section":"Sections 3.3-3.4"},{"comment":"The caption says the words are 'spoken by 2 subjects from VOCASET', but the figure and surrounding text describe Multiface; fix the caption.","section":"Section 4.3, Figure 4 caption"},{"comment":"'toal loss' should be 'total loss'.","section":"Section 3.4, Eq. (11)"},{"comment":"W denotes the adjacency matrix in Section 3.1 but denotes the network weights in Eq. (11); rename one of them to avoid ambiguity.","section":"Sections 3.1 and 3.4"},{"comment":"On VOCASET, removing the transformer gives E_head_mean 0.86 mm, slightly lower than the 0.87 mm of OT-Talk; the claim that 'other components also help' should be qualified to acknowledge this exception.","section":"Table 2"},{"comment":"Report the number of A/B responses and per-participant variance; with 20 volunteers, percentages such as 52.71% are based on a small sample and should be accompanied by confidence intervals.","section":"Section 4.4"},{"comment":"The claim of being 'the first approach to leverage optimal transportation' is stronger than Section 2.3 supports, since prior work already uses OT for mesh comparison; recommend saying 'first for audio-driven 3D talking-head animation'.","section":"Abstract and Section 2.3"}],"recommendation":"major_revision","confidential_remarks":"The main barrier to acceptance is evaluation rather than method design: with a defined DTW_lip, full baseline training details, and uncertainty quantification, the central claim would be far better supported. The paper appears to be a published ICMR paper; if it is being considered as a journal extension, the added value beyond the conference version should be clarified. I would not reject on the current evidence, but I would not accept without the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: OT-Talk is a competently assembled talking-head pipeline whose genuinely new piece is a sliced Wasserstein distance loss computed on mesh-as-probability-measure representations, and the ablation argues that this loss buys real lip-sync and geometry accuracy. The engineering is solid, but the paper's central 'best on all metrics' claim rests on evaluation details that are not verifiable from the text, especially the undefined DTW_lip metric and the Multiface baseline setup.\n\nWhat's good: The idea of treating meshes as varifold-type measures and using SWD as a training loss is clean and computationally sensible (O(L m log m)). The ablation study is helpful: removing ChebNet tanks performance, removing the OT loss hurts lip metrics on both datasets, and the transformer helps temporal alignment. That's an honest breakdown. The CoMA-style mesh resampling and PCA initialization are standard but credited, and the writing is straightforward.\n\nWhere it gets soft: DTW_lip appears in both tables but is never defined anywhere, which is a serious reproducibility gap. On Multiface, the baselines are 'official implementations' with no training details; if they weren't tuned to Multiface's 30 FPS or different mesh topology, the comparison could be unfair. The practice of evaluating baselines under the best of 8 identity conditions is unusual; if that identity is selected per metric, it's actually generous to the baselines, but the paper doesn't say whether the same identity is used in the user study. No error bars or significance tests; a margin like 0.87 vs 1.00 mm needs variance to mean anything. The mathematical transition from varifold measure (with normals) to SWD in Eqs. (4)-(10) is underspecified: are the measures on R^3 or R^3 x S^2? And the comparison omits ScanTalk, FaceDiffuser, DiffPoseTalk, and SelfTalk, all directly relevant and possibly affecting the 'first OT' and SOTA claims.\n\nNone of this is fatal to the core idea. The OT loss is novel and the positive ablation is the strongest evidence. But the superiority claim is not currently established.\n\nI'd send this to a serious referee if I were an editor; it deserves constructive review, with a strong request for code, defined metrics, and convergence-aware baseline comparisons. For a reading group, it's a decent loss-design paper, though I'd only bring it up if the group cares about talking heads or optimal transport in graphics.\n\nRecommendation: engage constructively; major revision before trusting the numbers.","headline":"A clean OT-loss idea with a solid ablation, but the superiority claim is undercut by an undefined metric and fair-comparison gaps.","tokens_in":14854,"tokens_out":2589,"would_cite":true,"duration_ms":26132,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Optimal transport metric sharpens lip sync in 3D talking-head animation","keywords":["optimal transport","sliced Wasserstein distance","speech-driven animation","3D talking head","Chebyshev graph convolution","mesh deformation","VOCASET","Multiface"],"falsifier":"Re-run VOCA, FaceFormer, and CodeTalker with the same training budget, evaluation script, and mask definitions as OT-Talk, report per-seed error bars, and compute DTW_lip from an explicit definition; if OT-Talk no longer wins on all metrics, or if the DTW advantage disappears under a different lip mask, the central claim fails.","tokens_in":1444,"feed_emoji":"🗣️","tokens_out":3111,"duration_ms":56006,"temperature":0.7,"pith_summary":"This paper introduces OT-Talk, a speech-driven 3D talking-head model whose training objective includes a sliced Wasserstein distance between predicted and ground-truth meshes represented as probability measures. It claims this optimal-transport loss, combined with Chebyshev graph convolutions on the mesh, produces smoother facial motion and better lip synchronization than existing models. Experiments on VOCASET and Multiface report the best scores on lip, face, and head reconstruction errors and on temporal alignment, and a 20-volunteer perception study favors it. The paper's central claim is stated directly: it achieves the best score on all metrics for both datasets. The claimed mechanism is that comparing surface distributions, not just per-vertex $\\ell^2$ differences, captures whole-mouth and global-shape errors.","feed_headline":"Optimal transport sharpens lip sync in 3D talking heads","feed_subtitle":"A sliced-Wasserstein loss plus mesh graph features beats three baselines on reconstruction and timing.","key_machinery":"The central object is the oriented varifold representation of a triangulated mesh: each face contributes a Dirac mass at its barycenter with weight equal to face area, embedded in a position-and-normal space. Projecting these measures onto random directions reduces comparison to one-dimensional Wasserstein distances, giving the sliced Wasserstein distance with $O(L m \\log m)$ cost. ChebNet graph convolution supplies geometric features on the mesh graph, and the full loss combines per-vertex reconstruction, velocity, sliced Wasserstein, and weight regularization. The SWD term does the load-bearing work of making the network optimize surface-level agreement rather than isolated vertex positions.","core_discovery":"On its own terms, the paper claims that a talking-head animation network can be trained to compare meshes not by vertex coordinates alone but as discrete probability measures supported on mesh faces, and that the sliced Wasserstein distance between those measures is a tractable training loss. With this loss plus a ChebNet mesh encoder, a HuBERT front-end, and a transformer, the model predicts vertex displacements that the paper reports as quantitatively closer to ground truth and more temporally aligned than VOCA, FaceFormer, and CodeTalker on two datasets. The claimed mechanism is that the optimal-transport distance measures the whole surface distribution, penalizing global shape and lip-location discrepancies in a way that per-vertex $\\ell^2$ and velocity losses miss.","pith_inferences":["Because the sliced Wasserstein distance compares the distribution of face-area mass, it should be particularly sensitive to whole-mouth opening and closing; a direct test would compare OT-Talk's predicted lip apertures to ground truth frame by frame.","The paper never defines the DTW_lip temporal metric, so its temporal-alignment advantage cannot be independently reproduced without an explicit definition and evaluation code.","The baselines were evaluated with their official pretrained weights on VOCASET and official implementations on Multiface, without matched training budgets or significance tests; a fairer head-to-head with comparable tuning could narrow the reported gap.","The varifold-plus-sliced-Wasserstein loss could plausibly benefit other mesh-deformation tasks such as cloth or soft-body simulation, though the paper does not test that."],"forward_implications":["If the best-on-all-metrics claim holds, OT-Talk offers real-time audio-driven animation at about 0.007 seconds per frame.","The sliced Wasserstein loss, being mesh-agnostic in how it forms probability measures, transfers across datasets with different mesh sizes, as demonstrated on VOCASET and Multiface.","The ablation indicates ChebNet is the largest contributor to reconstruction accuracy, while the OT loss specifically improves lip-region errors and temporal alignment.","Because identity is encoded from mesh geometry rather than one-hot labels, the method can animate unseen identities without retraining for a fixed identity set."],"supporting_citations":[{"why":"VOCA is the primary baseline and supplies the VOCASET training/validation/test split and the identity-conditioning setup.","marker":"[11]"},{"why":"FaceFormer is a transformer-based baseline whose official pretrained weights are used for comparison.","marker":"[17]"},{"why":"CodeTalker is a discrete-motion-prior baseline whose official implementation is used for comparison.","marker":"[51]"},{"why":"ChebNet supplies the Chebyshev spectral graph convolution that the mesh encoder is built on.","marker":"[14]"},{"why":"This reference defines the sliced Wasserstein distance used as the OT loss.","marker":"[4]"},{"why":"CoMA provides the mesh downsampling/upsampling transform strategy used for hierarchical mesh features.","marker":"[39]"},{"why":"HuBERT supplies the pretrained audio feature extractor used in the pipeline.","marker":"[24]"},{"why":"Multiface is one of the two evaluation datasets and supplies audio-mesh pairs.","marker":"[50]"}],"fun_headline_variants":["Wasserstein loss achieves smoother 3D talking heads","Optimal transport improves lip sync in 3D avatars","Sliced Wasserstein distance yields natural facial motion","Mesh probability measures sharpen talking-head animation","OT-Talk: Better lip sync via optimal transport"],"cache_read_input_tokens":17024,"weakest_assumption_plain":"The superiority claim rests on the comparison protocol: official pretrained baselines on VOCASET, official implementations on Multiface, no error bars or significance tests, and a temporal metric called DTW_lip that is never defined; if the baselines were not at their best or the metric was computed inconsistently, the claim of being best on all metrics would not be established.","fun_headline_variants_meta":{"raw":{"variants":["Wasserstein loss achieves smoother 3D talking heads","Optimal transport improves lip sync in 3D avatars","Sliced Wasserstein distance yields natural facial motion","Mesh probability measures sharpen talking-head animation","OT-Talk: Better lip sync via optimal transport"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00053,"raw_usage":{"total_tokens":2544,"prompt_tokens":927,"completion_tokens":1617,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":1541}},"tokens_in":543,"tokens_out":1617,"duration_ms":11948,"temperature":1.0,"reasoning_tokens":1541,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:06:45.397334+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run VOCA, FaceFormer, and CodeTalker with the same training budget, evaluation script, and mask definitions as OT-Talk, report per-seed error bars, and compute DTW_lip from an explicit definition; if OT-Talk no longer wins on all metrics, or if the DTW advantage disappears under a different lip mask, the central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FaceFormer is a transformer-based baseline whose official pretrained weights are used for comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This reference defines the sliced Wasserstein distance used as the OT loss."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CoMA provides the mesh downsampling/upsampling transform strategy used for hierarchical mesh features."}],"review_version":1}