Pith. sign in

REVIEW 2 major objections 4 minor 69 references

Text can drive precise 3D facial expressions by regressing 3D Morphable Model parameters through a language model token.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 07:48 UTC pith:AWFMU5NA

load-bearing objection Clean 3DMM-regression framing plus a dual-caption dataset and an MLLM <Expr> token that beats CLIP and diffusion text-to-3D heads on expression metrics inside the FLAME ceiling. the 2 major comments →

arxiv 2607.02674 v1 pith:AWFMU5NA submitted 2026-07-02 cs.CV

EmoteGPT: 3D Human Facial Expressions from Natural Language Descriptions

Haoran Wang , Mohit Mendiratta , Christian Theobalt , Adam Kortylewski This is my paper
classification cs.CV
keywords 3D facial expressiontext-to-3D3D Morphable ModelFLAMEmultimodal large language modelTxt2Emoteexpression controlavatar animation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that fine-grained control of 3D faces from language is better solved by predicting only expression parameters of a 3D Morphable Model, rather than jointly generating identity, expression, and texture. Existing resources lack the necessary paired language-to-expression data, so the authors build Txt2Emote: roughly 30k tracked 3D expressions annotated with both explicit facial-feature descriptions and implicit situational descriptions produced by GPT-4o. On top of this data they train EmoteGPT, a multimodal language model that emits a dedicated expression token whose hidden state is decoded into FLAME parameters. Adding large-scale image-to-expression supervision further strengthens the mapping. The resulting system is claimed to outperform prior CLIP regressors and diffusion-based text-to-3D head methods on emotion-recognition metrics and in a user preference study, while remaining compatible with existing avatar pipelines for photorealistic, stylized, and personalized faces.

Core claim

EmoteGPT, an MLLM equipped with a dedicated <Expr> token whose representation is decoded into 3DMM expression parameters, produces more accurate and expressive 3D faces from both explicit and implicit natural-language descriptions than existing text-to-3D face methods, once trained on the new Txt2Emote pairs plus image-to-3DMM data.

What carries the argument

The <Expr> token: a new vocabulary item whose final-layer hidden state is projected by a lightweight MLP into FLAME expression and jaw parameters, thereby grounding language inside the compact, disentangled 3DMM expression space.

Load-bearing premise

That monocular face-tracker reconstructions plus GPT-4o captions form accurate enough ground truth for both training and the held-out benchmark; if either systematically misrepresents expressions, the measured gains partly measure agreement with those proxies rather than true expression fidelity.

What would settle it

A controlled test set of faces whose expression parameters are obtained from multi-view or marker-based capture (not monocular EMOCAv2) and whose captions are written by human experts rather than GPT-4o; if EmoteGPT’s emotion-recognition and geometric advantages disappear on that set, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper formulates text-driven 3D facial expression synthesis as regression in the disentangled FLAME 3DMM parameter space rather than joint identity-expression-texture generation. It introduces the Txt2Emote dataset of ~30k expressions (from AffectNet via EMOCAv2) paired with GPT-4o-generated explicit (feature-level) and implicit (situational) captions, plus a 2.5k held-out benchmark. EmoteGPT builds on LLaVA-1.5 with a dedicated <Expr> token whose hidden state is decoded by a lightweight MLP into expression and jaw parameters; training mixes Txt2Emote text pairs, large-scale image-to-3DMM data, and VQA instruction data under a combined CE + parameter L2 + weighted mesh L1 objective. Experiments claim superior valence/arousal CCC, emotion accuracy, region-wise L1, and user preference (~75%) over CLIP/LLM regressors, Describe3D, HumanNorm and Portrait3D, with seamless integration into GaussianAvatar, HeadStudio and DiffusionRig pipelines.

Significance. If the empirical gains hold under independent scrutiny, the work supplies a practical, real-time interface for fine-grained expression control of existing FLAME-based avatars and a reusable paired resource that was previously missing. The multimodal training strategy and <Expr> grounding are clean engineering contributions that demonstrably improve generalization from explicit to implicit language; the honest Limitations section and public project page further strengthen reproducibility. These elements advance text-driven 3D human-centric generation beyond entangled diffusion pipelines and are of clear interest to the graphics and multimodal communities.

major comments (2)
  1. Sections 3.1, 4.3 and Supplementary B construct every training and test expression label by running EMOCAv2 on AffectNet images; Table 2 then reports L1 distances to those same parameters and trains a separate emotion-recognition MLP (valence/arousal CCC + 8-class ACC) on each method’s predicted parameters against the original AffectNet labels. Because EMOCAv2 itself was optimized with emotion-consistency losses, the automatic metrics are partly self-referential: any systematic under-expression or bias of the monocular tracker is shared by both the “ground truth” and the scoring networks. The 25-participant user study (Table 4) still compares FLAME meshes obtained by the identical tracker. This does not invalidate relative rankings inside the FLAME space, but it does weaken the absolute claim of surpassing SOTA “on emotion recognition metrics and in perceived expressiveness.” An independ
  2. Table 3 and the qualitative protocol of §5.2 evaluate diffusion-based full-head methods (HumanNorm, Portrait3D) only after fitting FLAME via the same EMOCAv2 tracker used for EmoteGPT’s own labels. While this isolates expression quality, it discards any geometric or textural advantages of the competing pipelines and reduces the comparison to “how well can we recover FLAME parameters from their outputs.” The paper should either (a) report additional metrics that do not require re-fitting (e.g., CLIP-score or human preference on the original textured heads) or (b) explicitly qualify that superiority is claimed only inside the FLAME expression subspace.
minor comments (4)
  1. §4.3 and Eq. (2): the region-dependent vertex weight map w is referenced to [55] but never defined or visualized; a short description or supplementary figure would aid reproducibility.
  2. Table 2 caption and surrounding text claim “best generalization ability even when trained without implicit descriptions,” yet the zero-shot implicit numbers for EmoteGPT (V-CCC 0.49) remain modest; a brief discussion of residual failure modes (Supplementary Fig. 9) would temper the claim.
  3. Several extraction artifacts appear in the supplied manuscript (e.g., “Theyalsostruggletocapturenuanced,” “CombiningEmoteGPT”); these should be cleaned for the camera-ready version.
  4. Supplementary B: the decision to train a separate emotion MLP for every method is non-standard and under-motivated; a single fixed classifier operating on mesh vertices or a common parameter space would make cross-method comparisons more transparent.

Circularity Check

0 steps flagged

No derivation-by-construction circularity; standard supervised regression on externally-sourced (EMOCAv2 + GPT-4o) labels with held-out evaluation and an independent user study.

full rationale

EmoteGPT is an empirical MLLM regression model (Eq. 1–2) that maps text (or images) to FLAME expression parameters via a learned <Expr> token + MLP decoder. Training and the Txt2Emote benchmark both obtain expression parameters from the external monocular tracker EMOCAv2 and captions from GPT-4o (Sec. 3.1, 4.3), and automatic metrics (V/A CCC, E-ACC, L1) are computed against those same parameters (Table 2, Supp. B). This shared label source can introduce common bias (acknowledged in Limitations), yet it does not make any claimed prediction equal to its inputs by definition or by a fitted free parameter: the model must still learn a non-trivial text-to-parameter mapping, is evaluated on held-out samples, and is compared against independently trained baselines (CLIP, Vicuna, Describe3D, HumanNorm, Portrait3D). The 25-person perceptual study (Table 4) supplies an external ranking signal. No self-definitional equation, no uniqueness theorem imported from the authors, no ansatz smuggled via self-citation, and no renaming of a known result appear. The residual self-referential flavor of the automatic metrics is a typical, low-severity limitation of supervised 3D face papers rather than circularity under the stated criteria; score therefore remains 0–2.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

Empirical CV paper whose central claim rests on the quality of automatically generated paired data and on the representational adequacy of FLAME. Hyper-parameters are conventional; the main load-bearing assumptions are domain-level (tracker fidelity, caption fidelity, model capacity).

free parameters (3)
  • loss weights (λ_txt, λ_expr, λ_mesh) = 1.0 / 1.0 / 0.01
    Hand-chosen coefficients (1.0, 1.0, 0.01) that balance text CE against parameter and mesh losses; no sensitivity analysis reported.
  • learning rate and LoRA rank = 4e-5
    AdamW 4e-5 with WarmupDecayLR and unspecified LoRA rank; standard but free choices that affect final numbers.
  • landmark filtering thresholds = 17.9 / 1.25
    Mean L1 landmark error cut-offs (17.9 full face, 1.25 lips) used to discard bad EMOCA reconstructions; chosen by inspection.
axioms (4)
  • domain assumption FLAME 2020 expression + jaw parameters form a sufficiently expressive and identity-disentangled space for the target task.
    Stated in §4.1 and Limitations; paper acknowledges incomplete disentanglement and missing wrinkles.
  • domain assumption EMOCAv2 monocular reconstructions supply accurate enough ground-truth expression parameters for both training and evaluation.
    Core of dataset construction (§3.1) and all quantitative tables; Limitations notes residual errors under extreme pose/lighting.
  • domain assumption GPT-4o captions (explicit and implicit) faithfully describe the facial expression present in each image.
    Used for all text supervision; prompts given in Supp. Figs. 11-12; Limitations notes possible cultural/contextual bias.
  • ad hoc to paper A lightweight MLP decoder on the <Expr> hidden state is adequate to map language embeddings into continuous 3DMM parameters.
    Architectural choice in §4.2; ablated against mean-pooling and <EOS> alternatives (Table 6).
invented entities (2)
  • Txt2Emote dataset no independent evidence
    purpose: Provide the missing paired text–3DMM expression supervision (explicit + implicit) at 30k scale.
    Constructed by the authors from AffectNet + EMOCAv2 + GPT-4o; no independent public release yet.
  • <Expr> vocabulary token no independent evidence
    purpose: Semantically ground continuous facial-expression representations inside the MLLM’s discrete token space so a decoder can extract them.
    Introduced in §4.2; performance gain shown by ablation, but the token itself has no existence outside this model.

pith-pipeline@v1.1.0-grok45 · 26802 in / 3051 out tokens · 33653 ms · 2026-07-12T07:48:08.560669+00:00 · methodology

0 comments
read the original abstract

Precise control of 3D facial expressions from text is crucial for virtual avatars, animation, and human-computer interaction, yet existing text-to-3D methods jointly generate identity, expression, and texture, making fine-grained expression control difficult. We instead formulate text-driven expression synthesis as a regression problem in the disentangled parameter space of a 3D Morphable Model (3DMM). This setting, however, requires paired data linking detailed language to precise expression parameters, which are missing from existing resources. To fill this gap, we introduce Txt2Emote, a benchmark of diverse 3D facial expressions with fine-grained textual annotations obtained from GPT-4o and a high-fidelity face tracker, providing both explicit descriptions detailing facial features and implicit descriptions referencing the situational context behind the expression. Leveraging this dataset, we present EmoteGPT, a text-to-3D expression framework based on a Multimodal Large Language Model (MLLM) with a dedicated <Expr> token to semantically ground expression representations, which are then decoded into 3DMM parameters. We further improve EmoteGPT by augmenting training with large-scale image-to-3DMM data, enabling it to surpass state-of-the-art text-to-3D face synthesis methods on emotion recognition metrics and in perceived expressiveness. Integrated into avatar pipelines, our method enables photorealistic and stylized 3D avatars, as well as expressive 3D-consistent 2D face synthesis from textual input.

Figures

Figures reproduced from arXiv: 2607.02674 by Adam Kortylewski, Christian Theobalt, Haoran Wang, Mohit Mendiratta.

Figure 1
Figure 1. Figure 1: EmoteGPT generates 3D facial expressions from text, supporting two types of descriptions: (1) Explicit ones, which detail physical facial features and overall emotional impressions; (2) Implicit ones, which reference events or situations that evoke facial expressions. The expressions are represented as 3DMM parameters and can be seamlessly integrated into existing frameworks [32, 48, 53] to enable expressi… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of EmoteGPT. Our framework generates 3D facial expressions from diverse inputs by combining a Multimodal Large Language Model (MLLM) with an expression decoder head (η). EmoteGPT is trained on diverse multimodal supervi￾sion, including explicit textual descriptions (e.g., “a broad smile with narrowed eyes”), implicit contextual prompts (e.g., “meeting an old friend unexpectedly”), and images. Duri… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative comparison of EmoteGPT with relevant methods. Results are vi￾sualized in both the 3DMM space and rendered 3D avatars using GaussianAvatar. Existing models often struggle with complex explicit descriptions and even simple implicit ones. In contrast, EmoteGPT produces semantically faithful and expressive results across diverse inputs. Please zoom in for more details. Portrait3D [46]. As these met… view at source ↗
Figure 4
Figure 4. Figure 4: Diverse applications of EmoteGPT. Combining EmoteGPT with GaussianA￾vatar, HeadStudio and DiffusionRig enables downstream tasks such as personalized expressive 3D avatar generation and 2D face image synthesis. – Expressive 3D-aware personalized image generation. By combining EmoteGPT with DiffusionRig [48], our method can replace the expression to synthesize identity-preserving, expression-aware face image… view at source ↗
Figure 5
Figure 5. Figure 5: Perceptual study comparing EmoteGPT and Flux.1[dev]. Participants pre￾ferred EmoteGPT in both explicit and implicit input settings. overall, EmoteGPT was selected in 77% of cases, compared with 16% for Flux [PITH_FULL_IMAGE:figures/full_fig_p023_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Ablation on the training data. Visual comparison of EmoteGPT trained with different training data on text-to-3D expression synthesis. The inclusion of facial images and VQA data improves expression synthesis, demonstrating how multi-modal inputs enhance the performance for both explicit and implicit-based expressions. G Setting of perceptual study We provide the screenshot of our perceptual study in [PITH… view at source ↗
Figure 7
Figure 7. Figure 7: Examples of paired data in Txt2Emote [PITH_FULL_IMAGE:figures/full_fig_p027_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Screenshot of our perceptual study [PITH_FULL_IMAGE:figures/full_fig_p028_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Ablation on the training data. Visual comparison of EmoteGPT trained with different training data on text-to-3D expression synthesis. The inclusion of facial images and VQA data improves expression synthesis, demonstrating how multi-modal inputs enhance the performance for both explicit and implicit-based expressions. Prompt: This person enjoys a pleasant conversation at a social event. Ours GT Ours GT Pro… view at source ↗
Figure 10
Figure 10. Figure 10: Additional visual comparisons with zoomed-in details. The examples cover diverse prompt-conditioned expressions, including positive and negative affective states [PITH_FULL_IMAGE:figures/full_fig_p029_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Prompts for querying GPT4o to generate explicit facial expression descriptions on face images. You are an expert in analyzing facial expressions, posture, and contextual clues to infer human actions and events. You will be provided with an image of a face, and your task is to predict the most likely action or event they are involved in. I will provide an image of a person's face. Based on this image, infe… view at source ↗
Figure 12
Figure 12. Figure 12: Prompts for querying GPT4o to generate implicit facial expression descrip￾tions on face images [PITH_FULL_IMAGE:figures/full_fig_p030_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Qualitative comparison of outputs from EmoteGPT and other relevant meth￾ods. For EmoteGPT, we additionally show results after exporting the generated expres￾sions to GaussianAvatar (GA), HeadStudio (HS), and DiffusionRig (DR). Compared with existing methods, EmoteGPT produces more semantically faithful and expres￾sive results across diverse inputs. It accurately captures subtle composite expressions speci… view at source ↗
Figure 14
Figure 14. Figure 14: Additional qualitative results generated by EmoteGPT from implicit textual descriptions. EmoteGPT effectively complements existing approaches by synthesizing vivid and realistic facial expressions across a range of contexts [PITH_FULL_IMAGE:figures/full_fig_p032_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

69 extracted references · 10 linked inside Pith

  1. [1]

    In: Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Proceedings

    Aneja, S., Thies, J., Dai, A., Niessner, M.: Clipface: Text-guided editing of tex- tured 3d morphable models. In: Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Proceedings. p. 1–11. SIG- GRAPH ’23, ACM (Jul 2023).https://doi.org/10.1145/3588432.3591566, http://dx.doi.org/10.1145/3588432.3591566

    doi:10.1145/3588432.3591566 2023

  2. [2]

    In: 26th An- nual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH 1999)

    Blanz, V., Vetter, T.: A morphable model for the synthesis of 3d faces. In: 26th An- nual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH 1999). pp. 187–194. ACM Press (1999)

    1999

  3. [3]

    In: International Conference on Computer Vision (2017)

    Bulat, A., Tzimiropoulos, G.: How far are we from solving the 2d & 3d face align- ment problem? (and a dataset of 230,000 3d facial landmarks). In: International Conference on Computer Vision (2017)

    2017

  4. [4]

    Comas-Massagué, A., Qiu, D., Chai, M., Bühler, M., Raj, A., Gao, R., Xu, Q., Matthews, M., Gotardo, P., Camps, O., Orts-Escolano, S., Beeler, T.: Magicmirror: Fast and high-quality avatar generation with a constrained search space (2024), https://arxiv.org/abs/2404.01296

    Pith/arXiv arXiv 2024

  5. [5]

    Daněček, R., Black, M.J., Bolkart, T.: Emoca: Emotion driven monocular face captureandanimation.In:ProceedingsoftheIEEE/CVFConferenceonComputer Vision and Pattern Recognition. pp. 20311–20322 (2022)

    2022

  6. [6]

    ACM (Dec 2023)

    Daněček, R., Chhatre, K., Tripathi, S., Wen, Y., Black, M., Bolkart, T.: Emotional speech-driven animation with content-emotion disentanglement. ACM (Dec 2023). https://doi.org/10.1145/3610548.3618183,https://emote.is.tue.mpg.de/ index.html

    doi:10.1145/3610548.3618183 2023

  7. [7]

    ACM Transactions on Graphics (ToG)39(5), 1–38 (2020)

    Egger,B.,Smith,W.A.,Tewari,A.,Wuhrer,S.,Zollhoefer,M.,Beeler,T.,Bernard, F., Bolkart, T., Kortylewski, A., Romdhani, S., et al.: 3d morphable face mod- els—past, present, and future. ACM Transactions on Graphics (ToG)39(5), 1–38 (2020)

    2020

  8. [8]

    In: 2021 16th IEEE International Con- ference on Automatic Face and Gesture Recognition (FG 2021)

    Egger, B., Sutherland, S., Medin, S.C., Tenenbaum, J.: Identity-Expression Am- biguity in 3D Morphable Face Models . In: 2021 16th IEEE International Con- ference on Automatic Face and Gesture Recognition (FG 2021). pp. 1–7. IEEE Computer Society, Los Alamitos, CA, USA (Dec 2021).https : / / doi . org / 10.1109/FG52635.2021.9667002,https://doi.ieeecompute...

    doi:10.1109/fg52635.2021.9667002 2021

  9. [9]

    In: CVPR (2024)

    Feng, Y., Lin, J., Dwivedi, S.K., Sun, Y., Patel, P., Black, M.J.: Chatpose: Chatting about 3d human pose. In: CVPR (2024)

    2024

  10. [10]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K.V., Joulin, A., Misra, I.: Imagebind: One embedding space to bind them all. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15180– 15190 (2023)

    2023

  11. [11]

    CoRRabs/1606.08415(2016),http://arxiv.org/ abs/1606.08415

    Hendrycks, D., Gimpel, K.: Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRRabs/1606.08415(2016),http://arxiv.org/ abs/1606.08415

    Pith/arXiv arXiv 2016

  12. [12]

    In: International Con- ference on Learning Representations (2022),https://openreview.net/forum?id= nZeVKeeFYf9

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: LoRA: Low-rank adaptation of large language models. In: International Con- ference on Learning Representations (2022),https://openreview.net/forum?id= nZeVKeeFYf9

    2022

  13. [13]

    Huang, X., Shao, R., Zhang, Q., Zhang, H., Feng, Y., Liu, Y., Wang, Q.: Human- norm: Learning normal diffusion model for high-quality and realistic 3d human generation (2024) EmoteGPT 17

    2024

  14. [14]

    In: Proceedings of the Thirtieth Annual ACM Symposium on Theory of Computing

    Indyk, P., Motwani, R.: Approximate nearest neighbors: towards removing the curse of dimensionality. In: Proceedings of the Thirtieth Annual ACM Symposium on Theory of Computing. p. 604–613. STOC ’98, Association for Computing Ma- chinery, New York, NY, USA (1998).https://doi.org/10.1145/276698.276876, https://doi.org/10.1145/276698.276876

    doi:10.1145/276698.276876 1998

  15. [15]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2021)

    Jiang, Y., Huang, Z., Pan, X., Loy, C.C., Liu, Z.: Talk-to-edit: Fine-grained facial editing via dialog. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2021)

    2021

  16. [16]

    IEEE Transactions on Pattern Analysis & Machine Intelligence43(12), 4217–4228 (Dec 2021).https://doi.org/10.1109/TPAMI

    Karras, T., Laine, S., Aila, T.: A Style-Based Generator Architecture for Gener- ative Adversarial Networks . IEEE Transactions on Pattern Analysis & Machine Intelligence43(12), 4217–4228 (Dec 2021).https://doi.org/10.1109/TPAMI. 2020.2970919,https://doi.ieeecomputersociety.org/10.1109/TPAMI.2020. 2970919

    doi:10.1109/tpami 2021

  17. [17]

    Labs, B.F.: Flux.https://github.com/black-forest-labs/flux(2024)

    2024

  18. [18]

    In: CVPR (2024)

    Lai, X., Tian, Z., Chen, Y., Li, Y., Yuan, Y., Liu, S., Jia, J.: Lisa: Reasoning segmentation via large language model. In: CVPR (2024)

    2024

  19. [19]

    ACM Transactions on Graphics, (Proc

    Li, T., Bolkart, T., Black, M.J., Li, H., Romero, J.: Learning a model of facial shape and expression from 4D scans. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia)36(6), 194:1–194:17 (2017),https://doi.org/10.1145/3130800.3130813

    doi:10.1145/3130800.3130813 2017

  20. [20]

    Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning (2023)

    2023

  21. [21]

    Advances in neural information processing systems36(2024)

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. Advances in neural information processing systems36(2024)

    2024

  22. [22]

    In: Proceedings of International Conference on Computer Vision (ICCV) (December 2015)

    Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of International Conference on Computer Vision (ICCV) (December 2015)

    2015

  23. [23]

    In: International Conferenceon LearningRepresentations(2019),https://openreview.net/forum? id=Bkg6RiCqY7

    Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conferenceon LearningRepresentations(2019),https://openreview.net/forum? id=Bkg6RiCqY7

    2019

  24. [24]

    Ma, Y., Wang, S., Ding, Y., Ma, B., Lv, T., Fan, C., Hu, Z., Deng, Z., Yu, X.: Talk- clip: Talking head generation with text-guided expressive speaking styles (2024), https://arxiv.org/abs/2304.00334

    Pith/arXiv arXiv 2024

  25. [25]

    In: SIGGRAPH Asia 2023 Technical Communications

    Manu, P., Srivastava, A., Sharma, A.: Clip-head: Text-guided generation of tex- tured neural parametric 3d head models. In: SIGGRAPH Asia 2023 Technical Communications. SA ’23, Association for Computing Machinery, New York, NY, USA (2023).https://doi.org/10.1145/3610543.3626169,https://doi.org/ 10.1145/3610543.3626169

    doi:10.1145/3610543.3626169 2023

  26. [26]

    ACM Trans

    Mendiratta, M., Pan, X., Elgharib, M., Teotia, K., R, M.B., Tewari, A., Golyanik, V., Kortylewski, A., Theobalt, C.: Avatarstudio: Text-driven editing of 3d dynamic human head avatars. ACM Trans. Graph.42(6) (dec 2023).https://doi.org/10. 1145/3618368,https://doi.org/10.1145/3618368

    doi:10.1145/3618368 2023

  27. [27]

    IEEE Trans

    Mollahosseini, A., Hasani, B., Mahoor, M.H.: Affectnet: A database for facial ex- pression, valence, and arousal computing in the wild. IEEE Trans. Affect. Com- put.10(1), 18–31 (Jan 2019).https://doi.org/10.1109/TAFFC.2017.2740923, https://doi.org/10.1109/TAFFC.2017.2740923

    doi:10.1109/taffc.2017.2740923 2019

  28. [28]

    Wang et al

    OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shapiro, G., Berner, C., Bogdonoff, L., Boiko, O., Boyd, M., Brakman, A.L., Brockman, ...

    2024

  29. [29]

    In: The Twelfth International Conference on Learning Representa- tions (2024),https://openreview.net/forum?id=di52zR8xgf

    Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., Rombach, R.: SDXL: Improving latent diffusion models for high-resolution im- age synthesis. In: The Twelfth International Conference on Learning Representa- tions (2024),https://openreview.net/forum?id=di52zR8xgf

    2024

  30. [30]

    In: The Eleventh International Conference on Learning Representations (2023),https://openreview.net/forum?id=FjNys5c7VyY

    Poole, B., Jain, A., Barron, J.T., Mildenhall, B.: Dreamfusion: Text-to-3d using 2d diffusion. In: The Eleventh International Conference on Learning Representations (2023),https://openreview.net/forum?id=FjNys5c7VyY

    2023

  31. [31]

    org/abs/2410.16395

    Prinzler, M., Zakharov, E., Sklyarova, V., Kabadayi, B., Thies, J.: Joker: Condi- tional 3d head synthesis with extreme facial expressions (2024),https://arxiv. org/abs/2410.16395

    Pith/arXiv arXiv 2024

  32. [32]

    Qian, S., Kirschstein, T., Schoneveld, L., Davoli, D., Giebenhain, S., Nießner, M.: Gaussianavatars:Photorealisticheadavatarswithrigged3dgaussians.In:Proceed- EmoteGPT 19 ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20299–20309 (2024)

    2024

  33. [33]

    In: Proceedings of the International Con- ference for High Performance Computing, Networking, Storage and Analysis

    Rajbhandari, S., Rasley, J., Ruwase, O., He, Y.: Zero: memory optimizations to- ward training trillion parameter models. In: Proceedings of the International Con- ference for High Performance Computing, Networking, Storage and Analysis. SC ’20, IEEE Press (2020)

    2020

  34. [34]

    In: Proceed- ings of the 26th ACM SIGKDD International Conference on Knowledge Discov- ery & Data Mining

    Rasley, J., Rajbhandari, S., Ruwase, O., He, Y.: Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In: Proceed- ings of the 26th ACM SIGKDD International Conference on Knowledge Discov- ery & Data Mining. p. 3505–3506. KDD ’20, Association for Computing Machin- ery, New York, NY, USA (2020).https://doi....

    doi:10.1145/3394486.3406703 2020

  35. [35]

    Retsinas, G., Filntisis, P.P., Danecek, R., Abrevaya, V.F., Roussos, A., Bolkart, T., Maragos, P.: 3d facial expressions through analysis-by-neural-synthesis (2024), https://arxiv.org/abs/2404.04104

    Pith/arXiv arXiv 2024

  36. [36]

    arXiv preprint arXiv:2305.16355 (2023)

    Su, Y., Lan, T., Li, H., Xu, J., Wang, Y., Cai, D.: Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305.16355 (2023)

    Pith/arXiv arXiv 2023

  37. [37]

    In: Proceedings of the 29th ACM International Conference on Multimedia

    Sun, J., Li, Q., Wang, W., Zhao, J., Sun, Z.: Multi-caption text-to-face synthesis: Dataset and algorithm. In: Proceedings of the 29th ACM International Conference on Multimedia. p. 2290–2298. MM ’21, Association for Computing Machinery, New York, NY, USA (2021).https://doi.org/10.1145/3474085.3475391,https: //doi.org/10.1145/3474085.3475391

    doi:10.1145/3474085.3475391 2021

  38. [38]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Tan, S., Ji, B., Pan, Y.: Style2talker: High-resolution talking head generation with emotion style and art style. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 38, pp. 5079–5087 (2024)

    2024

  39. [39]

    Taubner, F., Zhang, R., Tuli, M., Bahmani, S., Lindell, D.B.: MVP4D: Multi-view portrait video diffusion for animatable 4D avatars (2025),https://arxiv.org/ abs/2510.12785

    arXiv 2025

  40. [40]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Taubner, F., Zhang, R., Tuli, M., Lindell, D.B.: CAP4D: Creating animatable 4D portrait avatars with morphable multi-view diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5318–5330 (June 2025)

    2025

  41. [41]

    arXiv preprint arXiv:2511.22699 (2025)

    Team, Z.I.: Z-image: An efficient image generation foundation model with single- stream diffusion transformer. arXiv preprint arXiv:2511.22699 (2025)

    Pith/arXiv arXiv 2025

  42. [42]

    Nature Machine Intelligence (2021),https://www.nature.com/articles/s42256- 020- 00280-0

    Toisoul, A., Kossaifi, J., Bulat, A., Tzimiropoulos, G., Pantic, M.: Estimation of continuous valence and arousal levels from faces in naturalistic conditions. Nature Machine Intelligence (2021),https://www.nature.com/articles/s42256- 020- 00280-0

    2021

  43. [43]

    arXiv preprint arXiv:2311.00618 (2023)

    Wei, C., Liu, C., Qiao, S., Zhang, Z., Yuille, A., Yu, J.: De-diffusion makes text a strong cross-modal interface. arXiv preprint arXiv:2311.00618 (2023)

    Pith/arXiv arXiv 2023

  44. [44]

    In: 2023 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR)

    Wu, M., Zhu, H., Huang, L., Zhuang, Y., Lu, Y., Cao, X.: High-fidelity 3d face generation from natural language descriptions. In: 2023 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR). pp. 4521–4530. IEEE Computer Society, Los Alamitos, CA, USA (jun 2023).https://doi.org/ 10.1109/CVPR52729.2023.00439,https://doi.ieeecomputersociety....

    doi:10.1109/cvpr52729.2023.00439 2023

  45. [45]

    arXiv preprint arXiv:2309.05519 (2023)

    Wu, S., Fei, H., Qu, L., Ji, W., Chua, T.S.: Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519 (2023)

    Pith/arXiv arXiv 2023

  46. [46]

    Wang et al

    Wu, Y., Xu, H., Tang, X., Chen, X., Tang, S., Zhang, Z., Li, C., Jin, X.: Portrait3d: Text-guided high-quality 3d portrait generation using pyramid representation and 20 H. Wang et al. gans prior. ACM Trans. Graph.43(4) (Jul 2024).https://doi.org/10.1145/ 3658162,https://doi.org/10.1145/3658162

    doi:10.1145/3658162 2024

  47. [47]

    In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2021)

    Xia, W., Yang, Y., Xue, J.H., Wu, B.: Tedigan: Text-guided diverse face image gen- eration and manipulation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2021)

    2021

  48. [48]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023)

    Zheng, D., Cecilia, Z., Zhihao, X., Lars, J., Zhuowen, T., Xiuming, Z.: Diffusionrig: Learning personalized priors for facial appearance editing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023)

    2023

  49. [49]

    In: Proceedings of the 37th International Conference on Neural Information Processing Systems

    Zheng, L., Chiang, W.L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li,D.,Xing,E.P.,Zhang,H.,Gonzalez,J.E.,Stoica,I.:Judgingllm-as-a-judgewith mt-bench and chatbot arena. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. NIPS ’23, Curran Associates Inc., Red Hook, NY, USA (2023)

    2023

  50. [50]

    Zhong, Y., Wei, H., Yang, P., Wang, Z.: Expclip: bridging text and facial expres- sions via semantic alignment. In: Proceedings of the Thirty-Eighth AAAI Confer- ence on Artificial Intelligence and Thirty-Sixth Conference on Innovative Appli- cations of Artificial Intelligence and Fourteenth Symposium on Educational Ad- vances in Artificial Intelligence. ...

    doi:10.1609/aaai.v38i7.28594 2024

  51. [51]

    In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019)

    Zhou, Y., Barnes, C., Jingwan, L., Jimei, Y., Hao, L.: On the continuity of rotation representations in neural networks. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019)

    2019

  52. [52]

    In: 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021)

    Zhou, Y., Shimada, N.: Generative adversarial network for text-to-face synthesis and manipulation with pretrained bert model. In: 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021). pp. 01–08 (2021).https://doi.org/10.1109/FG52635.2021.9666791

    doi:10.1109/fg52635.2021.9666791 2021

  53. [53]

    Zhou, Z., Ma, F., Fan, H., Yang, Z., Yang, Y.: Headstudio: Text to animatable head avatars with 3d gaussian splatting (2024)

    2024

  54. [54]

    arXiv preprint arXiv:2304.10592 (2023)

    Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 (2023)

    Pith/arXiv arXiv 2023

  55. [55]

    This person sees their favorite cartoon character come to life in front of them

    Zielonka, W., Bolkart, T., Thies, J.: Towards metrical reconstruction of human faces. In: ECCV (2022) EmoteGPT 21 Supplementary Material A Details of data generation In this section, we describe the data generation procedure. Both the training and evaluation sets are produced using the automatic pipeline described be- low, while the test set is further ve...

    2022

  56. [56]

    Describe the facial expression precisely and comprehensively based on visible facial features

  57. [57]

    Focus only on the expression and avoid mentioning unrelated attributes such as head position, glasses, hairstyle, or back-ground

    Keep the description concise yet rich in detail. Focus only on the expression and avoid mentioning unrelated attributes such as head position, glasses, hairstyle, or back-ground

  58. [58]

    Use clear, natural language to convey the emotion or state of mind suggested by the facial features

  59. [59]

    Example Output Formats:

    If the expression conveys a mix of emotions, describe the balance and interplay between them. Example Output Formats:

  60. [60]

    The eyes appear bright and slightly narrowed, which is typical in expressions of happiness or joy

    The person has a facial expression characterized by an open, wide smile, showing teeth. The eyes appear bright and slightly narrowed, which is typical in expressions of happiness or joy

  61. [61]

    Their mouth is slightly open, with lips curved in a small smile

    The person has wide-open eyes with slightly raised eye-brows. Their mouth is slightly open, with lips curved in a small smile. This expression may suggest surprise or eager-ness

  62. [62]

    The mouth is relaxed without any discernible smile or frown, and the eyes appear calm and straightforward

    The person’s facial expression is neutral. The mouth is relaxed without any discernible smile or frown, and the eyes appear calm and straightforward

  63. [63]

    The person’s facial expression shows furrowed brows and a scrunched nose, with lips pressed together, possibly indicating disgust or displeasure

  64. [64]

    Now, generate a detailed facial expression description for the given image

    The person’s facial expression displays slight furrowing of the brow and slight parting of the lips, which may suggest concentration or concern. Now, generate a detailed facial expression description for the given image. Fig. 11:PromptsforqueryingGPT4otogenerateexplicitfacialexpressiondescriptions on face images. You are an expert in analyzing facial expr...

  65. [65]

    Be one specific activity or action (no multiple guesses)

  66. [66]

    Start with 'This person... ’

  67. [67]

    Avoid describing emotions—focus only on the action or situation

  68. [68]

    Avoid overly simple actions like 'laugh' or 'cry. ’

  69. [69]

    The person has their mouth slightly open with the top teeth visible, and the eyebrows are raised. The eyes appear somewhat wide, possibly indicating confusion

    Be concise and natural, similar to these examples: • This person catches someone taking credit for their hard work in front of others. • This person receives news of a canceled long-anticipated event. • This person bites into a piece of food that tastes completely spoiled. • This person realizes his fellow misplaced an important document moments before a ...