RECORD
GAN-Avatar: Controllable Personalized GAN-based Human Head Avatar
Computer Vision and Pattern Recognition · Submitted 22 Nov 2023
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim In a series of experiments, we compare our proposed technique to state-of-the-art monocular methods and show superior quality while not requiring expression tracking of the training data.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Digital humans and, especially, 3D facial avatars have raised a lot of attention in the past years, as they are the backbone of several applications like immersive telepresence in AR or VR. Despite the progress, facial avatars reconstructed from commodity hardware are incomplete and miss out on parts of the side and back of the head, severely limiting the usability of the avatar. This limitation in prior work stems from their requirement of face tracking, which fails for profile and back views. To address this issue, we propose to learn person-specific animatable avatars from images without assuming to have access to precise facial expression tracking. At the core of our method, we leverage a 3D-aware generative model that is trained to reproduce the distribution of facial expressions from the training data. To train this appearance model, we only assume to have a collection of 2D images with the corresponding camera parameters. For controlling the model, we learn a mapping from 3DMM facial expression parameters to the latent space of the generative model. This mapping can be learned by sampling the latent space of the appearance model and reconstructing the facial parameters from a normalized frontal view, where facial expression estimation performs well. With this scheme, we decouple 3D appearance reconstruction and animation control to achieve high fidelity in image synthesis. In a series of experiments, we compare our proposed technique to state-of-the-art monocular methods and show superior quality while not requiring expression tracking of the training data.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Digital humans and, especially, 3D facial avatars have raised a lot of attention in the past years, as they are the backbone of several applications like immersive telepresence in AR or VR. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| s1 | Despite the progress, facial avatars reconstructed from commodity hardware are incomplete and miss out on parts of the side and back of the head, severely limiting the usability of the avatar. | gap 0.65 | 0.08 | 0.04 | asserted 1.00 |
| s2 | This limitation in prior work stems from their requirement of face tracking, which fails for profile and back views. | background 0.49 | 0.05 | 0.03 | asserted 1.00 |
| s3 | To address this issue, we propose to learn person-specific animatable avatars from images without assuming to have access to precise facial expression tracking. | contribution 0.99 | 0.67 | 0.13 | asserted 0.92 |
| s4 | At the core of our method, we leverage a 3D-aware generative model that is trained to reproduce the distribution of facial expressions from the training data. | method 0.95 | 0.82 | 0.07 | asserted 1.00 |
| s5 | To train this appearance model, we only assume to have a collection of 2D images with the corresponding camera parameters. | method 0.99 | 0.63 | 0.04 | asserted 0.99 |
| s6 | For controlling the model, we learn a mapping from 3DMM facial expression parameters to the latent space of the generative model. | method 0.98 | 0.90 | 0.05 | asserted 0.99 |
| s7 | This mapping can be learned by sampling the latent space of the appearance model and reconstructing the facial parameters from a normalized frontal view, where facial expression estimation performs well. | method 1.00 | 0.80 | 0.05 | asserted 0.74 |
| s8 | With this scheme, we decouple 3D appearance reconstruction and animation control to achieve high fidelity in image synthesis. | method 0.62 | 0.85 | 0.16 | asserted 0.97 |
| s9 | In a series of experiments, we compare our proposed technique to state-of-the-art monocular methods and show superior quality while not requiring expression tracking of the training data. | result 0.99 | 0.96 | 0.76 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Digital humans and, especially, 3D facial avatars have raised a lot of attention in the past years, as they are the backbone of several applications like immersive telepresence in AR or VR. Despite the progress, facial avatars reconstructed from commodity hardware are incomplete and miss out on parts of the side and back of the head, severely limiting the usability of the avatar. This limitation in prior work stems from their requirement of face tracking, which fails for profile and back views. To address this issue, we propose to learn person-specific animatable avatars from images without assuming to have access to precise facial expression tracking. At the core of our method, we leverage a 3D-aware generative model that is trained to reproduce the distribution of facial expressions from the training data. To train this appearance model, we only assume to have a collection of 2D images with the corresponding camera parameters. For controlling the model, we learn a mapping from 3DMM facial expression parameters to the latent space of the generative model. This mapping can be learned by sampling the latent space of the appearance model and reconstructing the facial parameters from a normalized frontal view, where facial expression estimation performs well. With this scheme, we decouple 3D appearance reconstruction and animation control to achieve high fidelity in image synthesis. In a series of experiments, we compare our proposed technique to state-of-the-art monocular methods and show superior quality while not requiring expression tracking of the training data.
Human review
How the literature uses it
1 arXiv paper cite this in 1 sentence
Papers citing it
Cites it as background 1 paper
-
Recently, the 3D Gaussian Splatting (3DGS) [ref] based face avatars have shown success in rendering photo-realistic faces with controllability and real-time efficiency [ref] .
StyleMorpheus: A Style-Based 3D-Aware Morphable Face Model · Introduction · 0.81
What it cites
Cites as background 2 papers
-
Recent methods to recover an animatable digital double of a person either use monocular [ref] or multi-view inputs [ref] .
Learning Disentangled Avatars with Hybrid 3D Representations · Introduction · 1.00
-
Recent methods extend EG3D to also incorporate expression control [ref] .
3DFaceShop: Explicitly Controllable 3D-Aware Portrait Generation · Related Work · 0.86
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.