Introduction

We propose a 3D face generative model with local weights to increase the model's variations and expressiveness. The proposed model allows partial manipulation of the face while still learning the whole face mesh. For this purpose, we address an effective way to extract local facial features from the entire data and explore a way to manipulate them during a holistic generation. First, we factorize the latent space of the whole face to the subspace indicating different parts of the face. In addition, local weights generated by non-negative matrix factorization are applied to the factorized latent space so that the decomposed part space is semantically meaningful. We experiment with our model and observe that effective facial part manipulation is possible, and that the model's expressiveness is improved.

Locally Weighted Autoencoder

Our model is consisting of an encoder, projection, and a decoder. The encoder and decoder learn how to compress and decompress the data, respectively. In between them, the projection part factorizes the latent space into the subspace and applies local weights to make the subspace semantically meaningful.

\

Pre-Computed Local Weights from NMF

To make a part-based representation of the whole data, we employ the non-negative matrix factorization(NMF), which is a robust feature factorization method to represent data as part-based ones. We expect that local weights would make the part encodings more semantically meaningful. Given a feature matrix, V, W is a basis matrix that contains basis elements of V. We call the matrix W local weights.

Projection Matrix Layer & Applying Local Weights

When we factorize the whole encoding, we can generate part encodings corresponding to the shape structure of the part. Thereby, we disentangle different semantic part encodings from the encoding of the whole shapes. We then perform part-level shape manipulation.

We apply the pre-computed local weights to the part encodings that are factorized by the projection matrices. Thanks to this operation, each factorized latent vector has a localized weight, and the encodings lie on a part-based subspace.

Results

Part Interpolation

In this experiment, we tested the part manipulation results by applying interpolation between source and target as shown. We interpolated the source's part encodings to the target's corresponding part encodings obtained by factorized latent vectors.

Result shows that as the respective part of the face influence changes, the other parts of the face are not affected. Plus, we expected that each row’s changing part matches each local weight in the same row. As a result, we observed that each variation area corresponds to each local weight.

Diversity Visualization

To demonstrate the variety of data, we measured the diversity of generated data from our model. The result was visualized by projecting selected data onto a 2D plane using PCA and t-SNE.In our visualizations, our synthesis samples (green) cover wider areas in the encoding space. As a result, our proposed method shows a prominent performance to extend the model’s representation ability.

Generation Results with the same texture of face

Contacts