A morphable model for the synthesis of 3D faces
Summary (3 min read)
1 Introduction
- Computer aided modeling of human faces still requires a great deal of expertise and manual control to avoid unrealistic, non-face-like results.
- A limited number of labeled feature points marked in one face, e.g., the tip of the nose, the eye corner and less prominent points on the cheek, must be located precisely in another face.
- For this, human knowledge is even more critical.
1.2 Organization of the paper
- Along with a 3D reconstruction, the algorithm can compute correspondence, based on the morphable model.
- In Section 5, the authors introduce an iterative method for building a morphable model automatically from a raw data set of 3D face scans when no correspondences between the exemplar faces are available.
2 Database
- The laser scans provide head structure data in a cylindrical representation, with radiir(h; ) of surface points sampled at 512 equally-spaced angles , and at 512 equally spaced vertical stepsh.
- Additionally, the RGB-color values R(h; ), G(h; ),andB(h; ), were recorded in the same spatial resolution and were stored in a texture map with 8 bit per channel.
- All faces were without makeup, accessories, and facial hair.
- The subjects were scanned wearing bathing caps, that were removed digitally.
3 Morphable 3D Face Model
- Morphing between faces requires full correspondence between all of the faces.
- The algorithm for computing correspondence will be described in Section 5.
- A morphable face model was then constructed using a data set ofm exemplar faces, each represented by its shape-vectorSi and texturevectorTi.
- The deviation of a prototype from the average is added (+) or subtracted (-) from the average.
3.1 Facial attributes
- Shape and texture coefficients i and i in their morphable face model do not correspond to the facial attributes used in human language.
- While some facial attributes can easily be related to biophysical measurements [13, 10], such as the width of the mouth, others such as facial femininity or being more or less bony can hardly be described by numbers.
- The authors describe a method for mapping facial attributes, defined by a hand-labeled set of example faces, to the parameter space of their morphable model.
- At each position in face space (that is for any possible face), the authors define shape and texture vectors that, when added to or subtracted from a face, will manipulate a specific attribute while keeping all other attributes as constant as possible.
- A different kind of facial attribute is its “distinctiveness”, which is commonly manipulated in caricatures.
4 Matching a morphable model to images
- A crucial element of their framework is an algorithm for automatically matching the morphable face model to one or more images.
- For high resolution 3D meshes, variations inImodel across each trianglek 2 f1; :::; ntg are small, soEI may be approximated by EI ntX k=1 ak kIinput( px;k; py;k) Imodel;kk 2; whereak is the image area covered by trianglek.
- It then determinesak, and detects hidden surfaces and cast shadows in a two-pass z-buffer technique.
- With parameters j fixed, coefficients j and j are optimized independently for each segment.
5 Building a morphable model
- The authors describe how to build the morphable model from a set of unregistered 3D prototypes, and to add a new face to the existing morphable model, increasing its dimensionality.
- The key problem is to compute a dense point-to-point correspondence between the vertices of the faces.
- Since the method described in Section 4.1 finds the best match of a given face only within the range of the morphable model, it cannot add new dimensions to the vector space of faces.
- To determine residual deviations between a novel face and the best match within the model, as well as to set unregistered prototypes in correspondence, the authors use an optic flow al- gorithm that computes correspondence between two faces without the need of a morphable model [35].
5.1 3D Correspondence using Optic Flow
- Initially designed to find corresponding points in grey-level images I(x; y), a gradient-based optic flow algorithm [2] is modified to establish correspondence between a pair of 3D scansI(h; ) (Equation 8), taking into account color and radius values simultaneously [35].
- The authors therefore perform a smooth interpolation based on simulated relaxation of a system of flow vectors that are coupled with their neighbors.
- The quadratic coupling potential is equal for all flow vectors.
- On high-contrast areas, components of flow vectors orthogonal to edges are bound to the result of the previous optic flow computation.
- Given a definition of shape and texture vectorsSref andTref for the reference face,S andT for each face in the database can be obtained by means of the point-topoint correspondence provided by( h(h; ); (h; )).
5.2 Bootstrapping the model
- The texture estimate can be improved by additional texture extraction (4).
- Starting from an arbitrary face as the temporary reference, preliminary correspondence between all other faces and this reference is computed using the optic flow algorithm.
- Their average serves as a new reference face.
- The first morphable model is then formed by the most significant components as provided by a standard PCA decomposition.
- The current morphable model is now matched to each of the 3D faces according to the method described in Section 4.1.
6 Results
- The authors built a morphable face model by automatically establishing correspondence between all of their 200 exemplar faces.
- The authors interactive face modeling system enables human users to create new characters and to modify facial attributes by varying the model coefficients.
- The whole matching procedure was performed in105 iterations.
- To human observers who also know only the input image, the results obtained with their method look correct.
- The authors therefore apply a different method for transferring all details of the painting to novel views.
8 Acknowledgment
- The authors thank Michael Langer, Alice O’Toole, Tomaso Poggio, Heinrich Bülthoff and Wolfgang Straßer for reading the manuscript and for many insightful and constructive comments.
- In particular, the authors thank Marney Smyth and Alice O’Toole for their perseverance in helping us to obtain the following.
Did you find this useful? Give us your feedback
Citations
6,384 citations
2,187 citations
2,138 citations
Cites methods from "A morphable model for the synthesis..."
...Parameterized Appearance Models (PAMs), such as Active Appearance Models [11, 14, 2], Morphable Mod- els [6, 19], Eigentracking [5], and template tracking [22, 30] build an object appearance and shape representation by computing Principal Component Analysis (PCA) on a set of manually labeled data....
[...]
...Parameterized Appearance Models (PAMs), such as Active Appearance Models [11, 14, 2], Morphable Models [6, 19], Eigentracking [5], and template tracking [22, 30] build an object appearance and shape representation by computing Principal Component Analysis (PCA) on a set of manually labeled data....
[...]
1,775 citations
Cites background or methods from "A morphable model for the synthesis..."
...This class includes Active Appearance Models (AAMs) (Cootes et al., 1998a, 2001; Edwards, 1999; Edwards et al., 1998; Lanitis et al., 1997), Shape AAMs (Cootes et al., 1998b; Cootes and Kittipanyangam, 2002; Cootes et al., 2001), Direct Appearance Models (Hou et al., 2001), Active Blobs (Sclaroff and Isidoro, 2003), and Morphable Models (Blanz and Vetter, 1999; Jones and Poggio, 1998; Vetter and Poggio, 1997) as well as possibly others....
[...]
...Keywords: Active Appearance Models, AAMs, Active Blobs, Morphable Models, fitting, efficiency, GaussNewton gradient descent, inverse compositional image alignment...
[...]
...…Shape AAMs (Cootes et al., 1998b; Cootes and Kittipanyangam, 2002; Cootes et al., 2001), Direct Appearance Models (Hou et al., 2001), Active Blobs (Sclaroff and Isidoro, 2003), and Morphable Models (Blanz and Vetter, 1999; Jones and Poggio, 1998; Vetter and Poggio, 1997) as well as possibly others....
[...]
...For example, Levenberg-Marquardt was used in Sclaroff and Isidoro (1998) and a stochastic gradient descent algorithm was used in Blanz and Vetter (1999) and Jones and Poggio (1998)....
[...]
...Another thing that makes empirical evaluation hard is the wide variety of AAM fitting algorithms (Blanz and Vetter, 1999; Cootes et al., 1998a, 2001; Jones and Poggio, 1998; Sclaroff and Isidoro, 1998) and the lack of a standard test set....
[...]
1,269 citations
Cites methods from "A morphable model for the synthesis..."
...To test attentional modulation of object recognition beyond paper clips, we also tested stimuli consisting of synthetic faces rendered from 3D models, which were obtained by scanning the faces of human subjects ( Vetter&Blanz,1999 )....
[...]
References
17,446 citations
6,200 citations
3,905 citations
2,089 citations
1,069 citations
Related Papers (5)
Frequently Asked Questions (20)
Q2. What are the future works mentioned in the paper "A morphable model for the synthesis of 3d faces" ?
The authors plan to speed up their matching algorithm by implementing a simplified Newton-method for minimizing the cost function ( Equation 5 ). While the current database is sufficient to model Caucasian faces of middle age, the authors would like to extend it to children, to elderly people as well as to other races. The authors also plan to incorporate additional 3D face examples representing the time course of facial expressions and visemes, the face variations during speech. Automated reconstruction of hair styles from images is one of the future challenges.
Q3. What is the method for calculating the residual deviations between two faces?
To determine residual deviations between a novel face and the best match within the model, as well as to set unregistered prototypes in correspondence, the authors use an optic flow algorithm that computes correspondence between two faces without the need of a morphable model [35].
Q4. What are the parameters that are fixed to the values of the user?
Other parameters, such as camera distance, light direction, and surface shininess, remain fixed to the values estimated by the user.
Q5. What is the way to reduce the computation time of the matching algorithm?
Instead of the time consuming computation of derivatives for each iteration step, a global mapping of the matching error into parameter space can be used [9].
Q6. What is the definition of a morphable face model?
The morphable face model is a multidimensional 3D morphing function that is based on the linear combination of a large number of 3D face scans.
Q7. Why does the optic flow algorithm fail on some of the more unusual faces in the database?
Because the optic flow algorithm does not incorporate any constraints on the set of solutions, it fails on some of the more unusualfaces in the database.
Q8. How do the authors derive the attributes of a face?
The authors also derive parametric descriptions of face attributes such as gender, distinctiveness, “hooked” noses or the weight of a person, by evaluating the distribution of exemplar faces for each attribute within their face space.
Q9. What is the problem of finding the set of parameters with maximum posterior probability?
In terms of Bayes decision theory, the problem is to find the set of parameters (~ ; ~ ; ~ ) with maximum posterior probability, given an image Iinput.
Q10. What is the purpose of the morphable 3D face model?
The morphable 3D face model is a consequent extension of the interpolation technique between face geometries, as introduced by Parke [26].
Q11. How do the authors speed up their matching algorithm?
The authors plan to speed up their matching algorithm by implementing a simplified Newton-method for minimizing the cost function (Equation 5).
Q12. How did the authors test the expressive power of their morphable model?
The authors tested the expressive power of their morphable model by automatically reconstructing 3D faces from photographs of arbitrary Caucasian faces of middle age that were not in the database.
Q13. What is the goal of an extended morphable face model?
The goal of such an extended morphable face model is to represent any face as a linear combination of a limited basis set of face prototypes.
Q14. How did the authors construct the morphable face model?
A morphable face model was then constructed using a data set of m exemplar faces, each represented by its shape-vector Si and texturevector Ti. Since the authors assume all faces in full correspondence (see Section 5), new shapes Smodel and new textures Tmodel can be expressed in barycentric coordinates as a linear combination of the shapes and textures of the m exemplar faces:
Q15. What is the way to match a 3D face model to an image?
Most techniques for ‘face cloning’, the reconstruction of a 3D face model from one or more images, still rely on manual assistance for matching a deformable 3D face model to the images [26, 1, 30].
Q16. What is the method for combining 3D shapes with other objects?
The face can be combined with other 3D graphic objects, such as glasses or hats, and then be rendered in front of the background, computing cast shadows or new illumination conditions (Fig. 7).
Q17. What is the main problem in face modeling?
Computer aided modeling of human faces still requires a great deal of expertise and manual control to avoid unrealistic, non-face-like results.
Q18. What is the way to replace the missing part of the head?
For animation, the missing part of the head can be automatically replaced by a standard hair style or a hat, or by hair that is modeled using interactive manual segmentation and adaptation to a 3D model [30, 28].
Q19. What are the limitations of face synthesis?
Most limitations of automated techniques for face synthesis, face animation or for general changes in the appearance of an individual face can be described either as the problem of finding corresponding feature locations in different faces or as the problem of separating realistic faces from faces that could never appear in the real world.
Q20. What is the way to reconstruct a face?
As demonstrated in Figures 6 and 7, the results can be used for automatic post-processing of a face within the original picture or movie sequence.