Using [ ]v to denote the vectorization operation, we can vectorize B and C in Eq. (7.3), and concatenate them, as v= [B]v . [C]v (7.4)

This v is the vectorized bilinear basis for one shape (i.e., one object) with dimension Iv 1, where Iv = 7Nl MN (Nl MN for B and 6Nl MN for C). Given the 3D shape of Ii objects with Ie different deformations, we can compute this vectorized bilinear basis v for every combination. For faces, using the 3DMM [18] approaches, these instances can be obtained by choosing different coef cients of the corresponding linear basis functions. With the application to faces in mind, we will sometimes use the words deformation and expression interchangably. i We use ve to represent the vectorized bilinear basis of identity i with expression e. Let us rearrange them into a training data tensor D of size Ii Ie Iv with the rst dimension for identity, the second dimension for expression (deformation), and the third dimension for the vectorized, analytically derived bilinear basis for each training sample. Applying the N-Mode SVD algorithm [14], the training data tensor can be decomposed as D = Y 1 Ui 2 Ue 3 Uv = Z 1 Ui 2 Ue , where Z = Y 3 Uv . (7.5)

Y is known as the core tensor of size Ni Ne Nv , and Ni and Ne are the number of bases we use for the identity and expression. With a slight abuse of terminology, we will call Z (which is decomposed only along the identity and expression dimension with size Ni Ne Iv ) the core tensor. Ui and Ue , with sizes of Ii Ni and Ie Ne , are the left matrices of the SVD of 1T T IT 1T 1 v1 . . . vIe v1 . . . v1i and D(2) = , ... ... (7.6) D(1) = T T T Ii Ii 1 T . . . v Ii v1 . . . vIe vIe Ie where the subscripts of tensor D indicate the tensor unfolding operation2 along the rst and second dimension. According to the N-mode SVD algorithm and Eq. (7.5), the core tensor Z can be expressed as

(7.7)

an Nth-order tensor A CI1 I2 ... IN . The matrix unfolding A(n) CIn (In+1 In+2 ...IN I1 I2 ...In 1 ) contains the element ai1 i2 ...iN at the position with row number in and column number equal to (in+1 1)In+2 In+3 . . . IN I1 I2 . . . In 1 + (in+2 1)In+3 In+4 . . . IN I1 I2 . . . In 1 + + (iN 1)I1 I2 . . . In 1 + (i1 1)I2 I3 . . . In 1 + + in 1 .

7

e T T vi T = Z 1 ci 2 ce ,

(7.8)

where ci and ce are the coef cient vectors encoding the identity and expression. As e vi are the vectorized, bilinear basis functions of the illumination and 3D motion, the core tensor Z is quadrilinear in illumination, motion, identity, and expression. As an example, this core tensor Z can describe all the face images of identity ci with expression ce and motion (T, ) under illumination l. Due to the small motion assumption in the derivation of the analytical model of motion and illumination in Section 7.2.1, the core tensor Z can only represent the image of the object whose pose is close to the pose p under which the training samples of v are computed. To emphasize that Z is a function of pose p, we denote it as Zp in the following derivation. Since v is obtained by concatenating [B]v and [C]v , Zp also contains two parts, ZB with size (Ni Ne Nl MN) and ZC with size p p (Ni Ne 6Nl MN). The rst part encodes the variation of the image due to changes of identity, deformation, and illumination at the pose p, and the second part encodes the variation due to motion around p that is, the tangent plane of the manifold along the motion direction. Rearranging the two subtensors according to the illumination and motion basis into sizes of Nl 1 Ni Ne MN and Nl 6 Ni Ne MN (this step is needed to undo the vectorization operation of Eq. (7.4)), we can represent the quadrilinear basis of illumination, 3D motion, identity, and deformation along the rst, second, third, and fourth dimensions respectively. The image with identity ci and expression ce after motion (T, ) around pose p under illumination l can be obtained by I = ZB 1 l 3 ci 4 ce + ZC 1 l 2 p p T 3 ci 4 ce . (7.9)

Note that we did not need examples of the object at different lighting conditions and motion in order to construct this manifold these parts of the manifold came from the analytical expressions in Eq. (7.3). To represent the manifold at all the possible poses, we do not need such a tensor at every pose. Effects of 3D translation can be removed by centering and scale normalization, while in-plane rotation to a prede ned pose can mitigate the effects of rotation about the z axis. Thus, the image of object under arbitrary pose, p, can always be described by the multilinear object representation at a prede ned (Tpd , Tpd , Tpd , pd ), z x y z with only x and y depending upon the particular pose. Thus, the image manifold under any pose can be approximated by the collection of a few tangent planes on distinct j and j , denoted as pj . x y