USF Libraries
USF Digital Collections

Video-based person identification using facial strain maps as a biometric

MISSING IMAGE

Material Information

Title:
Video-based person identification using facial strain maps as a biometric
Physical Description:
Book
Language:
English
Creator:
Manohar, Vasant
Publisher:
University of South Florida
Place of Publication:
Tampa, Fla
Publication Date:

Subjects

Subjects / Keywords:
Face recognition
Face dynamics
Non-rigid motion
Optical flow
Elasticity
Dissertations, Academic -- Computer Science -- Masters -- USF
Genre:
bibliography   ( marcgt )
theses   ( marcgt )
non-fiction   ( marcgt )

Notes

Abstract:
ABSTRACT: Research on video-based face recognition has started getting increased attention in the past few years. Algorithms developed for video have an advantage from the availability of plentitude of frames in videos to extract information from. Despite this fact, most research in this direction has limited the scope of the problem to the application of still image-based approaches to some selected frames on which 2D algorithms are expected to perform well. It can be realized that such an approach only uses the spatial information contained in video and does not incorporate the temporal structure.Only recently has the intelligence community begun to approach the problem in this direction. Video-based face recognition algorithms in the last couple of years attempt to simultaneously use the spatial and temporal information for the recognition of moving faces. A new face recognition method that falls into the category of algorithms that adopt spatio-temporal representation and utilizes dynamic information extracted from video is presented. The method was designed based on the hypothesis that the strain pattern exhibited during facial expression provides a unique "fingerprint" for recognition. First, a dense motion field is obtained with an optical flow algorithm. A strain pattern is then derived from the motion field. In experiments with 30 subjects, results indicate that strain pattern is an useful biometric, especially when dealing with extreme conditions such as shadow light and face camouflage, for which conventional face recognition methods are expected to fail. The ability to characterize the face using the elastic properties of facial skin opens up newer avenues to the face recognition community in the context of modeling a face using features beyond visible cues.
Thesis:
Thesis (M.A.)--University of South Florida, 2006.
Bibliography:
Includes bibliographical references.
System Details:
System requirements: World Wide Web browser and PDF reader.
System Details:
Mode of access: World Wide Web.
Statement of Responsibility:
by Vasant Manohar.
General Note:
Title from PDF of title page.
General Note:
Document formatted into pages; contains 47 pages.

Record Information

Source Institution:
University of South Florida Library
Holding Location:
University of South Florida
Rights Management:
All applicable rights reserved by the source institution and holding location.
Resource Identifier:
aleph - 001795309
oclc - 151065438
usfldc doi - E14-SFE0001560
usfldc handle - e14.1560
System ID:
SFS0025878:00001


This item is only available as the following downloads:


Full Text

PAGE 1

Video-Based P erson Iden tication Using F acial Strain Maps as a Biometric b y V asan t Manohar A thesis submitted in partial fulllmen t of the requiremen ts for the degree of Master of Science in Computer Science Departmen t of Computer Science and Engineering College of Engineering Univ ersit y of South Florida Co-Ma jor Professor: Dmitry B. Goldgof, Ph.D. Co-Ma jor Professor: Rangac har Kasturi, Ph.D. Sudeep Sark ar, Ph.D. Date of Appro v al: April 13, 2006 Keyw ords: face recognition, face dynamics, non-rigid motion, optical ro w, elasticit y c r Cop yrigh t 2006,V asan t Manohar

PAGE 2

DEDICA TION T o m y lo ving family

PAGE 3

A CKNO WLEDGEMENTS I w ould lik e to express m y sincere thanks to Dr. Dmitry Goldgof for giving me an opp ortunit y to w ork on this topic and for his adept guidance through the course of m y Master's. I am grateful to Dr. Rangac har Kasturi for his supp ort during m y sta y at USF and for providing me with n umerous prosp ects to rourish as a researc her. I am deeply indebted to Dr. Sudeep Sark ar for b eing on m y committee and oering his v ast exp ertise in the area of face recognition. It w as a pleasure and honor to w ork with m y committee. Sp ecial thanks are due to Dr. Y ong Zhang and Sangeeta J. Kundu for their insigh tful discussions on understanding the concept of using material prop erties for ob ject recognition. Am also thankful to Dr. P admanabhan Soundarara jan and Pranab K. Mohan t y for their helpful and willing in teractions during the implemen tation of the pro ject. Finally I w ould lik e to extend m y deep est gratitude to m y mom, dad and brother for alw a ys b eing there in p erplexing situations and adding color to life through their lo v e and aection.

PAGE 4

T ABLE OF CONTENTS LIST OF T ABLES iii LIST OF FIGURES iv ABSTRA CT v CHAPTER 1 INTR ODUCTION 1 1.1 F ace as a Biometric 1 1.2 Need for No v el Biometrics 2 1.3 The Prop osed Approac h 2 1.4 Thesis Organization 4 CHAPTER 2 REVIEW OF F A CE RECOGNITION ALGORITHMS 6 2.1 Image-Based Algorithms 6 2.1.1 2D Algorithms 6 2.1.2 3D Algorithms 8 2.2 Video-Based Algorithms 9 2.2.1 Algorithms Using Spatio-T emp oral Represen tation 10 CHAPTER 3 THEORETICAL BA CK GR OUND 12 3.1 Theory of Elasticit y 12 3.1.1 Stress and Strain 12 3.1.2 Material Prop ert y 14 3.1.2.1 Geometric Discretization 14 3.1.3 Ph ysically-Based Mo dels 15 3.1.3.1 F ace Mo del 17 3.2 Motion Analysis 17 3.2.1 Optical Flo w 17 3.2.2 Flo w Computation 18 3.2.3 Handling F ailure Cases 20 3.3 Subspace Learning for F ace Recognition 21 3.3.1 Principal Comp onen t Analysis 21 3.3.1.1 Prepro cessing 21 3.3.1.2 T raining 22 3.3.1.3 T esting 23 3.3.1.4 Analysis 23 CHAPTER 4 COMPUT A TIONAL METHODS 25 4.1 Motion Field b y Optical Flo w Metho d 25 4.2 Strain Image b y Finite Dierence Metho d 26 i

PAGE 5

CHAPTER 5 ANAL YSIS OF STRAIN P A TTERN AS A BIOMETRIC 29 5.1 Uniqueness and Stabilit y of Strain P attern 29 5.2 Strain Measuremen t 30 5.3 Strain Computations Using Pro jections 31 CHAPTER 6 EXPERIMENTS AND RESUL TS 33 6.1 Video Collection 33 6.2 Results With Non-Camouraged F aces 34 6.3 Results With Camouraged F aces 35 6.4 Results With T ap ed F aces 38 6.5 In v ariance of F acial Strain P attern 39 CHAPTER 7 DISCUSSION AND CONCLUSIONS 41 REFERENCES 43 ii

PAGE 6

LIST OF T ABLES T able 6.1 Non-Camouraged F ace Exp erimen ts. 35 T able 6.2 Camouraged F ace Exp erimen ts. 37 T able 6.3 T ap ed F ace Exp erimen ts. 38 iii

PAGE 7

LIST OF FIGURES Figure 1.1 System Flo w of the Prop osed Approac h. 5 Figure 3.1 Flo w Field Pro duced b y a T ypical Optical Flo w Algorithm on an Image P air. 18 Figure 3.2 The Prepro cessing Step in Principal Comp onen t Analysis. 22 Figure 4.1 Deriv ed Motion and Strain Images from Tw o Video F rames. 28 Figure 5.1 Strain Maps Depicting In ter-P erson Strain V ariabilit y Under Same Ligh ting Conditions. 32 Figure 5.2 Strain Maps Depicting In tra-P erson Strain Consistency Under Dieren t Conditions. 32 Figure 6.1 Video Acquisition Conditions. 35 Figure 6.2 R OCs of Non-Camouraged F ace Exp erimen ts. 36 Figure 6.3 Gen uine and Imp ostor Scores Distributions of Non-Camouraged F ace Exp erimen ts. 36 Figure 6.4 Similarit y Score Distributions of Camouraged F ace Exp erimen ts. 37 Figure 6.5 Similarit y Score Distributions of Regular and T ap ed F ace Exp erimen ts. 38 Figure 6.6 Similarit y Distributions of In tra and In ter-Class Strains Under Normal Ligh ting Conditions. 40 Figure 6.7 Similarit y Distributions of In tra and In ter-Class Strains Under Lo w Ligh ting Conditions. 40 iv

PAGE 8

VIDEO-BASED PERSON IDENTIFICA TION USING F A CIAL STRAIN MAPS AS A BIOMETRIC V asan t Manohar ABSTRA CT Researc h on video-based face recognition has started getting increased atten tion in the past few y ears. Algorithms dev elop ed for video ha v e an adv an tage from the a v ailabilit y of plen titude of frames in videos to extract information from. Despite this fact, most researc h in this direction has limited the scop e of the problem to the application of still image-based approac hes to some selected frames on whic h 2D algorithms are exp ected to p erform w ell. It can b e realized that suc h an approac h only uses the spatial information con tained in video and do es not incorp orate the temp oral structure. Only recen tly has the in telligence comm unit y b egun to approac h the problem in this direction. Video-based face recognition algorithms in the last couple of y ears attempt to sim ultaneously use the spatial and temp oral information for the recognition of mo ving faces. A new face recognition metho d that falls in to the category of algorithms that adopt spatiotemp oral represen tation and utilizes dynamic information extracted from video is presen ted. The metho d w as designed based on the h yp othesis that the strain pattern exhibited during facial expression pro vides a unique \ngerprin t" for recognition. First, a dense motion eld is obtained with an optical ro w algorithm. A strain pattern is then deriv ed from the motion eld. In exp erimen ts with 30 sub jects, results indicate that strain pattern is an useful biometric, esp ecially when dealing with extreme conditions suc h as shado w ligh t and face camourage, for whic h con v en tional face recognition metho ds are exp ected to fail. v

PAGE 9

The abilit y to c haracterize the face using the elastic prop erties of facial skin op ens up new er a v en ues to the face recognition comm unit y in the con text of mo deling a face using features b ey ond visible cues. vi

PAGE 10

CHAPTER 1 INTR ODUCTION Recognizing p eople is a fundamen tal and routine activit y in our daily liv es b ecause ensuring the iden tit y and authen ticit y of p eople is a prerequisite for man y applic ations Biometric iden tication, or biometrics is the science of iden tifying a p erson based on his/her ph ysiological or b eha vioral c haracteristics. Ph ysiological biometrics, lik e face or ngerprin t, are ph ysical c haracteristics generally measured at some p oin t of time. On the other hand, b eha vioral biometrics, lik e signature or v oice, consist of the w a y some action is carried out and extend o v er time [14]. F ace, ngerprin t, hand geometry iris, signature and v oice are six biometrics most commonly used in to da y's automated authen tication systems. These biometric iden tiers are considered \mature" and generally nd widespread deplo ymen t at presen t. 1.1 F ace as a Biometric Researc h in face recognition has exp erienced a considerable surge not only b y the fundamen tal c hallenges this recognition problem p oses but also b y n umerous practical applications where h uman iden tication is needed. Rapid progress in sensoring tec hnology and increased demands on securit y ha v e escalated the imp ortance of face recognition as one of the primary biometric tec hnologies [40]. F ace recognition as a biometric has sev eral adv an tages o v er other suc h tec hnologies: It is natural, nonin trusiv e, and easy to use. Among the six biometric attributes considered b y Hietmey er [30 ], facial features scored the highest compatibilit y in a Mac hine Readable T ra v el Do cumen ts (MR TD) [1] system based on a n um b er of ev aluation factors, suc h as enrollmen t, renew al, mac hine requiremen ts and public p erception. 1

PAGE 11

A face recognition system can op erate in either or b oth of the follo wing t w o mo des: 1. F ace v erication It in v olv es a one-to-one matc h that compares a query face image against a template face image whose iden tit y is b eing claimed. 2. F ace iden tication It in v olv es one-to-man y matc hes that compares a query face image against all the template images in the database to determine the iden tit y of the query face. 1.2 Need for No v el Biometrics The p erformance of face recognition systems and the n um b er of suc h systems ha v e increased signican tly since the rst automatic face recognition system w as dev elop ed b y Kanade [35]. This is clearly evidenced b y the represen tation of face recognition pap ers in primary conferences and journals. Ho w ev er, main tec hnical diculties iden tied in the earlier studies still exist [18, 61], esp ecially those caused b y illumination and p ose c hanges. F or example, recen t studies with a large database indicate that outdo or en vironmen t p oses a serious c hallenge to the curren t metho ds [54, 53]. Another concern is that app earance-based approac hes require that a \clean" face b e presen t in images. This could b e problematic in situations where the app earance of a face is mo died b y mak eup or plastic surgery either in ten tionally or unin ten tionally T o deal with those issues, the trend is to searc h for information that is not used b y traditional metho ds. F or example, range image has b een used to pro vide 3D data [15]. In visible mo dalities suc h as infrared imagery w ere also in v estigated [63]. 1.3 The Prop osed Approac h Recen t psyc hological and neural studies indicate that c hanging facial expressions and head mo v emen ts pro vide additional evidences for recognition [51]. Therefore, b oth xed facial features and dynamic p ersonal c haracteristics are used in the h uman visual system (HVS) to recognize faces. It is clear that facial dynamics if exploited ecien tly can lead to impro v ed p erformance in recognition. By facial dynamics w e refer to the non-rigid mo v emen t of facial features, in addition to the rigid mo v emen t of the whole face (head) [27]. 2

PAGE 12

Inspired b y these ndings, a no v el metho d that uses dynamic strain pattern from video to recognize face is prop osed. The metho d has sev eral unique features: It is the strain pattern rather than the in tensit y pattern that is used for recognition. Strain pattern is extracted from a video sequence in whic h a sub ject's face ma y b e deformed and co v ered with camourage. Videos are acquired with a regular camcorder and no sp ecial imaging equipmen ts suc h as range and infrared cameras are needed. The study w as also partially motiv ated b y the observ ation that most kno w ho w to defeat an app earance-based photometric metho d. In an informal surv ey conducted here at the Univ ersit y of South Florida, college studen ts w ere ask ed to nd a w a y to fo ol a face recognition system. The participan ts w ere undergraduate studen ts with computer science bac kground ha ving no adv anced kno wledge of ho w a face recognition system w orks. Out of the 79 studen ts who participated in the surv ey whic h w as designed as a questionnaire with m ultiple c hoices, 79% c hose mak eup, 51% selected sunglasses and 28% suggested plastic surgery It w ould b e naiv e to b eliev e that a trained professional (or terrorist) cannot devise the same, if not b etter tactics, to a v oid b eing caugh t b y the curren t face recognition tec hnology F acial strain pattern is more robust in the presence of those adv erse factors b ecause of its strong ro ot in biomec hanics. Strain pattern is related to material prop erties of soft tissue that is unique for eac h individual. Sp ecically strain pattern has t w o adv an tages: 1. It is less vulnerable to mak eup or camourage, b ecause the strain pattern of a face (with or without mak eup) remains the same as long as accurate facial deformation is captured; 2. An y c hange of soft tissue prop erties caused b y plastic surgery will b e rerected in the strain pattern and hence will b e detected. In [66, 36], Kundu et al. use elastic strain maps for face recognition on still images. They obtain 99% v erication rate at 5% false alarm rate on a data set con taining 50 sub jects. This empirically sho ws that strain pattern has the discriminativ e p o w er to b e deplo y ed as a biometric. This thesis is a con tin uation of the earlier eort and mak es signican t adv ances: 3

PAGE 13

Motion computation is automated with an optical ro w algorithm that replaces the manual feature matc hing approac h. Strain computation is simplied with a more ecien t nite dierence metho d. A sequence of strain patterns, rather than a snap shot of strain pattern, is gathered, whic h enables us to study its in v ariance prop ert y A video sequence of a sub ject undergoing a particular facial expression is the essen tial input to the system. The apparen t facial motion is empirically estimated using an optical ro w algorithm. Since optical ro w traditionally fails on extensiv e motion, it is applied on subsequen t frame-pairs instead of the start and end frame of the expression. The displacemen t v ectors from frame-pairs are then com bined using a b ackwar d-mapp e d w arping approac h. Based on a few p oin ts man ually selected in the rst frame and the displacemen t v ectors obtained from the previous step, frames are registered to remo v e an y rigid motion of the face (head). The strain map of the sub ject's facial expression is calculated using a forw ard dierence appro ximation. The strain images of all the sub jects are then pro jected to a lo w er dimensional space using the Karh unen-Lo ev e transform or principal comp onen t analysis (PCA) [22]. The main motiv ation b ehind using this is the fact that the features in suc h subspace pro vide more salien t and ric her information for recognition than the ra w images themselv es [40]. In this space, a strain image is ecien tly represen ted as a v ector of w eigh ts (fe atur e ve ctor) of lo w dimensionalit y Subspace learning is the pro cess of learning these w eigh ts using a set of training images. During testing, when a query face is to b e iden tied with one of the faces in the database, distances to eac h of the training strain patterns are calculated in the pro jected subspace. This results in a matrix of similarit y scores (kno wn/enrolled or gal lery images vs. unkno wn or pr ob e images) whic h are p opularly analyzed using Receiv er Op erating Characteristic (R OC) curv es [47, 28]. Figure 1.1 giv es an o v erview of the complete system ro w. 1.4 Thesis Organization This thesis is organized in the follo wing manner. Chapter 2 presen ts a taxonom y of face recognition approac hes existing in the literature along with a brief description of the seminal w orks. Chapter 3 pro vides the reader with an in tro duction to the theory of elasticit y motion 4

PAGE 14

and Masking Similarity scores and ROC Distances in projected subspace Displacement vector for the complete sequence Training TestingCoordinate Point Extraction Repeated for each subject in the databaseOn the database containing strain maps of all subjectsGeometric Normalization Input video sequence Optic flow on two subsequent frames Displacement vectors for frame-pairs across sequence Linking flow values from each frame-pair Strain Map of a Subject (Strain to Intensity) Rigid registration across frames Forward difference Approximation Euclidean Subspace Principal Component Analysis Classifier Nearest Neighbor Figure 1.1 System Flo w of the Prop osed Approac h. analysis and subspace learning whic h are the in tegral comp onen ts of the prop osed approac h. Chapter 4 describ es the computational metho ds used in the designed tec hnique. Chapter 5 puts forw ard an analysis of strain pattern as a biometric. Chapter 6 details the exp erimen tal setup and rep orts the results on eac h of the conducted exp erimen ts. Chapter 7 concludes the ndings of this w ork along with a discussion on the scop e of using strain pattern as a soft biometric. 5

PAGE 15

CHAPTER 2 REVIEW OF F A CE RECOGNITION ALGORITHMS It is presumptuous to assume that one can giv e a complete review of existing face recognition tec hniques. There are coun tless pap ers in literature that dier in the data and mo dels they use to solv e the problem. Giv en the scop e of this w ork where a surv ey of curren t tec hniques is not the primary ob jectiv e, this section giv es an o v erview of the most imp ortan t w orks in this direction. In a broadsense, face recognition approac hes can b e classied in to t w o categories: 1. Image-Based F ace Recognition Algorithms 2. Video-Based F ace Recognition Algorithms The follo wing sections giv e a brief outline of algorithms that fall in to one of the t w o categories with an emphasis on video-based approac hes that extract dynamic information for impro v ed p erformance. 2.1 Image-Based Algorithms 2.1.1 2D Algorithms Princip al Comp onent A nalysis (PCA) [61, 52, 50] is a dimensionalit y reduction approac h whic h can b e used as a recognition tec hnique in the con text of learning a subspace whose basis v ectors corresp ond to the maxim um v ariance direction in the original image space. It is one of the rst and most p opular approac h to face recognition whic h has a reasonable p erformance in most scenarios. Indep endent Comp onent A nalysis (ICA) [6 43] attempts to nd the basis v ectors along whic h the data are statistic al ly indep endent This is done b y minimizing the second-order and higher-order dep endencies in the input data. Line ar Discriminant A nalysis (LD A) [45, 7, 67] nds the basis v ectors in the original space that results in maximal separation 6

PAGE 16

among dieren t classes. F or all samples of all classes, the goal of LD A is to maximize the b et w een-class scatter while minimizing the within-class scatter. It is generally b eliev ed that ob ject recognition algorithms based on LD A are sup erior to those based on PCA. Ho w ev er, in [46], Martinez and Kak argue that this is not the case alw a ys. Results on a face database sho w ed that PCA outp erforms LD A when the training data set is small and also that PCA is less sensitiv e to training data sets. Evolutionary Pursuit (EP) [44], a sp ecic kind of genetic algorithm, is an eigenspace-based dynamic approac h that searc hes for the b est set of pro jection axes in order to maximize an ob jectiv e function, while accoun ting for classication accuracy and generalization capabilit y of the solution. All of the ab o v e metho ds fall in the category of template-matching approac hes. Elastic Bunch Gr aph Matching (EBGM) [64] is a fe atur e-b ase d tec hnique in whic h faces are represen ted as graphs, with no des p ositioned at some anc hor p oin ts detected on the face (ey es, nose, etc...) and edges lab eled with 2-D distance v ectors. Recognition is then based on these lab eled graphs. T r ac e tr ansforms [34, 58] is a recen t image pro cessing tec hnique that is used to recognize ob jects under transformations, i.e. rotation, scaling and translation. First, a functional is computed along tracing lines of an image and then recognition is p erformed using a set of trace functionals. Bayesian Intr a/extr ap ersonal Classier (BIC) [49, 42] presen ts a probabilistic framew ork based on the Ba y esian b elief that the image in tensit y dierences are c haracteristic of t ypical v ariations in app earance of an individual. Similarit y among individuals is measured using the Ba y esian rule. Apart from these, there is a dieren t class of face recognition algorithms that are based on mo deling the subspace to b etter c haracterize the sp ecic case of faces. Kernel metho ds [65, 5, 71, 70] w ere dev elop ed based on the premise that the face manifold in the subspace need not necessarily b e linear. These metho ds essen tially explore direct non-linear manifold sc hemes to generalize the linear metho ds. Metho ds using Supp ort V e ctor Machines (SVM) [26, 29] handle face recognition as a binary classication problem. They nd a h yp erplane that separates the largest fraction of data samples from one class on the same side, while maximizing the distance from either classes to the h yp erplane. PCA is rst used to extract features of face images and 7

PAGE 17

then discrimination functions b et w een eac h pair of images are learned b y SVMs. An excellen t review of 2D face recognition algorithms can b e found in [18, 68]. 2.1.2 3D Algorithms 2D algorithms sho w v ery go o d recognition rates o v er a restricted set of inputs. Ho w ev er, they are not rexible with resp ect to p ose, illumination or scale c hanges, whic h require m ultiple templates to p erform recognition o v er v arying conditions. In ligh t of this, a need to explore new er information for face recognition w as felt necessary Using range data w as found to b e a promising direction b ecause curv ature data whic h, can b e computed from depth information, op ens up new horizons for face recognition metho ds in terms of c haracterizing a face and b eing viewp oin t indep enden t. 3D algorithms can b e further classied in to t w o branc hes: 1. Using range information only 2. Using range and texture information Metho ds that use depth information alone p erform recognition using geometric prop erties of the facial surface suc h as curv ature, depth maps, etc... On the other hand, algorithms that use b oth range and texture cues use in tegrated mo dels whic h com bine a mo del of shap e v ariation with a mo del of the app earance v ariations in a shap e-normalized frame. An A ctive App e ar anc e Mo del (AAM) [19] con tains a statistical mo del of the shap e and gra ylev el app earance of the face whic h can b e generalized to an y v alid face. Matc hing is done b y nding parameters of the mo del whic h minimize the dierence b et w een the original image and the pro jected image obtained from a syn thesized mo del. Morphable mo dels [12 13], prop osed b y Blan tz et al., is based on the h yp othesis that a h uman face is in trinsically a surface lying in 3-D space. They prop osed a face recogn tion metho d based on a morphable face mo del that enco des shap e and texture in terms of parameters of the mo del and an algorithm to reco v er these parameters from a single image of a face. Another seminal w ork in this direction is a recognition metho d based on c anonic al surfac es [16 ]. The no v el con tribution of this w ork is the abilit y to compare facial surfaces irresp ectiv e of deformations caused b y facial expressions. The range image is prepro cessed b y remo ving certain parts suc h as hair and ey ebro ws whic h can complicate the recognition step. A canonical form of the facial surface is then computed. 8

PAGE 18

Since suc h a represen tation is less sensitiv e to head orien tation and facial expression, the recognition pro cess is signican tly simplied. Though using 3-D information o v ercomes the problems p ose and illumination in tro duce, the latency in v olv ed in range sensoring equipmen ts mak e it unfa v orable for practical purp oses. In [15], Bo wy er et al. presen t an elab orate discussion on the existing 3D and m ulti-mo dal 2D + 3D face recognition approac hes and the c hallenges that need to b e addressed. 2.2 Video-Based Algorithms F ace recognition in image sequences has receiv ed more atten tion in the last three or four y ears. Video-based face recognition algorithms can b e classied in to t w o categories: metho ds that do not utilize the motion information presen t in video and metho ds that in tegrate the motion information for recognition. The rst category of algorithms are those that do not exploit dynamics of video ecien tly and apply still image-based tec hniques to some \go o d" frames selected from image sequences [18]. An example for suc h an approac h is [24], where a video-based face recognition system based on trac king the p ositions of the nose and ey es is prop osed. The lo cation of these p oin ts is used to mak e a decision whether the face p ose is acceptable for a still image-based recognition tec hnique to b e launc hed; otherwise the trac king con tin ues un til suc h a frame o ccurs. Other approac hes in this category include 3-D reconstruction and recognition via structure from motion or structure from shading. It is clear that the ab o v e approac hes exploit the abundance of frames in video and not essen tially the facial dynamics b y mainly using the spatial information. The second category of algorithms are those that attempt to sim ultaneously use the spatial and temp oral information for recognizing faces undergoing some mo v emen ts. These metho ds ecien tly exploit the temp oral information b y c ho osing a form of spatio-temp oral representation that includes b oth the facial structure and its dynamics. Some w orks using this principle include the condensation metho d [72] and the metho d based on Hidden Mark o v Mo dels (HMM) [2 ]. More recen tly Soatto et al. used a linear dynamic system mo del [56 ] to capture 9

PAGE 19

the spatio-temp oral information in image sequences [3]. A discussion of the second class of algorithms that use motion information in the follo wing section. 2.2.1 Algorithms Using Spatio-T emp oral Represen tation In [41], Li prop osed an approac h for mo deling facial dynamics using iden tit y surfaces. A set of mo del tra jectories constructed on iden tit y surfaces is matc hed with the face tra jectory constructed from the discriminating features. A recognition rate of 93 : 9% is rep orted on a dataset con taining 12 training sequences and testing sequences of 3 sub jects. In [39], Li and Chellappa rep ort enhancemen t o v er the frame-to-frame matc hing sc heme b y using tra jectories of trac k ed features in video. Gab or lters w ere used to extract the features of in terest. In the p opular w ork of [69], Zhou and Chellappa prop osed a framew ork for sim ultaneous trac king and recognition of faces b y including an iden tication v ariable to the state v ector of the mo del. Condensation algorithm is another w a y of c haracterizing temp oral information. Though traditionally used for trac king and recognizing m ultiple spatio-temp oral features, this has b een recen tly extended to video-based face recognition [72, 69]. In metho ds using Hidden Mark o v Mo dels [2], an HMM is created to learn b oth the statistics and temp oral structure of eac h sub ject during the training phase. In the testing phase, the temp oral dynamics of the sequence is analyzed o v er time b y the HMM corresp onding to eac h individual. The highest lik eliho o d score pro vided b y an HMM establishes the face iden tit y in the video. In the recen t w orks of Soatto [56], the auto-regressiv e and mo ving a v erage (ARMA) mo del w as used to c haracterize a mo ving face as a linear dynamical system. Other metho ds for video-based face recognition include algorithms that incorp orate manifold learning. Lee et al. prop osed one suc h an approac h based on probabilistic app earance manifolds [37]. The prop osed algorithm of this w ork falls in to the category of video-based face recognition metho ds that adopt a spatio-temp oral represen tation. F or a face undergoing some expression, non-rigid displacemen ts are used to c haracterize the prop erties of soft tissue and b one structure that constitute a face. The strain induced b y the deformation during the expression is used to dene the elastic prop erties of the facial skin whic h is in turn used to p erform recognition. 10

PAGE 20

This op ens up new er a v en ues to the face recognition comm unit y in the con text of mo deling a face using features b ey ond visible cues. 11

PAGE 21

CHAPTER 3 THEORETICAL BA CK GR OUND 3.1 Theory of Elasticit y When a b o dy is sub jected to external forces, the forces acting on it could either b e surfac e for c es whic h act o v er the surface of the solid, or b o dy for c es whic h are distributed o v er the v olume of the solid. The dissemination of these forces through a solid causes the generation of in ternal forces. T o study these forces, w e use a con tin uum mo del of the material in whic h matter is assumed to b e con tin uously distributed across the solid [21]. Con tin uum mec hanics deals with the mo v emen t of materials when sub jected to applied forces. The motion of a con tin uous and deformable solid can b e describ ed b y a con tin uous displacemen t eld resulting from a set of forces acting on the solid b o dy In general, the displacemen ts and forces ma y v ary con tin uously with time, but for the purp oses of this w ork w e use a t w o-state quasistatic mo del. The initial unloaded state of the material is referred to as the r efer enc e or undeformed state as the displacemen ts are zero ev erywhere. The material then recongures due to applied loads and reac hes an equilibrium state referred to as the deforme d state. The concepts of strain, a measure of length c hange or displacemen t gradien t, and stress, the force p er unit area on an innitesimally small plane surface within the material, are of fundamen tal imp ortance for nite deformation elasticit y theory 3.1.1 Stress and Strain W e dene the quan tit y stress to measure the in tensit y of eac h of the external forces when a solid b o dy is under equilibrium. Stress is used to measure the state of the force acting on the solid. It denes the force acting p er unit area. Stresses can b e decomp osed in to three comp onen ts based on the dieren t forces acting on the plane of the solid [21] as sho wn in 12

PAGE 22

Equation 3.1. xx = lim A 0 ( F x A ) xy = lim A 0 ( F y A ) xz = lim A 0 ( F z A ) (3.1) The comp onen t xx is the normal stress whic h measures the in tensit y of the normal force on the plane at a p oin t. The comp onen ts xy and xz are the shear stresses whic h measure the in tensit y of the shear force on the plane. Normal stress pro vides for c hange in the v olume of the material while shear stresses are resp onsible for the deformation of the material without aecting it's v olume. It can b e sho wn that the normal and the shear stresses on an y three orthogonal planes are sucien t to completely describ e the state of stress at a giv en p oin t. The stress tensor comprising the stress comp onen ts can b e expressed in a matrix form [48, 21] as sho wn in Equation 3.2. [ ] = 2 6 6 6 6 6 4 xx xy xz y x y y y z xx z y z z 3 7 7 7 7 7 5 (3.2) There are six distinct strain comp onen ts along with three complimen tary shear stresses along the diagonal whic h are iden tical, i.e. ( xy = y x ), ( z y = y z ) and ( xz = y z ). Str ain is another measure whic h has to b e considered when a b o dy undergo es some deformation. The eect on the b o dy's geometry under external forces can b e dened in terms of the displacemen ts of eac h p oin t in the b o dy There are t w o t yp es of displacemen ts p ossible: 1. Rigid b o dy displacemen ts 2. Deformation or non-rigid displacemen ts While rigid b o dy displacemen ts consists of translations and rotations of the b o dy as a whole, deformation consists of displacemen ts of p oin ts within the b o dy relativ e to one another [21]. Strain is used to quan tify the deformation undergone. The direct strain ( ) is dened as: = ds ds ds (3.3) where ( ds ) is the length b efore deformation and ( ds ) is the length after deformation. 13

PAGE 23

An innitesimal strain tensor is dened as: [ e ] = 1 2 [ r u + ( r u ) T ] (3.4) where u is a displacemen t v ector. 3.1.2 Material Prop ert y The stress measures the in ternal tension in the material whereas the strain measures its lo cal deformation. The dynamics of the medium ma y th us b e fully describ ed with an equation of motion relating these state v ariables to those represen ting the in v olv ed actions. The general form of this equation is giv en b y the Virtual Work principle [38]: ^ W acc = ^ W int + ^ W ext (3.5) where ^ W acc is the virtual w ork due to the acceleration, ^ W int is the virtual w ork due to the in ternal stresses and ^ W ext is the virtual w ork due to the external actions. In order to determine the state of the system resulting from the applied actions, the obtained equation m ust b e solv ed. Ho w ev er, as b oth state v ariables app ear to b e in v olv ed, this is not p ossible unless a supplemen tary equation relating stress and strain to eac h other is considered: the c onstitutive r elationship It is this relation that pro vides to the mo del the description of the sp ecic material prop erties of the con tin uous medium. 3.1.2.1 Geometric Discretization In principle, using the constitutiv e relationship of the material, the Virtual W ork equation 3.5 ma y b e solv ed for the stress or the strain, in order to determine the ev olution of the con tin uous medium under the external actions applied to it. Ho w ev er, in practice, a general solution v alid for an y p oin t of the con tin uous medium can b e extracted only in singular cases, and not for arbitrary deformations. In suc h a case, solutions ma y b e deriv ed for a nite set of sample p oin ts only Geometric discretization of the con tin uous medium is th us necessary so 14

PAGE 24

as to appro ximate the solution of the general con tin uous problem to that of a nite n um b er of equations dened for the mesh no des. V arious discretization metho ds ma y b e applied for this purp ose. Ho w ev er, all deriv e from the nite element metho d [73]. The basic idea of this metho d is to sub divide the con tin uous domain in to smaller elemen ts of v arious shap e and size. In general, the elemen ts ha v e simple geometric shap es suc h as segmen ts for curv es, triangles or quadrilaterals for surfaces, tetrahedrons or parallelepip eds for v olumes. An y p oin t in the con tin uous medium ma y th us b e expressed within an elemen t b y in terp olation b et w een its no des. As a result of this geometric discretization stage, the dynamic resp onse of the mesh under the dened external no dal loads ma y b e analyzed b y solving the system of equations for the no dal displacemen ts v ector U This w a y the solution of the con tin uous deformation problem is appro ximated to the solutions obtained for the no des of the mesh whic h appro ximates the con tin uous domain. Though geometric discretization has led to geometric linearization of the problem, ph ysical non-linearities, due in particular to the nonlinear ph ysical prop erties of the material still remain. Due to a lac k of resolution metho ds, still no solution can b e carried out directly from the discrete equation of motion, and the solution can only b e appro ximated step-b ystep b y means of incremen tal/iterativ e metho ds. It is therefore necessary to con v ert the time-con tin uous equation of motion and constitutiv e relation in to incremen tal forms, so that within eac h time in terv al the problem ma y b e handled as linear. 3.1.3 Ph ysically-Based Mo dels The rst and most famous ph ysically-based approac h for realistic deformation sim ulation in computer animation w as prop osed b y T erzop oulos et al. [59]. Instead of the Virtual W ork Principle, their mo del w as based on the L agr ange F ormalism Though the form ulation is dieren t, the mec hanical problem sta ys the same. Assuming a constan t uniform mass densit y and an isotropic h yp er-visco elastic material with constan t uniform damping densities the 15

PAGE 25

Lagrange equation of motion w as obtained as: d dt dq i dt + dq i dt + @ W int el ast @ q i = f q i ext (3.6) where, W int el ast = R V k C C 0 k 2 dV is the strain energy function of the h yp erelastic material f ext q i are pro jections on q i of the external force v ector C is the Cauc h y-Green righ t dilation tensor of the deformation T erzop oulos et al. prop osed the general strain energy function in the form of con trol con tin uit y generalized spline k ernels allo wing the mo deling of inelastic b eha viors suc h as viscoplasticit y fracture, etc. F or linear b eha vior, the strain energy function w as tak en as: W int el ast = Z L ( .stretc hing + .b ending + r .t wisting) dl for a curv e (3.7) W int el ast = Z S ( .stretc hing + .b ending) ds for a surface (3.8) where and r are parameters con trolling stabilit y Spatial discretization w as then ac hiev ed follo wing the nite dierence metho d, i.e. applying a grid discretization with a linear nite elemen t in terp olation leading to an equation of motion of the form: M U + D U + K ( U ) U = F ext (3.9) where, U U and U are no dal acceleration, v elo cit y and displacemen t v ectors resp ectiv ely M and D are mass and damping diagonal matrices resp ectiv ely K ( U ) is deformation dep enden t stiness matrix F ext is external no dal force v ector 16

PAGE 26

The nonlinear ordinary dieren tial system w as then con v erted in to a sequence of linear algebraic systems b y means of a nite dierence time discretization and w ere in tegrated through time using an in tegration pro cedure. 3.1.3.1 F ace Mo del The face mo del incorp orates a ph ysical appro ximation to h uman facial tissue [60]. The in ten tion is to appro ximate the elastic nature of the h uman skin tissues under a giv en load (facial expression). W e also try to estimate c hange in the material prop erties of the h uman skin (whether mo died b y surgery or mak e-up) based on the c hange in the strain pattern for a particular expression. The force exerted and the resultan t displacemen t of the facial tissues are in v estigated [36]. The strain induced b y activ ated m uscle bres in op ening the mouth is the primary en tit y studied. The mo del prop osed b y T erzop oulos et al. to em ulate facial expression motion is used. The strain resulting from the stretc hing of the skin is considered. This is an in v erse problem wherein w e kno w the displacemen ts and try to reco v er the forces that are exerted. 3.2 Motion Analysis 3.2.1 Optical Flo w Optical ro w rerects the c hanges in the image due to motion during a time in terv al dt It describ es the v elo cit y eld that represen ts the three-dimensional motion of ob ject p oin ts across a t w o-dimensional image. In [57], Sonk a et al. quote the follo wing salien t features of an optical ro w algorithm: Optical ro w should not b e sensitiv e to illumination c hanges and motion of unimp ortan t ob jects (e.g., shado ws) Non-zero optical ro w is detected if a xed sphere is illuminated b y a mo ving source Smo oth sphere rotating under constan t illumination pro vides no optical ro w 17

PAGE 27

Figure 3.1 1 giv es an example of the v ector eld that w ould b e pro duced b y a t ypical optical ro w algorithm. (a) Time t 1 (b) Time t 2 (c) Optical ro w Figure 3.1 Flo w Field Pro duced b y a T ypical Optical Flo w Algorithm on an Image P air. 3.2.2 Flo w Computation Optical ro w computation is based on the follo wing assumptions [31]: The observ ed brigh tness of an y ob ject p oin t is constan t o v er time. Nearb y p oin ts in the image plane mo v e in a similar manner (v elo cit y smo othness constrain t). Let f ( x; y ; t ) b e a con tin uous image. W e can use T a ylor series to represen t a dynamic image as a function of p osition and time. f ( x + dx; y + dy ; t + dt ) = f ( x; y ; t ) + f x dx + f y dy + f t dt + O ( @ 2 ) (3.10) where f x f y and f t are partial deriv ativ es of f and O ( @ 2 ) denotes the higher-order terms in the expansion. When an immediate neigh b orho o d of (x,y) is translated some small distance ( dx; dy ) during the in terv al dt the ev en t can b e mathematically expressed as: f ( x + dx; y + dy ; t + dt ) = f ( x; y ; t ) 1 This example is from the b o ok Image Pr o c essing: A nalysis and Machine Vision b y Sonk a et al. Chapter 14: Motion A nalysis. 18

PAGE 28

If dx dy dt are v ery small, the higher-order terms in equation v anish, and w e can reduce Equation 3.10 as: f t = f x dx dt + f y dy dt (3.11) The goal of optical ro w metho d is to determine the v elo cit y c = ( u; v ) = ( dx dt ; dy dy ) f x f y f t can b e computed, or at least appro ximated, from f ( x; y ; t ). Motion v elo cit y can then b e calculated as: f t = f x u + f y v = r f c (3.12) where r f is a t w o-dimensional image gradien t. F rom Equation 3.12, it can b e seen that the gra y-lev el dierence, f t at the same lo cation of the image at times t and t + dt is a pro duct of spatial gra y-lev el dierence and v elo cit y in this lo cation. Ho w ev er, Equation 3.12 do es not completely dene the v elo cit y v ector completely but rather pro vides the comp onen t in the direction of the brigh test gradien t. In order to handle this, a smo othness constrain t is in tro duced whic h states that the v elo cit y v ector eld c hanges slo wly in a giv en neigh b orho o d. Th us, the problem no w reduces to minimizing the squared error quan tit y: E 2 ( x; y ) = ( f x u + f y v + f t ) 2 + ( u 2 x + u 2 y + v 2 x + v 2 y ) (3.13) where u 2 x u 2 y v 2 x v 2 y denote partial deriv ativ es squared as error terms. The rst term in Equation 3.13 is the solution to Equation 3.12 and the second term is the smo othness criterion is a Lagrange m ultiplier. W e can reduce this to solving the dieren tial equations: ( 2 + f 2 x ) u + f x f y v = 2 u f x f t f x f y u + ( 2 + f 2 y ) v = 2 v f y f t (3.14) where u v are mean v alues of the v elo cit y in directions X and Y in some neigh b orho o d of ( x; y ). 19

PAGE 29

A solution [57] to the dieren tial equations of 3.14 is: u = u f x P D v = v f y P D (3.15) P = f x u + f y v ; D = 2 + f 2 x + f 2 y Measuremen t of the optical ro w is then based on a Gauss-Seidel iteration metho d [33] using pairs of consecutiv e images. 3.2.3 Handling F ailure Cases Errors in optical ro w computation o ccur when the brigh tness constancy and v elo cit y smo othness assumptions are violated. Suc h violations are quite common in real data. Highly textured regions, mo ving b oundaries, and depth discon tin uities are few examples where optical ro w computation fails dramatically In addition to constrain t violations, global relaxation metho ds of optical ro w computation also results in ro w estimation errors propagate across the solution. The reason for this is the fact that global metho ds nd the smo othest v elo cit y eld consisten t with the image data. Th us, a small n um b er of problem areas ma y cause widespread errors and p o or optical ro w estimates. Lo cal optical ro w estimation w as found to b e a natural solution to these problems. The basic idea is to divide the image in to small regions where the assumptions hold go o d. Though, this solv es the error propagation problem, it has it's o wn pitfall. In regions where the spatial gradien ts c hange slo wly the optical ro w estimation b ecomes ill-conditioned b ecause of lac k of motion information, and it cannot b e detected correctly If a global metho d is applied to the same region, the information from neigh b oring image parts propagates and w ould represen t a basis for optical ro w computation ev en if the lo cal information w as not sucien t b y itself. The conclusion of this discussion is that global sharing of information is b enecial in constrain t sharing but adv erse with resp ect to error propagation. A natural question then is \When to use a glob al metho d and when to use a lo c al appr o ach?" An answ er to this is b est obtained b y nding when the smo othness constrain t is 20

PAGE 30

violated. In order to detect regions in whic h the smo othness constrain ts hold, w e ha v e to select a threshold to decide whic h ro w v alue dierence should b e considered substan tial. This has it's o wn inheren t problems: If the threshold is to o lo w, man y p oin ts are considered p ositioned along ro w discon tin uities If the threshold is to o high, some p oin ts violating smo othness remain part of the computational net Blac k and Anandan [11] consider the problem of accurately estimating optical ro w from a pair of images using a no v el framew ork based on robust estimation whic h addresses violations of the brigh tness constancy and spatial smo othness assumptions. W e use this algorithm in our computations of facial motion. The details of this algorithm are summarized in Section 4.1. 3.3 Subspace Learning for F ace Recognition 3.3.1 Principal Comp onen t Analysis As discussed in Section 1.3, PCA is a dimensionalit y reduction tec hnique wherein the features along whic h maxim um v ariation in the dataset is captured are retained. The classication th us reduces from a higher dimension to a lo w er dimension called the eigen space, whic h is the space dened b y the principal comp onen ts or the eigen v ectors of the data set. In the PCA tec hnique, eorts ha v e b een on b oth fully automatic and partially automatic algorithms. P artially automatic algorithms are ones in whic h the co ordinates of landmark p oin ts on the image are supplied to the normalization routine i.e there is no automatic trac king of landmark p oin ts. In this w ork, w e use the partially automatic tec hnique. There are four steps in v olv ed in the partially automatic eigen approac h describ ed in the follo wing sections. 3.3.1.1 Prepro cessing 1. Lo cation of seed p oin ts. As men tioned earlier, the PCA approac h considered in this study requires that anc hor p oin ts b e supplied to the normalization routine. So w e need to lo cate 2 seed p oin ts whic h are presen t in all the sub jects in the dataset and whic h 21

PAGE 31

will b e further used for normalization. In our study for normalization of prole images, w e use the top of the nose and the corner of the ear as our 2 p oin ts as seen in Figure 3.2. 2. Geometric Normalization. In this step, the h uman c hosen seed p oin ts are lined up across the sub jects. Tw o xed lo cations (using the co de) lx, ly rx, ry w ere decided suc h that all the c hosen seed p oin ts for all the sub jects rest on these 2 xed p oin ts. F or aligning them translation, rotation and scaling are p erformed. 3. Masking. This is done to crop the (scaled and aligned from Step 2) image using a rectangular mask and the image b orders suc h that only the face from the forehead to c hin and ear to nose is visible. This is done to remo v e the un w an ted areas suc h as hair, bac kground etc. The mask for face/strain map is man ually sp ecied from the mean face/strain map image. In our case w e used images sequences of sizes 720x480 whic h w ere reduced to 200x250 pixels. (a) Seed p oin ts (b) Mask ed ra w image (c) Normalized strain map Figure 3.2 The Prepro cessing Step in Principal Comp onen t Analysis. 3.3.1.2 T raining In the training phase, the algorithm learns the subspace from the giv en inputs, i.e the eigen v alues and eigen v ectors of the training set are extracted. The eigen v ectors are c hosen based on the top eigen v alues whic h represen t the feature v ectors whic h retain the most v ariations across the images in the training set for discriminativ e purp oses. The training set should preferably con tain images whic h do not con tain m uc h of artifacts, suc h as sp ectacles, earrings, etc. and it should b e a set of images that do not ha v e an y duplicates. After extracting the most signican t v ectors ( m ), the images are then pro jected in to the eigen space of m dimen22

PAGE 32

sions. Eac h image is represen ted as a linear com bination of the m eigen v ectors in the reduced dimension. 3.3.1.3 T esting The testing phase is where the algorithm is pro vided a set of kno wn/enrolled faces (strain maps in our case) kno wn as the gallery set and a set of unkno wn faces/strain maps kno wn as the prob e set. The algorithm matc hes eac h prob e to its p ossible iden tit y in the gallery b y computing the Euclidean distance b et w een eac h prob e and eac h of the gallery images. 3.3.1.4 Analysis Dep ending on the mo de the biometric is op erated (See Section 1.1), the p erformance of the tec hnique is measured b y it's: V erication Rate. A v erication system has to tak e the measurable features of the sub ject (p) and compare it against the kno wn features of the p erson (g) whose iden tit y is b eing claimed, whic h mak es it a one-to-one matc hing problem. The p erformance of the system is measured using 2 statistics, rst is the probabilit y of v erication (( P V )) i.e accepting that the prob e p is actually the p erson g who the sub ject claims to b e, i.e. rep orting p = g when p g and second is the probabilit y of false alarm (( P F )), i.e rep orting p = g when p 6 = g The v erication rate is computed b y: P c;i V = 8 > < > : 0 if j D i j = 0 j s i ( k ) c giv en p k "D i j j D i j otherwise, and P c;i F = 8 > < > : 0 if j F i j = 0 j s i ( k ) c giv en p k "F i j j F i j otherwise, where s i ( k ) is the similarit y measure [55]. Th us a pair of ( P V ; P F ) are generated for a giv en cut-o v alue c The cut o v alue is selected b y v arying b et w een the minim um and maxim um distances obtained after pro jecting all the prob e images. By v arying c 23

PAGE 33

dieren t com binations of ( P V ; P F ) are pro duced. A plot of all ( P V ; P F ) is called the Relativ e (or Receiv er) Op erating Characteristics (R OC). Iden tication Rate. Iden tication scores are rep orted using the same n um b er of prob e and gallery images. A plot of all the p ercen tage of correct matc hes on the v ertical axis and the Rank along the horizon tal axis is called the Cum ulativ e Matc h Scores Curv e(CMC). The top rank matc h is at Rank 1 whic h indicates the fraction of prob es correctly iden tied. It has b een sho wn in [54] that CMC for small gallery sizes (30 in our case) dramatically underestimates the recognition p erformance for large gallery sizes. F or this reason, w e presen t our results as R OCs and similarit y score distributions of gen uine and imp ostor matc hes. An alternativ e to this is to emplo y metho ds that use similarit y scores of exp erimen ts conducted on small datasets to estimate the p erformance of the algorithm on a large dataset ha ving c haracteristics comparable to that of the smaller dataset [25 62]. 24

PAGE 34

CHAPTER 4 COMPUT A TIONAL METHODS 4.1 Motion Field b y Optical Flo w Metho d There are n umerous algorithms that ha v e b een dev elop ed to the solv e the Equation 3.14. These are usually accompanied with lo cal or global smo othness constrain ts explained in Section 3.2.3. The metho d adopted in this study is based on a robust estimation framew ork [11]. In [11], Blac k and Anandan ha v e considered the issues of robustness related to the reco very of optical ro w with m ultiple motions in an image sequence. It is understo o d from the discussions in Section 3.2.3 that measuremen ts are incorrect whenev er information is used from a spatial neigh b orho o d that spans a motion b oundary This holds go o d for b oth the brigh tness constancy and v elo cit y smo othness assumptions and violations of these constrain ts cause problems for the least-squares form ulation of optical ro w in Equation 3.13. The basic idea is to recast the least squared error form ulations with a dieren t err or-norm function instead of the standard quadr atic function. T o increase robustness, the error-norm function should b e more forgiving ab out outlying measuremen ts. One of the most common error-norm function in computer vision is the trunc ate d quadr atic where errors are w eigh ted quadratically up to a xed threshold but receiv e a constan t v alue b ey ond that. There are n umerous other error-norm functions that ha v e b een used in the literature, eac h with dieren t motiv ations and ecacies [9]. Loren tzian and Geman-McClure error-norm functions are used in [11]. The motiv ation for using these b eing the fact that they ha v e dieren tial inruential functions 1 whic h pro vide a more gradual transition b et w een inliers and outliers than do es the truncated quadratic. 1 An inruential function asso ciated with an error-norm function c haracterizes the bias that a particular measuremen t has on the solution. 25

PAGE 35

The authors explore n umerous optimization tec hniques to solv e the robust form ulations and found that deterministic con tin uation metho ds [10 ] to b e more ecien t and practical as a minimization tec hnique. T o cop e with large motions, a coarse-to-ne strategy [4 23] is emplo y ed in whic h a p yramid of spatially ltered and sub-sampled images is constructed. Beginning at the lo w est spatial resolution with the ro w c b eing zero, the c hange in the ro w estimate dc is computed. The new ro w eld, c + dc is then pro jected to the next lev el in the p yramid and the rst image at that lev el is warp e d to w ards the later image using the ro w information. The w arp ed image is then used to compute the dc at this lev el. The pro cess is rep eated un til the ro w has b een computed at the full resolution. Giv en a pair of frames, the algorithm pro duces t w o motion comp onen ts, u and v Examples of the generated motion eld are sho wn in Figure 4.1 (c,d). 4.2 Strain Image b y Finite Dierence Metho d A nite strain tensor is capable of describing large deformation of soft tissues: = 1 2 [ r u + ( r u ) T + ( r u ) T r u ] (4.1) where u ( u; v ) is the displacemen t v ector, and the gradien t op erator r is dened as: r u = 2 6 4 @ u @ x @ u @ y @ v @ x @ v @ y 3 7 5 (4.2) But the quadratic pro duct term in (2) in tro duces geometric nonlinearit y and amplies errors from motion eld. So, w e use Cauc h y tensor that is linear and adequate for biometric study: = 1 2 [ r u + ( r u ) T ] (4.3) 26

PAGE 36

In a 2D image co ordinate, it b ecomes: = 2 6 4 @ u @ x 1 2 ( @ u @ y + @ v @ x ) 1 2 ( @ v @ x + @ u @ y ) @ v @ y 3 7 5 (4.4) Giv en optical ro w data ( p; q ), w e can deriv e an image for eac h strain comp onen t ( @ u @ x ; @ u @ y ; @ v @ x ; @ v @ y ) with a nite dierence metho d. First, w e ensure that all frame pairs sen t to the optical ro w algorithm ha v e a xed time in terv al, so that w e can utilize motion v ector directly b y dropping the time v ariable ( 4 t = constant ): p = dx dt : = 4 x 4 t = u 4 t : = u (4.5) q = dy dt : = 4 y 4 t = v 4 t : = v (4.6) @ u @ x = u ( x ) u ( x + 4 x ) 4 x : = p ( x ) p ( x + 4 x ) 4 x (4.7) @ v @ y = v ( y ) v ( y + 4 y ) 4 y : = q ( y ) q ( y + 4 y ) 4 y (4.8) where 4 x and 4 y are preset distances (usually 1-3 pixels). The next step is to in tegrate all strain comp onen ts in to a single image. It has b een observ ed that, when a sub ject op ens his/her mouth, motion is mostly v ertical and strain pattern is dominated b y its normal comp onen ts ( @ u @ x ; @ v @ y ). Therefore, w e compute a strain magnitude image ( m ) with normal strains only: m = s @ u @ x 2 + @ v @ y 2 (4.9) A strain magnitude image is sho wn in Figure 4.1 (e). Using strain magnitude falls in line with the traditional 1D PCA analysis that is t ypically used in in tensit y-based approac hes. Alternativ ely strain comp onen ts could b e used whic h w ould require a m ulti-dimensional PCA approac h. Similarly sc hemes that incorp orate shear strains ma y also b e considered. The deriv ativ e of the motion v ector, strain is sensitiv e to motion discon tin uities, due to n umerical noise or the b oundary eect of a mo ving ob ject. As a result, high strain v alues 27

PAGE 37

are observ ed along the lo w er ja w line where computations span region b oundaries. Since it is not related to material prop erties of the facial tissues, suc h v alues are not included in the exp erimen ts. An alternativ e to this is to p erform segmen tation rst and extract the region of in terest b efore computing optical ro w. This w ould alleviate the problems caused b y surface discon tin uities. In studies with fron tal faces, images are usually normalized to ey e p ositions. Since w e used prole frames, w e selected the top of nose and the cen ter of ear as normalization landmarks. In addition, w e used the landmarks to create a mask that crops out a rectangle sub-section (200 x 250 pixels) from the original strain image (720 x 480 pixels). The sub-section w as then supplied to the recognition algorithm (Figure 4.1 (f )). Note that the region of in terest do es not include the lo w er ja w line and hence the motion discon tin uit y has no impact on the results. All exp erimen ts w ere conducted using the Principal Comp onen t Analysis (PCA) algorithm. More details ab out the implemen tation can b e found in [8]. (a) Video F rame 12 (b) Video F rame 15 (c) Motion Field ( p ) (d) Motion Field ( q ) (e) Strain Magnitude ( m ) (f ) Sub-section for PCA Figure 4.1 Deriv ed Motion and Strain Images from Tw o Video F rames. 28

PAGE 38

CHAPTER 5 ANAL YSIS OF STRAIN P A TTERN AS A BIOMETRIC A new biometric that can b e deplo y ed in practical settings should p ossess sev eral attributes: Its features b e unique and stable. F eatures b e measurable (directly or indirectly) with curren t sensoring tec hnology If features ha v e to b e extracted from ra w data, extraction algorithms m ust b e ecien t. W e will examine those asp ects of facial strain pattern in the follo wing sections. 5.1 Uniqueness and Stabilit y of Strain P attern Man y parts of h uman face ha v e b een studied in searc h for new biometrics. Iris and retina scans ha v e b een used in a restricted-access en vironmen t [63]. Ear also p ossesses information that can b e harnessed in a m ulti-classier framew ork [17 ]. Ho w ev er, soft tissue (m uscle and skin), the main anatomical unit of a face, has receiv ed little atten tion. F rom a mec hanical p oin t of view, the shap e of a face and its deformation is largely determined b y the strength of soft tissues and the b one structure. Since the visual pattern of a face allo ws us to establish its iden tit y it is natural to argue that b ones and soft tissues that actually mak e up a face m ust con tain unique information ab out the face. This t yp e of information can b e quan tied with a biomec hanical prop ert y suc h as elasticit y Using elasticit y directly as a biometric is not curren tly feasible in terms of real time applications due to the computational complexit y of solving nonlinear ill-p osed in v erse problems. F ortunately elasticit y of facial tissues can b e adequately represen ted b y strain pattern, pro vided that certain b oundary conditions are satised. 29

PAGE 39

Strain pattern is unique in the sense that it con tains information that is not presen t in other biometrics. Strain pattern enables us to \see through" the app earance of a face and analyze its iden tit y from b oth anatomical and biomec hanical p ersp ectiv es. It should b e p oin ted out that ph ysics-based tec hniques suc h as deformable mo deling has b een used in facial expression analysis [32, 20]. But their fo cus is on motion and mo del parameters, while w e mak e explicit use of strain pattern. Strain pattern is also stable, b ecause it is related to in tensit y dierence b et w een t w o frames rather than absolute in tensit y v alues of a single frame. If t w o frames are tak en under a similar ligh ting condition, the impact of illumination c hange or mak eup on strain pattern is m uc h less sev ere. W e demonstrate these argumen ts exp erimen tally 5.2 Strain Measuremen t The success of strain pattern as a biometric is dep enden t up on the qualit y of strain image. Although progress has b een made in the dev elopmen t of strain sensors, a non-con tact strain imaging system similar to the range camera is still not a v ailable. Instead, w e emplo y an indirect approac h that deriv es strain image in t w o steps: 1. A motion eld is obtained from t w o video frames that capture an ob ject's deformation. 2. A strain image is then computed from the motion eld. Eac h step has sev eral implemen tation alternativ es that are describ ed b elo w. First, a feature-based metho d can pro duce b etter corresp ondence in the presence of large deformation, but it giv es a sparse motion eld. More imp ortan tly it requires user in terv en tion in case of p o or feature qualit y On the other hand, optical ro w metho d generates a dense motion eld at pixel lev el and is fully automated. Since w e ha v e videos of small in terv al (30 frames p er second), optical ro w is a reasonable c hoice. The second issue is what t yp e of strain to use, 3D strain or 2D strain? It is ideal to ha v e a full 3D strain pattern, b ecause 3D data has b een pro v en to b e useful for face recognition [15]. A t presen t, w e do not ha v e equipmen ts that can capture range image at the same sp eed of a video recorder, whic h is necessary to compute 3D strain. So, our exp erimen ts are restricted to 30

PAGE 40

2D strain images. An in tuitiv e justication for the use of 2D strain o v er 3D strain is presen ted in Section 5.3. Giv en a description of face deformation, w e can use a nite elemen t metho d to compute its strain b y forw ard mo deling, assuming that the Diric hlet condition is satised. Finite elemen t metho d is go o d at handling irregular shap es, but it is not appropriate for pro cessing large amoun t of videos due to its computational cost. An alternativ e is to compute strain b y its denition in con tin uum mec hanics. As a tensor, strain can b e expressed as deriv ativ es of displacemen t v ector and can b e appro ximated b y a nite dierence metho d. Finite dierence metho d is v ery ecien t when carried out on a regular image grid. Because all our data are stored in image format (ra w image, motion image and strain image), w e c hose nite dierence metho d or Delaunay T riangulation 5.3 Strain Computations Using Pro jections In this section, w e presen t exp erimen tal evidence to subtly pro v e that the reasoning in Section 5.1 ab out the uniqueness and stabilit y of strain pattern will hold go o d for strain computed on image plane pro jections to o. While in 3D computations w e deal with absolute strain, in computations based on apparen t displacemen ts w e extract the relativ e strain. A fact that is w ell appreciated in the researc h comm unit y is p oin t that the success of a biometric tec hnology lies in iden tifying discriminativ e c haracteristics capable of dieren tiating individuals while at the same time retaining sucien t features to iden tify a particular sub ject across conditions. W e will analyze strain pattern computed from pro jections from these t w o p ersp ectiv es. Figure 5.1 sho ws the strain patterns of six dieren t individuals for the same expression under same ligh ting conditions using displacemen ts obtained from pro jections. It can b e sub jectiv ely analyzed that the strain pattern v aries among individuals whic h suggests that strain pattern satises the uniqueness criterion. Next, w e examine the stabilit y constrain t and c hec k if computation from pro jections qualies. Figure 5.2 sho ws the strain patterns of the same individual for the same expression under dieren t conditions using displacemen ts obtained from pro jections. 31

PAGE 41

Figure 5.1 Strain Maps Depicting In ter-P erson Strain V ariabilit y Under Same Ligh ting Conditions. (a) Normal Ligh t (b) Lo w Ligh t (c) Shado w Ligh t (d) Camourage F ace Figure 5.2 Strain Maps Depicting In tra-P erson Strain Consistency Under Dieren t Conditions. It can b e observ ed that strain pattern remains comparable across conditions whic h inheren tly conrms that strain pattern conforms to the stabilit y requiremen t to o. Th us, the ab o v e observ ations justify that relativ e strain pattern is sucien t for a biometric study This is more relev an t to the sp ecic case of this study as the exp erimen ts w ere conducted on prole views of the face where the b enets of using depth information, if an y is minimal. 32

PAGE 42

CHAPTER 6 EXPERIMENTS AND RESUL TS 6.1 Video Collection All videos w ere acquired using a Canon Optura 20 digital camcorder at a default sp eed of 30 frames p er second at a spatial resolution of (720 x 480) pixels. Sub jects sit ab out 2 meters in fron t of the camcorder against a plain grey b oard. In addition to normal indo or ligh t, a p oin t-ligh t source w as arranged ab o v e the sub ject's head to create a shado w eect. There w ere 11 acquisition conditions: 1. Normal Ligh t (NL) In this collection, t w o p oin t ligh t sources are placed in fron t of the sub ject so that the sub ject's face is w ell illuminated without casting an y shado ws. 2. Lo w Ligh t (LL) In this collection, the t w o p oin t ligh t sources used in the normal collection is turned o. In addition to this, the regular ligh ting of the ro om w as made dim. 3. Shado w Ligh t (SL) In this collection, the ligh ting conditions of the previous acquisition is main tained. Additionally a p oin t ligh t source ab o v e the sub ject's head is turned on to create a shado w eect. 4. Regular F ace (RF) This refers to the normal conditions where the sub ject pro vides data with his/her natural face. 5. Camouraged F ace (CF) During this collection, the sub ject's face is completely camouraged with a cosmetic face pain t. The camourage do es not follo w an y regular pattern and is randomly done for dieren t sub jects. The purp ose of this collection w as to completely suppress the natural skin color of the sub ject. 33

PAGE 43

6. Mo died F ace (MF) During this collection, a transparen t plastic tap e is glued to the c heek region of a sub ject's face. The purp ose of this collection w as to articially generate tissue abnormalities that em ulate the eect of plastic surgery 7. F ron tal View (FV) In this collection, the sub ject pro vides data facing headon to w ards the camera. 8. Prole View (PV) In this collection, the sub ject sits in a w a y suc h that only the face prole is the visible region. In all our collections, sub jects sho w their left side of the face. 9. Neutral Expression (NE) During this step, the sub ject mak es no expression. The purp ose w as to capture the undeformed state of the facial skin. 10. Op en Mouth (OM) During this step, the sub ject op ens his/her mouth as wide as they can. 11. Smile (SM) During this step, the sub ject expresses his/her natural smile. In all of these collections, the sub jects w ere ask ed not to mo v e their head excessiv ely This w as done to a v oid the optical ro w computation dominated b y the rigid motion of the face. V arious com binations of these conditions result in 14 acquisition sessions. F or example, one video sequence w as tak en under the conditions of: NL+RF+PV+(NE,OM,SM), while another sequence w as tak en under the conditions of: SL+CF+PV+(NE,OM,SM). Sev eral conditions are illustrated in Figure 6.1. A total of 30 sub jects participated in the exp erimen ts. All sub jects app eared on video sequences of Regular F ace. 10 sub jects w ere in v olv ed in sessions of Camouraged F ace (Figure 6.1 (d)). 6.2 Results With Non-Camouraged F aces One of the strengths of strain pattern is that it is less sensitiv e to illumination v ariation, b ecause strain is deriv ed from in tensit y dierence b et w een t w o frames rather than absolute in tensit y v alues. W e carried out t w o tests using Regular F ace with three ligh ting conditions: Normal Ligh t, Lo w Ligh t and Shado w Ligh t (T able 6.1). 34

PAGE 44

(a) Normal Ligh ting (b) Lo w Ligh ting (c) Indo or Shado w (d) Camourage F ace Figure 6.1 Video Acquisition Conditions. T able 6.1 Non-Camouraged F ace Exp erimen ts. Gallery Prob e T est-1 30 sub jects (RF,NL,PV,OM) 30 sub jects (RF,LL,PV,OM) T est-2 30 sub jects (RF,NL,PV,OM) 30 sub jects (RF,SL,PV,OM) T est-1 is to in v estigate whether strain pattern can b e used for face recognition under normal conditions, and T est-2 is to examine its p erformance under more adv erse conditions (Shado w Ligh t). PCA results are plotted as R OC curv es in Figure 6.2. Similarit y score distributions for the same set of tests are sho wn in Figure 6.3 (all v alues are in generic units). A t a false alarm rate of 1.5%, the rst test sho w ed a v erication rate of 84.6%, indicating that strain pattern has the basic discrimination p o w er as a biometric. More imp ortan tly a go o d p erformance w as observ ed in the second test (a 77.3% v erication rate at a 2.2% false alarm rate). It should b e stressed that shado w ligh t mark edly c hanges the lo ok of a face and can cause drastic p erformance degradation of an app earance-based metho d. The impact of shado w ligh t on strain's p erformance, ho w ev er, is m uc h less sev ere. The exp erimen t suggests that strain pattern can b e a v aluable alternativ e to traditional metho ds when the ligh ting condition gets w orse. 6.3 Results With Camouraged F aces Camourage and plastic surgery p ose serious c hallenges to face recognition study T o our kno wledge, no researc h has b een done on ho w to deal with those extreme y et realistic cases. W e 35

PAGE 45

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 True Acceptance Rate (TAR)False Acceptance Rate (FAR)ROC Curves (Experiment 1 and Experiment 2) Experiment 1 Experiment 2 Figure 6.2 R OCs of Non-Camouraged F ace Exp erimen ts. 0 5 10 15 20 25 30 35 0 0.05 0.1 0.15 0.2 Similarity score binsNormalized bin valuesGenuine and Impostor Score Distribution (Experiment 1) Impostor Score Distribution Genuine Score Distribution 0 5 10 15 20 25 30 35 0 0.05 0.1 0.15 0.2 Similarity score binsNormalized bin valuesGenuine and Impostor Score Distribution (Experiment 2) Impostor Score Distribution Genuine Score Distribution Figure 6.3 Gen uine and Imp ostor Scores Distributions of Non-Camouraged F ace Exp erimen ts. use next exp erimen t to demonstrate that strain pattern has the p oten tial to recognize faces inspite of disguise b y mak eup. The exp erimen t consists of t w o tests (T able 6.2). Because 36

PAGE 46

T able 6.2 Camouraged F ace Exp erimen ts. Gallery Prob e T est-1 10 sub jects (RF,NL,PV,OM) 10 sub jects (RF,LL,PV,OM) T est-2 10 sub jects (RF,NL,PV,OM) 10 sub jects (CF,LL,PV,OM) of the small n um b er of camouraged faces (10 sub jects), PCA results are presen ted as score distributions (Figure 6.4) (all v alues are in generic units). 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 0.1 0.2 0.3 0.4 Similarity score binsNormalized bin valuesSimilarity score distribution for regular faces normal lighting and regular faces low lighting Impostor Scores Genuine Scores 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 0.1 0.2 0.3 0.4 Similarity score binsNormalized bin valuesSimilarity score distribution for regular faces normal lighting and camouflage faces low lighting Impostor Scores Genuine Scores Figure 6.4 Similarit y Score Distributions of Camouraged F ace Exp erimen ts. Tw o observ ations can b e made from the results: 1. Similarit y distributions of t w o tests are more or less the same, suggesting that camourage do es not aect the p erformance of strain pattern v ery m uc h; 2. In fact, the second test sho w ed a sligh tly b etter separation b et w een imp oster class and gen uine class. 8 out of 10 camouraged faces w ere successfully iden tied. This is probably attributed to the additional features in camouraged images that lead to more accurate motion and strain images. 37

PAGE 47

6.4 Results With T ap ed F aces In this exp erimen t, w e articially mo dify the facial skin prop ert y b y pasting a plastic tap e to the c heek region of a sub ject. The motiv ation b ehind this exp erimen t w as to c hec k if the algorithm can dieren tiate the c hange in the material b eha vior of the facial skin. As in the previous cases, this exp erimen t had t w o tests (T able 6.3). T able 6.3 T ap ed F ace Exp erimen ts. Gallery Prob e T est-1 15 sub jects (RF,NL,PV,OM) 15 sub jects (RF,LL,PV,OM) T est-2 15 sub jects (RF,NL,PV,OM) 15 sub jects (TF,LL,PV,OM) Because of the small n um b er of camouraged faces (15 sub jects), PCA results are presen ted as score distributions (Figure 6.5) (all v alues are in generic units). 1 2 3 4 5 6 7 8 9 10 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 Similarity score binsNormalized bin values Regular Faces Modified Faces Figure 6.5 Similarit y Score Distributions of Regular and T ap ed F ace Exp erimen ts. The follo wing observ ations can b e made from the results: The strain pattern for the tap ed faces is dieren t from the regular faces. This can b e observ ed from the similarit y scores b et w een mo died faces and regular faces as sho wn in Figure 6.5. 38

PAGE 48

If this exp erimen t w ere to b e designed as a r eje ction exp erimen t, w e could then come up with a distance threshold in the subspace to iden tify surgical c hanges to the skin of a sub ject who claims a particular iden tit y The capabilit y of strain maps to detect cosmetic c hanges to the facial skin can b e of paramoun t imp ortance in forensic applications. 6.5 In v ariance of F acial Strain P attern Although w e used t w o frames of a xed in terv al to compute strain, there is no guaran tee that a sub ject op ened his/her mouth equally across all sequences. The concern is whether strain pattern remains in v arian t to the degree b y whic h the mouth is op ened. T o answ er this question, w e conducted an exp erimen t using shorter sequences subsampled from the original sequence. Giv en a video, Subsequence-1 comprises of one-third of the total frames from the b eginning of the video. Subsequence-2 consists of t w o-third of the total frames from the b eginning of the video. Subsequence-3 is the same as the original video. The gallery con tains 90 strain images, three for eac h sub ject with frame pairs sampled from subsequences. Similarly the prob e con tains 90 strain images from another video. The conditions are Regular F ace with Lo w Ligh t or Normal Ligh t. PCA results are plotted as similarit y distributions in Figures 6.6 and 6.7 (all v alues are in generic units). In this plot, the diagonal elemen ts of the similarit y matrix are remo v ed as they dene the distance of a sequence to itself (zero distance in the subspace). The in tra-class distances are the similarit y scores b et w een a subsequence of a sub ject to the other subsequences of the same sub ject. The in ter-class distances describ e the similarit y scores b et w een a subsequence of a sub ject to those of other sub jects. It is clear that strain patterns of the same sub ject (in tra-class) sho w m uc h stronger similarit y than strains of dieren t sub jects (in ter-class). This can b e partially explained b y the fact that strain is related to tissue elasticit y whic h is a constan t. Unless there is a mark ed c hange of material prop ert y (suc h as those caused b y plastic surgery), strain pattern can b e 39

PAGE 49

0 5 10 15 20 25 30 35 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 Similarity score binsNormalized bin values Inter-class distances Intra-class distances Figure 6.6 Similarit y Distributions of In tra and In ter-Class Strains Under Normal Ligh ting Conditions. 0 5 10 15 20 25 30 35 0 0.02 0.04 0.06 0.08 0.1 0.12 Similarity score binsNormalized bin values Inter-class distances Intra-class distances Figure 6.7 Similarit y Distributions of In tra and In ter-Class Strains Under Lo w Ligh ting Conditions. regarded as in v arian t to ho w m uc h a sub ject op ened his/her mouth. In other w ords, the impact of expression magnitude on the strain pattern as a biometric is minimal. 40

PAGE 50

CHAPTER 7 DISCUSSION AND CONCLUSIONS W e discussed a face recognition metho d that uses elastic information from video. The uniqueness of the metho d lies in the fact that, It utilizes strain pattern rather than ra w image to recognize face. The similarit y measures are th us calculated b et w een strain images instead of ra w in tensit y v alues. Because of its close tie with tissue prop ert y strain pattern adds a new dimension to our abilit y to c haracterize a face. The computational strategy is based on biomec hanics, and hence is ph ysically sound. It w orks w ell under unfa v orable conditions (shado w ligh t and mak eup). It is no w clear that displacemen ts are the essen tial input to the algorithm. F or the metho d to w ork, the input video sequence of facial expression should pro vide sucien t displacemen t in terms of non-rigid motion. The approac h do es not apply to video sequences where no non-rigid motion can b e observ ed whic h is needed to quan tify the c haracteristics (strain computation) of the deformed facial skin. In addition to the ab o v e failure case, p o or displacemen t estimates resulting from incorrect optical ro w computation will lead to erroneous strain maps that ev en tually inruence recognition accuracy Ho w ev er, this is not a problem inheren t to the recognition algorithm and dev elopmen ts to extraction algorithms will solv e this issue. In the computations of strain, the metho d uses displacemen ts from pro jections instead of absolute 3D displacemen ts. W e exp erimen tally justied that relativ e strain is adequate for iden tication purp oses. This approac h assumes more p ertinence in ligh t of the fact that curren t range sensoring equipmen ts cannot capture data at the sp eed of regular 2D camcorders. 41

PAGE 51

With adv ances in 3D acquisition tec hnology relativ e strain can b e replaced with absolute strain whic h w e b eliev e will impro v e the p erformance. Similarly the motiv ation b ehind using strain magnitude, instead of strain comp onen ts, is the fact that it neatly gels in to the traditional face recognition paradigm where 1D PCA analysis using in tensit y as the only feature is p opular. Ho w ev er, using strain comp onen ts subsequen tly follo w ed b y a m ulti-dimensional PCA analysis can also b e done and the ndings of this w ork will still b e v alid. Another impro v emen t to the metho d is to incorp orate the strain induced due to the b ending of the skin (shear strain). The algorithm in v estigates the strain resulting from the stretc hing of skin (normal strain) only Com bining shear strain to the mo del will completely describ e the deformed state of the facial skin for a giv en facial expression. W e w ould lik e to emphasize that the prop osed metho d is not in tended to replace or outp erform curren t metho ds. Hence, this thesis do es not attempt an y comparativ e study with existing metho ds. W e prop ose that the metho d w ould b e b est utilized when in tegrated with other biometrics. Though the p erformance on the most basic exp erimen t (Exp erimen t 1) is lesser than what curren t metho ds can ac hiev e, the fact that the algorithm gracefully degrades in terms of p erformance under drastic conditions (shado w ligh t and camourage) suggests that the metho d most certainly has the promise to supplemen t curren t tec hnology under suc h adv erse op erational conditions. In conclusion, it can b e said that the purp ose of this eort w as to explore the p ossibilit y of using strain pattern as a classication feature. Results from exp erimen ts sho w that strain maps can indeed b e used as a soft biometric. With that in mind, the in ten tion of this thesis is to establish a baseline p erformance using suc h an approac h. Impro v emen ts suc h as replacing or impro ving robustness of certain comp onen ts will denitely increase the p erformance of the system. Some of these enhancemen ts w ere earlier discussed in this c hapter. 42

PAGE 52

REFERENCES [1] Mac hine Readable T ra v el Do cumen ts (MR TD). Online: h ttp://www.icao.in t/mrtd/o v erview/o v erview.cfm. [2] X. L. 0002 and T. Chen. Video-Based F ace Recognition Using Adaptiv e Hidden Mark o v Mo dels. In Pr o c e e dings of the International Confer enc e on Computer Vision and Pattern R e c o gnition v olume 1, pages 340{345, 2003. [3] G. Aggarw al, A. R. Cho wdh ury and R. Chellappa. A System Iden tication Approac h for Video-based F ace Recognition. In Pr o c e e dings of the International Confer enc e on Pattern R e c o gnition v olume 4, pages 175{178, 2004. [4] P Anandan. A computational framew ork and an algorithm for the measuremen t of visual motion. Internation Journal of Computer Vision 2:283{310, 1989. [5] F. R. Bac h and M. I. Jordan. Kernel Indep enden t Comp onen t Analysis. Journal of Machinc e L e arning R ese ar ch 3:1{48, 2003. [6] M. S. Bartlett, J. R. Mo v ellan, and T. J. Sejno wski. F ace recognition b y indep enden t comp onen t analysis. IEEE T r ansactions on Neur al Networks 13(6):1450{1464, 2002. [7] P N. Belh umeur, J. P Hespanha, and D. J. Kriegman. Eigenfaces vs. Fisherfaces: Recognition Using Class Sp ecic Linear Pro jection. IEEE T r ansactions on Pattern A nalysis and Machine Intel ligenc e 19(7):711{720, 1997. [8] R. Bev eridge and B. Drap er. Ev aluation of F ace Recognition Algorithms. Online: h ttp://www.cs.colostate.edu/ev alfacerec/index.h tml. [9] M. J. Blac k. R obust Incr emental Optic al Flow PhD thesis, Y ale Univ ersit y 1992. [10] M. J. Blac k and P Anandan. A framew ork for the robust estimation of optical ro w. In Pr o c e e dings of International Confer enc e on Computer Vision pages 231{236, 1993. [11] M. J. Blac k and P Anandan. The robust estimation of m ultiple motions: P arametric and piecewise-smo oth ro w elds. Computer Vision Image Understanding 63(1):75{104, 1996. [12] V. Blanz and T. V etter. A morphable mo del for the syn thesis of 3D faces. In Pr o c e e dings of the 26th A nnual Confer enc e on Computer Gr aphics and Inter active T e chniques pages 187{194, 1999. [13] V. Blanz and T. V etter. F ace Recognition Based on Fitting a 3D Morphable Mo del. IEEE T r ansactions on Pattern A nalysis and Machinc e Intel ligenc e 25(9):1063{1074, 2003. 43

PAGE 53

[14] R. M. Bolle, J. H. Connell, S. P ank an ti, N. K. Ratha, and A. W. Senior, editors. Guide to Biometrics Springer, 2004. [15] K. W. Bo wy er, K. Chang, and P J. Flynn. A surv ey of approac hes and c hallenges in 3d and m ulti-mo dal 3d + 2d face recognition. Computer Vision and Image Understanding 101(1):1{15, 2006. [16] A. M. Bronstein, M. M. Bronstein, and R. Kimmel. Expression-In v arian t 3D F ace Recognition. In Pr o c e e dings of the International Confer enc e on A udio and Vide o-Base d Biometric Person A uthentic ation pages 62{69, 2003. [17] K. Chang, K. W. Bo wy er, S. Sark ar, and B. Victor. Comparison and com bination of ear and face images in app earance-based biometrics. IEEE T r ansactions on Pattern A nalysis and Machine Intel ligenc e 25(9):1160{1165, 2003. [18] R. Chellappa, C. L. Wilson, and S. Sirohey Human and mac hine recognition of faces: A surv ey Pr o c e e dings of the IEEE 83(5):705{740, 1995. [19] T. F. Co otes, K. N. W alk er, and C. J. T a ylor. View-Based Activ e App earance Mo dels. In Pr o c e e dings of the International Confer enc e on A utomatic F ac e and Gestur e R e c o gnition pages 227{232, 2000. [20] I. A. Essa and A. P P en tland. Co ding, analysis, in terpretation, and recognition of facial expressions. IEEE T r ansactions on Pattern A nalysis and Machine Intel ligenc e 19(7):757{763, 1997. [21] D. N. F enner. Engine ering Str ess A nalysis A Finite Element Appr o ach With F OR TRAN 77 Softwar e J. Wiley and Sons, 1987. [22] K. F ukunaga. Intr o duction to statistic al p attern r e c o gnition (2nd e d.) Academic Press Professional, Inc., 1990. [23] F. Glazer. Hier ar chic al motion dete ction PhD thesis, Univ ersit y of Massac h ussets, 1987. [24] D. O. Goro dnic h y On Imp ortance of Nose for F ace T rac king. In Pr o c e e dings of the International Confer enc e on A utomatic F ac e and Gestur e R e c o gnition pages 188{196, 2002. [25] P Grother and P J. Phillips. Mo dels of large p opulation recognition p erformance. In Pr o c e e dings of International Confer enc e on Computer Vision and Pattern R e c o gnition v olume 2, pages 68{75, 2004. [26] G. Guo, S. Z. Li, and K. L. Chan. F ace Recognition b y Supp ort V ector Mac hines. In Pr o c e e dings of International Confer enc e on A utomatic F ac e and Gestur e R e c o gnition pages 196{201, 2000. [27] A. Hadid and M. Pietik ainen. An exp erimen tal in v estigation ab out the in tegration of facial dynamics in video-based face recognition. Ele ctr onic L etters on Computer Vision and Image A nalysis 5(1):1{13, 2005. [28] J. A. Hanley and B. J. McNeil. The meaning and use of the area under a receiv er op erating c haracteristic (ro c) curv e. R adiolo gy 143:29{36, 1982. 44

PAGE 54

[29] B. Heisele, P Ho, and T. P oggio. F ace Recognition with Supp ort V ector Mac hines: Global v ersus Comp onen t-based Approac h. In Pr o c e e dings of the International Confer enc e on Computer Vision pages 688{694, 2001. [30] R. Hietmey er. Biometric iden tication promises fast and secure pro cessing of airline passengers. The International Civil A viation Or ganization Journal 55(9):10{11, 2000. [31] B. K. P Horn and B. G. Sc h unc k. Determining optical ro w. A rticial Intel ligenc e 17:185{204, 1981. [32] R. Jain and M. Shah. Motion-Base d R e c o gnition Klu w er Academic Publishers, 1997. [33] H. Jereys and B. S. Jereys. Metho ds of Mathematic al Physics Cam bridge Univ ersit y Press, third edition, 1988. [34] A. Kadyro v and M. P etrou. The T race T ransform and Its Applications. IEEE T r ansactions on Pattern A nalysis and Machine Intel ligenc e 23(8):811{828, 2001. [35] T. Kanade. Pictur e Pr o c essing by Computer Complex and R e c o gnition of Human F ac es PhD thesis, Ky oto Univ ersit y 1973. [36] S. J. Kundu. F acial strain maps as a biometric source. Master's thesis, Univ ersit y of South Florida, 2005. [37] K. C. Lee, J. Ho, M. H. Y ang, and D. J. Kriegman. Video-Based F ace Recognition Using Probabilistic App earance Manifolds. In Pr o c e e dings of the International Confer enc e on Computer Vision and Pattern R e c o gnition v olume 1, pages 313{320, 2003. [38] J. Lemaitre and J. L. Chab o c he, editors. Me chanics of solid materials Cam bridge Univ ersit y Press, 1990. [39] B. Li and R. Chellappa. F ace v erication through trac king facial features. Journal of the Optic al So ciety of A meric a 18:2969{2981, 2001. [40] S. Z. Li and A. K. Jain, editors. Handb o ok of F ac e R e c o gnition Springer, 2005. [41] Y. Li. Dynamic F ac e Mo dels: Construction and Applic ations PhD thesis, Univ ersit y of London, 2001. [42] C. Liu and H. W ec hsler. A Unied Ba y esian F ramew ork for F ace Recognition. In Pr o c e e dings of International Confer enc e on Image Pr o c essing v olume 1, pages 151{155, 1998. [43] C. Liu and H. W ec hsler. Comparativ e assessmen t of Indep enden t Comp onen t Analysis (ICA) for face recognition. In Pr o c e e dings of International Confer enc e on A udio and Vide o-b ase d Biometric Person A uthentic ation pages 211{216, 1999. [44] C. Liu and H. W ec hsler. Ev olutionary Pursuit and Its Application to F ace Recognition. IEEE T r ansactions on Pattern A nalysis and Machine Intel ligenc e 22(6):570{582, 2000. [45] J. Lu, K. Plataniotis, and A. V enetsanop oulos. F ace recognition using LD A based algorithms. IEEE T r ansactions on Neur al Networks 14(1):195{200, 2003. [46] A. M. Mart inez and A. C. Kak. PCA v ersus LD A. IEEE T r ansactions on Pattern A nalysis and Machine Intel ligenc e 23(2):228{233, 2001. 45

PAGE 55

[47] C. E. Metz. Basic principles of ro c analysis. Seminars in Nucle ar Me dicine 8(4):283{298, 1978. [48] S. Moa v eni. Finite Element A nalysis Pren tice Hall, 1999. [49] B. Moghaddam, T. Jebara, and A. P en tland. Ba y esian face recognition. Pattern R e c o gnition 33(11):1771{1782, 2000. [50] H. Mo on and P J. Phillips. Computational and p erformance asp ects of p ca-based face recognition algorithms. Per c eption 30:301{321, 2001. [51] A. J. O'T o ole, D. A. Roark, and H. Ab di. Recognizing mo ving faces: A psyc hological and neural syn thesis. T r ends in Co gnitive Scienc e 6(6):261{266, 2002. [52] A. P en tland, B. Moghaddam, and T. Starner. View-based and mo dular eigenspaces for face recognition. In Pr o c e e dings of IEEE Confer enc e on Computer Vision and Pattern R e c o gnition pages 84{91, 1994. [53] P J. Phillips, P J. Flynn, W. T. Scruggs, K. W. Bo wy er, J. Chang, K. Homan, J. Marques, J. Min, and W. J. W orek. Ov erview of the face recognition grand c hallenge. In Pr o c e e dings of International Confer enc e on Computer Vision and Pattern R e c o gnition pages 947{954, 2005. [54] P J. Phillips, P Grother, R. J. Mic heals, D. M. Blac kburn, E. T abassi, and J. M. Bone. F ace Recognition V endor T est (FR VT) 2002: Ov erview and Summary. Online: h ttp://www.frvt.org/FR VT2002/do cumen ts.h tm. [55] S. A. Rizvi, P J. Phillips, and H. Mo on. The FERET V erication T esting Proto col for F ace Recognition Algorithms. In Pr o c e e dings of International Confer enc e on A utomatic F ac e and Gestur e R e c o gnition pages 48{53, 1998. [56] S. Soatto, G. Doretto, and Y. W u. Dynamic T extures. In Pr o c e e dings of International Confer enc e on Computer Vision v olume 2, pages 439{446, 2001. [57] M. Sonk a, V. Hla v ac, and R. Bo yle. Image Pr o c essing: A nalysis and Machine Vision O'Reilly 1999. [58] S. Srisuk, M. P etrou, W. Kurutac h, and A. Kadyro v. A face authen tication system using the trace transform. Pattern A nalysis and Applic ations 8(1):50{61, 2005. [59] D. T erzop oulos, J. Platt, A. Barr, and K. Fleisc her. Elastically deformable mo dels. In Pr o c e e dings of the A nnual Confer enc e on Computer Gr aphics and Inter active T e chniques pages 205{214, 1987. [60] D. T erzop oulos and K. W aters. Analysis and syn thesis of facial image sequences using ph ysical and anatomical mo dels. IEEE T r ansactions Pattern A nalysis and Machine Intel ligenc e 15:569{579, 1993. [61] M. T urk and A. P P en tland. Eigenfaces for recognition. Journal of Co gnitive Neur oscienc e 3(1):71{86, 1991. [62] R. W ang and B. Bhan u. Learning mo dels for predicting recognition p erformance. In Pr o c e e dings of International Confer enc e on Computer Vision pages 1613{1618, 2005. 46

PAGE 56

[63] J. W a yman, A. K. Jain, D. Maltoni, and D. Maio, editors. Biometric Systems T e chnolo gy, Design and Performanc e Evaluation Springer, 2005. [64] L. Wisk ott, J. M. F ellous, N. Kr uger, and C. Malsburg. F ace Recognition b y Elastic Bunc h Graph Matc hing. IEEE T r ansactions on Pattern A nalysis and Machinc e Intel ligenc e 19(7):775{779, 1997. [65] M. H. Y ang. Kernel Eigenfaces vs. Kernel Fisherfaces: F ace Recognition Using Kernel Metho ds. In Pr o c e e dings of the International Confer enc e on A utomatic F ac e and Gestur e R e c o gnition pages 215{220, 2002. [66] Y. Zhang, S. J. Kundu, D. B. Goldgof, S. Sark ar, and L. V. Tsap. Elastic face, an anatom y-based biometrics b ey ond visible cue. In Pr o c e e dings of International Confer enc e on Pattern R e c o gnition v olume 2, pages 19{22, 2004. [67] W. Zhao, R. Chellappa, and A. Krishnasw am y Discriminan t Analysis of Principal Comp onen ts for F ace Recognition. In Pr o c e e dings of the International Confer enc e on A utomatic F ac e and Gestur e R e c o gnition pages 336{341, 1998. [68] W. Zhao, R. Chellappa, P J. Phillips, and A. Rosenfeld. F ace recognition: A literature surv ey. A CM Computing Surveys 35(4):399{458, 2003. [69] S. K. Zhou and R. Chellappa. Probabilistic Human Recognition from Video. In Pr o c e e dings of the Eur op e an Confer enc e on Computer Vision v olume 3, pages 681{697, 2002. [70] S. K. Zhou and R. Chellappa. Multiple-Exemplar Discriminan t Analysis for F ace Recognition. In Pr o c e e dings of the International Confer enc e on Pattern R e c o gnition v olume 4, pages 191{194, 2004. [71] S. K. Zhou, R. Chellappa, and B. Moghaddam. In tra-P ersonal Kernel Space for F ace Recognition. In Pr o c e e dings of the International Confer enc e on A utomatic F ac e and Gestur e R e c o gnition pages 235{240, 2004. [72] S. K. Zhou, V. Kr uger, and R. Chellappa. F ace Recognition from Video: A CONDENSA TION Approac h. In Pr o c e e dings of International Confer enc e on A utomatic F ac e and Gestur e R e c o gnition pages 221{228, 2002. [73] O. C. Zienkiewicz and R. L. T a ylor. The nite element metho d v olume 1. McGra w-Hill, 1989. 47


xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam Ka
controlfield tag 001 001795309
003 fts
005 20070703142814.0
006 m||||e|||d||||||||
007 cr mnu|||uuuuu
008 070703s2006 flu sbm 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E14-SFE0001560
040
FHM
c FHM
035
(OCoLC)151065438
049
FHMM
090
QA76 (ONLINE)
1 100
Manohar, Vasant.
0 245
Video-based person identification using facial strain maps as a biometric
h [electronic resource] /
by Vasant Manohar.
260
[Tampa, Fla] :
b University of South Florida,
2006.
3 520
ABSTRACT: Research on video-based face recognition has started getting increased attention in the past few years. Algorithms developed for video have an advantage from the availability of plentitude of frames in videos to extract information from. Despite this fact, most research in this direction has limited the scope of the problem to the application of still image-based approaches to some selected frames on which 2D algorithms are expected to perform well. It can be realized that such an approach only uses the spatial information contained in video and does not incorporate the temporal structure.Only recently has the intelligence community begun to approach the problem in this direction. Video-based face recognition algorithms in the last couple of years attempt to simultaneously use the spatial and temporal information for the recognition of moving faces. A new face recognition method that falls into the category of algorithms that adopt spatio-temporal representation and utilizes dynamic information extracted from video is presented. The method was designed based on the hypothesis that the strain pattern exhibited during facial expression provides a unique "fingerprint" for recognition. First, a dense motion field is obtained with an optical flow algorithm. A strain pattern is then derived from the motion field. In experiments with 30 subjects, results indicate that strain pattern is an useful biometric, especially when dealing with extreme conditions such as shadow light and face camouflage, for which conventional face recognition methods are expected to fail. The ability to characterize the face using the elastic properties of facial skin opens up newer avenues to the face recognition community in the context of modeling a face using features beyond visible cues.
502
Thesis (M.A.)--University of South Florida, 2006.
504
Includes bibliographical references.
516
Text (Electronic thesis) in PDF format.
538
System requirements: World Wide Web browser and PDF reader.
Mode of access: World Wide Web.
500
Title from PDF of title page.
Document formatted into pages; contains 47 pages.
590
Adviser: Dmitry B. Goldgof, Ph.D.
653
Face recognition.
Face dynamics.
Non-rigid motion.
Optical flow.
Elasticity.
690
Dissertations, Academic
z USF
x Computer Science
Masters.
773
t USF Electronic Theses and Dissertations.
4 856
u http://digital.lib.usf.edu/?e14.1560