Eigenfaces with OpenCV 2

Time to try and make this blog live up to it's domain name. I recently started playing around with face recognition. It seems there are fairly few Open Source libraries for face recognition. What I found after some searching online was:

• libface, Uses OpenCV's Eigenfaces.
• The Colorado State University FR Algorithm Evaluation suite. It includes some source code and provides the following algorithms: 
• Eigenfaces.
• Eigenfaces with Linear Discriminant Analysis.
• Bayesian Intrapersonal/Extrapersonal Classifier.
• Elastic Bunch Graph Matching.
• OpenCV, provides Eigenfaces.

Since I knew nothing about face recognition I began my quest by reading a pile of CS papers* and (inevitably) settled on starting with "Eigenfaces" which is the simplest of the holistic face recognition methods. There are several tutorials on how to implement Eigenfaces with OpenCV but most of the ones I found use the C AP and not the improved C++ API. On top of that the the C++ API was redesigned which means older tutorials that used the C API cannot always be quickly rewritten.

Some years ago Servo Magazine  published a fine series of OpenCV articles (made available online by cognotics.com). The series included two installments on Eigenfaces. The first installment did a good job of explaining Eigenfaces and Principal Component Analysis.  The second installment presented a simple implementation. I set myself the goal of rewriting this code in the new OpenCV C++ API and testing it. I recommend reading the introductory part of the Servo Magazine article and the Eigenfaces paper by Turk & Pentland, "Eigenfaces for Recognition" before proceeding. The paper is quite well written and relatively easy to understand.

* The face-rec.org site has a papers list that is a good place to start.

Eigenfaces in the OpenCV C++ API

In the OpenCV C++ API Eigenfaces is packaged into the PCA object  which in turn is neatly tucked away under array operations. This class can be used to easily implement Eigenfaces and is simpler to use than the old C API:

• PCA(const Mat& data, const Mat& mean, int flags, int maxComponents=0);
• data - The set of training faces (Γ = {Γ₁, Γ₂ ... Γ_M}).
• mean - You can provide a pre-calculated average face (Ψ) of the set of training faces (Γ) or an empty matrix in which case the average face is calculated from the set of training faces you provided.
• components - How many principal components to retain.
• Mat project(const Mat& vec) const;
• Projects a face vector into face-space: (ω_k=u_k^T(Γ-Ψ)).
• Mat eigenvectors;
• The eigenfaces (ω) which are the training faces (Γ) after projection into face-space.
• Mat eigenvalues;
• The eigenvectors (u) used to project faces to and from face-space.
• Mat mean;
• The average face (Ψ) .

The code

I wrote a simple program (not my cleanest code ever) that does more or less the same as the test program in the Servo Magazie tutorial did and it uses the same Cambridge University Computer Lab face database (preview here). The program was written for and compiled on a MacBook so Windows and Linux users may have to do a little tweaking of the includes and the makefile before the code compiles. The test program has several options:

   eigenfacetest [-vs] [-t training image list] [-r test images list]
      -v: Verbose mode.
      -s: The projected face vectors can be retrieved and rendered
          as images. This option shows the average face and a 
          few of the eigenfaces.
      -r: A list of test files with faces to try and recognize.
      -t: A List of training files with faces for the recognizer.

It expects two files: train_original.txt and test_original.txt which are the files that came with the original Servo Magazine article but you can specify your own test and training file sets with the -t and -r options.

The average face on the left and three eigenfaces.

A little experiment

Running the test sets from the original article gave me an encouraging 100% recognition rate which was way too good to be true. I decided to run the program on the entire set of test images using the first image for each individual as the training image and the rest as test images. The experiment was then repeated with two training images and eight test images etc.

Train/Test	1/9	2/8	3/7	4/6
Correct	66.4%	75.6%	78.6%	85%

The results were not quite as good as they were using the original Servo Magazine test-sets but rather consistent with what I thought they would be after reading all those CS papers i.e, that he accuracy would be between 60 and 70%. The accuracy goes up as the size of the training set increases. The accuracy with only one training image per subject is what one would expect from a simple Eigenfaces implementation and the fact that the Cambridge University face database, although small, contains a fair number of non frontal faces. The CU data set is, however, well lit and one would expect that variations in lighting conditions would cause the recognition rate to go down. To improve the results one might try further automatic preprocessing with eye detection, rotation and scaling.


Downloads

• The source code, it's makefile and the test set lists can be downloaded here. 
• You'll have to download the face database your self, here.