Eye blinking detection

This application detect eye blinking by calculating optical flow in area of eyes inside stabilized picture
of head. When the user have closed eyes for more than 5 seconds, the application starts to beep to
wake him up. This can be used inside vehicles to prevent driver sleeping behind the steering wheel.
For eye blinking detection is used generic webcam on computer.

First approach
The first approach to this problem was to detect face by using HAAR Cascade Classifier
(haarcascade_frontalface_alt2 ). Then the program would take eyes position (static, defined by
approximate position where the eyes should be or dynamic also by using HAAR Cascade).
This approach was not working well (or at all) because Optical Flow (calculated by Farenback algorithm)
was not stable. The Cascade Classifier was looking for face on every new frame and it was still changing
its size and position the image was shaky.

Image 1. Optical flow calculated by Farenback algorithm

As seen on figure 1 this is how the blink was calculated (visible just pixels that moved more than defined
length, calculated by Euclidean distance). This looks good, but problem was that the eyes was detected
as moving all the time even when I was looking without movement on the camera.

Image 2. Optical flow calculated by Farenback algorithm

Second approach
When I wanted to stabilize the image, I wanted to found darkest area of the eye image, and the other
similar (dark) pixels to redraw as absolutely black (0h). Later I wanted to show just these black pixels
and see just the one black circle. But this method is really unstable when there is not a good lightning
and there are darker shadows than the actual eye.

Final approach stabilization

This method is based also on HAAR Cascade Classifier to found face. But when the face is found, after
this first captured frame of face is activated stabilization. This first face is saved as rectangle and then
on the second frame I will calculate global optical flow on this two images of faces. When the face is
moved to the right by 5 pixels, first of all I will look at every pixel movement, add it to variable X and
later, when there are for example 100 pixels and movement by 5 pixels, value X will be 500. Now I will
divide this value by pixels and I get the movement 5 pixels to the right.
void calcFlow(const Mat& flow, Mat& cflowmap, int step, int &globalMovementX, int &globalMovementY)
{
int localMovementX = 0;
int localMovementY = 0;
for (int y = 0; y < cflowmap.rows; y += step)
{
for (int x = 0; x < cflowmap.cols; x += step)
{
const Point2f& fxy = flow.at<Point2f>(y, x);
localMovementX = localMovementX + fxy.x;
localMovementY = localMovementY + fxy.y;
}
}
globalMovementX = (localMovementX / (cflowmap.cols * cflowmap.rows))*2;
globalMovementY = (localMovementY / (cflowmap.rows * cflowmap.cols))*2;
}

Code 1. Calculating flow of head movement

Problem was that this calculated value was usually smaller than the real movement, so I multiplied this
value by 2 so the movement would be more accurate. Also when this method is used, a lot of
movement can move this rectangle of image where there is no face at all or is just moved that the eyes
are not where they should be (and Im detecting nose and ear). Because of this reason there is
implemented calibration, which is used every 100 frames.
Now I had face that is not shaky when the real face in front of the camera was holding still. There I
selected eyes by static values (estimated position of the eyes on the face). On each separate eye was
used optical flow calculation, so application could detect blinking for each eye separately. This
detection is not as with face, where is calculated real movement (for example real 5 pixels) but is used
absolute movement of the eye (for example 500 pixels moved by 1 pixel to the right so the output is
500).

void detectBlink(Mat &matEyePrevious, Mat &matEyeCurrent, String eye, bool &eyeOpen, int
&blinkNumber, clock_t &closeTime) {
Mat leftFlow, leftCflow;
calcOpticalFlowFarneback(matEyePrevious, matEyeCurrent, leftFlow, 0.5, 3, 15, 3, 5, 1.2, 0);
cvtColor(matEyePrevious, leftCflow, CV_GRAY2BGR);
int movementX, movementY;
calcFlowEyes(leftFlow, leftCflow, 1, movementX, movementY);

if (movementY == 0) {
return;
}
if (movementY > 0 && eyeOpen) {

//eye is now closed

closeTime = clock();

//time of eye closure

eyeOpen = false;
blinkNumber = blinkNumber + 1;

//increment blink count

}
else if (movementY < 0 && !eyeOpen){
eyeOpen = true;
}

//eye is now open

Code 2. Detecting if eye is open or not

Image 3. Top left and right eye on current frame, bottom previous frame

Now, when the variables for eyes (if each eye is opened or closed) are set, we can show it on the Result
window.

Image 4. GUI of the application

On final Result image is for each eye showing number of blinks, time of closed eye and color detection
if the eye is opened or closed (green for opened, red for closed).
Final results are publicly visible on YouTube:

https://youtu.be/v0UNbB_vyrM