INTRODUCTION TO INNOVATIVE PROJECTS

FACE RECOGNITION (OPENCV)

NAME REG. NO
Karthik Vajja 17BCE0462
Janani. J 17BCE0207
G.Pavan Varma 17BCE0270
Bapanna 17BCE0376
Lokavya Sharma 17BCE0562
Tushar Jaiswal 17BCE0591
Naga Deepak Bhushan 17BCE0791
V. Mohan Kishore 17BCE0501
JeyaVarshiniBharath 17BCE0200
EswarPrathap 17BCE0380

Submitted To-

Prof. Veluraja. k
PHY: 1999Introduction to Innovative Projects
J-Component
School of Computer Science and Engineering
VIT University
Vellore, Tamil Nadu.
Problem Definition
A general statement of face recognition problem can be formulated as follows:
Given still or video images of a scene, identify or verify one or more persons in the
scene using a stored database of faces. Available collateral information maybe as
race , age, gender, facial expression and speech may be used in narrowing the
search (enhancing recognition)The solution of the problem involves segmentation
of faces (face detection) from cluttered scenes,feature extraction from the face
region,recognition or verification. In identification problems ,the input to the
system is an unknown face, and the system reports back the decided identity from a
database of known individuals, whereas in verification problems , the system needs
to confirm or reject the claimed identity of the input face.So we can say that A
facial recognition system is a technology capable of identifying or verifying a
person from a digital image or a video frame from a video source. There are
multiple methods in which facial recognition systems work, but in general, they
work by comparing selected facial features from given image with faces within a
database. Some face recognition algorithms identify facial features by extracting
landmarks, or features, from an image of the subject's face. For example, an
algorithm may analyze the relative position, size, and/or shape of the eyes, nose,
cheekbones, and jaw. These features are then used to search for other images with
matching features.
Other algorithms normalize a gallery of face images and then compress the face
data, only saving the data in the image that is useful for face recognition. A probe
image is then compared with the face data.[14] One of the earliest successful
systems[15] is based on template matching techniques[16] applied to a set of
salient facial features, providing a sort of compressed face representation.
Recognition algorithms can be divided into two main approaches, geometric,
which looks at distinguishing features, or photometric, which is a statistical
approach that distills an image into values and compares the values with templates
to eliminate variances. Some classify these algorithms into two broad categories:
holistic and feature-based models. The former attempts to recognize the face in its
entirety while the feature-based subdivide into components such as according to
features and analyze each as well as its spatial location with respect to other
features.
ABSTRACT
In this face detection we are using different libraries in python like open cv,pickle
and numpy. We can execute this through the command propmpt or pycharm.These
libraries have different functionalities in code written. Open cv in python can be
used for face detection and face recognition.Face detection has attracted immense
attention because it has many applications in computer vision communication,
automatic control system and even in day to day life. Face detection is a method to
detect a face from an image which have several attributes in that image.This
technique also includes saliency prediction. To do the following work we have
used Python. The most used library that is used in this project is OpenCv. The
following libraries were also used in the completion of the project: Numpy, Pillow
and Pickle.

INTRODUCTION
A human face reveals a great deal of information to a perceiver. It can tell about
mood and intention and attentiveness, but it can also serve to identify a person. Of
course, a person can be identified by other means than the face. Voice, body shape,
gait or even clothing may all establish identity in circumstances where facial detail
may not be available. Nevertheless, a face is the most distinctive and widely used
key to a person’s identity, and the loss of ability to recognize faces experienced by
some neurological (prosopagnosic) patients has a profound effect on their lives.

In the past few years, face recognition owned significant consideration and
appreciated as one of the most promising applications in the field of image
analysis. Face detection can consider a substantial part of face recognition
operations. According to its strength to focus computational resources on the
section of an image holding a face. The method of face detection in pictures is
complicated because of variability present across human faces such as pose,
expression, position and orientation, skin colour, the presence of glasses or facial
hair, differences in camera gain, lighting conditions, and image resolution.

Object detection is one of the computer technologies, which connected to the

image processing and computer vision and it interacts with detecting instances of
an object such as human faces, building, tree, car, etc. The primary aim of face
detection algorithms is to determine whether there is any face in an image or not.
In recent times, a lot of study work proposed in the field of Face Recognition and
Face Detection to make it more advanced and accurate, but it makes a revolution in
this field when Viola-Jones comes with its Real-Time Face Detector, which is
capable of detecting the faces in real-time with high accuracy.

Face Detection is the first and essential step for face recognition, and it is used to
detect faces in the images. It is a part of object detection and can use in many areas
such as security, bio-metrics, law enforcement, entertainment, personal safety, etc.

It is used to detect faces in real time for surveillance and tracking of person or
objects. It is widely used in cameras to identify multiple appearances in the frame
Ex- Mobile cameras and DSLR’s. Facebook is also using face detection algorithm
to detect faces in the images and recognise them.

LITERATURE SURVEY
Face detection is a computer technology that determines the location and size of
human face in arbitrary (digital) image. The facial features are detected and any
other objects like trees, buildings and bodies etc are ignored from the digital image.
It can be regarded as a specific case of object-class detection, where the task is
finding the location and sizes of all objects in an image that belong to a given class.
Face detection, can be regarded as a more general case of face localization. In face
localization, the task is to find the locations and sizes of a known number of faces
(usually one). Basically there are two types of approaches to detect facial part in
the given image i.e. feature base and image base approach. Feature base approach
tries to extract features of the image and match it against the knowledge of the face
features. While image base approach tries to get best match between training and
testing images. We are using feature based approach for the detection and
recognition of images.

Knowledge-Based:-

The knowledge-based method depends on the set of rules, and it is based on human
knowledge to detect the faces. Ex- A face must have a nose, eyes, and mouth
within certain distances and positions with each other. The big problem with these
methods is the difficulty in building an appropriate set of rules. There could be
many false positive if the rules were too general or too detailed. This approach
alone is insufficient and unable to find many faces in multiple images.
Feature-Based:-

The feature-based method is to locate faces by extracting structural features of the

face. It is first trained as a classifier and then used to differentiate between facial
and non-facial regions. The idea is to overcome the limits of our instinctive
knowledge of faces. This approach divided into several steps and even photos with
many faces they report a success rate of 94%.

.Template Matching:-

Template Matching method uses pre-defined or parameterised face templates to

locate or detect the faces by the correlation between the templates and input
images. Ex- a human face can be divided into eyes, face contour, nose, and mouth.
Also, a face model can be built by edges just by using edge detection method. This
approach is simple to implement, but it is inadequate for face detection. However,
deformable templates have been proposed to deal with these problems.

Appearance-Based:-

The appearance-based method depends on a set of delegate training face images to

find out face models. The appearance-based approach is better than other ways of
performance. In general appearance-based method rely on techniques from
statistical analysis and machine learning to find the relevant characteristics of face
images. This method also used in feature extraction for face recognition.

METHODOLOGY
Here we basically took steps forward little by little going through how to work
with the opencv and finding the optimum solutions. We have used python to work
out with it and find the code. First we work out with Opencv that requires
Numpy,pickle also. We first try to work with pictures and how to make them
grayscale. Then we tried to draw on the pictures and put text on them. In this
project, we are going to use the haar cascades first to detect the face in the picture.
These haar cascades are already available with the python package as .xml file
extension. Then we made into yml extension file and trained our program to
recognize the faces of the people and then name them accordingly.

HAAR CASCADES
Haar features are similar to these convolution kernels which is used to detect the
presence of certain feature in an image. Each feature will give a single value result
which is calculated by subtracting the sum of the pixels under white rectangle from
that of the pixels under black rectangle. This algorithm uses a 24X24 window as
the base window size to start evaluating these features in any image. If we consider
all the features of haar features like position, scale and type we will end up
calculating 160000+ features in window.

Object Detection using Haar feature-based cascade classifiers is an effective object

detection method proposed by Paul Viola and Michael Jones in their paper, "Rapid
Object Detection using a Boosted Cascade of Simple Features" in 2001. It is a
machine learning based approach where a cascade function is trained from a lot of
positive and negative images. It is then used to detect objects in other images.First
we need to load the required XML classifiers. Then load our input image (or video)
in grayscale mode.

OPTICAL CHARACTER READER

Tesseract, a highly popular OCR(optical character recognition) engine, was
originally developed by Hewlett Packard in the 1980s and was then open-sourced
in 2005. Google adopted the project in 2006 and has been sponsoring it ever since.
Optical character recognition (OCR) refers to both the innovation and procedure
of perusing and changing over composed, printed or transcribed characters into
machine-encoded content or something that the PC can control. It is a subset of
picture acknowledgment and is generally utilized as a type of information section
with the information being some kind of printed archive or information record, for
example, bank proclamations, deals solicitations, travel papers, resumes and
business cards. The record is either checked or an image is taken and it is
dependent upon the program to perceive the characters and give a yield as a
content archive.
OCR (optical character recognition) is the acknowledgment of printed or
composed content characters by a computer. This includes photoscanning of the
text character-by-character, analysis of the checked in picture, and after that
interpretation of the character picture into character codes, for example, ASCII,
regularly used in data processing .In OCR preparing, the scanned in picture or
bitmap is analysed for light and dim regions in order to recognize each alphabetic
letter or numeric digit. At the point when a character is remembered, it is changed
over into an ASCII code. Extraordinary circuit sheets and PC chips structured
explicitly for OCR are utilized to accelerate the acknowledgment procedure.OCR
is being utilized by libraries to digitize and save their possessions. OCR is
likewiseused to process checks and charge card slips and sort the mail. Billions of
magazines and letters are arranged each day by OCR machines, extensively
accelerating mail conveyance.

SPEECH RECOGNITION:
Here we are doing speech recognition where we are taking in the audio and
converting to text. We are doing so by using a built in library in python, which
helps us to convert speech into text. Below is the explaanation how things work in
this pyttsx module. An application invokes the pyttsx3.init() factory function to get
a reference to a pyttsx3.Engine instance. During construction, the engine initializes
a pyttsx3.driver.DriverProxy object responsible for loading a speech engine driver
implementation from the pyttsx3.drivers module. After construction, an application
uses the engine object to register and unregister event callbacks; produce and stop
speech; get and set speech engine properties; and start and stop event loops.

INSTALLATIONS
On windows opencv can be installed through the following command---pip
install opencv –python

On anaconda->conda install –c menpoopencv

The required modules will be installed through the above commands.To avoid the
conflicts while installing opencv is to uninstall the manually installed modules in
the root of python’s site packages.We should not install multiple packages in same
environment. If there are multiple packages in same environment,uninstall them
with “pip uninstall” command and reinstall.

And later import them by “import cv2” command.

LIBRARIES

OPEN CV:
 Open source computer vision library.
 OpenCV is a library of programming functions mainly aimed at real-time
computervision.
  Originally developed by Intel, it was later supported by Willow Garage then
Itseez.
 The library is cross-platform and free for use under the open-source BSD
license.
 It has C++, Python and Java interfaces and supports Windows, Linux, Mac
OS, iOSand Android.
 Open cv supports deep learning frame work, tensor flow torch and
pytorchetc.
 Open cv applications include Human computer interaction,Ego
motionestimation,facial recognition system,facial recognition system,mobile
robotics and augmented reality etc.
 OpenCV is written in C++ and its primary interface is in C++, but it still
retains a lesscomprehensive though extensive older C interface. There are
bindingsin Python, Java and MATLAB/OCTAVE.

NUMPY LIBRARY

NumPy is the fundamental package for scientific computing with Python. It

contains among other things:
 a powerful N-dimensional array object
 sophisticated (broadcasting) functions
 tools for integrating C/C++ and Fortran code
 useful linear algebra, Fourier transform, and random number capabilities

SOFTWARE REQUIREMENTS
Pycharm
Opencv
Pillow
Cx_Oracle
Pyttsx3
Flow Chart

Code
import cv2
import numpy as np
from PIL import Image
import os
import pyttsx3
import time
import mysql.connector
import pytesseract
#from subprocess import call
con=mysql.connector.connect(
user="root",
password="karthikvajja",
host="127.0.0.1",
database="face",
auth_plugin="mysql_native_password"
)

cur = con.cursor()

listdata=list()
cur.execute("select * from comp_vis")
for i in cur:
listdata.append(i)
id=len(listdata)
def mouse():
#os.system(r"cd C:\Users\Intel\AppData\Local\Programs\Python\Python37-32>")
os.system(r"C:\Users\Intel\AppData\Local\Programs\Python\Python37-32\python
C:/Users/Intel/Desktop/mouse.py")
# os.system("python C:/Users/Intel/Desktop/mouse.py")
pytesseract.pytesseract.tesseract_cmd = r"C:\Users\Intel\Anaconda3\pkgs\tesseract.exe"
text1 = pytesseract.image_to_string(Image.open(r"C:\Users\Intel\Desktop\thres1.png"))
 engine = pyttsx3.init()
engine.say(text1)
engine.runAndWait()
def ocr():
pytesseract.pytesseract.tesseract_cmd = r"C:\Users\Intel\Anaconda3\pkgs\tesseract.exe"
engine = pyttsx3.init()
engine.say("Do you want to convert a part of a image")
engine.runAndWait()
print("Do you want to convert a part of a image")
g=speech()
if g=="no":
destdir = 'C:/users/Intel/Desktop/'
files = [ f for f in os.listdir(destdir) if os.path.isfile(os.path.join(destdir,f)) ]
for f in files:
if f.endswith(".jpg"):
print(f)
elif f.endswith(".png"):
print(f)
k=str(input("Enter the name of the file : "))
image = cv2.imread(r'C:/Users/Intel/Desktop/' + k)
text = pytesseract.image_to_string(Image.open(image))
print(text)
engine = pyttsx3.init()
engine.say(text)
engine.runAndWait()
elif g=="yes":
while True:
mouse()
 engine = pyttsx3.init()
engine.say("Do you want to continue ?")
engine.runAndWait()
h=speech()
if h=="no":
break
else:
engine.say("Sorry could not understand you !")
engine.runAndWait()
ocr()

def speech():
import speech_recognition as sr

r = sr.Recognizer()
with sr.Microphone() as source:
print("Say something!")
audio = r.listen(source)

try:
print("You said: " + r.recognize_google(audio))
return r.recognize_google(audio)
except sr.UnknownValueError:
print("Google Speech Recognition could not understand audio")
except sr.RequestError as e:
print("Could not request results from Google Speech Recognition service;
{0}".format(e))

def create_db():
engine = pyttsx3.init()
engine.setProperty("rate",200)
face_cascade = cv2.CascadeClassifier('C:/pro_/face/haarcascade_frontalface_default.xml')
global listdata
cap = cv2.VideoCapture(0)
global id
id=len(listdata)
id+=1
l=[]
l.append(id)

engine.say("Give your name.")

engine.runAndWait()
l.append(speech())

engine.say("give your reg no.")

engine.runAndWait()
l.append(speech())

l.append(input('Registration number:'))

engine.say("Taking pictures in a few moments, please look into the camera.")

engine.runAndWait()
sql="insert into comp_vis values(%d,'%s','%s')"%(id,l[1],l[2])
cur.execute(sql)
con.commit()
listdata.append(l)
sampleN=0;
 while 1:

ret, img = cap.read()

gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

faces = face_cascade.detectMultiScale(gray, 1.3, 5)

for (x,y,w,h) in faces:

sampleN=sampleN+1;

cv2.imwrite("C:/pro_/faces/faces."+str(id)+ "." +str(sampleN)+ ".jpg", gray[y:y+h,

x:x+w])

cv2.rectangle(img,(x,y),(x+w,y+h),(255,0,0),2)

cv2.waitKey(100)

cv2.imshow('img',img)

cv2.waitKey(1)

if sampleN > 30:

break

cap.release()
 cv2.destroyAllWindows()

path = 'C:/pro_/faces'

recognizer = cv2.face.LBPHFaceRecognizer_create()

detector = cv2.CascadeClassifier("C:/pro_/face/haarcascade_frontalface_default.xml");

def getImagesAndLabels(path):

imagePaths = [os.path.join(path,f) for f in os.listdir(path)]

faceSamples=[]
ids = []

for imagePath in imagePaths:

PIL_img = Image.open(imagePath).convert('L')
img_numpy = np.array(PIL_img,'uint8')

id = int(os.path.split(imagePath)[-1].split(".")[1])
faces = detector.detectMultiScale(img_numpy)

for (x,y,w,h) in faces:

faceSamples.append(img_numpy[y:y+h,x:x+w])
ids.append(id)

return faceSamples,ids
def train():

print ("\n [INFO] Training faces. It will take a few seconds. Wait ...")
faces,ids = getImagesAndLabels(path)
recognizer.train(faces, np.array(ids))

recognizer.write('C:/pro_/trash.yml')
print("\n [INFO] {0} faces trained. Exiting Program".format(len(np.unique(ids))))

def see():
global listdata
recognizer = cv2.face.LBPHFaceRecognizer_create()
recognizer.read('C:/pro_/face/trainer.yml')
cascadePath = "C:/pro_/face/haarcascade_frontalface_default.xml"
faceCascade = cv2.CascadeClassifier(cascadePath);

font = cv2.FONT_HERSHEY_SIMPLEX

id = 0

cam = cv2.VideoCapture(0)
cam.set(3, 640)
cam.set(4, 480)

minW = 0.1*cam.get(3)
minH = 0.1*cam.get(4)
c=0
c1=0
x1=0
while True:

ret, img =cam.read()

gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)

faces = faceCascade.detectMultiScale(
gray,
scaleFactor = 1.2,
minNeighbors = 5,
minSize = (int(minW), int(minH)),
)

for(x,y,w,h) in faces:

cv2.rectangle(img, (x,y), (x+w,y+h), (0,255,0), 2)

id, confidence = recognizer.predict(gray[y:y+h,x:x+w])

if (0 < confidence < 50 and id!=0):
m=id
id=(listdata[id-1][1])
print(id)
confidence = " {0}%".format(round(100 - confidence))
c=0
print(m,'---',x1,'--',c1)
 if(m==x1):
c1+=1
else:
c1=0
x1=m
if(c1==1):
engine = pyttsx3.init()
engine.setProperty("rate",200)
engine.say("Hey " +listdata[m-1][1] + "!")
engine.runAndWait()
while(1):
engine = pyttsx3.init()
engine.say("Do you want me to read something ? ")
engine.runAndWait()
print("Do you want me to read something")
a=speech()
if a=="yes":
ocr()
elif a=="no":
engine = pyttsx3.init()
engine.say("goodbye !")
engine.runAndWait()
print("Goodbye !")
time.sleep(5)
see()
else:
c+=1
id = "unknown"
 confidence = " {0}%".format(round(100 - confidence))
if(c==100):
cam.release()
cv2.destroyAllWindows()
engine = pyttsx3.init()
engine.setProperty("rate",200)
engine.say("Do you want me to remember you?")
engine.runAndWait()
j=speech()
#j=input("Do you want me to remember you? [y/n] ")

if(j=='y' or j=='Y' or j=='Yes' or j=='YES' or j=='yes'):

create_db()
engine = pyttsx3.init()
engine.setProperty("rate",200)
engine.say("Training Data, I am almost ready.")
engine.runAndWait()
train()
listdata =list(cur.execute("select * from comp_vis"))
engine.say("Nice to meet you"+listdata[len(listdata)-1][1])
engine.runAndWait()
c=0
see()
print(str(id))
cv2.putText(img, str(id), (x+5,y-5), font, 1, (255,255,255), 2)
cv2.putText(img, str(confidence), (x+5,y+h-5), font, 1, (255,255,0), 1)
 cv2.imshow('camera',img)

k = cv2.waitKey(10) & 0xff

k=k+10
if k == 27:
break

print("\n [INFO] Exiting Program and cleanup stuff")

cam.release()
cv2.destroyAllWindows()

print(listdata)

while 1:

see()

Conclusion
Thus we have used python and its libraries successfully for face detection or face
recognition. If a person goes in front of the web cam, the system will read out its
name and if it is not able to detect the person then it will get access of the images
to detect that person. This can be developed in to apps or webpage or can be used
in security systems in public places. This can also be used to monitor the drivers to
reduce the number of accidents and also can be implemented in attendance system.
The second part of our project reads out the name of the file or the image that a
user opens on the system.