SemesterCoreProject

FinalReport

Submittedby: Submittedto:
VaibhavSingh Mrs.Nishthaphutela

TABLEOFCONTENTS
AboutDeeplearning 3

LinearRegression 5
R2 6
Code: 7
Challenge 8
Code 8

UseofLinearregression 10

NeuralNetwork 10

Process 10
Code 11

UseofNeuralNetwork 22

TensorFlowforclassification 23

AboutDeeplearning
Traditionally, programing has been about defining every single step for a program to reach an
outcome. The machine learning flips that approach. With machine learning, we define the
outcome and program learn to steps get there. So if we build an app that can recognize street
signs so instead of writing the code to recognize hundreds of different feature of street signs
like this shape of letters and the colors. We will just say here are some examples of a street
signs, learn the shape you need to recognize it. Sometimes, we have not any idea what steps
could possibly be. Machine learning is already everywhere on the internet.Every major services
uses it in some way. Youtube uses it to decide which other videos we might like as we watch
anditsuseswillonlygrowovertime.
There are lot of machine learning models are there andoneofthemareneuralnetworks.When
we use a neural network thats not just one or two but many layers deep to make a prediction,
we called that deep learning. Its a subset of machine learning that has outperformed every
othertypeofmodelalmosteverytimeonahugerangeoftasks.

LinearRegression

Simplelinearregressionisastatisticalmethodthatallowsustosummarizeandstudy
relationshipsbetweentwocontinuous(quantitative)variables:

Onevariable,denotedx,isregardedasthepredictor,explanatory,orindependentvariable.

Theothervariable,denotedy,isregardedastheresponse,outcome,ordependentvariable.

Withsimplelinearregressionwewanttomodelourdataasfollows:

y=B0+B1*x(similartoy=mx+c)

Thisisalinewhereyistheoutputvariablewewanttopredict,xistheinputvariableweknow
andB0andB1arecoefficientsthatweneedtoestimatethatmovethelinearound.

R2
The R2 term is the coefficient of determination and it usually reflects how well the model
fits the observed data. The coefficient of determination is usually given by,

WhereYi is an observed response, is the mean of the observed responses, is a prediction of

theresponsemadebythelinearmodel,and(Yi -) is the residual, i.e., the difference between
theresponseandtheprediction.Also,SSEiscalledthesumofthesquarederror,orSSRthe
sumofthesquaredresiduals,andSSTiscalledthetotalsumofsquares.

Code:

import pandas as pd

from sklearn import linear_model

import matplotlib.pyplot as plt

# Read data

dataframe = pd.read_fwf('brain_body.txt')

print(dataframe)

x_values = dataframe[['Brain']]

y_values = dataframe[['Body']]

# train model on data

body_regression = linear_model.LinearRegression()

body_regression.fit(x_values, y_values)

# visualize results

plt.scatter(x_values, y_values)

plt.plot(x_values, body_regression.predict(x_values))

plt.show()

Challenge
challengeistousescikit-learntocreatealineofbestfitfortheincluded'challenge_dataset'.
Then,makeapredictionforanexistingdatapointandseehowcloseitmatchesuptotheactual
value.Printouttheerroryouget.Youcanusescikit-learn'sdocumentationformorehelp.

Code

import pandas as pd

import numpy as np

from sklearn import linear_model

import matplotlib.pyplot as plt

from pylab import figure

# Read dataset

dataframes = pd.read_csv('challenge_dataset.txt', sep=',', header=None)

dataframes.columns = ["X_data", "Y_data"]

x_values = dataframes[['X_data']]

y_values = dataframes[['Y_data']]

# Train data with linear regression model

model = linear_model.LinearRegression()

model.fit(x_values, y_values)

# function for calculating residuals

def calculate_residual(actual, predicted):

res = (np.array(actual)-np.array(predicted)**2)

return res

# Evaluating fit model

residuals = calculate_residual(y_values, model.predict(x_values))

print("Mean squared error: ", np.mean(residuals))

print("Coefficient of determination: ", model.score(x_values, y_values))

# Plotting on matplotlib

plt.figure(1)

plt.scatter(x_values, y_values, color="blue")

plt.savefig("Scatter.png")

plt.figure(2)

plt.plot(x_values, model.predict(x_values), color="black")

plt.figure(3)

plt.plot(residuals)

plt.savefig("residual.png")

plt.show()

UseofLinearregression

1. Inenvironmentalscienceitisusedtotrytoestablishhowmuchonequantity,say
atmosphericgreenhousegasses,influencesanother,sayglobalsurfacetemperature.
2. Inquantitativefinancelinearregressioniscoretoeverything,astheyusesomething
calledalinearfactormodel.Longstoryshortitestimateshowmuchthepriceofone
assetwillmovewhenanotherthing(oilprices)move.

NeuralNetwork

AnArtificialNeuralNetwork(ANN)isaninformationprocessingparadigmthatisinspiredbythe
waybiologicalnervoussystems,suchasthebrain,processinformation.Thekeyelementofthis
paradigmisthenovelstructureoftheinformationprocessingsystem.Itiscomposedofalarge
numberofhighlyinterconnectedprocessingelements(neurons)workinginunisontosolve
specificproblems.ANNs,likepeople,learnbyexample.AnANNisconfiguredforaspecific
application,suchaspatternrecognitionordataclassification,throughalearningprocess.
Learninginbiologicalsystemsinvolvesadjustmentstothesynapticconnectionsthatexist
betweentheneurones.ThisistrueofANNsaswell.

Process
Wewillgiveeachinputaweight,whichcanbeapositiveornegativenumber.Aninputwitha
largepositiveweightoralargenegativeweight,willhaveastrongeffectontheneuronsoutput.

1. Taketheinputsfromatrainingsetexample,adjustthembytheweights,andpassthem
throughaspecialformulatocalculatetheneuronsoutput.
2. Calculatetheerror,whichisthedifferencebetweentheneuronsoutputandthedesired
outputinthetrainingsetexample.
3. Dependingonthedirectionoftheerror,adjusttheweightsslightly.
4. Repeatthisprocess10,000times.

Code

from numpy import exp, array, random, dot

import numpy as np

class NeuralNetwork():

"""docstring for ."""

def __init__(self):

# seed the random number generation, so it generates the same number

# every time the program runs

random.seed(1)

# we model a single neuron, with 3 input connections and 1 output connection.

# we assign random weights to 3 x 1 matrix, with a value in the range

# of -1 to 1 a mean of 0

# our neuron have 3 input connections and 1 output connections

self.synaptic_weights = 2 * np.random.random((3, 1)) - 1

# Activation Function

# The sigmoid function, which describes an s shaped curve

# we pass the weighted sum of the inputs through this function

# to normalise them between 0 and 1

def __sigmoid(self, x):

return 1 / (1 + exp(-x))

# gradient of the sigmoid curve

def __sigmoid_deravative(self, x):

return x * (1 - x)

def train(self, training_set_inputs, training_set_outputs, number_of_training_iterations):

for iteration in range(number_of_training_iterations):

# pass the training set through our neural net

output = self.predict(training_set_inputs)

# calculate the error

error = (training_set_outputs - output)

# multiply the error by the input ad again by the gradient of the

# sigmoid curve

adjustments = dot(training_set_inputs.T, error *

self.__sigmoid_deravative(output))

# adjust the weights

self.synaptic_weights += adjustments

def predict(self, inputs):

# pass inputs through our neural network (our singe neuron)

return self.__sigmoid(dot(inputs, self.synaptic_weights))

if __name__ == '__main__':

# initialise a single neuron neural network

neural_network = NeuralNetwork()

print('Random starting synaptic weights: ')

print(neural_network.synaptic_weights)

# the training set. We have 4 examples, each consisting of 3 input values

# and 1 output value.

training_set_inputs = array([[0, 0, 1], [1, 1, 1], [1, 0, 1], [0, 1, 1]])

training_set_outputs = array([[0, 1, 1, 0]]).T

# train the neural network using the training sets

# Do it 10,000 times amd making small adjustments each time

neural_network.train(training_set_inputs, training_set_outputs, 10000)

print('New synaptic weights after training: ')

print(neural_network.synaptic_weights)

# Test the neural network

print('predicting...')

print('considering new situation [1, 0, 0] -> ?: ')

print(neural_network.predict(array([1, 0, 0])))

Challengeistocreatea3layerfeedforwardneuralnetworkusingonlynumpyasdependency.

Challenge.py

from numpy import exp, array, random, dot

import numpy as np

import matplotlib.pyplot as plt

# %matplotlib inline

class NeuralNetwork():

"""docstring for NeuralNetwork."""

def __init__(self):

# seed the random number generation, so it generates the same number

# every time

random.seed(1)

# setting the number of nodes in layer 2 and layer 3

# more the number of nodes in hidden layer more the confident the

# neural network in predicting

l2 = 5

l3 = 6

# assign random weights to the matrix

# (no. of nodes in previous layer) x (no. of nodes in following layer)

self.synaptic_weights1 = 2 * np.random.random((3, l2)) - 1

self.synaptic_weights2 = 2 * np.random.random((l2, l3)) - 1

self.synaptic_weights3 = 2 * np.random.random((l3, 1)) - 1

def __sigmoid(self, x):

return 1/(1 + exp(-x))

def __sigmoid_deravative(self, x):

return x * (1-x)

# train neural network, adjusting synaptics weight each time

def train(self, training_set_inputs, training_set_outputs, number_of_training_iterations):

for iteration in range(number_of_training_iterations):

# passing training set through our neual network

# a2 means activation fed to the second layer

a2 = self.__sigmoid(dot(training_set_inputs, self.synaptic_weights1))

print("============================================================")

print("Dot product output of 2nd Layer")

print(dot(training_set_inputs, self.synaptic_weights1))

print("Sigmoid function output")

print(a2)

print("============================================================")

a3 = self.__sigmoid(dot(a2, self.synaptic_weights2))

print("============================================================")

print("Dot product output of 3rd Layer")

print(dot(a2, self.synaptic_weights2))

print("Sigmoid function output")

print(a3)

print("============================================================")

output = self.__sigmoid(dot(a3, self.synaptic_weights3))

print("============================================================")

print("Dot product output layer")

print(dot(a3, self.synaptic_weights3))

print("Sigmoid function output")

print(output)

print("============================================================")

# calculatng error

delta4 = (training_set_outputs - output)

print("++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")

print("Error:")

print(delta4)

print("++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")

# finding errors in each layer

delta3 = dot(self.synaptic_weights3, delta4.T) * (self.__sigmoid_deravative(a3).T)

print("++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")

print("Error in Layer 3:")

print(delta3)

print("++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")

delta2 = dot(self.synaptic_weights2, delta3) * (self.__sigmoid_deravative(a2).T)

print("++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")

print("Error in LAyer 2:")

print(delta2)

print("++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")

# adjustments(gradient) in each layer

adjustment3 = dot(a3.T, delta4)

adjustment2 = dot(a2.T, delta3.T)

adjustment1 = dot(training_set_inputs.T, delta2.T)

# adjusting the weight accordingly

self.synaptic_weights1 += adjustment1

self.synaptic_weights2 += adjustment2

self.synaptic_weights3 += adjustment3

def forward_pass(self, inputs):

# passing our inputs into neural networks

a2 = self.__sigmoid(dot(inputs, self.synaptic_weights1))

a3 = self.__sigmoid(dot(a2, self.synaptic_weights2))

output = self.__sigmoid(dot(a3, self.synaptic_weights3))

return output

if __name__ == '__main__':

# initializing neural network

neural_network = NeuralNetwork()

# printing the initial weight assigned to the neural network

print("Random starting synaptics weights (layer 1): ")

print(neural_network.synaptic_weights1)

print("Random starting synaptics weights (layer 2): ")

print(neural_network.synaptic_weights2)

print("Random starting synaptics weights (layer 3): ")

print(neural_network.synaptic_weights3)

# loading the training set

training_set_inputs = array([[0, 0, 1], [1, 1, 1], [1, 0, 1], [0, 1, 1]])

plt.imshow(training_set_inputs)

plt.show()

# Interpolation calculates what the color or value of a pixel "should" be,

# according to different mathematical schemes. One common place that this

# happens is when you resize an image. The number of pixels change, but you

# want the same information. Since pixels are discrete, there's missing

# space. Interpolation is how you fill that space.

plt.imshow(training_set_inputs, interpolation='nearest')

plt.show()

training_set_outputs = array([[0, 1, 1, 0]]).T

# training the neural network using the training sets

# Doing 10,000 times and making small adjustments every time

neural_network.train(training_set_inputs, training_set_outputs, 10000)

print("\nnew synaptic weights (layer 1) after training: ")

print(neural_network.synaptic_weights1)

print("\nnew synaptic weights (layer 2) after training: ")

print(neural_network.synaptic_weights2)

print("\nnew synaptic weights (layer 3) after training: ")

print(neural_network.synaptic_weights3)

# Testing the neural network

print("Predicting...")

print("considering new situation [1, 0, 0] -> ?: ")

print(neural_network.forward_pass(array([1, 0, 0])))

UseofNeuralNetwork
CharacterRecognition-Theideaofcharacterrecognitionhasbecomeveryimportantas
handhelddeviceslikethePalmPilotarebecomingincreasinglypopular.Neuralnetworkscanbe
usedtorecognizehandwrittencharacters.

ImageCompression-Neuralnetworkscanreceiveandprocessvastamountsofinformationat
once,makingthemusefulinimagecompression.WiththeInternetexplosionandmoresites
usingmoreimagesontheirsites,usingneuralnetworksforimagecompressionisworthalook.

StockMarketPrediction-Theday-to-daybusinessofthestockmarketisextremely
complicated.Manyfactorsweighinwhetheragivenstockwillgoupordownonanygivenday.
Sinceneuralnetworkscanexaminealotofinformationquicklyandsortitallout,theycanbe
usedtopredictstockprices.

TravelingSaleman'sProblem-Interestinglyenough,neuralnetworkscansolvethetraveling
salesmanproblem,butonlytoacertaindegreeofapproximation.

Medicine,ElectronicNose,Security,andLoanApplications-Thesearesomeapplicationsthat
areintheirproof-of-conceptstage,withtheacceptionofaneuralnetworkthatwilldecide
whetherornottograntaloan,somethingthathasalreadybeenusedmoresuccessfullythan
manyhumans.

MiscellaneousApplications-Thesearesomeveryinteresting(albeitattimesalittleabsurd)
applicationsofneuralnetworks.

TensorFlowforclassification
Classificationisoneofthemostimportantpartsofmachinelearning,asmostofpeoples
communicationisdoneviatext.Wewriteblogarticles,email,tweet,leavenotesandcomments.
Allthisinformationistherebutisreallyhardtousecomparedtoaformordatacollectedfrom
somesensor.

TherebeenclassicNLPtechniquesdealingwiththis,bymostlyusingwordsassymbolsand
runninglinearmodels.Thistechniquesworkedbutwereverybrittle.Recentadoptionof
embeddingsanddeeplearningopenedupanewwaysofhandlingtext.

"""Supervised problem."""

import pandas as pd # work with data as tables

import numpy as np # use number matrics

import matplotlib.pyplot as plt

import tensorflow as tf

# step 1: Load data

dataframe = pd.read_csv('data.csv') # dataframe objects

# remove the features we not care about

dataframe = dataframe.drop(['index', 'price', 'sq_price'], axis=1)

# we only use first 10 rows of dataset

dataframe = dataframe[0:10]

print(dataframe)

# Step 2: Adding labels

# 1 is good buy and 0 is bad buy

dataframe.loc[:, ('y1')] = [1, 1, 1, 0, 0, 1, 0, 1, 1, 1]

# y2 is a negation of y1, opposite

dataframe.loc[:, ('y2')] = dataframe['y1'] == 0

# turns True/False values to 1s and 0s

dataframe.loc[:, ('y2')] = dataframe['y2'].astype(int)

# Step 3: Prepare data for tensorflow (tensors)

# tensors are a generic version of matrix and vectors

# vector - is a list of numbers (1D tensor)

# matrix is a list of list of numbers (2D tensor)

# list of list of list of numbers (3D tensors)

# converting features in input tensor

inputX = dataframe.loc[:, ['area', 'bathrooms']].as_matrix()

# convert labels into tensors

inputY = dataframe.loc[:, ['y1', 'y2']].as_matrix()

# print(inputX)

# print(inputY)

# Step 4: Write out our hyperparameters

learning_rate = 0.000001

training__epochs = 2000

display_steps = 50

n_samples = inputY.size

# Step 5: create our compuatatin graph/Neural network

# for feature input tensor, none means any number of examples

# placeholders are gateways for data in our computational graph

x = tf.placeholder(tf.float32, [None, 2])

# create weights

# 2x2 float matrix, that We'll keep updating through our training process

# variable in tf hold and update parameters

# in memory bufers contating tensors

w = tf.Variable(tf.zeros([2, 2]))

# add biases (exampe c in y = mx + c, c is a bias)

b = tf.Variable(tf.zeros([2]))

# multiply our weights by our inputs,

# weights are how the data flows in our compuattion graph

# multiply inputs with weights and biases

y_values = tf.add(tf.matmul(x, w), b)

# apply softmax to values we just created i.e "y_values"

# softmax is our activation function

y = tf.nn.softmax(y_values)

# feeding in a matrix labels

y_ = tf.placeholder(tf.float32, [None, 2])

# perform training

# create our cost function, mean squared error

# reduced sum computes the sum of elements across dimensions of a tensor

cost = tf.reduce_sum(tf.pow(y_-y, 2))/(2*n_samples)

# Gradient descent

optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)

# initialize variables and tensorflow sessions

init = tf.global_variables_initializer()

sess = tf.Session()

sess.run(init)

# training loop

for i in range(training__epochs):

sess.run(optimizer, feed_dict={x: inputX, y_: inputY}) # Take a gradient descent step

using our inputs and labels

# That's all! The rest of the cell just outputs debug messages.

# Display logs per epoch step

if (i) % display_steps == 0:

cc = sess.run(cost, feed_dict={x: inputX, y_:inputY})

print("Training step:", '%04d' % (i), "cost=", "{:.9f}".format(cc)) #, \"W=",

sess.run(W), "b=", sess.run(b)

print("Optimization Finished!")

training_cost = sess.run(cost, feed_dict={x: inputX, y_: inputY})

print("Training cost=", training_cost, "W=", sess.run(w), "b=", sess.run(b), '\n')

sess.run(y, feed_dict={x: inputX })

# So It's guessing they're all good houses. That makes it get 7/10 correct

sess.run(tf.nn.softmax([1., 2.]))