Commonly used Machine Learning Algorithms (with Python

and R Codes)
Sunil Ray, September 9, 2017
Login to Bookmark this article

Note: This article was originally published on August 10, 2015 and updated on Sept 9th, 2017

Overview
 Major focus on commonly used machine learning algorithms
 Algorithms covered- Linear regression, logistic regression, Naive Bayes, kNN, Random forest, etc.
 Learn both theory and implementation of these algorithms in R and python

Introduction
Google’s self-driving cars and robots get a lot of press, but the company’s real future is in machine learning, the
technology that enables computers to get smarter and more personal.

– Eric Schmidt (Google Chairman)

We are probably living in the most defining period of human history. The period when computing moved from
large mainframes to PCs to cloud. But what makes it defining is not what has happened, but what is coming our
way in years to come.

What makes this period exciting and enthralling for someone like me is the democratization of the various tools
and techniques, which followed the boost in computing. Welcome to the world of data science!

Today, as a data scientist, I can build data-crunching machines with complex algorithms for a few dollars per
hour. But reaching here wasn’t easy! I had my dark days and nights.

Are you a beginner looking for a place to start your data science journey? Presenting two comprehensive
courses, full of knowledge and data science learning, curated just for you to learn data science (using Python)
from scratch:

 Introduction to Data Science

 Certified Program: Data Science for Beginners (with Interviews)

Who can benefit the most from this guide?

What I am giving out today is probably the most valuable guide, I have ever created.

The idea behind creating this guide is to simplify the journey of aspiring data scientists and machine learning
enthusiasts across the world. Through this guide, I will enable you to work on machine learning problems and
gain from experience. I am providing a high-level understanding of various machine learning algorithms
along with R & Python codes to run them. These should be sufficient to get your hands dirty.
Essentials of machine learning algorithms with implementation in R and Python

I have deliberately skipped the statistics behind these techniques, as you don’t need to understand them at the
start. So, if you are looking for statistical understanding of these algorithms, you should look elsewhere. But, if
you are looking to equip yourself to start building machine learning project, you are in for a treat.

Broadly, there are 3 types of Machine Learning Algorithms

1. Supervised Learning

How it works: This algorithm consist of a target / outcome variable (or dependent variable) which is to be
predicted from a given set of predictors (independent variables). Using these set of variables, we generate a
function that map inputs to desired outputs. The training process continues until the model achieves a desired
level of accuracy on the training data. Examples of Supervised Learning: Regression, Decision Tree, Random
Forest, KNN, Logistic Regression etc.

2. Unsupervised Learning

How it works: In this algorithm, we do not have any target or outcome variable to predict / estimate. It is used
for clustering population in different groups, which is widely used for segmenting customers in different groups
for specific intervention. Examples of Unsupervised Learning: Apriori algorithm, K-means.

3. Reinforcement Learning:

How it works: Using this algorithm, the machine is trained to make specific decisions. It works this way: the
machine is exposed to an environment where it trains itself continually using trial and error. This machine learns
from past experience and tries to capture the best possible knowledge to make accurate business decisions.
Example of Reinforcement Learning: Markov Decision Process
List of Common Machine Learning Algorithms
Here is the list of commonly used machine learning algorithms. These algorithms can be applied to almost any
data problem:

1. Linear Regression
2. Logistic Regression
3. Decision Tree
4. SVM
5. Naive Bayes
6. kNN
7. K-Means
8. Random Forest
9. Dimensionality Reduction Algorithms
10. Gradient Boosting algorithms
1. GBM
2. XGBoost
3. LightGBM
4. CatBoost

1. Linear Regression
It is used to estimate real values (cost of houses, number of calls, total sales etc.) based on continuous variable(s).
Here, we establish relationship between independent and dependent variables by fitting a best line. This best fit
line is known as regression line and represented by a linear equation Y= a *X + b.

The best way to understand linear regression is to relive this experience of childhood. Let us say, you ask a child
in fifth grade to arrange people in his class by increasing order of weight, without asking them their weights!
What do you think the child will do? He / she would likely look (visually analyze) at the height and build of
people and arrange them using a combination of these visible parameters. This is linear regression in real life!
The child has actually figured out that height and build would be correlated to the weight by a relationship, which
looks like the equation above.

In this equation:

 Y – Dependent Variable
 a – Slope
 X – Independent variable
 b – Intercept

These coefficients a and b are derived based on minimizing the sum of squared difference of distance between
data points and regression line.

Look at the below example. Here we have identified the best fit line having linear equation y=0.2811x+13.9. Now
using this equation, we can find the weight, knowing the height of a person.
Linear Regression is mainly of two types: Simple Linear Regression and Multiple Linear Regression. Simple
Linear Regression is characterized by one independent variable. And, Multiple Linear Regression(as the name
suggests) is characterized by multiple (more than 1) independent variables. While finding the best fit line, you
can fit a polynomial or curvilinear regression. And these are known as polynomial or curvilinear regression.

Here’s a coding window to try out your hand and build your own linear regression model in Python:

R Code

#Load Train and Test datasets

#Identify feature and response variable(s) and values must be numeric and numpy arrays
x_train <- input_variables_values_training_datasets
y_train <- target_variables_values_training_datasets
x_test <- input_variables_values_test_datasets
x <- cbind(x_train,y_train)
# Train the model using the training sets and check score
linear <- lm(y_train ~ ., data = x)
summary(linear)
#Predict Output
predicted= predict(linear,x_test)

2. Logistic Regression
Don’t get confused by its name! It is a classification not a regression algorithm. It is used to estimate discrete
values ( Binary values like 0/1, yes/no, true/false ) based on given set of independent variable(s). In simple words,
it predicts the probability of occurrence of an event by fitting data to a logit function. Hence, it is also known as
logit regression. Since, it predicts the probability, its output values lies between 0 and 1 (as expected).

Again, let us try and understand this through a simple example.

Let’s say your friend gives you a puzzle to solve. There are only 2 outcome scenarios – either you solve it or you
don’t. Now imagine, that you are being given wide range of puzzles / quizzes in an attempt to understand which
subjects you are good at. The outcome to this study would be something like this – if you are given a trignometry
based tenth grade problem, you are 70% likely to solve it. On the other hand, if it is grade fifth history question,
the probability of getting an answer is only 30%. This is what Logistic Regression provides you.

Coming to the math, the log odds of the outcome is modeled as a linear combination of the predictor variables.
odds= p/ (1-p) = probability of event occurrence / probability of not event occurrence
ln(odds) = ln(p/(1-p))
logit(p) = ln(p/(1-p)) = b0+b1X1+b2X2+b3X3....+bkXk

Above, p is the probability of presence of the characteristic of interest. It chooses parameters that maximize the
likelihood of observing the sample values rather than that minimize the sum of squared errors (like in ordinary
regression).

Now, you may ask, why take a log? For the sake of simplicity, let’s just say that this is one of the best mathematical
way to replicate a step function. I can go in more details, but that will beat the purpose of this article.

Build your own logistic regression model in Python here and check the accuracy:

R Code

x <- cbind(x_train,y_train)
# Train the model using the training sets and check score
logistic <- glm(y_train ~ ., data = x,family='binomial')
summary(logistic)
#Predict Output
predicted= predict(logistic,x_test)

Furthermore..

There are many different steps that could be tried in order to improve the model:

 including interaction terms

 removing features
 regularization techniques
 using a non-linear model

3. Decision Tree
This is one of my favorite algorithm and I use it quite frequently. It is a type of supervised learning algorithm that
is mostly used for classification problems. Surprisingly, it works for both categorical and continuous dependent
variables. In this algorithm, we split the population into two or more homogeneous sets. This is done based on
most significant attributes/ independent variables to make as distinct groups as possible. For more details, you
can read: Decision Tree Simplified.

source: statsexchange

In the image above, you can see that population is classified into four different groups based on multiple attributes
to identify ‘if they will play or not’. To split the population into different heterogeneous groups, it uses various
techniques like Gini, Information Gain, Chi-square, entropy.

The best way to understand how decision tree works, is to play Jezzball – a classic game from Microsoft (image
below). Essentially, you have a room with moving walls and you need to create walls such that maximum area
gets cleared off with out the balls.

So, every time you split the room with a wall, you are trying to create 2 different populations with in the same
room. Decision trees work in very similar fashion by dividing a population in as different groups as possible.

More: Simplified Version of Decision Tree Algorithms

Let’s get our hands dirty and code our own decision tree in Python!
R Code

library(rpart)
x <- cbind(x_train,y_train)
# grow tree
fit <- rpart(y_train ~ ., data = x,method="class")
summary(fit)
#Predict Output
predicted= predict(fit,x_test)

4. SVM (Support Vector Machine)

It is a classification method. In this algorithm, we plot each data item as a point in n-dimensional space (where n
is number of features you have) with the value of each feature being the value of a particular coordinate.

For example, if we only had two features like Height and Hair length of an individual, we’d first plot these two
variables in two dimensional space where each point has two co-ordinates (these co-ordinates are known as
Support Vectors)

Now, we will find some line that splits the data between the two differently classified groups of data. This will
be the line such that the distances from the closest point in each of the two groups will be farthest away.
In the example shown above, the line which splits the data into two differently classified groups is the black line,
since the two closest points are the farthest apart from the line. This line is our classifier. Then, depending on
where the testing data lands on either side of the line, that’s what class we can classify the new data as.

More: Simplified Version of Support Vector Machine

Think of this algorithm as playing JezzBall in n-dimensional space. The tweaks in the game are:

 You can draw lines/planes at any angles (rather than just horizontal or vertical as in the classic game)
 The objective of the game is to segregate balls of different colors in different rooms.
 And the balls are not moving.

Try your hand and design an SVM model in Python through this coding window:

R Code

library(e1071)
x <- cbind(x_train,y_train)
# Fitting model
fit <-svm(y_train ~ ., data = x)
summary(fit)
#Predict Output
predicted= predict(fit,x_test)

5. Naive Bayes
It is a classification technique based on Bayes’ theorem with an assumption of independence between predictors.
In simple terms, a Naive Bayes classifier assumes that the presence of a particular feature in a class is unrelated
to the presence of any other feature. For example, a fruit may be considered to be an apple if it is red, round, and
about 3 inches in diameter. Even if these features depend on each other or upon the existence of the other features,
a naive Bayes classifier would consider all of these properties to independently contribute to the probability that
this fruit is an apple.

Naive Bayesian model is easy to build and particularly useful for very large data sets. Along with simplicity,
Naive Bayes is known to outperform even highly sophisticated classification methods.
Bayes theorem provides a way of calculating posterior probability P(c|x) from P(c), P(x) and P(x|c). Look at the
equation below:

Here,

 P(c|x) is the posterior probability of class (target) given predictor (attribute).

 P(c) is the prior probability of class.
 P(x|c) is the likelihood which is the probability of predictor given class.
 P(x) is the prior probability of predictor.

Example: Let’s understand it using an example. Below I have a training data set of weather and corresponding
target variable ‘Play’. Now, we need to classify whether players will play or not based on weather condition. Let’s
follow the below steps to perform it.

Step 1: Convert the data set to frequency table

Step 2: Create Likelihood table by finding the probabilities like Overcast probability = 0.29 and probability of
playing is 0.64.

Step 3: Now, use Naive Bayesian equation to calculate the posterior probability for each class. The class with
the highest posterior probability is the outcome of prediction.

Problem: Players will pay if weather is sunny, is this statement is correct?

We can solve it using above discussed method, so P(Yes | Sunny) = P( Sunny | Yes) * P(Yes) / P (Sunny)

Here we have P (Sunny |Yes) = 3/9 = 0.33, P(Sunny) = 5/14 = 0.36, P( Yes)= 9/14 = 0.64
Now, P (Yes | Sunny) = 0.33 * 0.64 / 0.36 = 0.60, which has higher probability.

Naive Bayes uses a similar method to predict the probability of different class based on various attributes. This
algorithm is mostly used in text classification and with problems having multiple classes.

Code a Naive Bayes classification model in Python:

R Code

library(e1071)
x <- cbind(x_train,y_train)
# Fitting model
fit <-naiveBayes(y_train ~ ., data = x)
summary(fit)
#Predict Output
predicted= predict(fit,x_test)

6. kNN (k- Nearest Neighbors)

It can be used for both classification and regression problems. However, it is more widely used in classification
problems in the industry. K nearest neighbors is a simple algorithm that stores all available cases and classifies
new cases by a majority vote of its k neighbors. The case being assigned to the class is most common amongst its
K nearest neighbors measured by a distance function.

These distance functions can be Euclidean, Manhattan, Minkowski and Hamming distance. First three functions
are used for continuous function and fourth one (Hamming) for categorical variables. If K = 1, then the case is
simply assigned to the class of its nearest neighbor. At times, choosing K turns out to be a challenge while
performing kNN modeling.

More: Introduction to k-nearest neighbors : Simplified.

KNN can easily be mapped to our real lives. If you want to learn about a person, of whom you have no
information, you might like to find out about his close friends and the circles he moves in and gain access to
his/her information!

Things to consider before selecting kNN:

 KNN is computationally expensive

 Variables should be normalized else higher range variables can bias it
 Works on pre-processing stage more before going for kNN like an outlier, noise removal
Python Code

R Code

library(knn)
x <- cbind(x_train,y_train)
# Fitting model
fit <-knn(y_train ~ ., data = x,k=5)
summary(fit)
#Predict Output
predicted= predict(fit,x_test)

7. K-Means
It is a type of unsupervised algorithm which solves the clustering problem. Its procedure follows a simple and
easy way to classify a given data set through a certain number of clusters (assume k clusters). Data points inside
a cluster are homogeneous and heterogeneous to peer groups.

Remember figuring out shapes from ink blots? k means is somewhat similar this activity. You look at the shape
and spread to decipher how many different clusters / population are present!

How K-means forms cluster:

1. K-means picks k number of points for each cluster known as centroids.

2. Each data point forms a cluster with the closest centroids i.e. k clusters.
3. Finds the centroid of each cluster based on existing cluster members. Here we have new centroids.
4. As we have new centroids, repeat step 2 and 3. Find the closest distance for each data point from new
centroids and get associated with new k-clusters. Repeat this process until convergence occurs i.e.
centroids does not change.

How to determine value of K:

In K-means, we have clusters and each cluster has its own centroid. Sum of square of difference between centroid
and the data points within a cluster constitutes within sum of square value for that cluster. Also, when the sum of
square values for all the clusters are added, it becomes total within sum of square value for the cluster solution.
We know that as the number of cluster increases, this value keeps on decreasing but if you plot the result you may
see that the sum of squared distance decreases sharply up to some value of k, and then much more slowly after
that. Here, we can find the optimum number of cluster.

Python Code

R Code

library(cluster)
fit <- kmeans(X, 3) # 5 cluster solution

8. Random Forest
Random Forest is a trademark term for an ensemble of decision trees. In Random Forest, we’ve collection of
decision trees (so known as “Forest”). To classify a new object based on attributes, each tree gives a classification
and we say the tree “votes” for that class. The forest chooses the classification having the most votes (over all the
trees in the forest).

Each tree is planted & grown as follows:

1. If the number of cases in the training set is N, then sample of N cases is taken at random but with
replacement. This sample will be the training set for growing the tree.
2. If there are M input variables, a number m<<M is specified such that at each node, m variables are selected
at random out of the M and the best split on these m is used to split the node. The value of m is held
constant during the forest growing.
3. Each tree is grown to the largest extent possible. There is no pruning.

For more details on this algorithm, comparing with decision tree and tuning model parameters, I would suggest
you to read these articles:

1. Introduction to Random forest – Simplified

2. Comparing a CART model to Random Forest (Part 1)
3. Comparing a Random Forest to a CART model (Part 2)
4. Tuning the parameters of your Random Forest model
Python Code:

R Code

library(randomForest)
x <- cbind(x_train,y_train)
# Fitting model
fit <- randomForest(Species ~ ., x,ntree=500)
summary(fit)
#Predict Output
predicted= predict(fit,x_test)

9. Dimensionality Reduction Algorithms

In the last 4-5 years, there has been an exponential increase in data capturing at every possible stages. Corporates/
Government Agencies/ Research organisations are not only coming with new sources but also they are capturing
data in great detail.

For example: E-commerce companies are capturing more details about customer like their demographics, web
crawling history, what they like or dislike, purchase history, feedback and many others to give them personalized
attention more than your nearest grocery shopkeeper.

As a data scientist, the data we are offered also consist of many features, this sounds good for building good
robust model but there is a challenge. How’d you identify highly significant variable(s) out 1000 or 2000? In such
cases, dimensionality reduction algorithm helps us along with various other algorithms like Decision Tree,
Random Forest, PCA, Factor Analysis, Identify based on correlation matrix, missing value ratio and others.

To know more about this algorithms, you can read “Beginners Guide To Learn Dimension Reduction
Techniques“.

Python Code

R Code

library(stats)
pca <- princomp(train, cor = TRUE)
train_reduced <- predict(pca,train)
test_reduced <- predict(pca,test)

10. Gradient Boosting Algorithms

10.1. GBM

GBM is a boosting algorithm used when we deal with plenty of data to make a prediction with high prediction
power. Boosting is actually an ensemble of learning algorithms which combines the prediction of several base
estimators in order to improve robustness over a single estimator. It combines multiple weak or average predictors
to a build strong predictor. These boosting algorithms always work well in data science competitions like Kaggle,
AV Hackathon, CrowdAnalytix.

More: Know about Boosting algorithms in detail

Python Code

R Code

library(caret)
x <- cbind(x_train,y_train)
# Fitting model
fitControl <- trainControl( method = "repeatedcv", number = 4, repeats = 4)
fit <- train(y ~ ., data = x, method = "gbm", trControl = fitControl,verbose = FALSE)
predicted= predict(fit,x_test,type= "prob")[,2]

GradientBoostingClassifier and Random Forest are two different boosting tree classifier and often people ask
about the difference between these two algorithms.

10.2. XGBoost

Another classic gradient boosting algorithm that’s known to be the decisive choice between winning and losing
in some Kaggle competitions.

The XGBoost has an immensely high predictive power which makes it the best choice for accuracy in events as
it possesses both linear model and the tree learning algorithm, making the algorithm almost 10x faster than
existing gradient booster techniques.

The support includes various objective functions, including regression, classification and ranking.

One of the most interesting things about the XGBoost is that it is also called a regularized boosting technique.
This helps to reduce overfit modelling and has a massive support for a range of languages such as Scala, Java,
R, Python, Julia and C++.

Supports distributed and widespread training on many machines that encompass GCE, AWS, Azure and Yarn
clusters. XGBoost can also be integrated with Spark, Flink and other cloud dataflow systems with a built in
cross validation at each iteration of the boosting process.

To learn more about XGBoost and parameter tuning, visit

https://www.analyticsvidhya.com/blog/2016/03/complete-guide-parameter-tuning-xgboost-with-codes-python/.

Python Code:

R Code:

require(caret)

x <- cbind(x_train,y_train)

# Fitting model

TrainControl <- trainControl( method = "repeatedcv", number = 10, repeats = 4)

model<- train(y ~ ., data = x, method = "xgbLinear", trControl = TrainControl,verbose =

FALSE)

OR

model<- train(y ~ ., data = x, method = "xgbTree", trControl = TrainControl,verbose =

FALSE)
predicted <- predict(model, x_test)

10.3. LightGBM

LightGBM is a gradient boosting framework that uses tree based learning algorithms. It is designed to be
distributed and efficient with the following advantages:

 Faster training speed and higher efficiency

 Lower memory usage
 Better accuracy
 Parallel and GPU learning supported
 Capable of handling large-scale data

The framework is a fast and high-performance gradient boosting one based on decision tree algorithms, used for
ranking, classification and many other machine learning tasks. It was developed under the Distributed Machine
Learning Toolkit Project of Microsoft.

Since the LightGBM is based on decision tree algorithms, it splits the tree leaf wise with the best fit whereas
other boosting algorithms split the tree depth wise or level wise rather than leaf-wise. So when growing on the
same leaf in Light GBM, the leaf-wise algorithm can reduce more loss than the level-wise algorithm and hence
results in much better accuracy which can rarely be achieved by any of the existing boosting algorithms.

Also, it is surprisingly very fast, hence the word ‘Light’.

Refer to the article to know more about LightGBM: https://www.analyticsvidhya.com/blog/2017/06/which-

algorithm-takes-the-crown-light-gbm-vs-xgboost/

Python Code:

data = np.random.rand(500, 10) # 500 entities, each contains 10 features

label = np.random.randint(2, size=500) # binary target

train_data = lgb.Dataset(data, label=label)

test_data = train_data.create_valid('test.svm')

param = {'num_leaves':31, 'num_trees':100, 'objective':'binary'}

param['metric'] = 'auc'

num_round = 10
bst = lgb.train(param, train_data, num_round, valid_sets=[test_data])

bst.save_model('model.txt')

# 7 entities, each contains 10 features

data = np.random.rand(7, 10)
ypred = bst.predict(data)

R Code:

library(RLightGBM)
data(example.binary)
#Parameters

num_iterations <- 100

config <- list(objective = "binary", metric="binary_logloss,auc", learning_rate = 0.1,
num_leaves = 63, tree_learner = "serial", feature_fraction = 0.8, bagging_freq = 5,
bagging_fraction = 0.8, min_data_in_leaf = 50, min_sum_hessian_in_leaf = 5.0)

#Create data handle and booster

handle.data <- lgbm.data.create(x)

lgbm.data.setField(handle.data, "label", y)

handle.booster <- lgbm.booster.create(handle.data, lapply(config, as.character))

#Train for num_iterations iterations and eval every 5 steps

lgbm.booster.train(handle.booster, num_iterations, 5)

#Predict
pred <- lgbm.booster.predict(handle.booster, x.test)

#Test accuracy
sum(y.test == (y.pred > 0.5)) / length(y.test)

#Save model (can be loaded again via lgbm.booster.load(filename))

lgbm.booster.save(handle.booster, filename = "/tmp/model.txt")

If you’re familiar with the Caret package in R, this is another way of implementing the LightGBM.

require(caret)
require(RLightGBM)
data(iris)

model <-caretModel.LGBM()

fit <- train(Species ~ ., data = iris, method=model, verbosity = 0)

print(fit)
y.pred <- predict(fit, iris[,1:4])

library(Matrix)
model.sparse <- caretModel.LGBM.sparse()

#Generate a sparse matrix

mat <- Matrix(as.matrix(iris[,1:4]), sparse = T)
fit <- train(data.frame(idx = 1:nrow(iris)), iris$Species, method = model.sparse, matrix
= mat, verbosity = 0)
print(fit)

10.4. Catboost

CatBoost is a recently open-sourced machine learning algorithm from Yandex. It can easily integrate with deep
learning frameworks like Google’s TensorFlow and Apple’s Core ML.

The best part about CatBoost is that it does not require extensive data training like other ML models, and can
work on a variety of data formats; not undermining how robust it can be.

Make sure you handle missing data well before you proceed with the implementation.

Catboost can automatically deal with categorical variables without showing the type conversion error, which
helps you to focus on tuning your model better rather than sorting out trivial errors.
Learn more about Catboost from this article: https://www.analyticsvidhya.com/blog/2017/08/catboost-
automated-categorical-data/

Python Code:

import pandas as pd
import numpy as np

from catboost import CatBoostRegressor

#Read training and testing files

train = pd.read_csv("train.csv")
test = pd.read_csv("test.csv")

#Imputing missing values for both train and test

train.fillna(-999, inplace=True)
test.fillna(-999,inplace=True)

#Creating a training set for modeling and validation set to check model performance
X = train.drop(['Item_Outlet_Sales'], axis=1)
y = train.Item_Outlet_Sales

from sklearn.model_selection import train_test_split

X_train, X_validation, y_train, y_validation = train_test_split(X, y, train_size=0.7,

random_state=1234)
categorical_features_indices = np.where(X.dtypes != np.float)[0]

#importing library and building model

from catboost import CatBoostRegressormodel=CatBoostRegressor(iterations=50, depth=3,
learning_rate=0.1, loss_function='RMSE')

model.fit(X_train,
y_train,cat_features=categorical_features_indices,eval_set=(X_validation,
y_validation),plot=True)

submission = pd.DataFrame()

submission['Item_Identifier'] = test['Item_Identifier']
submission['Outlet_Identifier'] = test['Outlet_Identifier']
submission['Item_Outlet_Sales'] = model.predict(test)

R Code:

set.seed(1)

require(titanic)

require(caret)

require(catboost)

tt <- titanic::titanic_train[complete.cases(titanic::titanic_train),]

data <- as.data.frame(as.matrix(tt), stringsAsFactors = TRUE)

drop_columns = c("PassengerId", "Survived", "Name", "Ticket", "Cabin")

x <- data[,!(names(data) %in% drop_columns)]y <- data[,c("Survived")]

fit_control <- trainControl(method = "cv", number = 4,classProbs = TRUE)

grid <- expand.grid(depth = c(4, 6, 8),learning_rate = 0.1,iterations = 100, l2_leaf_reg
= 1e-3, rsm = 0.95, border_count = 64)

report <- train(x, as.factor(make.names(y)),method = catboost.caret,verbose = TRUE,

preProc = NULL,tuneGrid = grid, trControl = fit_control)

print(report)

importance <- varImp(report, scale = FALSE)

print(importance)

Projects
Now, its time to take the plunge and actually play with some other real datasets. So are you ready to take on the
challenge? Accelerate your data science journey with the following Practice Problems:

Practice Problem: Food Demand Predict the demand of meals for a

Forecasting Challenge meal delivery company

Practice Problem: HR Analytics Identify the employees most likely

Challenge to get promoted

Predict number of upvotes on a

Practice Problem: Predict Number of
query asked at an online question &
Upvotes
answer platform

End Notes
By now, I am sure, you would have an idea of commonly used machine learning algorithms. My sole intention
behind writing this article and providing the codes in R and Python is to get you started right away. If you are
keen to master machine learning, start right away. Take up problems, develop a physical understanding of the
process, apply these codes and see the fun!

Did you find this article useful ? Share your views and opinions in the comments section below.
8 Machine Learning Algorithms in Python – You Must Learn
by DataFlair Team · September 28, 2018

1. Objective
Previously, we discussed the techniques of machine learning with Python. Going deeper, today,
we will learn and implement 8 top Machine Learning Algorithms in Python.

Let’s begin the journey of Machine Learning Algorithms in Python Programming.

Machine Learning Algorithms in Python – You Must LEARN

2. Machine Learning Algorithms in Python

Followings are the Algorithms of Python Machine Learning:

a. Linear Regression
Linear regression is one of the supervised Machine learning algorithms in Python that observes
continuous features and predicts an outcome. Depending on whether it runs on a single variable or
on many features, we can call it simple linear regression or multiple linear regression.

This is one of the most popular Python ML algorithms and often under-appreciated. It assigns
optimal weights to variables to create a line ax+b to predict the output. We often use linear
regression to estimate real values like a number of calls and costs of houses based on continuous
variables. The regression line is the best line that fits Y=a*X+b to denote a relationship between
independent and dependent variables.
Do you know about Python Machine Learning Environment Setup
Let’s plot this for the diabetes dataset.

1. >>> import matplotlib.pyplot as plt

2. >>> import numpy as np
3. >>> from sklearn import datasets,linear_model
4. >>> from sklearn.metrics import mean_squared_error,r2_score
5. >>> diabetes=datasets.load_diabetes()
6. >>> diabetes_X=diabetes.data[:,np.newaxis,2]
7. >>> diabetes_X_train=diabetes_X[:-30] #splitting data into training and test sets
8. >>> diabetes_X_test=diabetes_X[-30:]
9. >>> diabetes_y_train=diabetes.target[:-30] #splitting targets into training and test sets
10. >>> diabetes_y_test=diabetes.target[-30:]
11. >>> regr=linear_model.LinearRegression() #Linear regression object
12. >>> regr.fit(diabetes_X_train,diabetes_y_train) #Use training sets to train the model

LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False)

1. >>> diabetes_y_pred=regr.predict(diabetes_X_test) #Make predictions

2. >>> regr.coef_

array([941.43097333])

1. >>> mean_squared_error(diabetes_y_test,diabetes_y_pred)

3035.0601152912695

1. >>> r2_score(diabetes_y_test,diabetes_y_pred) #Variance score

0.410920728135835

1. >>> plt.scatter(diabetes_X_test,diabetes_y_test,color ='lavender')

<matplotlib.collections.PathCollection object at 0x0584FF70>

1. >>> plt.plot(diabetes_X_test,diabetes_y_pred,color='pink',linewidth=3)

[<matplotlib.lines.Line2D object at 0x0584FF30>]

1. >>> plt.xticks(())

([], <a list of 0 Text xticklabel objects>)

1. >>> plt.yticks(())
([], <a list of 0 Text yticklabel objects>)

1. >>> plt.show()

Machine Learning Algorithms in Python – Linear Regression

Python Machine Learning – Data Preprocessing, Analysis & Visualization

b. Logistic Regression
Logistic regression is a supervised classification is unique Machine Learning algorithms in Python
that finds its use in estimating discrete values like 0/1, yes/no, and true/false. This is based on a
given set of independent variables. We use a logistic function to predict the probability of an event
and this gives us an output between 0 and 1.
Although it says ‘regression’, this is actually a classification algorithm. Logistic regression fits data
into a logit function and is also called logit regression. Let’s plot this.

1. >>> import numpy as np

2. >>> import matplotlib.pyplot as plt
3. >>> from sklearn import linear_model
4. >>> xmin,xmax=-7,7 #Test set; straight line with Gaussian noise
5. >>> n_samples=77
6. >>> np.random.seed(0)
7. >>> x=np.random.normal(size=n_samples)
 8. >>> y=(x>0).astype(np.float)
9. >>> x[x>0]*=3
10. >>> x+=.4*np.random.normal(size=n_samples)
11. >>> x=x[:,np.newaxis]
12. >>> clf=linear_model.LogisticRegression(C=1e4) #Classifier
13. >>> clf.fit(x,y)
14. >>> plt.figure(1,figsize=(3,4))
15. <Figure size 300x400 with 0 Axes>
16. >>> plt.clf()
17. >>> plt.scatter(x.ravel(),y,color='lavender',zorder=17)

<matplotlib.collections.PathCollection object at 0x057B0E10>

1. >>> x_test=np.linspace(-7,7,277)
2. >>> def model(x):
3. return 1/(1+np.exp(-x))
4. >>> loss=model(x_test*clf.coef_+clf.intercept_).ravel()
5. >>> plt.plot(x_test,loss,color='pink',linewidth=2.5)

[<matplotlib.lines.Line2D object at 0x057BA090>]

1. >>> ols=linear_model.LinearRegression()
2. >>> ols.fit(x,y)

LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False)

1. >>> plt.plot(x_test,ols.coef_*x_test+ols.intercept_,linewidth=1)

[<matplotlib.lines.Line2D object at 0x057BA0B0>]

1. >>> plt.axhline(.4,color='.4')

<matplotlib.lines.Line2D object at 0x05860E70>

1. >>> plt.ylabel('y')

Text(0,0.5,’y’)

1. >>> plt.xlabel('x')

Text(0.5,0,’x’)

1. >>> plt.xticks(range(-7,7))
2. >>> plt.yticks([0,0.4,1])
3. >>> plt.ylim(-.25,1.25)

(-0.25, 1.25)
 1. >>> plt.xlim(-4,10)

(-4, 10)

1. >>> plt.legend(('Logistic Regression','Linear Regression'),loc='lower right',fontsize='small')

<matplotlib.legend.Legend object at 0x057C89F0>

Do you know How to Split Train and Test Set in Python Machine Learning

1. >>> plt.show()

Machine Learning Algorithms in Python -Logistic

c. Decision Tree
A decision tree falls under supervised Machine Learning Algorithms in Python and comes of use
for both classification and regression- although mostly for classification. This model takes an
instance, traverses the tree, and compares important features with a determined conditional
statement. Whether it descends to the left child branch or the right depends on the result. Usually,
more important features are closer to the root.

Decision Tree, a Machine Learning algorithms in Python can work on both categorical and
continuous dependent variables. Here, we split a population into two or more homogeneous sets.
Let’s see the algorithm for this-

1. >>> from sklearn.cross_validation import train_test_split

2. >>> from sklearn.tree import DecisionTreeClassifier
3. >>> from sklearn.metrics import accuracy_score
 4. >>> from sklearn.metrics import classification_report
5. >>> def importdata(): #Importing data
6. balance_data=pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-'+
7. 'databases/balance-scale/balance-scale.data',
8. sep= ',', header = None)
9. print(len(balance_data))
10. print(balance_data.shape)
11. print(balance_data.head())
12. return balance_data
13. >>> def splitdataset(balance_data): #Splitting data
14. x=balance_data.values[:,1:5]
15. y=balance_data.values[:,0]
16. x_train,x_test,y_train,y_test=train_test_split(
17. x,y,test_size=0.3,random_state=100)
18. return x,y,x_train,x_test,y_train,y_test
19. >>> def train_using_gini(x_train,x_test,y_train): #Training with giniIndex
20. clf_gini = DecisionTreeClassifier(criterion = "gini",
21. random_state = 100,max_depth=3, min_samples_leaf=5)
22. clf_gini.fit(x_train,y_train)
23. return clf_gini
24. >>> def train_using_entropy(x_train,x_test,y_train): #Training with entropy
25. clf_entropy=DecisionTreeClassifier(
26. criterion = "entropy", random_state = 100,
27. max_depth = 3, min_samples_leaf = 5)
28. clf_entropy.fit(x_train,y_train)
29. return clf_entropy
30. >>> def prediction(x_test,clf_object): #Making predictions
31. y_pred=clf_object.predict(x_test)
32. print(f"Predicted values: {y_pred}")
33. return y_pred
34. >>> def cal_accuracy(y_test,y_pred): #Calculating accuracy
35. print(confusion_matrix(y_test,y_pred))
36. print(accuracy_score(y_test,y_pred)*100)
37. print(classification_report(y_test,y_pred))
38. >>> data=importdata()

625
(625, 5)
01234
0B1111
1R1112
2R1113
3R1114
4R1115

1. >>> x,y,x_train,x_test,y_train,y_test=splitdataset(data)
 2. >>> clf_gini=train_using_gini(x_train,x_test,y_train)
3. >>> clf_entropy=train_using_entropy(x_train,x_test,y_train)
4. >>> y_pred_gini=prediction(x_test,clf_gini)

Machine Learning Algorithms in Python – Decision Tree

1. >>> cal_accuracy(y_test,y_pred_gini)

[[ 0 6 7] [ 0 67 18] [ 0 19 71]] 73.40425531914893

Machine Learning Algorithms in Python – Decision Tree

1. >>> y_pred_entropy=prediction(x_test,clf_entropy)

Machine Learning Algorithms in Python – Decision Tree

1. >>> cal_accuracy(y_test,y_pred_entropy)

[[ 0 6 7] [ 0 63 22] [ 0 20 70]] 70.74468085106383

Machine Learning Algorithms in Python – Decision Tree Algorithm i

Let’s explore 4 Machine Learning Techniques with Python

d. Support Vector Machines (SVM)

SVM is a supervised classification is one of the most important Machines Learning algorithms in
Python, that plots a line that divides different categories of your data. In this ML algorithm, we
calculate the vector to optimize the line. This is to ensure that the closest point in each group lies
farthest from each other. While you will almost always find this to be a linear vector, it can be other
than that.
In this Python Machine Learning tutorial, we plot each data item as a point in an n-dimensional
space. We have n features and each feature has the value of a certain coordinate.
First, let’s plot a dataset.

1. >>> from sklearn.datasets.samples_generator import make_blobs

2. >>> x,y=make_blobs(n_samples=500,centers=2,
3. random_state=0,cluster_std=0.40)
4. >>> import matplotlib.pyplot as plt
5. >>> plt.scatter(x[:,0],x[:,1],c=y,s=50,cmap='plasma')

<matplotlib.collections.PathCollection object at 0x04E1BBF0>

1. >>> plt.show()
Machine Learning Algorithms in Python – SVM

1. >>> import numpy as np

2. >>> xfit=np.linspace(-1,3.5)
3. >>> plt.scatter(X[:, 0], X[:, 1], c=Y, s=50, cmap='plasma')

<matplotlib.collections.PathCollection object at 0x07318C90>

1. >>> for m, b, d in [(1, 0.65, 0.33), (0.5, 1.6, 0.55), (-0.2, 2.9, 0.2)]:
2. yfit = m * xfit + b
3. plt.plot(xfit, yfit, '-k')
4. plt.fill_between(xfit, yfit - d, yfit + d, edgecolor='none',
5. color='#AFFEDC', alpha=0.4)

[<matplotlib.lines.Line2D object at 0x07318FF0>] <matplotlib.collections.PolyCollection object at

0x073242D0>
[<matplotlib.lines.Line2D object at 0x07318B70>] <matplotlib.collections.PolyCollection object at
0x073246F0>
[<matplotlib.lines.Line2D object at 0x07324370>] <matplotlib.collections.PolyCollection object at
0x07324B30>

1. >>> plt.xlim(-1,3.5)

(-1, 3.5)

1. >>> plt.show()

Machine Learning Algorithms in Python – Support Vector Machine

Follow this link to know about Python PyQt5 Tutorial

e. Naive Bayes
Naive Bayes is a classification method which is based on Bayes’ theorem. This assumes
independence between predictors. A Naive Bayes classifier will assume that a feature in a class is
unrelated to any other. Consider a fruit. This is an apple if it is round, red, and 2.5 inches in
diameter. A Naive Bayes classifier will say these characteristics independently contribute to the
probability of the fruit being an apple. This is even if features depend on each other.
For very large data sets, it is easy to build a Naive Bayesian model. Not only is this model very
simple, it performs better than many highly sophisticated classification methods. Let’s build this.

1. >>> from sklearn.naive_bayes import GaussianNB

2. >>> from sklearn.naive_bayes import MultinomialNB
3. >>> from sklearn import datasets
4. >>> from sklearn.metrics import confusion_matrix
5. >>> from sklearn.model_selection import train_test_split
6. >>> iris=datasets.load_iris()
7. >>> x=iris.data
8. >>> y=iris.target
9. >>> x_train,x_test,y_train,y_test=train_test_split(x,y,test_size=0.3,random_state=0)
10. >>> gnb=GaussianNB()
11. >>> mnb=MultinomialNB()
12. >>> y_pred_gnb=gnb.fit(x_train,y_train).predict(x_test)
13. >>> cnf_matrix_gnb = confusion_matrix(y_test, y_pred_gnb)
14. >>> cnf_matrix_gnb

array([[16, 0, 0],
[ 0, 18, 0],
[ 0, 0, 11]], dtype=int64)

1. >>> y_pred_mnb = mnb.fit(x_train, y_train).predict(x_test)

2. >>> cnf_matrix_mnb = confusion_matrix(y_test, y_pred_mnb)
3. >>> cnf_matrix_mnb

array([[16, 0, 0],
[ 0, 0, 18],
[ 0, 0, 11]], dtype=int64)

f. kNN (k-Nearest Neighbors)

This is a Python Machine Learning algorithms for classification and regression- mostly for
classification. This is a supervised learning algorithm that considers different centroids and uses a
usually Euclidean function to compare distance. Then, it analyzes the results and classifies each
point to the group to optimize it to place with all closest points to it. It classifies new cases using a
majority vote of k of its neighbors. The case it assigns to a class is the one most common among its
K nearest neighbors. For this, it uses a distance function.
i. Training and testing on the entire dataset
1. >>> from sklearn.datasets import load_iris
2. >>> iris=load_iris()
3. >>> x=iris.data
4. >>> y=iris.target
5. >>> from sklearn.linear_model import LogisticRegression
6. >>> logreg=LogisticRegression()
7. >>> logreg.fit(x,y)

LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,

intercept_scaling=1, max_iter=100, multi_class=’ovr’, n_jobs=1,
penalty=’l2′, random_state=None, solver=’liblinear’, tol=0.0001,
verbose=0, warm_start=False)

1. >>> logreg.predict(x)

array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
2, 1, 1, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])

1. >>> y_pred=logreg.predict(x)
2. >>> len(y_pred)

150

1. >>> from sklearn import metrics

2. >>> metrics.accuracy_score(y,y_pred)

0.96

1. >>> from sklearn.neighbors import KNeighborsClassifier

2. >>> knn=KNeighborsClassifier(n_neighbors=5)
3. >>> knn.fit(x,y)

KNeighborsClassifier(algorithm=’auto’, leaf_size=30, metric=’minkowski’,

metric_params=None, n_jobs=1, n_neighbors=5, p=2,
weights=’uniform’)

1. >>> y_pred=knn.predict(x)
2. >>> metrics.accuracy_score(y,y_pred)

0.9666666666666667
 1. >>> knn=KNeighborsClassifier(n_neighbors=1)
2. >>> knn.fit(x,y)

KNeighborsClassifier(algorithm=’auto’, leaf_size=30, metric=’minkowski’,

metric_params=None, n_jobs=1, n_neighbors=1, p=2,
weights=’uniform’)

1. >>> y_pred=knn.predict(x)
2. >>> metrics.accuracy_score(y,y_pred)

1.0

ii. Splitting into train/test

1. >>> x.shape

(150, 4)

1. >>> y.shape

(150,)

1. >>> from sklearn.cross_validation import train_test_split

2. >>> x.shape

(150, 4)

1. >>> y.shape

(150,)

1. >>> from sklearn.cross_validation import train_test_split

2. >>> x_train,x_test,y_train,y_test=train_test_split(x,y,test_size=0.4,random_state=4)
3. >>> x_train.shape

(90, 4)

1. >>> x_test.shape

(60, 4)

1. >>> y_train.shape

(90,)

1. >>> y_test.shape
(60,)

1. >>> logreg=LogisticRegression()
2. >>> logreg.fit(x_train,y_train)
3. >>> y_pred=knn.predict(x_test)
4. >>> metrics.accuracy_score(y_test,y_pred)

0.9666666666666667

1. >>> knn=KNeighborsClassifier(n_neighbors=5)
2. >>> knn.fit(x_train,y_train)

KNeighborsClassifier(algorithm=’auto’, leaf_size=30, metric=’minkowski’,

metric_params=None, n_jobs=1, n_neighbors=5, p=2,
weights=’uniform’)

1. >>> y_pred=knn.predict(x_test)
2. >>> metrics.accuracy_score(y_test,y_pred)

0.9666666666666667

1. >>> k_range=range(1,26)
2. >>> scores=[]
3. >>> for k in k_range:
4. knn = KNeighborsClassifier(n_neighbors=k)
5. knn.fit(x_train, y_train)
6. y_pred = knn.predict(x_test)
7. scores.append(metrics.accuracy_score(y_test, y_pred))
8. >>> scores

[0.95, 0.95, 0.9666666666666667, 0.9666666666666667, 0.9666666666666667,

0.9833333333333333, 0.9833333333333333, 0.9833333333333333, 0.9833333333333333,
0.9833333333333333, 0.9833333333333333, 0.9833333333333333, 0.9833333333333333,
0.9833333333333333, 0.9833333333333333, 0.9833333333333333, 0.9833333333333333,
0.9666666666666667, 0.9833333333333333, 0.9666666666666667, 0.9666666666666667,
0.9666666666666667, 0.9666666666666667, 0.95, 0.95]

1. >>> import matplotlib.pyplot as plt

2. >>> plt.plot(k_range,scores)

[<matplotlib.lines.Line2D object at 0x05FDECD0>]

1. >>> plt.xlabel('k for kNN')

Text(0.5,0,’k for kNN’)

1. >>> plt.ylabel('Testing Accuracy')

Text(0,0.5,’Testing Accuracy’)

1. >>> plt.show()

Machine Learning Algorithms in Python – k-Nearest Neighbors

Read about Python Statistics – p-Value, Correlation, T-test, KS Test

g. k-Means
k-Means is an unsupervised algorithm that solves the problem of clustering. It classifies data using
a number of clusters. The data points inside a class are homogeneous and heterogeneous to peer
groups.

1. >>> import numpy as np

2. >>> import matplotlib.pyplot as plt
3. >>> from matplotlib import style
4. >>> style.use('ggplot')
5. >>> from sklearn.cluster import KMeans
6. >>> x=[1,5,1.5,8,1,9]
7. >>> y=[2,8,1.7,6,0.2,12]
8. >>> plt.scatter(x,y)

<matplotlib.collections.PathCollection object at 0x0642AF30>

 1. >>> x=np.array([[1,2],[5,8],[1.5,1.8],[8,8],[1,0.6],[9,11]])
2. >>> kmeans=KMeans(n_clusters=2)
3. >>> kmeans.fit(x)

KMeans(algorithm=’auto’, copy_x=True, init=’k-means++’, max_iter=300,

n_clusters=2, n_init=10, n_jobs=1, precompute_distances=’auto’,
random_state=None, tol=0.0001, verbose=0)

1. >>> centroids=kmeans.cluster_centers_
2. >>> labels=kmeans.labels_
3. >>> centroids

array([[1.16666667, 1.46666667],
[7.33333333, 9. ]])

1. >>> labels

array([0, 1, 0, 1, 0, 1])

1. >>> colors=['g.','r.','c.','y.']
2. >>> for i in range(len(x)):
3. print(x[i],labels[i])
4. plt.plot(x[i][0],x[i][1],colors[labels[i]],markersize=10)

[1. 2.] 0
[<matplotlib.lines.Line2D object at 0x0642AE10>] [5. 8.] 1
[<matplotlib.lines.Line2D object at 0x06438930>] [1.5 1.8] 0
[<matplotlib.lines.Line2D object at 0x06438BF0>] [8. 8.] 1
[<matplotlib.lines.Line2D object at 0x06438EB0>] [1. 0.6] 0
[<matplotlib.lines.Line2D object at 0x06438FB0>] [ 9. 11.] 1
[<matplotlib.lines.Line2D object at 0x043B1410>]

1. >>> plt.scatter(centroids[:,0],centroids[:,1],marker='x',s=150,linewidths=5,zorder=10)

<matplotlib.collections.PathCollection object at 0x043B14D0>

1. >>> plt.show()
Machine Learning Algorithms in Python – K-Means

Have a Look at Python Descriptive Statistics – Measuring Central Tendency

h. Random Forest
A random forest is an ensemble of decision trees. In order to classify every new object based on its
attributes, trees vote for class- each tree provides a classification. The classification with the most
votes wins in the forest.

1. >>> import numpy as np

2. >>> import pylab as pl
3. >>> x=np.random.uniform(1,100,1000)
4. >>> y=np.log(x)+np.random.normal(0,.3,1000)
5. >>> pl.scatter(x,y,s=1,label='log(x) with noise')

<matplotlib.collections.PathCollection object at 0x0434EC50>

1. >>> pl.plot(np.arange(1,100),np.log(np.arange(1,100)),c='b',label='log(x) true function')

[<matplotlib.lines.Line2D object at 0x0434EB30>]

1. >>> pl.xlabel('x')

Text(0.5,0,’x’)

1. >>> pl.ylabel('f(x)=log(x)')

Text(0,0.5,’f(x)=log(x)’)
 1. >>> pl.legend(loc='best')

<matplotlib.legend.Legend object at 0x04386450>

1. >>> pl.title('A basic log function')

Text(0.5,1,’A basic log function’)

1. >>> pl.show()

Machine Learning Algorithms in Python – Random Forest

1. >>> from sklearn.datasets import load_iris

2. >>> from sklearn.ensemble import RandomForestClassifier
3. >>> import pandas as pd
4. >>> import numpy as np
5. >>> iris=load_iris()
6. >>> df=pd.DataFrame(iris.data,columns=iris.feature_names)
7. >>> df['is_train']=np.random.uniform(0,1,len(df))<=.75
8. >>> df['species']=pd.Categorical.from_codes(iris.target,iris.target_names)
9. >>> df.head()

sepal length (cm) sepal width (cm) … is_train species

0 5.1 3.5 … True setosa
1 4.9 3.0 … True setosa
2 4.7 3.2 … True setosa
3 4.6 3.1 … True setosa
4 5.0 3.6 … False setosa
[5 rows x 6 columns]

1. >>> train,test=df[df['is_train']==True],df[df['is_train']==False]
2. >>> features=df.columns[:4]
3. >>> clf=RandomForestClassifier(n_jobs=2)
4. >>> y,_=pd.factorize(train['species'])
5. >>> clf.fit(train[features],y)

RandomForestClassifier(bootstrap=True, class_weight=None, criterion=’gini’,

max_depth=None, max_features=’auto’, max_leaf_nodes=None,
min_impurity_decrease=0.0, min_impurity_split=None,
min_samples_leaf=1, min_samples_split=2,
min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=2,
oob_score=False, random_state=None, verbose=0,
warm_start=False)

1. >>> preds=iris.target_names[clf.predict(test[features])]
2. >>> pd.crosstab(test['species'],preds,rownames=['actual'],colnames=['preds'])

preds setosa versicolor virginica

actual
setosa 12 0 0
versicolor 0 17 2
virginica 0 1 15

So, this was all about Machine Learning Algorithms in Python Tutorial. Hope you like our
explanation.

3. Conclusion
Hence, today we discussed eight important Python Machine Learning Algorithms. Which one do
you think bears the most potential? Drop your suggestions in the comments below. We will surely
get back to you!
Related Topic- Python NumPy Tutorial
For reference
Statistical Data Analysis in Python
Share64
Tags: IPython, Jupyter, Pandas, Python, Statistical Analysis

This tutorial will introduce the use of Python for statistical data analysis, using data stored as Pandas DataFrame
objects, taking the form of a set of IPython notebooks.

By Christopher Fonnesbeck, Vanderbilt University School of Medicine.

Editor's note: This tutorial was originally published as course instructional material, and may contain out-of-
context references to other courses therein; this takes nothing away from the validity or usefulness of the
material.

Description

This tutorial will introduce the use of Python for statistical data analysis, using data stored as Pandas DataFrame
objects. Much of the work involved in analyzing data resides in importing, cleaning and transforming data in
preparation for analysis. Therefore, the first half of the course is comprised of a 2-part overview of basic and
intermediate Pandas usage that will show how to effectively manipulate datasets in memory. This includes tasks
like indexing, alignment, join/merge methods, date/time types, and handling of missing data. Next, we will cover
plotting and visualization using Pandas and Matplotlib, focusing on creating effective visual representations of
your data, while avoiding common pitfalls. Finally, participants will be introduced to methods for statistical data
modeling using some of the advanced functions in Numpy, Scipy and Pandas. This will include fitting your data
to probability distributions, estimating relationships among variables using linear and non-linear models, and a
brief introduction to bootstrapping methods. Each section of the tutorial will involve hands-on manipulation and
analysis of sample datasets, to be provided to attendees in advance.
The target audience for the tutorial includes all new Python users, though we recommend that users also attend
the NumPy and IPython session in the introductory track.

Student Instructions

For students familiar with Git, you may simply clone this repository to obtain all the materials (iPython
notebooks and data) for the tutorial. Alternatively, you may download a zip file containing the materials. A third
option is to simply view static notebooks by clicking on the titles of each section below.

Outline

Introduction to Pandas

 Importing data
 Series and DataFrame objects
 Indexing, data selection and subsetting
 Hierarchical indexing
 Reading and writing files
 Sorting and ranking
 Missing data
 Data summarization

Data Wrangling with Pandas

 Date/time types
 Merging and joining DataFrame objects
 Concatenation
 Reshaping DataFrame objects
 Pivoting
 Data transformation
 Permutation and sampling
 Data aggregation and GroupBy operations

Plotting and Visualization

 Plotting in Pandas vs Matplotlib

 Bar plots
 Histograms
 Box plots
 Grouped plots
 Scatterplots
 Trellis plots

Statistical Data Modeling

 Statistical modeling
 Fitting data to probability distributions
 Fitting regression models
 Model selection
 Bootstrapping
Required Packages

 Python 2.7 or higher (including Python 3)

 pandas >= 0.11.1 and its dependencies
 NumPy >= 1.6.1
 matplotlib >= 1.0.0
 pytz
 IPython >= 0.12
 pyzmq
 tornado

Optional: statsmodels, xlrd and openpyxl

For students running the latest version of Mac OS X (10.8), the easiest way to obtain all the packages is to
install the Scipy Superpack which works with Python 2.7.2 that ships with OS X.

Otherwise, another easy way to install all the necessary packages is to use Continuum Analytics' Anaconda.

Statistical Reading List

The Ecological Detective: Confronting Models with Data, Ray Hilborn and Marc Mangel

Though targeted to ecologists, Mangel and Hilborn identify key methods that scientists can use to build useful
and credible models for their data. They don't shy away from the math, but the book is very readable and
example-laden.

Data Analysis Using Regression and Multilevel/Hierarchical Models, Andrew Gelman and Jennifer Hill

The go-to reference for applied hierarchical modeling.

The Elements of Statistical Learning, Hastie, Tibshirani and Friedman

A comprehensive machine learning guide for statisticians.

A First Course in Bayesian Statistical Methods, Peter Hoff

An excellent, approachable book to get started with Bayesian methods.

Regression Modeling Strategies, Frank Harrell

Frank Harrell's bag of tricks for regression modeling. I pull this off the shelf every week.

Bio: Christopher Fonnesbeck is an Assistant Professor in the Department of Biostatistics at the Vanderbilt
University School of Medicine. He specializes in computational statistics, Bayesian methods, meta-analysis,
and applied decision analysis. He originally hails from Vancouver, BC and received his Ph.D. from the
University of Georgia.

Original. Reposted with permission.

7 Steps to Mastering Machine Learning With Python
There are many Python machine learning resources freely available online. Where to begin? How to proceed?
Go from zero to Python machine learning hero in 7 steps!

By Matthew Mayo, KDnuggets.

Getting started. Two of the most de-motivational words in the English language. The first step is often the
hardest to take, and when given too much choice in terms of direction it can often be debilitating.

Where to begin?

This post aims to take a newcomer from minimal knowledge of machine learning in Python all the way to
knowledgeable practitioner in 7 steps, all while using freely available materials and resources along the way.
The prime objective of this outline is to help you wade through the numerous free options that are available;
there are many, to be sure, but which are the best? Which complement one another? What is the best order in
which to use selected resources?

Moving forward, I make the assumption that you are not an expert in:

 Machine learning
 Python
 Any of Python's machine learning, scientific computing, or data analysis libraries

It would probably be helpful to have some basic understanding of one or both of the first 2 topics, but even that
won't be necessary; some extra time spent on the earlier steps should help compensate.
Step 1: Basic Python Skills

If we intend to leverage Python in order to perform machine learning, having some base understanding of Python
is crucial. Fortunately, due to its widespread popularity as a general purpose programming language, as well as
its adoption in both scientific computing and machine learning, coming across beginner's tutorials is not very
difficult. Your level of experience in both Python and programming in general are crucial to choosing a starting
point.

First, you need Python installed. Since we will be using scientific computing and machine learning packages at
some point, I suggest that you install Anaconda. It is an industrial-strength Python implementation for Linux,
OSX, and Windows, complete with the required packages for machine learning, including numpy, scikit-learn,
and matplotlib. It also includes iPython Notebook, an interactive environment for many of our tutorials. I would
suggest Python 2.7, for no other reason than it is still the dominant installed version.

If you have no knowledge of programming, my suggestion is to start with the following free online book, then
move on to the subsequent materials:

 Python The Hard Way, by Zed A. Shaw

If you have experience in programming but not with Python in particular, or if your Python is elementary, I
would suggest one or both of the following:

 Google Developers Python Course (highly recommended for visual learners)

 An Introduction to Python for Scientific Computing (from UCSB Engineering), by M. Scott Shell (a
great scientific Python intro ~60 pages)

And for those looking for a 30 minute crash course in Python, here you go:

 Learn X in Y Minutes (X = Python)

Of course, if you are an experienced Python programmer you will be able to skip this step. Even if so, I suggest
keeping the very readable Python documentation handy.

Step 2: Foundational Machine Learning Skills

KDnuggets' own Zachary Lipton has pointed out that there is a lot of variation in what people consider a "data
scientist." This actually is a reflection of the field of machine learning, since much of what data scientists do
involves using machine learning algorithms to varying degrees. Is it necessary to intimately understand kernel
methods in order to efficiently create and gain insight from a support vector machine model? Of course not. Like
almost anything in life, required depth of theoretical understanding is relative to practical application. Gaining an
intimate understanding of machine learning algorithms is beyond the scope of this article, and generally requires
substantial amounts of time investment in a more academic setting, or via intense self-study at the very least.

The good news is that you don't need to possess a PhD-level understanding of the theoretical aspects of machine
learning in order to practice, in the same manner that not all programmers require a theoretical computer science
education in order to be effective coders.

Andrew Ng's Coursera course often gets rave reviews for its content; my suggestion, however, is to browse the
course notes compiled by a former student of the online course's previous incarnation. Skip over the Octave-
specific notes (a Matlab-like language unrelated to our Python pursuits). Be warned that these are not "official"
notes, but do seem to capture the relevant content from Andrew's course material. Of course, if you have the
time and interest, now would be the time to take Andrew Ng's Machine Learning course on Coursera.

 Unofficial Andrew Ng course notes

There all sorts of video lectures out there if you prefer, alongside Ng's course mentioned above. I'm a fan of
Tom Mitchell, so here's a link to his recent lecture videos (along with Maria-Florina Balcan), which I find
particularly approachable:

 Tom Mitchell Machine Learning Lectures

You don't need all of the notes and videos at this point. A valid strategy involves moving forward to particular
exercises below, and referencing applicable sections of the above notes and videos when appropriate. For
example, when you come across an exercise implementing a regression model below, read the appropriate
regression section of Ng's notes and/or view Mitchell's regression videos at that time.

Step 3: Scientific Python Packages Overview

Alright. We have a handle on Python programming and understand a bit about machine learning. Beyond
Python there are a number of open source libraries generally used to facilitate practical machine learning. In
general, these are the main so-called scientific Python libraries we put to use when performing elementary
machine learning tasks (there is clearly subjectivity in this):

 numpy - mainly useful for its N-dimensional array objects

 pandas - Python data analysis library, including structures such as dataframes
 matplotlib - 2D plotting library producing publication quality figures
 scikit-learn - the machine learning algorithms used for data analysis and data mining tasks

A good approach to learning these is to cover this material:

 Scipy Lecture Notes, by Gaël Varoquaux, Emmanuelle Gouillart, and Olav Vahtras

This pandas tutorial is good, and to the point:

 10 Minutes to Pandas

You will see some other packages in the tutorials below, including, for example, Seaborn, which is a data
visualization library based on matplotlib. The aforementioned packages are (again, subjectively) the core of a
wide array of machine learning tasks in Python; however, understanding them should let you adapt to additional
and related packages without confusion when they are referenced in the following tutorials.

Now, on to the good stuff...

Step 4: Getting Started with Machine Learning in Python

 Python. Check.
 Machine learning fundamentals. Check.
 Numpy. Check.
 Pandas. Check.
 Matplotlib. Check.
The time has come. Let's start implementing machine learning algorithms with Python's de facto standard
machine learning library, scikit-learn.

The scikit-learn flow chart.

Many of the following tutorials and exercises will be driven by the iPython (Jupyter) Notebook, which is an
interactive environment for executing Python. These iPython notebooks can optionally be viewed online or
downloaded and interacted with locally on your own computer.

 iPython Notebook Overview from Stanford

Also note that the tutorials below are from a number of online sources. All Notebooks have been attributed to
the authors; if, for some reason, you find that someone has not been properly credited for their work, please let
me know and the situation will be rectified ASAP. In particular, I would like to tip my hat to Jake VanderPlas,
Randal Olson, Donne Martin, Kevin Markham, and Colin Raffel for their fantastic freely-available resources.

Our first tutorials for getting our feet wet with scikit-learn follow. I suggest doing all of these in order before
moving to the following steps.

A general introduction to scikit-learn, Python's most-used general purpose machine learning library, covering
the k-nearest neighbors algorithm:

 An Introduction to scikit-learn, by Jake VanderPlas

A more in-depth and expanded introduction, including a starter project with a well-known dataset from start to
finish:

 Example Machine Learning Notebook, by Randal Olson

A focus on strategies for evaluating different models in scikit-learn, covering train/test dataset splits:

 Model Evaluation, by Kevin Markham

Step 5: Machine Learning Topics with Python

With a foundation having been laid in scikit-learn, we can move on to some more in-depth explorations of the
various common, and useful, algorithms. We start with k-means clustering, one of the most well-known
machine learning algorithms. It is a simple and often effective method for solving unsupervised learning
problems:

 k-means Clustering, by Jake VanderPlas

Next, we move back toward classification, and take a look at one of the most historically popular classification
methods:

 Decision Trees via The Grimm Scientist

From classification, we look at continuous numeric prediction:

 Linear Regression, by Jake VanderPlas

We can then leverage regression for classification problems, via logistic regression:

 Logistic Regression, by Kevin Markham

Step 6: Advanced Machine Learning Topics with Python

We've gotten our feet wet with scikit-learn, and now we turn our attention to some more advanced topics. First
up are support vector machines, a not-necessarily-linear classifier relying on complex transformations of data
into higher dimensional space.

 Support Vector Machines, by Jake VanderPlas

Next, random forests, an ensemble classifier, are examined via a Kaggle Titanic Competition walk-through:

 Kaggle Titanic Competition (with Random Forests), by Donne Martin

Dimensionality reduction is a method for reducing the number of variables being considered in a problem.
Principal Component Analysis is a particular form of unsupervised dimensionality reduction:

 Dimensionality Reduction, by Jake VanderPlas

Before moving on to the final step, we can take a moment to consider that we have come a long way in a
relatively short period of time.

Using Python and its machine learning libraries, we have covered some of the most common and well-known
machine learning algorithms (k-nearest neighbors, k-means clustering, support vector machines), investigated a
powerful ensemble technique (random forests), and examined some additional machine learning support tasks
(dimensionality reduction, model validation techniques). Along with some foundational machine learning skills,
we have started filling a useful toolkit for ourselves.

We will add one more in-demand tool to that kit before wrapping up.

Step 7: Deep Learning in Python

 The learning is deep.

Deep learning is everywhere! Deep learning builds on neural network research going back several decades, but
recent advances dating to the past several years have dramatically increased the perceived power of, and general
interest in, deep neural networks. If you are unfamiliar with deep learning, KDnuggets has many articles detailing
the numerous recent innovations, accomplishments, and accolades of the technology.

This final step does not purport to be a deep learning clinic of any sort; we will take a look at a few simple network
implementations in 2 of the leading contemporary Python deep learning libraries. For those interested in digging
deeper into deep learning, I recommend starting with the following free online book:

 Neural Networks and Deep Learning by Michael Nielsen

Theano

Theano is the first Python deep learning library we will look at. From the authors:

Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving
multi-dimensional arrays efficiently.

The following introductory tutorial on deep learning in Theano is lengthy, but it is quite good, very descriptive,
and heavily-commented:

 Theano Deep Learning Tutorial, by Colin Raffel

Caffe

The other library we will test drive is Caffe. Again, from the authors:
Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by the
Berkeley Vision and Learning Center (BVLC) and by community contributors.

This tutorial is the cherry on the top of this article. While we have undertaken a few interesting examples above,
none likely compete with the following, which is implementing Google's #DeepDream using Caffe. Enjoy this
one! After understanding the tutorial, play around with it to get your processors dreaming on their own.

 Dreaming Deep with Caffe via Google's GitHub

I didn't promise it would be quick or easy, but if you put the time in and follow the above 7 steps, there is no
reason that you won't be able to claim reasonable proficiency and understanding in a number of machine learning
algorithms and their implementation in Python using its popular libraries, including some of those on the cutting
edge of current deep learning research.

Bio: Matthew Mayo is a computer science graduate student currently working on his thesis parallelizing machine
learning algorithms. He is also a student of data mining, a data enthusiast, and an aspiring machine learning
scientist.

Related:

 Top 20 Data Science MOOCs

 60+ Free Books on Big Data, Data Science, Data Mining, Machine Learning, Python, R, and more
 15 Mathematics MOOCs for Data Science