Journal of Petroleum Science and Engineering 175 (2019) 604–623

Contents lists available at ScienceDirect

Journal of Petroleum Science and Engineering

journal homepage: www.elsevier.com/locate/petrol

Application of hybrid artificial neural networks for predicting rate of T

penetration (ROP): A case study from Marun oil field
Seyed Babak Ashrafia, Mohammad Anemangelyb, Mohammad Sabaha,
Mohammad Javad Ameria,∗
a
Department of Petroleum Engineering, Amirkabir University of Technology (Tehran Polytechnic), 424 Hafez Avenue, Tehran, 15875-4413, Iran
b
School of Mining, Petroleum and Geophysics Engineering, Shahrood University of Technology, Shahrood, Iran

ARTICLE INFO ABSTRACT

Keywords: Rate of Penetration (ROP) can be considered as a crucial factor in optimization and cost minimization of drilling
Rate of penetration operations. In order to predict ROP with satisfactory precision, some hybrid artificial intelligence models were
Artificial neural network developed in the present research. To achieve this, dataset including petrophysical logs and drilling data was
Feature selection collected from a vertical well drilled in Marun oil field where is in southwest corner of Iran. Then, overall noise
Evolutionary algorithms
of the collected data was reduced with implementing Savitzky–Golay (SG) smoothing filter. In the next stage, a
feature selection method with application of genetic algorithm was employed for choosing the unrivaled features
and lessening the input vectors. Regarding results of the feature selection method, eight parameters including
weight on bit, bit rotational speed, pump flow rate, pump pressure, pore pressure, gamma ray, density log and
shear wave velocity were identified as the most influential parameters in drilling rate. After that, eight hybrid
artificial neural networks (ANN) were developed and trained by four evolutionary algorithms. These algorithms
are genetic algorithm (GA), particle swarm optimization (PSO), biogeography-based optimizer (BBO) and im-
perialist competitive algorithm (ICA). After model developments, to provide an objective assessment of per-
formances of the proposed hybrid models, their results were compared with results of conventional ANN models
and two multiple regression methods (NLMR and LMR) using different performance indices such as root mean
square error (RMSE), variance account for (VAF), and performance index (PI). Results showed that PSO-multi-
layer perception (PSO-MLP) and PSO-radial basis function (PSO-RBF) neural networks with RMSE of 1.12 and
1.4, respectively yielded the highest performance in prediction comparing to results of other developed models.
Concluding remark is that hybrid ANNs exhibit much more efficiency and reliability than conventional ANNs
and regression methods in terms of ROP prediction, indicating superiority of evolutionary algorithms over back
propagation (BP) methods in the ANNs training.

1. Introduction
Rate of penetration (ROP) is a direct measurement of drilling job
progression as it shows how much time it takes to drill a finite interval
of depth. To monitor and enhance drilling operations, drilling engineers
must consider ROP which itself relies upon many other independent
parameters such as geological and geotechnical parameters, operational
factors, and the machine characteristics. However, there is not a
transparent relationship between ROP and mentioned parameters,
rendering the behavior of ROP to be non-linear and difficult to predict
(Akgun, 2007; Darbor et al., 2017). Nevertheless, impact of different
parameters on drilling rate can be modeled in various correlations such
as non-linear, complex and stochastic relationships (Moradi et al.,
2010).
Over the past few decades, many researchers tried to formulate
these parameters and develop some models (Warren, 1987; Winters
et al., 1987; BINGHAM, 1965; Hareland and Rampersad, 1994;
Motahhari et al., 2010). These physics-based models are associated with
some limitations such as the requirement for auxiliary data (bit prop-
erties, mud properties, bit design, etc.), utilizing empirical coefficients
and their high dependency to lithology (since the empirical coefficients
heavily rely on lithology) (Hegde et al., 2017, 2018; Mendes et al.,
2007), which make them useless for prediction of drilling rate. There-
fore, other methods should be considered to approximate the ROP with
least possible errors. Novel solutions, in particular, like application of
smart systems, data mining methods, evolutionary algorithms and their

∗
Corresponding author.
E-mail address: Ameri@aut.ac.ir (M.J. Ameri).

https://doi.org/10.1016/j.petrol.2018.12.013
Received 31 July 2018; Received in revised form 24 November 2018; Accepted 5 December 2018
Available online 07 December 2018
0920-4105/ © 2018 Elsevier B.V. All rights reserved.
S.B. Ashrafi et al.
hybrids models have led into a robust performance in function ap-
proximation. Also, a broad range of uncertainty can be endured by
these approaches; Due to this fact, they have recently become highly
popular in various research fields (Yilmaz and Kaynar, 2011).
A lot of studies have been conducted in the drillling industry using
artificial intelligence methods. Jacintoa et al. (2013) (Jacintoa et al.,
2013) employed dynamic evolving neural-fuzzy inference system
(DENFIS) and Bayesian inference approach to predict and optimize the
ROP. Bataee et al. (2014) (Bataee et al., 2014) used artificial neural
networks and analytical methods for predicting the drilling rate and
optimizing the parameters affecting ROP. Basarir et al. (2014) (Basarir
et al., 2014) investigated the performance of linear regression, non-
linear regression and ANFIS model in terms of ROP prediction. Their
results showed superiority of ANFIS method over regression methods.
Bodaghi et al. (2015) (Bodaghi et al., 2015) established a relationship
between drilling variables and rate of penetration employing support
vector regression (SVR) which was optimized by cuckoo search algo-
rithm and genetic algorithm. Hegde et al. (2015) (Hegde et al., 2015)
predicted the rate of penetration by application of statistical learning
techniques such as trees, bagged trees and random forests which were
used for a data set with nine predictors. Moraveji and Naderi (2016)
(Moraveji and Naderi, 2016) evaluated the impact of drilling para-
meters on ROP using response surface methodology and established a
mathematical relationship between drilling rate and those parameters
with the greatest influence on ROP. Kahraman (2016) (Kahraman,
2016) found that neural network models are more reliable than the
regression models for predicting the drilling penetration rate. Jiang and
Samuel (2016) (Jiang and Samuel, 2016) trained a neural network with
application of ant colony optimization (ACO) algorithm eventuating the
optimum values for drilling rate. Shi et al. (2016) (Shi et al., 2016)
utilized a number of smart systems such as extreme learning machine,
upper-layer solution aware model and neural network. Their outcomes
showed that all of these approaches are adequately proper techniques
to predict the ROP. Khandelwal and Armaghani et al. (2016)
(Khandelwal and Armaghani, 2016) used multiple regression, neural
network (NN) and hybrid genetic algorithm (GA-NN) models to esti-
mate drilling rate index (DRI) based on rock's physical properties. Their
results showed that GA-NN technique yields better performance in
prediction compared to other implemented models. Bezminabadi et al.
(2017) (Bezminabadi et al., 2017) have estimated ROP employing
multiple nonlinear regression (MNR) and artificial neural network. In
their study, physical and mechanical properties of rocks have been in-
volved with operational parameters to improve performance of pre-
dicting models. Ansari et al. (2017) (Ansari et al., 2017) examined ROP
modeling utilizing committee support vector regression based on im-
perialist competitive algorithm (CSVR-ICA). Eskandarian et al. (2017)
(Eskandarian et al., 2017) applied data mining methods including
random forest and monotone multi-layer perceptron (MON-MLP) to
estimate the ROP. They implemented “fscaret” package (Szlek and
Mendyk, 2015) for feature ranking and determining the most influen-
cing variables on ROP. Hegde et al. (2017) (10) used physics-based and
data-driven models in ROP prediction which ended up with the con-
clusion that data-driven models showed better fitness compared to
traditional models. In a study conducted by Anemangely et al. (2018)
(Anemangely et al., 2018), two hybrid ANNs were developed by using
particle swarm optimization (PSO) and cuckoo optimization algorithm
(COA) as training functions to provide accurate prediction of ROP.
Yavari et al. (2018) (Yavari et al., 2018) investigated superiority of
different smart systems such as ANFIS and ANN over recently estab-
lished analytical ROP models. Their results showed that developed
smart systems are shown much more reliable results than analytical
models for drilling rate prediction. Regarding ANNs, this methods have
been used in a variety of different sections in the oil industry. Divisions
like characterizing reservoir-related properties (Vaferi et al., 2014,
2015, 2016; Ghaffarian et al., 2014), defining lithological properties
and analyzing source rock characteristics (Martin et al., 2017; Bolandi
605
Journal of Petroleum Science and Engineering 175 (2019) 604–623
et al., 2017), and enhancing oil recovery and production (Le Van and
Chon, 2017; Xue et al., 2014; Shan et al., 2018) are examples of ANN
implementations.
As it is obvious from the literature, many researchers utilized the
neural networks in ROP estimation. However, several scholars (Lee
et al., 1991; Wang et al., 2004; Armaghani et al., 2017) reported that
using back propagation (BP) algorithms to tune parameters of neural
networks has some disadvantages such as slow learning rate and trap-
ping in local minima. Utilizing evolutionary algorithms for training
neural networks instead of conventional (like BP) algorithms can pos-
sibly have many effectiveness of enhancing prediction performance in
its favor.
The objective of this study is to develop some hybrid neural net-
works that are trained by four different evolutionary algorithms in-
cluding particle swarm optimization (PSO), genetic algorithm (GA),
imperialism competitive algorithm (ICA) and Biogeography-based op-
timizer (BBO) for predicting drilling rate. Eventually, the performance
of developed hybrid models are compared with NNs trained by BP al-
gorithm to give an objective assessment of evolutionary algorithms’
performance in tuning neural networks parameters. Also, the perfor-
mance of two different ANNs were compared.
The workflow of this study is demonstrated in Fig. 1 which involves
5 main stages of data collection and up-scaling, noise reduction, feature
selection, developing proposed models and evaluation of the results.
2. Data collection
In the present research, collected data of a vertical well drilled into
Marun oil field in southwest of Iran was used. Fig. 2 shows some in-
formation about location and stratigraphy of the area. The database
consists of 1000 data points which are collected from Asmary formation
(one of primary reservoirs of Marun oil field), and covers 500 m of
drilling depth (3000 m–3500 m). The available data are comprised of
two different sources, mud logging and well logging (petrophysical)
data. Mud logging information encompasses weight on bit (WOB), bit
rotational speed (BRS), bit flow rate (BFR), pump pressure (PP) and
mud weight (MW) while petrophysical information includes compres-
sional wave velocity (Tp ), shear wave velocity (Ts ), density log, gamma
ray (GR), neutron porosity (NP) logs and formation pore pressure (Pp ).
It should be noted that two parameters of bit type (roller cone bit) and
formation type (Asmary) were constant in the selected range of depth
(3000 m–3500 m), so these parameters have not been considered as
input vectors in developing the relevant models.
On the other hand, due to collection of data from the two different
sources, they were recorded at disparate sampling rate (for petrophy-
sical logs measurements taken at 15.24 cm intervals, and for mud log-
ging data measurements taken at 50 cm intervals). Hence, it was ne-
cessary to implement an up-scaling method to contend with this
difficulty. In this approach, petrophysical data points were averaged
within each 50 cm depth interval. This averaging method made it
possible to present the well logging and mud logging data in a single
dataset with a same depth interval of 50 cm.
Table 1 gives a list of statistical information about eleven in-
dependent parameters and the dependent variable (ROP). Range of
changes, unit and average values are of some characteristics that pre-
sented in this Table. Furthermore, Fig. 3 Shows the relation (based on
regression coefficient) between independent variables and the depen-
dent variable (i.e. drilling rate), and the related p-values. Since the
calculated p-values do not exceed 0.05, it is concluded that effect of
proposed input parameters on drilling rate is significant. However, p-
values is not a sufficient criterion for selecting most influential para-
meters in ROP prediction, but it could be beneficial to use this value for
detecting and removing parameters with negligible effect on ROP
which are associated with p-value of more than 0.05.
S.B. Ashrafi et al.
Fig. 1. Schematic diagram of workflow applied to the ROP-prediction models developed (WOB = weight on bit; BRS = Bit Rotation Speed; BFR =Bit Flow Rate; PP
= Pump Pressure; MW = Mud Weight; Tp = compressional sonic velocity; Ts = shear sonic velocity; GR = Gamma Ray; NP = Neutron Porosity; Pp = Pore
Pressure).
2.1. Noise reduction
Noise of measurements is a common factor that affect most of raw
data (Wang et al., 1995). Scientists in statistical and machine learning
fields spot this as a phenomenon with different detrimental effects. A
large number of them are agree that learning process can be affected by
data which are contaminated by noise. Furthermore, noisy data can
affect learning process and increase learning time (Quinlan, 1986).
They also assert that noisy data can upset the balance between para-
meters and eventually lead to incorrect rules in machine learning
models as well as reducing overall function-approximation performance
(Garcia et al., 2015; Lorena and de Carvalho, 2004). What is more,
values of environmental properties are often affected by noisy mea-
surements acknowledged by some geophysicists (Cardiff and Kitanidis,
2010). Exploring source of the noise is of critical importance in dealing
with this incident. Some frequent cases can be said such as drilling tools
replacement, drill-string vibrations, washouts, work-overs, and also
shifting and changes in geological layers which can impose errors to the
acquired data.
There are various methods such as low-pass filter, moving average,
wavelet transform, and Savitzky–Golay (SG) smoothing filter in order to
reduce the noise from data (Vaferi and Eslamloueyan, 2015; Rudin
et al., 1992; Savitzky and Golay, 1964). In this research, noise reduction
was performed using Savitzky–Golay (SG) smoothing filter. The SG
technique is the common smoothing method that was widely used in
earth science for denoising petrophysical logs (Savitzky and Golay,
606
Journal of Petroleum Science and Engineering 175 (2019) 604–623
1964; Baba et al., 2014). In addition, signal's original properties will
maintain after smoothing process using this filter. In this approach, the
effect of noise on data is mitigated by a polynomial function (i.e. initial
values are replaced with much more smoothed values). SG polynomial
function of order n is fitted to a number of points which sets with an
interval founded upon the least squares error. Number of the points
within this interval should be odd and less than the order of the poly-
nomial. An increase in the order of polynomial or a decrease in the
number of points within the interval can further retain data structure
and make the extent to which data is smoothed lower. On the other
hand, the opposite brings about additional smoothing by wiping out a
number of the data. Hence, determination of proper polynomial order
and number of points within the interval is of paramount significance.
In order to obtain optimum values of these two parameters, polynomial
order and number of points within the interval, daily reports of the
studied well and geological setting of the field were evaluated. Ac-
cording to the results, optimal order of polynomial and number of
points within the interval for the drilling data were found to be 3 and
11, respectively. Same analysis on petro-physical logs showed the
amount of these parameters would be 4 and 11, respectively. Fig. 4
demonstrates a comparison between the data recorded of the well (in
red) along with denoised data (in black) using SG filter. According to
the figure, trend of changes in the denoised data is in good agreement
with that of recorded data, i.e. noise effect has been minimized.
S.B. Ashrafi et al. Journal of Petroleum Science and Engineering 175 (2019) 604–623

Fig. 2. Schematic map showing the location of the Marun oil field and the stratigraphy of the study area. The Marun oil field is hosted by the Asmary Formation.

Table 1
Statistical information of input and output parameters.
Coded factor Parameter Unit Minimum Maximum Average Standard deviation

F1 Neutron Porosity (NP) – 0.3 0.43 0.35 0.022

F2 Density kg / m3 2212 2424 2375 0.026
F3 Shear Wave Velocity (Ts ) us /ft 226.31 319.82 261.28 17.710
F4 Compressional Wave Velocity (Tp ) us /ft 100.48 121.45 108.3 4.105
F5 Gamma Ray (GR) GAPI 70 117 102.48 5.984
F6 Weight on Bit (WOB) Kgf 612.25 10086.53 4879.47 4.982
F7 Bit Rotational Speed (BRS) RPM 117.29 143.69 135.58 4.597
F8 Pump Pressure (PP) MPa 23.56 26.41 25.38 56.47
F9 Bit Flow Rate (BFR) m3/s 0.0544 0.0581 0.0572 5.4374
F10 Mud Weight (MW) kg / m3 1680.0 1720.0 1696.0 0.0196
F11 Pore Pressure (Pp ) kg / m3 1195 1598 1476 88.6
F12 Rate of Penetration (ROP) m/hr 2.27 35.12 24.86 4.3172

607
S.B. Ashrafi et al.
Fig. 3. Pair plot to analyze relationship of data.
3. Feature selection
Feature selection method is a problem of picking out a subset of the
most influential d parameters from a total set of D features according to
results of an optimization criterion. Shortening the training time, ease
of model interpretation and reduction in over-fitting are of the most
important benefits of applying feature selection method (James et al.,
2013).
There are several methods for applying feature selection and
choosing the most influential independent parameters. Some of these
approaches are filter methods, embedded methods, unsupervised
learning techniques, sequential and heuristic search algorithms
(Chandrashekar and Sahin, 2014; Law et al., 2004; Davoudi and Vaferi,
2018; Gholami et al., 2018). Since the difference between values of
prediction and observed data should be as small as possible, residual
error is the only important factor in selecting parameters. So, feature
selection methods that are based on correlation coefficient, which is a
subgroup of filter methods, are incompetent in selecting the most
608
Journal of Petroleum Science and Engineering 175 (2019) 604–623
influential features (Anemangely et al., 2017). Conversely, wrapped
methods which employ sequential or heuristic search algorithms
achieve more robust solutions, particularly in interacting with a large
number of features (Chandrashekar and Sahin, 2014; Vafaie and Imam,
1994). Accordingly, in this study, genetic algorithm (GA), which is a
heuristic search algorithm, was implemented to determine indis-
pensable inputs for ROP estimation.
Regarding feature selection using GA, firstly, 12 initial populations
containing a set of feasible solutions are generated. The generated po-
pulations include 11 members (chromosomes), each of which is a so-
lution of the desired problem. Actually, every member is indicative of
different combinations of input vectors that are defined as encoded
values. The fitness of these members (combinations) are determined
using a cost function (objective function). A MLP-NN with two hidden
layers was implemented as the cost function of genetic algorithm and
optimum number of neurons for each hidden layer was determined
through a trial and error approach. In other words, different archi-
tectures of MLP-NN were developed using all of the available features
S.B. Ashrafi et al.
Fig. 4. Recorded and denoised data of the well under study.
and their performance were evaluated based on MSE obtained through
10-folds cross validation method.
In 10-folds cross validation method, as it is evident in Fig. 5, the
entire data is split into 10 non-overlapped segments. One of these
segments is chosen as the testing data, and the remaining 9 folds are all
considered as the training set. Then, the neural network is trained using
all the training data for 100 times and the lowest error is recorded. This
procedure is performed for 10 times, and each time one of the folds is
considered as the testing set while the rest of folds are acting as the
training set. Eventually, MSE values for all 10 testing sets are averaged
and reported as the performance related to MLP-NN structure.
According to results (Fig. 6), it is found that the best configuration
609
Journal of Petroleum Science and Engineering 175 (2019) 604–623
of MLP-NN composed of two hidden layers with 6 and 5 neurons in first
and second layer, respectively. In the next step, the members were
ranked depending on MSE values resulted from predicted and measured
drilling rate. Again, the 10-fold cross validation method was also im-
plemented to determine the MSE value of each member and avoid oc-
currence of over fitting in MLP_NN training. After that, the most sui-
table members (combinations of inputs) were subjected to specific
genetic algorithm operators namely crossover, reproduction and mu-
tation. These operators were performed to create new populations and
guide the search toward the most suitable combinations of input vari-
ables.
It is important to note that all of the process described above can be
S.B. Ashrafi et al.
Fig. 5. Schematic diagram of the 10-fold cross validation method.
Fig. 6. Number of neurons in the two hidden layers of the developed MLP with
MSE values for feature selection.
Table 2
Applying feature selection on different number of input vectors and associated
RMSE values.
Number of inputs Selected inputs
1 F8
2 F11, F6
3 F11, F6 , F7
4 F11, F9 , F6 , F8
5 F11, F9 , F8 , F6 , F7
6 F7 , F6 , F9 , F11, F2 , F8
7 F2 , F8 , F7 , F11, F6 , F9 , F3
8 F6 , F5 , F9 , F11, F8 , F3 , F2 , F7
9 F5 , F3 , F2 , F1, F9 , F7 , F6 , F8 , F11
10 F8 , F2 , F1, F7 , F3 , F5 , F4 , F11 , F9 , F6
11 F1, F5 , F7 , F2 , F6 , F11, F4 , F3 , F9 , F10 , F8
performed to detect the best combination of features for different
number of inputs. Therefore, the feature selection was conducted for 11
rounds with different number of inputs (from 1 input to 11 inputs)
which provides more understanding about the effect of number of in-
puts on overall MSE. Table 2 presents results of applying feature se-
lection for different number of inputs and their best combinations. In
order to give a visual perception, associated errors are plotted in Fig. 7.
Regarding this figure, by increasing the number of input variables, the
curve of error tend to have a less-marked slope. Different stages with
regard to mentioned feature ranking method are delicately shown as a
flowchart in Fig. 8.
In conclusion, changes in errors would be negligible and only lead
to an increase in calculation time once the number of input parameters
exceeds eight. Hence, the model with the 8 input parameters was se-
lected for ROP approximation in subsequent model development.
Journal of Petroleum Science and Engineering 175 (2019) 604–623
4. Methods
4.1. Multi-layer perceptron neural network (MLP-NN)
Multi-layer perceptron neural network as one of the most common
and practical type of ANN is a branch of artificial intelligence (AI)
(Moghadassi et al., 2009; Gardner and Dorling, 1998). This network can
satisfactory estimate values for non-linear correlations (Esfe et al.,
2015a). The structure of a MLP-NN comprises three main sections, an
input layer, hidden layer(s), and an output layer. A relationship is es-
tablished between input and output parameters through hidden layer(s)
(Lashkarbolooki et al., 2012). Although number of neurons in input and
output layers are exactly the same as the number of input and output
parameters, respectively, number of hidden layers and also number of
neurons in each hidden layer can be chosen optimally (Ramchoun et al.,
2016). In each neuron, inputs are multiplied with a weight and summed
with a value called bias. Then, their summation are subjected to a
transformation through a function called transfer function (activation
function). Outputs of each layer are taken as inputs for the next layer
(Esfe et al., 2015b; Heidari et al., 2016). The output of MLP neural
network can be explained as in Eq. (1):
Nk 1
MSE
Yjk = Fk Wijk Yi (k 1) + Bik
19.4481 i=1 (1)
9.6154
6.6564 Where Yjk and Bik are the neuron j's output from k's layer and bias
5.3824
for neuron j in layer k, respectively. Wijk represents the weight that is
3.6123
3.2761 chosen on a random basis in the initial stage of training procedure. Fk is
3.1329 the transfer function which may be considered in many different forms
2.8951 such as identity function, binary step function, binary sigmoid, bipolar
2.8589 sigmoid, Gaussian, and linear functions (Fausett, 1994). Back-propa-
2.8488
gation (BP) algorithms are conventionally used for training MLP-NNs.
2.8472
Some of these BP algorithms are Scaled Conjugate Gradient (SCG),
Levenberg–Marquardt (LM), Gradient Descent (GD) and Resilient back
propagation (RP) (Ghoreishi and Heidari, 2013; Moghadassi et al.,
2011; Yetilmezsoy et al., 2011).
4.2. Radial basis function neural network (RBF-NN)
RBF-NNs began to function as a remedy in the late 1980s.
Practicability of this type of neural network in dealing with arbitrarily
scattered data, easily generalizing to several dimensional space, and
providing spectral accuracy have made it a particularly appropriate
alternative for MLP-NNs (Elsharkawy, 1998; Lashkenari et al., 2013).
Other advantages of this neural network are the ability of accurately
distinguishing patterns for non-linear models and the feasibility of
being trained in a single direct procedure which is unlike the process in
MLP-NN (Venkatesan and Anitha, 2006). Structure of RBF-NN is highly
similar to MLP-NN. Fundamentally, RBF-NNs has only one hidden layer
which consists of several nodes called RBF units.
Every RBF_NN include two tuning parameters representing location
610
S.B. Ashrafi et al. Journal of Petroleum Science and Engineering 175 (2019) 604–623

Fig. 7. Flow chart of feature ranking method.

of basis function's center and its width (spread). Distance between these Where, wT is the transposed weight, N is the number of neurons in
centers and input data are measured in the RBF units. The network hidden layer and (xi ) is the basis function which is typically in
reaches its highest performance when the measured distance becomes Gaussian form:
zero and degrades by increasing the distance. While RBF-NN has only a
(x c ) 2
single hidden layer, two sets of weights are exist, one for connection
(x i ) = e 2s 2 (3)
between hidden and input layers and the other for connection between
hidden and output layers. A non-linear function is used to transform the In Eq. (3), s is spread of Gaussian basis function and c represents the
input area into hidden area. Following this, the output of the network center of RBF unit.
are resulted from the output layer which is always linear (Wu et al.,
2012; Park and Sandberg, 1991). The output of the network can be
4.3. Genetic algorithm (GA)
shown in the following form (Eq. (2)):
Genetic algorithm (GA) as an evolutionary algorithm was firstly
N introduced by John Holland (Goldberg, 1989; Holland, 1992). GA is an
f (x i ) = wT (x i ) inquiry based optimization method inspired by natural selection or
j=1 (2) compatibility. This population based algorithm simulates the existed

611
S.B. Ashrafi et al.
Fig. 8. Different number of inputs versus corresponding MSE values.
process in a natural system where only the strongest participant can
survive. GA produces an initial population containing possible solutions
and recombines them in a way to steer the search toward high potential
areas of desirable answer. Every of those possible solution is encoded as
a chromosome, and a value called fitness is assigned to them by an
objective function (cost function). The high and low fitness values are
representative of the better solution for maximization and minimization
problems, respectively. Strong members of each population are trans-
ferred to a new population matrix undergoing some changes by specific
genetic operators such as crossover, reproduction, and mutation (Ab
Wahab et al., 2015). Crossover is recombination of two parents that are
elements of one generation which combine to generate new member for
next generation. Reproduction selects the most compatible candidates
of each generation, and mutation is another operator of GA which is
analogous to natural biological mutation, altering genes with new un-
expected genes (Grefenstette, 2013).
4.4. Particle swarm optimization (PSO)
Particle Swarm Optimization (PSO) is a population based stochastic
procedure initially put forward by Eberhart and Kennedy (1995)
(Eberhart and Kennedy, 1995). PSO implements an ordinary me-
chanism representing swarm behavior of birds flocking and fish
schooling to guide the particles to search the optimal solutions. PSO
was described with separation, alignment and cohesion by Del Valle
et al. (2008) (Del Valle et al., 2008). Separation avoid crowding local
flock-mates. While alignment is moving toward the mean direction of
local flock-mates, moving toward the mean position is the cohesion.
Three years after first implementation of PSO, inertia weight was in-
troduced by Shi and Eberhart (1998) (Shi and Eberhart, 1998). This
term regulate the impact of velocity which controls the exploration and
the exploitation behaviors of particle, and can be seen in following
formulas (Eq. (4) and Eq. (5)):
vidt+ 1 = w. vidt + c1. rand (0,1). (pidt x idt) + c2. rand (0,1). pgd
t
x idt
x idt+ 1 = x idt + vidt+ 1
Where in the above equations, and
vidt are particle velocity and
x idt
particle position respectively, i is the particle index, d is the dimension
in the search space, and t is the iteration number. Pi refers to the best
position of particle i and Pg represents the best position of the neighbors
of particle i . The random values between 0 and 1 are showed
Journal of Petroleum Science and Engineering 175 (2019) 604–623
asrand (0,1) . c1 and c2 are the speeds that regulate the length when going
towards the most optimal particles of the swarm, and w is the weight.
Higher weight value means bigger step size, resulting in more ex-
ploration behavior. If the weight value is low, the step size is small and
exploitation behavior is more dominant. In order to have a high ex-
ploration behavior at the beginning of optimization and the exploita-
tion behavior gradually increase through iterations, a damping ratio is
defined and multiplied by weight value at each iteration.
4.5. Biogeography-based optimizer
Simon (2008) (Simon, 2008) proposed a new evolutionary algo-
rithm called BBO for the first time. The fundamental inspiration of this
algorithm is got from assessing geographical distribution of biological
organisms over time and place. Like other evolutionary algorithms,
main motivation of BBO is the natural processes. A number of search
agents are employed by BBO, and are called habitats (Wang et al.,
2014a,b,c). These habitats are functioning the same as chromosomes in
GAs. A vector of habitants representing the variables of the problem is
assigned to each habitant in BBO algorithm (similar to genes in a GA).
Furthermore, overall fitness of a habitat can be determined as term of
habitat suitability index (HSI), so the higher the HSI, the more fit the
habitat. The fittest habitants are then subjected to some specific BBO
operators namely immigration, emigration, and mutation. In nature,
these concepts lead to a balance between various ecosystems. In other
words, nature has a tendency to provide the general stability of various
geographical areas. The BBO algorithm uses this concept to improve all
habitats’ HSI, and brings about development of the initial random so-
lutions for a particular problem (Mirjalili et al., 2014).
4.6. Imperialist competitive algorithm
Another evolutionary technique is Imperialist competitive algo-
rithm (ICA) which is based on human being's sociopolitical evolution
(Atashpaz-Gargari and Lucas, 2007). ICA like the other algorithms is
based on populations in which it starts with initial population named
(4) countries or colonies. Best countries (imperialists) in optimization are
those with the lowest cost. Therefore, the countries divide into two
(5)
categories of colony and imperialist which together form an empire.
Basic operations in ICA are assimilation, revolution and competition.
The imperialist with more power than other ones try to possess the
colonies of weaker empires and take their control. This process is called
imperialist competitive process. In this procedure, the imperialists with
more power will be aggravated in the power and the vice versa for the
612
S.B. Ashrafi et al.
weak ones. The empires that losses all of their colonies are demolished
and it is assumed that only one empire will remain in an ideal world
after passing a few decades. In other words, as it turned out, the most
powerful empire have all countries which are known as colonies (86).
Another case that makes ICA to stop is when no enhancement occurs in
an empire after a few decades (Ansari et al., 2017).
5. Development of ROP predictive models
This section presents developments of several ROP function ap-
proximations. As mentioned earlier, eight parameters including weight
on bit, pump pressure, bit rotational speed, bit flow rate, pore pressure,
density, shear wave velocity and gamma ray were recognized as the
most influential parameters on drilling rate. So, these variables were
utilized as input parameters for model development. Procedures of
developing the proposed models i.e. ANN, PSO-ANN, ICA-ANN, BBO-
ANN, GA-ANN, and regression models are described in the following
sections.
5.1. Simple ANNs
Process of developing two types of ANN (MLP and RBF) are de-
scribed in this section to predict the ROP. Firstly, data normalization
were performed with the aim of enhancing accuracy of models using
Eq. (6):
x in = 2 ×
xi xmin
1 neurons in the hidden layer, and their performances were evaluated
x max xmin (6)
In Eq. (6), i shows the number of parameters, and x max and x min
represent maximum and minimum values of x i , respectively. In the next
stage, for developing and evaluating the models, 10-fold cross valida-
tion method as described in section 3 was implemented. Among all of
the back propagation learning algorithms, Levenberg-Marquardt (LM)
algorithm was selected to train the MLP-NN. According to the litera-
ture, many scholars have confirmed efficiency of the LM algorithm in
approximation of geotechnical engineering problems (Armaghani et al.,
2017; Ornek et al., 2012; Ceryan et al., 2013). As it was highlighted by
several researchers, a MLP-NN with two or more hidden layers lead to
greater accuracy in extremely complex problems (Pham et al., 2017;
Surkan and Singleton, 1990; Openshaw and Openshaw, 1997; Li and
Yeh, 2002). Therefore, two hidden layers were considered in MLP-NN
to provide sufficient prediction performance. By evaluating perfor-
mance of different transfer (activation) functions, sigmoid and purelin
was selected as transfer function of hidden layers and output layer,
respectively. The number of neurons in input and output layer were
considered depending on the input and output variables. As it is stated
before, the performance of MLP-NN is significantly influenced by
number of neurons in the hidden layer(s). In this study, these para-
meters were determined through a trial and error procedure by evalu-
ating the performance of different network architecture (different
number of neurons in both hidden layers) based on MSE (average MSE
for 10 testing folds). MSE value versus number of neurons in both
hidden layers are shown in Fig. 9. According to the results, the optimum
case of 4 neurons in the first and 6 neurons in the second hidden layer
was chosen as the network structure which yield the minimum MSE.
Fig. 10 illustrates the structure of developed MLP-NN. A summary of
the properties of the developed MLP-NN are listed in Table 3. In ad-
dition, properties of evolutionary algorithms used for training the ANNs
are mentioned in Table 4.
Turning to simple RBF-NN training, Gaussian function was selected
as the basis function of RBF-NN. After determining centers of the RBF
units, the weights are tuned by Gradient-Descent (GD) back-propaga-
tion algorithm. The centers were chosen by application of a clustering
method based on the K-means algorithm, which is a commonly used
and accurate cluster analysis technique in data mining (Hartigan and
Wong, 1979). Maximum number of neurons (MNN) is a substantially
613
Journal of Petroleum Science and Engineering 175 (2019) 604–623
Fig. 9. Number of neurons in the two hidden layers of the developed MLP with
MSE values for the ROP prediction.
determinant tuning parameter in developing these types of ANNs. Here
a trial and error approach is presented to determine the optimum value
of this parameter. However, different optimization algorithms can be
used for this purpose (Najafi-Marghmaleki et al., 2017). Various
structures of RBF-NN have been developed by changing number of
according to the average MSE obtained through cross validation
method. Fig. 11 shows prediction performance of developed RBF-NNs
versus number of neurons in their hidden layer. It is obvious that using
40 neurons in hidden layer can be considered as optimum number of
RBF units in RBF-NN.
5.2. Hybrid ANNs
As mentioned earlier, each MLP has included some weights and
biases which should be optimized by an algorithm to achieve minimum
error in objective function. Here, aforementioned evolutionary algo-
rithms are applied to adjust weights and biases of MLP-NN. To reach
this goal, a set of randomly selected weights and biases were defined in
a matrix as follows:
[w1, w2, w3, wn, b1, b , b3, bn] (7)
Where w and b refer to weight and bias, respectively. The generated
matrix is called in various names in different evolutionary algorithm;
populations in GA, swarms in PSO, countries in ICA, and species in
BBO. Each row of this matrix (member) presents a solution for the
optimization problem, and number of columns are equal to sum of
number of weights and biases. After constructing the relevant matrix for
each algorithm, members of matrix were ranked based on the defined
objective function. In the next step, a new matrix including most fea-
sible solutions (weights and biases) is created using specific operators
which are diverse in each evolutionary algorithm. Regarding the de-
velopment of hybrid RBF-NN, there are three tuning parameters. These
parameters include weights, centers and spread, and are optimized by
application of the same approach used for adjusting weights and biases
in MLP-NN.
Maximum number of iterations for all evolutionary algorithms were
considered as 300 and 250 during training process of MLP-NNs and
RBF-NNs, respectively. In following, the size of generated matrix (po-
pulation size) was set to 80 for all of the optimization algorithms im-
plemented for training ANNs. It is important to note that the archi-
tecture of hybrid ANNs (number of hidden layers, number of neurons,
transfer functions and basis function) were considered as same as the
structure of ANNs trained using back propagation algorithm (see sec-
tion 4.1). Fig. 12 illustrate the convergence curves of meta-heuristic
S.B. Ashrafi et al. Journal of Petroleum Science and Engineering 175 (2019) 604–623

Fig. 10. Structure of developed MLP-NN for ROP prediction.

Table 3 estimated values versus the measured data for simple MLP-NN and
Summary of architecture of the developed MLP to predict ROP. hybrid MLP-NNs. Obviously, the values of regression coefficient for
Network type Multi-layer perceptron (MLP) developed hybrid MLP-NNs are highly desirable, indicating that the
Training function Levenberg-Marquardt (LM) proposed models yield reasonably accurate predictions for drilling rate.
Number of layers 3 Also, this figure reveals that prediction performance of hybrid MLP-NNs
Nodes in 1st hidden layer 4 are far higher than of MLP-NN trained by LM back propagation algo-
Transfer function of 1st hidden layer TANSIG
Nodes in 2nd hidden layer 6
rithm. Regression plots with regard to RBF-NN and hybrid RBF-NNs are
Transfer function of 2nd hidden layer TANSIG shown in Fig. 14. It is noticeable that hybrid RBF-NNs exhibit much
Neurons in output layer 1 better performance in prediction than RBF-NN trained by GD back
Transfer function of output layer PURELIN propagation algorithm. Moreover, relative deviations for developed
Performance evaluation MSE
ANNs and hybrid ANNs are shown in Figs. 15 and 16. According to
results, the lowest relative deviations are related to ANNs that use PSO
algorithm as training method, indicating the impressive power of this
Table 4
evolutionary algorithm in training ANNs.
Properties of training algorithms in MLP and RBF neural networks.
There are also various statistical performance indices which can be
Algorithm Parameters Value for MLP Value for RBF utilized to quantitatively assess performance of these models. In this
GA Selection Method Roulette Wheel Roulette Wheel
study, variance account for (VAF), root mean square error (RMSE),
Crossover Uniform (p = 1) Uniform (p = 1) performance index (PI), and regression coefficient (R2) are used to
Mutation Uniform Uniform compare the prediction capability of the developed models as they were
(p = 0.1) (p = 0.05) employed by Yılmaz and Yuksek (2008) (Yılmaz and Yuksek, 2008),
Mutation Rate 0.1 0.1
Basarir et al. (2014) (Basarir et al., 2014), and Boyacioglu and Avci
Selection Pressure (Roulette 3 2
wheel) (2010) (Boyacioglu and Avci, 2010):
PSO Cognitive Constant 0.5 0.5 1
p
Social Constant 2.5 2.5 1 2
Inertia Weight 5 5 RMSE = (yr zr )2
p (8)
Inertia Weight Damping 0.6 0.6 r=1
Ratio
BBO Keep Rate 1 1 VAF
PI = R + RMSE
Immigration Rate [0,1] [0,1] 100 (9)
Emigration Rate [0,1] [0,1]
Mutation Probability 0.01 0.01 var (yr z r )
ICA Number of Empires 15 15 VAF = 1 × 100
Assimilation Coefficient 2 2
var (yr ) (10)
Revolution Probability 0.3 0.3 p
Revolution Rate 0.1 0.1 r=1 r
(y zr )2
Colonies Mean Cost 0.2 0.1
R2 = 1 p
(
r=1 r
y yr , mean )2 (11)
Coefficient
In the preceding equations, z is output of the model, y is actual
output (target) and p is the number of data points in database. The
training algorithms used for training the both MLP and RBF neural accuracies of the proposed models are distinctive referring to the small
networks. In addition, properties of used algorithms are listed in details values of RMSE and high values of R2, PI and VAF. Table 5 indicates
in Table (5). The optimum value of these listed parameters was de- values of RMSE, VAF, PI and R2 for the developed models. Regarding
termined using trial and error approach. this table, PSO-MLP yielded the highest values of R2, VAF and PI and
the lowest RMSE corresponding to this model, showing the best per-
6. Results and discussion formance among other models in ROP prediction. The second most
accurate model is PSO-RBF that presents slightly lower prediction ef-
In this section, the predicting performance of developed models is ficiency compared to PSO-MLP. GA-MLP and GA-RBF come in third and
evaluated to choose most reliable model and analyze whether evolu- fourth level, respectively, in terms of prediction performance. Overall,
tionary algorithms can be a proper alternative to conventional back according to the results, PSO and GA algorithms perform much better
propagation methods for training. Fig. 13 presents regression plot of than other implemented algorithms in training ANNs.

614
S.B. Ashrafi et al. Journal of Petroleum Science and Engineering 175 (2019) 604–623

Fig. 11. MSE of the ROP prediction achieved by the RBF network versus the number of neurons in the hidden layer (MNN).

Fig. 12. Convergence of the hybrid neural networks to the optimum solution (minimum RMSE): (a) MLP, (b) RBF.

615
S.B. Ashrafi et al.
Fig. 13. Cross plot of measured versus predicted values of ROP for five developed machine-learning models: (a) MLP-PSO; (b) MLP-GA; (c) MLP-ICA; (d) MLP-BBO;
and, (e) MLP.
Comparing different results (RMSE, PI, VAF and R2) of hybrid ANNs
in Table 7, it is clear that MLP-NN exhibits more effectiveness in ROP
prediction than RBF-NN when using PSO or GA as training function. In
more detail, PSO-MLP and PSO-RBF having RMSE values of 1.12 and
1.4 for testing data, respectively, outperform the GA-MLP and GA-RBF,
with RMSE values of 1.43 for the former and 1.52 for the latter. Con-
versely, employing other evolutionary algorithms such as ICA and BBO
as training function leads to better performance of the RBF-NN than of
the MLP-NN in terms of ROP prediction.
Interestingly, ANNs trained by BP algorithms do not yield satisfac-
tory results compared with hybrid models in ROP prediction. The va-
lues of 1.68 and 1.76 are yielded as RMSE for testing data of simple
616
Journal of Petroleum Science and Engineering 175 (2019) 604–623
MLP and simple RBF model, respectively. This low performance can be
attributed to locally search behavior of BP algorithms which often leads
to fail in finding optimum parameters of ANNs. In fact, it is highly
possible that a BP algorithm converge into a local minimum, while
evolutionary algorithms such as those proposed in this study are suf-
ficiently able to avoid happening this phenomenon.
In addition, a comparison between the de-noised drilling rate
measurements and the results of the MLP-PSO model trained using the
pre-processed data are shown in Fig. 17-a. The model trained using the
de-noised data displays very good prediction performance and properly
follows the trend of changes observed in the ROP measured data in the
studied depth interval (3000 m–3500 m). However, figure for the model
S.B. Ashrafi et al. Journal of Petroleum Science and Engineering 175 (2019) 604–623

Fig. 14. Cross plot of measured versus predicted values of ROP for five developed machine-learning models: (a) RBF-PSO; (b) RBF -GA; (c) RBF eICA; (d) RBF eBBO;
and, (e) RBF.

trained using raw data (Fig. 17-b) exhibits large fluctuations and the
corresponding results are not in good accordance with the measured
drilling rate compared with trends in Fig. 17-a.
7. A comparison between stepwise multiple regression methods
and hybrid ANNs
Regression method can statistically establish some relationships
between different variables in a problem. This relationship is stated as a
function called regression function which presents relationship between
independent and dependent parameters (Bahrami et al., 2011;
Khandelwal et al., 2017). Finding this relationship is the goal of mul-
tiple regression method. Two types of multiple regression method are
exist, linear and non-linear multiple regression method. A linear re-
lationship between variables can be established with application of
linear multiple regression (LMR), and non-linear multiple regression
(NLMR) can provide a non-linear relationship between inputs and
output(s) parameters (Armaghani et al., 2016).
In this research, both of the LMR and NLMR methods were con-
sidered to probe any existed relationship between input and output
variables and also validate the results of proposed hybrid neural net-
works. In development stage, the eight parameters that were selected in

617
S.B. Ashrafi et al. Journal of Petroleum Science and Engineering 175 (2019) 604–623

Fig. 15. Relative deviations of measured versus predicted values of ROP for five developed machine-learning models-The horizontal axis is data record index number
for testing data set: (a) MLP-PSO; (b) MLP-GA; (c) MLP-ICA; (d) MLP-BBO; and, (e) MLP.

feature selection section were taken as input parameters into the re- equation provides more accurate results than LMR model.
gression models. Using the established data, equations for NLMR and
LMR models were obtained that are shown in Eq. (7) and Eq. (8), re- ROP = 1946.1 1968 × F2 4.58 × F3 + 14.71 × F5 13.77 × F6
spectively. The given terms for developed NMLR and LMR are pre- + 6.52 × F7 2.42 × F8 + 19.69 × F9 1039.5 × F11
sented in Tables 6 and 7, respectively. Table 8 also reports general
1.85 × F2 × F5 2.49 × F2 × F7 + 0.67 × F2 × F8
results of implementing the NLMR and LMR models on data of the well
under study. The R2 and RMSE for testing data reveal the superiority of 0.008 × F3 × F6 + 0.006 × F3 × F9 0.007 × F5 × F7
the developed hybrid neural networks models in predicting ROP com- 0.01 × F5 × F9 0.02 × F6 × F7 0.004 × F6 × F8
pared to linear and non-linear regression models. Confirming the fact
+ 0.05 × F6 × F9 3.45 × F6 × F11 0.005 × F7 × F8
that PSO-MLP and PSO-RBF have statistically quite impressive results
than two employed regression models for the dataset extracted from the + 0.02 × F7 × F9 + 0.003 × F8 × F9 + 0.22 × F8 × F11
studied well. Moreover, comparing the results of both regression + 1.0184 × F9 × F11 0.0009 × F32 0.032 × F62 0.009 × F72
models indicates that the relationship between independent parameters 0.0002 × F82 0.02 × F92 196.03 × 2
F11 (12)
and dependent parameter (ROP) is completely non-linear as NLMR

618
S.B. Ashrafi et al. Journal of Petroleum Science and Engineering 175 (2019) 604–623

Fig. 16. Relative deviations of measured versus predicted values of ROP for five developed machine-learning models-The horizontal axis is data record index number
for testing data set: (a) RBF-PSO; (b) RBF -GA; (c) RBF eICA; (d) RBF eBBO; and, (e) RBF.

ROP = 25.336 × F2 + 0.12148 × F5 + 0.87147 × F6 0.045999 × F8
+ 0.51205 × F9 + 50.046 × F11 305.62
8. Conclusion
In order to predict the ROP, a database consisting of 1000 data
points was collected from mud logging and petro-physical logs of a well
drilled in Marun oil field. At first, Savitzky–Golay filter as was applied
on the extracted data to reduce the effect of noise and to develop the
predictive models with high accuracy. In the next stage, a feature
ranking method based on heuristic search algorithm was applied on the
noise-free data to select the features with highest influence on ROP and
also to shorten the input vector. Accordingly, eight variables including
(13)
weight on bit, bit rotational speed, bit flow rate, pump pressure, pore
pressure, gamma ray, density log and velocity of sonic wave were se-
lected as model's inputs. Considering these eight parameters and using
all 1000 data points, hybrid MLP and RBF neural networks were de-
veloped using evolutionary algorithms including PSO, GA, ICA, and
BBO as learning algorithms. With the aim of giving better assessment of
the hybrid models' performance, simple ANNs (training with a back-
propagation method) and multiple regression methods were developed,
and the results of them were compared to the results of hybrid models.
Main achievements of this study are outlined as follows:

619
S.B. Ashrafi et al. Journal of Petroleum Science and Engineering 175 (2019) 604–623

Table 5 • ANNs (MLP and RBF) trained by PSO and GA meta-heuristic algo-
Statistical indicators of ROP-prediction performance accuracy for the simple rithms showed higher performances than other developed models,
ANNs and hybrid ANNs. indicating power of these algorithms in tuning parameters of ANNs
Model Data subset RMSE R VAF % PI in this specific problem.

MLP-PSO Test 1.12 0.933 93.39 0.7752

• MLP-NN exhibits more effectiveness in ROP prediction than RBF-NN
in condition of using PSO or GA as training function. However,
Train 0.9668 0.9496 94.96 0.9573
applying other evolutionary algorithms such as ICA and BBO as
MLP-GA Test 1.43 0.8931 89.44 0.4023
Train 1.23 0.9172 91.73 0.6387 training function leads to better results for RBF-NN than for MLP-NN
MLP-ICA Test 1.67 0.85 85.19 0.102 in terms of predicting ROP.

MLP-BBO
Train
Test
1.37
1.8
0.8976
0.8563
89.8
85.72
0.4657
−0.0209
• Hybrid ANNs are much more accurate than ANNs trained using back
propagation algorithms (LM algorithm for MLP and GD algorithm
Train 1.464 0.8778 87.79 0.3473
Simple MLP Test 1.68 0.83 83.75 0.0632
for RBF). It can be concluded that BP algorithms fail in discovering
Train 1.59 0.866 86.61 0.2034 global minimum and probability of trapping in local minima is high
RBF-PSO Test 1.4 0.8775 87.86 0.4093 for these algorithms.

RBF-GA
Train
Test
1.36
1.52
0.9028
0.8819
90.29
88.28
0.4856
0.2946
• Implementing the NLMR and LMR models on the established data
showed that these models cannot be considered as reliable ap-
Train 1.41 0.8908 89.12 0.4206
RBF-ICA Test 1.5 0.8601 86.81 0.2902 proaches for prediction of ROP. Confirming the fact that hybrid
Train 1.47 0.886 88.65 0.3487 ANNs yield much more accurate predictions than regression
RBF-BBO Test 1.61 0.8579 85.92 0.17 methods.
Simple RBF
Train
Test
1.5
1.76
0.878
0.8107
87.86
81.46
0.3084
−0.0539 • A comparison between the results of PSO-MLP trained using pre-
Train 1.69 0.84 84.93 0.0718
processed data and raw data showed that the noise reduction ap-
proach (SG filter) implemented in this study is very effective in in-
creasing accuracy of predictive models.

Fig. 17. Measured versus predicted ROP versus data record index numbers for the MLP-PSO model applied to the overall data subset.

620
S.B. Ashrafi et al. Journal of Petroleum Science and Engineering 175 (2019) 604–623

Table 6 References
Selected variables and obtained coefficients for each variable using NLMR
method. Ab Wahab, M.N., Nefti-Meziani, S., Atyabi, A., 2015. A comprehensive review of swarm
optimization algorithms. PLoS One 10 (5), e0122827.
added variables Estimated coefficients Standard error T-statistics
Akgun, F., 2007. Drilling rate at the technical limit. Int. J. Petrol. Sci. Technol. 1, 99–118.
Anemangely, M., Ramezanzadeh, A., Tokhmechi, B., 2017. Shear wave travel time esti-
Intercept −1946.1 1397.6 −1.3925 mation from petrophysical logs using ANFIS-PSO algorithm: a case study from Ab-
F2 −1968 288.84 −6.8134 Teymour Oilfield. J. Nat. Gas Sci. Eng. 38, 373–387.
F3 −4.5808 0.87679 −5.2245 Anemangely, M., Ramezanzadeh, A., Tokhmechi, B., Molaghab, A., Mohammadian, A.,
F5 14.713 3.0742 4.7861 2018. Drilling rate prediction from petrophysical logs and mud logging data using an
F6 −13.774 7.4501 −1.8488 optimized multilayer perceptron neural network. J. Geophys. Eng. 15 (4), 1146.
F7 6.5228 5.1346 1.2704 Ansari, H.R., Hosseini, M.J.S., Amirpour, M., 2017. Drilling rate of penetration prediction
F8 −2.4241 0.55334 −4.3808 through committee support vector regression based on imperialist competitive al-
F9 19.687 3.9658 4.964 gorithm. Carbonates Evaporites 32 (2), 205–213.
F11 −1039.5 301.41 −3.4487 Armaghani, D.J., Mohamad, E.T., Hajihassani, M., Yagiz, S., Motaghedi, H., 2016.
F2 F5 −1.8473 0.26677 −6.9247 Application of several non-linear prediction tools for estimating uniaxial compressive
strength of granitic rocks and comparison of their performances. Eng. Comput. 32 (2),
F2 F7 −2.487 0.89643 −2.7743
189–206.
F2 F8 0.67474 0.081699 8.2588
Armaghani, D.J., Mohamad, E.T., Narayanasamy, M.S., Narita, N., Yagiz, S., 2017.
F3 F6 −0.0080211 0.001093 −7.3389 Development of hybrid intelligent models for predicting TBM penetration rate in hard
F3 F9 0.0057337 0.00099838 5.743 rock condition. Tunn. Undergr. Space Technol. 63, 29–43.
F5 F7 −0.0066956 0.0028502 −2.3491 Atashpaz-Gargari, E., Lucas, C. (Eds.), 2007. Imperialist Competitive Algorithm: an
F5 F9 −0.01037 0.0030816 −3.365 Algorithm for Optimization Inspired by Imperialistic Competition. Evolutionary
F6 F7 −0.025033 0.0054302 −4.6099 Computation, 2007 CEC 2007 IEEE Congress on. IEEE.
F6 F8 −0.0041653 0.0012895 −3.23 Baba, K., Bahi, L., Ouadif, L., 2014. Enhancing geophysical signals through the use of
F6 F9 0.045603 0.010533 4.3294 Savitzky-Golay filtering method. Geofisc. Int. 53 (4), 399–409.
F6 F11 −3.4493 1.0739 −3.212 Bahrami, A., Monjezi, M., Goshtasbi, K., Ghazvinian, A., 2011. Prediction of rock frag-
F7 F8 −0.0052585 0.00052566 −10.004 mentation due to blasting using artificial neural network. Eng. Comput. 27 (2),
F7 F9 0.024154 0.0061148 3.95 177–181.
Basarir, H., Tutluoglu, L., Karpuz, C., 2014. Penetration rate prediction for diamond bit
F8 F9 0.0033364 0.00059644 5.5938
drilling by adaptive neuro-fuzzy inference system and multiple regressions. Eng.
F8 F11 0.21645 0.071884 3.0111
Geol. 173, 1–9.
F9 F11 1.0184 0.47716 2.1344 Bataee, M., Irawan, S., Kamyab, M., 2014. Artificial neural network model for prediction
F32 −0.00094814 0.00020179 −4.6986 of drilling rate of penetration and optimization of parameters. J. Jpn. Petrol. Inst. 57
F62 −0.031539 0.01055 −2.9894 (2), 65–70.
−0.0086858 0.003356 −2.5881 Bezminabadi, S.N., Ramezanzadeh, A., Jalali, S.-M.E., Tokhmechi, B., Roustaei, A., 2017.
F72
Effect of rock properties on ROP modeling using statistical and intelligent methods: a
F82 −0.0002487 3.4068 × 10 5 −7.3003 case study of an oil well in southwest of Iran. Arch. Min. Sci. 62 (1), 131–144.
F92 −0.020562 0.0030681 −6.7018 BINGHAM, G., 1965. A new approach to interpreting rock drillability. TECHNICAL
2 −196.03 34.971 −5.6055 MANUAL REPRINT. Oil Gas J. 93, 1965.
F11
Bodaghi, A., Ansari, H.R., Gholami, M., 2015. Optimized support vector regression for
drilling rate of penetration estimation. Open Geosci. 7 (1).
Bolandi, V., Kadkhodaie, A., Farzi, R., 2017. Analyzing organic richness of source rocks
Table 7 from well log data by using SVM and ANN classifiers: a case study from the Kazhdumi
formation, the Persian Gulf basin, offshore Iran. J. Petrol. Sci. Eng. 151, 224–234.
Selected variables and obtained coefficients for each variable using LMR Boyacioglu, M.A., Avci, D., 2010. An adaptive network-based fuzzy inference system
method. (ANFIS) for the prediction of stock market return: the case of the Istanbul stock ex-
change. Expert Syst. Appl. 37 (12), 7908–7912.
added variables Estimated coefficients Standard error T-statistics
Cardiff, M., Kitanidis, P.K., 2010. Fitting data under omnidirectional noise: a probabilistic
method for inferring petrophysical and hydrologic relations. Math. Geosci. 42 (8),
Intercept −305.62 21.157 −14.445 877–909.
F2 −25.336 3.9365 −6.436 Ceryan, N., Okkan, U., Kesimal, A., 2013. Prediction of unconfined compressive strength
F5 0.12148 0.016315 7.446 of carbonate rocks using artificial neural networks. Environmental earth sciences 68
F6 0.87147 0.037062 23.514 (3), 807–819.
F8 −0.045999 0.0027053 −17.003 Chandrashekar, G., Sahin, F., 2014. A survey on feature selection methods. Comput.
F9 0.51205 0.024267 21.1 Electr. Eng. 40 (1), 16–28.
F11 50.046 2.2713 22.034 Darbor, M., Faramarzi, L., Sharifzadeh, M., 2017. Performance assessment of rotary
drilling using non-linear multiple regression analysis and multilayer perceptron
neural network. Bull. Eng. Geol. Environ. 1–13.
Davoudi, E., Vaferi, B., 2018. Applying artificial neural networks for systematic estima-
Table 8 tion of degree of fouling in heat exchangers. Chem. Eng. Res. Des. 130, 138–153.
Results of implementing NLMR onto selected features. Del Valle, Y., Venayagamoorthy, G.K., Mohagheghi, S., Harley, R.G., Hernandez, J., 2008.
Particle Swarm Optimization: Basic Concepts, Variants and Applications in Power
Regression method RMSE R2 Adjusted R2 Systems.
A new optimizer using particle swarm theory. Micro Machine and Human Science. In:
Non-linear Model 1.91 0.81 0.804 Eberhart, R., Kennedy, J. (Eds.), MHS'95, Proceedings of the Sixth International
Linear Model 2.84 0.569 0.566 Symposium on; 1995. IEEE.
Modeling the properties of crude oil and gas systems using RBF network. In: Elsharkawy,
A.M. (Ed.), SPE Asia Pacific Oil and Gas Conference and Exhibition. Society of
Petroleum Engineers.
Acknowledgments Esfe, M.H., Wongwises, S., Naderi, A., Asadi, A., Safaei, M.R., Rostamian, H., et al., 2015a.
Thermal conductivity of Cu/TiO2–water/EG hybrid nanofluid: experimental data and
The authors thank National Iranian oil Company (NIOC) for pro- modeling using artificial neural network and correlation. Int. Commun. Heat Mass
Tran. 66, 100–104.
viding the mud Logging data. Esfe, M.H., Saedodin, S., Sina, N., Afrand, M., Rostami, S., 2015b. Designing an artificial
neural network to predict thermal conductivity and dynamic viscosity of ferromag-
netic nanofluid. Int. Commun. Heat Mass Tran. 68, 50–57.
Appendix A. Supplementary data Eskandarian, S., Bahrami, P., Kazemi, P., 2017. A comprehensive data mining approach to
estimate the rate of penetration: application of neural network, rule based models
and feature ranking. J. Petrol. Sci. Eng. 156, 605–615.
Supplementary data to this article can be found online at https://
Fausett, L.V., 1994. Fundamentals of Neural Networks: Architectures, Algorithms, and
doi.org/10.1016/j.petrol.2018.12.013.

621
S.B. Ashrafi et al.
Applications. Prentice-Hall Englewood Cliffs.
Garcia, L.P., de Carvalho, A.C., Lorena, A.C., 2015. Effect of label noise in the complexity
of classification problems. Neurocomputing 160, 108–119.
Gardner, M.W., Dorling, S., 1998. Artificial neural networks (the multilayer percep-
tron)—a review of applications in the atmospheric sciences. Atmos. Environ. 32
(14–15), 2627–2636.
Ghaffarian, N., Eslamloueyan, R., Vaferi, B., 2014. Model identification for gas con-
densate reservoirs by using ANN method based on well test data. J. Petrol. Sci. Eng.
123, 20–29.
Gholami, E., Vaferi, B., Ariana, M.A., 2018. Prediction of viscosity of several alumina-
based nanofluids using various artificial intelligence paradigms-Comparison with
experimental data and empirical correlations. Powder Technol. 323, 495–506.
Ghoreishi, S., Heidari, E., 2013. Extraction of epigallocatechin-3-gallate from green tea
via supercritical fluid technology: neural network modeling and response surface
optimization. J. Supercrit. Fluids 74, 128–136.
Goldberg, D., 1989. Genetic Algorithms in Optimization, Search and Machine Learning.
Addison-Wesley, Reading.
Grefenstette, J.J., 2013. Genetic Algorithms and Their Applications: Proceedings of the
Second International Conference on Genetic Algorithms. Psychology Press.
Drag-bit model including wear. In: Hareland, G., Rampersad, P. (Eds.), SPE Latin
America/Caribbean Petroleum Engineering Conference. Society of Petroleum
Engineers.
Hartigan, J.A., Wong, M.A., 1979. Algorithm AS 136: a k-means clustering algorithm.
Journal of the Royal Statistical Society Series C (Applied Statistics) 28 (1), 100–108.
Using trees, bagging, and random forests to predict rate of penetration during drilling. In:
Hegde, C., Wallace, S., Gray, K. (Eds.), SPE Middle East Intelligent Oil and Gas
Conference and Exhibition. Society of Petroleum Engineers.
Hegde, C., Daigle, H., Millwater, H., Gray, K., 2017. Analysis of rate of penetration (ROP)
prediction in drilling using physics-based and data-driven models. J. Petrol. Sci. Eng.
159, 295–306.
Hegde, C., Daigle, H., Gray, K.E., 2018. Performance comparison of algorithms for real-
time rate-of-penetration optimization in drilling using data-driven models. SPE J.
23 (05).
Heidari, E., Sobati, M.A., Movahedirad, S., 2016. Accurate prediction of nanofluid visc-
osity using a multilayer perceptron artificial neural network (MLP-ANN).
Chemometr. Intell. Lab. Syst. 155, 73–85.
Holland, J.H., 1992. Genetic algorithms. Sci. Am. 267 (1), 66–73.
Jacintoa, C.M., Freitas Filho, P.J., Nassarb, S.M., Roisenbergb, M., Rodriguesb, D.G.,
Limab, M.D., 2013. Optimization models and prediction of drilling rate (ROP) for the
Brazilian pre-salt layer. Chem. Eng. 33.
James, G., Witten, D., Hastie, T., Tibshirani, R., 2013. An Introduction to Statistical
Learning. Springer.
Optimization of rate of penetration in a convoluted drilling framework using ant colony
optimization. In: Jiang, W., Samuel, R. (Eds.), IADC/SPE Drilling Conference and
Exhibition. Society of Petroleum Engineers.
Kahraman, S., 2016. Estimating the penetration rate in diamond drilling in laboratory
works using the regression and artificial neural network analysis. Neural Process.
Lett. 43 (2), 523–535.
Khandelwal, M., Armaghani, D.J., 2016. Prediction of drillability of rocks with strength
properties using a hybrid GA-ANN technique. Geotech. Geol. Eng. 34 (2), 605–620.
Khandelwal, M., Faradonbeh, R.S., Monjezi, M., Armaghani, D.J., Majid, M.Z.B.A., Yagiz,
S., 2017. Function development for appraising brittleness of intact rocks using ge-
netic programming and non-linear multiple regression models. Eng. Comput. 33 (1),
13–21.
Lashkarbolooki, M., Hezave, A.Z., Ayatollahi, S., 2012. Artificial neural network as an
applicable tool to predict the binary heat capacity of mixtures containing ionic li-
quids. Fluid Phase Equil. 324, 102–107.
Lashkenari, M.S., Taghizadeh, M., Mehdizadeh, B., 2013. Viscosity prediction in selected
Iranian light oil reservoirs: artificial neural network versus empirical correlations.
Petrol. Sci. 10 (1), 126–133.
Law, M.H., Figueiredo, M.A., Jain, A.K., 2004. Simultaneous feature selection and clus-
tering using mixture models. IEEE Trans. Pattern Anal. Mach. Intell. 26 (9),
1154–1166.
Le Van, S., Chon, B.H., 2017. Evaluating the critical performances of a CO2–Enhanced oil
recovery process using artificial neural network models. J. Petrol. Sci. Eng. 157,
207–222.
The effect of initial weights on premature saturation in back-propagation learning. Neural
Networks. In: Lee, Y., Oh, S.-H., Kim, M.W. (Eds.), IJCNN-91-Seattle International
Joint Conference on; 1991. IEEE.
Li, X., Yeh, A.G.-O., 2002. Neural-network-based cellular automata for simulating mul-
tiple land use changes using GIS. Int. J. Geogr. Inf. Sci. 16 (4), 323–343.
Lorena, A.C., de Carvalho, A.C., 2004. Evaluation of noise reduction techniques in the
splice junction recognition problem. Genet. Mol. Biol. 27 (4), 665–672.
Martin, K.G., Totten, M.W., Raef, A., 2017. Characterization of a reservoir ooid shoal
complex and Artificial Neural Networks application in lithofacies prediction: mis-
sissippian St. Louis formation, Lakin fields, western Kansas. J. Petrol. Sci. Eng. 150,
1–12.
Mendes, J.R.P., Fonseca, T.C., Serapião, A., 2007. Applying a genetic neuro-model re-
ference adaptive controller in drilling optimization. World Oil 29–36.
Mirjalili, S., Mirjalili, S.M., Lewis, A., 2014. Let a biogeography-based optimizer train
your multi-layer perceptron. Inf. Sci. 269, 188–209.
Moghadassi, A., Parvizian, F., Hosseini, S., 2009. A new approach based on artificial
622
Journal of Petroleum Science and Engineering 175 (2019) 604–623
neural networks for prediction of high pressure vapor-liquid equilibrium. Australian
Journal of Basic and Applied Sciences 3 (3), 1851–1862.
Moghadassi, A., Hosseini, S.M., Parvizian, F., Al-Hajri, I., Talebbeigi, M., 2011. Predicting
the supercritical carbon dioxide extraction of oregano bract essential oil.
Songklanakarin J. Sci. Technol. 33 (5).
Moradi, H., Bahari, M.H., Naghibi Sistani, M.B., Bahari, A., 2010. Drilling rate prediction
using an innovative soft computing approach. Sci. Res. Essays 5.
Moraveji, M.K., Naderi, M., 2016. Drilling rate of penetration prediction and optimization
using response surface methodology and bat algorithm. J. Nat. Gas Sci. Eng. 31,
829–841.
Motahhari, H.R., Hareland, G., James, J., 2010. Improved drilling efficiency technique
using integrated PDM and PDC bit parameters. J. Can. Petrol. Technol. 49 (10),
45–52.
Najafi-Marghmaleki, A., Barati-Harooni, A., Tatar, A., Mohebbi, A., Mohammadi, A.H.,
2017. On the prediction of Watson characterization factor of hydrocarbons. J. Mol.
Liq. 231, 419–429.
Openshaw, S., Openshaw, C., 1997. Artificial Intelligence in Geography.
Ornek, M., Laman, M., Demir, A., Yildiz, A., 2012. Prediction of bearing capacity of
circular footings on soft clay stabilized with granular soil. Soils Found. 52 (1), 69–80.
Park, J., Sandberg, I.W., 1991. Universal approximation using radial-basis-function net-
works. Neural Comput. 3 (2), 246–257.
Pham, B.T., Bui, D.T., Pourghasemi, H.R., Indra, P., Dholakia, M., 2017. Landslide sus-
ceptibility assesssment in the Uttarakhand area (India) using GIS: a comparison study
of prediction capability of naïve bayes, multilayer perceptron neural networks, and
functional trees methods. Theor. Appl. Climatol. 128 (1–2), 255–273.
Quinlan, J.R., 1986. The effect of noise on concept learning. Mach. Learn.: An artificial
intelligence approach 2, 149–166.
Ramchoun, H., Amine, M., Idrissi, J., Ghanou, Y., Ettaouil, M., 2016. Multilayer per-
ceptron: architecture optimization and training. IJIMAI 4 (1), 26–30.
Rudin, L.I., Osher, S., Fatemi, E., 1992. Nonlinear total variation based noise removal
algorithms. Phys. Nonlinear Phenom. 60 (1–4), 259–268.
Savitzky, A., Golay, M.J., 1964. Smoothing and differentiation of data by simplified least
squares procedures. Anal. Chem. 36 (8), 1627–1639.
Shan, L., Cao, L., Guo, B., 2018. Identification of flow units using the joint of WT and
LSSVM based on FZI in a heterogeneous carbonate reservoir. J. Petrol. Sci. Eng. 161,
219–230.
A modified particle swarm optimizer. Evolutionary Computation Proceedings. In: Shi, Y.,
Eberhart, R. (Eds.), IEEE World Congress on Computational Intelligence, the 1998
IEEE International Conference on; 1998. IEEE.
Shi, X., Liu, G., Gong, X., Zhang, J., Wang, J., Zhang, H., 2016. An efficient approach for
real-time prediction of rate of penetration in offshore drilling. Math. Probl Eng. 2016.
Simon, D., 2008. Biogeography-based optimization. IEEE Trans. Evol. Comput. 12 (6),
702–713.
Neural networks for bond rating improved by multiple hidden layers. Neural Networks.
In: Surkan, A.J., Singleton, J.C. (Eds.), 1990 IJCNN International Joint Conference
on; 1990. IEEE.
Szlek, J., Mendyk, A., 2015. Package ‘fscaret’.
Feature selection methods: genetic algorithms vs. greedy-like search. Vafaie, H., Imam,
I.F. (Eds.), Proceedings of the International Conference on Fuzzy and Intelligent
Control Systems.
Vaferi, B., Eslamloueyan, R., 2015. Hydrocarbon reservoirs characterization by co-in-
terpretation of pressure and flow rate data of the multi-rate well testing. J. Petrol. Sci.
Eng. 135, 59–72.
Vaferi, B., Gitifar, V., Darvishi, P., Mowla, D., 2014. Modeling and analysis of effective
thermal conductivity of sandstone at high pressure and temperature using optimal
artificial neural networks. J. Petrol. Sci. Eng. 119, 69–78.
Vaferi, B., Eslamloueyan, R., Ayatollahi, S., 2015. Application of recurrent networks to
classification of oil reservoir models in well-testing analysis. Energy Sources, Part A
Recovery, Util. Environ. Eff. 37 (2), 174–180.
Vaferi, B., Eslamloueyan, R., Ghaffarian, N., 2016. Hydrocarbon reservoir model detec-
tion from pressure transient data using coupled artificial neural network—wavelet
transform approach. Appl. Soft Comput. 47, 63–75.
Venkatesan, P., Anitha, S., 2006. Application of a radial basis function neural network for
diagnosis of diabetes mellitus. Curr. Sci. 91 (9), 1195–1199.
Wang, R.Y., Storey, V.C., Firth, C.P., 1995. A framework for analysis of data quality re-
search. IEEE Trans. Knowl. Data Eng. (4), 623–640.
Wang, X., Tang, Z., Tamura, H., Ishii, M., Sun, W., 2004. An improved backpropagation
algorithm to avoid the local minima problem. Neurocomputing 56, 455–460.
Wang, G., Guo, L., Wang, H., Duan, H., Liu, L., Li, J., 2014a. Incorporating mutation
scheme into krill herd algorithm for global numerical optimization. Neural Comput.
Appl. 24 (3–4), 853–871.
Wang, G.-G., Gandomi, A.H., Alavi, A.H., 2014b. An effective krill herd algorithm with
migration operator in biogeography-based optimization. Appl. Math. Model. 38
(9–10), 2454–2462.
Wang, G.-G., Gandomi, A.H., Alavi, A.H., Hao, G.-S., 2014c. Hybrid krill herd algorithm
with differential evolution for global numerical optimization. Neural Comput. Appl.
25 (2), 297–308.
Warren, T., 1987. Penetration rate performance of roller cone bits. SPE Drill. Eng. 2 (01),
9–18.
Roller bit model with rock ductility and cone offset. In: Winters, W., Warren, T., Onyia, E.
(Eds.), SPE Annual Technical Conference and Exhibition. Society of Petroleum
Engineers.
S.B. Ashrafi et al.
Wu, Y., Wang, H., Zhang, B., Du, K.-L., 2012. Using radial basis function networks for
function approximation and classification. ISRN Applied Mathematics 2012.
Xue, Y., Cheng, L., Mou, J., Zhao, W., 2014. A new fracture prediction method by com-
bining genetic algorithm with neural network in low-permeability reservoirs. J.
Petrol. Sci. Eng. 121, 159–166.
Yavari, H., Sabah, M., Khosravanian, R., Wood, D., 2018. Application of an adaptive
neuro-fuzzy inference system and mathematical rate of penetration models to pre-
dicting drilling rate. Iranian Journal of Oil & Gas Science and Technology 7 (3),
623
Journal of Petroleum Science and Engineering 175 (2019) 604–623
73–100.
Yetilmezsoy, K., Ozkaya, B., Cakmakci, M., 2011. Artificial intelligence-based prediction
models for environmental engineering. Neural Netw. World 21 (3), 193.
Yilmaz, I., Kaynar, O., 2011. Multiple regression, ANN (RBF, MLP) and ANFIS models for
prediction of swell potential of clayey soils. Expert Syst. Appl. 38 (5), 5958–5966.
Yılmaz, I., Yuksek, A., 2008. An example of artificial neural network (ANN) application
for indirect estimation of rock parameters. Rock Mech. Rock Eng. 41 (5), 781–795.