670 IEEE TRANSACTIONS ON COMPONENTS, PACKAGING AND MANUFACTURING TECHNOLOGY, VOL. 8, NO.
4, APRIL 2018
SMT Solder Joint Inspection via a Novel Cascaded
Convolutional Neural Network
Nian Cai , Guandong Cen, Jixiu Wu, Feiyang Li, Han Wang, and Xindu Chen
Abstract— Due to the excellent self-learning ability of deep
learning, we propose a novel deep-learning-based method to
inspect surface-mount technology (SMT) solder joints in this
paper. In contrast to the state-of-the-art learning-based methods
in which low-level features are extracted before learning, our
method directly implements the inspection task without low-
level feature extraction, which is based on a novel cascaded
convolutional neural network (CNN). Three kinds of CNNs with
different network parameters compose the proposed cascaded
CNN. First, one kind of CNN is employed to adaptively learn
the regions of interest (ROIs) of SMT solder joint images. Then,
both the learned ROIs and the entire solder joint images are fed
into the other two kinds of CNNs, respectively. Finally, inspection
results are achieved by the learned cascaded CNN. Comparison
experiments indicate that our proposed method can achieve more
excellent inspection performance for SMT solder joints than that
of the state-of-the-art methods.
Index Terms— Cascaded network, convolutional neural net-
works (CNNs), regions of interest (ROIs), surface-mount tech-
nology (SMT) solder joint inspection, weighted-sum scheme.
I. I NTRODUCTION
S URFACE-MOUNT technology (SMT) is a technology for
producing electronic circuits in which components are
mounted or placed directly on the surfaces of printed circuit
boards (PCBs). The SMT solder joint is the joint between
the electronic component and the PCB through SMT. There
are many potential defects for SMT solder joints, which
may occur in the process of electronic component mounting.
If an SMT solder joint is bad or unreliable, it will result in
abnormality or even failure of the entire electronic equipment.
Thus, the quality of SMT solder joints on PCBs is a vital issue
of the reliability and practicality for the electronic production.
Manuscript received July 28, 2017; revised November 3, 2017 and
December 19, 2017; accepted December 30, 2017. Date of publication
January 30, 2018; date of current version April 12, 2018. This work
was supported in part by the National Natural Science Foundation
of China under Grant 61001179 and Grant 51305084, in part by the
Guangdong Natural Science Foundation under Grant 2015A030312008,
in part by the Guangdong Science and Technology Plan under Grant
2015B010104006, Grant 2015B010102014, Grant 2015B090903017, and
Grant 2015B010124001, and in part by the Guangzhou Science and
Technology Plan under Grant 201508010001, Grant 201604016022, and Grant
201604016064. Recommended for publication by Associate Editor I. Fidan
upon evaluation of reviewers’ comments. (Corresponding author: Nian Cai.)
N. Cai, G. Cen, J. Wu, and F. Li are with the School of Information Engi-
neering, Guangdong University of Technology, Guangzhou 510006, China
(e-mail: cainian@gdut.edu.cn; cenguandong@qq.com; 1183837300@qq.com;
651238355@qq.com).
H. Wang and X. Chen are with the School of Electromechanical Engineer-
ing, Guangdong University of Technology, Guangzhou 510006, China (e-mail:
wangh@gdut.edu.cn; chenxindu@gdut.edu.cn).
Color versions of one or more of the figures in this paper are available
online at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TCPMT.2018.2789453
2156-3950 © 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Vision-based inspection for solder joints has attracted more
and more researchers and engineers in the electronic industry
because of its excellent advantages of noncontact, reliability,
efficiency, and repeatability. Classifier-based methods are dom-
inant methods since the classifiers have excellent abilities
of robustness, nonlinear mapping, and self-learning [1]–[5].
Hongwei et al. [1] used an improved AdaBoost and a decision
tree to inspect chip components. In their method, the optimal
features are selected via an improved AdaBoost from various
features that are extracted from several subregions of the solder
joint. Then, a decision tree is employed for classification after
optimal feature selection. Wu et al. [2] employed a Bayes
classifier and a support vector machine (SVM) for classi-
fication of solder joints, in which color features, including
the average gray level, the percentage of highlights, and a
template-matching feature, are extracted for the classifiers.
Meyer et al. [3] established a neural network framework
including a backpropagation (BP) neural network and a radial
basis function neural network to predict the quality and relia-
bility of ball grid array solder joints. Yan and Zhou [4] utilized
several geometric features of solder joints and improved neural
network for defect inspection of plastic quad flat package sol-
der joints. Wu [5] adopted the features of section area, wetting
height, and contact angle to inspect solder joints based on
the Levenberg–Marquardt back propagation neural network.
The above classifier-based methods succeed in inspection of
some kinds of chip components, such as chip capacitors.
However, SMT solder joints are quite different from these
chip components, especially in size. Compared with other
components, such as a component with 0402 package, the sizes
of SMT solder joints are too small to extract the vital features,
which will result that the above classifier-based methods may
meet with the difficulties when inspecting.
Recently, there were only a few studies on the SMT solder
joint inspection [6]–[10], [24]. Wu and Zhang [6] proposed a
feature-extraction-based inspection algorithm for solder joints,
which is named shape digital logistic (SDL). In the SDL,
several digital features extracted from several subregions of
the solder joint image are used to evaluate the quality of
the solder joint, which include color, area, mass center, and
continuous pixels. Acciani et al. [7] employed a geometric
features vector extracted from input images, which is fed into
a multilayer perceptron (MLP) to inspect the quality of the
solder joint, named G-MLP. To achieve a better accuracy,
they made further research based on the research in [7]. They
extracted ten geometric features and eight wavelet features
as the inputs of an MLP to distinguish five kinds of solder
joints [8]. Although they claimed that their methods achieved
CAI et al.: SMT SOLDER JOINT INSPECTION VIA A NOVEL CASCADED CONVOLUTIONAL NEURAL NETWORK
good inspection performance for solder joints, their methods
should carefully design the feature extraction methods to
extract vital information from solder joint images. However,
it will inevitably meet with some difficulties because the size
of the SMT solder joint image is usually very small.
To avoid extracting the features from the small SMT solder
joints, Cai et al. [9], [10], [24] proposed three distinguished
kinds of SMT solder joint inspection methods. Another impor-
tant goal of their methods is to solve the imbalanced problem
inevitably existing at the learning stages of the classifier-
based methods in real assembly lines. Cai et al. [9] proposed
a statistical modeling-based method for SMT solder joint
inspection based on an improved visual background extrac-
tion (ViBe) algorithm. They built a solder joint model by using
the ViBe model updating scheme to detect potential defects.
A frequency map method and a metric named defect degree
are proposed to evaluate the qualities of the solder joints.
Experimental results indicated that their method achieved
high inspection accuracy and a reasonable execution time.
Cai et al. [24] made an attempt to use the Gaussian mixture
model (GMM)-based method to inspect the SMT solder joints.
Different from [9], they modeled the varying values of the
location of each pixel in an SMT solder joint image sequence
by means of a mixture of several Gaussian distributions. Also,
the GMM-based inspection method achieved a satisfactory
performance in SMT solder joint inspection. Although the
statistical modeling-based methods in [9] and [24] achieve fair
inspection performance of solder joints, they do not explore
any prior knowledge, which is often significant for the vision-
based method. Afterward, Cai et al. [10] proposed a compli-
cated inspection method for SMT solder joints based on the
robust principal component analysis (RPCA). Quite different
from [9] and [24], they formulated SMT solder joint inspection
as an optimization problem and introduced the RPCA, a cele-
brated optimization technique, to model an appearance model
of qualified SMT solder joints for subsequent inspection. They
incorporated the location prior knowledge related to human
perception into the defect score to evaluate the defects of SMT
solder joints. However, it is well known that the RPCA is a
very complicated mathematical method. There is no report on
applications of the RPCA in real industry. Also, the defect
score with location prior knowledge in [10] is defined as some
complicated mathematical formulas. Hence, the method in [10]
is incomprehensible for many researchers and engineers in the
field of SMT.
Recently, more and more researchers and engineers in the
field of computer vision devoted themselves to convolutional
neural network (CNN), first proposed by Lecun et al. [11], due
to its excellent self-learning ability. CNN can automatically
extract effective features from a large volume of data through
its special convolutional structures and achieve dimensionality
reduction and spatial invariance through pooling layers. This
means that it should not extract the features as the input of the
network compared with the traditional classifiers. Therefore,
applications of CNN increase tremendously, especially in
image processing, such as image classification [12], [13],
object detection [14], [15], and segmentation [16], [17].
Bian et al. [17] proposed a multiscale fully convolutional
671
TABLE I
N UMBER OF F EATURE M APS IN E ACH CNN
network combining networks trained at various scales for the
inspection of aircraft engine blades. Actually, as claimed by
them, they only performed an image segmentation task for
segmenting and quantifying the spallation of an aircraft engine
blade. However, their CNN-based method may be not suitable
to annotate the defects of SMT solder joints because there are
no distinguished differences between the qualified SMT solder
joint and the unqualified one. Also, their CNN-based method
should combine another classifier to implement the inspection
task.
In this paper, we propose a novel cascaded CNN for SMT
solder joint inspection, whose input is the whole SMT solder
joint image rather than its features. The proposed network
consists of three kinds of CNNs. One kind of CNN is designed
to adaptively annotate the regions of interest (ROIs) of SMT
solder joint images. The entire image and the learned ROI of
the SMT solder joint are simultaneously fed into two different
kinds of CNNs. Finally, inspection results are achieved by
weighting the outputs of the two CNNs.
The rest of this paper is organized as follows. Section II
introduces the proposed method, including the architecture of
the proposed cascaded CNNs, the inspection process, and the
training strategy at the training stage. Section III introduces the
data sets involved in this paper and shows experimental results
and discussions. Finally, we make a conclusion in Section IV.
II. P ROPOSED M ETHOD
A. Architecture of the Proposed Cascaded CNN
The architecture of the proposed cascaded CNN is illus-
trated in Fig. 1. It contains three kinds of CNNs (CNN-1,
CNN-2, and CNN-3 in Fig. 1), each of which has the same
structure with different numbers of feature maps. Each CNN
contains three convolutional layers, one max-pooling layer
and a full-connected layer with a softmax layer, as illustrated
in Fig. 2. The numbers of feature maps in different kinds of
CNNs are listed in Table I.
The kernel size of each convolution layer is set to 3 × 3.
This is because the SMT solder joint itself is very small
and slender so that the oversize convolutional kernel not
only increases the computation complexity but also maybe
ignores some elaborate information of the SMT solder joint
image. The pooling layer is a down sampling course aiming
at simplifying calculation and importing invariance, however,
at the same time results in losing some image information.
Hence, we only use one pooling layer in each CNN.
Each input image fed to CNN-1 is an entire RGB image
of an SMT solder joint with the size of 84 × 16. The input
of each CNN-2 is the patches of the SMT solder joint image
with the size of 16 × 16. The output of each CNN-2 is the
probability of the desired ROI of the SMT solder joint image.
672 IEEE TRANSACTIONS ON COMPONENTS, PACKAGING AND MANUFACTURING TECHNOLOGY, VOL. 8, NO. 4, APRIL 2018

Fig. 1. Architecture of the proposed cascaded CNN. The input and output of the networks are an entire image and the corresponding inspection result,
respectively.

of the sliding window and the stride to 16 × 16 pixels and

1 pixel, respectively. Thus, we can achieve 69 patches for each
solder joint image. All the patches are fed into CNN-2, and the
prediction results will be output from the softmax layer later.
The prediction result of each patch is the probability that the
patch is the ROI of the SMT solder joint image. If there is
a maximal probability for all the patches, the corresponding
patch is the desired ROI of the solder joint image.

C. Inspection by Combining CNN-1 and CNN-3

Fig. 2. Architecture of CNN-x (x = 1, 2, 3). Each kind of CNNs has the
same structure with different numbers of feature maps described in Table I. As illustrated in Fig. 1, we combine CNN-1 and CNN-3
to make a joint decision on the quality of the input SMT
solder joint (OK for qualified SMT solder joint and NG for
The desired ROI is achieved by analyzing the probability, unqualified one). The output of CNN-1 or CNN-3 is the
which will be described later. Thus, the input of CNN-3 is probability that the input SMT solder joint is qualified. This
the learned ROI of the SMT solder joint image, and its size joint decision can take full advantages of global and vital local
is 16 × 16. The weighted sum of the outputs of CNN-1 and information of the solder joint image. Assume the probabilities
CNN-3 is the inspection result for the SMT solder joint. This for an input solder joint achieved by CNN-1 and CNN-3 as
weighted-sum scheme takes full advantages of global and vital P1 and P3 , respectively, the final output Out of the proposed
local information of the SMT solder joint, which will result cascaded CNN is
in excellent inspection performance.
Out = (1 − ρ)P1 + ρ P3 (1)
σCNN-3
B. ROI of the Solder Joint Image Achieved by CNN-2 ρ = (2)
σCNN-1 + σCNN-3
The ROI of the solder joint image contains much vital N
information, which implies the potential defect of the solder σCNN-1 = [ϕ(i )(1 − p1(i )) + (1 − ϕ(i )) p1(i )] (3)
joint. Note that the ROIs may exist in the different parts of i=1
SMT solder joint images. In our proposed cascaded CNN, 
N
the learned CNN-2 is used to adaptively achieve the desired σCNN-3 = [ϕ(i )(1 − p3(i )) + (1 − ϕ(i )) p3(i )] (4)
ROI of the inspecting solder joint. i=1

At the training stage, positive patches (i.e., desired ROIs) 0, the training smaple i is qualified
and negative patches are fed into CNN-2 to train a two-class ϕ(i ) = (5)
1, the training smaple i is unqualified
classification model. These patches are achieved from the
solder joint images, whose sizes are 16 × 16. Please refer to where N represents the number of the training samples, and
Section III-B for how the patches are generated. px (i ) means the probability of CNN-x that the i th sample
At the inspecting stage, the patches input into CNN-2 are is qualified. Obviously, the probability for unqualified sample
achieved via a sliding-window scheme. Here, we set the size is 1 − p x (i ). σCNN-x (x = 1, 3) can be considered to be the
CAI et al.: SMT SOLDER JOINT INSPECTION VIA A NOVEL CASCADED CONVOLUTIONAL NEURAL NETWORK
Fig. 3. Image acquisition system for AOI.
impact of the CNN-x on the proposed cascaded CNN. ρ is
the weight factor that is calculated via σCNN-x (x = 1, 3). Out
represents the final probability that the input SMT solder joint
is qualified. If Out is greater than a predefined threshold, the
input SMT solder joint is considered to be qualified; otherwise,
it is unqualified. Here, we set the threshold to 0.5.
D. Training Strategy
In this paper, we implement our proposed method using
Caffe [18], an efficient framework for CNN. Stochastic gradi-
ent descent and BP algorithms are used to update the weights
of each layer by a linear combination of the negative gradient
and the previous weight updating [11], [19]. All weights are
initialized with the Gaussian distribution, whose mean value
and variance are 0 and 0.01, respectively. The learning rate
LR is 0.01 at the beginning and follows the descent formula
as:
L R = lr b × (1 + γ × iter)−power
where the initial learning rate lr b is 0.01, γ is 0.001, power
is 0.75, and iter is the current number of iterations. A gradual
decreasing learning rate can guarantee the network loss getting
close to minimum. The momentum and weight decay are
set to 0.9 and 0.005, respectively, so that the network can
avoid converging to local minimum and overfitting. A rectified
linear function ReLU [12] is used throughout as the activation
function.
All training processes are conducted in a computer with a
NVIDIA Quadro K4200 4-GB GPU. The testing experiments
are running with an Intel Xeon E5-2630V3 2.4-GHz CPU and
64-GB memory.
III. E XPERIMENTS AND D ISCUSSION
A. Image Acquisition System
In this paper, all the SMT solder joint samples are collected
from the electronic components mounted on the PCB by means
of SMT. The SMT solder joint images are collected by an
automatic optical inspection (AOI) image acquisition system,1
as illustrated in Fig. 3. The system consists of a charge-coupled
1 The AOI image acquisition system is provided by EKTion (ShenZhen)
Technology Co., LTD. Available at http://www.ekt-tech.com.
673
Fig. 4. Segmentation process used to crop an individual solder joint image
from a PCB image.
Fig. 5. Ten solder joint samples acquired by the image acquisition system.
device (CCD) color digital camera and a three-color (red,
green, and blue) hemispherical light-emitting diode array light
resource.
In the system, three different color lights irradiate the
electronic component directly and the component reflects the
light toward the CCD camera above. Different regions present
different colors toward the camera so that we are able to obtain
the 3-D shape and position information from the 2-D images
acquired from the camera [21], [22].
First, a PCB image is captured by the AOI system, a part of
which is illustrated in Fig. 4. The main body of each image is
(6)
an SMT component and surrounded by more than 100 solder
joints. Since there is a clear boundary between each solder
joint, each individual solder joint can be easily cropped and
extracted by template matching and projection segmentation
from the image of PCB, as illustrated in Fig. 4.
For simplicity, a part of the acquired SMT solder joint image
is illustrated in Fig. 5 to describe the differences between
the qualified and unqualified solder joints, which contains five
qualified solder joints (the left five) and five unqualified solder
joints (the right five). For more details, we select a qualified
sample and an unqualified sample to qualitatively distinguish
them in structure (shown in Fig. 6).
For qualified samples, the red light irradiates the pad region
vertically, and therefore, red color will appear on this region
if the pad region is flat enough. For the solder joint region,
the blue–green light is reflected toward the CCD camera
because of its slope structure. Thus, the solder joint region
of the qualified sample displays blue–green-like color. On the
contrary, unqualified SMT solder joint samples, such as false
welding and lack of weld, show different colors in the solder
joint regions compared with qualified ones. For example,
as shown in Figs. 5 and 6, all qualified samples reflect the
red lights on the pad regions and the blue–green lights on
the solder joint regions. However, all unqualified samples
674 IEEE TRANSACTIONS ON COMPONENTS, PACKAGING AND MANUFACTURING TECHNOLOGY, VOL. 8, NO. 4, APRIL 2018
Fig. 6. Zoomed-in view of two SMT solder joint samples.
reflect the red lights on the solder joint regions, but some
parts of the pad regions appear blue–green and some appear
red. Especially, the solder joint region, labeled by a rectangle
in Fig. 6, is the vital region for inspection, in which some
important prior knowledge internally exists. The differences
between qualified and unqualified samples mainly exist in the
solder joint regions. The unqualified samples are red-like in
appearance in the solder joint region, while the qualified ones
are blue–green. Hence, the solder joint region plays a decisive
role in the inspection process, while the pad region plays an
auxiliary role.
However, the acquired SMT solder joint image is quite
small in size, whose size is only 84 × 16. Thus, it is
difficult to extract the prior knowledge from the image by
means of traditional feature extraction methods. In this paper,
we propose a novel cascaded CNN for inspecting SMT solder
joints, which incorporates global information and local prior
knowledge.
B. Data Set
The 2002 SMT solder joint samples are acquired by the
image acquisition system mentioned above, which involve
1108 qualified SMT solder joint images and 894 unqualified
SMT solder joint images. Each image is an 84 × 16 RGB
image.
In order to train CNN-1, 2002 solder joint sample images
are randomly divided into two categories: the training samples
involving 966 qualified samples and 752 unqualified samples,
and the inspecting samples involving 142 qualified samples
and 142 unqualified samples. Then, all the sample images
in two categories are rotated at −3°, −2°, −1°, 1°, 2°,
and 3°. Thus, 6762 qualified SMT solder joint images and
5264 unqualified ones should compose the training data set
for CNN-1. The 994 qualified SMT solder joint images and
994 unqualified ones should compose the inspecting data set
for CNN-1. However, for the convenience of statistics, we ran-
domly move six qualified samples and six unqualified samples
from the training data set to the inspecting data set. Thus, there
are 6756 qualified samples and 5258 unqualified samples in
the training data set, and there are 1000 qualified samples and
1000 unqualified samples in the inspecting data set.
For CNN-2, 1105 samples are randomly selected from
2002 raw samples, which involve qualified and unqualified
samples. For each sample, an ROI is manually labeled as
a positive patch. Also, three or four negative patches are
randomly selected, which do not have any overlaps with
the positive patch. The sizes of the patches are 16 × 16.
Thus, we can obtain 1105 positive patches and 3675 negative
patches. Here, 981 positive patches and 3405 negative patches
are used for training and the rest is used for inspecting.
The desired ROIs acquired by CNN-2 are the input of
CNN-3, whose sizes are 16 × 16. It means that each solder
joint sample image composing the training and inspecting data
sets of CNN-1 has a corresponding desired ROI. That is to say,
there are 6756 ROIs for qualified samples and 5258 ROIs for
unqualified samples in the training data set of CNN-3, and
there are 1000 ROIs for qualified samples and 1000 ROIs for
unqualified samples in the inspecting data set.
C. Metrics
To objectively evaluate the performance of the inspection
methods, several metrics are used here, which are recall,
precision, and F-measure [23]. These metrics are usually
applied in real industries to judge the performance of the
inspection method, which are formulated as
TP
recall = (7)
TP + FN
TP
precision = (8)
TP + FP
precision · recall
F-measure = 2 · (9)
precision + recall
where FN (false negatives) denotes the number of qualified
SMT solder joints that are identified as unqualified ones, while
FP (false positives) means the opposite. TP (true positives)
represents the number of qualified samples identified correctly.
Recall and precision are usually introduced to judge the pros
and cons of inspection algorithms. F-measure is a tradeoff
between recall and precision, indicating the performance on
the overall samples.
D. Parameter Determination
The weight in (1) is quite important for the final inspec-
tion performance. To achieve a reliable and stable weight,
we adopted a cross-validation approach [20] to adaptively
obtain the exact value of ρ based on (1)–(5). In the pro-
posed cascaded CNN, the size of each convolutional kernel is
3 × 3. We randomly divided all the training samples into
four parts with an equal number, which is labeled 1, 2, 3,
and 4 in Table II, correspondingly. Three of them are used to
calculate ρ and one for verification every time. As indicated
in Table II, for the training data set in this paper, the optimal
weight ρ is 0.3041.
To verify whether the weight adaptively determined above
is an optimal one, we conducted an exhaustive experiment
in which the values of ρ vary from 0 to 1 with the stride
of 0.01. In Fig. 7, the vertical axis represents the number of
the qualified SMT solder joints identified as unqualified or the
opposite; the horizontal axis represents different values of ρ.
As indicated in Fig. 7, we can find that both the number of
CAI et al.: SMT SOLDER JOINT INSPECTION VIA A NOVEL CASCADED CONVOLUTIONAL NEURAL NETWORK
TABLE II
VALUES OF THE W EIGHT ρ C ALCULATED BY
D IFFERENT T RAINING S AMPLES
Fig. 7. Inspection results with respect to different values of ρ.
TABLE III
C OMPARISONS R ESULTS A CHIEVED BY THE CNN S W ITH D IFFERENT
C ASCADED S TRUCTURES AND C ONVOLUTIONAL K ERNELS
the qualified SMT solder joints identified as unqualified (FN)
and the number of unqualified ones identified as qualified (FP)
reach the lowest values when ρ = 0.3041. This fact indicates
that the optimal weight ρ can be determined based on (1)–(5).
Next, we discuss the influences of the size of the
convolutional kernel and the cascaded structure. Each
network in this experiment has three convolutional layers and
one pooling layer. Here, the weight ρ is adaptively determined
based on (1)–(5). As indicated in Table III, the network that
cascades CNN-1, CNN-2, and CNN-3 achieves better
inspection performance than that of the network that consists
of only CNN-1 and the network that cascades CNN-2 and
CNN-3. This is because the former cascaded network
combines the global and vital local information of the SMT
solder joint image compared with the other two networks.
675
TABLE IV
C OMPARISON R ESULTS A CHIEVED BY D IFFERENT I NSPECTION M ETHODS
Also, the proposed cascaded network with 3 × 3 convolutional
kernels achieves better inspection performance than that with
5 × 5 convolutional kernels. Especially, the precision achieved
by the proposed cascaded network with 3 × 3 convolutional
kernels is 1, which is quite valuable in practice.
E. Comparisons With the State-of-the-Art Inspection Methods
In this section, we compare our proposed cascaded
CNN with some state-of-the-art inspection methods, which
are Bayes- [2], SVM- [2], G-MLP- [7], ViBe- [9], and
GMM-based [24] methods. G-MLP-, Bayes-, and SVM-
based methods are traditional classifier-based methods, which
must extract some features from SMT solder joint images.
ViBe- and GMM-based methods are statistical-modeling
method without feature extraction.
As indicated in Table IV, three traditional classifier-
based methods achieve the worst inspection performance in
recall, precision, and F-measure. This is because the features
extracted from such small SMT solder joint images cannot
well characterize the defects of SMT solder joints. The
ViBe- and GMM-based methods avoid feature extraction and
inspect the SMT solder joints by comparing it with the quali-
fied model established based on an improved ViBe algorithm
and a GMM model, respectively. Thus, the two methods out-
perform three traditional classifier-based methods for SMT sol-
der joint inspection. Although the ViBe-based method achieves
a good recall rate, the precision rate is only 0.8625. This is
because the qualified model in the ViBe-based method is estab-
lished from qualified SMT solder joints, which results that
unqualified SMT solder joints tend to be identified as qualified.
The GMM-based method achieves an acceptable precision.
However, the initial values of GMM have a great influence on
inspection results, which are difficultly selected. The recall rate
for the GMM-based method is relatively low, although we have
carefully selected the initial values, which is also unacceptable
in the real industry. Our proposed cascaded CNN does not
need to extract features from SMT solder joints as the input
of the network. The network can implicitly combine the global
and vital local information of SMT solder joint images in its
carefully designed architecture. Thus, our proposed method
achieves the best inspection performance, especially the recall
and the precision are 0.998 and 1, respectively. This fact indi-
cates that our proposed method is feasible in real industries.
IV. C ONCLUSION
In this paper, we propose a novel SMT solder joint
inspection method via a novel cascaded CNN. The proposed
676 IEEE TRANSACTIONS ON COMPONENTS, PACKAGING AND MANUFACTURING TECHNOLOGY, VOL. 8, NO. 4, APRIL 2018
cascaded CNN consists of three kinds of CNNs with the
same structure and different network parameters. The proposed
method adaptively determines the ROI of the SMT solder
joint image via a sliding-window scheme based on one kind
of CNN. It utilizes the characteristic of CNN to combine
the global and vital local information of the SMT solder
joint image to make accurate predictions by using a cascaded
structure. The experimental results indicate that the proposed
method is superior to the state-of-the-art methods for SMT
solder joint inspection. Especially, the precision achieved by
the proposed method is 1 and much better than all the other
existing methods. This fact indicates that our proposed method
has the potential application for SMT solder joint inspection in
real industries. In addition, our proposed method is not limited
to a specific inspection task like SMT solder joint inspection.
It is also suitable for any inspection work combining the global
information and local vital information for accurate inspection.
In the future, we will apply the proposed method in the related
inspection fields.
R EFERENCES
[1] X. Hongwei, Z. Xianmin, K. Yongcong, and O. Gaofei, “Solder joint
inspection method for chip component using improved AdaBoost and
decision tree,” IEEE Trans. Compon., Packag., Manuf. Technol., vol. 1,
no. 12, pp. 2018–2027, Dec. 2011.
[2] H. Wu, X. Zhang, H. Xie, Y. Kuang, and G. Ouyang, “Classification of
solder joint using feature selection based on Bayes and support vector
machine,” IEEE Trans. Compon., Packag., Manuf. Technol., vol. 3, no. 3,
pp. 516–522, Mar. 2013.
[3] S. Meyer, H. Wohlrabe, and K. J. Wolter, “Neural network modeling
to predict quality and reliability for BGA solder joints,” in Proc.
60th Electron. Compon. Technol. Conf. (ECTC), Las Vegas, NV, USA,
Jun. 2010, pp. 1596–1603.
[4] T. Yan and D. Zhou, “The intelligent detection method study of PQFP
solder joint defects based on improved neural network,” in Proc. 12th
Int. Conf. Electron. Packag. Technol. High Density Packag., Shanghai,
China, Aug. 2011, pp. 1–4.
[5] Z. Wu, “Study on intelligent identification technology of solder joints
defects based on LMBP neural network,” in Proc. 12th Int. Conf.
Electron. Packag. Technol. High Density Packag., Shanghai, China,
Aug. 2011, pp. 1–6.
[6] F. Wu and X. Zhang, “Feature-extraction-based inspection algorithm
for IC solder joints,” IEEE Trans. Compon., Packag., Manuf. Technol.,
vol. 1, no. 5, pp. 689–694, May 2011.
[7] G. Acciani, G. Brunetti, and G. Fornarelli, “Application of neural
networks in optical inspection and classification of solder joints in
surface mount technology,” IEEE Trans. Ind. Informat., vol. 2, no. 3,
pp. 200–209, Aug. 2006.
[8] G. Acciani, G. Brunetti, and G. Fornarelli, “Automatic detection of
solder joint defects on integrated circuits,” in Proc. IEEE Int. Symp.
Circuits Syst., New Orleans, LA, USA, May 2007, pp. 1021–1024.
[9] N. Cai, J. Lin, Q. Ye, H. Wang, S. Weng, and B. W. K. Ling, “A new
IC solder joint inspection method for an automatic optical inspec-
tion system based on an improved visual background extraction algo-
rithm,” IEEE Trans. Compon., Packag., Manuf. Technol., vol. 6, no. 1,
pp. 161–172, Jan. 2016.
[10] N. Cai, Y. Zhou, Q. Ye, G. Liu, H. Wang, and X. Chen, “IC solder
joint inspection via robust principle component analysis,” IEEE Trans.
Compon., Packag., Manuf. Technol., vol. 7, no. 2, pp. 300–309,
Feb. 2017.
[11] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based
learning applied to document recognition,” Proc. IEEE, vol. 86, no. 11,
pp. 2278–2324, Nov. 1998.
[12] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification
with deep convolutional neural networks,” in Proc. Adv. Neural Inf.
Process. Syst., Stateline NV, USA, 2012, pp. 1097–1105.
[13] K. Simonyan and A. Zisserman, “Very deep convolutional networks
for large-scale image recognition,” in Proc. Comput. Sci., Sep. 2014,
pp. 1–14.
[14] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards
real-time object detection with region proposal networks,” IEEE Trans.
Pattern Anal. Mach. Intell., vol. 39, no. 6, pp. 1137–1149, Jun. 2017.
[15] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look
once: Unified, real-time object detection,” in Proc. IEEE Conf. Com-
put. Vis. Pattern Recognit. (CVPR), Las Vegas, NV, USA, Jun. 2016,
pp. 779–788.
[16] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks
for semantic segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern
Recognit. (CVPR), Boston, MA, USA, Jun. 2015, pp. 3431–3440.
[17] X. Bian, S. N. Lim, and N. Zhou, “Multiscale fully convolutional
network with application to industrial inspection,” in Proc. IEEE Winter
Conf. Appl. Comput. Vis. (WACV), Lake Placid, NY, USA, Mar. 2016,
pp. 1–8.
[18] Y. Jia et al. (2014). “Caffe: Convolutional architecture for fast feature
embedding.” [Online]. Available: https://arxiv.org/abs/1408.5093
[19] J. Bouvrie, “Notes on convolutional neural networks,” Neural Nets,
2006. [Online]. Available: http://cogprints.org/5869/
[20] R. Kohavi, “A study of cross-validation and bootstrap for accuracy
estimation and model selection,” in Proc. Int. Joint Conf. Artif. Intell.,
1995, pp. 1137–1145.
[21] F. Wu, X. Zhang, Y. Kuan, and Z. He, “An AOI algorithm for PCB
based on feature extraction,” in Proc. World Congr. Intell. Control
Autom. (WCICA), Jun. 2008, pp. 240–247.
[22] N. Li, K.-L. Pan, and P. Li, “Research on solder joint intelligent optical
inspection analysis,” in Proc. Int. Conf. Electron. Packag. Technol. High
Density Packag. (ICEPT-HDP), Jul. 2008, pp. 1–4.
[23] J. Davis and M. Goadrich, “The relationship between Precision-Recall
and ROC curves,” in Proc. Int. Conf. Mach. Learn., 2006, pp. 233–240.
[24] N. Cai, Q. Ye, G. Liu, H. Wang, and Z. Yang, “IC solder joint inspection
based on the Gaussian mixture model,” Soldering Surf. Mount Technol.,
vol. 28, no. 4, pp. 207–214, 2016.
Nian Cai received the B.Ed. degree in physics from
Nanjing University, Nanjing, China, in 1999, and
a direct-promotion program for the Ph.D. Diploma
degree in biophysics from the Institute of Bio-
physics, Chinese Academy of Sciences, Beijing,
China, in 2004.
He is currently a Professor with the School of
Information Engineering, Guangdong University of
Technology, Guangzhou, China. He held a postdoc-
toral program in pattern recognition and intelligent
system with Shanghai Jiaotong University, Shanghai,
China, in 2006. His current research interests include image processing,
pattern recognition, computer vision, and signal processing and related areas.
Guandong Cen received the B.E. degree in digital
media technology from the Communication Uni-
versity of China, Beijing, China, in 2016. He is
currently pursuing the M.E. degree in information
and communication engineering with the Guang-
dong University of Technology, Guangzhou, China.
His current research interests include deep learning
and computer vision.
Jixiu Wu received the B.E. degree in communica-
tion engineering with the Guangdong University of
Technology, Guangzhou, China, in 2017, where he
is currently pursuing the M.E. degree in information
and communication engineering.
His current research interests include computer
vision and machine learning.
CAI et al.: SMT SOLDER JOINT INSPECTION VIA A NOVEL CASCADED CONVOLUTIONAL NEURAL NETWORK 677

Feiyang Li is currently pursuing the B.E. degree
in information engineering with the Guangdong
University of Technology, Guangzhou, China.
His current research interests include deep learning
and computer vision.
ing, mechanotronics, and intelligent manufacturing system and related areas.
Xindu Chen received the B.S. and M.S. degrees
from the Department of Electrical Power, Huazhong
University of Science and Technology, Wuhan,
China, in 1989 and 1992, respectively, and the Ph.D.
degree in mechanical engineering with the School
of Mechanical and Science, Huazhong University of
Science and Technology, in 1995.
He is currently a Professor with the School of
Mechanical and Electrical Engineering, Guangdong
University of Technology, Guangzhou, China. His
current research interests include precision engineer-

Han Wang received the B.Ed. and the direct-

promotion program for Ph.D. Diploma degrees
in mechanical engineering with the School of
Mechanical and Electrical Engineering, Xiamen
University, Xiamen, China, in 2003 and 2010,
respectively.
He is currently a Professor with the School
of Mechanical and Electrical Engineering,
Guangdong University of Technology, Guangzhou,
China. His current research interests include
optical engineering, instrument engineering, and
micro–nano-processing technology and related areas.