26th Telecommunications forum TELFOR 2018
Convolutional Neural Network based SMS
Spam Detection
Milivoje Popovac, Mirjana Karanovic, Srdjan Sladojevic, Marko Arsenovic, Andras Anderla
Abstract — SMS spam refers to undesired text message.
Machine Learning methods for anti-spam filters have been
noticeably effective in categorizing spam messages. Dataset
used in this research is known as Tiago’s dataset. Crucial step
in the experiment was data preprocessing, which involved
reducing text to lower case, tokenization, removing
stopwords. Convolutional Neural Network was the proposed
method for classification. Overall model’s accuracy was
98.4%. Obtained model can be used as a tool in many
applications.
Keywords — CNN, Cost-sensitive classification,
Imbalanced dataset, Machine Learning, SMS Spam
I. INTRODUCTION
T EXT messages are used by youth and adults for
personal, social purposes and in business, hence, it is
becoming more popular for marketers to use in their
campaigns. Some of the most common means of sending
text messages is via SMS and email. According to [1],
SMS messages have a 98% open rate, while email only has
a 20% open rate, additionally, as stated in [2], SMS
messages have a 45% response rate, as opposed to email
which has a 6% response rate. For these reasons, marketers
in some regions choose SMS as their primary way of
advertising, which leads to spamming. SMS spam refers to
any undesired text message, which marketers send at
random for commercial purposes. It can take the form of a
simple message that contains a link to a number or a link to
a website etc. Some of the reasons why spam is bad,
includes: it's annoying, it wastes time, it misuses resources,
and it’s an invasion of privacy. However, the main reason
is that spamming costs its victims much more than it costs
its senders. Thus, SMS spam detection proves as a
Milivoje B. Popovac, Faculty of Technical Sciences, Trg Dositeja
Obradovica 6, University of Novi Sad, 21101 Novi Sad, Serbia (phone:
381-21-4852186, e mail: milivojepopovac@uns.ac.rs )
Mirjana M. Karanovic, Faculty of Technical Sciences, Trg Dositeja
Obradovica 6, University of Novi Sad, 21101 Novi Sad, Serbia (phone:
381-21-4852186, e mail: mkaranovic@uns.ac.rs )
Srdjan Sladojevic, Faculty of Technical Sciences, Trg Dositeja
Obradovica 6, University of Novi Sad, 21101 Novi Sad, Serbia (phone:
381-21-4852186, e mail: sladojevic@uns.ac.rs )
Marko Arsenovic, Faculty of Technical Sciences, Trg Dositeja
Obradovica 6, University of Novi Sad, 21101 Novi Sad, Serbia (phone:
381-21-4852186, e mail: arsenovic@uns.ac.rs )
Andras Anderla, Faculty of Technical Sciences, Trg Dositeja
Obradovica 6, University of Novi Sad, 21101 Novi Sad, Serbia (phone:
381-21-4852117, e mail: andras@uns.ac.rs )
978-1-5386-7171-9/18/$31.00 ©2018 IEEE
Serbia, Belgrade, November 20-21, 2018.
necessity.
SMS spam is a problem that doesn’t have clear and
simple solution yet and many efforts have been made to
make a model that will detect (classify) SMS spam.
Although these models can be helpful, there are still
opportunities for further enhancements. In this paper an
experiment was conducted in order to classify spam and
non-spam messages, by using Convolutional Neural
Network.
This research confirms that CNN can be used to make a
model for spam detection that categorize messages with
high accuracy. Moreover, adjusted model can be applied to
sentiment analysis, text categorization, or spam detecting
in other types of communication media such as emails,
social network systems, online reviews, etc.
The rest of the paper is organized as follows: Section II
presents the related work, Section III contains description
of the experiment, and Section IV presents the results and
discussion. Section V holds the conclusion, after which a
list of literature is given.
II. RELATED WORK
Detection of spam messages, as a classification problem,
can be solved by using different methods of data analysis.
With respect to the chosen method for this research, below
are listed papers in which artificial intelligence methods
have been used.
Authors of paper [3], also creators of Tiago’s dataset,
have tested multiple machine learning algorithms and
provided the foundation for further research. They used
two tokenizers (in order to recognize a domain and to
preserve symbols that help in separating spam from ham
messages) in combination with classifiers, where SVM,
Boosted NB, Boosted C4.5 and PART achieved best
results. SVM showed the best results, with an accuracy of
over 97.5%; it caught 83.10% of spams, and blocked only
0.18% of non-spam messages.
In paper [4], SMS Spam Corpus and SMS Spam
Collection datasets were used, individually and merged.
Both of the two used methods, Naive Bayes classifier and
FP-Growth Algorithm, accomplished an accuracy rate
superior than 90%. The accuracy best average (98.5%) was
obtained with the implementation of the FP-Growth
algorithm on Tiago’s dataset.
In paper [5], the research was conducted on the Tiago’s
dataset. In the experiment, Naïve Bayes outperformed
Random Forest algorithm and Logistic Regression
algorithm. NB provided the results of almost 98.5%
accuracy.
The same dataset was used in paper [6]. Authors
concluded that the boosting of Random Forest and SVM
algorithms gave the best results, and by using both
Linguistic Inquiry, Word Count (LIWC) and SMS-specific
content based features can positively affect results.
After comparing multiple algorithms, the authors of
paper [7] have chosen GentleBoost Classifier as the most
suitable to use on Tiago’s dataset. The specified algorithm
is a mix of LogitBoost and AdaBoostM1 algorithms, which
makes it convenient for binary Classification and
unbalanced data. This approach has led to an accuracy of
over 98.3%.
In paper [8], authors used the following algorithms: NB,
SVM, k-NN, RF and AdaBoost. All of the mentioned
algorithms accomplished an accuracy rate superior than
97%, whereas multinomial naive Bayes with Laplace
smoothing and SVM with linear kernel are among the best
classifiers.
Another research on Tiago’s dataset was presented in
paper [9]. Besides pre-processing and classification with
various classifiers, the experiment included Clustering
using K-Means algorithm or NMF Model. After the
mentioned steps, a solution for SMS Thread Identification
was proposed. The research led to the conclusion that the
SVM algorithm performs better in categorizing the SMS
messages and the combination of NMF and SVM
algorithm gave good results in thread identification.
Authors of paper [10] compared performance of
multiple algorithms on four spam datasets. They proposed
a novel spam filter integrating an N-gram tf.idf feature
selection, modified distribution-based balancing algorithm
and a regularized deep multi-layer perceptron NN model
with rectified linear units (DBB-RDNN-ReL). The
accuracy off suggested model on Tiago’s dataset was
approximately 98.5%, FP rate was around 0.0024, and the
auc was 0.961.
Similar methods can also be used for: email spam
detection, sentiment analysis, text categorization etc. The
authors of paper [11] trained CNN for sentence
classification; this paper proves that simple CNN with one
layer of convolution can achieve extraordinary results. In
paper [12], a study was made to test application of CNN-
RNN model for Multi-label Text Categorization. Feature
extraction was conducted using CNN, while RNN was
used to extract local semantic information and to model
label correlation. It is shown that the size of the dataset has
a significant impact on the performance of the applied
model. Small dataset may lead to overfitting, while large
dataset can achieve remarkable performance.
Considering presented basis, this study was conducted in
order to evaluate applicability of CNN to given problem.
III. MATERIALS AND METHODS
A. Dataset
The dataset used in this research is also known as
Tiago’s dataset [13]. It is composed of 5,574 English, real
and non-encoded messages labeled as ham (non-spam) or
spam.
The dataset represents a collection of 4 subsets:
• A subset extracted from the Grumbletext Web site
(UK forum). It contains 425 SMS spam messages.
• A collection of 3,375 SMS non-spam messages
provided by NUS SMS Corpus (NSC) and collected for
research at the Department of Computer Science at the
National University of Singapore.
• SMS Spam Corpus v.0.1 Big subset. It consists of
322 spam and 1,002 ham messages.
• A collection of 450 SMS ham messages from
Caroline Tag’s PhD Thesis.
B. Preprocessing
One of the essential steps for creating a good model for
spam classification is preprocessing text data. Converting
text into something an algorithm can work with represents
a complex process. Primarily, text often has a variety of
capitalization, which has no big impact on the final model,
so for the purpose of this research, text from the whole
dataset is reduced to lower case for simplicity.
In the next step, tokenization (splitting text into
individual words) was conducted in order to perform
stemming. By reducing words to a root, sentiment of the
text was preserved. Also, words which serve only for
connecting parts of given text (messages), rather than
influencing model (so-called stopwords), were removed.
Further, extraction of polarity and subjectivity for every
message was provided in order to interpret sentiment
analysis.
In the final phase of preprocessing, text from messages
was converted into a matrix of TF-IDF features. With this
step, the information about how important word is to a text
was reflected. The values within the given matrix increased
correspondingly to the number of times a word appeared in
messages. Fig. 1 shows the most often repeated words in
spam messages.
Fig. 1. Most frequent words in spam messages
Described preprocessing and following experiment are
implemented using Python programming language, in
Spyder IDE.
C. Experiment
CNNs proved to be useful for image classification [14]-
[16], nevertheless it can be quite useful for text and time-
series data [17], [18].
A CNN consists of an input and output layer, including
one or more hidden layers between them. The hidden
layers of a CNN typically consist of convolutional layers,
pooling layers, fully connected layers and normalization
layers [19].
For the purpose of this research, the model was
composed of two convolutional layers. Both of them had
32 filters with kernel size 3 and ReLU activation function.
After each of them, a MaxPooling with pool size 2 was
added. Flatten layer preceeded the fully connected layer
which contained 128 units with ReLU activation function.
Output layer consisted of one unit with sigmoid activation
function. Weights of CNN network were randomly
initialized. Described arhitecture of CNN is shown below,
in the Fig. 2.
Fig. 2. Proposed architecture of CNN
Due to the unbalance class distributions, cost-sensitive
classification is executed and class weights are assigned
accordingly, 1 for non-spam and 1.5 for spam.
Hyperparameters were assigned based on tests of multiple
models with different values.
Because of the problem’s nature, Adam optimizer and
binary crossentropy loss were used for compiling the
model. The model was updated through 10 epochs, since
fifth caused overfitting.
IV. RESULTS AND DISCUSSION
In order to estimate how accurate the predictions will be
in practice, the model was assessed using 10-fold cross-
validation.
TABLE 1:CONFUSION MATRIX
Predicted
0 1
0 1440 8
Actual
1 19 205
In each step, AUC, F1 score and confusion matrix were
evaluated. The final values were obtain by taking the mean
from each step.
Of the total 224 spam messages, the model correctly
classified 205, hence having sensitivity (TPR) of 91.51%.
Out of 1448 non-spam messages, 1440 messages were
classified accurately having specificity (TNR) of 99.44%.
Values obtained for AUC and F1 score are 0.955 and
0.938, respectively, as shown in table 2.
TABLE 2: PERFORMANCE MEASURES
TPR 0.915
TNR 0.994
AUC 0.955
F1 0.983
ACC 0.984
Taking into consideration the results obtained by
proposed model and comparing it with the similar papers
[3], [7], [8], listed in section II, CNN had produced better
results evaluated on the same dataset. Like in paper [11]
and [12] it is shown that CNN can be useful for sms spam
detection and similar classification tasks.
V. CONCLUSION
In this paper a CNN was applied on Tiago’s dataset in
order to distinguish spam from non-spam messages.
Dataset is composed of mostly ham messages, hence it is
described as strongly imbalanced.
In order to obtain a good model, preprocessing of data
was conducted. Primarily, text from the dataset is reduced
to lower case, and after that tokenization was applied. By
performing stemming, sentiment of text was preserved,
afterwards stopwords were removed. The last step
consisted of converting a text into matrix of TF-IDF
features.
There are many Machine Learning algorithms that can
be used for spam detection. This paper proposed CNN for
spam classifcation, with AUC score of 0.955 and accuracy
of 98.4%, it is proven that this model can perform better
compared to many other ML techniques.
The proposed model can be adjusted to fit many
correlated types of problems. Also, it can be used as a
background service in many applications.
Future research should focus on applying more complex
preprocessing techiques (spelling correction, term
frequency, N-grams etc.), or deciding on another
architecture of CNN network. Also, combining models
through ensemble learning can lead to exceptional results.
ACKNOWLEDGEMENT
This research was supported by the PanonIT company
[19].
REFERENCES
[1] Mobile marketing watch, "SMS Marketing Wallops Email with
98% Open Rate and Only 1% Spam," [Online]. Available:
https://mobilemarketingwatch.com/sms-marketing-wallops-email-
with-98-open-rate-and-only-1-spam-43866/. [Accessed 2
September 2018].
[2] A. Small, "How to Use SMS to Win Love, Leads, Revenue," 5 May
2013. [Online]. Available: https://martech.zone/text-messaging/.
[Accessed 1 September 2018].
[3] Almeida, T.A., Gómez Hidalgo, J.M., Yamakami, A.,
"Contributions to the Study of SMS Spam Filtering: New
Collection and Results," in Proceedings of the 11th ACM
symposium on Document engineering, Mountain View, 2011.
[4] D. Delvia Arifin, Shaufiah and M. A. Bijaksana, "Enhancing spam
detection on mobile phone Short Message Service (SMS)
performance using FP-growth and Naive Bayes Classifier," in IEEE
 Asia Pacific Conference on Wireless and Mobile (APWiMob),
Bandung, 2016.
[5] P. Sethi, V. Bhandari and B. Kohli, "SMS spam detection and
comparison of various machine learning algorithms," in
International Conference on Computing and Communication
Technologies for Smart Nation (IC3TSN), Gurgaon, 2017.
[6] A. Karami and L. Zhou, "Improving Static SMS Spam Detection by
Using New Content-based Features," in Twentieth Americas
Conference on Information Systems, Savannah, 2014.
[7] F. Akbari and H. Sajedi, "SMS spam detection using selected text
features and Boosting Classifiers," in 2015 7th Conference on
Information and Knowledge Technology (IKT), Umia, 2015.
[8] H. Shirani-Mehr, "SMS Spam Detection using Machine Learning
Approach," [Online]. Available:
http://cs229.stanford.edu/proj2013/ShiraniMehr-
SMSSpamDetectionUsingMachineLearningApproach.pdf.
[Accessed 27 August 2018].
[9] N. K. Nagwani and A. Sharaff, "SMS spam filtering and thread
identification using bi-level text classification and clustering
techniques," Journal of Information Science, vol. 43, no. 1, pp. 75-
87, 2017.
[10] A. Barushka and P. Hajek, "Spam filtering using integrated
distribution-based balancing approach and regularized deep neural
networks," Applied Intelligence, vol. 48, no. 10, pp. 3538-3556,
2018.
[11] Y. Kim, "Convolutional Neural Networks for Sentence
Classification," in Conference on Empirical Methods in Natural
Language Processing, Doha, 2014.
[12] G. Chen, D. Ye, Z. Xing, J. Chen and E. Cambria, "Ensemble
application of convolutional and recurrent neural networks for
multi-label text categorization," in International Joint Conference
on Neural Networks (IJCNN), Anchorage, 2017.
[13] T. A. Almeida and J. M. G. Hidalgo, "SMS Spam Collection v. 1,"
2011. [Online]. Available:
http://www.dt.fee.unicamp.br/~tiago/smsspamcollection/.
[Accessed 25 August 2018].
[14] C. Zhang, X. Pan, H. Li, A. Gardiner, I. Sargent, J. Hare and P. M.
Atkinson, "A hybrid MLP-CNN classifier for very fine resolution
remotely sensed image classification," ISPRS Journal of
Photogrammetry and Remote Sensing, vol. 140, pp. 133-144, 2018.
[15] A. Scherzinger, S. Klemm, D. Berh and X. Jiang, "CNN-Based
Background Subtraction for Long-Term In-Vial FIM Imaging," in
International Conference on Computer Analysis of Images and
Patterns, Ystad, 2017.
[16] M. Arsenovic, S. Sladojevic, A. Anderla and D. Stefanovic,
"FaceTime — Deep learning based face recognition attendance
system," in IEEE 15th International Symposium on Intelligent
Systems and Informatics (SISY), Subotica, 2017.
[17] L. Eren, T. Ince and S. Kiranyaz, "A Generic Intelligent Bearing
Fault Diagnosis System Using Compact Adaptive 1D CNN
Classifier," Journal of Signal Processing Systems, pp. 1-11, 2018.
[18] J. Liu, Y. Cheng, X. Wang and Y. Kong, "Joint Sample Expansion
and 1D Convolutional Neural Networks for Tumor Classification,"
in ICIC 2017: Intelligent Computing Theories and Application ,
Liverpool, 2017.
[19] "PanonIT: Homepage," PanonIT [Online]. Available:
http://panonit.com/. [Accessed 12 September 2018].