Anomaly Detection - Another Challenge For Artificial Intelligence

10/30/2019 Anomaly Detection — Another Challenge for Artificial Intelligence
Anomaly Detection — Another

Challenge for Arti cial Intelligence
Sciforce Follow
Jun 27 · 7 min read
Image Credit: unsplash.com
It is true that the Industrial Internet of Things will change the world someday. So far, it is
the abundance of data that makes the world spin faster. Piled in sometimes
unmanageable datasets, big data turned from the Holy Grail into a problem pushing
businesses and organizations to make faster decisions in real-time. One way to process
data faster and more efficiently is to detect abnormal events, changes or shifts in
Get one more story in your member preview when you sign up. It’s free.
datasets. Thus, anomaly detection, a technology that relies on Artificial Intelligence to
identify abnormal behavior within theSign
pool of collected
up with Google data, has become one of the
main objectives of the Industrial IoT.
Sign up with Facebook
Anomaly detection refers to identification of items or events that do not conform to an
Already
expected pattern or to other items in have an account?
a dataset that areSign in undetectable by a human
usually
https://medium.com/sciforce/anomaly-detection-another-challenge-for-artificial-intelligence-c69d414b14db 1/9
expert. Such anomalies can usually be translated into problems such as structural
defects, errors or frauds.
Examples of potential anomalies:

A leaking connection pipe that leads to the shutting down of the entire production
line;
Multiple failed login attempts indicating the possibility of fishy cyber activity;
Fraud detection in financial transactions.
Why is it important?
Modern businesses are beginning to understand the importance of interconnected
operations to get the full picture of their business. Besides, they need to respond to fast-
moving changes in data promptly, especially in case of cybersecurity threats. Anomaly
detection can be a key for solving such intrusions, as while detecting anomalies,
perturbations of normal behavior indicate a presence of intended or unintended induced
attacks, defects, faults, and such.
Unfortunately, there is no effective way to handle and analyze constantly growing

datasets manually. With the dynamic systems having numerous components in
perpetual motion where the “normal” behavior is constantly redefined, a new proactive
approach to identify anomalous behavior is needed.
Statistical Process Control

Statistical Process Control, or SPC, is a gold-standard methodology for measuring and
controlling quality in the course of manufacturing. Quality data in the form of product or
process measurements are obtained in real-time during the manufacturing process and
plotted on a graph with predetermined control limits that reflect the capability of the
process. Data that falls within the control limits indicates that everything is operating as
expected. Any variation within the control
Sign uplimits is likely due to a common cause — the
with Google
natural variation that is expected as part of the process. If data falls outside of the
control limits, this indicates that an assignable
Sign up withcause might be the source of the product
Facebook
variation, and something within the process needs to be addressed and changed to fix
Already have an account? Sign in
the issue before defects occur. In this way, SPC is an effective method to drive continuous
improvement. By monitoring and controlling a process, we can assure that it operates at

its fullest potential and detect anomalies at early stages.
Introduced in 1924, the method is likely to stay in the heart of industrial quality
assurance forever. However, its integration with Artificial Intelligence techniques will be
able to make it more accurate and precise and give more insights into the manufacturing
process and the nature of anomalies.
Tasks for Artificial Intelligence

When human resources are not enough to handle the elastic environment of cloud
infrastructure, microservices and containers, Artificial Intelligence comes in, offering
help in many aspects:
Tasks for Arti cial Intelligence
Automation: AI-driven anomaly detection algorithms can automatically analyze
datasets, dynamically fine-tune the parameters
Sign up with of normal behavior and identify breaches
Google
in the patterns.
Real-time analysis: AI solutions can interpret data activity in real time. The moment a
pattern isn’t recognized by theAlready have
system, an account?
it sends Sign in
a signal.
Scrupulousness: Anomaly detection platforms provide end-to-end gap-free monitoring

to go through minutiae of data and identify smallest anomalies that would go unnoticed
by humans
Accuracy: AI enhances the accuracy of anomaly detection avoiding nuisance alerts and
false positives/negatives triggered by static thresholds.
Self-learning: AI-driven algorithms constitute the core of self-learning systems that are
able to learn from data patterns and deliver predictions or answers as required.
. . .
Learning Process of AI Systems

One of the best things about AI systems and ML-based solutions is that they can learn on
the go and deliver better and more precise results with every iteration. The pipeline of
the learning process is pretty much the same for every system and comprises the
following automatic and human-assisted stages:
Datasets are fed to an AI system
Data models are developed based on the datasets
A potential anomaly is raised each time a transaction deviates from the model
A domain expert approves the deviation as an anomaly
The system learns from the action and builds upon the data model for future
predictions
The system continues to accumulate patterns based on the preset conditions

Sign up with Google
Learning Process of AI Systems
As elsewhere in AI-powered solutions, the algorithms to detect anomalies are built on

supervised or unsupervised machine learning techniques.
Supervised Machine Learning for Anomaly Detection

The supervised method requires a labeled training set with normal and anomalous
samples for constructing a predictive model. The most common supervised methods
include supervised neural networks, support vector machine, k-nearest neighbors,
Bayesian networks and decision trees.
Probably, the most popular nonparametric technique is K-nearest neighbor (k-NN) that
calculates the approximate distances between different points on the input vectors and
assigns the unlabeled point to the class of its K-nearest neighbors. Another effective
model is the Bayesian network that encodes probabilistic relationships among variables
of interest.
Supervised models are believed to provide a better detection rate than unsupervised
methods due to their capability of encoding interdependencies between variables, along
with their ability to incorporate both prior knowledge and data and to return a
confidence score with the model output.
Unsupervised Machine Learning for Anomaly Detection
Sign up with Google
Unsupervised techniques do not require manually labeled training data. They presume
that most of the network connectionsSign
are up
normal traffic and only a small amount of
with Facebook
percentage is abnormal and anticipate that malicious traffic is statistically different from
normal traffic. Based on theseAlready have an account?
two assumptions, Sign
groups ofinfrequent similar instances are
assumed to be normal and the data groups that are infrequent are categorized as
malicious.
The most popular unsupervised algorithms include K-means, Autoencoders, GMMs,

PCAs, and hypothesis tests-based analysis.
The most popular unsupervised algorithms
. . .
SciForce’s Chase for Anomalies

Like probably any company specialized in Artificial Intelligence and dealing with
solutions for IoT, we found ourselves hunting for anomalies for our client from the
Get oneindustry.
manufacturing more story in your
Using member
generative preview
models when you sign
for likelihood up. It’s free.
estimation, we detected
the algorithm defects, speeding up regular processing algorithms, increasing the system
Sign up with Google
stability, and creating a customized processing routine which takes care of anomalies.

For anomaly detection to be used commercially, it needs to encompass two parts:
anomaly detection itself and prediction of future anomalies.
Anomaly detection part

For the anomaly detection part, we relied on autoencoders — models that map input
data into a hidden representation and then attempt to restore the original input from
this internal representation. For regular pieces of data, such reconstruction will be
accurate, while in case of anomalies, the decoding result will differ noticeably from the
input.
Results of our anomaly detection model. Potential anomalies are marked in red.
In addition to the autoencoder model, we had a quantitative assessment of the similarity

between the reconstruction and the original input. For this, we first computed sliding
window averages for sensor
Get one more story inputs,
in your i.e. the average
member value
preview whenforyou
each sensor
sign over
up. It’s a 1-min.
free.
interval each 30 sec. and fed the data to the autoencoder model. Afterwards, we
calculated distances between the input data
Sign andGoogle
up with the reconstruction on a set of data and
computed quantiles for distances distribution. Such quantiles allowed us to translate an
Sign up with
abstract distance number into a meaningful Facebook
measure and mark samples that exceeded a
present threshold (97%) as an anomaly.
Sensor readings prediction

With enough training data, quantiles can serve as an input for prediction models based
on recurrent neural networks (RNNs). The goal of our prediction model was to estimate
sensor readings in future.
Though we used each sensor to predict other sensors’ behavior, we had trained a
separate model for each sensor. Since the trends in data samples were clear enough, we
used linear autoregressive models that used previous readings to predict future values.
Similarly to the anomaly detection part, we computed average each sensor values over 1-
min. interval each 30 sec. Then we built a 30-minute context (or the number of previous
timesteps) by stacking 30 consecutive windows. The resulting data was fed into
prediction models for each sensor and the predictions were saved as estimates of the
sensor readings for the following 1-minute window. To expand over time, we gradually
substituted the older windows with predicted values.
Sign up with Google

Results of prediction models outputs with historical data marked in blue and predictions in green.
It turned out that the context is crucial for predicting the next time step. With the scarce
data available and relatively small context windows we could make accurate predictions
for up to 10 minutes ahead.
. . .
Conclusion
Anomaly detection alone or coupled with the prediction functionality can be an effective
means to catch the fraud and discover strange activity in large and complex datasets. It
may be crucial for banking security, medicine, marketing, natural sciences, and
manufacturing industries which are dependent on the smooth and secure operations.
With Artificial Intelligence, businesses can increase effectiveness and safety of their
digital operations — preferably, with our help.
Machine Learning Arti cial Intelligence Data Science Technology Predictive Analytics
About Help Legal
Sign up with Google

Anomaly Detection - Another Challenge For Artificial Intelligence

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Anomaly Detection - Another Challenge For Artificial Intelligence

Încărcat de

Drepturi de autor:

Formate disponibile

10/30/2019 Anomaly Detection — Another Challenge for Artificial Intelligence

Anomaly Detection — Another

Image Credit: unsplash.com

Examples of potential anomalies:

Fraud detection in financial transactions.

Unfortunately, there is no effective way to handle and analyze constantly growing

Statistical Process Control

improvement. By monitoring and controlling a process, we can assure that it operates at

Tasks for Artificial Intelligence

Tasks for Arti cial Intelligence

Scrupulousness: Anomaly detection platforms provide end-to-end gap-free monitoring

Learning Process of AI Systems

Datasets are fed to an AI system

Data models are developed based on the datasets

A domain expert approves the deviation as an anomaly

The system continues to accumulate patterns based on the preset conditions

Sign up with Google

Sign up with Facebook

Already have an account? Sign in

Learning Process of AI Systems

As elsewhere in AI-powered solutions, the algorithms to detect anomalies are built on

Supervised Machine Learning for Anomaly Detection

The most popular unsupervised algorithms include K-means, Autoencoders, GMMs,

The most popular unsupervised algorithms

SciForce’s Chase for Anomalies

Sign up with Facebook

Anomaly detection part

In addition to the autoencoder model, we had a quantitative assessment of the similarity

Sensor readings prediction

Sign up with Google

Sign up with Facebook

About Help Legal

Sign up with Google

Sign up with Facebook

Already have an account? Sign in

S-ar putea să vă placă și