BROUGHT TO YOU IN PARTNERSHIP WITH

CONTENTS

Working with Time

ö ö What is time series data?

ö ö Time series data: use cases across

industries

Series Data
ö ö Open source is key

ö ö Purpose-built databases are better

ö ö Time series data full stack in a nutshell

ö ö Hands-on learning

WRITTEN BY DANIELLA PONTES

PRODUCT MARKETER TEAM AT INFLUXDATA

Working with Time Series Data

According to DB-Engines, since 2016, time series has been the
fastest growing database category. This popularity is fueled by
the "sensorification" of the physical world (i.e., IoT) and the rapidly
increasing instrumentation requirements of the next generation
of software.

As we enter the era of workflow automation, machine learning and

artificial intelligence, it is time for time series data.

What is Time Series Data?

What classifies the data around you as time series depends on what
A common question that comes to mind when the subject is
it represents: a collection of measurements of the same thing over
times series data is: "What actually is it?" The more you think
time, where you will find the time axis as you graph it. Bringing it
about it, the more you see it everywhere. Time is a constituent and
inexorable part of everything that is observable. If it does not have
a timestamp, it has no place in our universe. If time is always there
as a dimension, time series means to treat every piece of recorded
data for what it is: unique. Therefore, it should not be replaced, but
accumulated, appended as the next chapter in the "secret" time
series life of the data.

Think of weather records showing that the Earth is getting warmer

and ice caps are melting beyond historic limits. Or consider
economic indicators showing how well a current administration
is doing its job, such as changes in the GDP, income and wages,
unemployment rate, inflation, etc. Other common examples include
records of the evolving status of a patient under treatment, or tweets
per hour for a specific hashtag, e.g. #royalbaby. The reality is that
time series data is everywhere.

1
Time series is everywhere
and is an integral part of a
modern and sensorized
environment.
Open source to leverage collaboration and keep
freedom of choice and purpose-built for the required
leap in speed and efficiency

For more information, visit:

influxdata.com

to the tech world, time series are measurements that are tracked,
monitored, downsampled, and aggregated over time. This could be
server metrics, application performance monitoring, network data,
sensor data, events, clicks, trades in a market, and many other types
of analytics data.
Time series:
Time Series Example: Stock Ticker Prices Over Time
Also time series:
So, what do time series give you besides a gigantic collection of
data consuming your resources? If you can handle time series data
properly, you can have application environments with close to
zero-downtime, more and faster asset and resource consumption
predictions using machine learning, workflow automation integrated
with business KPIs, and ultimately, better customer experience
leading to happier users.
Time Series Data: Use Cases Across Industries
Time series data personas are those responsible for maintaining a
sustainable functional and evolving environment without operational
downtime, inefficiencies and performance issues, while providing
business units with means to monitor their own performance in
order to stay on track of their goals. By doing so, time series personas
provide the pillars for growth and success.
Time series projects usually start at DevOps and NetOps. They are
the ones that must keep up with the speed of growth and complexity
of the application environments, while coping with increasing
intolerance to downtime or low performance. Monitoring production
and pre-production as part of the application development lifecycle,
3 BROUGHT TO YOU IN PARTNERSHIP WITH
LOW-CODE APPLICATION DEVELOPMENT
as well as monitoring business performance indicators, is the way to
detect trends early, anticipate resource requirements, and provide
means to resolve issues in a timely manner.
Here are some case studies that illustrate specific scenarios where
operational and business challenges can be addressed with the
adoption of a time series platform solution.
SaaS companies looking for ensuring customer success find in time
series a good strategy to achieve the ultimate goal of 100% uptime.
They invest in DevOps monitoring solutions that provide visibility into
performance, detecting early trends that could negatively impact the
service. Monitoring is offered to the whole organization as a tool to
assist teams (not just operations) to collect metrics that are relevant
to them. Planning for scale, which includes not only being able to
collect high volumes of data, but also automate the data life cycle
while avoiding being locked in disparate systems is fundamental for a
comprehensive approach.
E-commerce companies also rely on time series to keep user
experience at its best. Ops monitoring teams work around the
clock, 24x7, assuring the health of IT infrastructure and storefront
applications. They integrate application monitoring as part of the
application development process, so they come to production under
the seal of "real user monitoring" (RUM) approval. For e-commerce
companies with multiple data centers, handling volume and having
HA becomes essential.
Companies that provide mobile services via smartphone applications
are also benefiting from time series monitoring. Many of those
companies have convenience and agility as core values. And
although the service sometimes relies on a single smartphone
app, the infrastructure and backend part of the application could
be massive, with many components and microservices. As being
fast, always ready, and at reach are all part of mobile services, they
must detect problems before their users experience them. Once a
user tries another service that works, this other service becomes the
preferred choice. So, it is paramount to monitor metrics from systems,
applications, and KPI at high sampling rates to catch anything that
goes wrong practically as it happens. Certainly, the volume of data
collected at rates of millions of reads per second will be high, but as
data gets cold, it could be aggregated for trending analysis. Therefore,
beyond high volume and HA, the data life cycle is also fundamental
for companies using time series monitoring to keep ahead of
competition in the mobile applications market.
Cloud service providers also turn to multi-tenant metric-driven
visibility to provide their customers with ways to monitor their
applications environment, regardless if it runs on Kubernetes clusters,
containers, virtual machines, bare metal, or a combination of these
options — which is very probable. Regardless of the diversity of

environments, everything needs to be ready and available from one
data source. Collecting all metrics in one time series platform that can
support multi-tenancy and HA is a necessary foundation for scalable
cloud hosting services.
In the time of time series, professionals are getting insights on how
change happens over time and ways to predict their course in the
future while monitoring the present. But in order to leverage the
benefits of time series to its full extent, it is necessary to be able to
collect, store, and analyze data in real-time and at scale.
Open Source is Key
Open source means, among other things, free to use, but, most
importantly, it means that ideas and information are shared openly
and the community is encouraged to collaborate transparently. Every
breakthrough is celebrated by all and takes everyone one step ahead,
driving innovation at a much faster pace. Because of the many eyes
and brains continuously testing and applying it in different use cases,
open source is more reliable and secure.
In complex environments, open source provides the necessary
freedom to adapt. With more shared code, APIs, plugins, and custom
scripts, the community helps you to implement solutions that fit
with your legacy applications and future choices, giving you more
bandwidth to concentrate on delivering the business logic for your
specific use cases.
The power of the open source community to drive innovation is
unsurpassed by any proprietary software solutions because it is not
about license, it is about collaboration that makes the whole greater
than the sum of its parts, and transparency, where nothing is hidden,
giving you a chance to make educated decisions. Open source also
keeps your options open by avoiding vendor lock-in.
Purpose-Built Databases are Better
Time series data platforms pose intrinsic challenges in scalability, high
availability, and usability when professionals try to use legacy database
and query engine models. Addressing these challenges properly led
to the development of time series databases. Those embarking on
time series data projects face a major decision: try to adapt an existing
relational database to manage times series data, or adopt time series
databases with purpose-built storage and query engines?
Purpose-built time series database engines reach benchmarks in
the order of hundreds of millions of time series, millions of writes
per second, and over thousands of queries per second. Time series
databases have a few properties that make them very different
than other data stores: automated data lifecycle management,
summarization, continuous queries, and large range scans of many
records. There is still, of course, the obvious differentiation of having
a time dimension present in everything done with the data: collecting,
storing, replicating, querying, transforming, and evicting.
4 BROUGHT TO YOU IN PARTNERSHIP WITH
LOW-CODE APPLICATION DEVELOPMENT
Although time must be a central point in the overall platform
architecture design of the database, just being able to query on
time doesn't cover all the requirements of an effective and efficient
solution. To achieve the envisioned scalability and usability paradigm
of a very large and granular data set, it is necessary to devise a
combined strategy in data model and storage engine design. Add
to that a query language that eases selection of series, continuous
queries, and transformation of queried data, and you have a complete
time series platform, such as InfluxData's TICK Stack.
Time Series Data Full Stack in a Nutshell
InfluxData is an open source, purpose-built full-stack time series
platform for managing time series data. We'll begin by explaining
each component that comprises the platform before moving into
examples of how it can be used with time series data.
The functional architecture consists of four components — Telegraf,
InfluxDB, Chronograf, and Kapacitor (TICK Stack). These components
are built as an end-to-end solution for time series data use cases.
Telegraf is built for data collection; InfluxDB is the database and
storage tier; Chronograf is for visualization; and Kapacitor is the rules
engine for processing, monitoring, and alerting. InfluxDB is a key
component in the time series data platform architecture because it
is designed to take the peculiar characteristics of time series data
as insights to better handle it, in contrast to legacy storage engines
and databases designs that only try to work around them, leading to
limitations and performance issues.
Getting Started with InfluxDB
INFLUXDB DATA MODEL DESIGN
Efficiency and effectiveness have to start in the data structure,
ensuring that time-stamped values are collected with the necessary
precision and metadata to provide flexibility and speed in graphing,
querying, and alerting. The InfluxDB data model has a flexible schema
that accommodates the needs of diverse time series use cases. The
data format takes the following form:
<measurement name>,<tag set> <field set> <timestamp>
The measurement name is a string, the tag set is a collection of key/

value pairs where all values are strings, and the field set is a collection
of key/value pairs where the values can be int64, float64, bool, or
string. There are no hard limits on the number of tags and fields.
Being able to have multiple fields and multiple tags under the same
measurement optimizes the transmission of the data, avoiding
multiple retransmissions which can render network protocols bloated
when transmitting data with shared tag sets. This design choice is
also particularly important for IoT use cases where the agent stored
on the monitored remote devices sending the metrics has to be
energy-efficient for longer lifespan.
Furthermore, support for multiple types of data encoding beyond
float64 values means that metadata can be collected along with the
time series, and not limited to monitor only numeric values. Precision
is another parameter that must be taken into account when defining
the data model. Timestamps in InfluxDB can be second, millisecond,
microsecond, or nanosecond precision.
The measurement name and tag sets are kept in an inverted index
which make lookups for specific series very fast.
See below CPU metrics mapped in InfluxDB line-protocol:
cpu,host=serverA,region=uswest
idle=23,user=42,system=12 1464623548s
INFLUXDB TIME SERIES DATABASE SPECIALTIES
A good strategy when adopting a platform for time series should take
into account the management of all types of time series data — not
only metrics (numeric values regularly collected or pulled), but also
events (pushed at irregular time intervals) like faults, peaks, and
human-triggered ones, such as clicks.
Since we are talking about a constant influx of granular data from
a number of data sources, a performant database solution for time
series has to handle high write rates with fast queries at scale, and
that is where most other types of databases stumble when used for
time series. In order to reach the bar raised raised by time series,
InfluxDB utilizes an architecture design with high compression, super-
fast storage and query engines, and a purpose-built stack.
InfluxDB uses an append-only file for new data arrival in order to
make new points quickly ingested and durable; a columnar on-disk
storage for efficient queries and aggregations over time; a time-bound
file structure that facilitates management of data in shards' sizes; a
reverse index mapping measurements to tags, fields, and series for
quick access to targeted data; and data compaction and compression
for read optimization and volume control.
InfluxDB was also designed to provide easy and automated data
lifecycle management: retention policies, sharding, replication, and
rollup capabilities are built-in. In time series it's common to keep high-
precision data around for a short period of time, and then aggregate
and downsample into longer-term trend data. This kind of data lifecycle
5 BROUGHT TO YOU IN PARTNERSHIP WITH
LOW-CODE APPLICATION DEVELOPMENT
management is difficult for application developers to implement on top
of regular databases, but it is fundamental for time series.
Hands-on Learning
The easiest way to get a feel for what time series data can do for you is
to try it. Since the TICK Stack is open source, you can freely download
and install Telegraf, InfluxDB, Chronograf, and Kapacitor. Let's start
with InfluxDB, feed some data, and build some queries.
With InfluxDB installed, it is time to interact with the database. Let's
use the command line interface (influx) to write data manually, query
that data interactively, and view query output in different formats.
To access the CLI, first launch the influxd database process and then
launch influx in your terminal. Once you've connected to an InfluxDB
node, you'll see the following output:
$influx
Connected to
[[http://localhost:8086]{.underline}](http://
localhost:8086) version
1.7x
InfluxDB shell version 1.7.x
Note that the version of InfluxDB and CLI must be identical.
Creating a Database in InfluxDB
A fresh install of InfluxDB has no databases (apart from the system
_internal ). You can create a database with the CREATE DATABASE
<db-name> InfluxQL statement, where <db-name> is the name of the
database you wish to create. Let's create a database called mydb.
> CREATE DATABASE mydb
To see if the database was created, use the following statement:
> SHOW DATABASES
name: databases
Name
----
_internal
mydb
Now, it is time to populate the mydb database.
Writing and Querying the Database
InfluxDB is populated with points from popular clients via HTTP /
write Endpoint POST or via the CLI. Datapoints can be inserted
individually or in batches.
Points consist of a timestamp, a measurement ("cpu_load", for
example), at least one key-value field (the measured value itself, e.g.
"value=0.64", or "temperature=21.2"), and zero to many key-value
tags containing any metadata about the value (e.g. "host=server01",
"region=EMEA", "dc=Frankfurt").
 LOW-CODE APPLICATION DEVELOPMENT

Conceptually, you can think of a measurement as an SQL table, where
the primary index is always time. Tags and fields are effectively
columns in the table. Tags are indexed, and fields are not.
InfluxDB is defined as the desired output.
telegraf -sample-config -input-filter cpu:mem
-output-filter influxdb > telegraf.conf

Points are written to InfluxDB using line protocol:

Start the Telegraf service and direct it to the relevant configuration file:

<measurement>[,<tag-key>=<tag-value>...] <field- MACOS HOMEBREW

key>=<field-value>[,<field2-key>=<field2-
telegraf -config telegraf.conf
value>...] [unix-nano-timestamp]
LINUX (SYSVINIT AND UPSTART INSTALL ATIONS)

To insert a single time series datapoint with the measurement name Sudo services telegraf start
of cpu and tags host and region, with the measured value of 0.64
LINUX (SYSTEMD INSTALL ATIONS)
into InfluxDB using the CLI, enter INSERT followed by a point in line
protocol format: Systemctl start telegraf

> INSERT cpu,host=serverA,region=us_west You will see the following output (Note: this output below runs
value=0.64 on a Mac):

Now query for the data written: NetOps-MacBook-Air:~ Admin$ telegraf --config
telegraf.conf
> SELECT “host”, “region”, “value” FROM “cpu”
2019-01-12T18:49:48Z I! Starting Telegraf 1.8.3
Name: cpu
2019-01-12T18:49:48Z I! Loaded inputs: inputs.cpu
--------
inputs.mem
Time host region 2019-01-12T18:49:48Z I! Loaded aggregators:
value 2019-01-12T18:49:48Z I! Loaded processors:
2015-10-21T19:28:07.5806643472 serverA us_west
2019-01-12T18:49:48Z I! Loaded outputs: influxdb
0.64
2019-01-12T18:49:48Z I! Tags enabled: host=NetOps-

Great! You successfully installed InfluxDB and can write and query MacBook-Air.local
the data. 2019-01-12T18:49:48Z I! Agent Config:
Interval:10s, Quiet:false, Hostname:”NetOps-
Next step is to collect data via Telegraf and send it to InfluxDB.
MacBook-Air.local”, Flush Internal:10s

Data Collection With Telegraf

Telegraf is a plugin-driven server agent for collecting and reporting
metrics. Telegraf has plugins to pull metrics from third-party APIs, or
to listen for metrics via StatsD and Kafka consumer services. It also
has output plugins to send metrics to a variety of datastores.
Install Telegraf from InfluxData download page: portal.influxdata.
com/downloads
Once Telegraf is up and running, it will start collecting data and
writing it to the desired output.
Returning to our sample configuration, we show what the cpu and
mem data look like in InfluxDB below. Note that we used the default
input and output configuration settings to get this data.
List all measurements in the Telegraf database:

Before starting the Telegraf server, you need to edit and/or create > SHOW MEASUREMENTS
an initial configuration that specifies your desired inputs (where the Name: measurements
metrics come from) and outputs (where the metrics go). Telegraf ------------------
can collect data from the system it is running on. That is just what is Name
needed to start getting familiar with Telegraf. Cpu
mem
The example below shows how to create a configuration file called
telegraf.conf with two inputs: List all field keys by measurement:

1. One input reads metrics about the system's cpu usage (cpu). > SHOW FIELD KEYS
name: cpu
2. Another input reads metrics about the system's memory ---------
usage (mem). fieldKey fieldType

6 BROUGHT TO YOU IN PARTNERSHIP WITH

 LOW-CODE APPLICATION DEVELOPMENT

usage_guest float • MacOS: tar zxvf chronograf-1.6.2_darwin_amd64.tar.gz

usage_guest_nice float
• Ubuntu & Debian: sudo dpkg -i chronograf_1.6.2_amd64.deb
usage_idle float
usage_iowait float • RedHat and CentOS: sudo yum localinstall > chronograf-
usage_irq float 1.6.2.x86_64.rpm
usage_nice float
usage_softirq float 3. Start Chronograf:
usage_steal float
• MacOS: tar zxvf chronograf-1.6.2_darwin_amd64.tar.gz
usage_system float
usage_user float • Ubuntu & Debian: sudo dpkg -i chronograf_1.6.2_amd64.deb

name: mem
4. Connect Chronograf to your InfluxDB instance or InfluxDB
--------- Enterprise cluster:
fieldKey fieldType
active integer • Point your web browser to > localhost:8888.
available integer • Fill out the form with the following details:
available_percent float
−− Connection String: Enter the hostname or IP of the
buffered integer machine that InfluxDB is running on, and be sure to include
cached integer InfluxDB's default port 8086.
free integer
−− Connection Name: Enter a name for your connection string.
inactive integer
total integer −− Username and Password: These fields can remain blank
used integer unless you've enabled authorization in InfluxDB.
used_percent float −− Telegraf Database Name: Optionally, enter a name for your
Telegraf database. The default name is Telegraf.
Select a sample of the data in the field usage_idle in the measurement
cpu_usage_idle: • Click “Add Source.”

> SELECT usage_idle FROM
total' LIMIT 5
name: cpu
---------
time
2016-01-16T00:03:00Z
2016-01-16T00:03:10Z
2016-01-16T00:03:20Z
2016-01-16T00:03:30Z
2016-01-16T00:03:40Z
cpu WHERE cpu = 'cpu- Pre-Canned Dashboards
Pre-created dashboards are delivered with Chronograf, and you just
have to enable the respective desired Telegraf plugins. In our example,
usage_idle we already enabled the cpu and mem plugins at the config file
97.56189047261816 creation. Let's take a look (Note: this example runs on a Mac):
97.76305923519121
97.32533433320835 Select the Dashboard icon in the navigation bar on the left, and then
95.68857785553611 select "System:”
98.63715928982245

That's it! You now have the foundation for using Telegraf to collect
and write metrics to your database.

Voilà!
Data Visualization and Graphing With Chronograf
Chronograf is the administrative interface and visualization engine
for the TICK Stack. It is simple to use and includes templates and
libraries to allow you to build dashboards of your data and to create
alerting and automation rules.

The Chronograf builds are available on InfluxData's Downloads page.

1. Choose the download link for your operating system.

• Note: If your download includes a TAR package, we
recommend specifying a location for the underlying datastore,
chronograf-v1.db, outside of the directory from which you start
Chronograf. This allows you to preserve and reference your
Taking a closer look at cpu measurement, you can see that it shows
existing datastore, including configurations and dashboards,
three measured fields (cpu.idle, cpu.user and cpu.system). You can
when you download future versions.
filter each and move the measurement line to show exact values
2. Install Chronograf: and timestamp.

7 BROUGHT TO YOU IN PARTNERSHIP WITH

 LOW-CODE APPLICATION DEVELOPMENT

The selected "System" App option for the example is just one of many
pre-created dashboards available in Chronograf. See the list below:

• apache • influxdb
• consul • kubernetes
• docker • memcached
• elasticsearch • mesos
• haproxy • mysql
• iis • nginx
• nsq • redis
• phpfpm • riak
• ping • system
Cpu.user metric filtered: • postgresql • varnish
• rabbitmq • win_system

Now that you are set to start exploring the world of time series data,
what is next? Learning from others experience is always a good idea.
Get more insights from case studies in various industry segments:
telecom and service providers, e-commerce, financial markets, IoT,
research, manufacturing, telemetry, and of course, the horizontal
case of DevOps and NetOps in any organization, and see how time
series monitoring generated positive results in the organizations,
from better resource management via automation and prediction to
five star customer experience.

Written by Daniella Pontes, Product Marketer at InfluxData

Daniella Pontes is part of the product marketing team in InfluxData. She started her career in telecommunications,
wireless technology, and global Internet service provisioning. As security became a major concern for enterprises, she
worked on enterprise policy management, SaaS, and data encryption solutions. Prior to joining InfluxData, she spent
some years living in Japan, Germany, and Brazil. Having worked in various market segments, from embedded smart
antenna technology to Internet security and e-commerce doing product management, partnerships, marketing, and
business development, she has a broad experience working cross-functionally and with customers and partners.

Devada, Inc.
600 Park Offices Drive
Suite 150
Research Triangle Park, NC
DZone communities deliver over 6 million pages each
month to more than 3.3 million software developers, 888.678.0399 919.678.0300
architects, and decision makers. DZone offers something for
Copyright © 2019 DZone, Inc. All rights reserved. No part of this publication
everyone, including news, tutorials, cheat sheets, research
may be reproduced, stored in a retrieval system, or transmitted, in any form
guides, feature articles, source code, and more. "DZone is a or by means electronic, mechanical, photocopying, or otherwise, without
developer’s dream," says PC Magazine. prior written permission of the publisher.

8 BROUGHT TO YOU IN PARTNERSHIP WITH