Documente Academic
Documente Profesional
Documente Cultură
Driven iPaaS
The Architect’s Guide to Building
a Modern Data Integration Platform
“
By 2020, event-sourced, real-time situational
awareness will be a required characteristic
for 70% of digital business solutions, and
80% of new business ecosystems will require
support for event processing.”
“
-SOURCE: GARTNER “TOP 10 STRATEGIC TECHNOLOGY TRENDS FOR 2018: EVENT-DRIVEN MODEL.” 8
MARCH 2018, YEFIM NATIS, DAVID CEARLEY, MIKE WALKER, BRIAN BURKE.
All rights reserved. No part of this work may be reproduced, or stored in a retrieval system, or transmitted in any form
or by any means, electronic, mechanical, photocopying, recording, or otherwise without permission from Solace.
Solace
535 Legget Drive, 3rd Floor
Ottawa, Ontario K2K 3B8
Canada
Phone: +1 613-271-1010
Web: solace.com
TABLE OF CONTENTS
Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Governance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
About Solace. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Companies around the world and in every Simply put, current architectures and
sector are striving for digital transformation— infrastructures are not able to keep up.
the incorporation of AI, hybrid cloud, mobile One key to success is evolving your
computing and IoT into their business architecture and business processes from
practices. Such transformation promises request/reply to event-driven. As analysts at
to improve situational awareness and Gartner put it, “By 2020, event-sourced, real-
operational agility, and to help companies time situational awareness will be a required
build close, customized relationships with characteristic for 70% of digital business
their customers. solutions....(1)”
But these benefits don’t come easy. The other key: consuming integration as a
For most enterprises, pursuing those critical service (also known as Integration Platform as
benefits drives an astronomical increase a Service, or iPaaS) so it can adapt and scale
both in the number of systems and devices to support whatever comes your way.
they need to integrate and in the amount of This paper describes how you can upgrade
information flowing between those systems from your on-prem-centric enterprise service
and devices. bus (ESB) to an event-driven iPaaS approach
that will facilitate and in many ways future-
proof your digital transformation.
segments.
Source: Gartner Newsroom (2)
INTEGRATION AND THE INTRODUCTION
OF AN ESB
Integration can be described as sharing state records and App B uses “CA.”
data between two otherwise separate This introduces the need to transform
systems, a process that consists of three that data at some point. You could
steps: transform California to CA as the data
moves directly from App A to App B, but
1. Data retrieval: read the customer
in systems with many producers and
record from App A;
consumers of information it makes sense
2. Data movement: move the customer to rationalize messages into a canonical
record from App A to App B, or normalized data format.
transforming payloads as necessary;
• Direct transformation: App A -> App B
and,
• Canonical transformation: App A ->
3. Data insertion: write the customer
Canonical -> App B, App C and App D
record into App B.
Canonical data formats allow for reuse
In the real world this becomes more
of data transformation mappings. They
complicated as App A usually represents
also add complications that are beyond
a record (a customer, for example)
our scope here, but suffice it to say
differently than App B.
complexity can grow to incorporate
Or let’s say App A uses “California” for not only complexities in the data
representation, such as nested data Infrastructure as a service (IaaS)
structures, but also in the relationships platforms provide hardware resources
between records. such as compute, disk, and network on
demand. And platform as a service (PaaS)
Think of a Return Material Authorization
added to that software platforms such
(when you want to return a product to a
as OS’s, databases, middleware, Java
store). There will be a customer record,
runtimes, and more.
maybe a warranty record, and a time of
purchase from the sales record—to name Then, with software as a service (SaaS),
just a few. complete applications were offered in
the cloud—first and still best exemplified
The complex relationships between those
by Salesforce.com. This evolution meant
records of information, both at rest and
that integration now had to extend
in motion, led to the advent of Enterprise
beyond the traditional LAN to include
Application Integration (EAI) software and
these cloud-based services and take
then enterprise service buses.
advantage of the benefits of cloud-native
ESBs retrieve data from source systems, deployment.
transform it as required, route it,
IPaaS addresses this need and brings
choreograph it, and write subsequent
traditional ESB capabilities to the cloud.
data into destination systems. This was
usually done over fast, reliable LAN
links within the walls of the enterprise
(corporate datacenters).
For companies with fairly simple, low- Things, events are what drive exchanges
volume, non-real-time integration of information. This means architects and
requirements, a simple integration developers need to shift their mindset
model might work fine: hub and spoke from being data-centric to event-centric.
architecture, request/reply interactions
Event-driven architectures enable greater
using RESTful HTTP.
agility, scalability and communication
But many companies—especially those across diverse cloud and datacenter
linking large numbers of applications and environments.
connected devices, serving multiple lines of
Here are some of the architectural and
business, or that have complex IT systems
operational challenges you will need to
that are distributed across datacenters
overcome to evolve toward an event-
and geographies—will need a more
driven architecture and ultimately achieve
sophisticated system.
success in your digital transformation
User expectations and information initiative if your system is complex, large or
technology are evolving from database- performance-sensitive.
centric request-based interactions to
event-centric, real-time interactions. From
social media platforms to the Internet of
BAKING AGILITY INTO YOUR across massive amounts of concurrent
INFRASTRUCTURE
connections.
The evolution of software development
Second, IoT devices exhibit different
practices and runtime environments means
characteristics and behavior than
applications can be created at a breakneck
applications, such as not being on or
pace. No app is an island, though, so if
connected all the time, and requiring very
you want to seize the advantages of rapid
low power/bandwidth communications
development and deployment you can’t
(many of which dictate the use of
forget rapid integration.
specialized APIs and communications
To do this, it’s critical to develop an event- protocols such as MQTT, which you need to
oriented data sharing infrastructure bridge with legacy APIs and protocols).
that makes it easy for apps to exchange
Command and control, especially at scale,
information with other systems so they can
is particularly difficult because you need to
play their part in business processes and
make sure that when a user clicks ”Unlock
transactions. Functionality is implemented
the door” that their car and only their car
as microservices, which consume and
reacts.
produce events.
First, IoT devices come in larger numbers This means they need to make new cloud-
than applications—tens of millions in native services, such as AI and machine
many cases. This demands sophisticated learning, accessible to legacy on-premises
aggregation, routing, filtration and fan-out systems across their enterprise.
IPaaS offerings from companies such as Companies of all sizes are using iPaaS
Dell Boomi, Informatica, Jitterbit, MuleSoft solutions to integrate a wide variety of
and SnapLogic are gaining a lot of traction on-premises and cloud-native applications,
in the integration space, and their adoption which generally encompass the following
is accelerating with the increasing four functions:
popularity of cloud-native deployments.
1. Adapters that enable connectivity via a variety of open and proprietary APIs and
protocols;
REASON #5 FROM 7
WARNING SIGNS YOU NEED
TO REPLACE YOUR ESB:
infrastructure.”
Source: Dell Boomi
HOW TO EVENT-ENABLE YOUR IPAAS WITH
AN ADVANCED EVENT BROKER
Now let’s talk about why and how you advanced event broker to handle the high-
can take things to the next level by speed, high-volume distribution of events
incorporating an advanced event broker between globally-distributed applications,
into your iPaaS architecture. devices and nodes of your iPaaS.
Simply put, you’ll be using your iPaaS Here’s what you should strive to achieve as
for frontline integration, application you event-enable your iPaaS as a means of
connectivity, service choreography, and realizing true digital transformation.
message transformation, and using an
Agility is king. If you aren’t agile, you won’t synchronous request/reply interactions
be competitive. Building applications as a tightly couple services.
collection of cloud-based and on-premises
If someone needs to define, encode, test
microservices is a great way to make your
and deploy specific paths for information
IT infrastructure more agile, but tight
to take, they won’t be able to quickly adapt
coupling between these microservices
to new requirements. An event-driven
through request/reply-based synchronous
architecture lets new services participate
interactions can keep you from realizing
in existing flows dynamically, without
those improvements.
encoding or deploying any rules.
That’s because event-driven architectures
Traditional ESB orchestration involves
generally enable the loose coupling
codifying service interactions in
of microservices, while traditional
technologies such as BPEL, which means a
One of the world’s leading consumer that would exist in their new architecture.
products companies wanted to reduce Generally speaking, they needed to
costs and improve corporate agility overcome many of the challenges described
by consolidating over 60 datacenters above:
around the world into just 15 by migrating
1. They needed to establish extremely
applications and workloads to the cloud.
reliable and real-time event-driven
They relied on TIBCO BusinessWorks as an communications between systems
on-premises ESB to integrate applications, running in a variety of cloud and on-
and TIBCO Enterprise Message Service premises environments, commonly
(EMS) for queueing, routing, transportation called hybrid cloud and multi-cloud
and persistence of information between a architecture. Complicating this hybrid
powerful SAP ERP system and a plethora of cloud connectivity is that the number
supporting 3rd-party applications. of producers of data is unknown and
constantly changing, as is the number
To meet their goal of migrating workloads
of consumers of data, which can include
from datacenters to the cloud, the
cloud and on-prem destinations such as
company selected Dell Boomi to give their
AWS S3, Hadoop, Postgres and SAP.
cloud-native systems and services easy
access to capabilities such as connectivity, 2. They needed to integrate legacy
transformation, orchestration and error applications with IoT devices and
handling, all of which Boomi provides for microservices, and do so in a way that
both on-premises and cloud-native systems. enabled gradual migration both from
datacenter to cloud, and from batch-
From there the company needed a way
based to event-driven.
to connect the different service endpoints
13 Upgrading Your ESB to Event-Driven iPaaS
3. They needed to ensure rapid, reliable, The first application was to synchronize a
guaranteed delivery over very long master data management (MDM) system
distances (the Internet and private across various internal and external
WANs), factoring for long and less applications in two public clouds (AWS and
consistent round trip times, less Azure) and an on-premises system of record
predictable availability, and the need to as diagrammed below.
minimize/optimize bandwidth.
Batch data was also transformed into
Using the combination of Dell Boomi and streams at source, so consumers could
Solace PubSub+, the company was able consume a filtered stream on what they
to deploy a hybrid and multi-cloud event needed via wildcarded topic routing.
distribution backbone for gradually moving
apps to the cloud.
Given this architecture and associated devices only need to publish data to a local
operational requirements, they needed the Solace node. From there, AWS analytics can
SAP MDM system to be able to distribute consume it locally in the cloud either via
data to downstream SaaS applications. And subscriptions or a RESTful push.
as more and more applications were being
The company is now looking to build
spun up in the cloud, they needed the
a global event mesh on this same
ability to easily fan-out data to many new
infrastructure so they can publish in one
destinations.
region and fan-out to all the others. This
They utilized an event-driven architecture could very well be a many-to-many data
and the publish/subscribe exchange exchange, made easy with Boomi providing
pattern to decouple senders and receivers, integration and application services and
allowing for the very fast subscription- Solace ensuring the real-time guaranteed
based addition of new destinations. delivery.
Beyond the master data management While a current Managed File Transfer
application, many other applications—such solution can provide the data movement,
as order management, order processing, it is unable to offer the granularity,
and customer channel integration— self-routing, real-time, and protocol
leveraged the same event-driven iPaaS. independence of a Solace/Boomi event-
driven iPaaS implementation. For that to
The company also needed to give
happen, bidirectional data delivery, which
applications and devices the ability to send
Solace PubSub+ natively supports, would
and receive messages using JMS and REST,
need to be coded into MFT-style solutions.
both of which are supported by Solace
PubSub+. And because they plan to use
this same infrastructure as the foundation
of an IoT platform, they needed support for
MQTT, which PubSub+ also supports.
This shift in mindset is what Gartner refers to as “event thinking(3)” and it’s critical for
enabling modern data integration and data movement across distributed enterprises.
Combining the easy, flexible, cloud-native integration that iPaaS offerings provide with
an event-driven architecture is the only way to achieve the maximum value from cloud-
native applications, hybrid cloud deployments, and microservices.
Endnotes
3. GARTNER “INNOVATION INSIGHT FOR EVENT THINKING.” 5 OCTOBER 2018. YEFIM NATIS,
ROY SCHULTE, ROB DUNIE, ELIZABETH GOLLUSCIO.
R
solace.com