Assessing Event-Driven Middleware Technology For Modern Application Architecture

VIEWKEYINSIGHTS(HTTPS://WWW.GARTNER.COM/DOCUMENT/3655417/KEYINSIGHT?
REF=DDISP)
This research note is restricted to the personal use of QuyetDo Bui (QuyetDB@fsoft.com.vn).
Assessing Event-Driven Middleware Technology for Modern

Application Architecture
Published: 27 March 2017 ID: G00319319
Analyst(s): Gary Olliffe
Summary
Demand for agile and exible solutions is driving growth in event-driven
architecture, but implementing EDA adds complexity to system design and delivery. Technical professionals architecting microservices,
integration and APIs must evaluate EDA and use appropriate middleware to support it.
Overview
Key Findings
EDA is a well-dened pattern, but this has not yet led to widespread standardization or codication as has happened with SOA and
REST. This is largely because it differs from common procedural programming patterns, making it harder to learn.
EDA is a natural t for modern distributed-application architectures, including microservice architecture and serverless application
architecture. This is because it is based on minimal coupling and enables exible composition of components.
Open-source middleware technology has lowered the commercial and technical barriers to EDA. Still, success is more dependent upon
learning the skills and practices for design and execution than on technology.
Recommendations
To modernize your application architecture for digital business solutions:
Harness existing pockets of EDA expertise and technology in your organization. EDA is not new, and it is likely you have some. Harvest
what you nd and apply it across your development teams.
Use a multiprotocol message broker as a general-purpose middleware for EDA to reduce technology proliferation and support protocol
mediation.
Use cloud messaging and event streaming platform capabilities to support advanced requirements, such as event-sourcing persistence
(storing application data as a log of events), and massively scalable distributed deployment.
Focus on developer experience when designing event-driven APIs. Use open-standard protocols and publish interface specications
that dene event types, payloads, topics and endpoints.
Analysis
In digital business, we face the challenge of building or composing distributed systems that take advantage of remote endpoints (e.g.,
mobile or web), use internal and third-party APIs, leverage cloud platforms, and integrate with existing business applications. These
systems also must be exible enough and scalable enough to cope with dynamically changing business needs.
Service-oriented architectures (SOAs), whether based on SOAP, REST or RPC-style interfaces, help to solve this problem. However, they
encourage request-response interaction patterns that have limitations:
Interactions are only driven by the client.

The client dictates which service will process the request.
A client must wait for a response to each request, even if it is able to continue other processing tasks in the interim.
These limitations constrain the responsiveness, exibility and scalability of a system, which are three key demands of digital business
applications. Introducing event-driven architecture (EDA) can address these limitations. EDA enables the components of a system to react
to events as they are received, enables multiple services to subscribe to and process events in parallel, and breaks the temporal coupling
between request-response clients and services. When an event notication is sent, it simply captures that something happened. It is not a
request, and it is immutable (it cannot change or "unhappen").
In applications built using event-driven architecture, one or more components produce events or consume them and trigger processing
logic. 1 (#dv_1_k_chandy) These components may be very simple, ne-grained services or very complex systems. However, the only
relationship between event-driven components is the one-way communication of events from a producer (also referred to as an event
source) to a consumer (also referred to as an event sink or event handler), as shown in Figure 1. Note that a component may be both a
producer and consumer. In addition, two components each can consume events emitted by the other, but these ows must be independent
and asynchronous.
Figure1.CommunicationinEDAIsOneWay
Source: Gartner (March 2017)
Your application must use appropriate communications middleware to manage the delivery of event notications. As a result, technical
professionals responsible for application architecture must answer the question:
"How can my organization use event-driven middleware to deliver more exible, extensible, scalable
and robust application architectures?"
This middleware must fulll the role of "event broker," an intermediary that manages the delivery of events from producers to subscribed
consumers.
The demand for solutions with more exibility, extensibility, scalability and robustness is also inuencing application architecture,
including:
Microservice architecture (see "How to Design Microservices for Agile Architecture"

(https://www.gartner.com/document/code/319890?ref=grbody&refval=3655417&latest=true) )
Modern mobile architecture (see "Decision Point for Selecting Your Mobile App Architecture"
Web application architecture (see "Assessing Event-Driven Architecture for Scalable and Reactive Web Applications"
Serverless application architecture (see "Adding Serverless Computing and fPaaS to Your Cloud-Native Architecture Toolbox"
Business intelligence and analytics (event stream processing [see "Hyperscaling Analytics: Comparing Streaming Analytics in the Cloud
With AWS, Microsoft Azure and IBM" (https://www.gartner.com/document/code/310019?ref=grbody&refval=3655417&latest=true) ])
Machine learning (see "Preparing and Architecting for Machine Learning" (https://www.gartner.com/document/code/317328?
ref=grbody&refval=3655417&latest=true) )
Lambda/Kappa architecture (see "Harness Streaming Data for Real-Time Analytics"

(https://www.gartner.com/document/code/315223?ref=grbody&refval=3655417) )
In each of these architectures, the role of events is not new; it is amplied.
In event-driven architecture, the capture, communication, processing and persistence of events are
core to the structure of the solution rather than a supplemental extension.
This does not mean that SOA, or traditional state persistence and data management go away. EDA is not a panacea there are scenarios
where a service consumer needs to request or retrieve information synchronously, or where the additional complexity of managing events
delivers little benet. Successful solutions using EDA must combine event-driven patterns with more familiar patterns to achieve optimal
developer productivity, integration and technical complexity.
Middleware Objectives for EventDriven Architecture

Using traditional request-response SOA patterns presents the following common challenges, which EDA can help with:
Pull-based interaction A client must pull information from the service by requesting it. This is ideal for accessing persisted data, but
much less efcient when a client needs to know about data created by a long-running process or updated via a distinct channel. Pull-
based status requests will resort to polling patterns, incurring overhead with each pull request, and delays dictated by the polling
interval. By contrast, EDA pushes events to consumers.
Coupling The client must know the details of the API or interface to the service that will process the request and how to discover the
endpoint at runtime. With effort, this coupling can be kept loose, but each client must know the details of each service to which it needs
to send data. In EDA, event producers have no knowledge of event consumers that ultimately receive event data. There is still some
minimal coupling, as consumers need to know the meaning of events, their formats and how they will be delivered (for example, the
names of topics and network addresses of middleware-hosted endpoints).
Service availability A service must be available and responsive at the time a request is made and remain so until the reply is sent to
the client. This applies even when asynchronous request handling is used. By contrast, events require no reply and are inherently
asynchronous. This allows events to be persisted for future consumption and processing of events to be delayed when consuming
services are busy or unavailable. With the right middleware, the event producer does not need to know whether there are event
consumers available. This provides resilience to network and compute failure, and allows event producers to avoid blocking when there
is high demand on event processing services.
Process modication and extension Process logic in request-response-based SOA is typically orchestrated by a process service
implemented in code or using an integration platform or framework. If the business process needs to change for example, to
introduce, remove or trigger a new service the denition of the process service must be updated. This introduces change
management and regression risk to all its dependencies. With EDA, processes can execute as implicit chains of events, and additional
event producers or consumers can be added to a system without changing any explicit process denition.
Consistency between process interaction and state management Request-response services manage state and processing based on
requests. A request to update an "order" record or resource mutates the state of the order. The request and the change in state are
distinct actions and change history must be managed separately. Event-driven architectures can support consistency between process
interaction and state management. Capturing and storing events as the primary persistence mechanism (e.g., using event logs and the
event-sourcing pattern) means that you are also capturing the triggers for any state transitions in your process. With a disciplined
approach to event payload denition, versioning and persistence, the current state of the system can be built or rebuilt from the log of
events. The stored events also can be replayed, analyzed and used to diagnose issues. Game and client application developers have
long understood the value of capturing actions as a rich log of events, supporting replay, "undo/redo" and debugging of complex
interactions.
For further explanation of EDA and the relationship between EDA and SOA, please see the Details section below.
While EDA does not explicitly require the use of middleware, using an intermediary between event producers and consumers will help to
implement required patterns and deliver more manageable and scalable solutions.
This research focuses on middleware to support three common uses cases for EDA in application architecture:
Microservice architecture using event-driven architecture to reduce coupling of microservices and improve scalability and exibility.
Event-driven APIs using event-driven architecture to support open, scalable, near-real-time, push-based communication in APIs
published to third parties.
Application integration using event-driven architecture to support application-to-application integration scenarios.
These three use cases are far from the only scenarios where EDA has powerful benets. For example, web applications, mobile
applications, Internet of Things (IoT) and analytics also can leverage event-driven architecture to create powerful solutions.
For more detailed analysis of other use cases, see the following Gartner research:
Web "Assessing Event-Driven Architecture for Scalable and Reactive Web Applications"
(https://www.gartner.com/document/code/310108?ref=grbody&refval=3655417&latest=true)
Mobile "Decision Point for Selecting Your Mobile App Architecture" (https://www.gartner.com/document/code/300055?
ref=grbody&refval=3655417&latest=true)
IoT "Assessing Integration Architecture for Internet of Things Solutions" (https://www.gartner.com/document/code/297261?

Analytics "Hyperscaling Analytics: Comparing Streaming Analytics in the Cloud With AWS, Microsoft Azure and IBM"
CommunicationPatternsforEventDrivenArchitecture
EDA demands a unidirectional ow of events from producers to consumers. This is most commonly implemented using the publish-
subscribe (pub-sub) pattern producers publish events, while consumers subscribe to receive events. These events are communicated as
messages that describe the event. Figure 2 shows how this logical ow is supported by implementing a pub-sub conversation ow and a
generalized view of how the technical ow takes place.
Figure2.TheLayersofCommunicationinEDA
ACK/NACK represents delivery acknowledgment/negative acknowledgment. It conrms delivery, not completion of processing
topic. A named logical channel for delivering events is specied by the event producer. A topic groups related events and enables a consumer to subscribe to that grouping.
Topic-based subscriptions (as shown in Figure 2) are the most widely supported. Some middleware platforms enable content-based
subscriptions. This allows event consumers to dene the patterns of event content (message header and/or body) they want to receive.
Content-based pub-sub requires the middleware to inspect the content of every message and compare it to every subscription pattern to
determine which consumers should receive it.
The technical ow (i.e., delivering events) may support or require a request-response interaction between participating components and
the event broker. For example, most message brokers expose a REST API for publishing and consuming messages in addition to their
proprietary interfaces and Java Message Service (JMS) providers. Increasingly, they also offer support for open messaging protocols such
as Advanced Message Queue Protocol (AMQP), Message Queued Telemetry Transport (MQTT) and the Simple, or Streaming, Text Oriented
Messaging Protocol (STOMP). Some cloud-hosted, message-oriented middleware currently provides only a REST API, for example,
Amazon SQS (https://aws.amazon.com/sqs/) and Google Cloud Pub/Sub (https://cloud.google.com/pubsub/docs/overview) . Google has
an alpha release of a gRPC-based interface for Pub/Sub (https://cloud.google.com/pubsub/docs/reference/rpc/) available at the time of
this writing.
Event Broker Middleware for EventDriven Solutions

One key challenge for teams or organizations adopting event-driven architecture is choosing the right kind of middleware to meet their
needs. Messaging and event-driven technology have a long history, and event broker capabilities have been implemented in several classes
of technology. The most prominent categories of middleware software and services that support EDA are:
Message brokers This is middleware that decouples message sources and destinations and provides routing logic to manage the
distribution of messages, including patterns that support EDA.
Event streaming platforms This middleware manages ingestion of events from event sources and provides read-only access to event
consumers.
Subscription-enabled data stores This involves data persistence technology that enables external event consumers to subscribe to
specic data changes or discrete events published by an event producer.
These technologies focus on message-based communication patterns that can be used to support EDA. Other product categories build
upon or incorporate the above technologies, including enterprise service buses (ESBs), integration platform as a service (iPaaS), mobile
back-end platforms and IoT platforms, but they are out of the scope for this assessment. Also out of scope are programming frameworks
for brokerless messaging and reactive applications.
MessageBrokers
Message brokers are the most mature and common form of middleware supporting EDA. Many of the products available today have a long
and proven heritage going back decades in some cases. The dening characteristic of a message broker is that the broker itself is a
discrete service producers and consumers communicate with the broker using standard or proprietary protocols.
Examples of products and technologies in this category include:
Commercial software: IBM MQ, Microsoft Service Bus for Windows, Microsoft MSMQ, Software AG Universal Messaging, TIBCO
Enterprise Message Service (EMS) and TIBCO Rendezvous
Commercially supported open-source software: Red Hat JBoss A-MQ (commercially supported ActiveMQ), Pivotal RabbitMQ and WSO2
Message Broker
Open-source software: Apache ActiveMQ, Mosquitto, Nats, NSQ, Pulsar and RabbitMQ
Cloud services: Microsoft Azure Service Bus, IronMQ, Amazon SQS, PubNub, Google Cloud Pub/Sub, CloudAMQP and MuleSoft
Anypoint MQ
Figure 3 shows a simplied view of a typical message broker, which implements pub-sub by routing event messages from the producer to
queues for each class of consumer. This mechanism provides support for the shared-subscription pattern described above. The internal
structure and implementation details vary from product to product.
Figure3.TheStructureofaMessageBrokerSupportingPublishSubscribe(EventBroker)

The broker itself should be deployed in a clustered conguration to provide resilience and horizontal scalability. Many message brokers
now support multiple protocols for publishing and consuming messages supporting varying degrees of openness, performance and
features. A typical multiprotocol message broker, such as IBM MQ or Apache ActiveMQ, will have support for at least three of the following:
Proprietary: This is when one or more proprietary binary network protocols, typically wrapped by a JMS provider and other language-
specic client libraries, are used to deliver optimal performance; for example, OpenWire is used by Apache ActiveMQ, but many vendors
do not publicly name or publish these protocols.
AMQP v.1.0: This is an open-standard, interoperable wire protocol that supports access over TCP/IP. It is an OASIS standard
(https://www.oasis-open.org/committees/amqp/) .
AMQP v.0.9.1: This is an open, interoperable wire protocol specication (https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf)

that supports access over TCP/IP. It shares very little with AMQP v.1.0 and should be considered a separate protocol.
MQTT v.3.1.1: This is an open-standard, interoperable, lightweight message protocol for constrained devices and network connections.
STOMP: 2 (#dv_2_stomp_github) This is an open-text-oriented messaging protocol intended to make client libraries for multiple
languages and platforms.
RESTful: This is a broker-specic HTTP API used to simplify publishing and consuming messages from web-based clients.
Message brokers support different levels of quality of service (QoS) that control how messages are persisted, routed and purged from the
queues managed by the broker. The QoS conguration determines whether the messages sent fulll requirements for once-and-only-once,
at-least-once or at-most-once delivery. It is also possible to dene the topics or subscriptions to be durable or transient. A durable
subscription will retain messages even when all consumers have disconnected, while a transient topic will be automatically destroyed by
the broker when all consumers have disconnected. This capability can help balance reliability and resource consumption on the broker as
consumers subscribe and disconnect. Message brokers will also support message ordering semantics that will ensure events are
published in the order they are received by the broker, apart from exceptional circumstances. These circumstances can include network
disconnections, full queues, 3 (#dv_3_ibm_mq) multiple consumers 4 (#dv_4_activemq_) and requeued messages. 5 (#dv_5_rabbitmq_)
Message brokers also support transactional semantics, allowing groups of messages within a session to be committed or rolled back as a
whole. In some cases, support for distributed transactions (for example, using the XA protocol 6 (#dv_6_distributed_transaction) ) enables
transaction contexts to include other resources, such as databases and other message brokers.
Some message brokers, such as TIBCO Rendezvous and Microsoft MSMQ, use a distributed broker model with a broker service or daemon
running on each application host. Applications send messages to the broker service, and the service routes the messages to the intended
destination servers.
EventStreamingPlatforms
Event streaming platforms are a relatively new category of technology aimed primarily at event stream processing and analytics scenarios.
Commercial software: Conuent Platform (commercially supported platform based on Apache Kafka) and SAP Hana Smart Data
Streaming
Open-source software: Kafka (commercial support also provided by Cloudera), Nats Streaming (support provided by Apcera) and
CorfuDB
Cloudstreams: Apache Kafka on Heroku, Amazon Kinesis Streams, Azure Event Hubs and IBM Message Hub
The distinguishing feature of these platforms is that the event messages that they process are not held in queues, and they are not
removed once read or delivered. Instead, event messages are held in append-only event logs. Each log represents an event topic. Events
published by a producer are always appended to the end of the log for the appropriate topic. Consumers read a topic log from an offset
position, a point in the log which they manage themselves or let the platform manage on their behalf. As events are consumed, the offset
into the log is incremented. Different consumers can have different offsets, allowing them to process the data at different paces or times,
and because the log is append-only and written events are immutable (read-only), there are fewer concurrency concerns. Figure 4 shows a
representative event stream platform architecture.
Figure4.TheStructureofPublishSubscribeinanEventStreamingPlatform
The log itself may be replicated across multiple nodes and sharded (split based on some key attribute of each event) for scalability. The
log is typically persisted to disk for durability and, due to its append-only semantics, can provide very efcient data persistence. This
capability makes the technology suitable for implementing the event sourcing pattern, which uses the log of events processed by an
application or service as the "source of truth" or master record (see "How to Design Microservices for Agile Architecture"
(https://www.gartner.com/document/code/319890?ref=grbody&refval=3655417&latest=true) for details). The events in the log can be
replayed, analyzed or reprocessed by new services at a later time or date. The other advantage of this log-based approach is that
consumers can have more control over the ow of events they must consume. In the case of Apache Kafka, reads are implemented as a
polling mechanism (abstracted from event consumer code by client-side libraries). This allows consumers to request events when they are
ready to process them and contrasts with message brokers, which typically rely on back-pressure feedback from consuming clients to
adjust the ow of outbound events.
The platforms provide APIs or software development kits (SDKs) to support the reliable publication of events to the platform and reading
events from the stream. As with message brokers, there is typically a proprietary and efcient network protocol supported by these SDKs
with additional support for other, more open protocols (for example, AMQP v.1.0 or REST over HTTP).
The most notable product in this market is Apache Kafka, which supports highly distributed and scalable processing by allowing
partitioning of topics and ensuring write consistency across replicas of each partition. It is used as the basis for a number of cloud-hosted
event stream services.
SubscriptionEnabledDataStores
Subscription-enabled data stores are databases and other data management tools that enable components of an EDA system to publish
and subscribe to events managed by the data store. In some cases, this is pure publish-subscribe support. For example, Redis Pub/Sub
(https://redis.io/topics/pubsub) and Hazelcast Topics (http://docs.hazelcast.org/docs/3.5/manual/html/topic.html) are independent
features built on top of the platforms' in-memory database capabilities. In other cases, the events relate to changes in data managed by
the platform. For example, Apache Geode (https://cwiki.apache.org/conuence/display/GEODE/Reliable+Messaging) , Hazelcast Maps
(http://docs.hazelcast.org/docs/3.7/manual/html-single/index.html#listening-for-map-events) and Oracle Streams
(https://docs.oracle.com/cd/B28359_01/server.111/b28321/strms_over.htm#i1006084) allow clients to register interest in (i.e., subscribe
to updates) specic key-value pairs managed by the platform. It publishes events with updated data to those clients when the values are
changed by another client. Figure 5 shows an example of publish-subscribe implemented in an in-memory data grid (IMDG) using both
patterns.
Figure5.TheStructureofPublishSubscribeinaSubscriptionEnabledDataStore
Topic-based pub-sub in these platforms provides simple message broker characteristics. Most implementations lack control over the QoS
and message durability, but the in-memory implementation model offers performance and latency benets that can be critical in some
scenarios.
The use of data-driven events, whether based on key-value data (as illustrated in Figure 5) or document-based or columnar data, can also
remove a level of indirection in processing when the events in your system are based on data change. An example would be when service
A creates an "order" object value in the store and other services subscribe to keys of that type or a naming structure using lters. Most
tools that support this capability are either IMDG or in-memory database (IMDB) products. You must rst assess whether adopting in-
memory technology for your application is appropriate (see "Architect Cloud-Native Solutions With In-Memory Computing"
(https://www.gartner.com/document/code/292676?ref=grbody&refval=3655417&latest=true) ).
Commercial software: Hazelcast IMDG Enterprise, MongoDB Enterprise Advanced, Red Hat JBoss Data Grid (commercially supported
Innispan), Oracle Streams AQ, Pivotal GemFire (commercially supported Apache Geode) and Redis Labs (Redis Enterprise).
Open-source software: Apache GemFire, Redis, Hazelcast IMDG Open Source, MongoDB open source (tailable cursors) and Innispan
Cloud services: Amazon ElastiCache, Hazelcast IMDG cloud deployment, Microsoft Azure Redis Cache and Redis Cloud
RelatedToolsandTechnologies
Message brokers, event streaming platforms and subscription-enabled data stores represent the most common event-driven middleware
in the marketplace. Other tools support very specic use cases or can be combined with the middleware to improve developer productivity.
BROKERLESS MESSAGING
Brokerless messaging tools provide a mechanism to support direct communication of events between processes or services. These tools
are aimed at scenarios with relatively simple delivery semantics but requirements for very low latency. They eliminate the overhead of a
broker, which requires a minimum of two application-layer network hops from producer to broker and broker to consumer(s). They also
suit scenarios where the complexity of a broker or the streaming capabilities of an event streaming platform are not required. The
implementations typically use some form of multicast message distribution.
Examples of products and tools in this category include:

Commercial: IBM WebSphere MQ Low Latency Messaging, Informatica Ultra Messaging, RTI Connext DDS, Software AG Universal
Messaging (peer-to-peer mode) and TIBCO FTL
OSS: ZeroMQ
DEVELOPMENT FRAMEWORKS
A growing number of application development frameworks are embracing event-driven patterns. These vary in their design goals, from
supporting the actor model 7 (#dv_7_akka_notes) of application design to enabling stream-based processing of (event) data in ight. The
actor model relies on message passing to communicate between components (actors), whether in process, out of process or remote.
These messages can be used to communicate event notications. 8 (#dv_8_distributed_publish) , 9 (#dv_9_event_processing)
Stream-based processing enables developers to connect to a variety of event sources and sinks. Processing logic can then be structured
as a chain of actions to be applied to a continuous stream of events processing individual events or ltering, aggregating and
propagating multiple events. These frameworks provide tools for addressing integration requirements that are owned by application
development teams.
In both styles of frameworks, messaging middleware can be used to manage distribution of events or as sources and sinks for the events
being processed.
Examples of frameworks in this category include:
Message-based programming models: Axon Framework, Akka Streams, Lightbend Lagom, Microsoft Azure Service Fabric Reliable
Actors, Seneca and Vert.x (event bus)
Message processing and integration frameworks: Apache Camel, Mule runtime, Spring Integration, Spring Cloud Stream and Spring
Cloud Data Flow
IOT PLATFORMS
The capture of data from connected devices in IoT solutions is inherently event-driven. Events captured from these devices represent
changes in the state of a physical object, system or environment. Connected devices can also react to these local events, or receive and
react to events from other devices or from other software components of the solution. IoT platforms provide turnkey capabilities that
enable robust deployment and operation of IoT solutions. These capabilities include device management, device identity management,
device connectivity, data ingestion, event stream processing and integration.
The devices and back-end services in an IoT solution provide an ideal opportunity to apply event-driven architecture. EDA can reduce
coupling between devices and other components, support scalability of consumers as device counts and usage change, and support
stream-based analytics.
For further details on IoT plaforms, please see "Assessing Integration Architecture for Internet of Things Solutions"
(https://www.gartner.com/document/code/297261?ref=grbody&refval=3655417&latest=true) and "Exploit the Strengths of the Public
Cloud for Your IoT Platform." (https://www.gartner.com/document/code/297298?ref=grbody&refval=3655417&latest=true)
Examples of platforms in this category include:
Amazon Web Services IoT (https://aws.amazon.com/iot-platform/) , AT&T M2X (https://m2x.att.com/) , GE Predix

(https://www.predix.io/) , IBM Watson IoT Platform (https://www.ibm.com/internet-of-things/iot-solutions/watson-iot-platform/) ,
Microsoft Azure IoT Hub (https://azure.microsoft.com/en-gb/services/iot-hub/) and Salesforce IoT Cloud
(https://www.salesforce.com/iot-cloud/)
Three Application Architecture Use Cases

Digital business requirements are driving application architects to address a number of critical use cases that can benet from event-
driven architecture and the middleware that supports it. We will focus on three of the most common:
Microservice architecture using event-driven architecture to reduce coupling microservices and improve scalability and exibility.
Application integration using event-driven architecture to support application-to-application integration scenarios.

Event-driven APIs using event-driven architecture to support scalable, near-real-time, push-based communication in APIs published to
third parties.
UseCaseNo.1:MicroserviceArchitecture
Microservice architecture (MSA) is an application architecture pattern that decomposes application functionality into loosely coupled, ne-
grained, independently deployable (and scalable) application services. The goal of MSA is to support high levels of delivery agility and
velocity by reducing the size and scope of the deployable components of a system. For further details, see "Assessing Microservices for
Cloud-Native Application Architecture and Delivery" (https://www.gartner.com/document/code/297262?
ref=grbody&refval=3655417&latest=true) and "How to Design Microservices for Agile Architecture."
It is critical that individual microservices are designed to be as loosely coupled as possible. This is where event-driven architecture comes
in. As described earlier, the components in an EDA are not just loosely coupled, but minimally coupled. The producer of events has no
knowledge of the consumers that may subscribe to and process those events. Equally, consumers have no knowledge of the producer(s)
of the events they are interested in; they are simply notied about each event and process it accordingly.
EDA plays a role in three scenarios in a microservice architecture. They are event-driven ow of processing between services, data
synchronization and event sourcing.
Event-driven ow of processing between services: This is the most impactful scenario for MSA. For some interactions, the dependencies
even if loosely coupled that are introduced using request-response interactions like REST or gRPC are not acceptable. This scenario
requires middleware to provide an event communication channel that can be shared by publishing and subscribing microservices in a
system, as shown in Figure 6. Multiple brokers can be used for scalability, isolation and operational alignment to service domains. Brokers
must be able to scale and perform to meet the general-purpose requirements of each event-driven microservice.
Figure6.EventDrivenFlowofProcessingBetweenServices
Data synchronization: The decoupling of microservices from each other should include decoupling their data models and data persistence.
This enables each service implementation to make independent decisions about changes to its data model or the data store it uses. This
split makes change easier, but data often must still be accessible to other services. One way to achieve this is to allow services to request
data from the services that own it when they need it. This request-response approach to data access is simple but can introduce
undesirable latency and runtime dependencies because the services being called must be available and performing well. Event-driven
architecture can be used to implement data synchronization between services, enabling services that need data to subscribe to updates
from the service that owns the data (see Figure 7). These services can then maintain a local replica or cache of the data they need to fulll
requests from their own clients without calling external services to retrieve the data.
Figure7.DataSynchronization
Event sourcing: Microservice architecture is often used for highly scalable platforms with high performance requirements. Event sourcing
is a design pattern for storing the state of a system as a log of events (like a ledger). This not only ts well with services that use an event-
driven ow of processing, but also can support very fast write performance because the event log is append-only. This pattern is frequently
combined with the Command Query Responsibility Segregation (CQRS) pattern. In this case, the command (update) persistence is handled
by the event source, and the Query (read) persistence is handled by one or more structured data stores optimized for read performance.
EDA is required to keep the write and read stores eventually consistent. An example of event sourcing used with CQRS is shown in Figure 8.
Figure8.EventSourcingUsedWithinaMicroserviceUsingCQRS
MAPPING MIDDLEWARE TO THE MICROSERVICE USE CASE

Table 1 describes how different microservice requirements for event-driven middleware map to the three classes of technology described
earlier in this research.
Table1.MiddlewareforEventDrivenMicroservices
Message Broker
Software Use for interservice choreography when message durability but not persistence is required and multiple protocols and patterns need to be supported.
Do not use for event sourcing.
Cloud Service Use for interservice choreography when targeting a public cloud environment.
Assess your portability needs when considering services delivered by general-purpose infrastructure as a service (IaaS) and PaaS providers.
Event Stream Platform
Software Use for implementing event sourcing.

Also use when event log semantics (append-only log and pull-based consumption) are required.
Cloud Service Use for event sourcing, but take regular snapshots.
Use when targeting a public cloud environment and cross-cloud portability is of low concern.
Subscription-Enabled Data Stores

Software Use for low latency, distributed data persistence and change notication within the scope of individual services.
Ensure that you retain decoupling between services when using key-value update notications as an event-driven communication channel.
Cloud Service Use for low latency, distributed data persistence and change notication within the scope of individual services.
Pay attention to cloud portability, noting that most services provide APIs compatible with open-source tools, such as Redis.
UseCaseNo.2:EnterpriseApplicationandDataIntegration
Every enterprise has integration requirements they are pervasive and highly varied. Many of these integration requirements can be
adequately handled by traditional data-centric or SOA-centric integration. However, as the demand for more real-time, reactive processing
of data crosses application and organizational boundaries, the need for event-driven integration will increase.
In data-centric integration, the movement of data between systems is often on a scheduled basis. This means that downstream systems
are almost always out of date compared with the source system. In many SOA-centric integrations, access to data or features of another
system only happens when the client system requests it. Event-driven architecture enables integration to be more reactive to change, when
it is needed. Events can trigger the movement of data and the events can even carry the data. Those same events can trigger processing
by subscribing systems, which might otherwise have only happened on some schedule.
Event-driven architecture also presents the opportunity to reduce the complexity of centralized integration logic. Enterprise application
integration typically uses an orchestration process to control the ow of tasks to complete an integration process, which is often
implemented and executed using an ESB or iPaaS (see "Choosing Application Integration Technology"
(https://www.gartner.com/document/code/314871?ref=grbody&refval=3655417&latest=true) ). In an event-driven architecture, some or all
of this ow can be managed by the ow of events, as shown in Figure 9. This reduces the complexity and fragility of the integration ow. It
moves the intelligence about the ow of execution to the endpoints in the process, allowing the process to change dynamically (for
example, by the addition of new consumers for existing events). However, there is no prescribed centralized denition of the process ow,
since it is composed of the topics, events, producers and consumers that make up the system at any point in time.
Figure9.ProcessOrchestrationandChoreography
As in the microservice use case, enterprises also need to synchronize data across systems to optimize business process or allow
consolidation analysis and reporting. Ideally, this integration should "touch" each system at a well-dened interface or API, rather than
going directly to the data managed by the system, but sometimes data-driven events are a necessary source. EDA can be used to reduce
the latency of this data exchange and enable the propagation of critical data to multiple downstream systems.
In some cases, this is implemented using message-oriented middleware to connect existing system repositories together. In other cases, a
data hub is created that acts as an aggregation and distribution point for data (see "Implementing the Data Hub: Architecture and
Technology Choices" (https://www.gartner.com/document/code/297674?ref=grbody&refval=3655417) ). A data hub with EDA support
uses middleware to subscribe to changes in data emitted by systems of record and then emits events to which downstream consumers
can subscribe, enabling them to keep in sync with the master data. This provides the enterprise with the possibility of maintaining a
centralized view of data in the hub, while further decoupling the subscribing systems from the systems of record for each data type. This
allows each system to change independently if it meets the requirement to emit or process events using appropriate topics in the agreed
formats.
Finally, digital business is driving the need for real-time analysis of continuous streams of events and the integration of these streams with
more traditional applications. Event stream processing, also known as complex-event processing, enables an organization to look for
patterns or trends in one or more streams of events and trigger actions based on them. For example, event stream processing can be used
to analyze customer behavior in mobile or web applications, and look for fraudulent behavior in nancial transactions or identify risk of
failure in business systems. This requires integration because the events from other systems must be routed into the appropriate streams.
When rules detect a pattern or aggregate a set of events into a high order event, those events may then need to trigger processing in other
business applications or services.
MAPPING MIDDLEWARE TO THE ENTERPRISE APPLICATION AND DATA INTEGRATION USE CASE
Table 2 describes how different integration requirements for event-driven middleware map to the three classes of technology described
earlier in this research.
Table2.MiddlewareforEventDrivenEnterpriseIntegration
Message Broker
Software An effective and proven solution.

Often combined with an integration platform or framework for payload transformation, mediation and orchestration.
Cloud Use in combination with an integration framework for integrating IaaS- or PaaS-hosted custom applications.
Service For integrating SaaS applications that expose event-driven interfaces, use an iPaaS. 10 (#dv_10_choosing_application)
Software Emerging capability when combined with a stream integration framework like Spring Cloud Data Flow (http://cloud.spring.io/spring-cloud-dataow/) or
Alpakka (http://developer.lightbend.com/docs/alpakka/current/) for simple, protocol and platform-level integration of enterprise applications and data
sources.
There are currently no application-specic connectors.
Cloud Integrating IoT or other high-volume and high-velocity event stream sources.
Service Use in combination with stream analytics to detect or aggregate complex events that can then be integrated with enterprise applications.
Software Event-driven support in data grid and database products is not suitable for general-purpose enterprise application integration.
A data store with event-driven support could be used as an effective data hub where the data structures can be mapped to the key-value or document-
based structures typically used by these tools.
Cloud Event-driven support in data grid and database products is not suitable for general-purpose enterprise application integration.
Service
UseCaseNo.3:EventDrivenAPIs
Web APIs have provided organizations with new ways to deliver services to their customers, partners and peer teams, as well as their own
applications. They have provided a mechanism to deliver highly productive and exible, platform-independent integration. However, the
ubiquitous RESTful model of APIs is typically used to implement request-response interactions. These interactions require the client
application to initiate requests to access resources or create new ones and trigger application logic. Users of applications and participants
in business processes are demanding ever-more-responsive experiences. They expect to be told when something has changed that could
inuence their decision making without hitting the refresh button. Information must be pushed rather than pulled.
Event-driven APIs provide the solution. Using publish-subscribe semantics allows clients to subscribe to an API and receive events as they
happen within the service they are using. This prevents the client from having to poll for updates or changes, reducing the burden on client,
service and network connection. It also enables near-real-time updates based on events taking place within the service, whether to support
reactive, engaging user experiences or faster decision making in a business process. An event-driven API requires two capabilities a
mechanism to allow a consumer to subscribe (this can be user controlled or programmatic) and the means to deliver events to consumers
that are subscribed (see Figure 10).
Figure10.EventDrivenAPIs
Of course, not every use case needs this level of responsiveness and the additional middleware it requires. See "A Guidance Framework for
Designing a Great Web API" (https://www.gartner.com/document/code/300000?ref=grbody&refval=3655417&latest=true) and continue to
follow the direction it provides to ensure that your API designs are fullling the needs of the various developer personas and scenarios.
Event-driven APIs require the same design discipline as RESTful APIs. While REST itself is not a standard, there are plenty of tools and
widely adopted interface specication formats to guide you. Examples include Open API specication (formerly known as Swagger) and
RESTful API Modeling Language (RAML).
For SOAP-based web services, WSDL 2.0 (https://www.w3.org/TR/wsdl20/#MessageExchangePattern) dened a number of message
exchange patterns (MEPs), including "In-Only" (https://www.w3.org/TR/2007/REC-wsdl20-adjuncts-20070626/#patterns) and extensibility
to support examples like "Out-Only" (https://www.w3.org/TR/wsdl20-additional-meps/#out-only) that could have been used to dene event-
driven interface semantics. However, platform and framework support for these features is inconsistent, and SOAP is not considered a
suitable option for EDA.
The webhook pattern is one pattern that can support event-driven HTTP-based APIs. While the industry has not standardized on this,
support for webhooks has been added to the next version of Open API specication in the form or "callbacks" 11 (#dv_11_callback_feature)
(OAS 3.0 was in development at the time of this writing).
Unless you are using the webhook pattern for event-driven APIs, there are no commonly used specication formats that allow the denition
of event types, payloads, topic structures and endpoints in a holistic manner. Denitions vary per use case and technology.
Today, you must document your own event-driven API specication, bringing together the content that denes how events are structured,
their semantics, how consumers can subscribe (including topic denition, if needed), how events are delivered by a producer or consumer,
and how endpoints are secured and authenticated. For examples of good design and documentation of event-driven APIs, look at
examples in the public domain, such as Slack's Events API (https://api.slack.com/events-api) (based-on webhooks) and Real Time
Messaging API (https://api.slack.com/rtm) (based on WebSockets), the Facebook Messenger Platform API
(https://developers.facebook.com/docs/messenger-platform) (using webhooks) and Questrade's Streaming API
(http://www.questrade.com/api/documentation/streaming) (based on WebSockets).
PAYLOAD FORMATS FOR EVENTDRIVEN APIS

When dening event payloads, rst you must choose whether to use a text or binary format. This choice will be based on balancing the
performance and payload size benets of binary formats with the readability and ease of use of text-based formats.
If you choose a binary format to optimize performance, you will need to choose a data format and a scheme for marshalling event data.
This marshalling scheme denes how event data is moved to and from your providers' and consumers' code into a payload that can be
sent through your middleware. The key here is to choose a format that has strong library support and simplicity for the intended users of
your event-driven API. There is no point in choosing the format with the fastest processing on your platform if it is hard to use or
unsupported for many of your target audience. Commonly used formats include Apache Avro (http://avro.apache.org/docs/current/) ,
BSON (http://bsonspec.org/) (the binary cousin of JSON), Protocol Buffers (https://developers.google.com/protocol-buffers/) , Cap'n Proto
(https://capnproto.org/) and Efcient Extensible Interchange (https://www.w3.org/XML/EXI/) (EXI).
In the world of request-response web APIs, text-based formats have dominated for years. They rst used XML, driven by the popularity of
HTML and, subsequently, used JSON, driven by its ease of use with JavaScript. Both formats now have near ubiquitous support across all
programming platforms as well as integration tools and low-code application development platforms.
For event-driven APIs that you expect to be widely used by developers outside your own development team, Gartner recommends you rst
evaluate whether you can meet your performance and bandwidth requirements using JSON or XML. This will give you the highest usability
and broadest consumer support. It will also provide a consistent developer experience between event-driven and request-response
interfaces.
If your use cases require the efciency of a binary format, begin by evaluating Protocol Buffers. This format is well-established as a cross-
platform marshalling scheme with good language support, small payload size and fast marshalling performance. Payload parity between
event-driven and request-response interfaces can include use of binary payloads. RESTful APIs can use binary resource representations, or
you might consider remote procedure call (RPC)-style protocols such as gRPC (http://www.grpc.io/) (which uses Protocol Buffers). Google
12
(#dv_12_google_cloud) , 13 (#dv_13_cloud_endpoints) and others 14 (#dv_14_machine_learning) are beginning to add support for gRPC
(a binary request-response API format that uses Protocol Buffers payloads), but this is still nascent.
PROTOCOL CHOICES FOR EVENTDRIVEN APIS

After deciding on a payload format, you must chose an application-layer communication protocol. Firstly, this must be an open and widely
supported standard. The most practical options today are:
HTTP combined with the webhook pattern An application protocol and usage pattern supporting outbound event notications based
on an HTTP request to a subscribing service.
Comet 15 (#dv_15_comet_low) model An abstraction model for server-side streaming of data to a wide variety of web clients based
on their capabilities. Frameworks that implement the comet model will typically revert to WebSockets when they are supported by the
client and server.
WebSockets This protocol supports bidirectional streaming of data between a web client and server.
MQTT v.3.1.1 This protocol supports lightweight, topic-oriented publish-subscribe messaging.
AMQP v.1.0 This protocol supports advanced, enterprise-grade messaging and quality of service.
AMQP v.0.9.1 This protocol supports advanced enterprise-grade messaging and quality of service but has a different wire protocol
and distinct programming model from v.1.0. It must be considered a different protocol from AMQP v.1.0.
Table 3 provides a more detailed comparison of these options.
Table3.OpenProtocolOptionsforEventDrivenAPIs
Client
Protocol Network Library
Protocol Adoption for Overhead Size Ease of
Protocol Type APIs (Relative) Built-In QoS (Relative) Use Authentication Sample Tools
HTTP plus Notication- Universal High to None Medium Good Use HTTP ASP.NET webhooks
Webhooks only medium mechanisms (https://blogs.msdn.microsoft.com/webdev/2016/12/14/introducing-

application (usually text- microsoft-asp-net-webhooks-preview-2/) , REST Hooks
transport; based, but (http://resthooks.org/)
Transient binary
connection payload and
compression
supported).
Comet Bidirectional Universal uses High to Framework Framework Medium Use HTTP CometD (https://docs.cometd.org/current/reference/) , SignalR
application HTTP or medium (Will dependent dependent (client and mechanisms (https://www.asp.net/signalr) , Socket.io (https://socket.io/)
transport; WebSockets, but default to provider
Persistent implementations HTTP and frameworks

connection are framework- will must be
specic negotiate to compatible)
use
WebSockets,
if available.
Most
frameworks
can support
binary and
text
payloads.)
WebSockets Bidirectional Widely Medium None Small Medium Use HTTP JEE 7 (https://docs.oracle.com/javaee/7/tutorial/websocket.htm)
data supported (text or mechanisms NET 4.5 and later (https://msdn.microsoft.com/en-

transport binary us/library/system.net.websockets.aspx)
Persistent payload) Node.js ws package (https://www.npmjs.com/package/ws)

connection GoLang WebSocke (https://godoc.org/golang.org/x/net/websocket)
ts package
MQTT Bidirectional Some support Low (text or Basic three Small Good ID + Password Mosquitto (https://mosquitto.org/)
application (IoT-focused) binary levels, no ow or X509 Eclipse Paho (https://eclipse.org/paho/) (includes .NET and Java
transport; payload, can control Certicate libraries)

Persistent operate over
connection TLS, TCP or

WebSockets)
AMQP v.1.0 Bidirectional Some support Medium Advanced three Large Adequate TLS with SASL Apache Qpid (https://qpid.apache.org/index.html)
application (enterprise- (text or levels with ow Azure AMQP.NET Lite (http://azure.github.io/amqpnetlite/)
transport; focused) binary control Node.js amqp10 package

Persistent payload) (https://www.npmjs.com/package/amqp10)
connection
AMQP Bidirectional Limited support Medium Advanced Medium Adequate TLS with SASL Spring AMQP (https://projects.spring.io/spring-amqp/)
v.0.9.1 application (enterprise- (text or Acknowledgment- Node.js amqplib package
transport; focused) binary based, with (https://www.npmjs.com/package/amqplib)

Persistent payload) message prefetch
connection support
HTML5 Server-Sent Events 16 (#dv_16_serversent_events) are not listed because they are specic to browser clients and not suitable for general-purpose APIs.
TLS = Transport Layer Security
SASL = Simple Authentication and Security Layer
Each of these options has pros and cons. You must choose a protocol that meets your technical requirements (e.g., built-in support for
QoS or low-network and processing overhead) and the usability needs of your API consumers.
MAPPING MIDDLEWARE TO THE EVENTDRIVEN API USE CASE

Table 4 describes how different event-driven API requirements map to the three classes of technology described earlier in this research.
Table4.MiddlewareforEventDrivenAPIs
Message Broker
Software For front-end (consumer) integration, use the message broker to expose MQTT and AMQP v1.0 endpoints.
When exposing WebSockets/webhook interfaces implemented in an application runtime, use a message broker to implement and manage event-driven
processing.
Cloud Use as the front end for private, event-driven APIs exposed over REST, MQTT or AMQP v1.0. The focus here is on internal/private APIs because you will lack
Service control over some middleware conguration when delivered as a cloud service.
When exposing WebSockets/webhook interfaces implemented in an application runtime or function platform as a service (fPaaS), use a cloud message broker
to implement and manage event-driven processing.
Software Not suitable. Primarily useful for managing inbound streams.

Outbound event notications required by event-driven APIs would require polling semantics.
Cloud Not suitable. Primarily for managing inbound streams.

Service Integration with fPaaS (serverless functions) could be used to trigger notications for webhook subscribers.
Software Use as the back end for data-centric event notications for WebSockets/webhook interfaces implemented in a discrete application runtime.
Cloud Use as the back end for data-centric event notications for WebSockets/webhook interfaces implemented in a discrete, cloud-hosted application runtime. Pay
Service attention to cloud portability requirements.
OtherRelatedUseCases
The three core application use cases above are not the only scenarios where EDA is valuable. Its ability to provide high-performance,
reactive connectivity between distributed components, the evolution of cloud platforms using event-driven architecture and growth in
analytics tools that can process events as streams of data in motion make EDA ever more common.
These additional use cases and the Gartner research that covers them include:
Reactive real-time web applications Event-driven architecture is not just for the back end. Reactive web applications use multiple
levels of event-driven architecture to provide responsive and interactive user experiences and deliver information to users in near real
time. For more details, see "Assessing Event-Driven Architecture for Scalable and Reactive Web Applications."
Event stream analytics Events are data about things that have happened. That data can be analyzed to aggregate it, identify patterns
and make decisions based on the events over time. Event stream processing and analytics enables organizations to gain insight from
data in ight that cannot be gleaned from analysis of the current state of a system. For more information, see "Hyperscaling Analytics:
Comparing Streaming Analytics in the Cloud With AWS, Microsoft Azure and IBM." (https://www.gartner.com/document/code/310019?
Internet of Things solutions and connected device integration Much of the communication in IoT solutions is event-driven. This
communication includes streams of events from eets of connected devices carrying sensor data, such as a GPS location, temperature
of the environment or vibration data. It also can be more sophisticated events generated by algorithms running at the edge on the
devices or gateways making up the solution. IoT solutions also make extensive use of event stream processing to gather insight and
make decisions. For further details, see "Assessing Integration Architecture for Internet of Things Solutions"
(https://www.gartner.com/document/code/297261?ref=grbody&refval=3655417&latest=true) and "Guide to Deploying IoT Analytics
From Edge to Enterprise." (https://www.gartner.com/document/code/317014?ref=grbody&refval=3655417&latest=true)
Serverless application architecture The rise of fPaaS has led to the emergence of event-driven serverless application architectures
that implement their application logic in discrete functions hosted and executed in the cloud. These functions are triggered by events
and can trigger further events. They can be triggered directly from events published by other services offered by the cloud provider (for
example, a cloud hosted message broker, event streaming platform or data store). For further details, see "Adding Serverless Computing
and fPaaS to Your Cloud-Native Architecture Toolbox." (https://www.gartner.com/document/code/318340?
In summary, EDA is a widely applicable and powerful architecture than can be applied at many levels of abstraction and in many scenarios.
It is most often a complement to more familiar service-oriented approaches to application architecture and brings both benets and new
challenges.
Strengths
The benets of using middleware that supports EDA patterns align with the demands of many digital business applications. The main
strengths are:
Minimal coupling of service components By keeping event producers and consumers completely decoupled, they can be developed,
deployed, changed, enhanced and evolved on independent life cycles. Using middleware to broker the subscriptions and delivery of
events between the producer and any subscribed consumers ensures that there is a separation of concerns between these roles. The
producers and consumers can also focus on the functionality that generates events or is triggered by them.
Ability to handle concurrency more consistently Concurrency in distributed systems is one of the additional challenges that
application architects must tackle. If data is shared and mutable, then challenges around the handling of concurrent updates arise. EDA
helps to address this because events are immutable they record something that has already happened and cannot be changed. An
ordered stream of events also can be processed concurrently by multiple subscribers without coordination. However, the system must
be able to cope with eventual rather than immediate consistency when events trigger changes in state.
Playback testing, debugging and diagnostics Recording the stream of events processed by an event broker provides a powerful tool
for debugging and diagnosing issues. Event streams can be replayed to test and debug new versions of consumers, and diagnose the
behavior of the system.
Scalability of application processing The competing consumers pattern can be used to simply and effectively scale out and distribute
the processing of events. Additionally, the asynchronous nature of EDA means that producers do not need to block waiting for a reply
once they know that their event is sure to be delivered. They can simply subscribe to a topic that will notify them when a processing has
completed or failed (which are just more events). When supporting front-end services, this nonblocking asynchronous processing
improves resource utilization and scalability.
Extensibility of applications The minimal coupling of producers and consumers means that extending an application can be done
both dynamically and with minimal regression risk to the rest of the system. A new piece of functionality that adds some processing for
a specic event can be implemented as a new subscribing consumer and deployed without changing existing components or
subscriptions. It is simply another destination for the middleware to deliver the relevant events.
Agility in deployment and development Disciplined design of event-driven interfaces and APIs decouples event producers and
consumers. In the microservice architecture use case, the goal is to increase delivery cadence and application scalability by keeping
services as loosely coupled from each other as possible. The minimal coupling of EDA is a powerful tool to achieve this agility, but
loosely coupled request-response interfaces should not be overlooked when their simplicity makes sense.
Resilience can be engineered in The asynchronous nature of EDA means that systems will be more resilient to variations in
performance or outages of individual components. With strong engineering discipline and the right choice and conguration of
middleware, it is possible to ensure that events are not lost and that systems can self-heal when outages or issues occur.
Maturity of technology EDA is a mature and established pattern that permeates technology and has for decades. Much of the
technology is battle-tested and proven over years of use in very specic scenarios and environments. None of this is new. What is new
is the nature of the "average" system we build today. The digital business applications demanded by today's users, markets and
ecosystems will all be, at least in part, event-driven. EDA is a pattern that every architect must understand and evaluate as they design
new systems.
Weaknesses
Delivering systems using EDA and the middleware that supports it are not without challenges and pitfalls. The main weaknesses of this
approach are:
Complexity of design and operations Event-driven systems are distributed, asynchronous and open-ended. They have many moving
parts, and those parts are loosely or minimally coupled. They must also work in harmony to solve the problem for which they were
designed. They are open-ended in the sense that subscribers to events can be added or removed independently, dynamically changing
the scope of the system. This means that systems using EDA tend to be more complex and harder to understand as a whole than their
more familiar request-response-based equivalents. Much of the middleware available today is mature and very powerful, but with it
often comes complexity and learning curves that new users often have little time for it is easier to revert to what you know.
Debugging, tracing and event correlation While the fact that event streams can be captured and replayed is a strength, in event-driven
architectures, the outcomes of a system are often the result of a chain of events. This means that one event triggers some processing
that results in one or more events that trigger yet more processing, and so on. To monitor, debug or diagnose an overall process
execution, you need to be able to correlate these chains of events and view them as one virtual "path" through the system. This is
analogous to the call stack or call graph in a request-response architecture. The tools and techniques to solve this problem are still
maturing, and you may need to create a custom solution to meet your needs. Example technologies include OpenTracing
(http://opentracing.io/) , Zipkin (http://zipkin.io/) , AWS X-Ray (https://aws.amazon.com/xray/) (in preview at the time of this writing)
and Event Tracing for Windows (https://blogs.msdn.microsoft.com/cloud_solution_architect/2016/10/06/distributed-tracing-in-service-
fabric-using-application-insights/) .
Unexpected consequences The choreography of events in an EDA is not centrally dened. It is organic and depends on the
operational state of the system rather than the structure of application code. This means that there can be unexpected consequences
when unexpected chains of events take place for example, triggering a race condition. Disciplined testing, monitoring and operational
controls need to be in place to make sure that these situations are minimized and avoided as much as possible, and handled effectively
when they do occur.
Limited standardization As noted earlier in this research, there is no standard for dening an event-driven API. There are multiple
protocols that support EDA, and there are many different classes of technology and middleware that can be used as part of a system
built using EDA. Add to this the use of the EDA pattern in a plethora of distinct solutions, including user interfaces, microservices, the
internals of application runtimes and enterprise integration. With this variety, it is no wonder there is little standardization. Distilling the
world of EDA to nd the best path for your specic requirements can be challenging.
Lack of runtime governance tools for EDA API gateways are a critical component in request-response-based distributed systems,
whether public APIs, enterprise integration or microservice architecture. The API gateway provides a control plane that not only
enforces encapsulation of the API implementation, but also centralizes management of policy-based controls for security,
authentication, trafc management and resource consumption. At present, there is limited support for EDA protocols and patterns
among API gateway products, and no clear equivalent in the middleware market. You must ensure that you are consistently applying
policy-based controls within your chosen middleware (if it supports them) or within your service implementations.
Lack of developer experience, training and learning resources There is an imbalance between the ubiquity of use cases for EDA of all
kinds and the material for developers and architects to learn about it. Of the few books on the topic, most are now out of print. Few
public resources are available to provide foundational advice on how to design and build event-driven systems. There is also little
evidence that EDA is being taught in university undergraduate courses. Developers and computer scientists are learning object-oriented
and sometimes functional programming. This familiarity with request-response programming models makes them much more
comfortable dealing with REST and RPC-style APIs.
Guidance
Get a leap on your competitors by learning EDA as part of a foundational architecture and design skill set. EDA is a necessary skill, but not
all developers and architects know how to use it effectively to design and build enterprise-class business solutions. By adding EDA to your
teams' capabilities so they can use it effectively, you give your organization the opportunity to get ahead of competitors that have not
realized its impact. You also stand a better chance of keeping pace with competitors that have adopted this architecture.
Identify event-driven business patterns that can drive EDA. When designing systems, we recommend that your analysis begins by looking
for event-driven patterns in the problem domain and the business processes you must support. A business process that is naturally event-
driven is a much better candidate for leveraging EDA. Supporting such a process without using EDA may introduce unnecessary
constraints on business operations. SaaS business applications and tools are increasingly offering event-driven interfaces that can help
support event-driven business processes with the right architecture in place.
Use EDA in combination with SOA. Don't assume that EDA will replace more common request-response models. Not every application
interaction, business process or transaction is a good t for EDA. Use EDA where decoupling for agility and scalability are critical, and the
ability to implement reactive 17 (#dv_17_the_reactive) architecture will give your systems competitive advantage. For example, consider
augmenting a RESTful API with support for webhooks or web-socket-based events for features that will benet from pushing data to your
API consumers. Don't try to replace REST with event-driven APIs.
Microservice outer architecture 18 (#dv_18_assessing_microservices) should include an event broker. When building microservice
architecture, you must prepare to support event-driven communication. You may not need it initially, but it is highly likely that, as the
number of services grows, you will want to support event-driven service interactions. This requires shared middleware services accessible
to all microservices in your environment. Also recognize that some events will need to be exposed to or received from external
sources/sinks. You can abstract this through REST APIs or expose event-driven APIs.
Dene design standards and guidelines for event structure, payloads, protocols and queue/topic denition. Due to the lack of
standardization in the marketplace, architects must take the lead. They must provide guidance to the teams they are working with on how
to design events and topic structures as well as the payload formats, protocols and middleware to use. Failure to do so will lead to
duplicate efforts, as teams work this out for themselves, as well as incompatible and inconsistent design decisions.
The Details
EDA and Application Architectures for Digital Business
EDA is centered on the concept of events. An event is something that has happened or is thought to happen. Every action of your
customers or users triggers events. Everything that happens in the environment around us represents an event. Everything that happens in
a business process or system represents an event. Examples of events vary widely and include bank transactions, stock trades, customer
orders, address changes, shipment deliveries, birthdays, moon landings, the Great Depression and anything that occurs in a simulation or
dream. Events may be instantaneous, or they may happen over time.
The more we digitize processes and devices, the more events we capture, communicate and process. Events have increasing value, letting
us understand and learn from the behavior in a system (see "Preparing and Architecting for Machine Learning"
(https://www.gartner.com/document/code/317328?ref=grbody&refval=3655417&latest=true) ). Events will be central to the operation and
understanding of every critical digital business system.
Data about an event can be recorded in an event object. (An event object may be called an "event," which overloads the term and
sometimes leads to confusion.) Event objects exist in many forms, such as a JSON object describing an alert from a smart thermostat; an
XML document containing a customer order; an email conrmation of an airline reservation; a database row containing a new customer
address; data from an RFID sensor reading; or a nancial data feed message that reports an equity trade. An event object generally
includes some attributes of the event, one or more time stamps, and, sometimes, an identier of the event source and related context data.
If the event is instantaneous, then there may be one time stamp noting when it happened; otherwise, there may be separate time stamps
for the beginning and end of the event. There may be yet another time stamp for when the event object was created or sent. When an event
object is transmitted in the form of a message, the combination is called an "event notication" or "notication."
EDA is dened as an architectural style in which one or more components of a system execute in response to receiving one or more event
notications. The ow of work through the system is determined, in part, by transmitting notications. EDA applications must implement
these three principles:
Notications are pushed by the event source, not pulled from the source by the event consumer. In other words, the event source
determines when the message that contains the event object is sent.
The arrival of a notication causes the event consumer to act immediately. In some cases, the action is merely to save the event object
for subsequent processing. The consumer is waiting for a notication and is driven to do something by its arrival.
A notication does not specify what action the event consumer will perform; it is a report, not a request. The consumer contains the
logic that determines how it will respond. In this respect, notications are fundamentally different from procedure calls or method
invocations, which explicitly specify the particular functions that the message recipient will perform.
WhyDigitalBusinessDemandsEDA
Application architects have long recognized the power of events. It is not surprising that when we look at information systems and their
architectures and designs, we see these patterns being replicated and codied stretching back many years from interrupts to actors, and
from UI frameworks to message brokers. Event-driven patterns and implementations are not new (see Figure 11). Events and the
technology that supports them are diverse. They are often viewed as "implementation details" rather than being at the center of how
enterprises design information systems. However, this has changed, driven by the elevation of digitization from the scope of individual
applications to entire enterprises and business ecosystems.
Figure11.MilestonesintheHistoryofEventDrivenArchitecture
19
TCAM (#dv_19_os_tcam)
The demands of digital business mean that event-driven architectures and patterns are now needed to solve problems that are much more
visible to business stakeholders and users.
Digital business solutions:
Need to respond to complex combinations of events rather than offer just a point-in-time view of the application state.
Are inherently distributed, demanding resilient and exible communication patterns.
Must give the end users the expected reactive user experiences providing near-real-time feedback and notication.
Demand that the systems are able to adapt to support innovation and scale as they evolve and mature.
CombiningEDAandServiceOrientation
As a general principle, architectural styles often are composable. In other words, multiple architectural styles can apply simultaneously to
different aspects of one system. For example, an application may use relational data architecture, object-oriented architecture and model-
driven architecture all at once because these styles are not mutually exclusive.
This principle applies because service-oriented architectures (including web APIs, REST and microservice architecture) and EDA are
complementary and composable. Most new applications that use EDA with business or domain events 20 (#dv_20_domain_event) should
be implemented as SOA simply because SOA is helpful in most distributed software systems. The key qualier is the term "business" or
"domain." A business event is an event that is relevant to the business. It is a meaningful change in something in the company or related to
its activities. A domain event impacts the state of the system or application as a whole. Examples of these events include bank
transactions, stock trades, customer orders, address changes and shipment deliveries. These events typically are of the right level of
granularity to be implemented as SOA service interfaces. An EDA application is an SOA application when it implements all ve principles in
the denition of SOA:
Modular: An EDA system always has two or more components (usually dozens), including at least one component that acts as an event
source and another that acts as an event consumer.
Distributable: EDA components often are designed to run on disparate infrastructure and communicate with each other by sending
notications over a network at runtime. However, not all EDA systems are distributable; thus, not all EDA systems use SOA.
Dened interfaces: The schema of event objects and other aspects of the notication communication (for example, topic denitions)
should be documented using interface metadata to facilitate application maintenance and expansion. (This is for the same reason that
request-response-based SOA applications use interface denitions.)
Loosely coupled: An event source, which is the provider component in an EDA relationship, can be swapped out for another event
source implementation. This is because notications are dened separately from the event source and component's internals.
Shareable: Event sources are usable by multiple, disparate event consumers in the sense that each emitted event notication can be
delivered to multiple disparate event consumers (for example, using event broker middleware to implement the publish-and-subscribe
pattern).
For further details on the relationship between EDA and SOA, see "Tutorial for EDA and How It Relates to SOA."
(https://www.gartner.com/document/code/155163?ref=grbody&refval=3655417)
The Patterns of EDA

Implementing EDA using middleware is based on four complementary communication patterns. When used together, these patterns deliver
the semantics for message delivery that allows both logical and temporal decoupling of components:
Fire and forget: This describes the simplest possible conversation pattern 21 (#dv_21_conversations_between) between a producer and
a consumer. In this pattern, the event (or message) producer has no knowledge of the consumers that may ultimately process the
event. It also has no expectation of a response. The fact that there is no response to an event is the foundation of EDA.
Publish-subscribe: This pattern builds upon the re-and-forget pattern, but allows multiple consumers to subscribe to receive events.
When an event is produced, each subscribed consumer is notied of the event.
Point to point: In this pattern, a client sends a message that must be processed by one and only one consumer. This is often mistakenly
assumed to be a one-to-one relationship between producer and consumer; however, there may be multiple consumers subscribed of
which only one can receive the message. In this case, the pattern is analogous to the anycast 22 (#dv_22_operation_of) model of
network addressing.
Shared subscriptions: In a typical resilient and scalable EDA, we need a combination of publish-subscribe and point-to-point semantics.
We need multiple types of consumers to subscribe, but we also need to ensure that only one instance of each type receives a copy of
the event. The shared subscriptions pattern solves this problem. Consumers of the same type share a subscription to the events they
are interested in, and only one instance of each consumer type processes those events (for example, using a competing consumers
pattern). Some middleware supports this pattern natively (for example, shared subscriptions in JMS 2.0, 23 (#dv_23_java_message)
RabbitMQ nonexclusive named consumer queues, Microsoft Azure Event Hub consumer groups 24 (#dv_24_event_consumers) and
Microsoft Azure Service Bus topic subscriptions 25 (#dv_25_azuresamples_github) ).
Gartner Recommended Reading

"Assessing Event-Driven Architecture for Scalable and Reactive Web Applications" (https://www.gartner.com/document/code/310108?
ref=ggrec&refval=3655417&latest=true)
"Assessing Integration Architecture for Internet of Things Solutions" (https://www.gartner.com/document/code/297261?

"Assessing Microservices for Cloud-Native Application Architecture and Delivery" (https://www.gartner.com/document/code/297262?

"How to Design Microservices for Agile Architecture" (https://www.gartner.com/document/code/319890?

"How to Architect and Design Cloud-Native Applications" (https://www.gartner.com/document/code/296114?

"A Guidance Framework for Designing a Great Web API" (https://www.gartner.com/document/code/300000?

"Adding Serverless Computing and fPaaS to Your Cloud-Native Architecture Toolbox" (https://www.gartner.com/document/code/318340?
"Hyperscaling Analytics: Comparing Streaming Analytics in the Cloud With AWS, Microsoft Azure and IBM"
(https://www.gartner.com/document/code/310019?ref=ggrec&refval=3655417&latest=true)
"2017 Planning Guide for Application Platform Strategies" (https://www.gartner.com/document/code/313640?

Additional Gartner Recommended Reading Outside Your Current Subscriptions
"CTO Alert: Master Event-Driven IT to Master Digital Business" (https://www.gartner.com/document/code/308964?ref=ggrec&refval=3655417)
"Follow the Leaders: Digital Business Innovation Is Event-Driven" (https://www.gartner.com/document/code/323880?ref=ggrec&refval=3655417)
Evidence
1
K. Chandy and W. Schulte, "Event Processing: Designing IT Systems for Agile Companies," McGraw-Hill Education, 2010.
2
"STOMP," (http://stomp.github.io/) GitHub.
3
"IBM MQ, Version 9.0 Message Ordering,"
(https://www.ibm.com/support/knowledgecenter/SSFKSJ_9.0.0/com.ibm.mq.dev.doc/q026660_.htm) IBM.
4
"ActiveMQ How Do I Preserve Order of Messages," (http://activemq.apache.org/how-do-i-preserve-order-of-messages.html) the Apache
Software Foundation.
5
"RabbitMQ Broker Semantics," (https://www.rabbitmq.com/semantics.html) Pivotal.
6
"Distributed Transaction Processing: The XA Specication," (http://pubs.opengroup.org/onlinepubs/009680699/toc.pdf) X/Open Co.
7
"Akka Notes Introducing Actors," (http://rerun.me/2014/09/11/introducing-actors-akka-notes-part-1/) Rerun.me.
8
"Distributed Publish Subscribe in Cluster," (http://doc.akka.io/docs/akka/current/scala/distributed-pub-sub.html) Akka.
9
"Event Processing," (http://www.axonframework.org/docs/2.4/event-processing.html#distributed-event-bus) Axon Framework.
10
"Choosing Application Integration Technology." (https://www.gartner.com/document/code/314871?
11
"Callback Feature," (https://github.com/OAI/OpenAPI-Specication/pull/763) GitHub.
12
"Google Cloud Pub/Sub API," (https://cloud.google.com/pubsub/docs/reference/rpc/) Google Cloud Platform.
13
"Cloud Endpoints for gRPC APIs," (https://cloud.google.com/endpoints/docs/grpc) Google Cloud Platform.
14
"Machine Learning Powered Search as a Service," (https://www.sajari.com/) Sajari.
15
"Comet: Low Latency Data for the Browser," (http://infrequently.org/2006/03/comet-low-latency-data-for-the-browser/) Infrequently
Noted.
16
"Server-Sent Events," (https://html.spec.whatwg.org/multipage/comms.html#server-sent-events) HTML.
17
"The Reactive Manifesto," (http://reactivemanifesto.org/) The Reactive Manifesto.
18
18
"Assessing Microservices for Cloud-Native Application Architecture and Delivery." (https://www.gartner.com/document/code/297262?
19
"OS TCAM Concepts and Facilities," (http://bitsavers.trailing-edge.com/pdf/ibm/360/os/tcam/GC30-2022-
2_OS_TCAM_Concepts_and_Facilities_Rel_21.6_Jan73.pdf) IBM.
20
"Domain Event," (https://martinfowler.com/eaaDev/DomainEvent.html) Martin Fowler.
21
"Conversations Between Loosely Coupled Systems," (http://www.enterpriseintegrationpatterns.com/patterns/conversation/index.html)
Enterprise Integration Patterns.
22
"Operation of Anycast Services," (https://tools.ietf.org/pdf/rfc4786.pdf) Abley and Lindquist.
23
" Java Message Service," (http://download.oracle.com/otn-pub/jcp/jms-2_0-fr-eval-spec/JMS20.pdf) Oracle.
24
"Event consumers," (https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-what-is-event-hubs#event-consumers) Microsoft.
25
"Azure-Samples," (https://github.com/Azure-Samples/azure-servicebus-messaging-samples/tree/master/PrioritySubscriptions) GitHub.
2017 Gartner, Inc. and/or its afliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. or its afliates. This publication may
not be reproduced or distributed in any form without Gartner's prior written permission. If you are authorized to access this publication, your use of it
is subject to the Usage Guidelines for Gartner Services (https://www.gartner.com/technology/about/policies/usage_guidelines.jsp) posted on
gartner.com. The information contained in this publication has been obtained from sources believed to be reliable. Gartner disclaims all warranties
as to the accuracy, completeness or adequacy of such information and shall have no liability for errors, omissions or inadequacies in such
information. This publication consists of the opinions of Gartner's research organization and should not be construed as statements of fact. The
opinions expressed herein are subject to change without notice. Although Gartner research may include a discussion of related legal issues, Gartner
does not provide legal advice or services and its research should not be construed or used as such. Gartner is a public company, and its
shareholders may include rms and funds that have nancial interests in entities covered in Gartner research. Gartner's Board of Directors may
include senior managers of these rms or funds. Gartner research is produced independently by its research organization without input or inuence
from these rms, funds or their managers. For further information on the independence and integrity of Gartner research, see "Guiding Principles on
Independence and Objectivity. (https://www.gartner.com/technology/about/ombudsman/omb_guide2.jsp) "

Assessing Event-Driven Middleware Technology For Modern Application Architecture

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Assessing Event-Driven Middleware Technology For Modern Application Architecture

Încărcat de

Drepturi de autor:

Formate disponibile

VIEWKEYINSIGHTS(HTTPS://WWW.GARTNER.COM/DOCUMENT/3655417/KEYINSIGHT?

Assessing Event-Driven Middleware Technology for Modern

Interactions are only driven by the client.

Source: Gartner (March 2017)

Microservice architecture (see "How to Design Microservices for Agile Architecture"

Lambda/Kappa architecture (see "Harness Streaming Data for Real-Time Analytics"

In each of these architectures, the role of events is not new; it is amplied.

Middleware Objectives for EventDriven Architecture

Application integration using event-driven architecture to support application-to-application integration scenarios.

IoT "Assessing Integration Architecture for Internet of Things Solutions" (https://www.gartner.com/document/code/297261?

Event Broker Middleware for EventDriven Solutions

Examples of products and technologies in this category include:

Source: Gartner (March 2017)

AMQP v.0.9.1: This is an open, interoperable wire protocol specication (https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf)

Examples of products and technologies in this category include:

Source: Gartner (March 2017)

Source: Gartner (March 2017)

Examples of products and technologies in this category include:

Examples of products and tools in this category include:

Examples of frameworks in this category include:

Examples of platforms in this category include:

Amazon Web Services IoT (https://aws.amazon.com/iot-platform/) , AT&T M2X (https://m2x.att.com/) , GE Predix

Three Application Architecture Use Cases

Application integration using event-driven architecture to support application-to-application integration scenarios.

MAPPING MIDDLEWARE TO THE MICROSERVICE USE CASE

Event Stream Platform

Software Use for implementing event sourcing.

Subscription-Enabled Data Stores

Source: Gartner (March 2017)

Source: Gartner (March 2017)

Software An effective and proven solution.

Event Stream Platform

Subscription-Enabled Data Stores

Source: Gartner (March 2017)

Source: Gartner (March 2017)

PAYLOAD FORMATS FOR EVENTDRIVEN APIS

PROTOCOL CHOICES FOR EVENTDRIVEN APIS

MQTT v.3.1.1 This protocol supports lightweight, topic-oriented publish-subscribe messaging.

Table 3 provides a more detailed comparison of these options.

Webhooks only medium mechanisms (https://blogs.msdn.microsoft.com/webdev/2016/12/14/introducing-

Persistent implementations HTTP and frameworks

data supported (text or mechanisms NET 4.5 and later (https://msdn.microsoft.com/en-

Persistent payload) Node.js ws package (https://www.npmjs.com/package/ws)

transport; payload, can control Certicate libraries)

connection TLS, TCP or

transport; focused) binary control Node.js amqp10 package

transport; focused) binary based, with (https://www.npmjs.com/package/amqplib)

Source: Gartner (March 2017)

MAPPING MIDDLEWARE TO THE EVENTDRIVEN API USE CASE

Event Stream Platform

Software Not suitable. Primarily useful for managing inbound streams.

Cloud Not suitable. Primarily for managing inbound streams.

Subscription-Enabled Data Stores

Source: Gartner (March 2017)

Digital business solutions:

Are inherently distributed, demanding resilient and exible communication patterns.

The Patterns of EDA

Gartner Recommended Reading

"Assessing Integration Architecture for Internet of Things Solutions" (https://www.gartner.com/document/code/297261?

"Assessing Microservices for Cloud-Native Application Architecture and Delivery" (https://www.gartner.com/document/code/297262?

"How to Design Microservices for Agile Architecture" (https://www.gartner.com/document/code/319890?

"How to Architect and Design Cloud-Native Applications" (https://www.gartner.com/document/code/296114?

"A Guidance Framework for Designing a Great Web API" (https://www.gartner.com/document/code/300000?