Ships

at Sea
Hybrid Cloud Integration Session HHM-6451
IBM InterConnect 2017 Conference

Gary Gershon
Intermedia Sciences LLC
Gary@IntermediaSciences.com
908-969-1119
Agenda
A case study in hybrid cloud architecture with analytics, IoT, and reactive
programming

• Project Goals
• Problem Domain
• Hybrid Cloud Decisions
• Analytics
• IoT MQTT
• Akka Reactive
• Other Takeaways

Intermedia Sciences - Ships at Sea 2

Intermedia Sciences LLC
Architecture and design consultancy

• Enterprise architectures
• Application designs
• Cloud, Big Data, and messaging
technologies
• POC and team mentoring
Principal – Gary Gershon
• New York based
Senior Architect and Data Scientist
Prior VP Application Architecture – Chubb
Group of Insurance Companies

Intermedia Sciences - Ships at Sea 3

Project Goals
All the goals have been prototyped, with iterations in production

• Create a distributed platform for

processing vessel navigation
data
• Implement predictive
applications of these models
• Enable interesting mobile, social,
and voice-text applications
• Prototype a scalable cloud
platform for 100x vessels

Intermedia Sciences - Ships at Sea 4

Problem Domain
For 15 years, large ships and yachts have broadcast their position, course, speed, and other
information as part of the mandatory global Advanced Identification System (AIS)

• Data is captured by land stations

and send to distribution hubs
• About 250K AIS equipped
vessels worldwide
• Thousands of messages per
day/vessel
• Future technology expansion
plans

Intermedia Sciences - Ships at Sea 5

Hybrid Cloud Perspectives
Hybrid cloud is now well established with patterns and products to extend
enterprise systems of record to the cloud to enable Digital IT solutions
• Hybrid Architecture – On Premises
• Most often legacy systems-of-record
integration
• Integrated with diverse cloud resources,
including cloud-to-cloud
• Hybrid Architecture – Cloud-to-Cloud
• Cloud-to-Cloud integration – applications,
storage, big data analytics
• Utilities – logging, notifications, monitoring
• Today’s Cloud App may be tomorrow’s legacy
premises equivalent
• Enterprise Hybrid Pattern would typically
include:
• IBM Secure Gateway
• IBM Integration Bus (in cloud or on-premises)
• IBM API Gateway
• IBM MQ
Intermedia Sciences - Ships at Sea 6
Ships at Sea Hybrid Cloud Topology
On-premises and legacy cloud migrating new development to hybrid multi-cloud
Legacy
Mobile Apps Mobile Apps Future Apps
Mobile Apps

Legacy Cloud
Filter – Route MQTT MQTT
Transform - Persist
Bluemix

On-Premises
Vessel Messages & Metadata IBM Message Hub
Station Data

IBM DSE Analytics

Intermedia Sciences - Ships at Sea 7

Bluemix Differentiators & Decisions
Our programming skills and preferences, combined with
anticipated future integrations, fit well with Bluemix
• IBM Guidance and Assistance • Mobile
• Architectural assets and sessions • MobileFirst
• Integration and Hybrid Experience • Cloudant (mobile sync)
• Global Presence • API Management
• Data Science • IBM API Gateway
• Spark • StrongLoop (Node.js)
• IBM Data Science Experience • Future JavaScript clients
• Jupyter Notebooks • Core Components for Bluemix
• Cognitive • IBM Containers (Docker)
• IBM Cognitive Watson Services • IBM Message Hub (Kafka)
• Java. JEE and Swift Skills • Spark Analytics (on IBM DSE)
• Akka Reactive Technology (microservices)
• Liberty Profile -> WAS (microservices)
• IBM Watson Services
• Server Side Swift
• Scala and Java
• Messaging
• IBM Message Hub -> MQ light (AMQP)
• Hybrid Integrated with Legacy Systems
• Global data center message hub
• MQTT -> IBM Message Sight
• Ship data registry
• Multi-cloud message ingestion and persistence

Intermedia Sciences - Ships at Sea 8

Hybrid Reference Architecture - Layering
A hybrid cloud would typically have two API layers – one for SOR and one for SOE

• Assembling Systems of Engagement for

Digital IT using:
• Proximity Data
• Data Synchronization
• Likely REST/JSON
• Messaging via AMQP and MQTT

• Empowering Digital IT using:

• SOA/ESB/Integration Bus
• SOR adapters, engine to integrate
and transform data formats and
protocols
• Likely SOAP/XML & Proprietary
• Messaging via MQ
• Key Concepts:
• Two layers of API services
• Top SOE layer contains duplicated
data coming from and going to
bottom SOR layer
© 2016 IBM • API layers likely to use differing
data formats and protocols

Intermedia Sciences - Ships at Sea 9

Hybrid Reference Architecture - Products
The hybrid layers are enabled by prior and new products

• Assembling SOEs for Digital IT

• IBM Liberty (Java) or Node.js
• Bluemix Containers
• IBM MQ Light (AMQP) Message Hub
• IBM Message Sight (MQTT)
• IBM API Connect
• IBM DataPower Gateway
• Proximity Data
• IBM Cloudant Data Synchronization

• Empowering Digital IT
• IBM Integration Bus (IIB)
• IBM MQ
• Bluemix Secure Gateway
• IBM DataPower API Gateway
• Proximity Data
• Your favorite NoSQL DB
© 2016 IBM
• In-memory DB (e.g. Redis)
• In-memory Actors (Akka)
• Caching or event sourcing persistence
• Synchronization via messaging and ETL
Intermedia Sciences - Ships at Sea 10
Target Architecture
The Ships at Sea architecture has similar layering to Enterprise Systems

API – MQTT API – Akka IBM Message

API – REST/HTTP
Stream Message Hub
API
Kafka Real-Time
Akka Actors Transform - Spark Kafka Publish Spark
Kafka Scala
Persist - Spark Persist - Spark Kafka Persist
Service Client
API API – MQTT Stream API – REST/HTTP
Lambda Architecture
IoT Optimize Enqueue Physics

Ingest via MQTT Parse and Filter Fetch and Persist

IoT Vessels AISHub VHF

SOR Vessel Repository
Message Stream Message Stream
Intermedia Sciences - Ships at Sea 11
Lambda Architecture - Analytics
The Lambda Architecture supports big-data analytics for both batch and real-time
needs – implemented using IBM Message Hub and Spark

Object Store – Hourly Archive (Gzip)

Java Sub

MQTT Spark

Java Sub

Models
© 2014 MapR Technologies
IBM Message Hub (Kafka) - Spark

Intermedia Sciences - Ships at Sea 12

Message Hub (Kafka)
IBM Message Hub provides a logging message ingestion and distribution hub.
Effective for data analytics with Spark as well as MQ Light worker models

• 300,000 msg/sec
• 24 hour persistence
• Subscribers
• Real time Analytics
• Persistence
• Pub-Sub
• Multiple subscribers per
message, or,
• One subscriber per
© 2016 IBM message

Intermedia Sciences - Ships at Sea 13

MQTT Two-way Message Bus
MQTT supports a flexible publish-subscribe bus which facilitates transformations as
well as very-efficient collection of vessel data by suppressing redundant transfers

• Encoded vessel station

messages are received by Stations MQTT AIS Hub
MQTT, transformed, and
republished as new topics. Message Hub
Validate
(Kafka)
• Smart IoT stations (Linux-
based) significantly optimize Message
bandwidth usage by Suppression Persistence
suppressing messages from
vessels that have already been Transform and Physics &
Routing
received from other stations. Filter Prediction

Intermedia Sciences - Ships at Sea 14

Akka – In Memory Domain Services
Akka is an open source framework which supports in-memory data management
and both REST and queued-message services for fast response times

• Runs in Docker Container

• Programmed in Scala and Java
• Non-blocking, multi-threaded
and horizontally scalable
• Based on Actors and Messages
• Highly reliable – threading is
fully implemented by Akka, not
by programmer © 2014 Richard Doyle

Intermedia Sciences - Ships at Sea 15

Akka Domain Models
Each vessel and geographic tile has its own actor. These actors have functions,
methods, and hold data – and safely invoke other actors via messages

HTTP REST API Remote Message API

Actor Message Management (configureable)

All Vessels Actor All Tiles (Geo) Actor Support Actors

Vessel 01 Actor Vessel 02 Actor Tile AA Actor Tile AB Actor Services Actors
Vessel 03 Actor Vessel 04 Actor Tile AC Actor Tile AD Actor

Intermedia Sciences - Ships at Sea 16

Architectural Design Notes
Some takeaways from implementation experience

Persistence Messaging Transformations

• Facilitate data evolution • Employ messaging at • Transform data in pipelines

with NoSQL and JSON multiple layers for solving multiple needs
• Fully embrace eventual • Deliver availability and • Use both dataset and
consistency scaling via asynchronous message stream pipes
• Utilize multiple and worker patterns • Chaining sequences of
synchronization patterns • Enable evolution and transformations is a core
• Explore Event Streaming flexibility via Pub-Sub concept for Spark analytics
and in-memory data for • Mix technologies as needed: • Rapidly deploy new
performance MQ – MQTT – MQ Light transformations via pub-sub
(AMQP) – Message Hub - message streams
Akka

Intermedia Sciences - Ships at Sea 17

DevOps Implementation Notes
A multi-cloud implementation greatly benefits from a DevOps mindset

Scripts Monitoring Logging

• Embrace DevOps for • Employ internet monitoring to • Utilize one of many available
productivity and availability isolate code and performance internet logging sites for rapid
• Develop task scripts to restart issues analysis
containers, etc. – quickly and • Recognize value of graphs and • Recognize that when things fail,
with repeatability alerts your servers may be unavailable
• Employ scripts to verify • Invoke monitoring APIs directly to retrieve logs
connections (ping), queue from application code as well as • Install simple extensions to
depth, and queue flow monitor containers and servers frameworks like log4j to write
• Create synthetic transaction • Create metrics based on to remote logging systems
scripts to verify entire business activity to recognize • Employ alerts based on
subsystems are working stalls centralized log scanning

Intermedia Sciences - Ships at Sea 18

Thank you!

Gary Gershon
Intermedia Sciences LLC
Gary@IntermediaSciences.com
908-969-1119

Intermedia Sciences - Ships at Sea 19