Native Docker Clustering with Swarm
()
About this ebook
Related to Native Docker Clustering with Swarm
Related ebooks
Learning Docker Rating: 5 out of 5 stars5/5Learning Docker - Second Edition Rating: 0 out of 5 stars0 ratingsDeveloping with Docker Rating: 5 out of 5 stars5/5DevOps. How to build pipelines with Jenkins, Docker container, AWS ECS, JDK 11, git and maven 3? Rating: 0 out of 5 stars0 ratingsNginx Essentials Rating: 0 out of 5 stars0 ratingsLearn Kubernetes - Container orchestration using Docker: Learn Collection Rating: 4 out of 5 stars4/5DevOps for Networking Rating: 0 out of 5 stars0 ratingsGetting Started with Terraform Rating: 5 out of 5 stars5/5Mastering OpenStack Rating: 1 out of 5 stars1/5Extending Docker Rating: 0 out of 5 stars0 ratingsMonitoring Docker Rating: 0 out of 5 stars0 ratingsDocker Orchestration Rating: 0 out of 5 stars0 ratingsLearn Kubernetes & Docker - .NET Core, Java, Node.JS, PHP or Python Rating: 0 out of 5 stars0 ratingsHands-On Microservices with Kubernetes: Build, deploy, and manage scalable microservices on Kubernetes Rating: 5 out of 5 stars5/5Infrastructure as Code (IAC) Cookbook Rating: 0 out of 5 stars0 ratingsDocker Deep Dive: Zero to Docker in a single book Rating: 0 out of 5 stars0 ratingsHybrid Cloud Management with Red Hat CloudForms Rating: 0 out of 5 stars0 ratingsKubernetes Native Microservices with Quarkus and MicroProfile Rating: 0 out of 5 stars0 ratingsKubernetes Handbook: Non-Programmer's Guide to Deploy Applications with Kubernetes Rating: 4 out of 5 stars4/5Learn Docker - .NET Core, Java, Node.JS, PHP or Python: Learn Collection Rating: 5 out of 5 stars5/5DevOps and Containers Security: Security and Monitoring in Docker Containers Rating: 0 out of 5 stars0 ratingsLearning Ansible 2 - Second Edition Rating: 5 out of 5 stars5/5OpenStack Orchestration Rating: 5 out of 5 stars5/5Infrastructure as Code, Patterns and Practices: With examples in Python and Terraform Rating: 0 out of 5 stars0 ratingsContainerization with LXC Rating: 0 out of 5 stars0 ratingsAnsible For Containers and Kubernetes By Examples Rating: 0 out of 5 stars0 ratingsLearning RHEL Networking Rating: 0 out of 5 stars0 ratingsPipeline as Code: Continuous Delivery with Jenkins, Kubernetes, and Terraform Rating: 3 out of 5 stars3/5
System Administration For You
Practical Data Analysis Rating: 4 out of 5 stars4/5Linux Bible Rating: 0 out of 5 stars0 ratingsImprove your skills with Google Sheets: Professional training Rating: 0 out of 5 stars0 ratingsLearn PowerShell Scripting in a Month of Lunches Rating: 0 out of 5 stars0 ratingsCybersecurity: The Beginner's Guide: A comprehensive guide to getting started in cybersecurity Rating: 5 out of 5 stars5/5Linux: Learn in 24 Hours Rating: 5 out of 5 stars5/5CompTIA A+ Complete Review Guide: Core 1 Exam 220-1101 and Core 2 Exam 220-1102 Rating: 5 out of 5 stars5/5Learn Git in a Month of Lunches Rating: 0 out of 5 stars0 ratingsLinux for Beginners: Linux Command Line, Linux Programming and Linux Operating System Rating: 4 out of 5 stars4/5Ethical Hacking Rating: 4 out of 5 stars4/5Learn Windows PowerShell in a Month of Lunches Rating: 0 out of 5 stars0 ratingsLinux Command-Line Tips & Tricks Rating: 0 out of 5 stars0 ratingsOperating Systems DeMYSTiFieD Rating: 0 out of 5 stars0 ratingsBuilding a Plex Server with Raspberry Pi Rating: 0 out of 5 stars0 ratingsMastering Windows PowerShell Scripting Rating: 4 out of 5 stars4/5Bash Command Line Pro Tips Rating: 5 out of 5 stars5/5Git Essentials Rating: 4 out of 5 stars4/5Mastering Active Directory Rating: 0 out of 5 stars0 ratingsMastering Linux Shell Scripting Rating: 4 out of 5 stars4/5ConfigMgr - An Administrator's Guide to Deploying Applications using PowerShell Rating: 5 out of 5 stars5/5The Complete Powershell Training for Beginners Rating: 0 out of 5 stars0 ratingsLearn SQL Server Administration in a Month of Lunches Rating: 3 out of 5 stars3/5Bash for Fun: Bash Programming: Principles and Examples Rating: 0 out of 5 stars0 ratingsLearn PowerShell in a Month of Lunches, Fourth Edition: Covers Windows, Linux, and macOS Rating: 0 out of 5 stars0 ratingsPowerShell in Depth Rating: 0 out of 5 stars0 ratingsPowerShell: A Comprehensive Guide to Windows PowerShell Rating: 4 out of 5 stars4/5
Reviews for Native Docker Clustering with Swarm
0 ratings0 reviews
Book preview
Native Docker Clustering with Swarm - Fabrizio Soppelsa
Table of Contents
Native Docker Clustering with Swarm
Credits
About the Authors
About the Reviewer
www.PacktPub.com
Why subscribe?
Dedication
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Errata
Piracy
Questions
1. Welcome to Docker Swarm
Clustering tools and container managers
Swarm goals
Why use Swarm
Real world use case examples
Pet versus cattle models
Swarm features
Similar projects
Kubernetes
CoreOS Fleet
Apache Mesos
Kubernetes versus Fleet versus Mesos
Swarm versus all
The Swarm v1 architecture
Terminology
Getting started with Swarm
Docker for Mac
Docker for Windows
Getting ready with Linux
Check that Docker Machine is available - all systems
Swarm, yesterday
Boot2Docker
Create 4 cluster nodes with Docker Machine
Configuring the Docker hosts
Starting Docker Swarm
Test your Swarm cluster
Swarm, today
Summary
2. Discover the Discovery Services
A discovery service
Token
Re-architecting the example of Chapter 1 with token
Token limitations
Raft
Raft theory
Raft in practice
Etcd
Re architecting the example of Chapter 1 with Etcd
ZooKeeper
Consul
Re architecting the example of Chapter 1 with Consul
Towards a decentralized discovery service
Summary
3. Meeting Docker Swarm Mode
Swarmkit
Versions and support
Swarmkit architecture
How a manager chooses the best node for a task
The heart of SwarmKit: swarmd
The controller of SwarmKit: swarmctl
Provisioning a SwarmKit cluster with Ansible
Creating a service on SwarmKit
Swarm mode
Old versus new Swarm versus SwarmKit
Swarm Mode zoom in
Integration into the engine
docker swarm command
docker node
docker service
Docker Stack
Etcd's Raft is integrated already
Load balancing and DNS
Promotion and demotion
Replicas and scale
Services and tasks
docker service scale
Summary
4. Creating a Production-Grade Swarm
Tools
An HA Topology for Swarm2k
Managers specifications
In case of Raft recovery
Raft files
Running tasks
Manager topologies
Provisioning the infrastructure with belt
Securing Managers with Docker Machine
Understanding some Swarm internals
Joining workers
Upgrading Managers
Live Upgrading the Managers
Monitoring Swarm2k
InfluxDB Time-Series Database
Telegraf Swarm plugin
Swarm3k
Swarm3k Setup and Workload
Swarm performance at a scale
Swarm2k and Swarm3k lessons learned
Summary
5. Administer a Swarm Cluster
Docker Swarm standalone
Docker Swarm Mode
Manually adding nodes
Managers
Workers number
Scripted nodes addition
Belt
Use Ansible
Cluster management
Nodes operations
Demotion and promotion
Tagging nodes
Remove nodes
Remove workers
Remove managers
Swarm health
Backing up the cluster configuration
Disaster recovery
Graphical interfaces for Swarm
Shipyard
Portainer
Summary
6. Deploy Real Applications on Swarm
Microservices
Deploy a replicated nginx
A minimal Swarm
Docker service
Overlay networks
Integrated load balancing
Connecting services: A WordPress example
Swarm scheduling strategies
Now, WordPress
Docker Compose and Swarm mode
Introducing Docker stacks
Distributed Application Bundles
Docker deploy
Another app: Apache Spark
Why Spark on Docker
Spark standalone without Swarm
Spark standalone on Swarm
Spark topology
Storage
Prerequisites
Start Spark on Swarm
Summary
7. Scaling Up Your Platform
The Spark example, again
Docker plugins
The lab
A unique key
Docker Machine
Security groups
Networking configuration
Storage configuration and architecture
Installing Flocker
Generating Flocker certificates
Installing software
Installing the control node
Installing the cluster nodes
Testing whether everything is up and running
Installing and configuring Swarm
A volume for Spark
Deploying Spark, again
Testing Spark
Using Flocker storage
Scaling Spark
Monitoring Swarm hosting apps
Prometheus
Installing a monitoring system
Importing Prometheus in Grafana
Summary
8. Exploring Additional Features of Swarm
Libnetwork
Networking plugins
Container Networking Model
Encryption and routing mesh
MacVLAN
Overlay networks
Network Control Plane
Libkv
How to use libkv
Summary
9. Securing a Swarm Cluster and the Docker Software Supply Chain
Software Supply Chain
Securing Swarm cluster
Securing a Swarm: Best practices
Certification Authorities
Certificates and Mutual TLS
The join token
Add TLS with Docker Machine
Form a cluster on a private network
Docker Notary
Introducing Docker secrets
Summary
10. Swarm and the Cloud
Docker for AWS and Azure
Docker for AWS
Docker for Azure
Docker Datacenter
Swarm on OpenStack
OpenStack Nova
The (deprecated) nova-docker driver
The reality - OpenStack the friendly way
OpenStack Heat
OpenStack Magnum
Architecture and core concepts
Install HA Magnum on Mirantis OpenStack
Configure an HA Magnum installation
Create a Swarm cluster on Magnum
Summary
11. What is next?
The challenge of provisioning
Software defined infrastructure
Infrakit
TUF - The Update Framework
Docker stacks and Compose
CaaS - Containers as a Service
Unikernels
Contribute to Docker
Github
File issues
Code
Belt and other projects
Summary
Native Docker Clustering with Swarm
Native Docker Clustering with Swarm
Copyright © 2016 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
First published: December 2016
Production reference: 1151216
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham
B3 2PB, UK.
ISBN 978-1-78646-975-5
www.packtpub.com
Credits
About the Authors
Fabrizio Soppelsa works as an Escalations Engineer for Mirantis Inc., the OpenStack company. Docker activist and advocate since Docker 0.3 and author of several articles on the Docker tools in three languages, he is also a concrete contributor to the projects, especially Machine. He currently lives in Moscow, Russia, where he is the Docker Meetup organizer with his spider Mosha.
I would like to thank ClusterHQ folks for their help on Flocker, especially, Ryan Wallner. the Yandex team and Denis Kutin for having given me a free OpenStack lab on which I could easily access. Mirantis for creating the (I think) best OpenStack distribution. The Docker team and the Docker community, for all the fun.
Chanwit Kaewkasi is an Assistant Professor at the School of Computer Engineering, Suranaree University of Technology, Thailand. Chanwit started contributing to the Docker Swarm project since 0.1 where he co-designed and implemented the strategy filters, the ZooKeeper discovery, and other features. He currently serves as Docker Swarm Maintainer and Docker Captain.
I'd like to thank my wife, Pitchaya, for her encouragement and best support to everything I'm working on, including this book.
Special thanks to the Docker Engineering team for their great software, Suranaree University of Technlogy for being my great workplace, and my parents for their support and lastly, Fabrizio for inviting me to co-author this great book.
About the Reviewer
Baohua Yang is a senior researcher at IBM. His interests include key areas in Cloud Computing, Fintech, distributed system and analytics, especially on emerging technologies, e.g., SDN/NFV, Container, BigData, Blockchain and cognitive computing.
As the chief architect, he has lead the architecture design and system implementation of enterprise productions, and helps solve key technical challenges for industrial solutions.
As an Open-Source contributor, he has submitted code, proposals, and presentations to numbers of projects, e.g., OpenStack, Hyperledger, OpenvSwitch, Docker, OpenDaylight, Kubernetes, and has lead several projects including easyOVS, Hyperledger Fabric-SDK-py and Cello. He is now the chairperson of Hyperledger Technical Working Group in China.
He has published 10s of papers in top-quality international conference and journals (e.g., IEEE INFOCOM, IEEE Trans on Computers), with several technical books and patents. He is now TPC member of numbers of academic conferences and journals.
His homepage is at https://yeasy.github.com.
www.PacktPub.com
For support files and downloads related to your book, please visit www.PacktPub.com.
Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at service@packtpub.com for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.
https://www.packtpub.com/mapt
Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt books and video courses, as well as industry-leading tools to help you plan your personal development and advance your career.
Why subscribe?
Fully searchable across every book published by Packt
Copy and paste, print, and bookmark content
On demand and accessible via a web browser
Dedication
I want to dedicate this book to the memory of my father, Donato.
Preface
Welcome to Native Docker clustering with Swarm! This is a book about containers and distributed systems. We’ll show how to use the native Docker tools to model micro services, spawn tasks, scale up the size of your application, and push your containers to the Docker clustering limit! In a word, we’ll discuss Docker orchestration.
With the recent rise of Swarm Mode and the enablement of Swarm inside the Docker Engine itself, it turned out that the best way to orchestrate Docker is… Docker!
Good, but what do we mean by orchestrate Docker
? What is orchestration? Better, what is an orchestra?
An orchestra is an ensemble of musicians led by a conductor, who dictates tempo, rhythm and shapes the sound. Strings, woodwinds, percussions, keyboards and other instruments follow the conductor’s direction to perform an astonishing symphony, for example Beethoven’s The Ninth.
Similarly, in a containers orchestration system the musicians are tasks, and the conductor is a leader service (Swarm primitives). Tasks don’t play a symphony, or at least not only: More abstractly they execute some computational job, for example they run a webserver. The conductor, Swarm, is responsible for their provisioning, their availability, their linkage, their scaling. This (and more) is what we mean by Docker orchestration
.
This book shows how to provision such Docker orchestras
, how to guarantee the availability of the service, how to connect tasks and how to scale the platform, to play the symphony of your application.
What this book covers
Chapter 1, Welcome to Docker Swarm, introduces Swarm, and explains why you need a clustering solution for your containers. It illustrates the Swarm features, giving a high-level description of its architecture. We define some use cases and describe how Swarm is different from Fleet, Kubernetes and Mesos. The chapter proceeds with the Docker tools installation and finally with two Swarms provisionings: A local Swarm Standalone and a remote Swarm Mode cluster on DigitalOcean.
Chapter 2, Discover the Discovery Services, is a descriptive and mostly abstract chapter. We’ll learn what discovery mechanisms and consensus algorithms are, and why they are crucial for distributed systems. We’ll describe in detail Raft and its implementation Etcd, the consensus mechanism included in Swarm Mode. We will also show the limitations of the discovery mechanism used in Chapter 1, Welcome to Docker Swarm, by extending the local tiny example with Consul, re deploying it.
Chapter 3, Meeting Docker Swarm Mode, is about the new Docker kit that allows to create task clusters of any size. We will introduce Swarmit, the foundation of Docker Swarm Mode, showing how it works in Docker 1.12+, discuss its architecture, its concepts, how it’s different from the old
Swarm, and how it organizes workloads by abstracting services and tasks.
Chapter 4, Creating a Production-Grade Swarm, shows and discusses the community-driven projects Swarm2k and Swarm3k, our 2,300 and 4,800 nodes Swarm clusters experiments, which ran hundreds of thousands of containers. We demonstrate how such huge clusters were planned, provisioned, and summarize the lessons we learned.
Chapter 5, Administer a Swarm Cluster, is a chapter about infrastructure. We will show how to increase or decrease Swarms sizes, how to promote and demote nodes, and how to update clusters and nodes properties. We’ll introduce Shipyard and Portainer.io as graphical UIs for Swarm.
Chapter 6, Deploy Real Applications on Swarm, is where we will put real applications in motion on Swarm and where we add to the discussion some notes about Compose, Docker Stacks and Docker Application Bundles. We will show the typical deployment workflow, how to filter and schedule containers over the cluster, launch them as services, handle containers as tasks. We’ll start defining a web service with Nginx, then we’ll deploy a mandatory Wordpress with MySQL example. We’ll finally move on with a more realistic app: Apache Spark.
Chapter 7, Scale Up Your Platform, will develop new topics from the previous chapter. Here we’ll introduce Flocker to add storage capacity to Spark on Swarm, and we’ll show how to install and use it automatically at a scale in conjunction with Swarm. We’ll refine our Spark example by running some real big data jobs and setting up a basic monitoring system for this infrastructure.
Chapter 8, Exploring Additional Features to Swarm, discusses some advanced topics important to Swarm, such as Libnetwork and Libkv.
Chapter 9, Securing a Swarm Cluster and Docker Software Supply Chain, will focus on security considerations for Swarm clusters. Among the arguments, certificates, firewalling concepts for platform, and a mention to Notary.
Chapter 10, Swarm and the Cloud, is a chapter illustrating the most popular options for running Swarm on cloud providers. We’ll install Swarm on AWS and Azure before introducing the Docker Datacenter, and we’ll finally move to OpenStack showing how to install and manage Swarms on the top of Magnum, the Container as a Service solution for OpenStack.
Chapter 11, What is Next?, concludes the discussion with an overview on the next Docker orchestration trends, such as software defined infrastructures, Infrakit, unikernels, Containers as a Service. The adventure continues!
What you need for this book
We assume the reader to have some experience using Docker from the command line: throughout the book we’ll continuously pull images, run containers, define services, expose ports and create networks.
Also, the ideal reader possesses a basic understanding of networking protocols and is familiar with public and private cloud concepts like virtual machines and tenant networks.
To follow the examples in the text, you will need Docker and its tools. Chapter 1, Welcome to Docker Swarm, covers their installation.
Also, to get the most from the examples, you will need access to a a public (for example AWS, Azure or DigitalOcean) or private (for example OpenStack) cloud to instantiate Virtual