OpenStack images that fit your imagination -

deep dive into container images in Kolla

OpenStack Summit, Sydney, Nov 8 2017

Rich Wellum, Software Developer, Cloud Architecture, Lenovo, rwellum

Vikram Hosakote, Cloud Software Engineer, Cisco Systems, vhosakot
Agenda
• OpenStack Kolla
• What OpenStack Kolla solves
• Benefits of containers in OpenStack
• Architecture of Kolla
• How to use Kolla
• Deploy OpenStack in minutes – do not build
• How to customize Kolla images
• Demo
OpenStack Kolla
• Deploys OpenStack services in Docker containers
• Two ways to deploy:
• Ansible
• Kubernetes

• Three sub-projects:
• Kolla
• Kolla-ansible
• Kolla-kubernetes
What OpenStack Kolla solves
• Deploy OpenStack binaries or build from source
• Faster deployment
• Easy maintenance, re-configuration, patching and upgrades
• Containerized OpenStack services in registry
• Only one tool to do multiple things
• Pin:
• OpenStack version of each service
• Configuration files for each service
Which projects are users most interested in adopting in the future? Which projects are used by OpenStack deployments?

OpenStack User Survey (April 2017) - https://www.openstack.org/assets/survey/April2017SurveyReport.pdf

Kolla is easy to use
• DevStack
• Juju
• PackStack
• Fuel
• OpenStack-ansible
• TripleO
• Openstack-puppet
• OpenStack-chef
Benefits of containers in OpenStack
• Reproduce golden state easily – maintain, version, tag, upgrade
• No more “Works in DevStack” ;)
• Production-ready images
• Easy to override/highly flexible:
• Dockerfiles
• Jinja templates (configuration files)

• Dev mode (do Dev and Test quickly using Kolla)

• Portable, tested, replicated images in secure registries – DockerHub,
Quay, etc
Architecture of Kolla
 GitHub
(kolla, kolla- Registry
ansible, kolla- (DockerHub)
kubernetes)
Deploy node
Dockerfiles, ansible playbooks, (Docker, Ansible, images, layers
jinja templates (config files), Docker python,
helm charts, k8s jobs Jinja2)
deploy, re-configure, deploy, re-configure,
upgrade deploy, re-configure, upgrade
upgrade
Compute Controller Storage
node node node
Compute
(Docker, OpenStack Compute
(Docker, OpenStack Compute
(Docker, OpenStack
Computeservices)
node Computeservices)
node Computeservices)
node Target
node node node nodes

Management/admin network
API/service network
Storage/data network
OOB network
Not just for vanilla OpenStack with default configs
• Kolla can be used for advanced configurations like
• Jumbo frames
• SR-IOV (PCI passthrough)
• Massive scale (hundreds of nodes)
• New networking technologies - OpenDaylight, OVS-DPDK, VPP*, OVN, NFV,
Tacker, Service Function Chaining (SFC)
• Supports plugins
• Ceph storage
• EFK (Elasticsearch, Fluentd, Kibana)
• Prometheus – cloud-native monitoring and alerting
• Customizations and template overrides
How to use Kolla
• Install:
• Install Ansible, Docker, Docker python and Jinaj2 on the deploy node
• Install Docker and Docker python on all the target nodes (bootstrap-servers.yml)
• Clone the kolla repos
• Build:
• Create /etc/kolla/kolla-build.conf (optional)
• kolla-build (lots of build options)
• Deploy kolla-ansible:
• Create Ansible inventory file and update globals.yaml
• kolla-ansible prechecks -i <inventory file>
• kolla-ansible deploy -i <inventory file>
Deploy OpenStack in minutes – do not build
• Pull pre-built golden images from DockerHub
(https://hub.docker.com/u/kolla/) instead of building images
Deploy OpenStack in minutes – do not build
• Kolla images are tagged per-release on DockerHub
• kolla-ansible –i inventory file pull
Building and Orchestrating Custom Kolla
Images and Containers (On Kubernetes!)

● Official Kolla Image building documentation:

https://docs.openstack.org/kolla/latest/admin/image-building.html

● These slides: http://bit.ly/2zl7WFF

● Orchestration tool used in this demo:

○ https://github.com/RichWellum/k8s.git
○ https://review.openstack.org/#/c/487972/

● Other orchestration options:

○ Ansible Playbook: https://review.openstack.org/#/c/457317/
○ Deployment Guide: https://docs.openstack.org/kolla-
kubernetes/latest/deployment-guide.html
 Reasons why you might build custom Kolla
1. Case example 1; your company buildsImages?
and produces a proprietary OpenStack. Some services
are touched, some are not. For example Horizon has plugins and addons that uniquely identify
your company's look and feel, but changes to many other Services are not required.
○ Kolla provides tools to build Docker images containing an OpenStack Service. And a single
image for a Service up to the entire stack can be built.
2. Case example 2; as an OpenStack contributor you work with a Service - say Ironic. The bulk of
your work is done in devstack, a simulator. How cool would it be to develop in an OpenStack
environment running with production level code?
○ Kolla sub-projects: Kolla-Ansible and Kolla-Kubernetes provide the orchestration to run
containerized OpenStack.
3. Case example 3; Your company develops drivers unique to their hardware. Most of OpenStack
should just work, be easy to build and orchestrate in a real environment.
○ With Kolla the hard-work per Service has already been done; facilitating a low cognitive
overhead.
src code changed: Helm
delete / Helm install Building and Orchestrating
DEVELOPMENT KOLLA BUILD ORCHESTRATION and OPERATIONS
Local Source Code Tools to build Kolla-Kubernetes
images
Kube Kube
src/Keystone Schedul Controll Neutron Nova
+ LDAP or Keystone Image er er
SAML
Kube Kube Glance
Heat
etcd DNS
src/Horizon
+ Custom Horizon Image
logo/skins Kube Kube
Cinder Ironic
SDN Proxy

Tiller Kube Keystone Horizon

src/<Service> <Service> Image API

Reconfiguration
 Demo
● Start with source code for Horizon, make a customization change.
● Install Kolla.
● Compile a Horizon docker image with Kolla Build tools and store locally.
● Bring up Kubernetes Cluster and verify it is operational.
● Orchestrate a combination of UpStream(Ocata) Kolla containers from
Dockerhub and our new Horizon container on the Kubernetes Cluster.
● View the results on the Horizon GUI.
● Demo Video: https://www.youtube.com/watch?v=jyjQ_qy7wPU
● Demo Notes with all the steps are in the slides below for referenced.
DEMO
Wrap up - how to get involved
● Need a feature? Find a bug? Let us know!
● Gerrit workflow -
http://docs.openstack.org/infra/manual/developers.html
● Kolla IRC channel - #openstack-kolla
● File bugs, blueprints, track releases on Launchpad -
https://launchpad.net/kolla
● Attend weekly meetings -
https://wiki.openstack.org/wiki/Meetings/Kolla
● Contribute code - https://github.com/openstack/kolla
● Mailing list - openstack-dev@lists.openstack.org [kolla]
Demo Notes
 Start with pulling your source code
● For the purpose of the rest of this presentation - the assumption is that
you are working from a VM…
● We are going to build custom Kolla Images for three services, Keystone,
Horizon and Cinder.
● Grab the custom OpenStack code - for example your company's source
code for keystone, horizon and cinder may reside on a git server
somewhere - clone it to a work-space, E.g:
○ git clone git@10.1.2.3:myopenstack/keystone.git
○ git clone git@10.1.2.3::myopenstack/horizon.git
○ git clone git@10.1.2.3::myopenstack/cinder.git
 Grab Kolla Repository
● Grab the kolla repo which will give you access to the tools to build
images
○ git clone http://github.com/openstack/kolla

● Compile Kolla
○ sudo pip install -r ./kolla/requirements.txt
○ sudo -H pip install -U kolla
 Generate files
● Generate a kolla-build.conf
○ sudo -H pip install tox
○ cd kolla; sudo -H tox -e genconfig
■ Note: (places build-conf in /etc/kolla !):
● Edit kolla-build.conf and add in source
○ vi etc/kolla/kolla-build.conf
○ install_type = source
■ But this is optional as it can be supplied as a parameter to kolla build
■ Note binary means that OpenStack will be installed from apt/yum. And the
source means that OpenStack will be installed from source code.
 Edit kolla-build.conf and add pointer to source
● Edit kolla-build.conf and add in source pointers
○ vi etc/kolla/kolla-build.conf ⇐ note local etc not /etc
○ Lets add references to keystone, horizon and cinder source code
○ Note the reference - what will the rest of your OpenStack be running?
○ Type can also be git or url.
[cinder-base]
+ type = local
[keystone-base] + location = /home/rwellum/cinder/
+ type = local + reference = stable/ocata
+ location = /home/rwellum/keystone/
+ reference = stable/ocata

[horizon]
+ type = local
+ location = /home/rwellum/horizon/
+ reference = stable/ocata
 Create a Docker Registry if needed
● For some scenarios (multi-node, sharing etc) it’s helpful to create a docker registry to store these images, E.g:
● kolla/tools/start-registry
● Or:

docker run -d \
--name registry \

--restart=always \

-p 4000:5000 \

-v registry:/var/lib/registry \

registry:
 Now we are ready to build images (1/2)...
● You can choose to build the images locally (Like an All In One (AIO) Dev environment) or push to a
Docker Registry - if you intend to share these images or running multi-node.
● Change the base distro with the -b option:
○ kolla-build -b ubuntu
● Local build (4 different ways - why not?)
○ sudo kolla-build -t source horizon cinder keystone --config-dir
./kolla/etc/kolla/
○ Or: sudo which klollkolla-build -t source horizon cinder keystone --
config-file ./kolla/etc/kolla/kolla-build.conf
○ Or: copy kolla-build.conf to /etc/kolla (from ./kolla/etc/kolla/kolla-
build.conf
○ Or: sudo kolla/tools/build.py -t source --config-file
./kolla/etc/kolla/kolla-build.conf
 Now we are ready to build images (2/2)...
● Registry multi-node
○ Add: “ --registry 172.22.2.81:5000 --push”
○ E.g. sudo kolla-build --registry 172.22.2.81:5000 --push -t source
horizon cinder keystone
○ To push images to a Dockerhub repository named mykollarepo:
○ sudo kolla-build -n mykollarepo --push -t source horizon cinder
keystone
● Add in build logs
○ Any issues in building this is where you start
○ Add: “--logs-dir horizon-log --debug“
○ E.g. sudo kolla-build --registry 172.22.2.81:5000 --push -t source
horizon cinder keystone --logs-dir horizon-log --debug
 Lets check our images built...
● Lets check for valid generated images
rwellum@ubuntuk8s:~$ sudo docker images | grep horizon
kolla/centos-source-horizon 4.0.2 5182810deebb 2 minutes ago
1.077 GB
● The Tag ‘4.0.2’ is very important in Kolla…
● We’ll use this and the location of the image to orchestrate
How do we Orchestrate our new images (1/2)?
● So we have our custom images. Now we’d like to Orchestrate…
● Kolla is the main project - the tools to build OpenStack Images
● Kolla-Ansible is a sub-project to orchestrate with Kolla images
○ Industry proven, widely used, very reliable!
● Kolla-Kubernetes is another sub project to orchestrate with Kolla
images
○ Dev project, to run on Kubernetes cluster, new, exciting, be careful!
● This example is based on Kolla-Kubernetes and using a tool I wrote to
stand up an AIO kolla-kubernetes cluster…
● But it applies to any method, manual (following the bare-metal guide:
https://docs.openstack.org/kolla-kubernetes/latest/deployment-
guide.html) even Kolla-Ansible
How do we Orchestrate our new images (2/2)?
● Kolla-Kubernetes relies on a ‘cloud.yaml’ file to stand up
OpenStack
● In our example we want to run with our custom keystone,
horizon and cinder images
● All other Services will be pulled from dockerhub
● Grab k8s.py - deployment / orchestration tool:
○ git clone https://github.com/RichWellum/k8s.git
○ Replace with hopefully merged version
 Orchestration continued...
● Standup OpenStack
○ Create a working directory - like ‘os’
○ Run k0:
■ ../k8s/ko.py eth0 eth1 --image-version ocata --edit-config
○ Note two interfaces required as per normal (MGMT and Neutron)
○ Note edit-config option
○ --help will show all the options available.
● Ko.py will proceed to install Kubernetes and start to bring up
OpenStack
● It will pause at the appropriate point to allow the user to edit the
cloud.yaml - where you will point the image tag to your local
custom image….
 Changes to Cloud.yaml
keystone:

all:
+ image_full: kolla/centos-source-keystone:4.0.2
horizon:
all:
+ image_full: kolla/centos-source-horizon:4.0.2
cinder:
all:
+ image_full: kolla/centos-source-cinder-base:4.0.2
 Once OpenStack is up...
● Lets confirm we actually orchestrated with local images
○ kubectl get pods --all-namespaces | grep horizon
○ kubectl describe pod horizon-blah -n kolla | grep Image

● And/or connect to your container and look for that proprietary code
○ docker run -tu root -ti horizon-blah /bin/bash
○ kubectl -n kolla exec -it horizon-blah -- ls
 So you make some changes to your source
code..
● Repeat the steps to build the image
● Kill the container: kubectl delete pod horizon-blah -n kolla
● WHAT NEXT???
 And commence OpenStacking...
● At this point you have a running Kolla-Kubernetes OpenStack running
with custom containers where needed.
 Building and Orchestrating
DEVELOPMENT KOLLA BUILD ORCHESTRATION and OPERATIONS
Local Source Code Tools to build KUBERNETES
containers
Kube Kube
Schedul Controll Neutron Nova
src/Keystone Keystone Image er er

Kube Kube Glance

Heat
etcd DNS
src/Cinder Cinder Image

Kube Kube
Cinder Ironic
SDN Proxy
src/Horizon Horizon Image

Tiller Kube Keystone Horizon

src/<Service> <Service> Image
API

Trash and restart Helm upgrade /

update config
Helm delete / Helm install
Building and Orchestrating
DEVELOPMENT KOLLA BUILD ORCHESTRATION and OPERATIONS
Local Source Code Tools to build Kolla-Kubernetes
images
Kube Kube
Schedul Controll Neutron Nova
src/Keystone Keystone Image er er

Kube Kube Glance

Heat
etcd DNS
src/Cinder Cinder Image

Kube Kube
Cinder Ironic
SDN Proxy
src/Horizon Horizon Image

Tiller Kube Keystone Horizon

src/<Service> <Service> Image
API

Helm upgrade /
update config