VMware Big Data Extensions

Provide Hadoop, Kubernetes and Mesos

clusters on vSphere

2014 VMware Inc. All rights reserved.

Agenda
Brief Introduction to the BDE architecture
Demo: (Classic) BDE for provisioning Hadoop clusters with Cloudera Manager
Introduction to the Mesos Architecture
Demo: BDE deploys a Mesos cluster and allows Docker containers to be run
Using Chronos
Using Marathon

Demo: BDE deploys a Kubernetes cluster

How it Works

BDE is a virtual appliance that is integrated as an extension to vCenter Server.

based on the open source Project Serengeti, founded by VMware
works on vSphere 5.0 Standard/Enterprise/Enterprise Plus Edition or later.
clones VMs from the template VM and controls/configures VMs through vCenter
uses Chef to do software provisioning (Hadoop/others) into the cloned VMs

BDE Client

Virtual Appliance
Management
Management
Server
Server

Template
Template
Cluster
Cluster Node
Node

Cluster
Cluster Node
Node

Virtualization Platform

vCenter
Host
Host

Host
Host

Host
Host

Host
Host

Host
Host

Overview
Developers in the enterprise are building applications in containers
Mesos and Kubernetes are popular container cluster schedulers
BDE is extended to make it simple to run Kubernetes and Mesos on vSphere
Enterprise IT can leverage extended BDE to build an on-premise container

offering

Demo (1):Deploy Mesos cluster easily

User can create a Mesos cluster use BDE GUI (i.e. vSphere Web Client)
Watch the recorded video demo

User can create a Mesos cluster use BDE CLI as well

Login to BDE server and launch BDE CLI shell.
serengeti> connect --host localhost:8443

Create a Mesos cluster with a single command

serengeti> cluster create --name mesos --distro mesos --specFile

/opt/serengeti/samples/mesos_cluster.json
After the cluster is created,
Go to http://<mesos master ip>:5050/ to visit the Mesos homepage.
Or go to http://<mesos master ip>:8080/ to visit the Marathon homepage, and submit jobs.
Or go to http://<mesos master ip>:4400/ to visit the Chronos homepage, and submit jobs.
Or login to one of the mesos slave node to submit a simple job

CONFIDENTIAL

Mesos Architecture

CONFIDENTIAL

Offers and Scheduling in Mesos

CONFIDENTIAL

Demo (2): Deploy Kubernetes cluster easily

User can create a K8S cluster use GUI (i.e. vSphere Web Client) or CLI
Watch the recorded video demo

User can create a K8S cluster use BDE CLI as well (more details on wiki )
Login to BDE server and set VC connection params in sample cluster spec file

/opt/serengeti/samples/kubernetes_cluster.json

Launch BDE CLI shell.

serengeti> connect --host localhost:8443

Create a K8S cluster with a single command

serengeti> cluster create --name mykube --distro kubernetes --specFile

/opt/serengeti/samples/kubernetes_cluster.json yes
After the cluster is created, login the K8S Workstaion node to launch docker containers.
cd $KUBE_HOME; kubecfg.sh list minions
kubecfg.sh -p 9080:80 run 10.141.73.118:5000/nginx 2 myNginx

CONFIDENTIAL

VMs in the cluster

Demo of the Prototype (Contd)

CONFIDENTIAL

Kubernetes Cluster Spec File used by BDE

/opt/serengeti/samples/kubernetes_cluster.json

{
"nodeGroups":[
{
"name": "workstation",
"roles": [
"kubernetes_workstation"
],
"instanceNum": 1,
"cpuNum": 1,
"memCapacityMB": 7500,
"storage": {
"type": "LOCAL",
"sizeGB": 1
},
"haFlag": "off"
},

{
"name": "master",
"roles": [
"kubernetes_master"
],
"instanceNum": 1,
"cpuNum": 1,
"memCapacityMB": 7500,
"storage": {
"type": "SHARED",
"sizeGB": 1
},
"haFlag": "on"

},

{
"name": "minion",
"roles": [
"kubernetes_minion"
],
"instanceNum": 4,
"cpuNum": 1,
"memCapacityMB": 7500,
"storage": {
"type": "LOCAL",
"sizeGB": 1

},
"placementPolicies": {
"instancePerHost": 1
},
"haFlag": "off"

}
CONFIDENTIAL

10

Kubernetes Cluster Spec File (Contd)

/opt/serengeti/samples/kubernetes_cluster.json

"configuration": {
"kubernetes": {
"env": {
"GOPATH": "/opt/kubernetes/src/go",
"GOVC_URL": "https://root:vmware@10.111.88.43/sdk",
"GOVC_DATACENTER": "SingaporeDataCenter",
"GOVC_DATASTORE": "kubernetes",
"GOVC_RESOURCE_POOL": "/SingaporeDataCenter/host/BigDataCluster/Resources/",
"GOVC_NETWORK": "vlan883",
"GOVC_GUEST_LOGIN": "serengeti:password",
"GOVC_INSECURE": "1",
"KUBE_USER": "serengeti",
"INSTANCE_RESOURCE_POOL": "/SingaporeDataCenter/vm/SERENGETI-vApp-M9GA"
}
}
}
}

CONFIDENTIAL

11

Demo (3) : Wrapping BDE CLI in a single command

BDE provides CLI and REST API to allow DevOps to do further

integration and automation with other systems.

Download and unzip kube-bde-tool.tar
wget http://10.141.73.118/kube-bde-tool.tar

Modify the vSphere information in the configuration file

Create cluster with a single command
./kube.sh cluster create <cluster_name>

SSH to Kubernetes workstation node

Create a docker container with nginx docker image
kubecfg.sh -p 9080:80 run 10.141.73.118:5000/nginx 2 nginx

CONFIDENTIAL

12

Why BDE
For users
Create/manage a cluster of VMs in an scale-out application as a whole
A simple language describes the connection between application and
infrastructure
A extensible framework for DevOps in virtual infrastructure
For VMware
BDE is a proven tool to manage clustered workload at scale
BDE is a automation layer to make the scale-out workload oriented VM level
primitives usable to end users.
BDE is a very agile release vehicle that can meet the ever changing market.
Async release with vSphere
Backed by OSS project Serengeti

CONFIDENTIAL

13

Future work
Make the developers experience consistent
Reduce/remove the vSphere information in the developer tools
Use lightweight OS for containers (e.g. CoreOS)
Use VMFork technology to provision clusters super fast

CONFIDENTIAL

14

Goal: help enterprise IT to provide a container offering onpremises

Infrastructure team
Enable rapid delivery of Kubernetes and Mesos clusters on-premises
Satisfy the agility requirement of developers while maintaining control
The service can be shared by multiple teams
Developers
Should have consistent development & deployment experience
Shouldnt care about the infrastructure

CONFIDENTIAL

15