FI-ware :: Cloud Hosting

Task 4.1 :: Virtualisation Layer Intel: Joe Butler, Andy Edmonds

T4.1 Overview
T4.5 Self-service & PaaS

T4.4 Advanced Management

T4.2 Basic Management

T4.1 Virtualisation Layer

Intel IBM-IL FT INRIA

48 PMs 20 PMs 20 PMs 10 PMs

Fundamental physical Resource virtualisation Instrument and Monitor Resources Secure and Performant Resources Holistic Management of Resources

T4.5 Cloud Proxy

T4.1 Dependencies & Composition

T4.1
T4.2 Basic Management T4.1D Management

Uses T4.1 Virtualisation Layer

Uses
WP7::T7.3 Interface to Open Networking Entities WP7::T7.4 Network Services

Composed of

T4.1C Security & Performance

T4.1B Monitoring Uses

WP8::T8.1 Security Monitoring WP8::T8.2 Generic Security Enablers

T4.1A Resources

T4.1A: Physical Resource Virtualisation

3 Types of Resources to Support
1. Compute:
Hypervisor based virtualisation (Types 1 & 2)

2. Storage:
Object and block level: data and VM image storage

3. Network:

Intra and Inter physical host virtual networks. Provisioning of related services (Firewall, DHCP, DNS) Integrate with T7.4 Network Services

Major challenge today

Expose holistic management capabilities (API)

T4.1B: Instrumentation & Monitoring

Essential aspect of manageability if its not monitorable, its not manageable Expose resource attributes as metrics (API)
Upper layers can build upon these (computed metrics) Metrics fully configurable (sample rate, reported unit, resolution, period etc.) The fundamental building block to SLA-enabled systems

Minimise overhead impacts on monitored resources

T4.1C: Secure & Performant Resources

Secure
Isolated (partitioned) resources Integrate work where appropriate from T8.1 Security Monitoring and T8.2 Generic Security Enablers

Performance
There will be trade-offs
Typical example: with power savings

Major challenge here is I/O

Investigate and prototype the use of IOMMU, SR-IOV

T4.1D: Comprehensive Manageability

Consistent standardised interfaces. Management of the 3 types of resources. An API to allow T4.2 CRUD and monitor T4.1 instantiated resources. Provide end-to-end (client-to-provider) capabilities.

Intel Assets
T4.1A
Hypervisors: KVM, XEN, VMware, OS Containers (?) Resource management: FlexMigration, Apache Tashi & Zoni, OpenStack: Nova & Swift, OpenVswitch

T4.1C
Security: TXT, TPM, Trusted Compute Pools Performance: IOMMU, SR-IOV, VMDQ, IOLanes (?)

T4.1D
Management API: OCCI Service Interface Intel Web API SLA@SOI Messaging (XMPP, AMQP)

T4.1B
Monitoring: collectl, ganglia, IOLanes-developed kernel probes

Overarching: CloudBuilder, Open Data Centre Alliance

Note: what follows can be used (elaborated) as inputs to SotA for WP4

T4.1A Assets Review

Hypervisors (not discussed today)
Note on Intel FlexMigration Would be interested in OS Container opinions

Open Source Virtualised Resource Managers

Apache Tashi (compute) OpenStack (compute and storage) Eucalyptus (will not be discussed) OpenvSwitch (network)

And yes there are others! ;-)

T4.1A Intel FlexMigration

AMD Analogue: Extended Migration Problem: You attempt to migrate a VM to a server that expects certain CPU functions (e.g. SSE2,3) and theyre not Kernel Panic Solution: use the facilities provided.
Through a bitmask that limits the reporting of new features Gives the hypervisor a way to trap and emulate the instruction

Currently exposed by VmWare ESX (vMotion) & XEN (patch Apr 2010) Why? Reliable migrations

T4.1A Apache Tashi

Manages clustered virtualised compute resources Used in SLA@SOI Pain points:
Storage management Network Management

Implements QoS using Cgroups Large changes in scheduler for power-aware resource mgt*
Monitoring, QoS, Scheduler, OCCI Service manager enable an SLA-aware infrastructure service. Apache 2.0

T4.1A Apache Tashi Zoni

Physical node deployment system
Creates multiple isolated physical resource clusters

Registers, configures and deploys clusters Converts existing virtual deployments to physical ones
Functionality:
Isolation Allocation Provisioning OOB Management

Components:
DHCP & DNS PXE & HTTP Image Store (NFS) Configurable switches Remote Access (iLO, DRAC, PDUs)

T4.1A OpenStack Nova

Large community, Apache 2.0 Main favoured cloud research platform within Intel
Number of internal test beds already exist Intel sponsored blueprints
Manages virtualised resources
Compute (libvirt based)
Hypervisors and OS Containers

Storage
Volume controller SAN, iSCSI, Image Repository glance,

Networking
Number of modes: flat, flat DHCP and VLAN DHCP

RBAC AuthZ EC2 user tools compat

T4.1A OpenStack Nova: Core Components

API Server: front end to the system, EC2, Rackspace, OCCI. Message Queue: currently RabbitMQ. Compute Controller: run, terminate, reboot, (de)attach storage, console. Network Controller: allocate IP, config VLAN, network config. Volume Controller: create, delete, swap in/out volumes. Scheduler: Simple & Random. Image Store: Glance {registry, store[S3, Swift, FS, HTTP]}.

T4.1A Object Storage: OpenStack Swift

"Swift is a highly available, distributed, eventually consistent object/blob store Comprises of multiple loosely coupled components
Compliments Nova License: Apache 2.0

T4.1A OpenStack Swift: Core Components

Proxy server:
front end to the system. provides API Maps entity names to their storage location stores, retrieves and deletes objects stored on local devices. Stores objects' metadata using xattrs (major OSes support these). stored using a path derived from the object names hash and the operations timestamp. Last write always wins - latest object version will always be served Deletion is treated as a version of the file replication within cluster is by hash lists,
object replicator has responsibility

The Ring Object Server:

Container Server:
Lists objects, listings are replicated using hash lists and shared high watermarks, DB replicator has responsibility

T4.1A OpenStack Swift: Core Components

Account Server:
Lists containers. Listings are replicated using hash lists (checks for change) and shared high watermarks (establishes content change).
DB replicator has responsibility.

Account Reaper:
Removes data from deleted accounts in the background. Runs on the account server.

Updaters:
Load is high or failure conditions: replication is deprioritised. Updaters implement the eventual consistency characteristic of the system.

Auditors:
Logs and check for integrity. If corruption is found an auditor will replace from another replica.

T4.1A OpenStack Swift: Other Components

AuthN/Z:
Can be internal or external to swift, User passes auth token with each request via HTTP header
token does not change but expires (time-based) Note: swift caches user data until the token expires

Intercept point for work out of T8.1, T.8.1 Access Logs, Account Stats Log, supports log processing plugins performed on write requests to the account and container dbs requires synchronised time across proxy server(s) Upload is defaults to max 5GB Uploads can be segmented (client side chunking, multiple parallel connections) Download is unlimited (via segmentation)

Stats System (Monitoring):

Rate Limiting (QoS):

Large Object Support:

TA4.1A OpenvSwitch
A L2/L3 Managed Distributed Virtual Switch Provides inter- and intra-physical host virtual networking capabilities
Allows for cross subnet migrations via virtualisation Allows for cross subnet vVLAN per customer

Software-only and Hybrid

Exports Linux bridging & VDE compatible interfaces Almost identical performance* Virtual Machine Networking State is Mobilised OpenFlow sflow, NetFlow support. Standard(s) to be supported in WP7

IPv6 Support Tunnelling (VPN, GRE) Per VM traffic policing NIC bonding 802.1ag link monitoring Fine-grained min/max rate QoS

*B. Pfaff, J. Pettit, T. Koponen, and K. Amidon, Extending networking into the virtualization layer, Proc. HotNets 09

T4.1B: Monitoring
Expose Metrics across all 3 types of Resources
Raw Metrics, no computed metrics (upper layers) Make accessible via T4.1D

Metrics will come from different systems (e.g. Ganglia, collectl) SLA@SOI monitoring source agnostic FP7 IOLanes kernel probes

TA4.1C: Security & Performance

TPM, TXT, Trusted Compute Pools
TXT Trusted Execution Technology
Protected launch & execution, attestation (remote also)

TPM Trusted Platform Module:

Standardised (Trusted Compute Group) interface in front of TXT implemented as 3rd party hardware module

Trusted Compute Pools

TXT/TPM technology integrated with hypervisor and resource mgt system. Enables config and mgt of TPM/TPM features for more trustworthy virt Ensures hypervisor, VM images & critical executables are not tampered with (via MLEs) Blueprint registered with OpenStack from Intel*

Isolation offered by OpenvSwitch (see previous) IOMMU: I/O Memory Management Unit
Dedicates complete PCIe device to one VM, minimal hypervisor overhead. Need use case to motivate IOMMU - is this an edge case?
Dynamic Allocation of Graphics Processors based on work load type? Adding additional compute power to an existing VM?
* https://blueprints.launchpad.net/nova/+spec/trusted-computing-pools

TA4.1C: Security & Performance

Network-related Performance

Intel VMDq can provide hardware accelerated switching in the NIC (push fast paths into silicon) OpenvSwitchs Fast path can reside here

SR-IOV support in PCIe devices Device supports PFs and VFs PF full config and data VF only data, exposed as device

OVS: All in software

OVS:
Hardware & software Management via OVS

TA4.1D: Holistic Management

OCCI Open Cloud Computing Interface
Categorise, identify, link and operate on RESTful resources Adaptable, discoverable, extensible, truly open
Initially targeted at IaaS but can accommodate PaaS*

Impact:
Recommended by UK G-Cloud Only IaaS standard considered by NIST To be submitted to DMTF-CMWG, Work charter established Many implementations** Monitoring & SLA extensions DGSI, SLA@SOI OVF and JSON interop output of DMTF meeting (mid-may) Planned collaboration with FP7 SAIL focused on Networking OCCI over AMQP transport (REST does not mandate HTTP)

Upcoming

* A. Edmonds, T. Metsch, and A. Papaspyrou, Open Cloud Computing Interface in Data Management-related Setups, Springer Grid and Cloud Database Management, Apr. 2011. ** http://occi-wg.org/community/implementations/

TA4.1D: Holistic Management

OGF DCI-Fed Distributed Compute Infrastructure, Federated
Tasked with Interoperability profiles (Cloud/Grid)

SLA@SOI Infrastructure Service Manager

OCCI Front-end to infrastructure resources Pluggable backend

Intel Web API

Allows service access to client hardware features
Allows for the application of service side policies to client devices
E.g. if power low, migrate VM, report RTTs to service

JavaScript library to a OS specific plugin

Overarching Assets
Intel CloudBuilder
Library of infrastructure HOWTOs Avenue of dissemination for FI-ware

Open Data Centre Alliance

Source of industry use cases and needs
Timeline current now and +2 years

Channel to Influence Industry

Key Criteria for T4.1

Remove lock-in, minimise proprietary solutions, adopt existing open standards, interoperability Share-nothing, decomposable architecture (loose coupling) Automate everything
give a thought for config mgt (puppet, chef, CFEngine)!

Reuse and extend with critical evaluation Influence standards and initiatives Drive by real world needs:
FI-ware partner project use cases Open data centre use cases

Quick Thought Experiment

Architecture!
But; more code, less boxes

Create a basic framework to plug-in all our assets Most IaaS systems follow a similar pattern For this experiment lets use:
Tashi Nova Swift and for some balance: CloudFoundry

What Have They In Common?

Decoupled Components!
Entry Point: API Front-end
Request Management Request Distribution Redundant Resource Providers Monitoring across all resources
Enabling adapability
And Probably More!

The boundary between consumer and internal system

These are even present in PaaS

Look (quickly!) at CloudFoundry

CloudFoundry Arch

Generalised Architecture
A start! Theres a place for all WP4 assets
Requests for Provisioning Monitoring CRUD Metrics Bus

Collect Metrics

Entr y Point

Resource Provider
Request Management Provisionings Service Consumer

Loose ends
How to characterise typical work loads, typical customer needs? (Guerilla Capacity Planning Method?) Source code - where? license? Non-viral please Packaging of outputs - all in one? runnable on a laptop? only in a massive DC? Probably More Testbeds - who has what? Some might be Dedicated WP4 F2F? redundant by Tues.

Cheers!

OpenStack Nova Architecture

T4.1A OpenStack Nova: Networking

Flat Mode
1 virtual bridge per physical host. Static IP config injected at provisioning.

Flat DHCP Mode

1 virtual bridge per physical host. IP config is set to DHCP, DHCP server manages IPs.

T4.1A OpenStack Nova: Networking

VLAN DHCP Mode

OCCI Features
Discovery system for supported Resources
Types (kind, mixins) offered for instantiation are advertised

Kinds: basic types of a offering (compute, storage etc.) Mixins

Dynamic composition Tagging OS and Resource Templating Extension mechanism

Full CRUD on Resources and Links Resources are linkable (Link) Resources are actionable (Action) Batch atomic operations are supported (multipart) Current transport == HTTP, resources rendered in header or body

OCCI Core Model

OCCI Infrastructure Model

Note OCCI::Core Resource and Link are extended by this Model

Infra Svc Mgr

T4.5

T4.5

T4.4

T4.2

T4.1