VXLAN

By Anand Nande

AGENDA

What is VXLAN ?
Why VXLAN ?
How does it work ?
So now we can migrate VMs across subnets?
What about routing across VXLANs?
Any Performance Impact?
Demo

What is it ?
Virtual eXtensible Local Area Network

 Tunneling protocol co-developed by a group of companies.

Original description as per the RFC:

Used to address the need for overlay networks
within virtualized data centers accommodating multiple tenants. The
scheme and the related protocols can be used in networks for cloud
service providers and enterprise data centers
Encapsulation method to extend L2 traffic over a L3 network. VXLAN uses
multicast in the backend. It uses traditional flood and learn for MAC learning
and Address Resolution Protocol (ARP) resolution.

Why VXLAN?
What problems does it address and how am I selling it

It's intended to solve the following issues

Limitations Imposed by Spanning Tree and VLAN Ranges

STP blocks the use of links to avoid the replication and looping of frames.
This is a prob to some DC admins who pay for ports and links
Resiliency due to multipathing - not available.
2^12 = 4096
A 12-bit VLAN ID used to divide multiple broadcast domains
STP blocks few ids from this domain

Multi-tenant Environments

Is it possible to address need for multiple VLANs per tenant w/ 4096 limit?

L3 networks not a comprehensive solution. 2 tenants might use the same set of Layer 3
addresses within their networks.

Inadequate Table Sizes at ToR Switch

How does it work?

Lets not draw conclusions yet..theres a lot more to it

 VXLAN encapsulates Layer 2 frames into Layer 3

packets (using UDP)
Adds a 24-bit VXLAN Network Identifier (VNI) that allows
for up to 16 million unique combinations
VXLAN Segments are built between VXLAN Tunnel
Endpoints (VTEPs)...more on this in coming slides
Not understood by any physical networking devices (the
transport that carries the encapsulated frames only needs
an IP-based network).

VXLAN Encapsulation
The L2 Ethernet Frame

The Final packet that goes out on L-3.

Note how inner-MAC is payload here
and outer-MAC + UDP is added.

The VNI is the only distinguished

identifier for a VXLAN segment

VTEPs across a L3 network

Hypervisor 2

Hypervisor 1
10.0.0.1

10.0.0.2

VM2-1

VM1-1
VTEP-1

172.16.1.0/24

VNI=10

192.168.1.0/24 VNI=20
IP=172.16.1.10
MAC=52:54:00:0e:08:b3
VNI=10

VTEP-2
IP=172.16.1.12
MAC=52:54:00:30:de:e3
VNI=10

L3 Network

VM1-2
IP=192.168.1.100
MAC=00:0C:29:2F:32:A0
VNI=20

VXLAN segment-1 w/ VNI=10

VXLAN segment-2 w/ VNI=20

VTEPs responsible for

encap/decap packets at
both ends of the wire

VM2-2
IP=192.168.1.111
MAC=00:0C:29:2F:23:A0
VNI=20

VXLAN Multicast group

VTEP-3s table
MAC

VNI ID

REMOTE VTEP

52:54:00:a0:1b:bb

10

192.168.122.186

52:54:00:8a:bd:ff

10

192.168.122.101

VTEP-1s table
MAC

VNI ID

REMOTE VTEP

52:54:00:60:18:f9

10

192.168.122.141

52:54:00:8a:bd:ff

10

192.168.122.101

MAC

VNI ID

REMOTE VTEP

52:54:00:a0:1b:bb

10

192.168.122.186

52:54:00:60:18:f9

10

192.168.122.141

VTEP-2s table

packet
capture
on one
of the
VTEPs

Additional
Wireshark
plugin
required to
analyse the
UDP data here

So now can we migrate

VMs across subnets?
..the answer is YES - we can

Whats the most important thing when we migrate a VM ?

- VMs ip and mac addresses should not change even if the VM has been
migrated to a new hypervisor.
- We use the reference controller which learns the new VM-placement using
ovsDB.
# ps
-ef | grep controller | grep -v color
root
1396 1325 0 Feb17 pts/3 00:00:01 controller -v
ptcp:6633
-

This reference controller is located on each VTEP.

VM communication via VTEP post-migration

Hypervisor 2

Hypervisor 1
10.0.0.1

10.0.0.2

VM2-1
Flows related to
vm1-1 removed on
VTEP-1
VTEP-1 and added to
VTEP-2

172.16.1.0/24

VNI=10

VTEP-2
IP=172.16.1.12
MAC=52:54:00:30:de:e3
VNI=10

* The VM's port is updated to

point to the new host.

VM1-2
IP=192.168.1.100
MAC=00:0C:29:2F:32:A0
VNI=20

* When the port is updated by

Nova, Neutron executes ML2
mechanism drivers, including
l2pop, which then sends RPC
messages to the new host.

VM1-1
IP=172.16.1.10
MAC=52:54:00:0e:08:b3
VNI=10

What about routing across

VXLANs ?

Lets look at the 2 way communication that happens

physical:virtual
[VLAN-to-VXLAN]
- also known as vxlan-bridging that extends
the l2-domain over a vast l3-network.
- VTEPs are responsible in this case to the
check for the VNI id in the VXLAN header. If
it matches the one of the VNIs thats in its
table, forward the packet to the relevant
vxlan-segment to take care of it.

virtual:physical
[VXLAN-to-VLAN]
- vlan to router
- hardware VTEP(router) only does bridging
- on physical router:
- in_ports, out_ports, loopback_ports
1. in_ports > bridge into a vlan > lo_ports
2. router_fib > next_hop > physical_world

- Switch should be capable of vxlan handling.

- performance loss because of loopback

Any Performance Impact?

cliche but true Your loss is someone else's gain

An overlay network is simply a computer network which is built on top of another network.

The added flexibility comes at a cost, due to the additional processing overhead for encapsulation
and de-encapsulation of packets. This consumes both CPU resources and degrades network
performance, especially for high speed connections.

By introducing hardware offloading capabilities that can be found in some of todays modern NICs,
the added overhead for packet processing can be offloaded to the NIC hardware, resulting in
improved CPU utilization and higher throughput.

Improve performance of VXLAN w/ hardware offload: https://access.redhat.com/articles/1390483

Demo
Lets put this into action by creating a simple topology

192.168.122.141
192.168.122.186

192.168.122.101

Ignore for the purpose of demo.

You can do this later as an advanced exercise

Hands on
[On your KVM host or laptop]
# yum install virt-manager
libvirt* qemu-kvm openvswitch
# cd /var/lib/libvirt/images
## Get the mininet vmdk image
from the mininet website
## write the spawn.sh
# chmod +x spawn_vm.sh
# ./spawn_vm.sh <vmname>
(note you need to spawn 2
mininet VMs, hence use the above
script 2wice)

# cat > spawn.sh << EOM

> #!/bin/sh
> export VM_NAME=$1
> export portgroup=$2
> IMAGES_BASE=/var/lib/libvirt/images
> cp $IMAGES_BASE/mininet.vmdk
$IMAGES_BASE/$VM_NAME.vmdk
>
> virt-install -r 256 \
>
-n $VM_NAME \
>
--vcpus=1 \
>
--import \
>
--autostart \
>
--memballoon virtio \
>
--network network=host-only\
>
--disk $IMAGES_BASE/$VM_NAME.vmdk \
> EOM

[Inside your Mininet VM-1]

Create a simple topology:
mininet@mininet-vm:~$ sudo mn --topo single,1
Create vxlan interface on each side:
mininet> sh ovs-vsctl add-port s1 vxlan
Setup vxlan interface on each side
mininet> sh ovs-vsctl set interface vxlan type=vxlan option:
remote_ip=192.168.122.101
Setup IP for hosts:
mininet> h1 ifconfig h1-eth0 10.0.0.1
[Inside your Mininet VM-2]
mininet@mininet-vm:~$ sudo mn --topo single,1
mininet> sh ovs-vsctl add-port s1 vxlan
mininet> sh ovs-vsctl set interface vxlan type=vxlan option:
remote_ip=192.168.122.186
mininet> h1 ifconfig h1-eth0 10.0.0.2

Advanced VXLAN configuration

Manually assign VXLAN Network ID (VNI) and/or OpenFlow port number
e.g. VNI=20, OF_PORT=9
# ovs-vsctl set interface vxlan type=vxlan option:remote_ip=192.168.122.186
option:key=5566 ofport_request=9
Assign VNI flow by flow (Useful when implementing multi-tenant environment)
# ovs-vsctl set interface vxlan type=vxlan option:remote_ip=140.113.215.200
option:key=flow ofport_request=9
and then setup your flow entries by
# ovs-ofctl add-flow s1 in_port=1,actions=set_field:5566->tun_id,output:9
# ovs-ofctl add-flow s1 in_port=9,tun_id=5566,actions=output:1

References
-

IETF RFC : tools.ietf.org/html/rfc7348

Scott Lowes blog : http://blog.scottlowe.org/
David Mahlers youtube channel : https://www.youtube.com/user/mahler711
VMware VXLAN Performance Evaluation : https://www.vmware.com/files/pdf/techpaper/VMwarevSphere-VXLAN-Perf.pdf
TCP/IP over VXLAN Bandwidth Overheads: http://packetpushers.net/vxlan-udp-ip-ethernetbandwidth-overheads/