Table

of Contents
Concepts and Components of NSX for vSphere platform ................................................. 10
NSX for vSphere Networking and Security Core Management and Control Components .......... 11
Management Plane - NSX for vSphere Manager ......................................................................... 12
Control Plane NSX for vSphere Controller ................................................................................ 13
Data Plane ................................................................................................................................... 14
NSX for vSphere Networking and Security Logical Switch Networks ......................................... 15
NSX for vSphere Networking and Security Dynamic Routing Capabilities using the NSX for
vSphere Edge Appliance ........................................................................................................... 18
OSPF Protocol .............................................................................................................................. 18
ISIS Protocol ................................................................................................................................. 19
BGP Protocol ................................................................................................................................ 19
Distributed Logical Router ........................................................................................................... 20
Distributed Logical Router: Logical View ..................................................................................... 21
NSX for vSphere Networking and Security Features of the VMware NSX Edge Services Gateway
................................................................................................................................................ 23
Network Address Translation (NAT) ............................................................................................ 23
Load Balancing ............................................................................................................................. 23
High Availability ........................................................................................................................... 24
Virtual Private Networking .......................................................................................................... 24
Layer 2 Bridging ........................................................................................................................... 25
NSX for vSphere Security .......................................................................................................... 25
NSX for vSphere Edge Firewall ..................................................................................................... 26
NSX for vSphere Distributed Firewall .......................................................................................... 26
Service Composer ........................................................................................................................ 27
Introduction to the NSX Networking and Security Architecture ....................................... 29
Overview of VMwares NSX Networking and Security Components ........................................ 30
High Level Overview .................................................................................................................... 30
The NSX for vSphere Platform Solution ....................................................................................... 30
NSX for vSphere Manger ............................................................................................................. 31
NSX for vSphere Edge Appliance ................................................................................................. 33
NSX for vSphere Security Features ............................................................................................ 35
NSX for vSphere Edge Services Gateway VPN Services ............................................................... 36
NSX for vSphere SSL VPN-Plus Services ....................................................................................... 44
NSX for vSphere Edge Services Gateway SSL VPN-Plus Secure Management Access Server ...... 46
L2 VPN and Stretched Logical Networks ...................................................................................... 46
NSX for vSphere Edge Services Gateway L2 VPN Use Case ......................................................... 47
NSX for vSphere Edge Services Gateway L2 VPN Topology ......................................................... 48
NSX for vSphere Edge Services Gateway Firewall ...................................................................... 49
Flow Monitoring ....................................................................................................................... 53
NSX for vSphere Hardening Guide ............................................................................................ 53
Deployment of NSX for vSphere ...................................................................................... 56
Installation of Management Plane ............................................................................................ 56
Installation of Control plane ..................................................................................................... 59
Installation of Data Plane ......................................................................................................... 63
Edge Services Gateway .................................................................................................... 90
Scalability and Redundancy NSX Edge ...................................................................................... 90
Path Determination .................................................................................................................. 91
Failure scenario of Edge Devices ............................................................................................... 93
 Troubleshooting and visibility .................................................................................................. 94
Useful CLI for Debugging ECMP ................................................................................................ 97
ECMP Deployment Consideration ............................................................................................. 99
ECMP and Edge Firewall NSX ..................................................................................................... 100
NSX Edge and DRS Rules ......................................................................................................... 101
Edge NAT ............................................................................................................................... 113
SNAT .......................................................................................................................................... 113
DNAT .......................................................................................................................................... 114
Firewall rules and SNAT ............................................................................................................. 116
Firewall rules and DNAT ............................................................................................................ 116
DNAT Configuration ................................................................................................................... 116
SNAT configuration .................................................................................................................... 121
PAT ............................................................................................................................................. 124
NAT Order .................................................................................................................................. 126
Distributed Firewall ....................................................................................................... 128
Micro-segmentation ............................................................................................................... 128
How does the Distributed Firewall Operates? ......................................................................... 130
Alignment of slots for the Distributed Firewall ....................................................................... 135
What happens in case of VM mobility? .................................................................................. 136
2-Way Traffic Introspection .................................................................................................... 140
What happens in case of a failure of this DFW functionality? ................................................. 140
Configuring Fail-Safe for Distributed Firewall ............................................................................ 140
Visibility and Packet Walks ..................................................................................................... 141
Rule Priority ............................................................................................................................... 142
Distributed Firewall Static Objects ....................................................................................... 146
Distributed Firewall Dynamic Object .................................................................................... 146
Creation of Security Groups .................................................................................................... 147
Security Policy ........................................................................................................................ 149
Applied To Field .................................................................................................................. 157
Identity Firewall ..................................................................................................................... 160
Create Identity Based Firewall Rules ....................................................................................... 163
Application Level Gateway (ALG) ............................................................................................ 165
Backup and Recovery ............................................................................................................. 166
Working with Distributed Firewall API .................................................................................... 169
Troubleshooting ..................................................................................................................... 173
Integration of NSX for vSphere with Cisco UCS .............................................................. 180
Cisco Unified Computing System Architecture ........................................................................ 180
Fabric Interconnect .................................................................................................................... 182
Chassis ....................................................................................................................................... 183
Fabric Extender .......................................................................................................................... 183
Blade Servers ............................................................................................................................. 184
Network Adaptors ..................................................................................................................... 185
UCS Manager ............................................................................................................................. 186
Running NSX on UCS ............................................................................................................... 186
Hardware Requirements ........................................................................................................... 187
Software Requirements ............................................................................................................. 187
VLAN and IP Addressing ............................................................................................................. 189
Distributed Logical Router ............................................................................................. 191
Overview of the Distributed Logical Router (DLR) ................................................................... 191
DLR Interfaces type ................................................................................................................ 193
 Logical Interfaces, virtual MAC & Physical MAC ...................................................................... 194
DLR Kernel module and ARP table .......................................................................................... 195
DLR and local routing ............................................................................................................. 196
Multiple Route Instances ........................................................................................................ 198
Logical Router Port ................................................................................................................. 199
Routing information Control Plan Update Flow ...................................................................... 200
DLR Control VM communications ........................................................................................... 201
DLR High Availability .............................................................................................................. 203
Protocol Address and Forwarding Address ............................................................................. 204
DLR Control VM Firewall ........................................................................................................ 205
DLR Validation Scenarios ........................................................................................................ 206
DLR Control Plane validation .................................................................................................. 206
Validate Routing in DLR ............................................................................................................. 207
NSX-v Controller ..................................................................................................................... 210
Verify LIFs exist on the NSX-v Controller ................................................................................... 211
ESXi Host ................................................................................................................................ 212
Validate the DLR LIFs are present on the ESXi host .................................................................. 214
Validate the Forwarding table on the ESXi host ........................................................................ 215
Validate ARP Entries in the ESXi host of the DLR ....................................................................... 216
Capture Packets on the VDR Port ........................................................................................... 218
Dynamic Routing Verification ................................................................................................. 222
Whats new in NSX for vSphere version 6.2 ................................................................... 224
Introduction ........................................................................................................................... 224
Cross vCenter Deployments through Egress Optimization ........................................................ 225
Increased Support for Cross vCenter vMotion .......................................................................... 227
Introduction of Roles ................................................................................................................. 228
When is egress optimization possible? ................................................................................... 229
How is egress optimization possible? ..................................................................................... 230
NSX for vSphere Networking and Security Universal Objects .................................................. 230
Applying Firewall rules using Universal Section of Distributed Firewall ................................... 233





















About the Authors

Prasenjit Sarkar (@stretchcloud) is a Member of the CTO Ambassador Program &
Staff Solutions Architect at VMware and part of Professional Services Center of Excellence
Team where his primary role is to work directly with Product Development and Field/Partner
Sales & Delivery organizations to incubate and help bring to market the next generation of
deep technical architectures, solutions and services.

He is working on advanced product and solution incubation development, which is well in

advance of market availability. He provides expert technical architectural support and
guidance for cloud in the product incubation phase and during the development of emerging
solutions and services.

He also provides training on emerging solutions and services offerings for Field Specialists &
Consultants and Partner Sales & Delivery organizations.

He has also worked in vCloud Air R&D Team. He has an extensive background in designing
and implementing cloud solutions. He holds several certifications including VCP3/4/5,
VCAP-DCA, VCAP-DCD, VCAP-CIA, VCP-NV, VCIX-NV. He has been awarded the
VMware vExpert award for 4 years. He is also the author of the blog http://stretch-cloud.info
and Author of 4 other books including one as Amazon Best Seller. He is also part of many
inventions and research papers and have 1 Granted and 11 pending patents on his name.

I would like to thank and dedicate this book to my mom and dad.
Without their endless and untiring support, this book would not
have been possible.
Michael Haines (@michaelahaines) is a Senior Member of Technical Staff who
specializes in Cloud Networking and Security at VMware where his primary role is to
architect and implement Software-Defined Data Center (SDDC) cloud, networking and
security solutions. Michael also has extensive knowledge of VMware's SDKs and APIs, and
in particular the NSX for vSphere API, vCloud Networking and Security API and vCloud API.

Michael also authored and co-authored various books and technical papers (over 25),
including the NSX for vSphere Hardening Guide, vCloud Director Security Hardening Guide,
VMware vCloud Architecture Toolkit (vCAT I and II), and was also the co-author of the first
VMware Cloud published book, called Cloud Computing with VMware vCloud Director which
provides use cases, design considerations, and technology guidance to answer your
questions about cloud computing.

In addition to this, Michael is also a member of the DMTF's Open Cloud Standards
incubator, which develops a suite of DMTF informational specifications that deliver
architectural semantics to unify the interoperable management of enterprise computing and
cloud computing. These DMTF specifications primarily include the Open Virtualization
Format (OVF), the vCloud API and Cloud Security. Michael was also a member of the IETF,
Internet Engineering Task Force and participated most noticeably in the LDAPv3 Working
Group.

Prior to his current role at VMware, Michael was the Chief Architect of the Data Center
Optimization and Virtualization Solutions team within the Sun Microsystems Solutions
Development Group. His primary role within this group was as the Lead Architect and
Engineer. Michael focused on the Data Center infrastructure architecture design,
development and implementation of these solutions. In addition to being the Lead Data
Center Architect and Engineer, a large part of this role also involved having a
comprehensive and detailed understanding of not just Data Center Solutions, but also
Virtualization (Sun/Oracle and VMware), Cloud Computing, Cloud Security, Network
Security, Solaris and OpenSolaris (including internals) Security, Naming Services and
Identity Management.

Michael holds certifications from VMware, Oracle, and Fortinet (Network Security Expert)
and blogs on http://stretch-cloud.info.
Roie Ben Haim (@roie9876) is a PSO Consultant in VMware PSO who specializes
in Networking and Security at VMware and who is currently focused on implementing
solutions, which incorporate VMwares NSX platform as well as integrating with various
Cloud platforms on VMwares infrastructure.

Roie works in VMwares Consulting (PSO) team whose focus is on the delivery of
Networking Virtualization and Security solutions. In this role Roie provides technical
leadership in all aspects, including the installation, configuration, and implementation of
VMwares products and services. This is also includes being involved from the inception of
these project, through requirements assessment, design and deployment phases and then
into production which ensures continuity for VMwares customers.

Roie has over a 15 years of experience working on data center technologies, and providing
solutions for global enterprises, which primarily focus on Network and Security.

A highly motivated and enthusiastic MSc graduate Roie holds a wide range of industry
leading certificates, including his most recent Network Virtualization (VCDX-NV). Roie is not
only a strong team member, but is also able to demonstrate his skills and experience
working in various fields.

As a well known and respected blogger, Roie maintains an impressive blog at
http://routetocloud.com



























Ajit Sharma is a consultant at VMware Professional Services India with over 6 years
of rich experience in designing, deployment and support of virtualization and cloud based
solutions using VMware product stack. His technical experience includes Software Defined
Networking Solutions, Cloud Automation and Operation Solutions and Disaster Recovery
Solutions. He comes with prior experience which includes vulnerability assessment,
malware/spyware analysis, Security Policies assessment and auditing, Network Support &
Designs and Wintel Engineering.

Ajit currently leads the Network Virtualization practice across India to ensure delivery of
complex network virtualization designs and implementations. He holds some of the industry
leading certifications by VMware like VCAP-DCD, VCAP-CID and VCIX-NV. Having served
several industries like Telecom, retail, Healthcare, Financial Services etc. He has a proven
track record of providing impeccable consulting and advisory services to some of the large
customers across India.
Anuj Modi (@vConsultant) is a Unified Computing and Virtualization Consultant for
Cloud & Network Service team at Cisco Systems. He works with Cisco Global Clients to
strategize, plan and design, implement and deploy a secure, agile and highly automated
data center and cloud-computing infrastructure. He has more than 12 years of expertize in
Professional Services/Consulting, Post & Pre-Sales for architecting and designing of mission
critical applications and virtualization solutions.

He has lot of passion in new technology initiatives on data center and cloud automation &
orchestration, SDN, NFV, OpenStack, VMware and Cisco products.

He is an author, reviewer and blogger (anujmodi1.wordpress.com) for various communities
on Cisco and VMware and holds certifications from VMware, Microsoft, Citrix and EXIN.

He feels great pleasure in traveling and meets the people across the world. He likes cricket,
football, and mountain hiking.

I would like to thank and dedicate my first book to my family.
 1
Concepts and Components of NSX for
vSphere platform
Server virtualization has been accepted well in our IT industry over the past decade. As a result, we
have seen a completely new way of provisioning and managing workloads in the data center. This
concept of server virtualization saved businesses from spending billions of dollars.
However, the way these workloads are connected to the network has not been so agile.
The solution that every one is now looking at is to virtualize the network. Because when you
virtualize the network layer, it abstracts the physical network and simplifies the provisioning and
consumption of networking moving forward. In addition to this, security services are built-in, and do
not require purpose-built hardware, and can scale as the network expands.
NSX for vSphere removes the operational barrier the network has become for IT. Programmatic
provisioning transforms service delivery times from weeks to a matter of minutes. NSX for vSphere
does not change the laws of physics, packets do not move faster through the network. NSX for
vSphere transforms the operational model of networking, which combined with compute and storage
virtualization delivers never before possible IT speed and agility for the business. This is depicted in
this diagram below.

As the main motto of the Network Virtualization remains same, NSX for vSphere can be deployed
over any existing physical network. NSX for vSphere reproduces exactly the networking model in
software. As a result your existing application workloads operate unmodified and existing network
monitoring and troubleshooting tools view and process virtual network traffic just as if they would in
the physical network.
This chapter will describe at a high-level the concepts and components of the NSX for
vSphere Networking and Security platform.

List of topics that will be covered in the chapter:

Discuss the NSX for vSphere Networking and Security Core Management and
Control Components: Providing powerful control over the Networking and Security
functionality
Discuss the NSX for vSphere Networking and Security Logical Switch Networks:
Covering layer 2 networking and the basics of configuring, deploying, and using the
logical switch networks
Discuss the NSX for vSphere Networking and Security Dynamic Routing
Capabilities using the NSX Edge Appliance: Covering the NSX Logical Distributed
Router Appliance to establish East-West traffic and North-South traffic.
Discuss the NSX for vSphere Networking and Security Features of the VMware NSX
Edge Services Gateway: Cover all the specific components including, NAT (Network
address translation) Load balancing, High availability, Virtual private networking, which
covers Layer 2 VPN, IPsec VPN, SSL VPN-Plus and VLAN-to-VXLAN bridging
Discuss NSX for vSphere Security: This will cover the security components including
Role-based access control, NSX Edge firewall, NSX distributed firewall, NSX data
endpoint, Flow Monitoring and Service Composer

NSX for vSphere Networking and Security Core Management and

Control Components

NSX for vSphere as a product uses 3 planes to keep the functionalities modularized. It has a
Management Plane that is managed by the NSX for vSphere Manager, Control Plane that is managed
by the NSX for vSphere Controller/Controllers Cluster and Data Plane, which runs on the ESXi hosts.
This model of modularity just gives us a much greater deal of control and which allows us the
minimal effect on the functions of the other planes. This is depicted in the diagram below.
Let us describe each section of the above diagram.

Management Plane - NSX for vSphere Manager

The NSX for vSphere Manager communicates directly and securely with the vCenter Server and is
the northbound interface for the NSX for vSphere REST API and for third party applications that
integrate with NSX for vSphere, such as Palo Alto Network (PAN), TrendMicro and many more. The
NSX for vSphere Controller instances are deployed by the NSX for vSphere Manager instance
through requests to the vCenter Server system to deploy the NSX for vSphere Controller virtual
machines from OVA files. In the below image we can see that we have a NSX Manager deployed
along with just one NSX Controller. However, as a best practice of VMware we should have 3
controllers deployed.
NSX for vSphere Manager helps configure and manage logical routing services, both East-West
routing, which is handled by the Distributed Logical Router, and North-South routing which is
handled by the NSX Edge Services router. During the configuration process you have the choice to
deploy a Distributed or Centralized Logical Router. If the Distributed Logical Router is selected, the
NSX Manager instance deploys the Logical Router Control virtual machine and pushes the logical
interface configurations to each host through the controller clusters. In essence, the logical control
virtual machine is responsible for managing the routing network interactions, whereby giving the
routing table to the NSX for vSphere Manager instance.
In the case of centralized routing, the NSX for vSphere Manager deploys the NSX for vSphere Edge
services router virtual machine. The REST API interface for NSX Manager helps automate the
deployment and management of these logical routers through a Cloud management platform, such as
VMware vCloud Automation Center (vCAC), as well as a using a third-party solution or custom
cloud management platform.
The only thing actually installed in the traditional sense is NSX for vSphere Manager. NSX for
vSphere Manager handles all the management tasks, including its initial configuration, such as setting
up SSO, NTP and other core services. There is a direct correlation of one vCenter Server system to
one NSX for vSphere Manager, so if vCloud Automation Center is present with multiple vCenter
Server systems, each of those vCenter Server systems has an NSX for vSphere Manager instance.
NSX for vSphere deploys into vSphere Clusters. The NSX for vSphere platform has a few basic
requirements. Any server on which you can install ESXi 5.5 can run NSX for vSphere on, connected
to any physical network. Multicast over the physical infrastructure is an added benefit but not
required. After you deploy the NSX for vSphere Manager, you deploy the NSX for vSphere
Controller instances, VIBs, and then configure the virtual networks and security services.
When you install the NSX for vSphere Manager it has the OVA files to deploy the NSX for vSphere
Edge gateways, the NSX for vSphere Controller, and the VIBs that get pushed to the ESXi hosts for
the distributed switches. The NSX for vSphere Manager also uses the REST API when
communicating to and from third party applications such as firewalls from Palo Alto Networks and
Checkpoint to name but a few and other networking and security services. The REST API is used
extensively by the broader networking and security services ecosystem, and it is primarily this
interface that used to integrate with the NSX for vSphere platform.

Control Plane NSX for vSphere Controller

Before we start explaining the concept of the control plane and NSX for vSphere controller, let us
look at various terminologies that we will use through out this section.
Controller Cluster A set of virtual units operating in unison to perform distributed
controller functions.
Controller Node A single virtual unit of controller cluster member, each node can have
one or more role. Each node can be upgraded, powered down and fail independently
Transport Node A virtual entity outside the controller-performing packet forwarding
functions and interacting with controller for logical network topology

The NSX for vSphere controller is responsible for two key functions. Firstly it serves as a control
plane for all the forwarding elements (transport nodes) and it exposes REST APIs for the
consumption of northbound entities, which could be vCAC or OpenStack for example, or any third
party management and monitoring infrastructure. The NSX for vSphere Controller internally
maintains states of the logical network and its mapping to the physical infrastructure. This logical
state is then distributed to the respective transport nodes thus allowing VMs (and physical hosts) to
see logical networks. Northbound entities can then obtain the states and operational visibility of the
entire logical network from the NSX for vSphere controller.
So, in essence the NSX for vSphere Controller provides the control plane to distribute VXLAN and
Logical Routing network information to ESXi hosts. The Controller cluster also provides multi-node
high-availability, whereby every controller node is Active (as opposed to an Active-Standby model,
providing for both scale out and high availability. It is important to note that say for example you
loose one of the controller nodes, then once the controller node comes back online it then re-syncs the
forwarding plane entries (flows) without any data plane outage. Also and just as important, is the
security of all the control messages between the Controllers and transport nodes, which are encrypted.
In addition to this the Controller also provides an Audit Log facility for security compliance.
Major Problems today that we are facing in terms network virtualization are:
Need to dynamically distribute workloads across all available cluster nodes
Redistribute workloads when a new cluster member is added
Ability to sustain failure of any cluster node with zero impact
Perform all of the above transparent to application

Solution for this problem is using slicing. The first NSX for vSphere Controller instance deployed
requests a password and all future NSX for vSphere Controller instances deployed use this password.
A user then uses this password to SSH into the NSX for vSphere Manager or NSX for vSphere
Controller. The NSX for vSphere Controller must be connected to the same vCenter Server system as
NSX for vSphere Manager. It is recommended that the NSX for vSphere Controller instances be
deployed in clusters of three nodes, which is currently the supported maximum. The reason is that the
NSX for vSphere Controllers can scale with no trouble to the limit of the vCenter Server, which is
currently 10k active VMs and 10k logical switches. So, apart from not being supported, at this point
there is no value is say adding 5 NSX for vSphere Controllers, and that is why we currently support 3
controllers.
Each NSX for vSphere Controller instance in a cluster must be deployed individually. With NSX for
vSphere there is currently a requirement to synchronize time (NTP) between the ESXi hosts and the
NSX for vSphere Manager for deployment of the controllers.

Data Plane

The Distributed Switch that is provided by the vSphere platform fundamentally defines the Data
plane. The distributed switch only performs layer 2 switching and hosts have to be on the same layer
2 network for virtual machines on each host to communicate with virtual machines on the other host.
NSX for vSphere installs three vSphere Installation Bundles (VIB) to the host to enable the NSX for
vSphere functionality. One VIB enables the layer 2 VXLAN functionality and another VIB enables
the distributed router, while the final VIB enables the distributed firewall. After adding the VIBs to a
distributed switch that distributed switch is then referred to as an NSX Virtual Switch. On an NSX
Virtual Switch, the hosts are not restricted to being on the same layer 2 domains for virtual machine to
virtual machine communication across hosts.
The NSX for vSphere Edge Services Gateway (ESG) is not distributed so it does not have a control
entity. The NSX for vSphere ESG handles control traffic itself.
NSX for vSphere Networking and Security Logical Switch Networks

Historically we have noticed some of the challenges in the traditional network that we wanted to
eliminate. Those were:
Multi tenancy in network segment
L2 stretch for VM mobility across clusters/datacenters
Network sprawl in a large L2 environment because of STP
So we wanted to eliminate these using logical switch networks that bring VXLAN into the picture.
Let us look at the various different VXLAN terminologies.

VTEP A Virtual Tunnel End Point (VTEP) is an entity that encapsulates an Ethernet
frame in a VXLAN frame or de-encapsulates a VXLAN frame and forwards the inner
Ethernet frame.
VTEP Proxy A VTEP proxy is a VTEP that forwards VXLAN traffic to its local
segment from another VTEP in a remote segment.
Transport Zone A transport zone defines members or VTEPs of the VXLAN overlay:
o Can include ESXi hosts from different vSphere clusters
o A cluster can be part of multiple transport zones
VNI - A VXLAN Number Identifier (VNI) is a 24-bit number that gets added to the
VXLAN frame:
o The VNI uniquely identifies the segment to which the inner Ethernet frame belongs
o Multiple VNIs can exist in the same transport zone
o VMware NSX for vSphere starts with VNI 5000

Before we go into some of the high level details of VXLAN, lets take a brief look at VXLANs
history.
Pre-VXLAN: DVFilter CHF (VCDNI), BFN, Overlay CHF, vTunnel
Introduced in vCloud Networking and Security v5.1 (SPOCK) (SP1) tech preview for
selected service providers in 2012
vCloud Director v5.1 (T2) optimization in full network stack and platform 2012
2013 release VXLAN control plane!
o A separate (reliable and secure) control plane to distribute VXLAN mapping.
o Suppress broadcast/multicast in VXLAN network:
Replace conventional broadcast / multicast based control plane protocols.
Remove dependency on physical multicast routing / PIM.
o Discover and publish virtual network information.
o Broadcast suppression (replacing bcast/mcast CP protocols)
 ARP (85% broadcast traffic), RARP, DHCP, IGMP, etc.
Optimize VM multicast
Multicast proxy - fully distributed software replication.

Virtual eXtensible Local Area Network (VXLAN) is a network overlay that encapsulates layer 2
traffic within layer 3. VXLAN provides us the following benefits:

It provides us unique capability of managing overlapping addresses between multiple

tenants
It supports vMotion of your virtual machines that is independent of the physical network
Unlimited number of virtual networks
It provides us the capability to decouple the network service provided to servers from the
technology used in the physical network
It provides us a unique capability to isolate the issues such as MAC table size in physical
switches.
VXLAN provides up to 16 million virtual networks in contrast to the 4094 limit of
VLANs
Application agnostic, all work is performed in the ESXi host.

Let us take a look at the high level physical deployment of VXLAN as the following illustration.
VXLAN has gone through some modifications in NSX for vSphere. Let us take a look at the table to
understand what are the changes it has gone through in both Data Plane and Control Plane.

Data Plane Control Plane

Support for multiple VXLAN vmknics per host A highly available and secure control plane to
to provide additional options for uplink load distribute VXLAN network information to
balancing ESXi hosts

DSCP & COS Tag from internal frame copied Removes dependency on multicast routing/PIM
to external VXLAN encapsulated header in the physical network

Dedicated TCP/IP stack for VXLAN Suppress broadcast traffic in VXLAN networks

Ready for VXLAN hardware offloading to
network adapters (in future)

Unlike vCloud Networking and Security Manager, NSX for vSphere uses three different modes of
VXLAN traffic replication.
Multicast
Unicast
Hybrid

Replication mode relates to the handling of broadcast, unknown unicast, and multicast (BUM) traffic.
Unicast has no Physical Network requirements apart from the MTU. All traffic is replicated by the
VTEPs. In the same VXLAN segment its the source VTEP, in remote VXLAN segments the NSX
Controller selects a proxy VTEP. Hybrid mode uses IGMP layer 2 multicast to offload local
replication to the physical network. Remote replication uses unicast proxies so there is no need for
multicast routing. Hybrid is recommended for the majority of deployments. Multicast is seen
frequently in upgrade scenarios from vCloud Networking and Security v5.x or environments that
already have multicast routing. The following image illustrates these three available options while
creating a logical switch.
The logical switch is a distributed port group on the distributed switch. The Logical switch can
expand distributed switches by being associated with a port group in each distributed switch. The
creation of the port group is performed by the vCenter Server on behalf of the NSX for vSphere
Manager. vSphere vMotion is supported but only among those hosts that are part of the same
distributed switch.

NSX for vSphere Networking and Security Dynamic Routing

Capabilities using the NSX for vSphere Edge Appliance

The distributed routing capability in NSX for vSphere provides an optimized and scalable way of
handling East-West traffic of a data center. The NSX for vSphere Edge services router provides the
traditional centralized routing support in the NSX for vSphere platform.

The TCP/IP protocol suite offers different routing protocols that provide a router with methods for
building valid routes. This module will discuss three routing protocols:
OSPF: Open Shortest Path First is a link state protocol, which uses a link state routing
algorithm and is an interior routing protocol.
IS-IS: Intermediate System to Intermediate System determines the best route for
datagrams through a packet switched network.
BGP: Border Gateway Protocol is an exterior gateway protocol that is designed to
exchange routing information between autonomous systems on the Internet.

OSPF Protocol

OSPF is a link-state protocol meaning each router maintains a database describing the Autonomous
Systems topology. When you enable OSPF there are two areas created by default. Area 0 and area
51. Area 51 can be deleted and replaced with a desired area.
By default, OSPF adjacency negotiations happen in clear authentication assuming trust in the
segment. If installed in an insecure segment, enabling authentication ensures a third party cannot
corrupt the routing table or hijack connection by injecting a compromised default route.
OSPF maintains a link-state database that describes the autonomous systems topology. Each
participating router has an identical database. The router shares this database with routers in the
autonomous system by a mechanism knows as flooding. All routers in the autonomous system run the
exact same algorithm used to construct the shortest path between itself and the root. This in turn gives
each router the route to each destination in the autonomous system. When multiple paths to a
destination exist and those paths are equal cost, traffic is distributed equally among those paths.
Sets of networks grouped together are called areas. Areas are a collection of routers, links, and
networks that have the same area identification. Each OSPF area can combine with other areas and
form a backbone area. Backbone areas combine multiple independent areas into one logical routing
domain. This backbone area is given the ID of 0 or (0.0.0.0). The primary responsibility of the
backbone area is to distribute routing information between non-backbone areas.

ISIS Protocol

Intermediate System to Intermediate System (IS-IS) is an inter-domain dynamic routing protocol used
to support large routing domains. OSPF is designed to support only TCP/IP networks whereas IS-IS
started as an ISO protocol. Both protocols are interior gateway protocols (IGP) but IS-IS runs over
layer 2 and was intended to support multiple routed protocols.
IS-IS uses a two level hierarchy for managing and scaling large networks. A routing domain is
partitioned into areas. Level 1 routers know the topology of their area including all routers and
endpoints in their area. Level 1 routers do not know the identity of routers or destinations outside their
areas. Level 1 routers forward all traffic outside of their area to a level 2 router in their area.
Level 2 routers know the level 2 area and know which addresses it can reach by contacting other level
2 routers. A level 2 router does not know the topology of a layer 1 area. Level 2 routers can exchange
packets or routing information directly with external routers located outside of the routing domain.
Level 1 routers belonging to a level 1 area only form neighbor adjacencies with level 1 routers in the
same area and have full visibility of their area. Level 2 routers belonging to a level 2 area can form
neighbor adjacencies with any level 2 router, including in other areas and advertise inter-area routes.
Level 1-2 routers belong to both level 1 and level 2 areas at the same time. Similar to OSPFs area
border router, level 1-2 routers can form neighbor adjacencies with any other router in any area. Level
1-2 router takes level 1 area routing updates and propagates them to level 2 areas and vice versa. Only
level 2 routers can connect to an external network.

BGP Protocol

The Border Gateway Protocol (BGP) is an inter-autonomous system routing protocol. There are two
different kinds of BGP. Internal and external BGP known as iBGP and eBGP. External BGP is used
when talking to a router that has an AS number different from its own. Internal BGP is used with
routers within the same local AS. Neighbor level configurations allow a variety of settings to be
configured to customize the BGP configuration.
An autonomous system is a set of routers under a single technical administration, using an interior
gateway protocol (IGP) and common metrics to determine how to route packets in the AS, and using
an inter-AS routing protocol to determine how to route packets to other autonomous systems. Each of
these autonomous systems is uniquely identified using an autonomous system number (ASN).
Peers are manually configured to exchange routing information and form TCP connections. There is
no discovery in BGP. A peer in a different AS is referred to as an external peer, while a peer in the
same AS is referred to as an internal peer.

Distributed Logical Router

Routing between virtual networks, layer 3 is distributed in the hypervisor. The distributed logical
router optimizes the routing and data path, as well as supports single tenant or multitenant
deployments. For example, a network that contains two VNIs that have the same IP addressing. This
requires deploy two different distributed routers with one distributed router connecting to tenant A
and one to tenant B.

Distributed routing is provided by a logical element called Distributed Logical Router (DLR),
which has two main components:

DLR control plane is managed through the DLR control VM that supports dynamic
routing protocols, which is in our case BGP & OSPF. Its main function is to exchange
routing updates with the next layer 3 hop device (usually the NSX Edge) and
communicates with the NSX Manager and the Controller Cluster. You can also achieve
HA for the DLR Control VM through Active-Standby configuration.

Kernel modules known as VIBs will be installed on the ESXi hosts and form the data-
plane level. These modules have routing information that is pushed through the controller
cluster. These modules will do the route lookup, ARP entry lookup. The kernel modules
are equipped with logical interfaces (called LIFs) connecting to the different logical
switches. Each LIF has assigned an IP address representing the default IP gateway for the
logical L2 segment it connects to and a vMAC address. The IP address has to be unique
per LIF, whereas same vMAC can be assigned to all the defined LIFs.

A logical representation of this Logical Distributed Router is depicted below.

Without the distributed router, routing is done in one of two ways. The first method uses a physical
appliance, which means all traffic has to go to a physical appliance and come back regardless of
whether or not the virtual machines are on the same host.

The other method performs routing on a virtual router such as the NSX for vSphere Edge gateway.
This method uses a virtual machine running on one of the hosts to act as the router.
If virtual machines running on a hypervisor are connected to different subnets, the communication
between these virtual machines has to go through a router. This non-optimal traffic flow is sometimes
called hair pinning.

Distributed Logical Router: Logical View

The distributed logical router routes between VXLAN subnets. If two virtual machines are on the
same host and the Web VM on VXLAN 5001 wants to communicate with the App VM on VXLAN
5002, the distributed logical router will route traffic between the two virtual machines on the same
host. This logical representation has been depicted in the following illustration.
The distributed logical router can also route between physical and virtual subnets.
The NSX for vSphere Manager configures and manages the routing service. During the configuration
process, NSX for vSphere Manager deploys the Logical Router Control VM and pushes the logical
interface configurations to each host through the control cluster.
The Logical Router Control VM is the control plane component of the routing process. It supports the
following dynamic routing protocols:
OSPF
BGP

There is a kernel module required for the logical router and it is configured as part of the preparation
through the NSX for vSphere Manager. You can think of these kernel modules as line cards in a
modular chassis supporting layer 3 routing. This kernel module stores the routing information, which
is pushed from the controller cluster.
The controller cluster is responsible for distributing routes learned from the Logical Router Control
VM across the hypervisors. Controller nodes in a controller cluster divides the load of distributing the
routes information when there is multiple distributed logical router instances deployed.
NSX for vSphere Networking and Security Features of the VMware
NSX Edge Services Gateway
VMware NSX for vSphere provides you several Edge Gateway features. Those are:

NAT (Network Address Translation)

Load Balancing
High Availability (HA)
Virtual Private Network (VPN)
Layer 2 Bridging
Domain Name Service (Authoritative DNS)
Dynamic Host Configuration Protocol (DHCP Relay)
Firewall
Syslog

In this section we will take you through the new features those are newly introduced in NSX for
vSphere 6.1. Other features such as DNS, DHCP Relay, Firewall and Syslog remains the same as
it was in vCloud Networking and Security. However, the firewall feature is explained in the
below section within NSX Security.

Network Address Translation (NAT)

NSX for vSphere Edge provides network address translation (NAT) service to assign a public address
to a computer or group of computers in a private network. Using this technology limits the number of
public IP addresses that an organization or company must use, for economy and security purposes.
You must configure NAT rules to provide access to services running on privately addressed virtual
machines. The NAT service configuration is separated into source NAT and destination NAT rules.
There are mainly two types of NAT services it provides.
Source NAT - Source NAT is used to translate a private internal IP address into a public
IP address for outbound traffic.
Destination NAT - Destination NAT is commonly used to publish a service located in a
private network on a publicly accessible IP address.

Load Balancing

The NSX for vSphere Edge load balancer distributes incoming requests evenly among multiple. Load
balancing helps in achieving optimal resource utilization, maximizing throughput, minimizing
response time, and avoiding overload. NSX for vSphere Edge Devices provides load balancing up to
layer 7.
The load balancer does not perform global balancing; it only performs local load balancing. If you
have multiple virtual machines providing a Web service, the load balancer, which is a service of the
NSX for vSphere Edge device, can provide load balancing across those virtual machines. Another
feature is that when one of the virtual machines being load balanced to becomes unreachable, or the
service becomes unresponsive, the load balancer service detects that condition and removes that Web
server from the load balance rotation.
Clients do not open a Web browser and go to the IP address of the Web server. Instead the client
points to an IP that is owned or hosted by the load balancer itself. The load balancer then takes the
client traffic and redirects it by changing the destination IP address from the load balancers IP address
to the IP address of the Web server that was selected to establish your session. The IP address that
was used by the client to connect to the Web site is called the virtual IP (vIP).

High Availability

The NSX for vSphere Edge High Availability (HA) feature ensures that an NSX for vSphere Edge
appliance is always available by installing an active pair of NSX for vSphere Edge gateways on your
virtualized infrastructure. You can enable HA either when installing an NSX for vSphere Edge
appliance or on an already installed/and or deployed NSX for vSphere Edge instance.
Primary Edge appliance is in the active state and the secondary appliance is in the standby state.
Primary Edge Appliance replicates the configuration to the standby appliance. It is VMware best
practice to deploy the primary and secondary appliances on separate resource pools and datastores, so
that the underlying infrastructure does not become single point of failure for the edge appliances.
High Availability (HA) ensures that a NSX for vSphere Edge appliance is always available on your
virtualized network. NSX Edge High Availability supports two NSX Edge appliances (peers) per
cluster, running in active-standby mode.
NSX Manager manages the lifecycle of both peers and pushes user configurations as they are made to
both NSX Edge instances simultaneously.
NSX Edge High Availability peers communicate with each other for heartbeat messages as well as
runtime state synchronization. Each peer has a designated IP address to communicate with the other
peer. The IP addresses are for HA purposes only and cannot be used for any other services. The IP
addresses must be allocated on one of the internal interfaces of the NSX Edge.
Heartbeat and data synchronization both use the same internal vNIC. Layer 2 connectivity is through
the same port group.
The heartbeat and sync channel requires L2 adjacency. The internal Portgroup is a vNic of type
internal on the NSX for vSphere Edge, which can be shared or dedicated, but not on a management
network.
When and if a failover occurs there is a heartbeat deadtime, which is the latency to detect a dead peer.
Beyond that, there is extra latency for the Standby to completely take over, including reconfiguration
and restarting non-hot standby services like VPN. This extra latency depends on NSX for vSphere
Edge configuration. TCP flows can tolerate the downtime but there will be packet loss for udp traffic.
In NSX for vSphere, the default (minimum) deadtime is increased to 15 seconds since there are
customer cases where a 6 second deadtime is too short.

Virtual Private Networking

In NSX for vSphere v6.1 you will see the following VPN:
 SSL VPN Plus SSL VPN-Plus being a user based solution; you can use one Edge and
configure SSL VPN on it. Then the Private network behind the edge will be reachable via
the end users machine once connected via the SSL VPN client.
IPSec VPN For IPSec VPN you can configure two edges and create a site-to-site tunnel
between the two NSX for vSphere edges. Then the networks behind the two edges would
be reachable to each other. This is a Layer 3 Site-to-Site solution and provides the ability
to interconnect two different networks.
L2 VPN For L2 VPN this is used to extend a network across boundaries such that the
VMs being extended are not aware or require any change in their routing or MAC
addresses etc. This is a Layer 2 Site-to-Site solution and basically lets you move VM's
between two different networks without reconfiguring the VMs. And L2VPN is used by
vCloud Connector for Data Center Extensions.

Layer 2 Bridging

In Layer 2 bridges, you create a bridge between a logical switch and a VLAN. This helps you to
migrate virtual workloads to physical devices with no impact on IP addresses. A logical network can
leverage a physical gateway and access existing physical network and security resources by bridging
the logical switch broadcast domain to the VLAN broadcast domain.
Bridging can also be used in a migration strategy where you might be going P2V and you don't want
to change subnets.
VXLAN to VXLAN bridging or VLAN to VLAN bridging is not supported. Bridging between
different data centers is also not supported. All of the participants the VLAN and VXLAN bridge
must be in the same data center.
The layer 2 bridge runs on the host that has the NSX Edge logical router virtual machine. The layer 2
bridging path is entirely in the VMkernel. This is where is where the sink port connection is for the
distributed router to connect to the distributed port group. The sink port steers all interesting traffic
related to bridging on to the switch. You cannot have routing enabled on those interfaces that you
connect to the distributed router.
The distributed router that performs the bridging cannot perform routing on that logical switch. This
means the virtual machines on that switch cannot use the distributed router as their default gateway.
Because logical switches cannot have more than one distributed router connected to a logical switch,
those virtual machines have to have a default gateway either outside in the physical network or in an
appliance, such as the NSX Edge gateway, connected to the logical switch on the port group.

NSX for vSphere Security

NSX for vSphere Security feature comes with several dynamics. In this section we will cover:
NSX for vSphere Edge Firewall
NSX for vSphere Distributed Firewall
Service Composer
NSX for vSphere Edge Firewall

The NSX for vSphere Edge Appliance provides a stateful firewall for North-South and East-West
traffic flows.
The NSX for vSphere Edge supports stateful firewalling capabilities, which complements the
Distributed Firewall (DFW) enabled in the Kernel of the ESXi hosts. While the DFW is
mostly utilized to enforce security policies for communication between workloads connected
to logical networks (the so called east-west communication), the firewall on the NSX Edge is
usually filtering communications between the logical space and the external physical network
(north-south traffic flows).

NSX for vSphere Edge Firewall Features:

Industry-standard UI Optimized for NetOps

Advanced Connection Tracking
Attack and Malformed Packet checks
Object-based rules
Multiple objects per rule
Application Definition Groups
VMware Dynamic Objects
Cert-compliant logging and structured syslog
Traffic Stats per rule
Comment & Name fields for Change Management

The NSX for vSphere Edge services gateway provides several virtual machine form factors.

Total Number of Number of

Size vCPU RAM Firewall Connections Firewall Rules Comments
Compact 1 64 MB 64000 2000 Suitable for basic Firewall
Suitable for medium level
Large 2 1 GB 1000000 2000 Firewall
Suitable for high
performance Firewall +
XLarge 6 8 GB 1000000 2000 Load Balancer

Suitable for high

Quad Large 4 1 GB 1000000 2000 performance Firewall

NSX for vSphere Distributed Firewall

The firewall has evolved in recent years. Originally the firewall was a physical device that was placed
at the perimeter of the network to inspect traffic entering the data center.
The next stage in the evolution was firewall appliances running in virtual machines. From a
hypervisor perspective, one virtual machine was talking to another virtual machine. The virtual
machine acting as the firewall had to be the default gateway for the other virtual machines running on
that host. In some cases, firewalls also used to run inside the virtual machine to provide an additional
layer of security.
The Distributed Firewall is a hypervisor kernel-embedded firewall that provides visibility and control
for virtualized workloads and networks. The Distributed Firewall offers multiple sets of configurable
rules for network layers 2, 3, and 4. The Distributed Firewall focuses on East-West access. The
following image illustrates the logical representation of the Distributed Firewall.

NSX Distributed Firewall provides security-filtering functions on every host, inside the hypervisor
and at kernel level.
NSX Distributed Firewall offers a centralized configuration (using the vSphere Web Client) with
distributed enforcement of policy rules (i.e applied down to vNIC level). It integrates Spoof Guard
functionality to prevent spoofing in virtual environment and protects against IP and MAC spoofing.
The Distributed Firewall provides distributed enforcement of policy rules. The Distributed Firewall is
configured using the vSphere Web Client. The Distributed Firewall is independent of the distributed
router.
The Distributed Firewall is meant for East-West traffic or horizontal traffic. The NSX for vSphere
Edge firewall focuses on the North-South traffic enforcement at the tenant or data center perimeter.
The Distributed Firewall policy is independent of where the virtual machine is located. If a virtual
machine is migrated to another host using vMotion, the firewall policy will follow the virtual
machine.
Distributed firewall rules are enforced at the vNIC layer before encapsulation or after decapsulation.
The distributed firewall policies are independent of whether a virtual machine is connected to a
VXLAN or VLAN. Distributed firewall rules are independent of virtual machine location.
The Distributed Firewall can enforce rules even if the virtual machines are on the same layer 2
segment. Policy rules always follow a virtual machine if the virtual machine is migrated to another
host.

Service Composer

Service Composer helps you provision and assign network and security services to applications in a
virtual infrastructure. This is illustrated in the following diagram below.
You map services to a security group, and the services are applied to the virtual machines in the
security group. Define security policies based on service profiles already defined (or blessed) by the
security team. Apply these policies to one or more security groups where your workloads are
members.
A security policy is a collection of the following service configurations:
Firewall rules that define the traffic to be allowed to, from, or within the security group
that apply to vNICs.
Endpoint service, which is data security or third party solution provider services such as
antivirus or vulnerability management services that apply to virtual machines.
Network introspection services, which are services that monitor your network such as
intrusion prevention systems that apply to virtual machines.

If a virtual machine belongs to more than one security group, the services that are applied to the
virtual machine depends on the precedence of the security policy mapped to the security groups.
Traffic leaves the virtual machine and is sent to the integrated partner product. Some partners have
integrated products into NSX. This traffic flow happens before the traffic reaches the network.

Summary

In this chapter we have discussed an overview and high-level representation of network virtualization
concepts and where NSX for vSphere fits in. We have discussed about NSX for vSphere core
components, logical switching, distributed routing, edge gateway features and security from a high-
level. In the next chapter we will introduce the overview of the NSX for vSphere security architecture,
NSX for vSphere networking and security connectivity. Also we will discuss about the use cases of
NSX for vSphere networking and security features.





 2
Introduction to the NSX Networking
and Security Architecture
In this chapter we are going to focus on the VMware NSX for vSphere networking and security
platform which delivers the entire networking and security model in software, and which is decoupled
from the traditional networking hardware, representing a transformative leap forward in a data center
networking and security architecture.
Networking and Security is fundamentally a very real and serious concern for any organization today,
and the thought of a security breach of any type is unimaginable! In this ever-increasing multi-tenant
environment, it is just not viable that any organization could have data from a virtual machine that
exists in one tenant being accessed by another virtual machine from another tenant. One of the core
features of network and security virtualization is the ability to provide extensibility, which can be
implemented in various ways, as in the case of networking; NSX for vSphere has the capability to
extend a network across boundaries such that the virtual machines (VMs) being extended are not
aware or require any change in their routing or MAC addresses. This is a layer 2-network extensibility
solution and basically lets you move virtual machines (VMs) between two different networks without
reconfiguring the virtual machines (VMs). While this ability to dynamically provision networks and
associated security services across the entire data center is a critical factor for any organization today,
it does potentially introduce and open up the data center to a myriad of security risks.

With the introduction of NSX for vSphere in your organization, the network and security
virtualization is abstracted from the application workload communication and from the physical
network hardware and topology, allowing the network security to break free from these physical
constraints and whereby we can apply granularity in the network security, based on user, application,
and business context.

When implementing any networking and security virtualization technology, such as NSX for vSphere,
organizations need to ensure that they can continue to maintain a secure environment and one that
meets their regulatory and compliance obligations. In order to achieve this, organizations are
required to evaluate the risks that might affect protected information and mitigate those risks through
risk-appropriate standards, processes, and best practices.

VMware considers standards compliance as one of the main motivations behind the management of
information security. Compliance requires you by law or rule to abide by specific standards meant to
protect specific types of information. Over the past several years, the compliance landscape in IT
security has become quite complex. Many governments and industry associations worldwide have put
in place regulations and standards that mandate the protection of various types of information. For
example, in the United States, there are industry-specific privacy regulations, such as the Health
Insurance Portability and Accountability Act (HIPPA) in health care and the Gramm-Leach-Bliley
Act (GLBA) in financial services, as well as a patchwork of state regulations that dictate requirements
for handling personal information. Companies operating in Europe also have to abide by the European
Unions Data Protection Directive and its specific implementations by individual countries. Privacy
regulations are also now emerging in other geographies, such as India and China. Globally, credit card
data must be protected as per the Payment Card Industry (PCI) Data Security Standard.
With a continuous stream of new, such as the Criminal Justice Information Services, CJIS or updated
regulations and standards on the horizon, the compliance landscape will continue to be complex.

Overview of VMwares NSX Networking and Security Components

We are now going to turn our attention and look at the different components that make up VMwares
NSX for vSphere Software Networking and Security Virtualization Platform. We will primarily focus
on the NSX for vSphere Edge, Distributed Firewall, Flow Monitoring Role-Based Access Control,
Service Composer and Monitoring options available for you and your organization to use. This is by
no means a complete inventory of all the network and security features that are available for you to
consume in the NSX for vSphere product, merely an appetizer!

High Level Overview

VMwares NSX for vSphere is undeniably the leading network and security virtualization platform in
the industry today. Similar to the concept of virtual machines for compute, virtual networks and
security services are programmatically provisioned and managed independently of the underlying
hardware. NSX for vSphere reproduces the entire network and security model in software, enabling
any network topology from simple to complex multi-tier networks to be created and provisioned in a
matter seconds. It enables a library of logical networking elements and services, such as logical
switches, routers, firewalls, load balancers, VPN, and workload security. Users can create isolated
virtual networks and security services through using custom combinations of these features and
capabilities.
When you look at VMwares NSX for vSphere from a high level view you can start to picture and
imagine the power of network and security virtualization, and the benefits of abstracting these
network and security functions into software which provides a rich and diverse amount of control,
scalability and automation to such a solution. VMware s NSX for vSphere product and solution is
one of the key building blocks of a software-defined data center (SDDC) approach.

The NSX for vSphere Platform Solution

Before we take a look at some of the individual network and security components of the NSX for
vSphere product, let us take a brief look at what it takes and the process needed to build the NSX for
vSphere Platform solution.
Before we take a brief look at some of the individual network and security components, lets take a
look at the BIG NSX for vSphere component picture, which depicts the core technology used in
delivering these network and security features.
Diagram 1 NSX for vSphere Component Overview



NSX for vSphere Manger

It all starts with NSX for vSphere Manager, which as you see in Diagram 1 NSX for vSphere
Component Overview; the NSX manager sits in the management plane and is responsible for various
aspects of configuration, such as the Appliance itself, which includes Time Settings, Syslog Server
Settings, Network Settings, TLSv1.2/SSL Certificate Settings, Backup and Restore Settings,
Upgrading to newer release Settings. In addition to this, there is also the NSX for vSphere
Management Service Components, which provides the configuration settings for both the vCenter
Server and SSO Lookup Service.

The NSX for vSphere Manager communicates with a singular vCenter Server and is the interface that
is used for the NSX for vSphere API as well as for third-party applications that integrate with NSX
for vSphere. As you can see here, the NSX for vSphere Manager is handling all the management tasks
and as stated above there is a one direct correlation of one vCenter Server to one NSX for vSphere
Manager. As an example, if the vRealize Automation Center (formerly vCloud Automation Center) is
present with multiple vCenter Servers, each of those vCenter Servers will have an NSX for vSphere
Manager instance.

When you initially log in to the NSX for vSphere Manager UI here is what you will see:

Diagram 2 NSX for vSphere Manager UI

After logging in to the NSX for vSphere Manager, you would go the Manage Appliance Settings
where you would configure the initial settings as mentioned previously.

The following points highlight the main functions of the NSX for vSphere Manager, which are:

A single point of configuration and control for the NSX for vSphere platform solution

Packaged as a Virtual Appliance, containing all components required to run NSX on vSphere

Deployed in the vCenter Server to provide the NSX for vSphere functionally

Deployed with one instance per vCenter Server

The vSphere Web Client UI is a required component

Optionally configured with SSO server to share authentication with the vCenter Server

The following diagram represents the relationship between the different components.

Diagram NSX for vSphere Manager User Interface

As you can see in the diagram, the NSX for vSphere components require a number of ports to be open
in order for successful communication between the NSX for vSphere Manager, vCenter Server and
REST client that may be used:
Port 443 between a REST client and NSX for vSphere Manager.

Port 443 to access the NSX for vSphere Manager Management User Interface.

Port 9443 to access the vSphere Web Client

As you are reading this section and looking at the Diagram 3, the NSX for vSphere Manager User
Interface you may be asking yourself, what about high availability of the NSX for vSphere Manager!
Is that not a single point of failure? You would be right in thinking this and need to take the necessary
measures in order to make sure the NSX for vSphere Manager is highly available. A common
approach and guideline to this would be to use the vSphere HA feature, as after all, the NSX for
vSphere Manager is a virtual machine (VM).

NSX for vSphere Edge Appliance

In NSX for vSphere there are a number of networking and security capabilities that are available for
both user configuration and consumption, and can be accessed by using the NSX for vSphere
Manager Plugin that gets instantiated in the vSphere Web Client, which we will see later, and by
using the NSX for vSphere API, which is a REST API. It is also worth noting that the NSX for
vSphere features are also consumed and exposed by other VMware products, such as vRealize
Automation (formerly vCloud Automation Center) and vCloud Director. These network and security
services are once again provided through the integration with the vSphere Web Client. When you
deploy a NSX for vSphere Edge appliance you will observe that it has two personalities! What that
means, is you can configure the NSX for vSphere Edge appliance as either an Edge Services
Gateway (ESG) or Logical Distributed Router (DLR). Please see the following diagram that depicts
this.
Lets take a brief look at the capabilities that are included in the NSX for vSphere Edge Appliance:

Firewall Providing a 5 tuple rule configuration with IP, Port ranges, Grouping Objects, and
vCenter Server Containers
Network Address Translation Source and Destination NAT capabilities
DHCP Configuration of IP Pools, gateways, DNS servers and search domains
Routing Both Static and Dynamic Routing protocols are supported, including: OSPF, BGP
and ISIS
Site-to-Site VPN IPsec site-to-site VPN between two Edges or a third party vendors VPN
terminator, such as Juniper and Cisco as well as others.
SSLVPN Allow remote users to access the private networks behind the Edge Services
Gateway (ESG)
L2VPN Stretch your layer 2 networks across datacenters
High Availability Active-Standby High Availability (HA) capability which works well with
vSphere HA
DNS Allow the configuring of DNS as a relay
Syslog Allow remote syslog servers

These networking and security features in NSX for vSphere are key to providing multi-tenancy
capabilities at the virtual datacenter layer, while still supporting self-service for consumers of these
services. This is important to provide the required functionality for Enterprise and Service Providers
who are looking to build and offer secure networking and security services.
NSX for vSphere Security Features

We are now going to take a look at the NSX for vSphere Edge Appliance security features, namely
the IPsec VPN, Layer 2 VPN and SSLVPN, and in particular we are going to describe what features
are available, where you would typically deploy them and finally give you an example of how you
would configure them.
But before we do, it is important to understand that the VMware NSX for vSphere Platform is not
only the network virtualization platform but also the security platform to for the software-defined data
center. The NSX for vSphere Platform brings both Virtualization and Security to your existing
network and transforms network and security operations and economics.

The software-defined data center extends the virtualization and security concepts like abstraction,
pooling, and automation to all data center resources and services. Components of the software-defined
data center can be implemented together, or in a phased approach:

Compute virtualization, network virtualization, security, and software-defined storage deliver

abstraction, pooling, and automation of the compute, network, security and storage infrastructure
services.

Automated management delivers a framework for policy-based management of data center

application and services.

In the following diagram you can see how security is extended across the software-defined data
center.

As you are aware virtualizing the network abstracts application workload communications from the
physical network and hardware topology. This virtualization is critical in allowing network security to
break free from the physical constraints. Virtualization enables the network security to be based on
user, application, and business context.

In the NSX for vSphere Platform we can think of security that is split into the following categories:

NSX for vSphere Edge Services Gateway Firewall (VPN Services)

NSX for vSphere Distributed Firewall
NSX for vSphere Flow Monitoring
NSX for vSphere Role-Based Access Control
NSX for vSphere Service Composer

We will cover these in brief, but will spend more time on the NSX for vSphere Edge Services
Gateway Firewall (VPN Services).

NSX for vSphere Edge Services Gateway VPN Services

VMware customer experience and independent analyst research demonstrate it is possible to build a
fully virtualized DMZ which is secure, scalable and cost effective using VMwares NSX for vSphere
platform. In this section we are going to look at how we provide some guidance around developing an
architecture, and the design the deployment that is required to both realize the benefits and mitigate
the risks.
There has been a great deal of confusion on what VPN features actually exists in the VMwares NSX
for vSphere platform. With the NSX for vSphere v6.1.x product you will see the following features
available. Please see the diagram VPN Services:

VPN
o IPSec VPN
o L2 VPN

SSL VPN-Plus

Diagram: VPN Services

Now of these '3' VPN features is as follows:

IPSec VPN - For IPSec VPN you can configure two Edge Services Gateways (ESG) and
create a site-to-site tunnel between the two Edge Services Gateways. Then the networks
behind the two Edge Services Gateways would be reachable to each other. This is a Layer 3
Site-to-Site solution and provides the ability to interconnect two different networks.
L2VPN - For L2VPN this is used to extend a network across boundaries such that the VMs
being extended are not aware or require any change in their routing or MAC addresses etc.
This is a Layer 2 Site-to-Site solution and basically lets you move VM's between two
different networks without reconfiguring the VMs.
 SSL VPN-Plus - SSL VPN-Plus being a user based solution, you can use one Edge Services
Gateways and configure SSL VPN on it. Then the Private network behind the Edge Services
Gateway will be reachable via the end users machine once connected via the SSL VPN client.

Note that the L2VPN feature is used by vCloud Connector for its Data Center Extensions (DCE).

NSX Edge Services Gateway IPSec VPN

The NSX for vSphere Edge Services Gateway (ESG) supports certificate authentication, pre-shared
key mode, IP unicast traffic, and no dynamic routing protocol between the NSX for vSphere Edge
Gateway instance and remote VPN routers. Behind each remote VPN router, you can configure
multiple subnets to connect to the internal network behind an NSX for vSphere Edge Services
Gateway instance through IPsec VPN tunnels. These subnets and the internal network behind an NSX
for vSphere Edge Services Gateway instance must have address ranges that do not overlap.

You can deploy an NSX for vSphere Edge Services Gateway (ESG) behind a NAT device. In this
deployment, the NAT device translates the VPN address of a NSX for vSphere Edge Services
Gateway instance to a publicly accessible address facing the Internet. Remote VPN routers use this
public address to access the NSX for vSphere Edge Services Gateway instance. You can also place
remote VPN routers behind a NAT device. You must provide the VPN native address and the VPN
Gateway ID to set up the tunnel. On both ends, static one-to-one NAT is required for the VPN
address. You can have a maximum of 64 tunnels across a maximum of 10 sites.

A common question that arises based on the above, is the IPSec Tunnels says refer to have a
maximum of 64 tunnels across a maximum of 10 sites. But what exactly defines a site? What this
means is that the NSX for vSphere Edge Services Gateway in the datacenter is limited by a vSphere
VMs limitation of 10 interfaces. This limits the number of remote NSX for vSphere Edge Services
Gateway s that can connect to a single data center Edge (10 being the maximum). Once connectivity
is available, NSX for vSphere Edge Services Gateway can support 64 IPsec tunnels (carrying Logical
Switch traffic). So, using a single NSX for vSphere Edge Services Gateway you can connect 10
offices with 64 Logical Switches tunneled.

Encapsulating Security Payload (ESP) tunnel mode is used:

64 tunnels are supported across a maximum of 10 sites.
Internet Key Exchange v1
Multiple non-overlapping local and peer subnets can be configured.
Industry standard IPsec implementation:
Full interoperability with Cisco, Juniper, Sonicwall, and others
Supports both the pre-shared key (PSK) and certificate authentication mode.
Supported encryption algorithms are AES (default), AES256, and TripleDES.

IPsec Security Protocols Internet Key Exchange

IPsec is a framework of open standards. Many technical terms are in the logs of the NSX for vSphere
Edge Services Gateway instance and other VPN appliances that you can use to troubleshoot the IPsec
VPN. You might encounter some of these standards.

Internet Security Association and Key Management Protocol (ISAKMP): This protocol is defined by
RFC 2408 for establishing Security Associations (SA) and cryptographic keys in an Internet
environment. ISAKMP only provides a framework for authentication and key exchange and is
designed to be key exchange independent.
Oakley: This protocol is a key agreement protocol that allows authenticated parties to exchange
keying material across an insecure connection by using the Diffie-Hellman key exchange algorithm.

Internet Key Exchange (IKE): This protocol is a combination of ISAKMP framework and Oakley.
The NSX for vSphere Edge Services Gateway Edge provides IKEv2.

IKE has two phases. Phase 1 sets up mutual authentication of the peers, negotiates cryptographic
parameters, and creates session keys. Phase 2 negotiates an IPsec tunnel by creating keying material
for the IPsec tunnel to use. Phase 2 either uses the IKE phase one keys as a base or performs a new
key exchange. The following phase 1 parameters are used by NSX for vSphere Edge Services
Gateway:

Main mode
3DES or AES (configurable)
SHA-1
MODP group 2 (1024 bits)
Pre-shared secret (configurable)
Security association lifetime of 28800 seconds (eight hours)
ISAKMP aggressive mode disabled

The following IKE phase 2 parameters are supported by NSX for vSphere Edge Services Gateway:

3DES or AES (matches the phase 1 setting)

SHA-1
ESP tunnel mode
MODP group 2 (1024 bits)
Perfect forward secrecy for rekeying
Security association lifetime of 3600 seconds (one hour)
Selectors for all IP protocols, all ports, between the two networks, using IPv4 subnets

The Diffie-Hellman (DH) key exchange protocol is a cryptographic protocol that allows two parties
that have no previous knowledge of one another to jointly establish a shared secret key over an
insecure communications channel. The NSX for vSphere Edge Services Gateway supports DH group
2 (1024 bits) and group 5 (1536 bits).

IPsec Security Protocols Encapsulating Security Payload

Encapsulating Security Payload (ESP) is a member of the IPsec protocol suite. In IPsec, it provides
origin authenticity, integrity, and confidentiality protection of packets. ESP in Tunnel Mode
encapsulates the entire original IP packet with a new packet header. ESP protects the whole inner IP
packet (including the inner header). The outer header remains unprotected. ESP operates directly on
IP, using IP protocol number 50.

ESP tunnel mode:

Confidentiality (encryption)
Connectionless integrity
Data origin authentication
Protection against replay attacks
IPsec ESP Tunnel Mode Packet

Encapsulating Security Payload (ESP) processes a packet in tunnel mode, the entire packet is
surrounded by the ESP header, ESP trailer, and ESP authentication data.

ESP header: Contains two fields, the SPI and Sequence Number, and comes before the
encrypted data.
ESP trailer: Placed after the encrypted data. The ESP trailer contains padding that is used to
align the encrypted data through a Padding and Pad Length field.
ESP authentication data: Contains an integrity check value.

What you are seeing in the diagram above is the original packet that is transmitted and is both
encrypted and authenticated.

Security is one of the top concerns among organizations evaluating any networking and security
product, and the NSX for vSphere platform is no different. Offering customers, the ability to
implement their own security measures, with a comprehensive set of security tools is a very attractive
proposition. Among these tools, one of the most important to enterprise customers and service
providers alike is the ability to securely interconnect physical and virtual datacenters with virtual
private networks (VPNs).

Virtual Private Networks are an important feature, as this enables IT organizations to securely connect
their own physical, virtual, and cloud environments to virtual datacenters hosted by service provider
as an example. With connectivity to multiple organizations secured by VPN, organizations can freely
move their date securely in an out without having to worry about loss or corruption of data in transit.
From the enterprise datacenters perspective, a virtual datacenter in the public Cloud is simply another
subnet in its network topology. For cloud service providers, supporting VPNs makes it easier to
attract customers and garner more of their workloads, increasing revenue and strengthening
partnerships with customers.

Fortunately, VMware provides a suite of Networking and Security products, whereby one of these
features is the NSX for vSphere platform, which has a comprehensive IPsec VPN feature. Using this
feature, you can securely interconnect your enterprise datacenters with virtual datacenters. With the
NSX for vSphere platform IPsec VPN Edge Services Gateway, customers can secure communication
between their datacenters. The result is that customers can treat this feature as providing a seamless
extension of their own datacenters, making this inter-connect straightforward and secure.
IPsec VPN Network Topologies

To interconnect multiple private organizations or sites over public networks securely, requires the
functionality of the NSX for vSphere Platform and in particular the IPsec VPN Edge Services
Gateway, thus making them work as if they are extensions of a single datacenter. The network
topologies could include the following an example:

Single Site Topology - NSX for vSphere platform Edge Services Gateway IPsec VPNs can for
example connect different virtual datacenters hosted by the same service provider, even hosted in the
same vCloud Director instance (See Figure 1 above). This example secures communication between
networks hosted on shared infrastructure.

Multi-Site Topology NSX for vSphere platform Edge Services Gateway IPsec VPNs can connect
to multiple deployments. For example, an enterprise private cloud can be securely connected to the
organizations virtual datacenter in a service providers public cloud (See Figure 1 above). Similarly,
virtual datacenters hosted by multiple service providers can be interconnected. These examples secure
communication between clouds over public networks.

Enterprise Site Topology - NSX for vSphere platform Edge Services Gateway Edge VPNs can
securely connect enterprises with fixed router or firewall-based VPNs to virtual datacenters hosted by
providers of vCloud Powered services. Because NSX for vSphere platform Edge Services Gateway
supports industry-standard IPsec-based VPNs, a wide range of devices, including those from Check
Point, Cisco, and Juniper, can be used to terminate the VPN at the enterprise location.

Edge Services Gateway IPsec VPN Use Case

The network topologies that most enterprise and service provider customers are most likely to
encounter involve two use cases for NSX for vSphere platform Edge Services Gateway IPsec VPNs:

Connecting multiple virtual datacenters regardless of location. This single use case supports both
multi-site and single-site deployments and connecting enterprise environments with virtual
datacenters hosted by service providers.
Connecting enterprise datacenters with virtual datacenters.

From the standpoint of implementing these use cases with NSX for vSphere platform Edge Services
Gateway IPsec VPNs, the main difference is the endpoints. In the first case, both endpoints are NSX
for vSphere platform Edge Services Gateway appliances located at the perimeter of a virtual
datacenter. In the second case, a NSX for vSphere platform Edge Services Gateway Edge appliance
establishes a VPN with a physical device located in an enterprise datacenter.
Edge Services Gateway IPsec VPN Prerequisites

In order to establish a site-to-site IPsec VPN, a small number of prerequisites must be fulfilled:

Each VPN appliance, whether a NSX for vSphere platform Edge Services Gateway instance or a
physical appliance, must have a fixed IP address that makes the appliance visible to each other. In
the case of multi-site VPNs, this requires public IP addresses. In the case of single-site VPNs,
private addresses can be used as long as the appliances are on the same network or the addresses
are routable.
The NSX for vSphere platform Edge Services Gateway appliance must allow the following
protocols to pass: Encapsulating Security Payload (ESP), Internet Key Exchange (IKE), and NAT
traversal.

Edge Services Gateway IPsec VPN Protocols

Firstly, there are in fact two versions of IKE, IKE version 1 (IKEv1) and IKE version 2 (IKEv2). The
NSX for vSphere platform Edge Services Gateway IPsec VPN feature in v6.1.x only supports IKEv1.
IKEv1 is based on the Internet Security Association and Key Management Protocol (ISAKMP)
framework.

IKEv1

UDP port 500

IKE has two phases:
Phase 1: Authenticated communication channel for the key exchange
Phase 2: Security Associations for both sides

Auto-plumbing of the firewall for the control channel the NSX for vSphere platform Edge Services
Gateway automatically adds the required firewall rules to accept communication with the peer, both
for tunnel control and data traffic.

Encapsulating Security Payload (ESP) Tunnel Mode

Confidentiality (encryption)
Connectionless integrity
Data origin authentication
Protection against replay attacks
Tunnel mode

Edge Services Gateway IPsec VPN Key Characteristics

ESP Tunnel mode is used to provide a VPN between two networks or between a host and a
network. With the version NSX for vSphere platform Edge Services Gateway v6.1.x 64 tunnels
are supported
Multiple non-overlapping local and peer subnets can be configured
Only IKEv1 is supported
NAT Traversal (NAT-T) is used in situations where network address translation is interposed
between the two-NSX for vSphere platform Edge Services Gateway devices. NAT Traversal
overcomes the problems inherent in encrypting IPsec ESP packets that include translated
addresses that must be modified in the payload, thus causing checksum errors and other
 incompatibilities. NAT Traversal and all the other IPsec protocols including IKE and ESP only
pass between the NSX for vSphere platform Edge Services Gateway Edge devices. The internal
virtual machines communicating to the NSX for vSphere platform Edge Services Gateway
devices do not need to be aware of the existence of the tunnel
Based on Industry Standard IPsec implementation
Fully interoperable with standard vendors, such as Cisco, Juniper, Sonicwall, etc
Support for both pre-shared key (PSK) and certificate authentication mode - in this model both
the peers use a pre-shared key for encryption and authentication. These keys are agreed to and
configured on each peer prior to starting the VPN
Supported Encryption Algorithms include, 3DES and AES (128 and 256 bits)

AES-NI support, which was added in NSX for vSphere platform Edge Services Gateway v6.1.x,
provides higher performance - AES-NI is Intels Advanced Encryption Standard New Instructions,
which uses a hardware dependent Westermere microarchitecture, such as the Xeon E56xx model. No
user configuration is required in order to enable AES-NI. The encryption overhead for packet traffic
in a VPN application can be high. The Intel AES-NI feature can substantially reduce the demand on
the CPUs of the ESXi hosts. The NSX for vSphere platform Edge Services Gateway offloads the AES
encryption of data to the hardware on supported Intel Xeon and second-generation Intel Core
processors. In the following example it is anticipated that up to a 40 percent performance increase
could be achieved by supporting the Intel AES-NI (AES New Encryption Instruction Set). In addition
to this, and from a security perspective there is no user configuration necessary as AES-NI support in
hardware is auto detected. The NSX for vSphere platform Edge Services Gateway also supports
certificate authentication, pre-shared key mode, and IP unicast traffic.

Diagram: VPN Services using AES-NI

In order to make the NSX for vSphere platform Edge Services Gateway IPsec VPN work end-to-end,
the following ports need to be opened on both peers and all firewall devices in between, This includes
UDP 500 for IKE, UDP 4500 for NAT-T and ip proto 50 for ESP.

Adding an Edge Services Gateway IPsec VPN

Launch your favorite browser, such as Firefox, Chrome or Internet Explorer and enter the IP Address
or name of the NSX for vSphere Manager. Once you have successfully logged in go to the
Networking and Security View -> NSX Edges.
Open the NSX Edges, in this example Edge Id edge-7, and you will see the following:

Now, click on the Add button and you will be presented with the following:

What is important to note here is that you must configure at least one external IP address on the NSX
for vSphere Edge Services Gateway to provide the IPsec VPN service.

For the local NSX for vSphere Edge Services Gateway instance, you need to provide an ID, the
external IP address, and the CIDR block for the local subnets. You will also be required to enter the
same set of information for the peer endpoint also.

For the remote NSX for vSphere Edge Services Gateway instance, you will need to provide the same
information, but from the remote perspective.

Finally, you will need to select an encryption algorithm, type of authentication, Diffie- Hellman
Group, and Extension is required.

NSX for vSphere SSL VPN-Plus Services

Conventional full access SSL VPNs send TCP/IP data in a second TCP/IP stack for encryption over
the Internet. The result is that application layer data is encapsulated twice in two separate TCP
streams. When packet loss occurs, which happens even under optimal Internet conditions), a
performance degradation effect called TCP-over-TCP meltdown occurs. In essence, two TCP
instances are correcting a single packet of IP data, undermining network throughput and causing
connection timeouts. TCP optimization eliminates this TCP-over-TCP problem, ensuring optimal
performance.

The NSX for vSphere SSL VPN-Plus Service fundamentally enables individual remote users to
connect securely to private networks behind an NSX for vSphere Edge Services Gateway, and thus
allows remote users to access applications and servers from the private networks. This service also
provides two access modes, one being Web access mode, where there is no client installed, and full
network access mode, which requires that a client is installed.

The NSX for vSphere Edge Services Gateway provides users with access to protected resources by
establishing an SSL encrypted tunnel between a laptop, such as Mac OS X or Windows and the NSX
for vSphere Edge Services Gateway. The NSX for vSphere SSL VPN-Plus Service is intended to be
deployed as a substitute for more complicated IPsec client-to-site or jump server solutions. The NSX
for vSphere SSL VPN-Plus Service does not support mobile clients, and does not deliver common
end-user features such as reverse proxy, custom portal, and SSL offload. The primary use cases and
capabilities of the NSX for vSphere SSL VPN-Plus Service are somewhat different from capabilities
that are provided by the Horizon View. View is the VMware comprehensive approach to virtual
desktop infrastructure, secure mobility, and end-user remote access.

One of the real advantages of the NSX for vSphere SSL VPN-Plus Service is you can access your
corporate LAN by using the Web-access mode or with a downloadable SSL client, but there is no
special hardware or software that is required.
An important aspect of the NSX for vSphere SSL VPN-Plus Service is its ability to provide
administrative users with full tunnel access to protected resources by establishing an SSL encrypted
tunnel between a laptop and the NSX for vSphere Edge Services Gateway, which in turn provides you
with a Secure Management Access Server.
NSX for vSphere Edge Services Gateway SSL VPN-Plus Secure Management Access
Server

L2 VPN and Stretched Logical Networks

We have covered both IPSec VPN and SSL-VPN Services, but will now take a look at the lesser-
known feature of the NSX for vSphere Edge Services Gateway, which is the L2 VPN. The L2 VPN
enables you to stretch a Layer 2 subnet over Layer 3, tunnelled through an SSL VPN. The two sites
form a Layer 2 adjacency; this could either be within the same datacenter or across
datacenter locations.
With the current version of the NSX for vSphere 6.1.x platform it is now possible to trunk multiple
logical networks, whether that be a VLAN to VLAN, VLAN to VXLAN or VXLAN to VXLAN. It is
also possible to deploy a standalone NSX for vSphere Edge Services Gateway on a remote site
without that site being NSX Enabled that is, connecting to a VLAN. As this is not a particularly
well-understood area, it is important we cover two new abstracts.

Trunk interface, this allows multiple internal networks, which could be either a VLAN or
VXLAN to be trunked. Interfaces that are configured to support 802.1Q Ethernet frames are
called Trunk interfaces.
 Local Egress Optimization enables the NSX for vSphere Edge Services Gateway to route any
packets sent towards the Egress Optimization IP address locally, and send everything else
over the tunnel. This is important for VM Mobility, as if the default gateway for the virtual
machines, which belongs to the subnets you are stretching, is the same across the two sites
you need this setting to ensure traffic will be locally routed on each site. If you decide you
need to migrate a Virtual Machine (VM) from one site to another, you can do so without
touching the Guest OS network configuration.

NSX for vSphere Edge Services Gateway L2 VPN Use Case

A very common use case for L2 VPN is Cloud Bursting, where a Private Cloud service bursts into a
Public Cloud when the demand is required. Effectively this is a Hybrid Cloud solution.

The following diagram is from the NSX for vSphere v6.1 Administration Guide. As you can see in
this scenario and diagram below, VLAN 10 on site A is stretched to VXLAN 5010 on site B.
Similarly, for VLAN 11 is stretched to VXLAN 5011 on site B. Again, this is an example where an
NSX data centre is extended to a non-NSX data centre.
In the diagram and scenario below, this could be used for a Private Cloud to Private Cloud Migration
or DR Site. As you can see the VXLAN 5010 and 5011 have been stretched to site B and mapped to
the same VNI. A VXLAN Number Identifier (VNI) is a 24-bit number that gets added to the VXLAN
frame and uniquely identifies the segment to which the inner Ethernet frame belongs. You can have
multiple VNIs, which can exist in the same transport zone. In the NSX for vSphere Platform the VNI
start at 5000.

NSX for vSphere Edge Services Gateway L2 VPN Topology

The following diagram and topology depicts a VXLAN to VXLAN extension. Here we are stretching
VXLAN 5004 at Site B, which is the Branch Web Tier, to VXLAN 5003 at Site A, which is the Web
Tier.
This will allow an administrator to extend his datacenter to public clouds in such a way that he can
manage resources (VMs, vApps, templates) in public clouds as if they where in the local datacenter.

NSX for vSphere Edge Services Gateway Firewall

A logical firewall provides security mechanisms for dynamic virtual data centers, and includes
components to address different deployment use cases.

The NSX for vSphere Edge Services Gateway Firewall focuses on the North-South traffic
enforcement at the tenant or data center perimeter as shown below. There is also the NSX for vSphere
Distributed Firewall, which focuses on East-West traffic, and together, these components address the
end-to-end firewall needs of virtual data centers. It is also possible to deploy either or both of these
technologies.
The NSX for vSphere Edge Services Gateway firewall provides perimeter security functionality,
which is a stateful firewall for North-South traffic flows. It supports dynamic routing, is virtualization
context aware and the configurations and management is performed from one central location where
Firewall rules are applied in ascending number order.

The types of firewall rules that the NSX for vSphere Edge Services Gateway firewall supports are as
follows:

When you first deploy NSX for vSphere Edge Services Gateway a set of default rules are created
during the deployment. The following example depicts this:

Internal rules are created by the NSX for vSphere Edge Services gateway in support of services
configured by the user, and the user creates user rules, as depicted below. The firewall rule type does
not affect the application of the rule.

Virtualization context awareness refers to the ability for the NSX for vSphere Edge Services gateway
firewall to filter traffic flows based on IP and TCP/UDP header information. The NSX for vSphere
Edge Services gateway firewall can also filter traffic flows based on virtualization specific
information, such as Cluster, Logical switch, Security Group and Virtual machine for example. Please
see the following diagram that depicts this:

Populating the NSX for vSphere Edge Services Gateway firewall rules requires that you enter specific
information when creating the firewall rule. For example, you are required to assign the following
information:

Name This is just a free form description.

Source
Destination The source and destination rules can include the IP address in the packet or
information provided by the VMware vCenter Server, such as virtual machine name or
resource pool.
Service - A service in the firewall context is a collection of TCP/UDP ports that form the
 components for successful communication with an end system.
Action - The Action option allows the rule to accept or deny the traffic. This can include
logging for the rule, Network Address Translation support can be enabled and the rule can be
applied on ingress or egress.

This complete rule is also shown in the diagram below:

This diagram also shows the action option for Network Address Translation, which is not commonly
known:

After the NSX for vSphere Edge Services Gateway firewall rule(s) has been created, you are required
to publish the changes. When this process has taken place the changes take effect immediately. This is
a common issue whereby after completing the creation of the NSX for vSphere Edge Services
Gateway firewall rule(s) you forget to publish the rule. As soon as you make a change to a rule you
will see the following dialog:
Also note that when you are creating your NSX for vSphere Edge Services Gateway firewall rule(s)
that they are processed in order. This means that the first rule that matches the traffic being examined
is applied and the traffic is passed or dropped.

Flow Monitoring

Flow Monitoring is a traffic analysis tool that provides a detailed view of the traffic on virtual
machines. The Flow Monitoring output defines which machines are exchanging data and over which
application. This data includes the number of sessions, packets, and bytes transmitted per session.
Session details include sources, destinations, direction of sessions, applications, and ports being used.
Session details can be used to create firewall allow or block rules. You can use this information to
audit network traffic, define and refine firewall policies and identity threats to your network.

The NSX for vSphere Distributed Firewall has visibility of all traffic flows that have taken place in
the logical switches. By drilling down into the traffic data, it is possible to evaluate the use of your
resources and send session information to the Distributed Firewall to create a rule or block rule at any
level. This is a very power security feature.

NSX for vSphere Hardening Guide

In this section we look at the high level guidance for securing or evaluating the NSX for vSphere
platform and infrastructure. The recommendations expressed herein are not exhaustive. Instead, they
intend to represent high value actions one may take toward securing or evaluating the NSX for
vSphere platform infrastructure. Additionally, these recommendations are intend to be specific for
NSX for vSphere v6.1.x. The recommendations are categorized in the following sections:

1. Common Recommendations
2. Management Plane Recommendations
3. Control Pane Recommendations
4. Data Plane Recommendations

The above sections are defined in the VMware NSX for vSphere Hardening Guide. To obtain the
latest version of this guide, please visit https://communities.vmware.com/docs/DOC-28142. If you
have questions, comments, or have identified ways to improve this guide, please send comments in
the comments area.

This VMware NSX for vSphere Hardening Guide is intended for various roles, including network and
security architects, security officers, virtual infrastructure administrators, cloud infrastructure
architects, cloud administrators, cloud customers, cloud providers, and auditors. Additionally,
individuals and organizations that seek a starting point for the network and security controls to
consider when adopting or building a network and security infrastructure may find the
recommendations in this section valuable.

VMware engages with various partners to perform the security assessment of the NSX for vSphere
platform. These reviews provide an assessment as well as focusing on major, and newly added
features such as the integration of Software Defined Networking (SDN) and Software Defined Data
Center (SDDC). The VMware NSX for vSphere Hardening assessment of the NSX for vSphere
platform is primarily focused on the networking and security attacks, configuration issues, secure
defaults, and protocols in use. Using a combination of targeted source code review, active testing,
fuzzing and discussions we are able to locate and determine if any significant vulnerabilities exists.
Separately or in concert, many of these issues can result in a complete datacenter compromise.

I think it is fait to say that the NSX for vSphere platform is complex, and is not a platform that you
can just drop in, and requires a great deal of thought before you go building or deploying such an
environment.

There is no doubt in anyones mind that there are inherent risks of the Software Defined Networking
however, software defined networking paired with network and security virtualization promises to
offer a myriad of benefits and allows for entire Software Defined Data Centers (SDDC), a key part of
VMware's vision for the current and future products.

It is also important to know and understand the potential and inherent risks of this new platform in
order to help guide you as you work with VMwares NSX platform technology.

One of the true values of software defined networking and security is it allows agile movement of
virtual machines and networks and security services between physical hosts and the datacenter(s). The
dynamic nature of this technology requires that underlying hosts be fully connected at the physical
and IP layer. However, with this mass of connectivity comes a myriad of associated risks. All
software has flaws, and the reimplementation of core networking protocols, parsers, and switching
methods will repeat and likely inherit historic vulnerabilities from the age of physical networking and
security.

A Denial-of-Service (DoS) becomes a much greater issue now. In the physical networking world,
dedicated hardware handles much of the parsing and routing of packets. In a software networking and
security world, it is the software component that must parse, reparse, perform table lookups, and
generally be aware and about the encapsulation and fragmentation and so on., and overall spending
much more CPU time deciding how to handle each packet. A potential software bug in any stage of
this packet handling can lead to resource exhaustion, software crashes, and other scenarios that result
in a total Denial-of-Service (DoS) and thus potentially a loss of networking and security services for
hundreds of hosts, and also to potentially affecting an entire datacenter!

It is also true that software-defined networking and security also extends traditional network and
security attacks to multiple datacenters. Traditionally local attacks, such as ARP spoofing, can now be
conducted across layer three networks in geographically diverse locations. Additionally, if any
vulnerability in the software network and security stack allows these attacks to leak onto the physical
network, physical hosts in multiple datacenters affecting multiple customers can also be
compromised.

In a very real sense software-defined networking and security as it is currently designed everything is
in the basket of VM containment. If a virtual machine escape is ever performed or if an attacker
discovers a technique for sending un-encapsulated packets on physical networks, expected security is
lost. As described above, every physical host must be completely connected at the IP and physical
layer, exposing an extremely broad attack surface. Once an attacker has a method of sending and
receiving data on this physical network, the attacker can move laterally between hosts unabated by
firewalls or routers, as these are no longer security relevant devices. Software-defined networking and
security is a powerful technology that is necessary to organizations and companies now and in the
long-term future. However, like all software, it can be fragile and networking and security
vulnerabilities have broad ramifications not traditionally realized in physical networking platforms.

Throughout your security assessments, it should be noted that you look at recurring weaknesses, as
these are good candidates for systematic fixes as well as areas that you should look at and be subject
to additional testing. These can also be considered in secure guidelines and threat modeling.

Here are some security measures to consider.

Much of the NSX for vSphere platform can be protected with TLSv1/SSL if properly configured, but
consistent usage and strong defaults are still elusive. When protecting the NSX for vSphere Manager
as well as the management REST APIs we use TLS v1.2, as the control plane uses TLS in all other
communications.

Summary

In this chapter we have discussed about the overview of the NSX for vSphere security architecture,
NSX for vSphere networking and security connectivity. Also we have discussed about the use cases
of NSX for vSphere networking and security features. In the next chapter we will discuss about the
deployment process of NSX for vSphere and walk you through the process of various different
component deployment.





































3
Deployment of NSX for vSphere
This deployment of NSX will be divided into three sections.

Installation of Management Plane

Installation Control Plane
Installation of Data Plane

Each section will be described with stepby-step procedure from installing the first component of
NSX Management Server to VM Production ready network for L2-L3 Services. Integration of L4-L7
services will not cover in this book, this will discuss in the upcoming books.

Installation of Management Plane

NSX Manager is the management plane for vCenter Server to provide the administration through
plugin interface. NSX Manager is available as appliance from VMware and provides easy to
download and deploy methodology. NSX Manager requires an IP address to communicate with
vCenter Server in the same subnet. Once NSX Manager is installed, the plugin will be active on the
vCenter Server after registration of its FQDN or IP Address with the NSX Manager.

To install the NSX Manager, download all the required binary images for VMware NSX from
download section on the VMware portal.

https://my.vmware.com/web/vmware/info/slug/networking_security/vmware_nsx/6_x

This download will require valid login account ID with VMware NSX subscription.

Import the NSX Manager (ovf file) from your server on the selected cluster or host in the vCenter
Server, accept the extra configuration, EULA and provide the details for desired destination.
Configure the CLI admin User and Privilege Mode Password, Network Properties like hostname, IP
address, Netmask, Gateway, DNS & NTP setting.

After completing all the steps, NSX Manager appliance will be deployed in few minutes and power
on the NSX Manager Virtual Machine in the vCenter Server.

Login into NSX Manager IP address using any web browser with admin user and password defined
during installation.
This will open NSX Manager Virtual Appliance Management. Click on Manage vCenter Registration
to install the Network and Security Plugin into vCenter Server.

On the left side under Components select NSX Management Service and select Edit under vCenter
Server. Enter the required information to register vCenter Server with NSX Manager like vCenter
Server Name or IP Address, vCenter User Name and Password, and click ok to proceed.
Accept the Trust Certificate and proceed with this certificate. Once registration is completed, Network
and Security icon will show up in the vCenter Server home section. This icon will provide you all the
features to manage the NSX.

Installation of Control plane

NSX Controllers are the control plane for all logical switches and at least three controllers cluster is
recommended for any production environment. These controllers maintain database and information
of all the virtual machines, hosts, logical switches, logical routers and VXLANs. The controller
cluster will be distributed across hosts to provide the high availability for all the logical switches in
case of failure of any host. Define the anti-affinity rule to separate the controllers on separate ESXi
servers.

Click on Network & Security plugin in vCenter Server to open the management of NSX. Select the
Installation on left side, and click on green plus symbol to add a Controller under NSX Controller
nodes.

This will open Add Controller window; select the relevant Datacenter, Cluster/Resource Pool,
Datastore, and Host from drop down option.
Select to Connect this controller to Management Distributed vSwitch PortGroup for network
connectivity.

Create the IP pool required for network connectivity. This IP pool will be used by this controller and
subsequent controllers installed. Enter the Pool name, Gateway, Prefix Length, Primary & Secondary
DNS, DSN Suffix and Static IP Pool.
After this is done we just need to set a management password for the controller.

Once the controller will be deployed. Repeat the steps for controller 2 and 3. It is recommended to
have minimum 3 controllers in production environment, however for testing purpose one or two
controllers can be used.
Installation of Data Plane
After installation of controllers, the clusters will be configured for VXLAN transport traffic. The host
preparation will install VXLAN interface on each host and enable it for VXLAN communication.
This interface will encapsulate the network segment packets from Virtual Machines to transport it to
other hosts and this make virtual machines to communicate over L3 boundaries. The VMware
Distributed Switch will become VMware NSX Switch, once below three VIBs will installed on each
hosts.

VXLAN
Distributed Router
Distributed Firewall

Once the hosts will be enabled with VXLAN Transport,

Click on Network & Security plugin in vCenter Server to open the management of NSX. Select the
Installation on left side, and click on Host Preparation Tab, click on Install option for each cluster.
This install will enabled the VXLAN feature for that cluster and hosts inside the cluster.

Click on Yes option to install on the selected cluster.

This will take some time to do the installation. Sometime, this installation will show warning message
and unable to do the installation. In this case, refresh the vCenter Server and verify Firewall and
VXLAN column should be green and enabled for that cluster.
Click on the Logical Network Preparation Tab, Under VXLAN Transport sub tab, Click on
configuring the host for VXLAN transport. Select the Distributed Switch, enter the desired VLAN,
MTU 1600, create IP Address pool for VMKNic and assign Static Etherchannel Policy for each
cluster.

Once each cluster or host will have configured with VXLAN Tranport IP address, verify that each
host get another vmkernal IP address and enabled with vxlan in TCPIP stack column.
Once all the clusters and hosts are configured for VXLAN transport IP address, click on Segment ID
sub tab. Segment ID pools are used by logical segments for VXLAN Network Identifier (VNI). You
can enable multicast address, when configuring the Hybrid or Multicast Control plan modes.

After adding Segment ID Pool, Click on Transport Zones sub tab under Logical Preparation Tab, to
create the global transport zone for all the clusters, click on Green Sign to add transport zone.

Enter the name, Description, Control Plane Mode and Select the clusters to add into transport zone.
After this basic setup for VMware NSX is ready for all clusters and hosts. Addition of new hosts and
clusters will follow the above process for Virtual Machine network provisioning.

Next steps are to create the logical switches, edge router and distributed logical routers.

Under the Network and Security plugin within vCenter click on the Logical Switches menu, deploy a
Logical Switch, click the Green plus. Enter Name as Logical-Transport, Description and Control
Plane Mode as Unicast.
Repeat the steps for Logical-Web, Logical-App and Logical-DB switches. Each logical switch will
get one Segment ID assigned earlier in the pool.

Attach the Web, App & DB VMs with created above respective logical switches. Click on little plus
icon with three blue boxes, attach the virtual machines associate with web logical switch.

Select vNICs associated to the VM need to be attached to the logical switch.

Login into Virtual Machines console, verify the IP Address of the Web VM1 and ping with IP
address of Web VM2. Ping will work, these 2 Web VMs running on two different hosts are connected
to layer 2 network and communicating over VXLAN logical switch.

In the vCenter Server, click on the Networking and Security plugin. Select NSX edge, click the green
plus. Select the Logical (Distributed) Router from the Install Type and fill the name, hostname and
description, click Next.
Set an administrative password and username. Choose Enable SSH access to access through CLI.

Select the Cluster, Datastore, Host and Folder to place this Edge VM, click ok.
Select the Management Port group on Distributed Switch, click Ok
Click the Green plus on the add interface, fill the Primary IP address and subnet prefix length for
management traffic, click ok.

Add the Uplink for DLR, Select the Type Uplink, connected to Logical-Transport switch created
earlier, click on green plus to add the IP address, subnet prefix length for this interface, click ok.
Similarly, add the interface for Web, App & DB tier applications and respective IP addresses.
After adding all the required interfaces, click on Finish.
Once the DLR route is deployed, the kernel routing will be enabled for Virtual Machines.

Ping gateway IP address of respective Web, App & DB tier and it should be reachable.

Verify IP route in the DLR router and execute show ip route command. The routes should be visible
from different tiers.
Next step is to deploy the Edge Service Gateway (ESG) to connect the application to corporate or
external network. The ESG can provide routing, firewall, load balancer, VPN, Layer 2 bridging and
more services. We will deploy here routing Services.

In the vCenter Server, click on the Networking and Security plugin. Select NSX edge, click the green
plus. Select the Edge Services Gateway (ESG) from the Install Type and fill the name, hostname and
description, click Next.
Fill the username and password and select the Enable SSH access, click Next.

Select the Datacenter from the dropdown menu, Appliance Size as per your requirements. Select the
Enable auto rule generation option, click on NSX Edge Appliance to place in the vCenter Server.
Choose the Resource Pool, Select Datastore, Host and Folder for ESG VM placement, click ok and
Next.
Add the internal and uplink interfaces for ESG Appliance. Click on Add Edge NSX interface, fill the
name, and select type Uplink. Click on Connected to change distributed port group for uplink
interface. Click on green plus sign, add the IP address and Subnet prefix length, click ok twice.

Similarly configured the internal interface to communicate with DLR appliance in the same subnet
and management interface.

Choose Configure the Default Gateway option, select the Uplink Network Interface in vNIC drop
down, add the gateway details of SVI Interface created on Nexus Switch for communication to
external network to internal network, MTU 1500, click Next.
In Firewall and HA, choose Configure Firewall default policy and select Default Traffic Policy with
Accept and logging with Disable option. If HA has been configured, then here you can specify the
keep alive link and relevant configurations and click Next.

Verify the details and Select Finish to deploy the ESG appliance.
Now ESG and DLR appliances are ready, we will configure dynamic routing between the two routers.

Now we will configure the Open Shortest Path First (OSPF) dynamic routing protocol between NSX
Edge and LDR.

Select NSX Edges and double-click on the Logical Distributed Router that was deployed previously.
Under the Manage tab, select Routing, Global Configuration and select Edit on Default Gateway and
enter the default gateway IP address of NSX Edge in the same subnet, click save.
Edit on Dynamic Router Configuration, Select the Router ID as Uplink interface and click on Enable
OSPF, click on Save.

Accept the changes and click Publish Changes.

Select the OSPF tab on the left side; click the Edit button for OSPF Configuration.

Select the Enable OSPF option; fill the Protocol Address and Forwarding Address, Click OK to
finish.
Enter the Area ID, Select Type as Normal and Authentication as None. Click on Ok.

Next click the Green Plus under Area Definitions. OSPF neighbors need to peer with routers with the
same area ID. We defined Area 10 earlier and therefore we need to use this again.

Next click the Green Plus under Area Definitions. Select the Uplink interface and Area as defined
earlier.
Review the changes and now click Publish Changes. This will enable OSPF on Logical Router.
Click the Route Redistribution, verify that OSPF is enabled and there is Route Distribution Rule with
permit. This will allow the OSFP route to be permitted in the hypervisor kernel and present the OSPF
route to ESG router.
Similarly, we need to enable OSPF on the Edge Service Gateway (ESG), double click the ESG, Select
the Routing sub tab under Manage Tab. Notice the Default Gateway is already populated from the
deployment window.

Select the Edit button next to Dynamic Routing Configuration. Select the Router ID from drop down
menu, choose Enable OSPF option, click Save.

Click Publish Changes to save the changes and publish them.

Select the OSPF menu on the left side.

Click the Green Plus under the Area Definitions, select vNIC and Area from drop down menu. Keep
Advance parameter as default, click Save.
Publish Changes window will pop up after save it and click on Publish Changes to populate the
setting to other routers.
Lets verify that all the routes from internal network are published on the ESG router.

Open the console session of ESG route and execute the show ip route command. You will see that all
Web, App & DB tier networks will be listed in the ESG router and it will make OSPF adjacency with
DLR router.
Summary
In this chapter, we have discussed about the minimum requirements for VMware NSX deployment
and walked through the deployment of management, control and data components of VMware NSX.
In the next chapter we will discuss about VMware NSX Edge Services Gateway and its different
functions, such as NAT, firewall, DHCP, HA etc.

4
Edge Services Gateway
The VMware NSX Edge services gateway provides services such as routing, perimeter firewall,
network address translation (NAT), DHCP, VPN, load balancing, and high availability.

The NSX Edge services gateway is a key component in the communication link providing network
function virtualization in an agile, software-defined data center while providing high-speed
throughput.

NSX Edge provides the following benefits:

Near real-time service instantiation

Support for dynamic service differentiation per tenant or application

Scalability and Redundancy NSX Edge

Equal Cost Multi-Path functionality (ECMP) introduced in VMware NSX release 6.1 ECMP has the
potential to offer substantial increases in bandwidth by load-balancing traffic over multiple paths as
well as providing fault tolerance for failed paths. This is a feature which is available on physical
networks but we are now introducing this capability for virtual networking as well. ECMP uses a
dynamic routing protocol to learn the next-hop towards a final destination and to converge in case of
failures. For a great demo of how this works, you can start by watching this video, which walks you
through these capabilities in VMware NSX.

To keep pace with the growing demand for bandwidth, the data center must meet scale out
requirements, which provide the capability for a business or technology to accept increased volume
without redesign of the overall infrastructure. The ultimate goal is avoiding the rip and replace of
the existing physical infrastructure in order to keep up with the growing demands of the applications.
Data centers running business critical applications need to achieve near 100 percent uptime. In order
to achieve this goal, we need the ability to quickly recover from failures affecting the main core
components. Recovery from catastrophic events needs to be transparent to end user experiences.

ECMP with VMware NSX 6.1 allows you to use up to a maximum of 8 ECMP Paths simultaneously.
In a specific VMware NSX deployment, those scalability and resilience improvements are applied to
the on-ramp/off-ramp routing function offered by the Edge Services Gateway (ESG) functional
component, which allows communication between the logical networks and the external physical
infrastructure.

External users traffic arriving from the physical core routers can use up to 8 different paths (E1-E8)
to reach the virtual servers (Web, App, DB).

In the same way, traffic returning from the virtual servers hit the Distributed Logical Router (DLR),
which can choose up to 8 different paths to get to the core network.

Path Determination

When a traffic flow needs to be routed, the round robin algorithm is used to pick up one of the links as
the path for all traffic of this flow. The algorithm ensures to keep in order all the packets related to
this flow by sending them through the same path. Once the next-hop is selected for a particular Source
IP and Destination IP pair, the route cache stores this. Once a path has been chosen, all packets related
to this flow will follow the same path.

There is a default IPv4 route cache timeout, which is 300 seconds. If an entry is inactive for this
period of time, it is then eligible to be removed from route cache. Note that these settings can be tuned
for your environment.

Distributed Logical Router (DLR):

The DLR will choose a path based on a Hashing algorithm of Source IP and Destination IP.

Failure scenario of Edge Devices

In order to work with ECMP the requirement is to use a dynamic routing protocol: OSPF or BGP. If
we take OSPF for example, the main factor influencing the traffic outage experience is the tuning of
the OSPF timers. OSPF will send hello messages between neighbours, the OSPF Hello protocol is
used and determines the Interval as to how often an OSPF Hello is sent.

Another OSPF timer called Dead Interval is used, which is how long to wait before we consider an
OSPF neighbour as down. The OSPF Dead Interval is the main factor that influences the
convergence time. Dead Interval is usually 4 times the Hello Interval but the OSPF (and BGP) timers
can be set as low as 1 second (for Hello interval) and 3 seconds (for Dead interval) to speed up the
traffic recovery.

ECMP failed Edge

In the example above, the E1 NSX Edge has a failure; the physical routers and DLR detect E1 as
Dead at the expiration of the Dead timer and remove their OSPF neighbour relation with him. As a
consequence, the DLR and the physical router remove the routing table entries that originally pointed
to the specific next-hop IP address of the failed ESG.

As a result, all corresponding flows on the affected path are re-hashed through the remaining active
units. Its important to emphasize that network traffic that was forwarded across the non-affected
paths remains unaffected.

Troubleshooting and visibility

With ECMP its important to have introspection and visibility tools in order to troubleshoot optional
point of failure. Lets look at the following topology.
 Troubleshooting

A user outside our Datacenter would like to access the Web Server service inside the Datacenter. The
user IP address is 192.168.100.86 and the web server IP address is 172.16.10.10. This User traffic will
hit the Physical Router (R1), which has established OSPF adjacencies with E1 and E2 (the Edge
devices). As a result, R1 will learn how to get to the Web server from both E1 and E2 and will get two
different active paths towards 172.16.10.10. R1 will pick one of the paths to forward the traffic to
reach the Web server and will advertise the user network subnet 192.168.100.0/24 to both E1 and E2
with OSPF.

E1 and E2 are NSX for vSphere Edge devices that also establish OSPF adjacencies with the DLR. E1
and E2 will learn how to get to the Web server via OSPF control plane communication with the DLR.
From the DLR perspective, it acts as a default gateway for the Web server. This DLR will form an
OSPF adjacency with E1 and E2 and have 2 different OSPF routes to reach the user network. From
the DLR we can verify OSPF adjacency with E1, E2.

We can use the command: show ip ospf neighbor:

From this output we can see that the DLR has two Edge neighbours: 198.168.100.3 and
192.168.100.10. The next step will be to verify that ECMP is actually working.

We can use the command: show ip route

The output from this command shows that the DLR learned the user network 192.168.100.0/24 via
two different paths, one via E1 = 192.168.10.1 and the other via E2 = 192.168.10.10. Now we want to
display all the packets which were captured by an NSX for vSphere Edge interface.

In the example below and in order to display the traffic passing through interface vNic_1, and which
is not OSPF protocol control packets, we need to type this command:

debug packet display interface vNic_1 not_ip_proto_ospf

We can see an example with a ping running from host 192.168.100.86 to host 172.16.10.10

If we would like to display the captured traffic to a specific ip address 172.16.10.10, the command
capture would look like:

debug packet display interface vNic_1 dst_172.16.10.10

* Note: When using the command debug packter display interface we need to add
underscore between the expressions after the interface name.

Useful CLI for Debugging ECMP

To check which ECMP path is chosen for a flow

debug packet display interface IFNAME

To check the ECMP configuration

show configuration routing-global

To check the routing table

show ip route
To check the forwarding table

show ip forwarding

Useful CLI for Dynamic Routing

show ip ospf neighbor

show ip ospf database
show ip ospf interface
show ip bgp neighbors
show ip bgp

ECMP Deployment Consideration

Equal Cost Multipath, ECMP traffic involved Asymmetric routing between Edges and DLR or
between Edge and physical routers. In Asymmetric routing, a packet traverses from a source to a
destination in one path and takes a different path when it returns to the source. Because of this ECMP
currently implies stateless behavior. This means that there is no support for stateful services such as
the Firewall, Load Balancing or NAT on the NSX Edge Services Gateway.

ECMP with Asymmetric routing is not a problem by itself, but will cause problems when more than
one NSX Edge in place and stateful services inserted in the path of the traffic.

Problem Statement

User from outside try to access Web VM inside the Data Center. the traffic will pass through E1
Edge. From E1 the traffic will go to DLR transverse NSX distributed firewall and get to Web VM.
When Web VM respond back the traffic will hit the DLR default gateway. DLR have two option to
route the traffic E1 or E2. If DLR choose E2 the traffic will get the E2 and will Dropped!

The reason for this is E2 does not aware the state of session started at E1, replay packet from Red VM
arrived to E2 are not match any existing session at E2.

From E2 perspective this is new session need to validate, any new TCP session should start with
SYN, since this is not the begin of the session E2 will drop it!
 Asymmetric Routing with Edge Firewall Enabled

Note: NSX Distributed firewall is not part of this problem. NSX

Distributed firewall implements at the vNic level, all traffic get
in/out through the same vNic. There is no Asymmetric route in the
vNic level, and this is the reason when we vMotion VM, the
Firewall Rule, Connection state is move with the VM itself.

ECMP and Edge Firewall NSX

In version 6.1 when we enable ECMP on NSX Edge, we get the following message
The firewall service disabled by default:

Starting from 6.1.2 Edge Firewall not disabled automatic on ESG when ECMP is enabled,
administrator will need to turn off Firewall when enable ECMP.

NSX Edge and DRS Rules

The NSX Edge Cluster Connect Logical to the Physical world and contain an NSX Edge gateway and
DLR Control VM. There are cases in design that the Edge Cluster contains the NSX Controllers.
Important design key point of edge cluster is to survive different host failure or entire chassis with
minimal impact of the workload connectivity.

In the figure below we deploy NSX Edges, E1 and E2, in ECMP mode and they are active/active. The
DLR Control VMs runs active/passive while both E1 and E2 running a dynamic routing protocol with
the active DLR Control VM. When the DLR learns a new route from E1 or E2, it updates this info to
the NSX Controller. The NSX Controller updates the routing tables to each of the ESXi hosts, which
are running this DLR instance.
So that happens in this scenario when the ESXi host, which contains the Edge E1 failed? Let me
explain what happens.

The active DLR will update the NSX controller to remove E1 as next hop, the NSX Controller will
update the ESXi host and as a result the Web VM traffic will be routed to Edge E2. The time it
takes to re-route the traffic depends on the dynamic protocol converge time.
In the scenario when the ESXi or Chassis contains both the Edge E1 and the active DLR fails, we will
face a longer outage in the forwarded traffic.

The reason for this is that the active DLR is down and cannot update the Controller on failures. The
ESXi will continue to forward traffic to Edge E1 until the passive DLR becomes active, learns that the
Edge E1 is down and updates the NSX Controller.
The Golden Rule is that we must ensure that when the Edge gateway and the DLR belong to the same
tenant they will not reside in the same ESXi host. It is better to distribute them between ESXi hosts
and reduce the affected functions.

By default, when we deploy a NSX Edge or DLR in active/passive mode, the system takes care of
creating a DRS anti-affinity rule and this prevents the active/passive VMs from running in the same
ESXi host.
 DRS anti affinity rules

We need to build new DRS rules as these default rules will not prevent us from getting to the previous
scenario.

The figure below describes my Lab logical view. This topology is built from two different tenants
where each tenant is being represented in a differenced color and has its own Edge and DLR.

I removed the connectivity to the physical world in order to simplify the diagram.
My physical Edge Cluster has four ESXi hosts which are distributed over two physical chassis:

Chassis A: esxcomp-01a, esxcomp-02a

Chassis B: esxcomp-01b, esxcomp-02b

You need to create DRS Host Group for each Chassis, we start with creating a container for all the
ESXi hosts in Chassis A, this container group configured is in DRS Host Group.

Edge Cluster -> Manage -> Settings -> DRS Groups

Click on Create Add button and call this group Chassis A.

Container type need to be Host DRS Group and Add ESXi host running on Chassis A, in my lab its
esxomp-01a and esxcom-02a.
Another group Chassis B, in my lab its esxcomp-01b and esxcomp-02b:

VMs DRS Group for Chassis A:

We need to create a container for VMs that will run in Chassis A. At this point we just name it as
Chassis A, but we are not actually put the VMs in Chassis A.

This Container type is VM DRS Group:

VM DRS Group for Chassis B:
At this point we have four DRS groups:

Now we need to take the DRS object we created before: Chassis A and VM to Chassis A and tie
them together. The next step is to do the same for Chassis b and VM to Chassis B

* This configuration needs to be part of DRS Rules.

Edge Cluster -> Manage -> Settings -> DRS Rules

Click on the Add button in DRS Rules, in the name enter something like: VMs Should Run on
Chassis A

In the Type select Virtual Machine to Hosts because we want to bind the VMs group to the Hosts
Group. In the VM group name choose VM to Chassis A object.

Below the VM group selection we need to select the group & hosts binding enforcement type. We
have two different options:

Should run on hosts in group

Must run on hosts in group

If we choose Must option, in the event of the failure of all our ESXi hosts in this group (for
example Chassis A as critical power outage) other ESXi hosts in the cluster (Chassis B) will not be
considered as option for vSphere HA recovery VMs.

Should option will take other ESXi hosts as recovery option.

Same for Chassis B:

Now the problem with the current of the DRS rules and the VM placement in this Edge cluster is that
the Edge and DLR are actually running in the same ESXi host. We need to create anti-affinity DRS
rules.
Anti-Affinity Edge and DLR:

An Edge and DLR that belong to the same tenant should not run in the same ESXi host.

For Green Tenant:

For Blue Tenant:

The Final Result:

In the case of a failure of one of the ESXi hosts we dont face the problem where Edge and DLR are
on the same ESXi host, even if we have a catastrophic event of a chassis A or B failure.

Edge NAT

One of the most important NSX Edge features is NAT. With NAT (Network Address Translation) we
can change the Source or Destination of the IP address and TCP/UDP port. Combined NAT and
Firewall rules can lead to confusion when we try to determine the correct IP address to apply the
firewall rule.

To create the correct rule, we need to understand the packet flow inside the NSX Edge in details. In
NSX Edge we have two different type of NAT, SNAT (Source NAT) and DNAT (Destination NAT).

SNAT

Allow to translate internal IP address (for example private IP describe RFC 1918) to public External
IP address. In figure bellow any VMs from VXLAN 5001 that need outside connectivity to WAN
can translated to an external IP address which is configured on the Edge. For example, VM1 with IP
address 172.16.10.11 need to communicate with WAN internet, NSX Edge can translate it to a
192.168.100.50 IP address configured on Edge external interface.

Users from outside are not aware of the internal Private IP address.
DNAT

Allow to access internal private server from outside world. In the example figure below, users from
the WAN need to communicate with Server 172.16.10.11. NSX Edge DNAT was created so that the
users from outside connect to 192.168.100.51 and NSX Edge will translate this IP address to
172.16.10.11.
The following figure is the outline of the Packet flow process inside the Edge. The important parts are
where the SNAT/DNAT Action and firewall decision action are being taken.

We can see from this process that the ingress packet will evaluate against firewall rules before
SNAT/DNAT action.

Note: The actual packet flow details are more complicated with
more action/decisions in Edge flow, but the emphasis is on the NAT
and Firewall. NAT function will work only if firewall service is
enable.
Firewall rules and SNAT

Because of this packet flow the firewall rule for SNAT need to be apply at internal IP address object
and not NAT IP address.

For example, when a VM1 172.16.10.11 needs to communicant with WAN, the firewall rule needs to
be:

Firewall rules and DNAT

Because of this packet flow the firewall rule for DNAT need to be apply at NAT IP address object
and not Private IP address after the NAT. When a user from the WAN will send traffic to
192.168.100.51, this packet will be checked against this F.W rule and then the NAT will change it to
172.16.10.11

DNAT Configuration

User from outside need to access to internal web server with public IP address.
The server internal IP address is 172.16.100.11, the NAT IP address is 192.168.100.6
First step is to Create External IP on the Edge, this IP is secondary because this edge already has main
IP address. Btw the main IP address is mark with black Dot (192.168.100.3).

For this example, the DNAT IP address is 192.168.100.6.

Create DNAT Rule in the Edge:

Now pay attention to firewall rule need to be open one the Edge. The user coming from outside and
try to access 192.168.100.6, so the firewall rule need to allow this access.

Verification:

There are few ways to verify NAT. in our example users from any source address need to access IP
address 192.168.100.6, after the DNAT action happen the translated packet is 172.16.10.11.
The output of the command:

show nat

The output of the command:

Show firewall flow

We can see that packet entered to edge designated to 192.168.100.6, the return packet coming from
different IP address 172.16.10.11. That mean NAT happens here.

We can capture the traffic and see the actual packet.

Capture Edge traffic on its outside interface vNic_0, in this example user source IP address is
192.168.110.10 and destination is 192.168.100.6

The command to capture this is as below:

Debug packet display interface vNic_0

port_80_and_src_192.168.110.10

Capture edge traffic on its internal interface vNic_1, we can see destination IP address as changed to
172.16.10.11 because of DNAT:

SNAT configuration

All server from VXLAN 172.16.10.0/24 need to NAT on the outside interface of the Edge with IP
address 192.168.100.3
SNAT Configuration:

Edge Firewall Rules:

Allow to 172.16.10.0/24 to go out

For verification, use the following command

show nat
PAT

PAT Port Address translation allow to change Layer4 TCP/UDP port. For example, we would like
to mask our internal SSH server port for all users from outside.
The new port will be TCP/222 instead of regular SSH TCP/22 port.

The user establishes to Server with port TCP/222 but NSX Edge will change it to TCP/22.
NAT Order

For this scenario, we need to create two different SNAT rule.

SNAT Rule 1:

All VXLAN 5001 172.16.10.0/24 need to translate to the outside interface of Edge device, which is
192.168.100.3

SNAT Rule 2:

Web-SRV-01a is on VXLAN 5001 with IP address 172.16.0.11 that need to translate on outside
interface of Edge device, which is 192.168.100.4

In this example traffic will never hit rule number 4 because 172.16.10.4 is part of subnet
172.16.10.0/24.

We need to re-order the NAT and put the more specific NAT before the wide network.
After re-order:

Summary
In this chapter, we have discussed about the VMware NSX Edge Services Gateway and its different
functions, such as NAT, firewall, DHCP, HA etc in detail. We have discussed about design choices
and how will they impact the configuration as well. In the next chapter we will discuss about one of
the most powerful feature of VMware NSX and that is Distributed Firewall and its various use cases
and troubleshooting when it goes wrong.

 5
Distributed Firewall
With the release of NSX-v, a very strong and agile security technology was introduced to primarily
address and overcome certain limitations that were inherent in the vCloud Networking and Security
App product (also known and referred to as vShield App). One of the most noticeable and eagerly
awaited aspects, was the new version of firewall capabilities, and in particular the ability to deliver
close to line rate throughput due to an in-kernel firewalling module that was moved from a
previously service virtual machine based architectural security model.

The Distributed Firewall now provides a very flexible model as it is now embedded in the VMkernel
of the ESXi host and provides a secure juncture between the virtual machines and with the physical
world.

Micro-segmentation

Fortune 500 enterprises have been losing the war against cyber-criminal forces, according to the 2013
State of Cybercrime Survey from PwC and CSO magazine which included responses from 500 U.S.
executives, security experts, and others from both private and public sectors. These organizations are
turning to the latest and best tools for self-defense while trying to determine what economic impact
fighting cyber-crime will have on their organizations.

Hence, it requires you to continuously re-evaluate the current security standards and compliance
adherence in your datacenter strictly with a lot of CAPEX involved to achieve the desired level of
security. With the evolution of the NSX-v platform, this has been simplified with the Micro-
segmentation security model.

Micro-segmentation is not new, but conceptually introduces zones in your NSX for vSphere
environment. This enables you to manage your workload security without having to introduce
hardware firewalls to track sessions for your file servers, RDP or SSH to critical business servers and
so on.
Micro-segmentation helps you to critically define zones in your environment to retain workload
security with mobility without having to perform tedious reconfiguration when the workload moves
from one rack to another or even one datacenter to another. To achieve this functionality, the NSX-v
product provides an out-of-the-box Distributed Firewall feature, which can be leveraged to define
rules for the East-West traffic generated in your datacenter.
Generally, it is seen in the datacenter environment that the East-West traffic is the one that is
growing with the amount of application and business needs that continue to increase at a brisk pace.
Due to the growing business demands and concerns about security in modern cyber security warfare,
it has become critical to understand the points surrounding security hardening. According to
NIST/Forrester research recently which gave birth to the Zero trust security model, most of the
breaches in the cyber security have occurred due to the uncontrolled access or inefficient RBAC (Role
based access control) which pose a high security threat if a specific asset is compromised or stolen
which gives rise to most of the datacenter network exploits.

To further complicate the situation, most of the security devices today operate at the datacenter core
network that creates a traffic trombone effect directly impacting the performance. It is the hypervisor-
embedded nature of the distributed firewall that delivers close to line rate throughput to enable higher
workload consolidation on physical servers. The distributed nature of the distributed firewall provides
a scale-out architecture that extends firewall capacity when additional hosts are added to a data center.

The distributed firewall enables you to perform the traffic introspection at the virtual machines vNIC
level at almost line rate performance, which overcomes the above said constraint.
How does the Distributed Firewall Operates?

In the legacy security application for virtual environment, for example vCloud Networking and
Security App, formerly vShield App and before that known as vShield Zones leveraged a Service
VM deployed on every single ESXi host to perform the inspection.

Actually, vCloud Networking and Security App Firewall comprised of three entities and the vCloud
Networking and Security Manager (aka vShield Manager) which provided the centralized
management for all vCloud Networking and Security (aka vShield) products.

vCloud Networking and Security Manager: Within the context of the vCloud Networking and
Security App Firewall, the vCloud Networking and Security Manager is a centralized management
console which allows users to:

Define firewall policies to control the traffic in/out of the environment.

Define Spoofguard Policies
Define Namespace configuration (also known as realm)
View historical flow data going in/out of the environment.
Lifecycle management of the vCloud Networking and Security Manager App appliance

The three components of the vCloud Networking and Security App Firewall are:

1. vCloud Networking and Security dvFilter module: An ESX Loadable Kernel Module (LKM)
that sits in the ESX hypervisor and provides hooks to channelize virtual machine traffic to the
vCloud Networking and Security App service VM using VMware dvfilter APIs. The LKM
module is also known as fastpath while its counterpart the Service VM is known as slowpath.
 vCloud Networking and Security App Service VM aka VSA is a Service VM or appliance
which performs the network traffic introspections. It reports, allows or denies traffic for a
virtual machine based on the policies configured by end user via the vCloud Networking and
Security Manager. The vCloud Networking and Security dvFilter module redirects all
the traffic for protected VMs to this service VM.

2. The vSA service VM provides control and datapath processes. The control path process
also known as Sysmgr communicates with the vCloud Networking and Security Manager (a
java process) over a secured, encrypted channel to receive the control information i.e. firewall
configurations, Spoofguard configuration, flow configuration and other data.

3. vCloud Networking and Security dvFilter properties on protected virtual machines: Three
properties and associated values are added to the vmx configuration file of every vnic of
every virtual machine that is on a host where vCloud Networking and Security App has been
installed. These values inform the VMs on a given ESXi host that vCloud Networking and
Security App is present. All vCloud Networking and Security appliances are excluded from
receiving these properties.

NOTE: slow path and fast path are just VMware internal names to
developers. A fast path is a Linux Kernel Module (LKM) sitting in
the kernel/hypervisor/ESXi and performs processing of the traffic
in real time and its pretty fast, while slow path is a Service VM
getting its data from fast path and then processing traffic, there
is a round trip involved so it will slow the traffic.

This again raised concerns about the network performance as it was limited to less than 2 Gbps. NSX-
v deploys an in-kernel firewall module called vSIP (VMware Internetworking Service Insertion
Platform) that handles all traffic inspection sitting at the vNIC level of all virtual machine.
The components of distributed firewall are described below:

Message Bus based on AMQP: Advanced Message Queuing Protocol

Message Bus is used by the NSX-v Manager to reliably and securely transfer
firewall rules down to the ESXi host.
AMQP is an open standard application layer protocol for message-oriented
middleware. The defining features of AMQP are message orientation,
queuing, routing (including point-to-point and publish-and-subscribe),
reliability and security.

vSIP: VMware Internetworking Service Insertion Platform

vSIP is the distributed firewall kernel module component.
vSIP receives firewall rules from NSX-v Manager through the User World
Agent (UWA) and downloads them down to each virtual machine vNIC
(using vNIC-FW constructs)
Note: The VMware Internetworking Service-Insertion Platform is also a
framework that provides the ability to dynamically introduce 3rd party as well
as VMwares own virtual and physical security and networking services into
VMwares virtual network.
vNIC-FW:
Construct (or memory space) where firewall rules are stored and enforced.
This space contains rules table and connections tracker table

The above diagram explains the communication of the components involved in Distributed Firewall
Solution.
The above screen shot shows NSX-v Manager running RabbitMQ Server on port 5671 establishes a
connection with vsfwd (User world agent) running RabbitMQ client on the ESXI host.

The above screen shot shows VMware Internetworking Service Insertion Platform module installed
on an ESXI host.

Before we look at how RabbitMQ fits into the NSX-v and ESXi platform, we will start with a brief
introduction about what RabbitMQ actually is!

RabbitMQ provides robust messaging for applications, in particular for products such as NSX-v and
vCloud Director for Service Providers to name but a few. Messaging describes the sending and
receiving of data (in the form of messages) between systems. Messages are exchanged between
programs or applications, similar to the way people communicate by email but with selectable
guarantees on delivery, speed, security and the absence of spam.

A messaging infrastructure (a.k.a. message-oriented middleware or enterprise service bus) makes it

easier for developers to create complex applications by decoupling individual program components.
Rather than communicating directly, the messaging infrastructure facilitates the exchange of data
between components. The components need know nothing about each others status, availability or
implementation, which allows them to be distributed over heterogeneous platforms and turned off and
on as required.

In a NSX-v deployment, the NSX-v platform uses the open standard AMQP protocol to publish
messages associated with Blocking Tasks or Notifications. AMQP is the wire protocol natively
understood by RabbitMQ and many similar messaging systems, and defines the wire format of
messages, as well as specifying the operational details of how messages are published and consumed.

A RabbitMQ server or _broker_, runs within the NSX-v network environment, for example deployed
into NSX-v platform underlying installation as a virtual appliance. Clients belonging to the NSX-v
infrastructure itself, as well as other applications interested in notifications connect to the RabbitMQ
broker. Such clients then publish messages to, or consume messages from the broker.

The RabbitMQ broker is written in the Erlang programming language and runs on the Erlang virtual
machine. Notes on Erlang-related security and operational issues are presented later in this chapter.

The following RabbitMQ parameters are created on the ESXi host when prepared from NSX-v
Manager to create the message bus. This message bus is responsible for pushing down the rules to
every single ESXi host. The NSX-v Manager establishes this connection over port 5671.


When an ESXi host is being prepared from vSphere web client using the Networking and Security
plugin, the modules or vSphere Installation bundle (VIB) gets installed in every esxi host which is
added to the cluster. These VIBs are available in the NSX-v Manager and gets deployed on every
ESXi host using the ESX Agent Manager (EAM) component available in vCenter Server which
comes bundled with vCenter Management Web Services. EAM acts as a broker between any solution
and the ESXi hosts to deploy their bundles and modules, which can cohesively work to provide any
services to the virtual machines. For example: vCloud Networking and Security App deploys a virtual
machine on every ESXi host and also deploys a kernel dvfilter module (if not already present) and
they together work in fastpath-slowpath model.

The above RabbitMQ parameters are also configured as a part of the preparation process and includes
some critical parameters; for example: NSX-v Manager IP Address, certificate thumbprint, RabbitMQ
port etc. These parameters help vsfwd or the user world agent running the RabbitMQ client software
to connect to NSX-v Manager and operate together.

You could easily traverse through this set of parameters that are available using either the command
line or GUI. Heres an example of esxcfg-advcfg to check the value of /UserVars/RmqIpAddress
which shows the IP address of NSX-v Manager:

Note: Modifying any of these parameters could cause adverse

effects on the solution. Hence it is strictly recommended that
none of the above parameters should be modified unless advised by
VMware Technical Support.
Alignment of slots for the Distributed Firewall

A Slot are the points of injection of services or modules which acts as intermediary between the Guest
Operating System and the physical networking world. You could resemble a slot to a tap in the
networking however the difference here is a tap only monitors the traffic and a slot has a module
attached to it which can redirected or process the traffic as required. This redirection or processing is
based on the policies enforced either by NSX or any third party for advanced services like Deep
Packet Inspection. IO Chains are in kernel buffers which holds packet during processing of packets
before it is handed over the distributed switch.

The following slots are created in the ESXI kernel to provide services for the Distributed Firewall and
redirection capabilities when using NSX for vSphere.

Slot 0: DVFilter (Distributed Virtual Filter)

Distributed Virtual Filter or DVFilter is the vmkernel module between the protected vNIC at Slot 0
associated Distributed Virtual Switch (DVS) port, and is instantiated when a virtual machine with a
protected virtual NIC gets created. It monitors the incoming and outgoing traffic on the protected
virtual NIC and performs stateless filtering.

Slot 1: Switch Security (sw-sec)

sw-sec module learns VMs IP and MAC address. sw-sec is a critical component which captures
DHCP ACK and ARP broadcast messages and forwards this information as unicast to the NSX for
vSphere Controller to maintain the ARP table in order perform the ARP suppression feature. sw-sec is
the layer where NSX IP spoofguard is implemented,

NOTE: The Distributed Firewall (DFW) currently does not perform

any inspection in DHCPv6 packets. The spoofguard feature is
limited to MAC and IPv4/IPv6 addresses.

Slot-2: VMware-sfw:
This is the place where DFW firewall rules are stored and enforced, VMware-sfw contains rules table
and connections table.


Slot-4: 3rd Party Integration:
This is the place where the redirection rules are written to send traffic to a third party firewall for
traffic introspection. The point to note here is, only if a traffic is allowed by DFW at slot 2, the traffic
is redirected to a third party firewall (for instance PAN) to be inspected.

The below screenshot shows the Distributed Firewall module which is attached to slot 2 and switch
security module which is attached to slot 1.



What happens in case of VM mobility?

Distributed Firewall comes with a feature referred to as vMotion Callback which moves the security
rules with VM as it migrates within the datacenter or even across datacenters which are prepared for
NSX. This feature helps you to seamlessly migrate the virtual machine across racks without having
the needs to perform reconfiguration on your firewall and other security devices to maintain and meet
the compliance. It maintains rules and connection tracker tables to ensure that even if the VM is
migrated to the last most rack in the environment, no connections are passed uninspected the security
of the datacenter is still maintained across all levels.

The below diagram shows the tables maintained by dvfilter module on ESXI host. Rule table contains
all the rules enforced to a specific object (for instance: a virtual machine) and the connection tracker
tables holds all the live flows which were allowed on the ESXI host.
Heres a small depiction of how the rules move along with the virtual machine:
The machine was initially on a different host and hence the rules and filters were associated to that
specific host.
If you look through the command line, you would ideally see the filter associated with the virtual
machine on the ESXI host where this machine is powered ON. This filter is a unique name which is
one per virtual machine running on each host. The filter is the complete set of policies (rules, address
sets etc) which are defined through the distributed firewall pane on the Networking and Security
section of vSphere web client.

The filter associated with the virtual machines enlists all the rulesets which has to be processed for
any traffic originated or destined for a virtual machine.

The UUID mentioned in the filter shows which virtual machine this filter is associated with it. You
can retrieve this uuid from the virtual machine configuration file.
What you can also retrieve is the different rulesets and address sets that are associated with this
specific filter through command line and identify what are the different rules that are applied to each
object.

The above screen shot shows the different rules that are applied to the virtual machine.

The below screen shot shows different address sets associated with the filters and applied to specific
rules above. For instance, rule 1009 in the above screen shot has ip-securitygroup-13 and the ip
address associated with the security group are shown below.
2-Way Traffic Introspection

Since Distributed firewall sits exacts in between Guest and the Distributed Virtual switch and is
enforced on the vNIC layer, the inspection of the traffic happens on all the vNICs it crossed to reach
the guest. For instance, in the fig, the source VM sends out a packet to the destination is checked first
at the Sources vNIC level and then passed to the Distributed switch. Before the packet could be
passed to the Destination VMs GOS, it is again checked at the Destination VMs vNIC level. This
ensures that all the check happens diligently before the packet is delivered to the appropriate machine.
For an instance, if a rule RULE1 (using applied to option) is available only at source vm and a rule
RULE2 (using applied to option) is available only at the destination vm, this mechanism ensures
that both the rules are honoured before the packet is delivered.

What happens in case of a failure of this DFW functionality?

Since this is a module which operates at the kernel level, it is highly unlikely that the module would
fail as it gets loaded as a part of the boot image. However, in case of any failures of the distributed
firewall functionality; for an instance an ESXi host maxed out on the CPU, the traffic would be
blocked by default and packets would start dropping for the VMs which are protected. You can
change this default behavior to allow all traffic to go through using an API call which would allow
traffic to go through in the event of failure of distributed firewall:

Configuring Fail-Safe for Distributed Firewall:

PUT https://<nsxmgr fqdn or ip>/api/2.1/app/failsafemode

Request Body: FAIL_OPEN

Visibility and Packet Walks

For a fresh session, once the packet is generated by the source, the lookup is followed in the below
order:

Lookup happens against the connection tracker table

If no entry is found, the lookup is then done against the rule table
If a matching rule is found, action is taken appropriately.

Rule Lookup (first packet)

For an existing session, once the packet is generated by the source, the lookup is followed in the
below order:

Lookup is done against the connection tracker table

Since the Connection tracker table already has entry in it, the packet is forwarded.
 Rule Lookup (subsequent packets)

Rule Priority

Rules are prioritized from top to bottom as placed in the Distributed Firewall section in the
Networking and Security plugin. The first match is taken up and actioned appropriately (allow or
deny). For example, if you have an allow rule for SSH for a VM on top and then you have a deny all
traffic rule for the same VM below, SSH is allowed.
Also, L2 based rules (Data link layer attributes) takes priority over L3 based rules (network layer
attributes). Definition of rule is 5-tuple** based.

And finally there are other rules which are generated as third party solutions (for example, Palo Alto
Networks) that are used to provide advanced L7 based firewall rules.

** A 5-tuple refers to a set of five different values that

comprise a Transmission Control Protocol/Internet Protocol
(TCP/IP) connection. It includes a source IP address/port number,
destination IP address/port number and the protocol in use.

The above screen shot shows the L3 section of the Distributed Firewall which correlates to Network
and Transport layer parameters that can be applied to inspect traffic.

The above screen shot show the L2 section of the Distributed Firewall which correlates to the Data
link layer parameters that can be applied to a rule to inspect the traffic.
If you look carefully through the command line, you will see the segregation of the rules sets.

The rules are under ruleset domain-c25_L2 are Ethernet based rules and takes precedence over the
L3 based rules. In this case we applied a block rule on L2 and and allow rule on L3 between 2 VMs.
The end-result as expected:

Also, the rules are prioritized in the order they are created or applied in the Distributed Firewall
window. The rules are applied from a top-down order and the first match of the rule takes precedence
over the rule available at the bottom. You can move a rule up or down using the icons provided in the
Distributed Firewall window.
 Move a rule up

Move a rule down

Partner Security Services are intended to populated rules created by the partner solutions that work in
conjunction with the NSX solution. Distributed Firewall and Partner security services works side by
side to provide advanced firewall services (for instance; Palo Alto Networks for DPI)

The traffic from the NSX environment is steered to a PAN based firewall appliance, which performs
an advanced level of packet introspection and provides secured services across the complete
environment. However, the traffic is only steered to the PAN firewall if it is allowed in the
Distributed Firewall. Thats exactly why I said they work side by side!

Partner Security Services

Distributed Firewall Static Objects

Firewall rules can be applied at various levels and different objects which DFW understands. Listed
are some of the objects on which you can apply the policies:
1. vCenter Containers (Clusters, Datacenters portgroups etc)
2. IP Sets(IPV6 Complaint) and MAC sets
3. Active Directory Groups

Distributed Firewall Dynamic Object

NSX comes with a very powerful and dynamic container called security group which can help
maintain security for growing workloads with similar attributes to be applied with similar policy. It is
a predefined set of policies which can cater to needs of future workload without the need of
associating policies to the workloads once they are created.
For example: If a NSX administrator wants all future Windows workloads to be a part of the same
security group with a set of firewall policy to block all connections except RDP, he can do so by
defining a Security Group with the dynamic membership with Windows as Guest OS and and
associating a Security Policy with the RDP enabled and default policy to be blocked.

If a new VM is created with Windows OS, it automatically moves to this newly created security group
and applies the pre-created firewall policies
 Security Groups

Creation of Security Groups

To create a security group, go to Security Groups under Service Composer and Click on the symbol.

a. Type a name for the security group (recommended to follow a specific naming convention for
ease of administration for example, Win2k8-DCs)

b. Select a criterion for Dynamic membership. For example: for a Guest OS type select
Windows Server 2008 R2. This defines your future membership also and hence you need to
be very careful during the definition.
c. Select static objects to be a part of the group. For example, there is a legacy DC running
Windows Server 2003 which is also required to be a part of the Domain Controller Group.
 d. Select the objects to be excluded from the complete list. For example, a Domain Controller
which is undergoing migration and soon decommissioned.

The security group membership can be defined by the following formula:

(Expression result + Inclusions result) Exclusions results

Security Policy

With Security policy we can create templates containing DFW policy approved by security admins
and this becomes how you want to protect your environment, then apply this on security groups
WHAT you want to protect.


We can apply security policy to more than one security group, for example below apply Web
Security Policy to both Security Group 1 and Security Group 2.

Different option even contrary is to apply two different security policy to same security groups. For
example, Security Policy 1 and Security Policy 2 to WEB Security Groups:

The Precedency of Security policy will be determined by Weight value configure by Security
Admins. Heres a small depiction of the above use case:
We have created 2 security policies namely:

Allow ICMP SP and Allow HTTP SP and apply this to pre-Created Security Group Web
Servers

Create Allow ICMP SP:

In Create Firewall Rules the Action is Allow.

1. The Weight is 4300

2. The source filed is: any
3. Destination is: Policy Security Groups. This is the interesting part, because this security
policy work as template we may reuse it for different security groups, the goal is to no tied
this template to specific security group.
4. Service: ICMP ECHO Request, ICMP ECHO Replay.


Ready to compute and click Finish:

At firewall tab we can note this point we did not apply this Security Policy to any security groups, this
policy is not activated yet. We can see it as gray policy in the firewall tab, another point here to see in
the designation there is no security group:
Create Allow WEB SP is the same way:

Pay Attention to Weight Field is 1300, is lower than previously ALLOW ICMP SP 4300. Create
the WEB rule, same concept like previous:

The Firewall policy order Allow ICMP before Allow WEB

Now we Applied both security policy on same security group. With Apply Policy:

Chose Allow ICMP Security Policy:

And do the same for second security policy called Allow WEB SP.
In Security Policy tab view, we can see the results of this action:

From firewall tab we can see that now we have two activated service composer security rules.

In the service composer Canvas view, we have excellent summery of security services applied to
security group:

Applied To Field

Applied to field helps to narrow down the objects on which a policy applies to. By default, if the
rule is written without an Applied To field select with an object, the rule gets applied to all objects
referenced in the rule.

For instance, if we create a specific rule on Distributed Firewall, and do not filter the objects where it
has to be applied to, it gets applied to all the Clusters that have been prepared from NSX Manager to
provide the Firewall services. Heres a small explanation about it.

I have 4 VMs on the ESXi host and each of these filters are associated to each VM.
However, I have only created rules related to the last 2 VMs that you see in the pictures for which the
UUID is mentioned i.e., web-sv-01a and web-sv-02a as shown in the below figure:

However, when I check the rule sets of VMs that are not a part of the rules (either source or
destination), I can see even their filters having the rules, which refers to these VMs.
Look at the rule 1006 which shows it is a part of web-sv-02a as well as an another machine which is
hosted on the same ESXi host but not part of the rule. This is where the applied to filter comes into
picture:

After applying the Applied To field only for web VMs, this is what it looks like:
The rule 1006 is no more applied to any other VM except the web based VMs, in this case web-sv-
02a.

Identity Firewall

The coolest feature of NSX that brings you with more granularity of control over the users traffic
through integration with Active Directory. Identity Firewall allows you to control what a user can
access amongst the network resources.

For example: If you want to allow only the Domain Admins to map a shared drive or if you want only
specific users to RDP into a specific set of servers.

This fine grain control is achieved by integration your Active Directory with NSX domain. Heres
how you can accomplish the tasks:

Integrate your NSX with Active Directory

Go to NSX Manager and click on the NSX Manager that you wish to integrate with AD.
Under Domains, click on the plus symbol to start adding your NSX manager to a domain.

Provide the Name of the Active Directory Domain

Provide Details and Credentials about AD

Provide the Security Event Log Access Details

Summary of the information provided

Create Identity Based Firewall Rules

Adding rules with Security Groups with the membership of AD Groups.

Create a Security Group

Define the membership and add a group from AD

Write rules with the created security groups.

Now, you might have a question. How does it write rules in the backend?
Heres the answer:
The Address Set shows IP address of NSX Manager.

Application Level Gateway (ALG)

Application Level Gateway (ALG) is the ability of a firewall or a NAT device that can either allow or
block applications that uses dynamic ephemeral ports to communicate. In the absence of ALG, it
could be a nightmare for security and network administrators with the options of trade off between
communication and security. A network administrator can suggest opening a large number of ports
which would pose security threat for the network or the given server while a security administrator
can suggest blocking all other ports except the known ports which again breaks the communication.
ALG reads the network address found inside the application payload and opening respective ports for
preceding communication and also synchronizing data across multiple sessions across different ports.
For example: FTP uses different ports for session initiation/control connection and actual data
transfers. An ALG would manage any information passed on the control connection as well as data
connection in the above case.

NSX-v acts as ALG for few protocols such as FTP, CIFS, ORACLE TNS, MS-RPC, SUN-RPC.

Backup and Recovery

NSX offers you a very agile solution with various options of recoverability in case of errors or
misconfigurations.

You can export your complete set of working rule set and then go ahead and make any changes at any
point of time with the option to revert back to the working state. This helps you to gain a snapshot-
oriented operation wherein you can quickly fall back to a working configuration if in case a change in
any rules causes disruption to the services.

To export your current firewall configuration, all you need to do is hit Export Configuration icon on
the Distributed Firewall option on the left hand pane.

NSX also saves the complete configuration during modifications or specific intervals as shown in the
picture.
To be able to use the saved configuration, you need to load these saved configurations by using the
icon.

You can also upload a previously exported firewall configuration to the saved configuration and load
that specific configuration.
However, if in case during a restore operation of firewall configuration, if there were other rules
which were applied by partner security solutions which are not a part of these saved configurations,
they would be removed from the partner security services. To sync the new set of rules from the
partner security solutions, you need to click Synchronize Firewall Configuration from the Service
Composer tab.
You can also perform this sync operation using the following API:

URL: https://<nsxmgr-ip>/ api/2.0/services/policy/serviceprovider/firewall

Request: GET
Request Body:
<keyValues>
<keyValue>
<key>forceSync</key>
</keyValue>
</keyValues>

Working with Distributed Firewall API

NSX offers Restful APIs for Distributed Firewall Management. You can customize your
operations using these API calls by integration these into your home grown CMP or using
these as custom workflows in the vRealize Automation Platform. For instance, if you want a
specific rule to be added the moment a VM gets provisioned, you can use these API calls to
trigger updates to the existing rule sets.

This way you can make extensive use of the firewall rules and objects to automate your operations
and management.

You can download any rest client for your browser and start working with NSX APIs to configure the
Distributed Firewall.

The simplest of all is Rest Client for Mozilla:

https://addons.mozilla.org/en-US/firefox/addon/restclient/

You can set various headers while calling these API calls (for example, credentials, type of request
etc)
The first thing that you will have to do is set the Authentication.

These credentials would be used to authenticate and authorize your API calls to perform any changes
in the Distributed Firewall section.

Also if you are trying to POST or PUT any configurations using REST client, you also need to set a
custom header for Content-Type: application/xml

Alright, so we now were all geared to get set and go!

Lets look at how you can use the rest client to work with Distributed Firewall.
For an instance if you wish to query the complete configuration of your distributed firewall, heres
what you would do?

Request: GET
The above query would return all the rules you have created (including L2, L3 or any partner
solutions rules) along with the default rule.

How to identify your rules, the default rule and the Partner Solution Rules? Very easy, heres the
trick!

If you havent created any rules yet on the distributed firewall, this is how your Response Body for
the above API call would look like. Mark the Name of the rule which shows default rule

HTTP/1.1 200 OK
Cache-Control: private
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: no-cache
Set-Cookie: JSESSIONID=4CAE025C868939C35245B2553079807A; Path=/
ETag: 1395341576368
Date: Wed, 02 Oct 2013 20:58:39 GMT
Server: vShield Manager
Content-Type: application/xml
Transfer-Encoding: chunked
<?xml version="1.0" encoding="UTF-8"?>
<firewallConfiguration timestamp="1360144793284">
<contextId>globalr
oot-0</contextId>
<layer3Sections>
<section id="2" name="defaultSectionLayer3" gene
rationNumber="1360144793284" timestamp="1360144793284">
<rule id="2" disabled="false" logged="false">
<name>Default Rule</name>
<action>DENY</action>
<appliedToList>
<appliedTo>
<name>DISTRIBUTED_FIREWALL</name>
<value>DISTRIBUTED_FIREWALL</value>
<type>DISTRIBUTED_FIREWALL</type>
<isValid>true</isValid>
</appliedTo>
</appliedToList>
<sectionId>2</sectionId>
</rule>
</section>
</layer3Sections>
<layer2Sections>
<section id="1" name="defaultSectionLayer2" gene
rationNumber="1360144793284" timestamp="1360144793284">
<rule id="1" disabled="false" logged="false">
<name>Default Rule</name>
<action>ALLOW</action>
<appliedToList>
<appliedTo>
<name>DISTRIBUTED_FIREWALL</name>
<value>DISTRIBUTED_FIREWALL</value>
<type>DISTRIBUTED_FIREWALL</type>
<isValid>true</isValid>
</appliedTo>
</appliedToList>
<sectionId>1</sectionId>
</rule>
</section>
</layer2Sections>
</firewallConfiguration>

Every Rule is associated with a specific section that would be created. You can create multiple
sections on Distributed Firewall and every section would be denoted with a section id in this case
<sectionId>2</sectionId> for L3 rules and <sectionId>1</sectionId> for L2 based rules.
Hence, if you wish to create any section in DFW, you can perform it using the following query:

POST https://<nsxmgr-ip>/api/4.0/firewall/globalroot-0/config/layer3sections | layer2sections

<?xml version="1.0" encoding="UTF-8"?>

<section name="TestSection">
Rules Details
</section>

A distributed firewall rule consists of the options that you would see on the GUI. Not all fields are
mandatory and hence you may need to pass each value appropriately when adding or modifying any
rules. Heres how the API to add a firewall rule would look like:

REST API CALL:

POST

<?xml version="1.0" encoding="UTF-8"?>

<rule disabled="false" logged="false">
<name>AddRuleTest</name>
<action>allow</action>
<notes />
<appliedToList>
<appliedTo>
<value>datacenter-26</value>
<type>Datacenter</type>
</appliedTo>
</appliedToList>
<sectionId>2</sectionId>
<sources excluded="true">
<source>
<value>datacenter-26</value>
<type>Datacenter</type>
</source>
</sources>
<services>
<service>
<value>application-216</value>
</service>
</services>
</rule>

Troubleshooting

Distributed Firewall behaves like any other hardware firewalls but offers a set of commands which
you can run to easily troubleshoot connectivity issues or packet drops.

Installation Issues

In order to prepare your clusters for Distributed Firewall, there are multiple components which
interacts with each other to install the kernel modules and setup the message bus between NSX
manager and every single ESXi host.

For any solution which deploys VIBs or modules into ESXi host, ESX Agent Manager (EAM) plays
an important role and worth understanding in case you run into any issues.

EAM is an important component of vCenter Server and is a part of vSphere Management

WebServices. EAM acts as a broker for any solution that integrates with vCenter Server to provide
services for the virtual machines. To view details around ESX Agent manager, go to vCenter home >
vCenter Solutions Manager > vSphere ESX Agent Manager.

Here are the steps in which the deployment of Distributed Firewall VIBs will happen across all the
host in a cluster once you click on the Install option as shown in the figure.
 NSX Manager connects to vCenters EAM over port 80
EAM creates an agency for the Cluster being prepared for NSX.
EAM pulls down and caches the VIBs locally and then performs installation of the VIBs on
each ESXi host.

Now, if you run into any issues during the installation, you can check three logs to narrow down the
issues encountered:

1. NSX Manager logs

Use the following command on the SSH session to NSX Manager
#show manager log | follow
2. ESX Agent Manger logs
Goto C:\ProgramData\VMware\vCenter Server\logs and open eam.log
3. ESXi logs
Goto /var/log and open esxupdate.log

Before you go to the above mentioned logs, it worth checking the EAM status on GUI using vSphere
client/web client.

EAM creates an agency for any solution to maintain the status information of the solution.
On vSphere ESX Agent Manager, check the status of Agencies prefixed with _VCNS_153. If
any of the agencies has a bad status, select the agency and view its issues.
In case of any failures, NSX manager might show some logs similar to this. Now to dig further, we
could also look into other logs as mentioned above.

2014-12-01 07:40:01.529 UTC ERROR taskScheduler-34 VcEamConnection:85 - Error

connection to eam:Exception occurred while getting Tunneled VcConnection.; nested
exception is java.util.concurrent.ExecutionException:
com.vmware.vim.vmomi.client.exception.ConnectionException:
org.apache.http.conn.HttpHostConnectException: Connection to http://10.0.16.28:80
refused
com.vmware.vshield.vsm.vcserver.VcConnectionNotAvailableException:
core-services:500:vCenter Connection is not
available.:com.vmware.vim.vmomi.client.exception.ConnectionException:
org.apache.http.conn.HttpHostConnectException: Connection to http://10.0.16.28:80
refused
Hence, moving to eam.log we should be able to track down the issue of any failures. In this case the
vCenter Server Managed IPv4 Address was not set correctly and hence you see null:80 in the URL. If
in case, you see any other errors

ERROR | | host-1 | VibTask.java | 677 | Unhandled response code: 4

ERROR | | host-1 | VibTask.java | 683 | PatchManager operation failed:
<esxupdate-response>
<version>1.40</version>
<error errorClass="MetadataDownloadError">
<errorCode>4</errorCode>
<errorDesc>Failed to download metadata.</errorDesc>
<url>http://null:80/eam/vib?id=1ff30a9a-d773-4880-b98a-90bfcb8afd41-1</url>
<localfile>None</localfile>
<msg>('http://null:80/eam/vib?id=1ff30a9a-d773-4880-b98a-90bfcb8afd41-1',
'/tmp/tmpoLgjit', '[Errno 4] IOError: &lt;urlopen error [Errno -2] Name or service
not known&gt;')</msg>
</error>
</esxupdate-response>

INFO | | host-1 | IssueHandler.java | 167 | Unique issue added:

AgentIssueHandler:agent-4
eam.issue.VibNotInstalled {
key = 1,
time = 2014-08-17 00:43:48,111,
description = 'XXX uninitialized',
solutionName = 'vShield Manager',
agency = 'Agency:agency-0',
agencyName = 'GenericFastPath',
solutionId = 'com.vmware.vShieldManager',
host = 'HostSystem:host-5316',
hostName = '10.226.200.190',
agent = 'Agent:agent-4',
agentName = 'not-initialized',
}
Location: VibTask.java:processErrorCode:685

We set the Managed IP address correctly and restarted vSphere Management WebServices post which
it showed up the URL correctly and the installation completed perfectly fine.

INFO | | vc-callback | Utils.java | 986 | Setting option:

VirtualCenter.AutoManagedIPV4 := 10.1.160.30
INFO | | vc-callback | Utils.java | 986 | Setting option:
VirtualCenter.AutoManagedIPV6 :=
INFO | | vc-callback | Utils.java | 986 | Setting option: VirtualCenter.FQDN :=
wwvc0.cloud.solnetsolutions.co.nz
INFO | | vc-callback | Utils.java | 986 | Setting option: VirtualCenter.ManagedIP
:= 10.1.160.30
INFO | | vlsi-2 | ClientAuthenticator.java | 280 | Authenticating session with
physical vCenter cookie. SHA1: E027F8B3
And finally you could see something around errors in esxupdate.log as well.
2014-08-16T19:13:46Z esxupdate: esxupdate: INFO: ---
Command: scan
Args: ['scan']
Options: {'nosigcheck': None, 'retry': 5, 'loglevel': None, 'cleancache': None,
'viburls': None, 'meta': ['http://null:80/eam/vib?id=1ff30a9a-d773-4880-b98a-9
0bfcb8afd41-1'], 'proxyurl': None, 'timeout': 30.0, 'cachesize': None, 'hamode':
True, 'maintenancemode': None}
2014-08-16T19:13:47Z esxupdate: BootBankInstaller.pyc: INFO: Unrecognized value
"title=Loading VMware ESXi" in boot.cfg
2014-08-16T19:13:47Z esxupdate: vmware.runcommand: INFO: runcommand called with:
args = '['/sbin/bootOption', '-rp']', outfile = 'None', returnoutput = 'True'
, timeout = '0.0'.
2014-08-16T19:13:47Z esxupdate: vmware.runcommand: INFO: runcommand called with:
args = '['/sbin/bootOption', '-ro']', outfile = 'None', returnoutput = 'True'
, timeout = '0.0'.
2014-08-16T19:13:47Z esxupdate: downloader: DEBUG: Downloading
http://null:80/eam/vib?id=1ff30a9a-d773-4880-b98a-90bfcb8afd41-1 to
/tmp/tmpoLgjit...
2014-08-16T19:13:47Z esxupdate: esxupdate: ERROR: An esxupdate error exception was
caught:
2014-08-16T19:13:47Z esxupdate: esxupdate: ERROR: Traceback (most recent call
last):
2014-08-16T19:13:47Z esxupdate: esxupdate: ERROR: File "/usr/sbin/esxupdate",
line 216, in main
2014-08-16T19:13:47Z esxupdate: esxupdate: ERROR: cmd.Run()
2014-08-16T19:13:47Z esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-
1331820/bora/build/esx/release/vmvisor/sys-boot/lib/python2.6/site-packages/
vmware/esx5update/Cmdline.py", line 106, in Run
2014-08-16T19:13:47Z esxupdate: esxupdate: ERROR: File "/build/mts/release/bora-
1331820/bora/build/esx/release/vmvisor/sys-boot/lib/python2.6/site-packages/
vmware/esximage/Transaction.py", line 71, in DownloadMetadatas
2014-08-16T19:13:47Z esxupdate: esxupdate: ERROR: MetadataDownloadError:
('http://null:80/eam/vib?id=1ff30a9a-d773-4880-b98a-90bfcb8afd41-1', None,
"('http://
null:80/eam/vib?id=1ff30a9a-d773-4880-b98a-90bfcb8afd41-1', '/tmp/tmpoLgjit',
'[Errno 4] IOError: <urlopen error [Errno -2] Name or service not known>')")
2014-08-16T19:13:47Z esxupdate: esxupdate: DEBUG: <<<
2014-08-16T19:14:01Z esxupdate: vmware.runcommand: INFO: runcommand ca

Communication Issues:

Audit Logs

Audit Logs can be really useful to identify the logins of different enterprise admins or any sort of
configuration changes which could have resulted in catastrophic conditions. This helps in keeping a
track of multiple admins who would be performing similar changes with similar rights also trace back
to a situation which could have been the cause of some major environment outage.

Example of user bob login to system:

The information regarding any policy changes is also available in vsfwd.log.

Example for output:

2015-03-10T02:43:12Z vsfwd: [INFO] Received vsa message of RuleSet, length 67

2015-03-10T02:43:12Z vsfwd: [INFO] Processed vsa message RuleSet: 67
2015-03-10T02:43:12Z vsfwd: [INFO] L2 rule optimization is enabled
2015-03-10T02:43:12Z vsfwd: [INFO] applying firewall config to vnic list on host
host-10
2015-03-10T02:43:12Z vsfwd: [INFO] Enabling TCP strict policy for default drop
rule.
2015-03-10T02:43:12Z vsfwd: [INFO] sending event: applied vmware-sfw ruleset
1425955389291 for vnic 500e519a-87fd-4acd-cee2-c97c2c6291ad.000
2015-03-10T02:43:12Z vsfwd: [INFO] successfully saved config to file
/etc/vmware/vsfwd/vsipfw_ruleset.dat
2015-03-10T02:43:12Z vsfwd: [INFO] Sending vsa reply of domain-c7 host host-10: 0

Firewall Logs

Starting from NSX for vSphere version 6.1, DFW have dedicated log file to view start/termination
session and drop/pass packets. The log contains rule id associated with vCenter objects and their
actions taken for a session. The logs can come very handy when troubleshooting issues about
connectivity and it shows detailed information about every rule which got applied to a session.

File name is: dfwpktlogs.log

File location ESXi host file: /var/log/dfwpktlogs.log
Example of output file:
#more /var/log/dfwpktlogs.log
2015-03-10T03:22:22.671Z INET match DROP domain-c7/1002 IN 242 UDP
192.168.110.10/138->192.168.110.255/138

1002 is the DFW rule-id

domain-c7 is the vCenter object
192.168.110.10/138 is the source IP
192.168.110.255/138 is the destination IP

By default, when we create DFW rule there is no logging enabled. To view the Log Filed we need to
enable the Log option filed.
Live Flow

With DFW we have ability to view live flows. These flows are pulled by the vsfwd from the vsip
kernel module and aggregated appropriately. NSX Manager pulls normal flows from vsfwd every 5
min, real time flows every 5 secs.

To Enabled the Flow Monitoring Go to Flow Monitoring -> Configuration and click on the Enable.
Global Flow Collection should change to greed Enabled status

To View vNIC flow go to Live Flow tab and browse specific VM and vNIC.

Click the start button

Live flow will show up in the screen as shown in the figure. The refresh Rate is 5 second.

Summary
In this chapter, we have discussed about the Distributed Firewall and its various use cases and
troubleshooting when it goes wrong. We have discussed Microsegmentation, Identity based firewall,
troubleshooting and packet walk as well. In the next chapter we will discuss about Integration of
Cisco UCS with VMware NSX. We will discuss the UCS Components and how they all fit together
with NSX.

 6
Integration of NSX for vSphere with
Cisco UCS
In recent years there have been a number of changes observed in the data center and cloud computing
industry. Only faster virtual machine provisioning is not helping the current requirements, virtual
network and security would be needed with similar pace to meet the demands of current industry trends.
VMware NSX is an ideal platform for virtualizing the network and security running on top of Cisco
UCS converged physical infrastructure. VMware NSX doesnt provide only virtual network and
security but integrated with Cisco UCS simplifies infrastructure, reduces costs, automates operations
and increases efficiency. Distributed virtual switching, routing, bridging, load balancing and security
capabilities are inbuilt into VMware NSX can provide highly scalable and secure cloud architecture.

Running VMware NSX on Cisco UCS doesnt require any changes to the underlying hardware and
network infrastructure. NSX can be installed in a customers existing or new environment smoothly
without the need to modify any changing any physical topology. It will require minimum connectivity
to existing physical network topology. Once installed & configured, it can automatically provision the
required layer 2 to layer 7 services for virtual infrastructure, with simple easy to go steps.

This chapter will provide the brief introduction to Cisco Unified Computing System and integration of
VMware NSX with Cisco UCS.

Discuss Cisco Unified Computing System Architecture: Understand the Cisco unified
compute architecture and its various components

Discuss Running NSX on UCS: Understand the system/network requirements for NSX to
deploy on UCS.

Cisco Unified Computing System Architecture

Cisco Unified Computing System commonly known as UCS, is an integral part of the compute strategy
for any data center and plays a vital role in designing the application services either bare-metal or
virtualized environment. UCS is the first converged platform that combines industry-standard, Intel
x86-architecture servers with networking, storage and virtualization into a single cohesive system.

A typical blade server system adopts the mini-rack approach, where complex and expensive
switching technology is included in each deployed chassis. This might include both Fibre Channel
(FC) and Ethernet switching, which can significantly increase the cost to scale your system as an
additional chassis may then cost tens of thousands of dollars before a single blade is added to the
system. While switching within the chassis does offer the benefit of allowing blade to blade switching
to be kept within the chassis, it also means management of such a system is much more complex.
Each chassis requires additional on-board management and configuration, and the cost of
management software alone can add hundreds of thousands of dollars to the system cost.
Cisco UCS does not include switching elements in each chassis, keeping the cost to scale the system
extremely low and making the system much easier to manage. The low-latency and high bandwidth
10Gbe connectivity throughout UCS also reduces intra-chassis switching to a negligible benefit in
competing systems, which can be offset by reduced OPEX costs due to simple system management
and configuration.
The Cisco UCS solution reduces the number of components by up to 30%, reducing power and
cooling requirements while simultaneously simplifying system configuration, growth and
management. The key differentiator, the Fabric Interconnects, converge functionality of FC and
Ethernet switching into a single component while also embedding system management in a fully
redundant, clustered manner.
Cisco UCS is a set of pre-integrated data center components that comprises of following components:

Fabric Interconnect
Chassis
Fabric Extender
Blade Server
Converged Network Adaptor
UCS Manager

All the components are well managed with one single pane of glass embedded into Fabric
Interconnect called Cisco UCS Manager (UCSM). We will discuss about each components in
details.
Fabric Interconnect

Cisco UCS Fabric Interconnect is the brain of the Unified Computing System and makes a perfect
compute solution for any data center and cloud computing solutions. Fabric Interconnect is specialized
switch, which provide uniform access to compute, network and storage, eliminating multiple
components to separately access them. Fabric Interconnect is based on Cisco Nexus 5000 Series
switching platform, combining power of customized NX-OS system and embedded with management
capabilities referred as Cisco UCS 6000 Series Fabric Interconnects.

The first generation of Fabric Interconnect starts with 6100 series switches and 6200 is second
generation series provide better throughput, low-latency & unified fabric ports while offering higher
capacity, higher port density, and lower power consumption. The Fabric Interconnect provide the data,
management and storage connectivity to Cisco UCS B-Series Blade servers through UCS Fabric
Extenders inside the Blade Chassis on unified fabric. The unified fabric provides the same channel to
support LAN & SAN connectivity for all blade servers. The LAN channel includes the data and
management traffic for blade servers to communicate with outer world.

Cisco has designed the fully redundant solution and deployed as a pair of Fabric Interconnect to
provide the high availability. The ports on the Fabric Interconnect can be connected either to server
side called server ports and uplink to the network switches called network ports. The network ports
providing LAN and SAN connectivity to all the blade connected to the Fabric Interconnect.

The Cisco UCS 6200 Fabric Interconnect is currently comprised of two models.

Cisco UCS 6248UP Fabric Interconnect

Cisco UCS 6296UP Fabric Interconnect

Cisco UCS 6248UP Fabric Interconnect

The Cisco UCS 6248UP 48-Port Fabric Interconnect is a one RU, 10-GE, FCoE and native Fiber
Channel switch providing more that 960 Gbps throughput with low latency. The switch has 32
1/10Gbps fixed unified ports and one expansion module slot can be used up to 16 additional ports.
The unified ports can be configured as traditional Ethernet, Fiber Channel or FCoE ports and provide
greater flexibility to overall solution.

Cisco UCS 6296UP Fabric Interconnect

The Cisco UCS 6296UP 96-Port Fabric Interconnect is a two RU, 10-GE, FCoE and native Fiber
Channel switch providing more that 1920 Gbps throughput with low latency. The switch has 48
1/10Gbps fixed unified ports and three expansion module slots can be used up to 48 additional ports.

Chassis

The Cisco UCS 5108 Series Blade Server Chassis is a 6 RU that can accommodate up to max eight
half-width Cisco UCS B-Series Blade servers or up to four full-width Cisco UCS B-Series Blade
Servers, or combination of half-width and full-width blade servers. The blade servers are connected
through midplane inside the chassis and can be connected with two Fabric Extenders to provide
connectivity to Fabric Interconnect. These Fabric Extenders are configured in redundant mode to
provide failure of any ports or Fabric Extender Switch.

The UCS Chassis is managed with Chassis Management Controllers (CMC), which are embedded in
the UCS Fabric Extenders. The Fabric Interconnect communicates with CMC to manage all the
components through UCS Manager.

The UCS Chassis is designed in such a way that it provides maximum free space for blade servers
cooling. The chassis can be powered with four 2500 W power supplies in non-redundant, n+1
redundant or n+n redundant mode. For cooling of all the components inside the chassis, 8 hot
swappable fans provide front to rear cooling.

A single Cisco UCS can scale up to 40 individual chassis and 320 half-width blade servers with one
uplink. At this time Cisco support up to 20 individual chassis and 160 half-width blade servers.

Fabric Extender

Cisco UCS Fabric Extender commonly known as FEX or I/O Module is interlink switch between
blade servers and Fabric Interconnect. FEX forward all traffic from blade servers in a chassis to
Fabric Interconnect over unified fabric links.

The first generation of Fabric Interconnect starts with 2100 series switches and 2200 is second
generation series provide more throughput for blade server traffic.

The Cisco UCS 2200 Fabric Extender is currently comprised of two models.

Cisco UCS 2204XP Fabric Interconnect

Cisco UCS 2208XP Fabric Interconnect

Cisco UCS 2204XP Fabric Extender

The Cisco UCS 2204XP Fabric Extender has four 10GE, FCoE capable uplink ports that connect the
blade chassis to the fabric interconnect and 16 10GE port connected through midplane to each blade
slot in the chassis. Two Fabric Extenders provide redundant solution with 80 Gbps of I/O to the single
chassis.

Cisco UCS 2208XP Fabric Extender

The Cisco UCS 2208XP Fabric Extender has eight 10GE, FCoE capable uplink ports that connect the
blade chassis to the fabric interconnect and 32 10GE port connected through midplane to each blade
slot in the chassis. Two Fabric Extenders provide redundant solution with 80 Gbps of I/O to the single
chassis.

Blade Servers

The Cisco UCS Blade Server further extends the capabilities of Cisco Unified Computing System by
delivering new levels of manageability, performance, energy efficiency, reliability, security, and I/O
bandwidth for enterprise-class virtualization and other mainstream data center workloads. Cisco UCS
Blades servers power the compute industry with x86 Architecture based on next generation Intel Xeon
processors adapt to application demands, intelligently scale energy use, and offer best-in-class
virtualization. Each Cisco UCS B-Series blade server utilizes converged network adapters for access
to the unified fabric with various levels of transparency to the operating system. Half width and full
width blade servers are designed for various kinds of different workloads.

Based on features and capabilities, following the Cisco B Series Blade Servers is available.

Cisco UCS B460 M4 Blade Server

Cisco UCS B440 M2 Blade Server
Cisco UCS B420 M3 Blade Server
Cisco UCS B260 M4 Blade Server
Cisco UCS B230 M4 Blade Server
Cisco UCS B200 M3 Blade Server
Cisco UCS B22 M3 Blade Server

Network Adaptors

Cisco UCS Network Adapters are offered in a mezzanine-card form factor. Three types of adapters
offer a range of options to meet application requirements, including adapters optimized for
virtualization, compatibility with existing driver stacks, or efficient, high-performance Ethernet.

Ethernet Adaptors: Intel & Broadcom based network silicon provides the conventional Ethernet
network capabilities to the blade servers called Ethernet Adaptors. Cisco UCS System supports
following Ethernet Adaptors in earlier generations.

Cisco UCS CNA M61KR-I Intel

Cisco UCS NIC M51KR-B Broadcom

Converged Network Adapters: Emulex and Qlogic based network silicon provides both Fibre
Channel and Ethernet adaptors on same adaptors called Converged Network Adaptors (CNA). Cisco
UCS supports following Converged Network Adaptors.

Cisco UCS CAN M73KR-E Emulex

Cisco UCS CAN M73KR-Q QLogic
Cisco UCS CAN M72KR-E Emulex
Cisco UCS CAN M72KR-Q QLogic

Virtual Interface Cards: Cisco based network silicon provides both Fibre Channel and Ethernet
adaptors up to 256 total virtual network adaptors. This supports VM-Link/802.1BR technology, which
extends the Cisco UCS fabric interconnect ports to virtual machines, simplifying server virtualization
deployment. Cisco UCS supports following Converged Network Adaptors

Cisco UCS Virtual Interface Card (VIC) 1240

Cisco UCS Virtual Interface Card (VIC) 1280
Cisco UCS Virtual Interface Card (VIC) 1340
Cisco UCS Virtual Interface Card (VIC) 1380
UCS Manager

Cisco UCS Manager serves as the central nervous system of the Cisco Unified Computing System and
manages all software and hardware components of UCS including Fabric Interconnect, Chassis, Fabric
Extender, Blades & adaptors. No separate management software need to be installed on the system, it
is embedded into Fabric Interconnect. The UCS Manager is fully redundant solution and work as active-
standby mode in case of failure on one Fabric Interconnect. UCS Manager is single point of
management for services like server provision, device discovery, inventory management, fault
monitoring & statistic analysis. It uses the service profiles, pool, policies & templates to provision the
servers and provide full control to server, network and storage admin to manage the respective roles
inside the UCS Manager.

Running NSX on UCS

VMware NSX decouples the network services from physical networks and provides the layer 2 to
layer 7 services for virtual machines, when integrated with Cisco UCS Converged Infrastructure,
customer can get better performance and faster deployment of virtual machines without changing
anything on the infrastructure. UCS Manager Plugin with VMware vCenter can provide single glass
of pane for managing & monitoring the physical and virtual infrastructure.

The VMware NSX can be deployed on any model discussed above in the UCS Architecture. Based on
the application and scalability requirements, customer can choose various form factor of blades
servers, fabric interconnect, fabric extenders and converged network adaptors. Other than Cisco UCS
B-Series Architecture, customer can select C-Series Rack Mounted Servers, can be managed or
unmanaged independently with Cisco UCS Manager. The Cisco C-Series blade server can provide
scalability and higher performance with directly connecting to network & storage switches.

We will discuss about the pre-requisite to integrate the VMware NSX with UCS Converged
Infrastructure and show how to install, configure and NXS Manager on Cisco UCS Server, then how
to create a logical switch and connect with virtual machines. This section will not cover any
installation of VMware ESXi on Cisco UCS blade servers and vCenter Server, assumptions will be
virtual infrastructure is ready on Cisco UCS infrastructure. For more detail on design considerations
for your infrastructure, refer to VMware NSX on Cisco Nexus 7K and UCS Design Guide:
www.vmware.com/files/pdf/products/nsx/vmware-nsx-on-cisco-n7kucs-design-guide.pdf and

VMware NSX Network Virtualization Design Guide

www.vmware.com/files/pdf/products/nsx/vmw-nsx-network-virtualization-design-guide.pdf

VMware NSX reduces the dependency on network administrators for provisioning the number of
VLANs, VRF, and Security Policies on various physical devices to quickly provision the instant
virtual machines with desired switching, routing and security policies. The basic network
configuration is required to connect the Cisco UCS Fabric Interconnect to Nexus Switch. Fabric
Interconnect can be connected either in vPC or non-vPC mode with Nexus Switches. The Unified
Computing System provides the basis foundation for hosting the management, Control, data and
virtual machines components. The management components can create in separate cluster to isolate
from the production virtual machine and offering multi-tenancy services.

Hardware Requirements

VMware NSX can be deployed on any generation of Cisco Unified Computing System and Nexus
Switches. The Nexus 70xx/77xx series is the ideal platform for connecting VMware NSX running on
Cisco Unified Computing System providing better performance and scalability. However, customer
can use any Cisco server model and Nexus switches for this deployment.

For this book, we have considered the following hardware devices

Cisco Nexus 7018 2

UCS Fabric Interconnect 6248 2
UCS Fabric Extender 2204 - 2
UCS Chassis 5108 - 1
UCS B200 M2 4
UCS VIC M81 2

Software Requirements

VMware NXS 6.x would require VMware vSphere ESXi 5.5 platform and firmware considerations
for Cisco UCS should be 2.x and for Cisco Nexus 6.0.x

For this installation, we have used the following software versions

VMware NSX 6.1.4

 VMware ESXi 6.0
vCenter Server Appliance 6.0
UCS Manager 2.2
Nexus 6.0.2

VMware NSX provides flexibility in designing the multi-tier application architecture and gives better
control to manage layer 2 to layer 7 traffic for Virtual Machines running on UCS Infrastructure. The
multiple environments like production, development and test can run the same IP addressing on the
same underlying physical infrastructure, still provide the isolations between them. The UCS Fabric
Interconnect works in end-host mode and provides network isolation for Virtual Machine Traffic. The
high I/O intensive applications, which require east-west traffic, perform better when integrated with
VMware NSX running on Cisco UCS infrastructure. VMware Distributed Logic Router can route the
east-west traffic internally to Fabric Interconnect.

The architecture diagram below illustrates the tier 3-application architecture on deployed on VMware
NSX and Cisco UCS Infrastructure.
The logical architecture diagram below illustrates the virtual machines communicate through logical
switches, logical distribute routers, edge service routes to corporate network.

Corp%Network%
GW:%10.105.220.1%

10.105.220.118%

GW:%192.168.10.2%

192.168.10.3%

GW:%172.16.10.1% GW:%172.16.30.1%

GW:%172.16.20.1%

Web%Switch% App%Switch% DB%Switch%

Web% Web% App% App% DB% DB%

VM01% VM01% VM01% VM02% VM01% VM02%
172.16.10.11% 172.16.10.12% 172.16.20.11% 172.16.20.12% 172.16.30.11% 172.16.30.12%

VLAN and IP Addressing

VLANs need to be enabled on the Nexus 7K and Cisco UCS to configure the VMware NSX on
unified infrastructure. Minimum 2 VLANs for VXLAN transport and Edge Gateway need to be
defined on the Nexus 7K and UCS to communicate to the corporate network. Once these VLANs are
configured, the VMware NSX supports multiple VLANS for Virtual machine traffic to forward traffic
through these VLANs.

The below tables can be used for planning the VLANs and IP Address for your deployment.
Table: ESXi Host Server IP Addressing

ESXi Host Name IP Address Subnet Gateway VLAN VMKNIC

IP address
ESXi Server 1
ESXi Server 2
ESXi Server 3
ESXi Server 4

Table: VMware NSX Manager and Components IP Addressing

Virtual Machine IP Address Subnet Gateway VLAN

Name
vCenter Server
NSX Manager
NSX Controller 1
NSX Controller 2
NSX Controller 3
Primary DNS
Secondary DNS
Primary NTP
Secondary NTP

Table: LDR, ESG and Virtual Machine IP Addressing

Tenant 1 IP Address Subnet Gateway VLAN

ESR01
DLR01
Web VM01
Web VM02
App VM01
App VM02
DB VM01
DB VM02

Summary
In this chapter, we have discussed about the Integration of Cisco UCS with VMware NSX. We will
discuss the UCS Components and how they all fit together with NSX. In the next chapter we will
deep dive into the most powerful Distributed Logical Router.
 7
Distributed Logical Router
In todays modern Datacenter, the physical router is essential for building a workable network design.
As in the physical infrastructure, we need to provide similar functionality in virtual networking.
Routing between IP subnets can be performed in a logical space without traffic going out to the
physical router. This routing is performed in the hypervisor kernel with a minimal CPU and memory
overhead. This functionality provides an optimal data-path for routing traffic within the virtual
infrastructure. Distributed routing capability in the NSX-v platform provides an optimized and
scalable way of handling East - West traffic within a datacenter. East - West traffic is the
communication between virtual machines within the datacenter. The amount of East - West traffic in
the datacenter is growing. The new collaborative, distributed, and service oriented application
architecture demands a higher bandwidth for server-to-server communication.

If these servers are virtual machines running on a hypervisor, and they are connected to different
subnets, the communication between these servers has to go through a router. Also, if a physical
router is used to provide routing services the virtual machine communication has to go out to the
physical router and get back in to the server after the routing decisions have been made. This is
obviously not an optimal traffic flow and is sometimes referred to as hair pinning.

The distributed routing on the NSX-v platform prevents the hair-pinning by providing hypervisor
level routing functionality. Each hypervisor has a routing kernel module that performs routing
between the Logical Interfaces (LIFs) defined on that distributed router instance.

The distributed logical router possesses and manages the logical interface (LIF). The LIF idea is
similar to interfaces VLAN on a physical router. But on the distributed logical router, the interfaces
are called LIFs. The LIF connects to the logical switches or distributed port groups. A single
distributed logical router can have a maximum of 1,000 LIFs.

Overview of the Distributed Logical Router (DLR)

The best way to explain the DLR is to look at how the Backbone Data Center works today. Inside this
big router we have one or two supervisor cards, which is the brain of the chassis (Control Plane) and
we have many line cards performing the forwarding (Data Plane). The DLR is basically made up of
two components; the first component is the DLR Control VM (Control Plane) that is a virtual
machine. The Dynamic routing protocol runs between the DLR Control VM and upper router; this can
be a NSX Edge Services Gateway (which will now be referred to as the ESG throughout this book)
device or physical router. The second component is the distributed kernel module that acts like a line
card (Forwarding plane). The distributed kernel module runs on each ESXi host.
When the DLR Control VM learns its route(s), the routing table is pushed to the NSX controller and
then to all ESXi hosts running the DLR kernel module.
DLR Interfaces type

With the DLR we have three types of interfaces. These are called Uplink, LIFs and Management.

Uplink: This is used by the DLR Control VM to connect the upstream router. In most of the
documentation you will see, it is also referred to as transit, and this interface is the transit interface
between the logical space to the physical space. The DLR supports both OSPF and BGP on its Uplink
Interface, but cannot run both at the same time. OSPF can be enabled only on single Uplink Interface.

LIFs: LIFs exist on the ESXi host at the kernel level; LIFs are the Layer 3 interface that act as the
default gateway for all VMs connected to logical switches.

Management: DLR management interface can be used for different purposes. The first one is to
manage the DLR control VM remote access like SSH. Another use case is for High Availability. The
last one is to send out syslog information to a syslog server. The management interface is part of the
routing table of the control VM; there is no separate routing table. When we configure an IP address
for the management interface only devices on the same subnet as the Management subnet will be able
to reach the DLR Control VM management IP, and the remote device will not be able to contact this
IP.
 Note: If we just need the IP address to manage the DLR remotely we
can SSH to the DLR Protocol Address explain later in this
chapter, there is no need to configure new IP address for
management interface.

Logical Interfaces, virtual MAC & Physical MAC

Logical Interfaces (LIFs) including IP address of the DLR Kernel module inside the ESXi host. For
each LIF we will have an associated MAC address called virtual MAC (vMAC). This vMAC is not
visible to the physical network. The virtual MAC (vMAC) is the MAC address of the LIF and is the
same across all the ESXi hosts and is never seen by the physical network, only by virtual machines.
The virtual machines use the vMAC as their default gateway MAC address. The physical MAC
(pMAC) is the MAC address of the uplink through which traffic flows to the physical network, and in
this case when the DLR needs to route traffic outside of the ESXi host it is the Physical MAC
(pMAC) address that will be used.

In the following figure, inside esxcomp-01a that is an ESXi host, we have the DLR kernel module,
this DLR instance will have two LIFs. Each LIF is associated with a logical switch VXLAN 5001 and
5002. From the perspective of VM1, the default gateway is LIF1 with IP address 172.16.10.1, VM2
has a default gateway that is LIF2 172.16.20.1 and vMAC is the same mac address for both LIFs.

The LIFs IP address and vMAC will be the same across all NSX-v hosts for the same DLR instance.

When VM2 is vMotioned from esxcomp-01a to esxcomp-01b, VM2 will have the same default
gateway (LIF2), which is associated with vMAC, and from the perspective of VM2 nothing has been
changed.

DLR Kernel module and ARP table

The DLR does not communicate with the NSX-v Controller to figure out the MAC address of VMs.
Instead it sends an ARP request to the entire ESXi host VTEPs members on that logical switch The
VTEPs that receive this ARP request forward it to all VMs on that logical switch.
In the following figure, if VM1 needs to communicate with VM2, this traffic will route inside the
DLR kernel module at escomp-01a, this DLR needs to know the MAC address of VM1 and VM2.
The DLR will then send an ARP request to all VTEP members on VXLAN 5002 to learn the MAC
address of VM2. In addition to this, the DLR will also keep the ARP table entry for 600 seconds,
which is called its aging time.

Note: The DLR instance may have different ARP entries between
different ESXi hosts. Each DLR Kernel module maintains its own ARP
table.

DLR and local routing

Since the DLR instance is distributed, each ESXi host has a route instance that can route traffic. When
VM1 need to send traffic to VM2, theoretically both DLR in esxcomp-01a and esxcomp-01b can
route the traffic as in the following figure. In NSX-v the DLR will always perform local routing for
VMs traffic!

When VM1 sends a packet to VM2, the DLR in esxcomp-01a will route the traffic from VXLAN
5001 to VXLAN 5002 because VM1 has initiated the traffic.

The following illustration shows that when VM2 replies back to VM1, the DLR at esxcomp-01b will
route the traffic because VM2 is near to the DLR at esxomp-01b.

Note: the actual traffic between the ESXi hosts will flow via
VTEPs.


Note: the actual traffic between the ESXi hosts will flow via
VTEPs.
Multiple Route Instances

The Distributed Logical Router (DLR) has two components, the first one is the DLR Control VM that
is a virtual machine and the second one is the DLR Kernel module that runs in all ESXi hypervisor.
This DLR Kernel module, which is called, route-instance has the same copy of information in each
ESXi host. The Route-instance works at the kernel level. We will have at least one unique route-
instance of the DLR kernel module inside the ESXi host but not limited to just on ESXi host.

The following figure shows two DLR control VMs, with the DLR Control VM1 on the right and DLR
Control VM2 on the left. Each Control VM has its own route-instance in the ESXi hosts. In esxcomp-
01a we have the route-instance1, which is managed by the DLR control VM1, and route-instance 2,
which is managed by the Control VM2, and the same also applies to escomp-01b. The DLR instance
has its own range of LIFs that it manages. The DLR control VM1 manages the LIF in VXLAN 5001
and 5002. The DLR control VM2 manages the LIF in VXLAN 5003 and 5004.

Logical Router Port

Regardless of the amount of route-instances we have inside the ESXi hosts we will have one special
port called the Logical Router Port or vdr Port.
This port works like a route in stick concept. That means all routed traffic will pass through this
port. We can think of route-instance like vrf lite because each route-instance will have its own LIFs
and routing table, even the LIFs IP address can overlap with others.
In the following figure we have an example of an ESXi host with two route-instances where in route-
instance-1 we have the same IP address as route-instace-2, but with a different VXLAN.

Note: Different DLRs cannot share the same VXLAN

Routing information Control Plan Update Flow

We need to understand how a route is configured and pushed from the DLR control VM to the ESXi
hosts. Lets look at the following figure to understand the flow.

Step 1: An end user configures a new DLR Control VM. This DLR will have LIFs (Logical
interfaces) and a static or dynamic routing protocol peer with the NSX-v Edge Services gateway
device.

Step 2: The DLR LIFs configuration information is pushed to all ESXi hosts in the cluster that have
been prepared by the NSX-v platform. If more than one route instance exists, the DLR LIFs
information will be sent to that instance only.
At this point VMs in a different VXLAN (East - West traffic) can communicate with each other.

Step 3: The NSX-v Edge Services gateway (ESG) will update the DLR control VM about new routes.

Step 4: The DLR control VM will update the NSX-v controller (via UWA) with Routing Information
Tables (RIBs).

Step 5: Then NSX-v controller will push RIBs to all ESXi hosts that have prepared by the NSX-v
platform. If more than one route instance exists, RIBs information will send to that instance only.

Step 6: Route Instance on the ESXi host creates Forwarding Information Base (FIB) and handles the
data path traffic.

DLR Control VM communications

The DLR Control VM is a virtual machine that is typically deployed in the Management or Edge
Cluster. When the ESXi host has been prepared by the NSX-v platform, one of the VIBs creates the
control plane channel between the ESXi hosts to the NSX-v controllers. The service demon inside the
ESXi host which is responsible for this channel, is called netcpad, and which is also more commonly
referred to as the User World Agent (UWA).

The netcpad is responsible for communication between the NSX-v controller and ESXi host learns
MAC/IP/VTEP address information, and for VXLAN communications. The communication is
secured and uses SSL to communicate with NSX-v controller on the control plane. The UWA can
also connect to multiple NSX-v controller instances and maintains its logs at /var/log/ netcpa.log

Another Service demon called the vShield-Statefull-Firewall is responsible for interacting with the
NSX-v Manager. This service daemon receives configuration information from the NSX-v Manager
to create (or delete) the DLR Control VM, create (or delete) the ESG. Beside that, this demon also
performs NSX-v firewall tasks: Retrieve the DFW policy rules, gather the DFW statistics information
and send them to the NSX-v Manager, send audit logs and information to the NSX-v Manager. Part of
host preparation processes SSL related tasks from the NSX-v Manager.

The DLR control VM runs two VMCI sockets to the user world agents (UWA) on the ESXi host it is
residing on. The first VMCI socket is to the vShield-Statefull-Firewall service daemon on the host for
receiving update configuration information from the NSX-v Manager to the DLR control VM itself,
and the second to netcpad for control plane access to the controllers.

The VMCI socket provides the local communication whereby the guest virtual machines can
communicate to the hypervisor where they reside but cannot communicate to the other ESXi hosts.
On this basis the routing update happens in the following manner:

Step (1) DLR Control VM learn new route information (from the dynamic routing as an
example) to update the NSX-v controller,
Step (2) the DLR will use the internal channel inside the ESXi01 host called the Virtual
Machine Communication Interface (VMCI). VMCI will open a socket to transfer learned
routes as Routing Information Base (RIB) information to the netcpa service daemon.
Step (3) The netcpa service demon will send the RIB information to the NSX-v controller.
The flow of routing information passes through the Management VMkernel interface of the
ESXi host, which means that the NSX-v controllers do not need a new interface to
communicate to the DLR control VM. The protocol and port used for this communication is
TCP/1234.
Step (4) NSX Controller will forward the DLR RIB to all netcpa service daemons on the
ESXi host.
Step (5) netcpa will forward the FIBs to the DLR route instance.

DLR High Availability

The High Availability (HA) DLR Control VM allows redundancy at the VM level. The HA mode is
Active/Passive where the active DLR Control VM holds the IP address, and if the active DLR Control
VM fails the passive DLR Control VM will take ownership of the IP address (flip event). The DLR
route-instance and the interface of the LIFs and IP address exists on the ESXi host as a kernel module
and are not part of this Active/passive mode flip event.

The Active DLR Control VM sync-forwarding table to secondary DLR Control VM, if the active
fails, the forwarding table will continue to run on the secondary unit until the secondary DLR will
renew the adjacency with the upper router.

The HA heartbeat message is sent out through the DLR management interface. We must have L2
connectivity between the Active DLR Control VM and the Secondary DLR Control VM. IP address
of Active/Passive assign automatic as /30 when we deploy HA. The default failover detection
mechanism is 15 seconds but can be lowered down to 6 seconds. The heartbeat uses UDP Port 694 for
its communication.

You can also verify the HA status by running the following command:
DLR HA verification command:

show service highavailability

show service highavailability connection-sync
show service highavailability link

Protocol Address and Forwarding Address

The Protocol address is the IP address of the DLR Control VM. This Control Plane actually
establishes the OSPF or BGP peering with the ESGs. The following figure shows OSPF as example:

The following figure shows that the DLR Forwarding Address is the IP address that uses as the next-
hop for ESGs.

DLR Control VM Firewall

The DLR Control VM can protect its Management or Uplink interfaces with the built in firewall. For
any device that needs to communicate with the DLR Control VM itself we will need a firewall rule to
approve it.

For example, SSH to the DLR control VM or even OSPF adjacencies with the upper router will need
to have a firewall rule. We can Disable/Enable the DLR Control VM firewall globally.

Note: do not confuse DLR Control VM firewall rule with

NSX-v distributed firewall rule. The following image
shows the firewall rule for DLR Control VM.

DLR Validation Scenarios

In this section we will explain how to validate routing information, this validation will illustrate all
the concepts that are explained in this chapter.

The validation scenario between two VMs in the same ESXi host. Lets look at the following image
where web-sv-01a and app-sv-01a are located in esxcomp-01a. The DLR control VM learns BGP
dynamic routes from the ESG not shown here.

1. Validate DLR Control plan:

a. DLR Control VM
b. NSX-v Controllers
c. ESXi host:
2. Capture on the vdr-port

DLR Control Plane validation

As we described earlier in this chapter in the Routing information Control Plan Update Flow step 1
and 2 The DLR route-instance LIFs and vMAC configuration need to be pushed from the NSX-v
Manager to the DLR Control VM then to the NSX-v controller and finally to the ESXi hosts DLR
kernel Module. We need to validate the control plane status of this path.
Compare between the LIFs configuration in the GUI (NSX-v Manager) to the DLR Control VM. In
rare cases we may face issues with LIFs interfaces configured in the NSX-v Manager GUI, but
actually are not showing in the DLR Control VM.

SSH to the DLR Control VM or open up the console of this VM and run the following command:

DLR-01-0> show interface

[IFNAME] Inteface name

tlr-01-0> show interface
Interface VDR is up, line protocol is up
index 2 metric 1 mtu 1500 <UP,BROADCAST,RUNNING,NOARP>
HWaddr: 82:71:35:08:9e:b3
inet6 fe80::8071:35ff:fe08:9eb3/64
inet 172.16.10.1/24
inet 172.16.20.1/24
proxy_arp: disabled
Auto-duplex (Full), Auto-speed (1341Mb/s)
input packets 0, bytes 0, dropped 0, multicast packets 0
input errors 0, length 0, overrun 0, CRC 0, frame 0, fifo 0, missed 0
output packets 0, bytes 0, dropped 0
output errors 0, aborted 0, carrier 0, fifo 0, heartbeat 0, window 0
collisions 0

The VDR port is not a real vNIC interface on the DLR Control VM, this VDR port exist on the ESXi
host contain all the LIF internal interfaces, From the output above line 5, 6 & 7 we can see that we
have 3 LIFs with correct IP address and subnet mask.

Validate Routing in DLR

We can verify routing information base RIB and forwarding information base FIB.
Show RIB with the command from the DLR Control VM show ip route

dlr-01-0> show ip route

Codes: O - OSPF derived, i - IS-IS derived, B - BGP derived,

C - connected, S - static, L1 - IS-IS level-1, L2 - IS-IS level-2,
IA - OSPF inter area, E1 - OSPF external type 1, E2 - OSPF external type 2,
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

Total number of routes: 7

B 0.0.0.0/0 [200/0] via 192.168.10.1

C 172.16.10.0/24 [0/0] via 172.16.10.1
C 172.16.20.0/24 [0/0] via 172.16.20.1
C 192.168.10.0/29 [0/0] via 192.168.10.6
B 192.168.100.0/24 [200/0] via 192.168.10.1
B 192.168.101.0/24 [200/0] via 192.168.10.2
Show FIB with the command show ip forwarding

dlr-01-0> show ip forwarding

Codes: C - connected, R - remote,

> - selected route, * - FIB route

R>* 0.0.0.0/0 via 192.168.10.1, vNic_2

C>* 172.16.10.0/24 is directly connected, VDR
C>* 172.16.20.0/24 is directly connected, VDR
C> 172.16.30.0/24 is directly connected, VDR
C>* 192.168.10.0/29 is directly connected, vNic_2
R>* 192.168.100.0/24 via 192.168.10.1, vNic_2
R>* 192.168.101.0/24 via 192.168.10.2, vNic_2
In the case we need to sync the configuration for NSX-v Manager to DLR control VM we have two
option: Force Sync or Redeploy.

Force Sync option will reboot the DLR control VM and push the configuration to from NSX-v
Manager.

Redeploy will terminate and delete the DLR control VM, Create new one with same configuration
from NSX-v Manager.

In this case we need to use redeploy option in the GUI to redeploy the DLR Control VM as shown in
the image below.

Note: this action is disruptive to the forwarding path!

NSX-v Controller

Routing information Control Plan Update Flow step 4 we need to verify whether the NSX-v
controller knows the DLR LIFs and vMAC. We are expecting to have the same LIFs and RIBs from
the DLR control VM in the NSX-v controller.

We may have more than one DLR control VM in NSX-v that means we will have more than one
route-instances. Each route-instance will have a unique id and we need to determine what this ID is.
Log in to your DLRs Control VM, and display its configuration to find out the DLR ID.

tlr-01-0> show configuration

-----------------------------------------------------------------------
vShield Edge Global Config:
{
"global" : {
"edgeAssistId" : 1460487509,
"enableTcpLoose" : false,
"hostname" : "tlr-01-0",
"hypervisorAssist" : false,
"fips" : {
"enable" : false
},
.
snip

From the above output, you can see that the DLR ID is 1460487509. Lets log in to any of the NSX-v
controllers to find the DLRs name and which service controller manages this route-instance as shown
in the following figure.

nvp-controller # show control-cluster logical-routers instance 1460487509

LR-Id LR-Name Hosts[] Edge-Connection Service-Controller

1460487509 default+edge-1 192.168.110.52 192.168.110.201

192.168.110.51

192.168.210.51

192.168.210.56

192.168.210.57

192.168.210.52
LR-ID: DLR Control VM unique ID number in the system.
LR-Name: DLR Control VM unique name in the system.
Host: This is the IP address of the management interfaces of all the ESXi hosts running this routing-
instance.
Service-Controller: This is the IP address of the NSX-v controller, which manages this route-instance.

Verify LIFs exist on the NSX-v Controller

Log into the NSX-v controller and look for the DLR (see above how to find it), and display the DLRs
LIFs table:

nvp-controller # show control-cluster logical-routers interface-summary

1460487509

Interface Type Id IP[]

570d455500000002 vxlan 5000 192.168.10.2/29

570d45550000000b vxlan 5002 172.16.20.1/24

570d45550000000c vxlan 5003 172.16.30.1/24

570d45550000000a vxlan 5001 172.16.10.1/24

If the LIFs are present in the NSX-v Manager and DLR Control VM, but not on the Controller, there
may be a synchronization issue between the NSX-v DLR control VM and NSX-v Controller.

ESXi Host

Validated the DLR route instance information is pushed from the NSX controller to the ESXi host.
Log into the source and destination ESXi hosts and list their DLRs.

~ # net-vdr -I l

VDR Instance Information :

---------------------------

VDR Instance: default+edge-1:1460487509

Vdr Name: default+edge-1
Vdr Id: 1460487509

Number of Lifs: 4

Number of Routes: 5
State: Enabled

Controller IP: 192.168.110.201

Control Plane Active: Yes

Control Plane IP: 192.168.210.51

Edge Active: No


We can see that the vdr exists and that vdr name is: default+edge-1, the vdr name will be needed
later.

From early in this chapter we learned that the ESXi host communicate with the NSX-v controller
happens via a process called the netcpad that runs inside the ESXi host. We need to verify this
process is actually running and has established a connection to the NSX-v controllers.

The command to verify netpcad is running:

# /etc/init.d/netcpad status
netCP agent service is running

Verify that the host has ESTABLISHED a connection to the NSX-v Controller(s). To do this, search
for port 1234 to show only TCP port 1234.

esxcomp-01a # esxcli network ip connection list | grep 1234

tcp 0 0 192.168.210.51:54153 192.168.110.202:1234 ESTABLISHED 35185 newreno netcpa-worker
tcp 0 0 192.168.210.51:34656 192.168.110.203:1234 ESTABLISHED 34519 newreno netcpa-worker
tcp 0 0 192.168.210.51:41342 192.168.110.201:1234 ESTABLISHED 34519 newreno netcpa-worker

The following example depicts the problem with communication from the ESXi host to the NSX-v
Controller(s):


1 esxcli network ip connection list | grep 1234
2 tcp 0 0 192.168.210.51:54153 192.168.110.202:1234 TIME_WAIT 0
3 tcp 0 0 192.168.210.51:34656 192.168.110.203:1234 FIN_WAIT_2 34519 newreno
4 tcp 0 0 192.168.210.51:41342 192.168.110.201:1234 TIME_WAIT 0

If we cannot see ESTABLISHED connection we need to check the following:

1. IP connectivity from the ESXi host Management interface to all the NSX-v controllers.
2. If you have a firewall between the ESXi hosts to the NSX-v Controllers, then make sure
TCP/1234 port is open.

If netcpad is not running on the ESXi host then start netcpad:

$ /etc/init.d/netcpad status
netCP agent service is not running

$ /etc/init.d/netcpad start
Memory reservation set for netcpa
netCP agent service starts

Check the netcpa status again:

$ /etc/init.d/netcpad status
netCP agent service is running

Check that the host has the correct IP address for the NSX-v Manager:

# esxcfg-advcfg -g /UserVars/RmqIpAddress

Validate the DLR LIFs are present on the ESXi host

We need to verify DLR LIFs existence for route-instance in ESXi host. The command to verify it is
net-vdr. net-vdr -L -l default+edge-1

The parameter for this command are:

-l = vdr-name which is default+edge-1 learned previously, and L = for LIFs

esxcomp-01a # net-vdr -L -l default+edge-1

VDR default+edge-1 LIF Information :
Name: 570d45550000000a
Mode: Routing, Distributed, Internal
Id: Vxlan:5001
Ip(Mask): 172.16.10.1(255.255.255.0)
Connected Dvs: Compute_VDS
VXLAN Control Plane: Enabled
VXLAN Multicast IP: 0.0.0.1
State: Enabled
Flags: 0x2388
DHCP Relay: Not enabled

Name: 570d45550000000c
Mode: Routing, Distributed, Internal
Id: Vxlan:5003
Ip(Mask): 172.16.30.1(255.255.255.0)
Connected Dvs: Compute_VDS
VXLAN Control Plane: Enabled
VXLAN Multicast IP: 0.0.0.1
State: Enabled
Flags: 0x2288
DHCP Relay: Not enabled
Name: 570d45550000000b
Mode: Routing, Distributed, Internal
Id: Vxlan:5002
Ip(Mask): 172.16.20.1(255.255.255.0)
Connected Dvs: Compute_VDS
VXLAN Control Plane: Enabled
VXLAN Multicast IP: 0.0.0.1
State: Enabled
Flags: 0x2388
DHCP Relay: Not enabled

Name: 570d455500000002
Mode: Routing, Distributed, Uplink
Id: Vxlan:5000
Ip(Mask): 192.168.10.5(255.255.255.248)
Connected Dvs: Compute_VDS
VXLAN Control Plane: Enabled
VXLAN Multicast IP: 0.0.0.1
State: Enabled
Flags: 0x2308
DHCP Relay: Not enabled


From this command we can see we have 4 different LIFs for route-instance default+edge-1.

If any LIFs are missing but we can see them in the NSX-v Manager and on the Controller, then the
host may be out of synchronization with the Controller.

To try and rectify this, run the following command on the Controller:

nvp-controller # sync control-cluster logical-routers interface-

to-host <logical-router-id> <host-ip>

The host-ip can be found by issuing the net-vdr -I -l command on the host, where its displayed
against the Control Plane IP

If the above did not help, try restarting netcpa on the host:

# /etc/init.d/netcpad restart

Validate the Forwarding table on the ESXi host

To validate DLR Routes on the ESXi host we need to run the net-vdr command with the R switch.
Note that missing forwarding routes will affect VMs which cannot communicate outside the
Datacenter. For VMs communicating inside the Datacenter we just need to have the DLR LIFs.
Log in into the source and destination hosts, and display their DLR forwarding tables:



~ # net-vdr -R -l default+edge-1

VDR default+edge-1:1460487509 Route Table

Legend: [U: Up], [G: Gateway], [C: Connected], [I: Interface]

Legend: [H: Host], [F: Soft Flush] [!: Reject]

Destination GenMask Gateway Flags Ref Origin UpTime Interface

----------- ------- ------- ----- --- ------ ------ ---------


0.0.0.0 0.0.0.0 192.168.10.1 UG 1 AUTO 370902 570d455500000002

172.16.10.0 255.255.255.0 0.0.0.0 UCI 1 MANUAL 370902 570d45550000000a

172.16.20.0 255.255.255.0 0.0.0.0 UCI 1 MANUAL 370902 570d45550000000b

172.16.30.0 255.255.255.0 0.0.0.0 UCI 1 MANUAL 370902 570d45550000000c

192.168.10.0 255.255.255.248 0.0.0.0 UCI 1 MANUAL 370902 570d455500000002

 Check if all prefixes are present and point to the correct interfaces. Entries with the Origin =
MANUAL are directly connected.
If any of these are missing, check if the LIFs are all present and configured correctly.
Check if all expected routes are present on the Controller.
If any are missing but present on the DLR Control VM, try to re-sync. Command is as
follows:

nsx-controller # sync control-cluster logical-routers route-from-

edge <logical-router-id>

Validate ARP Entries in the ESXi host of the DLR

In order for the DLR to be able to route packets to the next hop or between VMs, the DLR needs to
know the MAC address of the directly connected VMs, the remote VMs.

Verify that the DLR route-instance has ARP entries for Directly Connected VMs.

esxcomp-01a # net-vdr -N -l default+edge-1

VDR default+edge-1 ARP Information :

Legend: [S: Static], [V: Valid], [P: Proxy], [I: Interface]

Legend: [N: Nascent], [L: Local], [D: Deleted]

Network Mac Flags Expiry SrcPort Refcnt Interface

------- --- ----- ------ ------- ------ ---------

172.16.10.1 02:50:56:56:44:52 VI permanent 0 1 570d45550000000a

172.16.10.12 00:50:56:a6:a1:e3 V 536 50331650 3 570d45550000000a

172.16.10.11 00:50:56:a6:7a:a2 VL 266 50331657 3 570d45550000000a

172.16.30.1 02:50:56:56:44:52 VI permanent 0 1 570d45550000000c

172.16.20.11 00:50:56:a6:84:52 VL 271 50331659 3 570d45550000000b

172.16.20.1 02:50:56:56:44:52 VI permanent 0 1 570d45550000000b

192.168.10.5 02:50:56:56:44:52 VI permanent 0 1 570d455500000002

L = Local, this ARP entry that has be learnt Local to the DLR (not from a Remote route-
instance)

I= Interface means the L3 DLR LIF.

From Line 4 we can observe the DLR has Entry for:

web-sv-01a IP address 172.16.10.11 MAC 00:50:56:a6:7a:a2. ARP entry will aged out in 266 sec
app-sv-01a IP address 172.16.20.11 MAC 00:50:56:a6:84:52. ARP entry will aged out in 271 sec

If the ARP entry has not been learnt, this can be because the VMs are not connected to the correct
VXLAN or that L2 firewall rules are blocking this ARP message.

Capture Packets on the VDR Port

Capture traffic in or out from the VDR port can help us to troubleshoot L3 problems.

With the pktcap-uw command we can capture traffic in many places in the vDS or VM vNIC level.
This is depicted in the following illustration.

The capture packet received by vdr-port:

pktcap-uw --capture VdrRxTerminal --switchport <VDR_port> --lifID

<vni> [--proto <num>] -o <filename>

VdrRxTerminal = packet arrived to vdr port

VdrTxTerminal = packet sent out from vdr port
Switchport = PorID of the vdrPort
lifID = VNI = VXLAN Network Identifier.
Proto = Protocol for example 0x1 for icmp.
O = outpout to file

With the command pktcap-uw A we can view all capture points. Lets look at line 17 and 18
which shows it is the VDR port.

esxcomp-01a # pktcap-uw -A

Supported capture points:

1: Dynamic -- The dynamic inserted runtime capture point.

2: UplinkRcv -- The function that receives packets from uplink dev

3: UplinkSnd -- Function to Tx packets on uplink

4: Vmxnet3Tx -- Function in vnic backend to Tx packets from guest

5: Vmxnet3Rx -- Function in vnic backend to Rx packets to guest

6: PortInput -- Port_Input function of any given port

7: IOChain -- The virtual switch port iochain capture point.

8: EtherswitchDispath -- Function that receives packets for switch

9: EtherswitchOutput -- Function that sends out packets, from switch

10: PortOutput -- Port_Output function of any given port

11: TcpipDispatch -- Tcpip Dispatch function

12: PreDVFilter -- The DVFIlter capture point

13: PostDVFilter -- The DVFilter capture point

14: Drop -- Dropped Packets capture point

15: VdrRxLeaf -- The Leaf Rx IOChain for VDR

16: VdrTxLeaf -- The Leaf Tx IOChain for VDR

17: VdrRxTerminal -- Terminal Rx IOChain for VDR

18: VdrTxTerminal -- Terminal Tx IOChain for VDR

19: PktFree -- Packets freeing point

This is the Vdr status before web-sv-01a and app-sv-01a were powered-off

esxcomp-01a # net-vdr -N -l default+edge-1

VDR default+edge-1 ARP Information :

Legend: [S: Static], [V: Valid], [P: Proxy], [I: Interface]

Legend: [N: Nascent], [L: Local], [D: Deleted]

Network Mac Flags Expiry SrcPort Refcnt Interface

------- --- ----- ------ ------- ------ ---------

172.16.10.1 02:50:56:56:44:52 VI permanent 0 1 570d45550000000a

172.16.30.1 02:50:56:56:44:52 VI permanent 0 1 570d45550000000c

172.16.20.1 02:50:56:56:44:52 VI permanent 0 1 570d45550000000b

192.168.10.1 00:50:56:8e:a6:ff V 363 50331650 1 570d455500000002

192.168.10.5 02:50:56:56:44:52 VI permanent 0 1 570d455500000002

The vdr-port does not know the ARP entry for web-sv-01a 172.16.10.11 and app-sv-01a
172.16.20.11.

This is the Vdr status before web-sv-01a and app-sv-01a were powered-on. Now we power on both of
these.

esxcomp-01a # net-vdr -N -l default+edge-1

VDR default+edge-1 ARP Information :

Legend: [S: Static], [V: Valid], [P: Proxy], [I: Interface]

Legend: [N: Nascent], [L: Local], [D: Deleted]

Network Mac Flags Expiry SrcPort Refcnt Interface

------- --- ----- ------ ------- ------ ---------

172.16.10.1 02:50:56:56:44:52 VI permanent 0 1 570d45550000000a

172.16.10.11 00:50:56:a6:7a:a2 V 416 50331650 3 570d45550000000a

172.16.30.1 02:50:56:56:44:52 VI permanent 0 1 570d45550000000c

172.16.20.11 00:50:56:a6:84:52 VL 416 50331660 1 570d45550000000b

172.16.20.1 02:50:56:56:44:52 VI permanent 0 1 570d45550000000b

192.168.10.5 02:50:56:56:44:52 VI permanent 0 1 570d455500000002

Find the vdr-port id we use the esxtop command then press n for network:

Now we capture at the vdr-port for ICMP traffic and start to ping from web-sb-01a to
app-sv-01ap


pktcap-uw --capture VdrRxTerminal --switchport 50331657 --lifID
5001 --proto 0x1 -o VdrRxTerminal.pcap

pktcap-uw --capture VdrTxTerminal --switchport 50331657 --lifID
5001 --proto 0x1 -o VdrTxTerminal.pcap


Dynamic Routing Verification

SSH to the NSX-v Edge Services Gateway and type the following command:

show ip ospf neighbor or show ip bgp neighbor

The NSX-v Edge Services Gateway device has OSPF neighbor adjacency with 192.168.10.3 this is
the Control VM IP address.

The NSX-v Edge Services Gateway device has received OSPF Routes from the DLR.

From the NSX-v Edge Services Gateway device perspective the next-hope to the DLR is the
Forwarding Address 192.168.10.2
More useful commands:

- debug packet display interface <>

- debug packet capture interface <>
- show ip ospf neighbor
- show ip ospf database
- show ip ospf interface
- show ip bgp neighbors
- show ip bgp

Summary
In this chapter, we have done a deep dive into the most powerful Distributed Logical Router. We have
discussed about the routing techniques, troubleshooting, various command lines, packet walk etc. In
the last chapter we will discuss about some of the most significant new features introduced in
VMware NSX for vSphere 6.2.
 8
Whats new in NSX for vSphere version
6.2
Okay, so I think by now were all on the same page trying to see various abstracts of Software
Defined Network through VMwares binoculars; NSX for vSphere. But I am sure some of you by
now would have the same question which I had in my mind 2 months before I started writing this
book:

Whats next?

Exactly! And as you know we at VMware are called Architects of whats next and then we went
to Ready for any, I am sure I can answer that question to you in the best possible way I could.

Throughout this complete book weve been talking about features and components that evolved from
NSX for vSphere v6.0 till NSX for vSphere 6.1.4. This chapter majorly emphasizes on the features
and enhancements that came out in NSX for vSphere 6.2.

Introduction

NSX for vSphere has been through iterations in features set that it covers over a period of time and
the scalability to support enterprise grade networks. Version 6.2 of the product was to overcome the
limitations encountered with the pre 6.2 versions and the idea was to evolve the complete platform to
better support the Software Defined Datacenter. The major changes were mainly around the following
areas:

Hopping the fence of single vCenter management model

Support Multi-Site environment with Egress Optimization through Universal Objects
(Universal Distributed Logical Router & Universal Logical Switches)
Filling up security loopholes and ensuring
Better Operations and Visibility

These points are covered in details in the following sections. However, it is noteworthy to understand
the deep dive explanations of each component covered in earlier chapters in order to understand the
enhancements that were introduced with NSX for vSphere 6.2.
Cross vCenter Deployments through Egress Optimization

With the increased requirements around multi-site deployments with several datacenters serving
application needs simultaneously across a business, NSX for vSphere 6.2 really is an eye opener with
its unique capabilities around this used case.

You could now think of a centrally located Control and Management Plane with distributed workload
clusters serving the needs for applications.

The architectures can be as complex as shown in the following diagram which would serve for a
business continuity plan and business availability plan.

Figure: Sample Multi-Site Architecture

Egress Optimization is a technique or mechanism that is used to exit the local traffic through the local
gateway in a datacenter interconnect architecture. This is important in a L2 extension environment
wherein a VLAN is across multiple sites and 2 gateways with same IP can exists in both the
infrastructure. However, the local traffic should never take an non-optimal path through the DCI
connectivity and then exit out through the other sites gateway. Hence, egress optimization plays a
major role when considering multi-site architectures.
 Figure: Egress Optimization using L2 extension

With the introduction of NSX for vSphere, the constraints of having 1 NSX manager to 1 vCenter
Server is surpassed and now you can integrate multiple vCenter Servers to a centralized management
and control plane of NSX environment.

This becomes crucial with the need of extending software-defined networks and services across
multiple sites to seamlessly integrate with existing vSphere Platform managing huge environments.

Prior to 6.2, the DCI was achieved using manual affinity rules and configuration but the limit was still
a single vCenter Server. With the release of NSX for vSphere 6.2, Universal Logical Switch and
Universal Logical router helps in achieving DCI seamlessly with egress optimization.

Figure: Egress Optimization using NSX Universal Objects

Though this architecture is more latency driven, it doesnt really require any dedicated hardware (like
Cisco OTV) to achieve VLAN extensions across sites.
Increased Support for Cross vCenter vMotion

With increasing applications demands, most of the organizations are concerned about availability and
uptime of enterprise applications like ERP systems, databases, billing solutions etc. Most of the
organizations do have BCP for these applications cloud bursting to accommodate applications
increasing connections, auto scaling, active-passive model for applications with different sites etc.
However, in most of these cases theres some consideration that needs to be catered during the design
phase as the IP and MAC of the virtual machines needs to be exposed to the physical networking
world and hence the need of availability of VLANs, QoS configuration, routes redistributions in
WAN space etc should be well designed and planned.

Figure: Migration in a Multi-Site Environment with Universal Logical Switch

With the release of vSphere 6.0, VMware has realized that the limitation of vMotion needs within a
single vCenter needs to be widened and should cater to dynamic needs of agility, security and
availability. However, this would again require you to position your networking team to be able to
provide support for those applications on the physical networking layers as they would require similar
operating environment when the applications are de-rooted and moved across.

Figure: Migration in extended L2 using NSX Universal Logical Switch

With the availability of VXLAN where the VMs MAC and IP address is not really advertised on the
physical network, workloads can be seamlessly migrated to any datacenter without the need to IP
readdressing or VLAN provisioning. Multi-VC can be managed and prepared for clusters that would
fall in the same Transport Zone and can easily provision networks on both sites thereby providing the
same operating environment to all the datacenters where the virtual machine migrates.

This would really create a wave for the organizations that have been struggling with extending their
boundaries of applications from one datacenter to another. Also with the DLRs and edges running
dynamic routing protocols with the physical network, we can easily redistribute the routes in BGP that
can result in automatic failover of connections for an application stretched across sites.

Introduction of Roles

As you would have already noticed in the last diagram, we now have NSX Manager with roles. One
of the lucky NSX Manager in the datacenter would be designated as Primary NSX Manager which
means that this guy would be holding the ownership of something called as Universal Controller
Cluster.

Figure: Roles Definition for NSX Manager

So lets start with an analogy here which I thought would correlate to certain terms that you would be
often listening going forward.

In Medieval period, there used to be a Feudal System in the royal ownership.

At the top of the Feudalism Pyramid was the King

The King claimed ownership of the land
The King granted the land to important nobles these nobles then pledged their loyalty by
swearing to serve and protect the king
The king also granted land to the less powerful military men (the knights) who were called
vassals
The vassals also agreed to fight for the king in exchange for their land
The land was worked by the peasants or serfs. They belonged to the land and could not leave
without permission - the bottom of the Feudalism pyramid.
Lets consider the same analogy here. The King here is Primary NSX Manager and the important
nobles are the Universal Controller Cluster who would always advice the king about any war situation
as well as would send important messages to all parts of the kingdom through their channels. The
Knights here are the Secondary NSX Managers who would hold on to their land as in this case the
individual sites where these secondary NSX Managers are deployed. The user world agents on each
sites here could be actually related to messengers whom the nobles used to send information through
different channels. And last but not the least the peasants! Yes these are ESXi with Distributed Switch
with VXLAN and DLR capabilities which serves the data plane. Also, the complete Kingdom could
be referred to as the Universal Transport Zone.

Isnt that easier to understand? Well, thats the best I could come up with in order to complete the
explanation.

To summarize:
Primary NSX Manager manages and deploys the Universal Controller Cluster and would be
responsible for the creation of Universal components
Secondary NSX Manager would synchronize back with the primary NSX Manager to get
information on all universal objects deployed as well as it would be responsible for importing
the Universal Controller Cluster into its configuration which would be necessary to notify
user world agents where exactly they should be reporting.
Universal Transport Zone would consist of all the Clusters across multiple vCenters where
the need to stretch logical switches and DLR would be necessary.
Universal Logical Switch would extend across multiple vCenter Servers to provide a single
L2 domain.
Universal Distributed Logical router would assist in, but not necessary, egress optimization
for each local site.

When is egress optimization possible?

Now that we know what egress optimization is, you should also be aware on what circumstances you
would be able to use egress optimization. Considering the fact that weve been talking about multi-
site deployment scenarios, we need to ensure that the deployment model should be designed correctly
so as to effectively use this technique in your datacenter.

Deployment Model # 1: Multi-site deployment with 2 vCenter Servers with Primary-Secondary DLR
Control VMs deployed.

In this deployment model, there is a primary and a secondary DLR Control VM deployed in each site
connected to different set of transit networks and both learning routes in their own site.

Deployment Model # 2: Multi-site deployment with all the sites managed by a single vCenter Server
In this deployment model, we can think about a ROBO environment wherein all the sites are using
mostly Metro Clustering. All the clusters in this case are managed by a single vCenter Server and we
need to manipulate locale id to ensure that the traffic from each site is existing through their own
gateways.

How is egress optimization possible?

With the introduction of locale id, the routes learned within a single site through the DLR control VM
is sent to the Universal Controller Cluster with the locale id. The Controller Cluster then looks into
the locale id of the route and filters it accordingly so that the routes learnt via the DLR of site A is
only sent to the hosts available in site A.

Figure: Filtering of routes based on locale ID

NSX for vSphere Networking and Security Universal Objects

Since I have been talking about the Universal Objects through this chapters, let us have a closer look
at the various different terminologies.

Universal Synchronization Service This is the service that runs in the NSX Manager
as soon as the manager is designated as Primary NSX Manager. It remains stopped all the
time in the standalone role. This service is responsible of replicating the universal objects
across all secondary NSX Managers.
 Figure: NSX Universal Synchronization Service

Universal Transport Zone Universal Transport Zone is a collection of all the clusters
across all vCenter Servers in the environment. This defines the boundary or scale of the
Universal Logical Switches in the environment. The globe icon on the transport zone and
the Scope can help you identify whether its a local transport zone or a universal transport
zone. During the time of the release of NSX 6.2, the maximum number of Universal
Transport Zone is limited to 1.

Figure: Universal Transport Zone

Universal Segment ID Pool With the introduction of universal switches, the need to
keep VNI IDs consistent for all VXLAN networks across all the vCenter Servers gave
birth to Universal Segment ID Pool. This pool is responsible to provide VNIs to ULS and
also for LIFs on the DLR.

Figure: Filtering of routes based on locale ID

Universal Logical Switch This is the VXLAN VNI which would be stretched across
all the Clusters that are a part of Universal Transport Zone. In order to create a Universal
logical switch, you need to select a Universal Transport Zone in the create new logical
 switch wizard. Also, there should be Universal Segment Pool that should be available
when creating a Universal Logical Switch.

Universal Logical Router In order to provide east-west routing in multi-site

topologies, Universal Logical Router can now be created which would span across all the
Clusters prepared across vCenter Servers. This instance of DLR would run in kernel
across all the ESXi hosts which has been added to the NSX environment. The Control
VM can either be deployed in just one site with Active across all the other sites, or can be
deployed in primary-secondary model which would cater to the needs of egress
optimization.

Figure: Creating a Distributed Logical Router

Universal IP Sets/MAC sets/Security Groups/Service/Service Groups To use the

Universal Section of Firewall to publish rules across multiple vCenter Servers, you
require either Universal IP sets, MAC sets or Security Groups (which consists of IP sets
and MAC sets) in NSX environment. Please note that when considering a multiple site,
the rules cannot be written with vCenter containers as they are limited to vCenter
boundaries and cannot be published in a Universal section. However, the Universal
Firewall objects can be referenced in the local section of distributed firewall. You can
also create a Universal Service that can be referenced in the 5 tuple firewall rule which
would be explained in the How to apply a Universal Firewall rule later.
 Figure: Creating a new IP Set

Applying Firewall rules using Universal Section of Distributed

Firewall

Since Distributed firewall also supports applying rules across multiple vCenter Servers, please have
some key considerations in mind before applying rules using Universal Section.

You can create a Universal Section by creating a new section in Distributed Firewall and selecting the
checkbox Mark this section for Universal Synchronization as shown in the figure.

Figure: Creating a Universal Section in Distributed Firewall

Also, when applying rules, you can only use Universal objects in the Firewall rule also. Universal
Objects that can be used in the firewall rule are Universal IP Sets/MAC Sets, Universal Security
Groups, Universal Services and Universal Logical switches (in the applied to section for L3 based
rules).

The below picture represents the objects that you can use in the L2 or an L3 based rules.

Figure: Components of a Universal Firewall rule

Summary
In this last chapter, we have discussed about some of the most significant new features introduced in
VMware NSX for vSphere 6.2, mainly Egress Optimization, Cross vCenter vMotion, Universal
Distributed Firewall, Universal Logical Switches, Universal Controller Cluster, Universal, Distributed
Logical Router etc.