Sunteți pe pagina 1din 38

Isilon

OneFS
Version 8.1.0

External Network Connectivity Guide


Copyright © 2014-2018 Dell Inc. or its subsidiaries. All rights reserved.

Published July 2018

Dell believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS-IS.“ DELL MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND
WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. USE, COPYING, AND DISTRIBUTION OF ANY DELL SOFTWARE DESCRIBED
IN THIS PUBLICATION REQUIRES AN APPLICABLE SOFTWARE LICENSE.

Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be the property of their respective owners.
Published in the USA.

EMC Corporation
Hopkinton, Massachusetts 01748-9103
1-508-435-1000 In North America 1-866-464-7381
www.EMC.com

2 OneFS 8.1.0 External Network Connectivity Guide


CHAPTER 1
External network connectivity overview

This chapter includes the following topics.

l About this Guide.................................................................................................. 4


l Isilon networking.................................................................................................. 4

External network connectivity overview 3


External network connectivity overview

About this Guide


This guide provides information and best practices to help network and storage
architects and administrators plan, configure, monitor, and manage their EMC Isilon
network configurations.
Network configuration topics include:
l Network connectivity planning considerations
l Best practices for preventing data unavailable (DU) conditions
l Isilon cluster management
l Network topologies and IP routing
l Source-based routing in OneFS 8.x.x.x releases
l Planning, implementation, and best practices for EMC Isilon OneFS SmartConnect
and SmartConnect Advanced (licensed modules)
l Backup and disaster recovery planning
This guide refers to the following products and releases.
l OneFS 8.x.x.x
l Isilon InsightIQ 3.x (licensed module)
l Isilon InsightIQ 4.x (licensed module)
To get the most benefit from this guide, you should be familiar with Isilon networking
concepts and have basic IP networking knowledge.
You should also become familiar with Isilon documentation resources, including:
l EMC Community Network (ECN) info hubs, such as the OneFS 8.1.0
Documentation - Isilon Info Hub.
l EMC Online Support.
The OneFS release notes, available on EMC Online Support, contain important
information about resolved and known issues.

Isilon networking
This section reviews basic principles of Isilon networking.
An Isilon cluster requires two networks.
l Internal—A high-speed, low-latency network for communications and data
transfer among nodes in a cluster. The internal network can either be InfiniBand or
a backend Ethernet that is supported in OneFS 8.1.0.0 and later.
l External—An Ethernet connection for client users and applications that access
the cluster through an external, or front-end, network.
The Isilon OneFS operating system supports a single cluster on the internal network.
The Isilon OneFS operating system supports standard network communication
protocols IPv4 and IPv6, as well as client access protocols, including NFS, SMB,
HTTP, HDFS, FTP, and OpenStack Swift. Different Isilon nodes include several
external Ethernet connection options, providing flexibility for a wide variety of
network configurations.
The building blocks of Isilon networking are groupnets, subnets, and pools. Groupnets
are networking objects that allow for portions of the cluster to have different DNS

4 OneFS 8.1.0 External Network Connectivity Guide


External network connectivity overview

resolution settings. For more information about subnets and groupnets, refer to the
Subnet and groupnet overview section in this guide. Pools are logical partitions of
subnets. How you set up your external network subnets depends on your network
topology. A basic network topology where all client-node communication occurs
through a single gateway requires only a single external subnet. If clients connect
through multiple subnets or internal connections, you must configure multiple external
network subnets. By default, when you create a cluster, OneFS automatically creates
the initial subnet or pool for your network. It also creates a provisioning rule
automatically for the interface type that you selected.

Note

If you created your cluster using pre-Generation 6 hardware, and have set a
provisioning rule for ext-1, you must create a second provisioning rule for mgmt-1
before adding your Generation 6 nodes in case you want the nodes to get
automatically added.

In general, keeping the network configuration simple provides the best results with the
lowest amount of administrative overhead. OneFS offers network provisioning rules to
automate the configuration of additional nodes as clusters grow.
The following licensed Isilon modules provide advanced features to help you to
manage and monitor your network.
l The Isilon SmartConnect Advanced module provides IP address allocation
management and connection balancing.
l The Isilon InsightIQ virtual appliance provides cluster performance monitoring and
analysis to help you optimize storage resources and forecast capacity. InsightIQ
provides a view of network throughput and rates at the moment and over time.
For more information about Isilon networking, monitoring, and management, see the
Isilon OneFS Web Administration Guide, Isilon OneFS CLI Administration Guide, and Isilon
InsightIQ User Guide for your version of Isilon OneFS and Isilon InsightIQ.

Isilon networking 5
External network connectivity overview

6 OneFS 8.1.0 External Network Connectivity Guide


CHAPTER 2
Network planning and design considerations

This chapter includes the following topics.

l General cluster best practices..............................................................................8


l Link aggregation.................................................................................................. 9
l Managing Isilon clusters......................................................................................10
l Calculating IP address requirements................................................................... 11
l Ports used by OneFS.......................................................................................... 16
l Migration strategies for network consolidation...................................................16

Network planning and design considerations 7


Network planning and design considerations

General cluster best practices


This section describes best practices for designing your cluster network.
l OneFS balances a cluster’s connections among all the nodes that service external
(front-end) connections. We recommend that you regularly monitor cluster
connections with InsightIQ. If the number of connections frequently approaches
the maximum number of connections that the node can support, consider adding
another node.
l SmartConnect balances incoming network connections across all the configured
network interfaces in a SmartConnect zone or pool with one of several load-
balancing techniques. The most common load-balancing technique is round robin.
Round robin is effective for most workflows, but it is important to understand
whether your front-end connections are being evenly distributed, either in count
or bandwidth. Monitor front-end connection distribution with InsightIQ.
The following table can help you to choose a connection balancing policy that’s
best for your environment.

Workload characteristics

Load Not sure Heavy Lots of NFS Lots of NFS


balancing constant and SMB short-lived automount
policy activity on long-lived connection and/or
a few connection s such as UNIX paths
clients s HTTP and
FTP
Round robin X X

CPU X X X
utilization

Connection X X
count

Network X X X
throughput

l Observe SMB connection limits. For more information about limitations such as
the recommended maximum number of SMB connections, see the EMC Isilon
OneFS technical specifications guide for your version of OneFS.
l Regularly monitor cluster usage with InsightIQ, the web administration interface,
or the command-line interface (CLI). If available disk space approaches 20 to 15
percent (disk space usage reaches 80 percent to 85 percent of capacity), consider
adding a node. When disk space usage reaches 90 percent, we strongly
recommend adding additional capacity. This helps to ensure that there is sufficient
usable space for successful OneFS failover and to achieve performance goals for
the cluster. When disk space usage reaches 95 percent of capacity, the risk for file
system anomalies increases. For more information, see Best Practices Guide for
Maintaining Enough Free Space on Isilon Clusters and Pools.
l Many cluster configuration settings are global and have cluster-wide effects. If
you consider changing cluster-wide configuration settings, be sure that you fully
understand the global settings and their implications. For information about global
cluster configuration settings, see the Isilon OneFS Web Administration Guide or

8 OneFS 8.1.0 External Network Connectivity Guide


Network planning and design considerations

Isilon OneFS CLI Administration Guide for your version of OneFS. You can deploy an
Isilon OneFS Simulator to test the changes before you make changes that affect
production.
l Confirm that remote support functions work correctly through EMC Secure
Remote Support (ESRS), SupportIQ, or internal email/SNMP notifications.
l Always use at least the default data protection scheme. Inadequate data
protection, or none at all, can cause transient failures for the duration of a job. As
a result, data might be lost or scrambled. Recheck the cluster’s data protection
levels as the cluster grows.
l Calculate the recommended protection level based on cluster configuration. The
suggested protection level strikes the best balance between data protection and
storage efficiency.
l Recommend to your client system administrators that they turn off client DNS
caching wherever possible. All the currently shipping versions of Windows DNS
servers will automatically change the DNS caching value to one second. To handle
client requests properly, SmartConnect requires that clients use the latest DNS
entries. If clients cache SmartConnect DNS information, they might connect to
incorrect SmartConnect zone names. In this situation, SmartConnect might not
appear to be functioning properly.
l Use SmartConnect Advanced to manage multipath I/O.
l If you plan to use VLANs for improved redundancy and availability, set up Link
Aggregation Control Protocol (LACP) and create VLANs on top of the aggregated
interface.
l Use consistent frame sizes across your network. For example, if you enable jumbo
frames, enable them uniformly across your network infrastructure to prevent
packet fragmentation or dropped data when you try to write through a smaller
switch.
l For information about configuring LACP between a Cisco switch and an Isilon
cluster, see article 000304472 on the EMC Online Support site.

Link aggregation
This section describes the advantages and disadvantages of link aggregation.
Link aggregation combines multiple physical connections (ports) in parallel to provide
failover functionality in case one of the aggregated ports fails. Link aggregation does
not increase bandwidth beyond a single link for a single client. However, it provides
bandwidth that is equivalent to all the links when multiple clients are involved.
For more information about link aggregation, see IEEE 802.3ad Link Bundling.

Note

With dynamic SmartConnect zones for NFSv3 clients, link aggregation does not
provide any discernible improvement in functionality or in reliability.

Link aggregation advantages


This section describes the advantages of link aggregation.
l Stateful protocols such as NFSv4 and SMB can benefit from link aggregation as a
failover mechanism between the two interfaces of any single node.

Link aggregation 9
Network planning and design considerations

l Link aggregation provides switch redundancy on the Isilon node.


l Link aggregation can enable transparent failover of:
n Individual SFP+ Optics in an Isilon node
n Single network cables in a bonded pair
n SFP+ Optics on the switch side of the connection
l If the two connections in the link aggregate are spread across two physical
switches using a technology such as Cisco VPC, maintenance or reboots of one
switch will not affect client I/O.

Link aggregation considerations


This section describes considerations for using link aggregation.
l Link aggregation is only per node, not across nodes. Because OneFS is a clustered
file system, each node of the cluster is an independent unit with its own operating
system. Link aggregation across more than one node is not available or supported.
l Link aggregation does not increase overall throughput for a single client. This is
because load-balancing is performed based on either a source/target IP address
hash or a source/target MAC address hash, depending on the configuration of the
port channel configured on the switch. For a single client, you can use a bandwidth
that is greater than that of a single link, through the SMBv3 protocol.

Managing Isilon clusters


This section describes the methods you can use to manage Isilon clusters.
You can manage Isilon clusters using the following methods.
l Command line interface (CLI) using SSH
l Web interface (GUI) using HTTP and HTTPS
l Platform API (PAPI) using HTTP and HTTPS

Managing in-band networks


All OneFS cluster management is performed in-band.
You may be familiar with scale-up NAS platforms that use separate network interfaces
for out-of-band management and configuration. In contrast, Isilon handles all network
management in-band. The following table summarizes common questions and answers
about in-band network management.

Question Answer
Does connecting extra network interfaces for No. Disks are the limiting factor, not the
management increase the total network network interface cards (NICs).
throughput to the cluster?

Does using dedicated 1 Gbe network No. Although the motherboards on Isilon
interfaces for management give out-of-band nodes have IPMI interfaces, their use is not
access to the cluster? supported at this time. Therefore, the 1 GbE
interfaces do not provide that functionality.

Does connecting extra 1 GbE network No. Management traffic is only a tiny fraction
interfaces for management minimize the of the traffic that users generate for normal
impact of management traffic on user traffic? data access.

10 OneFS 8.1.0 External Network Connectivity Guide


Network planning and design considerations

Question Answer
Security: Can management on the 10 GbE No.
NICs be shut off so that it acts like a VNX
datamover, with the 1 GbE acting as the
control station?

Out-of-band node management


OneFS supports using serial ports as out-of-band management interfaces.
For example, serial ports can provide command-line interface access for on-site
service staff to perform maintenance or installation operations. Although you can
connect serial console servers to all of your nodes to extend this capability over your
internal network, be aware of possible security risks inherent in the serial console
server that you purchase.

Sending replication traffic on a dedicated WAN link


Some network topologies require sending all replication traffic for storage devices
across a dedicated WAN circuit that is separate from user traffic.
This is a common scenario with Celerra/VNX Replicator, DMX/VMAX SRDF, and
similar platforms. If you have this requirement for Isilon SyncIQ traffic, we recommend
that you use a different subnet on the Isilon cluster for replication traffic that is
separate from the subnet used for user data access.
Whether source-based routing (SBR) is enabled or not, we recommend that you set
up the replication subnet with no configured gateway on both the source and the
target clusters for SyncIQ. On both clusters, configure a static route for the other side
and designate the gateway of the current subnet as the next-hop IP address.

Calculating IP address requirements


This section provides guidance for determining the number of IP addresses for a new
cluster implementation.
Consider the following points when determining your IP address requirements.
l Calculate the number of IPs that are needed based on how big the cluster might
be in the future, not on how big the cluster is in the beginning.
l Never share a subnet with other application servers. If you eventually need more
IP addresses and the range is full, re-addressing an entire cluster and then moving
it into a new VLAN or moving servers to another subnet, can be very disruptive.
With proper planning, you can prevent these complications.
l Static IP pools require one IP address for each logical interface that will be in the
pool.
l For optimal load-balancing if a node fails, dynamic IP pools require N*(N-1) IP
addresses, where N is the number of nodes that will participate in the pool. For
larger clusters, you can use a smaller number of IP addresses. The imbalance
created by using fewer IP addresses is negligible.

Note

The recommended largest allocation per cluster is a /23 subnet, or 510 usable
addresses.

Out-of-band node management 11


Network planning and design considerations

Suppose that you have a cluster with three IP pools. The cluster will be used for
NFSv3 and SMB client access and will have a dedicated IP pool for backup and
replication. Use the following factors to calculate the number of IP addresses that the
cluster requires (assuming 2x10 GbE per node and no LACP).
l 1 for the SmartConnect service IP (SSIP)
l 2*N for static pool0 (with two logical interfaces per node) (for stateful
connections [SMB])
l N*(N-1) for pool1 (for stateless connections [NFS])
l 2*N for pool2 (with two logical interfaces per node (for backup and replication)
The extrapolated formula is:

1+2N+2N+N(N-1)=x

The simplified formula is:

1+4N+N²-N=x

or:

N²+3N+1=x

Where:
l N=the number of nodes
l x=the total number of IPs needed for the current cluster size
Given a four-node cluster, this works out as:

4²+3*4+1=x
16+12+1=x
x=29

Use this formula as a guide, keeping in mind the recommended largest allocation per
cluster is a /23 subnet, or 510 usable addresses.
The following table contains results for a partial extrapolation of the formula, starting
with three nodes. The table shows how the number of IP addresses needed for a
cluster grows as the number of nodes increases.

# of # of logical SmartCon Static Dynamic Static Total # of


nodes connections nect SSIP pool 0 pool 1 pool 2 IPs
(N) per node needed
(x)
3 2 1 6 6 6 19

4 2 1 8 12 8 29

5 2 1 10 20 10 41

6 2 1 12 30 12 57

7 2 1 14 42 14 71

.....

12 OneFS 8.1.0 External Network Connectivity Guide


Network planning and design considerations

# of # of logical SmartCon Static Dynamic Static Total # of


nodes connections nect SSIP pool 0 pool 1 pool 2 IPs
(N) per node needed
(x)
20 2 1 40 380 40 461

21 2 1 42 420 42 504

The number of IPs expands exponentially. At a certain point, the benefits of having
large numbers of IP addresses with dynamic SmartConnect zones have diminishing
returns. There are very few cases that require an IP allocation larger than the
recommended /23 subnet, or 510 usable addresses.
From a load-balancing perspective, it is ideal (though optional) that all the interfaces
for dynamic pools have the same number of IP addresses whenever possible.

Using multiple IP addresses with Dynamic SmartConnect zones


The examples in this section show how using multiple IP addresses with dynamic
SmartConnect zones helps to handle failover events.
About the sample scenarios
Dynamic SmartConnect zones require many IP addresses to handle failover behavior.
The formula used to calculate the number of IP addresses your network will need is
N*(N-1). The following scenarios explain how the formula works and why.
Each scenario is based on a three-node cluster with one network connection per node
and one dynamic SmartConnect zone.

Example 1 Three IP addresses

In this scenario, the three-node cluster with one network connection per node and one
dynamic SmartConnect zone has only three IP addresses. After a few days, 100 clients
are actively connected to each node over NFS using a round robin connection policy.
One IP address will be assigned to each node, as shown in the following illustration.

Most NFSv3 mountd clients do an nslookup only the first time that they perform a
mount, so they never do another nslookup to check for an updated IP address. If the
IP address changes, the NFSv3 clients have a stale mount and retain that IP address.

Example 2 Failure event: one node fails

Using multiple IP addresses with Dynamic SmartConnect zones 13


Network planning and design considerations

Example 2 Failure event: one node fails

Suppose that one of the nodes fails, as shown in the following illustration.

A SmartConnect Zone with a dynamic allocation strategy immediately hot-moves the


one IP address on the failed node to one of the other two nodes in the cluster, and
sends out a number of gratuitous address resolution protocol (ARP) requests to the
connected switch so that client I/O continues uninterrupted.

However, even though all three IP addresses are still online, two of them—and 200
clients—are now connected to one node. That’s because SmartConnect can fail only
one IP to one other place, and one IP address and 100 clients are already connected to
the other node. This means that one node failing has just doubled the load on one of
the two remaining nodes, while doing nothing to the third node. The result is that
client performance declines, but not equally. The goal of any scale-out NAS solution
must be consistency. To double the I/O on one node and not on another is
inconsistent.

Example 3 Six IP addresses

In this scenario, the three-node cluster has the same configuration but with two IP
addresses per node instead of one IP address per node. The cluster has the same
number of clients.

14 OneFS 8.1.0 External Network Connectivity Guide


Network planning and design considerations

Example 3 Six IP addresses (continued)

This cluster follows the rule of N*(N-1), in this case, 3*(3-1) = 6, or two IPs per node.
When the same failure event occurs, the two IP addresses are spread over all the
other nodes in that SmartConnect zone. This means that each remaining node has 150
clients and three IP addresses. So although performance might degrade to some
degree, that degradation will not be as drastic as the failure in the first scenario. The
experience is consistent for all users, as shown in the following illustration.

Accessing a SmartConnect Zone using both IPv4 and IPv6 protocols


Follow the steps in this section to access a SmartConnect zone using both IPV4 and
IPV6 protocols.
1. Create two subnet definitions as shown in the following examples:
l IPv4 subnet: example subnetv4
l IPv6 subnet: example subnetv6
2. Configure the SmartConnect service address in the IPv4 subnet only. For example,
for the 1.2.3.4 SmartConnect service address, the configuration appears as
follows:
l subnetv4: -sc-service-addr: 1.2.3.4
l subnetv6: <will not have -sc-service-addr>
3. In both the IPv4 and IPv6 address pools, configure the SmartConnect DNS zone
to be the DNS name of the cluster and the SmartConnect subnet to be the IPV4
subnet as shown in the following example:
subnetv4.poolv4:
l -sc-dns-zone: cluster.com
l -sc-subnet: subnetv4
subnetv6.poolv6:
l -sc-dns-zone: cluster.com
l -sc-subnet: subnetv4
4. In the DNS infrastructure, delegate cluster.com to 1.2.3.4.
With this setup, if a client using the IPv6 protocol issues a DNS request for an IPv6
address against cluster.com, it is directed to send that request to 1.2.3.4 and receives

Using multiple IP addresses with Dynamic SmartConnect zones 15


Network planning and design considerations

an IPv6 address. A client using the IPv4 protocol requests an IPv4 address and
receives it from 1.2.3.4.

Ports used by OneFS


OneFS uses a number of TCP and UDP ports.
For the list of ports, see the OneFS Security Configuration Guide for your version of
OneFS.

Migration strategies for network consolidation


This section describes strategies for consolidating legacy file servers to Isilon.
A common requirement for consolidating legacy file servers to Isilon is to keep the
names used by clients to minimize the impact on users and applications. There are two
valid approaches, one of which is preferred.
SmartConnect zone aliases and name server (NS) records/delegations (preferred)
This approach requires you to create Service Principal Name (SPN) records in
Active Directory or in MIT Kerberos for the SmartConnect zone names, as a
component of the cluster’s machine account. To create the SPN records, use the
isi auth command after you add the zone alias, similar to the following:

isi auth ads spn fix <provider-name>

DNS CNAMES (not preferred)


The DNS CNAMES approach is the most common method of redirecting clients
from an old platform to a new one. However, DNS CNAMES are nearly impossible
to track. It is not possible to examine an Isilon cluster and discover which
CNAMES point to a given SmartConnect zone name. Common DNS tools,
including dig and nslookup, cannot do this.
With the DNS CNAMES approach, failover to a disaster recovery cluster requires
updating a potentially large number of potentially unknown CNAMES to point to a
disaster recovery SmartConnect zone name.

We strongly advise against keeping the IP addresses of legacy platforms when you
migrate your network to Isilon. SmartConnect functions properly when you use only
DNS names for client data access.

16 OneFS 8.1.0 External Network Connectivity Guide


CHAPTER 3
Designing for specialized workloads

This chapter includes the following topics.

l Designing for specialized workloads overview.....................................................18


l Media and entertainment best practices.............................................................18

Designing for specialized workloads 17


Designing for specialized workloads

Designing for specialized workloads overview


This section presents considerations and best practices for specialized workloads such
as media and entertainment.
The recommended practices in this section help to ensure optimal functioning of your
Isilon networks.

Media and entertainment best practices


This section describes best network design practices for media and entertainment
workloads.
l In media and entertainment workflows where low-latency networking is required
for real-time video ingest or playback, use a dedicated media network for your
workflow and dual-home all your clients to keep the switch caches clear of
extraneous packets. We recommend low-latency switches with a minimum of 1 MB
of cache per port for the dedicated media LAN.
l In workflows such as media content creation where there is no automation
component, store heavyweight, multi-gigabyte media files on the cluster, and
store metadata files locally. Users rarely make significant changes to original media
files, but when they do, the best practice is to save changes back to the cluster.
For example, users performing video editing should work on a local copy of the
media file, then save the copy back to the cluster.
l In media and entertainment environments, use a Media Access Management
(MAM) system, if possible, to handle version control and collaboration among
multiple users.

18 OneFS 8.1.0 External Network Connectivity Guide


CHAPTER 4
Cluster stability and data integrity

This chapter includes the following topics.

l Data unavailable causes and preventive actions overview.................................. 20


l Causes and preventive actions for data unavailable conditions.......................... 20
l Planning for backup and disaster recovery.........................................................20

Cluster stability and data integrity 19


Cluster stability and data integrity

Data unavailable causes and preventive actions overview


This section presents best practices to help ensure cluster stability, data integrity, and
optimal network performance.

Information includes:
l Common causes of data unavailable (DU) conditions and the actions you can take
to prevent them.
l Best practices for backup and disaster recovery planning with the OneFS
SmartConnect Advanced module.

Causes and preventive actions for data unavailable


conditions
The table in this section lists common causes of data unavailable (DU) conditions and
the actions that you can take to help prevent them.

DU cause Preventive action


Overbooked nodes Design redundant network paths.

Protocol holes in NFSv3 and unconfigured Change the SmartConnect IP rebalance delay
SmartConnect IP rebalance delay to a value other than zero.

Setting up an address only on a part of the If your SmartConnect network spans multiple
SmartConnect network networks, set up a separate address for each
SmartConnect network, including the
spanning network.

Inadequate free space Monitor cluster disk space usage with


InsightIQ, the web administration interface, or
the CLI.

Using inadequate data protection schemes or Always use at least the default data
running upgraded clusters at a protection protection scheme.
level that is lower than the recommended data
protection scheme level

Some of the other common causes that could result in data unavailable (DU)
conditions are as follows:
l Interfaces are down or a specific gateway cannot be reached.
l Protocols (SMB, NFS, HDFS, or Swift) are down.
l A node is down or suspended.

Planning for backup and disaster recovery


This section describes best practices for backup and disaster recovery planning with
the OneFS SmartConnect Advanced module. SmartConnect Advanced requires an
active license.
We recommend that you create a dedicated static SmartConnect zone for SyncIQ
backup and replication jobs on clusters that have SmartConnect Advanced installed.
Like any static SmartConnect zone, the dedicated backup and replication zone

20 OneFS 8.1.0 External Network Connectivity Guide


Cluster stability and data integrity

requires one IP address for each active logical interface. For example, if you have two
active physical interfaces, 10gige-1 and 10gige-2, you need two IP addresses. But if
you configure a link aggregation with LACP, 10gige-agg-1, you need only one IP
address. You can source-restrict all SyncIQ jobs to use your dedicated static
SmartConnect zone.
For example, consider a 10-node cluster for which the requirement is to have all
SyncIQ traffic run across five of the nodes. You can restrict the SmartConnect subnet
and pool that the job can run against. Because you can limit which nodes participate in
that pool, you can restrict on the source which nodes will be used for SyncIQ. For
example, you can use archive nodes as your SyncIQ source and keep the faster nodes
for client I/O.
By restricting SyncIQ backup and replication jobs to a dedicated static SmartConnect
Zone, you can easily redirect backup and replication traffic from certain nodes to
reduce the impact of SyncIQ jobs on user or client I/O. You can redirect traffic
without reconfiguring or modifying the interfaces participating in the SmartConnect
zone.
For example, consider a data ingest cluster for a sports television network. The cluster
must ingest large amounts of data recorded in 4K video format. The data must be
active immediately, and the cluster must store the data for a long time. The sports
television network administrators want to keep data ingestion and data archiving
separate, to maximize performance. The sports television network purchased two
types of nodes: H500s for ingesting data, and A200s for the long-term archive.
Because the data set is so large, SyncIQ jobs to replicate the data to the disaster
recovery site have a lot of work to do on each pass. The front-end interfaces are
saturated on the H500 nodes for either ingesting data or performing immediate data
retrieval. The CPUs of those nodes must not be affected by the SyncIQ jobs. By using
a separate static SmartConnect pool, the network administrators can force all SyncIQ
traffic to leave only the A200 nodes and provide maximum throughput on the H500
nodes.

Planning for backup and disaster recovery 21


Cluster stability and data integrity

22 OneFS 8.1.0 External Network Connectivity Guide


CHAPTER 5
SmartConnect considerations

This chapter includes the following topics.

l SmartConnect usage considerations.................................................................. 24


l DNS delegation best practices........................................................................... 25
l Using SmartConnect in isolated network environments..................................... 26
l Considerations for SmartConnect with multiple node pools............................... 26
l Protocols and network allocation policies...........................................................27
l SmartConnect service IPs overview................................................................... 27

SmartConnect considerations 23
SmartConnect considerations

SmartConnect usage considerations


SmartConnect acts as a DNS delegation server to return IP addresses for
SmartConnect zones, generally for load-balancing connections to the cluster.
IP routing principles are the same with or without SmartConnect.
DNS lookups of SmartConnect zone names involve four separate DNS operations as
shown in the following illustration:

1. A client makes a DNS request for example.domain.com by sending a DNS request


packet to the site DNS server.
2. The site DNS server has a delegation record for example.domain.com and sends a
DNS request to the defined nameserver address in the delegation record, the
SmartConnect service (SmartConnect Service IP Address).
3. The cluster node hosting the SmartConnect Service IP (SSIP) for this zone
receives the request, calculates the IP address to assign based on the configured
connection policy for the pool in question (such as round robin), and sends a DNS
response packet to the site DNS server.
4. The site DNS server sends the response back to the client.
Keep in mind the following considerations.
l If you have firewalls, make sure that the appropriate ports are open. For example,
if you open UDP port 53, make sure that you also open TCP port 53.
l The client never sends a DNS request directly to the cluster. The site DNS servers
handle DNS requests from clients and route the requests appropriately.
l In order to successfully distribute IP addresses, the OneFS SmartConnect DNS
delegation server answers DNS queries with a time-to-live (TTL) of 0 so that the
answer is not cached. All the currently shipping versions of Windows DNS servers
will fix the value to one second. If you have many clients requesting an address
within the same second, this will cause all of them to receive the same address. If
you encounter this problem, you may need to use a different DNS server, such as
bind.
l Certain clients perform DNS caching and might not connect to the node with the
lowest load if they make multiple connections within the lifetime of the cached

24 OneFS 8.1.0 External Network Connectivity Guide


SmartConnect considerations

address. For example, this issue occurs in Mac OS X for certain client
configurations.
l The site DNS servers must be able to communicate with the node that is currently
hosting the SmartConnect service. This is the node with the lowest DevID with an
active interface in the subnet that contains the SSIP address. This behavior
cannot be modified.
l The stats used by the connection count, CPU usage, and throughput policies are
updated approximately every 5 seconds.
l Site DNS servers might not exist in the regular local subnets, or in any of the
subnets that clients occupy. To enable the SmartConnect lookup process, make
sure that the DNS servers use a consistent route to the cluster and back. If the
site DNS server sends a lookup request that arrives through one local subnet on
the cluster, but the configured cluster routing causes the response to be sent
through a different subnet, it’s likely that the packet will be dropped and the
lookup will fail. The solutions and considerations for SmartConnect are similar to
the client scenarios. Additionally, the DNS server might benefit from a static route
to the subnet that contains the SSIP address(es).
l SmartConnect makes it possible for different nodes to have different default
routes, but this is fundamentally determined by connectivity. SmartConnect
enables you to define multiple gateways: 1 gateway per subnet. Each gateway is
assigned a priority when it is defined. On any node, SmartConnect attempts to use
the highest priority gateway. The highest priority gateway is usually the one that
has the lowest number and has an available functioning interface in a subnet that
contains the gateway address.

DNS delegation best practices


This section describes best practices for DNS delegation for Isilon clusters.
l Configure your delegation to point to a hostname (A/AAAA record) and not to an
IP address.
The SmartConnect service IP on an Isilon cluster must be created in DNS as an
address (A) record, also called a host entry. An address record maps a URL such
as www.emc.com to its corresponding IP address. Delegating to an address record
means that if you ever need to failover the entire cluster, you can do so by
changing just one DNS record. All other name server delegations can be left alone.
In many enterprises, it is easier to have an address record updated than to update
a name server record, because of the perceived complexity of the process.
l Use one name server record for each SmartConnect zone name or alias.
We recommend creating one delegation for each SmartConnect zone name or for
each SmartConnect zone alias on a cluster. This method permits failover of only a
portion of the cluster's workflow—one SmartConnect zone—without affecting
any other zones. This method is useful for scenarios such as testing disaster
recovery failover and moving workflows between data centers.
We do not recommend creating a single delegation for each cluster and then creating
the SmartConnect zones as sub records of that delegation. Although using this
method would enable Isilon administrators to change, create, or modify their
SmartConnect zones and zone names as needed without involving a DNS team, this
method causes failover operations to involve the entire cluster and affects the entire
workflow, not just the affected SmartConnect zone.

DNS delegation best practices 25


SmartConnect considerations

Using SmartConnect in isolated network environments


To use SmartConnect in an isolated network environment where no DNS
infrastructure is available (such as a DMZ), configure your client systems to use the
SmartConnect service IP address as the primary DNS server.
SmartConnect is, effectively, a limited implementation of a custom DNS server: it
answers only for the SmartConnect zone names or aliases configured on it.
Configuring your client systems to use the SmartConnect service IP address as the
primary DNS server helps to ensure that:
l Requests to connect to Isilon clusters with SmartConnect zone names will
succeed.
l The isolated network benefits from SmartConnect features, such as load-
balancing and rerouting traffic to prevent unavailable nodes, will work as expected
in a normal, non-isolated deployment.
The following commands show how you can simulate and test a configuration that
uses the SmartConnect service IP address as the primary DNS server.

C:\>nslookup
Default Server: 10.123.17.60
Address: 10.123.17.60
> isi01-s0.domain.com
Server: [10.123.17.60]
Address: 10.123.17.60
Name: isi01-s0.domain.com
Address: 10.123.17.64
> isi01-s0.domain.com
Server: [10.123.17.60]
Address: 10.123.17.60
Name: isi01-s0.domain.com
Address: 10.123.17.63

Considerations for SmartConnect with multiple node pools


This section presents performance considerations for using SmartConnect with
multiple nodes or node types.
From a client or application perspective, the goals for all scale-out NAS deployments
are consistency and availability. Consistency, in this context, implies that every time a
client connects, whether that client is an application server or a user opening their
home directory, they get an equal level of performance. A number of different Isilon
node types with different performance profiles are available for use.
Performance in network-attached storage is determined by many factors. In an Isilon
cluster, key components are the front-end performance, which consists of the
network card, CPU, and memory in the node that is serving the relevant protocol
(such as SMB or NFS), and the back-end performance, which, in this case, is the disk
tier or pool where the data resides. In the context of SmartConnect configuration,
nodes with different performance characteristics should not be placed in the same
pool. For example, H600 and A200 nodes should not be combined and placed in the
same pool. Regardless of the storage tier, the protocol performance will be noticeably
different.

26 OneFS 8.1.0 External Network Connectivity Guide


SmartConnect considerations

Protocols and network allocation policies


This section describes Isilon and client access protocols.
Client access protocols on Isilon can be divided into the following categories.
l Stateful. The client/server relationship usually has a session state for each open
file. Failing over IP addresses to other nodes for these types of workflows means
that the client assumes that the session state information was carried over.
Session state information for each file is not shared among Isilon cluster nodes.
l Stateless. Stateless protocols are generally accepting of failover without session
state information being maintained (except for locks).
SynqIQ can add nodes while a sync job is running.
Following are the recommended IP allocation strategies for SmartConnect Advanced
for each supported protocol.

Protocol Protocol category Recommended allocation


policy
NFSv3 Stateless Dynamic

NFSv4 Stateful Static

Note

Though NFSv4 is supported


with dynamic IPs, there could
be a potential performance
impact.

SMBv1 Stateful Static

SMBv2/2.1 Stateful Static

SMBv3 Stateful Static

FTP/FTPs Stateful Static

SFTP/SSH Stateful Static

HDFS Stateful (But protocol is Static


tolerant of failures)

HTTP/HTTPS/RAN Stateful Static

SyncIQ Stateful Static

SmartConnect service IPs overview


Each cluster needs at least one SmartConnect service IP (SSIP), as long as there are
no firewalls between the infrastructure DNS servers and the SSIP that block TCP and
UDP port 53.
It doesn’t matter how many domains or subnets the cluster is joined to or participates
in. SmartConnect is essentially a very selective DNS server that answers only for the
SmartConnect zone names and SmartConnect zone aliases that are configured on it.

Protocols and network allocation policies 27


SmartConnect considerations

28 OneFS 8.1.0 External Network Connectivity Guide


CHAPTER 6
Routing in OneFS

This chapter includes the following topics.

l OneFS routing overview.....................................................................................30


l Subnet and groupnet overview.......................................................................... 30
l Multiple cluster interfaces in the same subnet................................................... 30
l Source-Based Routing (SBR) overview.............................................................. 31
l Destination-based routing overview................................................................... 33
l Advantages and disadvantages of source-based routing.................................... 35
l NIC affinity overview......................................................................................... 35
l How routing tables, SBR and NIC affinity interact............................................. 36

Routing in OneFS 29
Routing in OneFS

OneFS routing overview


Routing is the process of determining how to get IP packets from a source to a
destination. When responding to client computers, OneFS IP routing attempts to find
a matching route, starting with the most specific match. If no specific match is found,
IP routing uses the default route (if there is one). There is only one active default
outbound route on any particular node at any one time.

Subnet and groupnet overview


Subnet refers to a contiguous range of IP addresses.
A subnet's contiguous range of IP addresses is usually denoted in classless
interdomain routing (CIDR) notation (address/bits, such as 10.0.0.0/8 or
192.168.0.0/24). An interface on a OneFS node can be a plain network interface card
(NIC) port (em0/ext-1, cxgb1/ext-3), or a virtual LAN (VLAN) port, or an aggregate/
bond, or a combination of these. A OneFS cluster node can be directly connected to
one or more IP subnets, and can use routers to communicate with many others. Local
refers to directly-connected subnets. Foreign refers to all other subnets.
A groupnet is a new networking object that represents a configuration object for DNS
options. Different groupnets are used to allow portions of the cluster to have different
networking properties for the DNS namespace resolution. You should create a
groupnet for each individual DNS namespace that you want to use. For example, if you
have a group of clients that would like to use AD provider AD1.FOO.COM and DNS
servers 1.2.3.4 and 1.2.3.5, and another group of clients that would like to use AD
provider AD2.FOO.COM and DNS servers 5.6.7.8 and 5.6.7.9, you must set up the
groupnet configuration as follows:

isi network groupnet create groupnet1 --dns-servers=1.2.3.4,1.2.3.5


isi network groupnet create groupnet2 --dns-servers=5.6.7.8,5.6.7.9
isi auth ads create AD1.FOO.COM --groupnet=groupnet1
isi auth ads create AD2.FOO.COM --groupnet=groupnet2
isi zone zones create zone1 /ifs/data/zone1 --groupnet=groupnet1 --
auth-providers=AD1.FOO.COM
isi zone zones create zone2 /ifs/data/zone2 --groupnet=groupnet2 --
auth-providers=AD2.FOO.COM
isi network subnet create groupnet1.subnet1 <additional subnet
arguments>
isi network subnet create groupnet2.subnet2 <additional subnet
arguments>
isi network pool create groupnet1.subnet1.pool1 --access-zone=zone1
<additional pool arguments>
isi network pool create groupnet2.subnet2.pool2 --access-zone=zone2
<additional pool arguments>

Based on the above configuration, the cluster should query the IP address of
AD1.FOO.COM from DNS server 1.2.3.4, and should query the IP address of
AD2.FOO.COM from DNS server 5.6.7.8.

Multiple cluster interfaces in the same subnet


Legacy routing configuration mechanisms allow you to use multiple interfaces in the
same subnet. We recommend that you do not use these mechanisms with clusters

30 OneFS 8.1.0 External Network Connectivity Guide


Routing in OneFS

created using OneFS 8.0.0.0 and later releases. Instead, we recommend that you use
link aggregation with the Link Aggregation Control Protocol (LACP) for these clusters.
Historically, many clusters were configured with multiple interfaces in the same
subnet, with IP pools split among the interfaces. Because of the way IP routing works,
while incoming traffic tended to be balanced, outgoing traffic used only one interface:
the first in the routing table. OneFS sysctl .inet.ip.choose_ifa_by_ipsrc
was added to address this issue. If sysctl .inet.ip.choose_ifa_by_ipsrc is
enabled (set to 1), OneFS IP routing determines the interface to send from based on
the source address in the IP packet.

Note

This configuration does not affect routing, which is determined by the destination IP
address.

Source-Based Routing (SBR) overview


Source-Based Routing (SBR) uses the source IP address of packets sent from the
cluster to determine the next hop (router) for IP packets that are destined for non-
local addresses.
There must be an existing gateway definition for each local subnet on the cluster.
l SBR inserts routing rules in the cluster that cause the cluster to send packets
destined for a non-local IP address to the gateway on the subnet that contains the
source IP that the packet is being sent from.
l SBR supports IPv4.
l SBR addresses a scenario that often occurs on Isilon systems. Suppose that your
cluster has 1 GbE and 10 GbE interfaces. You dedicate the 1 GbE interfaces for
management purposes and the 10 GbE interfaces for handling all data access.
Suppose that client C1 must send data to server A1. They must use the high speed
10 GbE interfaces, and they must connect to a different Isilon subnet to do so. The
following diagram illustrates the packet flow for this scenario.

1. Client C1 must send a packet to server A1 at IP address 10.3.1.90.


a. Client C1 determines that the destination IP address is not local and it does
not have a static route defined for that address.
b. Client C1 sends the packet to its default gateway, Router C, for further
processing.
2. Router C receives the packet from Client C1.
a. Router C examines the packet’s destination IP address and determines that
it has a route to the destination through the router at 10.1.1.1, Router A.

Source-Based Routing (SBR) overview 31


Routing in OneFS

b. Router C sends the packet to Router A through its external interface.


The packet can pass through several routers as it travels through the network.
3. Router A receives the packet on its external interface.
a. Router A determines that it has a direct connection to the destination IP
address, 10.3.1.90.
b. Router A sends the packet directly to 10.3.1.90 using its internal interface
on the 10GbE switch.
4. Server A must send a response packet to client C1.
a. Server A determines that the destination IP address, 10.2.1.50, is not local
and that it does not have a static route defined for that address.
b. Server A determines which gateway to send the response packet to based
on its default gateways’ priority numbers. Gateways with lower priority
numbers have precedence over gateways with higher priority numbers.
Server A has two default gateways: 10.1.1.1 with a priority of 0 and 10.3.1.1
with a priority of 10.
c. Server A chooses the gateway with priority 0: 10.1.1.1.
d. Server A sends the packet to gateway 10.1.1.1 through the 1 GbE interface,
not the 10 GbE interface.
5. Router A receives the response packet from Server A. However, the response
packet arrives on the 1 GbE switch.
a. Router A evaluates the destination IP address, 10.2.1.50, and determines
that it has a route to the subnet through the router at 10.2.1.1, Router C.
b. Router A sends the response packet to Router C through its external
interface.
c. The response packet can pass through more routers as it travels through
the network.
6. Router C receives the packet on its external interface. Router C determines
that it has a direct connection to the destination IP address.
a. Router C sends the packet directly to 10.2.1.50 on its internal interface.
Isilon has only one global routing table for all interfaces. With destination-based
IP routing, packets can be sent using the wrong interface. In some situations,
even a static route will not work properly because the destination can be
reached over multiple networks, as is the case in this example.
l SBR enables sending a packet through the same interface on which it arrived.
Instead of relying on the destination IP, SBR creates dynamic forwarding rules
using the sender's IP address and the subnet that the packet arrives on. It then
creates a reverse rule so that packets going to that IP address will always be
forwarded to the default gateway for that subnet. For example, if you have a
subnet designated IP 10.3.1.x with a gateway of 10.3.1.1, whenever a packet arrives
at the cluster destined for any IP address in the 10.3.1.x subnet, SBR creates a rule
to send return packets to the gateway 10.3.1.1 regardless of what is in the routing
table or gateway priorities.

Note

SBR bypasses the routing table and static routes.

The following diagram shows how the packet flow for the previous example
changes with SBR.

32 OneFS 8.1.0 External Network Connectivity Guide


Routing in OneFS

Destination-based routing overview


This section illustrates traditional, destination-based routing.
Each IP packet header contains a set of four values that identify key routing
information for a network connection. The routing information includes the source IP
address, source port number, destination IP address, and destination port number.
The most common routing method makes all routing decisions based on the
destination (IP address, port) of a packet and does not consider any of the other
fields.
The following diagrams illustrate the IP packet flow from a source to a destination and
back for a simple routing scenario that uses destination-based routing.

Destination-based routing overview 33


Routing in OneFS

In this traditional, destination-based routing scenario, client C1 must send an IP packet


to server A1. The following steps describe how the packet travels from its source,
client C1, to its destination, server A1. This scenario ignores name resolutions and uses
only IP addresses. Note that each router receives packets from any number of other
routers.
1. Client C1 must send an IP packet to server A1 at IP address 10.1.1.80.
a. Client C1 determines that the destination IP address is not local and that it does
not have a static route defined for that address.
b. Client C1 sends the packet to its default gateway, Router C, for further
processing.
2. Router C receives the packet from Client C1.
a. Router C evaluates the destination IP address, 10.1.1.80, and determines that it
has a route to the destination through Router A at IP address 10.1.1.1.
b. Router C sends the packet to Router A through its external interface.
The packet can pass through several routers as it travels through the network.

34 OneFS 8.1.0 External Network Connectivity Guide


Routing in OneFS

3. Router A receives the packet on its external interface.


a. Router A evaluates the IP packet’s 5-tuple to determine the packet’s
destination address.
b. Router A determines that it has a direct (internal) connection to the
destination IP address, 10.1.1.80: server A.
c. Router A sends the packet directly to server A on its internal interface.
4. Server A must send a response to the IP packet’s source, Client C1, at IP address
10.2.1.50.
a. Server A determines that the destination is not local and that it does not have a
static route defined for that address.
b. Server A sends the response packet to its default gateway, Router A, for
further processing.
5. Router A receives the packet from Server A.
a. Router A evaluates the response packet’s IP address, 10.2.1.50, and determines
that it has a route to the subnet through the router at 10.2.1.1: Router C.
b. Router A sends the response packet to Router C at 10.2.1.1 through its external
interface.
c. The response packet can pass through several routers as it travels through the
network.
6. Router C receives the packet on its external interface.
a. Router C determines that it has a direct connection to the response packet’s
destination IP address.
b. Router C sends the response packet directly to 10.2.1.50 on its internal
interface.
All routing decisions are made by using the packet’s destination IP address. At each
hop, each device in the path first examines its directly connected networks, and then
examines its routing table to determine where to send the packet to next.

Advantages and disadvantages of source-based routing


This section describes the advantages and disadvantages of source-based routing
(SBR).
l SBR provides support for very complex network topologies, but works only with
incoming packets. A packet that originates from the cluster that is not a response
from a client still requires processing through standard routing tables. Requests
from the cluster such as DNS lookups, LDAP lookups, AD lookups, e-mail, SNMP,
and other outgoing traffic are not aware of SBR and continue to follow standard
routing rules, including static routes.
l SBR does not support multiple routing tables and does not support using static
routes for the return path.
l SBR might impact the network traffic performance.

NIC affinity overview


NIC affinity is a sysctl that can be configured in OneFS.
The NIC affinity setting applies only when there are multiple NICs on the same node
connected to the same subnet. The NIC affinity setting is enabled automatically when

Advantages and disadvantages of source-based routing 35


Routing in OneFS

there are multiple NICs on the same subnet to enable response packets to go out
using the same NIC that they arrived on, based on the source IP address of the
response packet. The interface that is currently configured with that IP address is the
interface that the packet will be sent on.

How routing tables, SBR and NIC affinity interact


For data that originates from the cluster, standard routing rules always apply.
When SBR is enabled, response packets are sent back using dynamically generated
forwarding rules. With SBR enabled, the routing table and all static routes are
bypassed. NIC affinity is usually enabled or disabled by OneFS. OneFS does this to
balance the outgoing traffic so that not all the traffic leaves on a single interface.
All three features have their place and work together to route packets as efficiently as
possible.

36 OneFS 8.1.0 External Network Connectivity Guide


CHAPTER 7
Recommended reading

This chapter includes the following topics.

l Isilon networking references list.........................................................................38

Recommended reading 37
Recommended reading

Isilon networking references list


This section lists reference materials that can help you understand, plan for, and
troubleshoot your Isilon network implementation.
l Isilon Networking Info Hub: Videos, white papers, Knowledgebase articles,
troubleshooting guides, and more
l OneFS Web Administration Guides for your version of OneFS: OneFS Web
Administration Guides
l OneFS CLI Administration Guides for your version of OneFS: OneFS CLI
Administration Guides
l OneFS API Reference for your version of OneFS: Isilon SDK Info Hub
l Isilon InsightIQ User Guide for your version of InsightIQ: InsightIQ - Isilon Info Hub
l For information about the TCP and UDP ports that OneFS uses, see the OneFS
Security Configuration Guide for OneFS 8.1.0.
l For information about maintaining free space on Isilon clusters and storage pools,
see Best Practices Guide for Maintaining Enough Free Space on Isilon Clusters and
Pools.
Isilon OneFS product documentation for each version of OneFS is available on the
OneFS documentation Info Hubs, including the release-specific EMC Isilon OneFS
Technical Specifications Guide and OneFS Security Configuration Guide.
l OneFS 8.1.0 Documentation - Isilon Info Hub
l OneFS 8.0.1 Documentation - Isilon Info Hub
l OneFS 8.0.0 Documentation - Isilon Info Hub

38 OneFS 8.1.0 External Network Connectivity Guide

S-ar putea să vă placă și