Documente Academic
Documente Profesional
Documente Cultură
OneFS
Version 8.1.0
Dell believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS-IS.“ DELL MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND
WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. USE, COPYING, AND DISTRIBUTION OF ANY DELL SOFTWARE DESCRIBED
IN THIS PUBLICATION REQUIRES AN APPLICABLE SOFTWARE LICENSE.
Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be the property of their respective owners.
Published in the USA.
EMC Corporation
Hopkinton, Massachusetts 01748-9103
1-508-435-1000 In North America 1-866-464-7381
www.EMC.com
Isilon networking
This section reviews basic principles of Isilon networking.
An Isilon cluster requires two networks.
l Internal—A high-speed, low-latency network for communications and data
transfer among nodes in a cluster. The internal network can either be InfiniBand or
a backend Ethernet that is supported in OneFS 8.1.0.0 and later.
l External—An Ethernet connection for client users and applications that access
the cluster through an external, or front-end, network.
The Isilon OneFS operating system supports a single cluster on the internal network.
The Isilon OneFS operating system supports standard network communication
protocols IPv4 and IPv6, as well as client access protocols, including NFS, SMB,
HTTP, HDFS, FTP, and OpenStack Swift. Different Isilon nodes include several
external Ethernet connection options, providing flexibility for a wide variety of
network configurations.
The building blocks of Isilon networking are groupnets, subnets, and pools. Groupnets
are networking objects that allow for portions of the cluster to have different DNS
resolution settings. For more information about subnets and groupnets, refer to the
Subnet and groupnet overview section in this guide. Pools are logical partitions of
subnets. How you set up your external network subnets depends on your network
topology. A basic network topology where all client-node communication occurs
through a single gateway requires only a single external subnet. If clients connect
through multiple subnets or internal connections, you must configure multiple external
network subnets. By default, when you create a cluster, OneFS automatically creates
the initial subnet or pool for your network. It also creates a provisioning rule
automatically for the interface type that you selected.
Note
If you created your cluster using pre-Generation 6 hardware, and have set a
provisioning rule for ext-1, you must create a second provisioning rule for mgmt-1
before adding your Generation 6 nodes in case you want the nodes to get
automatically added.
In general, keeping the network configuration simple provides the best results with the
lowest amount of administrative overhead. OneFS offers network provisioning rules to
automate the configuration of additional nodes as clusters grow.
The following licensed Isilon modules provide advanced features to help you to
manage and monitor your network.
l The Isilon SmartConnect Advanced module provides IP address allocation
management and connection balancing.
l The Isilon InsightIQ virtual appliance provides cluster performance monitoring and
analysis to help you optimize storage resources and forecast capacity. InsightIQ
provides a view of network throughput and rates at the moment and over time.
For more information about Isilon networking, monitoring, and management, see the
Isilon OneFS Web Administration Guide, Isilon OneFS CLI Administration Guide, and Isilon
InsightIQ User Guide for your version of Isilon OneFS and Isilon InsightIQ.
Isilon networking 5
External network connectivity overview
Workload characteristics
CPU X X X
utilization
Connection X X
count
Network X X X
throughput
l Observe SMB connection limits. For more information about limitations such as
the recommended maximum number of SMB connections, see the EMC Isilon
OneFS technical specifications guide for your version of OneFS.
l Regularly monitor cluster usage with InsightIQ, the web administration interface,
or the command-line interface (CLI). If available disk space approaches 20 to 15
percent (disk space usage reaches 80 percent to 85 percent of capacity), consider
adding a node. When disk space usage reaches 90 percent, we strongly
recommend adding additional capacity. This helps to ensure that there is sufficient
usable space for successful OneFS failover and to achieve performance goals for
the cluster. When disk space usage reaches 95 percent of capacity, the risk for file
system anomalies increases. For more information, see Best Practices Guide for
Maintaining Enough Free Space on Isilon Clusters and Pools.
l Many cluster configuration settings are global and have cluster-wide effects. If
you consider changing cluster-wide configuration settings, be sure that you fully
understand the global settings and their implications. For information about global
cluster configuration settings, see the Isilon OneFS Web Administration Guide or
Isilon OneFS CLI Administration Guide for your version of OneFS. You can deploy an
Isilon OneFS Simulator to test the changes before you make changes that affect
production.
l Confirm that remote support functions work correctly through EMC Secure
Remote Support (ESRS), SupportIQ, or internal email/SNMP notifications.
l Always use at least the default data protection scheme. Inadequate data
protection, or none at all, can cause transient failures for the duration of a job. As
a result, data might be lost or scrambled. Recheck the cluster’s data protection
levels as the cluster grows.
l Calculate the recommended protection level based on cluster configuration. The
suggested protection level strikes the best balance between data protection and
storage efficiency.
l Recommend to your client system administrators that they turn off client DNS
caching wherever possible. All the currently shipping versions of Windows DNS
servers will automatically change the DNS caching value to one second. To handle
client requests properly, SmartConnect requires that clients use the latest DNS
entries. If clients cache SmartConnect DNS information, they might connect to
incorrect SmartConnect zone names. In this situation, SmartConnect might not
appear to be functioning properly.
l Use SmartConnect Advanced to manage multipath I/O.
l If you plan to use VLANs for improved redundancy and availability, set up Link
Aggregation Control Protocol (LACP) and create VLANs on top of the aggregated
interface.
l Use consistent frame sizes across your network. For example, if you enable jumbo
frames, enable them uniformly across your network infrastructure to prevent
packet fragmentation or dropped data when you try to write through a smaller
switch.
l For information about configuring LACP between a Cisco switch and an Isilon
cluster, see article 000304472 on the EMC Online Support site.
Link aggregation
This section describes the advantages and disadvantages of link aggregation.
Link aggregation combines multiple physical connections (ports) in parallel to provide
failover functionality in case one of the aggregated ports fails. Link aggregation does
not increase bandwidth beyond a single link for a single client. However, it provides
bandwidth that is equivalent to all the links when multiple clients are involved.
For more information about link aggregation, see IEEE 802.3ad Link Bundling.
Note
With dynamic SmartConnect zones for NFSv3 clients, link aggregation does not
provide any discernible improvement in functionality or in reliability.
Link aggregation 9
Network planning and design considerations
Question Answer
Does connecting extra network interfaces for No. Disks are the limiting factor, not the
management increase the total network network interface cards (NICs).
throughput to the cluster?
Does using dedicated 1 Gbe network No. Although the motherboards on Isilon
interfaces for management give out-of-band nodes have IPMI interfaces, their use is not
access to the cluster? supported at this time. Therefore, the 1 GbE
interfaces do not provide that functionality.
Does connecting extra 1 GbE network No. Management traffic is only a tiny fraction
interfaces for management minimize the of the traffic that users generate for normal
impact of management traffic on user traffic? data access.
Question Answer
Security: Can management on the 10 GbE No.
NICs be shut off so that it acts like a VNX
datamover, with the 1 GbE acting as the
control station?
Note
The recommended largest allocation per cluster is a /23 subnet, or 510 usable
addresses.
Suppose that you have a cluster with three IP pools. The cluster will be used for
NFSv3 and SMB client access and will have a dedicated IP pool for backup and
replication. Use the following factors to calculate the number of IP addresses that the
cluster requires (assuming 2x10 GbE per node and no LACP).
l 1 for the SmartConnect service IP (SSIP)
l 2*N for static pool0 (with two logical interfaces per node) (for stateful
connections [SMB])
l N*(N-1) for pool1 (for stateless connections [NFS])
l 2*N for pool2 (with two logical interfaces per node (for backup and replication)
The extrapolated formula is:
1+2N+2N+N(N-1)=x
1+4N+N²-N=x
or:
N²+3N+1=x
Where:
l N=the number of nodes
l x=the total number of IPs needed for the current cluster size
Given a four-node cluster, this works out as:
4²+3*4+1=x
16+12+1=x
x=29
Use this formula as a guide, keeping in mind the recommended largest allocation per
cluster is a /23 subnet, or 510 usable addresses.
The following table contains results for a partial extrapolation of the formula, starting
with three nodes. The table shows how the number of IP addresses needed for a
cluster grows as the number of nodes increases.
4 2 1 8 12 8 29
5 2 1 10 20 10 41
6 2 1 12 30 12 57
7 2 1 14 42 14 71
.....
21 2 1 42 420 42 504
The number of IPs expands exponentially. At a certain point, the benefits of having
large numbers of IP addresses with dynamic SmartConnect zones have diminishing
returns. There are very few cases that require an IP allocation larger than the
recommended /23 subnet, or 510 usable addresses.
From a load-balancing perspective, it is ideal (though optional) that all the interfaces
for dynamic pools have the same number of IP addresses whenever possible.
In this scenario, the three-node cluster with one network connection per node and one
dynamic SmartConnect zone has only three IP addresses. After a few days, 100 clients
are actively connected to each node over NFS using a round robin connection policy.
One IP address will be assigned to each node, as shown in the following illustration.
Most NFSv3 mountd clients do an nslookup only the first time that they perform a
mount, so they never do another nslookup to check for an updated IP address. If the
IP address changes, the NFSv3 clients have a stale mount and retain that IP address.
Suppose that one of the nodes fails, as shown in the following illustration.
However, even though all three IP addresses are still online, two of them—and 200
clients—are now connected to one node. That’s because SmartConnect can fail only
one IP to one other place, and one IP address and 100 clients are already connected to
the other node. This means that one node failing has just doubled the load on one of
the two remaining nodes, while doing nothing to the third node. The result is that
client performance declines, but not equally. The goal of any scale-out NAS solution
must be consistency. To double the I/O on one node and not on another is
inconsistent.
In this scenario, the three-node cluster has the same configuration but with two IP
addresses per node instead of one IP address per node. The cluster has the same
number of clients.
This cluster follows the rule of N*(N-1), in this case, 3*(3-1) = 6, or two IPs per node.
When the same failure event occurs, the two IP addresses are spread over all the
other nodes in that SmartConnect zone. This means that each remaining node has 150
clients and three IP addresses. So although performance might degrade to some
degree, that degradation will not be as drastic as the failure in the first scenario. The
experience is consistent for all users, as shown in the following illustration.
an IPv6 address. A client using the IPv4 protocol requests an IPv4 address and
receives it from 1.2.3.4.
We strongly advise against keeping the IP addresses of legacy platforms when you
migrate your network to Isilon. SmartConnect functions properly when you use only
DNS names for client data access.
Information includes:
l Common causes of data unavailable (DU) conditions and the actions you can take
to prevent them.
l Best practices for backup and disaster recovery planning with the OneFS
SmartConnect Advanced module.
Protocol holes in NFSv3 and unconfigured Change the SmartConnect IP rebalance delay
SmartConnect IP rebalance delay to a value other than zero.
Setting up an address only on a part of the If your SmartConnect network spans multiple
SmartConnect network networks, set up a separate address for each
SmartConnect network, including the
spanning network.
Using inadequate data protection schemes or Always use at least the default data
running upgraded clusters at a protection protection scheme.
level that is lower than the recommended data
protection scheme level
Some of the other common causes that could result in data unavailable (DU)
conditions are as follows:
l Interfaces are down or a specific gateway cannot be reached.
l Protocols (SMB, NFS, HDFS, or Swift) are down.
l A node is down or suspended.
requires one IP address for each active logical interface. For example, if you have two
active physical interfaces, 10gige-1 and 10gige-2, you need two IP addresses. But if
you configure a link aggregation with LACP, 10gige-agg-1, you need only one IP
address. You can source-restrict all SyncIQ jobs to use your dedicated static
SmartConnect zone.
For example, consider a 10-node cluster for which the requirement is to have all
SyncIQ traffic run across five of the nodes. You can restrict the SmartConnect subnet
and pool that the job can run against. Because you can limit which nodes participate in
that pool, you can restrict on the source which nodes will be used for SyncIQ. For
example, you can use archive nodes as your SyncIQ source and keep the faster nodes
for client I/O.
By restricting SyncIQ backup and replication jobs to a dedicated static SmartConnect
Zone, you can easily redirect backup and replication traffic from certain nodes to
reduce the impact of SyncIQ jobs on user or client I/O. You can redirect traffic
without reconfiguring or modifying the interfaces participating in the SmartConnect
zone.
For example, consider a data ingest cluster for a sports television network. The cluster
must ingest large amounts of data recorded in 4K video format. The data must be
active immediately, and the cluster must store the data for a long time. The sports
television network administrators want to keep data ingestion and data archiving
separate, to maximize performance. The sports television network purchased two
types of nodes: H500s for ingesting data, and A200s for the long-term archive.
Because the data set is so large, SyncIQ jobs to replicate the data to the disaster
recovery site have a lot of work to do on each pass. The front-end interfaces are
saturated on the H500 nodes for either ingesting data or performing immediate data
retrieval. The CPUs of those nodes must not be affected by the SyncIQ jobs. By using
a separate static SmartConnect pool, the network administrators can force all SyncIQ
traffic to leave only the A200 nodes and provide maximum throughput on the H500
nodes.
SmartConnect considerations 23
SmartConnect considerations
address. For example, this issue occurs in Mac OS X for certain client
configurations.
l The site DNS servers must be able to communicate with the node that is currently
hosting the SmartConnect service. This is the node with the lowest DevID with an
active interface in the subnet that contains the SSIP address. This behavior
cannot be modified.
l The stats used by the connection count, CPU usage, and throughput policies are
updated approximately every 5 seconds.
l Site DNS servers might not exist in the regular local subnets, or in any of the
subnets that clients occupy. To enable the SmartConnect lookup process, make
sure that the DNS servers use a consistent route to the cluster and back. If the
site DNS server sends a lookup request that arrives through one local subnet on
the cluster, but the configured cluster routing causes the response to be sent
through a different subnet, it’s likely that the packet will be dropped and the
lookup will fail. The solutions and considerations for SmartConnect are similar to
the client scenarios. Additionally, the DNS server might benefit from a static route
to the subnet that contains the SSIP address(es).
l SmartConnect makes it possible for different nodes to have different default
routes, but this is fundamentally determined by connectivity. SmartConnect
enables you to define multiple gateways: 1 gateway per subnet. Each gateway is
assigned a priority when it is defined. On any node, SmartConnect attempts to use
the highest priority gateway. The highest priority gateway is usually the one that
has the lowest number and has an available functioning interface in a subnet that
contains the gateway address.
C:\>nslookup
Default Server: 10.123.17.60
Address: 10.123.17.60
> isi01-s0.domain.com
Server: [10.123.17.60]
Address: 10.123.17.60
Name: isi01-s0.domain.com
Address: 10.123.17.64
> isi01-s0.domain.com
Server: [10.123.17.60]
Address: 10.123.17.60
Name: isi01-s0.domain.com
Address: 10.123.17.63
Note
Routing in OneFS 29
Routing in OneFS
Based on the above configuration, the cluster should query the IP address of
AD1.FOO.COM from DNS server 1.2.3.4, and should query the IP address of
AD2.FOO.COM from DNS server 5.6.7.8.
created using OneFS 8.0.0.0 and later releases. Instead, we recommend that you use
link aggregation with the Link Aggregation Control Protocol (LACP) for these clusters.
Historically, many clusters were configured with multiple interfaces in the same
subnet, with IP pools split among the interfaces. Because of the way IP routing works,
while incoming traffic tended to be balanced, outgoing traffic used only one interface:
the first in the routing table. OneFS sysctl .inet.ip.choose_ifa_by_ipsrc
was added to address this issue. If sysctl .inet.ip.choose_ifa_by_ipsrc is
enabled (set to 1), OneFS IP routing determines the interface to send from based on
the source address in the IP packet.
Note
This configuration does not affect routing, which is determined by the destination IP
address.
Note
The following diagram shows how the packet flow for the previous example
changes with SBR.
there are multiple NICs on the same subnet to enable response packets to go out
using the same NIC that they arrived on, based on the source IP address of the
response packet. The interface that is currently configured with that IP address is the
interface that the packet will be sent on.
Recommended reading 37
Recommended reading