Do

c
ID
Summary of Typical Solaris IP Multipathing (IPMP) Configurations (Doc ID
1010640.1)

To Bottom

APPLIES TO:

Solaris SPARC Operating System - Version 8 10/00 U2 to 10 8/11 U10 [Release 8.0 to 10.0]
Solaris x64/x86 Operating System - Version 8 10/00 U2 to 10 8/11 U10 [Release 8.0 to 10.0]
All Platforms
GOAL

Provide examples for different IPMP configuration scenarios.

SOLUTION

Steps to Follow
Contents:
1. Production and test interfaces in the same IP subnet
1.1 With default router
1.2 Without default router
1.3 With dedicated hosts acting as test targets with "host-routes"
1.4 Configuration example for 1.1, 1.2 and 1.3
2. Production and test interfaces in different IP subnets but the same
physical network
2.1 With default router in production subnet and test subnet
2.2 With default router in production subnet but without default router in test subnet
2.3 With dedicated hosts acting as test targets with "host-routes"
2.4 Configuration example for 2.1, 2.2 and 2.3
2.5 Configuration example for 2.2 if you use IPMP on the test subnet
A. Start script for adding static "host routes" on Solaris 8 and 9
B. Adding persistent "host routes" via 'route -p' option on Solaris 10
C. Summary
Note: This document does not cover the basic knowledge of IP Multipathing. If you would need it, please search in Oracle
Documentation.
All IPMP patches mentioned below should be installed.
OS
Release Sparc Intel Comments
----------------------------------------------------------------------------------7 or earlier N/S N/S IPMP was introduced with Solaris[TM] 8 U2
----------------------------------------------------------------------------------8 108528 108529 kernel update patch (in.mpathd, ip, icmp, ifconfig)
116965 116966 ip/arp/tcp/udp/tun patch
109900 109901 /etc/init.d/network and /sbin/ifparse patch
109902 109903 in.ndpd patch
110378 110396 mipagent patch Mobile IP
116991 116992 rcm_daemon patch
----------------------------------------------------------------------------------9 112233 112234 Kernel Patch
114344 119435 arp, dlcosmk, ip, and ipgpc Patch
" 114348 /usr/sbin/in.routed patch
112912 N/P libinetcfg Patch
113073 114733 ufs and fsck patch
----------------------------------------------------------------------------------10 118833-36 118855-36 kernel patch
120011-14 120012-14 kernel patch (min. required for IPMP patch 141532)

141532 141533 in.mpathd patch

10 U4 (8/07) 127111 127112 kernel patch
10 U5 (5/08) 137111 137112 kernel patch
10 U6 (10/08) 138888 138889 kernel patch
10 U7 (5/09) 141414 141415 kernel patch
10 U8 (10/09) 142900 142901 kernel patch
10 U9 (9/10) 144488 144489 kernel patch
10 U10 (9/10) 147440 147441 kernel patch
----------------------------------------------------------------------------------Key
N/S - Product not supported on this OS
N/P - No patch applicable for this OS
Legend:
IPMP system with IPMP group
T target host
p network/host partof IP address of interface in production subnet
t network/host part of IP address of interface in test subnet
--- data link subnet (i.e broadcast domain) with one IP subnet
=== data link subnet (i.e broadcast domain) with two (or more) IP subnets

Good to know:
The operation of IP Multipathing (in.mpathd) depends on the routing configuration. Therefore in.mpathd always refers to the routingtable (IRE-cache) to distinguish which test partner(s) are going to be used. Test partners are required for deciding if the interface is
working properly.
in.mpathd by default chooses the default router as single test-target for probing. If no default router exists for the test-interface ip
address, arbitrary hosts on the link are detected by sending out "all hosts" multicast packets (224.0.0.1) on the wire to detect its
test-partners. An "all routers" multicasts (224.0.0.2) will never be sent! The first five hosts that are responding to the echo packets
are chosen as targets for probing. In such a non-default router environment, the in.mpathd always tries to find five probe targets via
an "all hosts" multicast.
The in.mpathd detects failures and repairs by sending out 'icmp-echo' probes (like pinging) from all interfaces that are part of the
IPMP group. If there are no replies to five consecutive probes, the interface is considered to have failed and performs a failover of
the network access to another interface in the IPMP group. The probing rate depends on the failure detection time which is defined
in /etc/default/mpathd. By default, failure detection time is 10 seconds. Thus the five probes will be sent within the failure detection
time.
1. Production and test interfaces in the same IP subnet
1.1 With default router

+----------------+
| default router |
+-------o--------+
| p=t:172.20.20.1
|
|
----------+-+------------------------+------------------------| | p=t:172.20.20/24
||
p:172.20.20.10 | |
t:172.20.20.210 | | t:172.20.20.220
+---o-o---+
| IPMP |
+---------+

IPMP uses the default router as a single probe target if it is configured to the same IP subnet. Each test interface of the IPMP group
will send ICMP requests to the default router only. To get the configuration, IPMP looks to the routing table and is independent of
the /etc/defaultrouter file. There will be NO "all hosts" multicast (224.0.0.1) sent out.

Advantages:

Easiest configuration for IPMP.

Disadvantages:

If the default router is down then IPMP failover does not work anymore. The in.mpathd does NOT send out multicasts to
get other probe targets, therefore all interfaces in the IPMP group get the state "failed". You can ignore this bug/feature
when you have a default router which is 100% online! Please look to RFE 4431511 and 4489960 for further information.

If you have a lot of IPMP groups, the default router has to reply to many ICMP requests. Take care about the default router.
Do not overload the default router.

The default router has to reliably answer ICMP echo requests. (e.g. firewalls sometimes do not)

1.2 Without default router

+-----+ +-----+
| T 1 | ... | T 5 |
+--o--+ +--o--+
| p=t:172.20.20.110 | p=t:172.20.20.150
||
||
-----------+-+--------------+---------------------+------| | p=t:172.20.20.0/24
||
p:172.20.20.10 | |
t:172.20.20.210 | | t:172.20.20.220
+---o-o---+
| IPMP |
+---------+

Without any router probe target IPMP sends out an "all hosts" multicast (224.0.0.1) to dynamically determine up to five arbitrary
hosts from the multicast reply. At least, you will need one probe target to get IPMP working but be aware that one probe target is not
reliable enough. As long as there are less than two targets available, the in.mpathd daemon will continuously sent out "all hosts"
multicasts until it has 2 (min) to 5 (max) host probe targets.
Advantages:

easiest configuration for IPMP in a subnet without a default router.

very reliable with up to five targets.

Disadvantages:

a subnet without an default router is very rare.

1.3 With dedicated hosts acting as test targets with "host-routes"

+----------------+ +-----+ +-----+
| default router | | T 1 | ... | T 5 |
+-------o--------+ +--o--+ +--o--+
| | p=t:172.20.20.110 | p=t:172.20.20.150
|||
|||
-----------+-+--------------+---------------------+------| | p=t:172.20.20.0/24
||
p:172.20.20.10 | |
t:172.20.20.210 | | t:172.20.20.220
+---o-o---+
| IPMP |
+---------+

Solaris 8 and 9 - Some "host routes" will be defined with a start script in /etc/rc2.d/S70staticroutes (refer to appendix A). When IPMP
refers to the routing table it will choose the first five defined "host routes" as probe targets. This is due to the fact that normally the
"host routes" are before the default router in the routing table. If you have less than five "host routes", the default router (when
available) will be used as a probe target too.
Examples:
1.

Configuration with host1, host2 ... hostN (with N=5 or N>5), defaultrouter :
==> The first five hosts (host1 ... host5) will be defined as target, not the default router

2.

Configuration with less than 5 hosts : for instance, host1, host2, defaultrouter :
==> The three systems (host1, host2, defaultrouter) will be defined as target.
The in.mpathd continuously tries to get five probe targets from the routing table but it will NOT send "all hosts"
multicasts!

Solaris 10 - Use the "route -p" option to add persistent routes to the kernel (refer to appendix B). No additional script will be
necessary. If there is a default router defined in /etc/defaultrouter this will always be used as a probe target plus up to four additional
probe targets defined via "host routes".
Advantages:

The default router is not a single point of failure anymore because you still have some other "host routes" as probe
targets.

Higher availability.

Disadvantages:

More administrative work to do.

Due to static configuration you should regularly check that the probe targets are available.

1.4 Configuration examples for 1.1, 1.2 and 1.3

/etc/hosts
127.0.0.1 localhost
172.20.20.10 host10 loghost
172.20.20.210 host10-test-qfe0
172.20.20.220 host10-test-qfe4
/etc/hostname.qfe0
host10 netmask + broadcast + group ipmp0 up \
addif host10-test-qfe0 netmask + broadcast + deprecated -failover up
/etc/hostname.qfe4
host10-test-qfe4 netmask + broadcast + deprecated -failover group ipmp0 up
ifconfig output:
qfe0: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 172.20.20.10 netmask ffffff00 broadcast 172.20.20.255
groupname ipmp0
ether 8:0:20:e8:88:dc
qfe0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
inet 172.20.20.210 netmask ffffff00 broadcast 172.20.20.255
qfe4: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 4
inet 172.20.20.220 netmask ffffff00 broadcast 172.20.20.255
groupname ipmp0
ether 8:0:20:e8:89:34

Note: The example describes the setup of an active-active IPMP configuration which is best practices. But you can also configure an
active-standby configuration. You simple have to add the "standby" flag to the /etc/hostname.qfe4 file. More Details available in the

following Technical Instruction Document 1012450.1 "Solaris 8 and up: How to Disable Outbound Load Balancing in an IPMP group".
2. Production and test interfaces in different IP subnets
If you don't have enough additional IP addresses available, you can configure the IPMP test interfaces to a different IP subnet than
your production network (e.g. 192.168., 10. ..). But you have to make sure that there are always enough test partners in that test
network to respond to the ICMP probes. In such a configuration in.mpathd will use the test IP addresses only as source address for
the outgoing probe packets.
Note: The in.mpathd only checks the test network. If there is no responding probe target in that IP subnet the IPMP group will fail
even though the production subnet is still available.
2.1 With default router in production subnet and test subnet
+----------------+
| default router |
+-------o--------+
| p:172.20.20.1
| t:192.168.1.1
|
| t:192.168.1.0/24
----------+-+------------------------+------------------------| | p:172.20.20.0/24
||
p:172.20.20.10 | |
t:192.168.1.210 | | t:192.168.1.220
+---o-o---+
| IPMP |
+---------+

IPMP uses the default router as a single probe target if it is configured to the same IP subnet. Each test interface of the IPMP group
will send ICMP requests to the default router only. To get the configuration, IPMP looks to the routing table and is independent of
the /etc/defaultrouter file. There will be NO "all hosts" multicast (224.0.0.1) sent out.
Advantages:

No additional production IP addresses required for the test interfaces

Disadvantages:

The interface of the default router has to reside in both the production AND test subnet.

Exceptional configuration of default router.

All other disadvantages already mentioned in section 1.1

2.2 With default router in production subnet net but without default router in test subnet

+----------------+ +-----+ +-----+

| default router | | T 1 | ... | T 5 |
+-------o--------+ +--o--+ +--o--+
p:172.20.20.1 | | p:172.20.20.110 | p:172.20.20.150
| | t:192.168.1.110 | t:192.168.1.150
|||
| | | t:192.168.1.0/24
-----------+-+--------------+---------------------+------| | p:172.20.20.0/24
||
p:172.20.20.10 | |
t:192.168.1.210 | | t:192.168.1.220
+---o-o---+
| IPMP |
+---------+
Without any router probe target in the test subnet IPMP sends out an "all hosts" multicast (224.0.0.1) to dynamically determine up

to five arbitrary hosts from the multicast reply. At least, you will need one probe target to get IPMP working but be aware that one
probe target is not reliable enough. As long as there are less than two targets available, the in.mpathd daemon will continuously
sent out "all hosts" multicasts until it has 2 (min) to 5 (max) host probe targets.
Advantages:

easiestIPMPconfiguration if youdon't have enough additional IP addresses availablein the production subnet.

very reliable with up to five targets.

Disadvantages:

the probe targets must be available before you can setup the IPMP host.

more administrative work because you have to setup the probe targets as an additional interface in the test subnet.
Recommendation: Setup an additional logical network interface on target host.
(e.g. add a new interface to /etc/hostname.qfe0 with the 'addif' option)
If your test partners are on systems which run also IPMP you must add an logical network interface NOT flagged
deprecated and NOT flagged nofailover. An address associated with this interface will then be used as source address by
responding properly to "all hosts" multicast packets which are used for the automatic IPMP test partner detection. Be
aware that Solaris 8 does NOT require the additional logical network interface in the test subnet on the target host. So, in
case of an upgrade from Solaris 8 to Solaris[TM] 9 or higher you have to change your configuration.

Please refer to 2.5 for a detailed example.

2.3 with dedicated hosts acting as test targets with "host-routes"

+----------------+ +-----+ +-----+

| default router | | T 1 | ... | T 5 |
+-------o--------+ +--o--+ +--o--+
p:172.20.20.1 | | p:172.20.20.110 | p:172.20.20.150
| | t:192.168.1.110 | t:192.168.1.150
|||
| | | t:192.168.1.0/24
-----------+-+--------------+---------------------+------| | p:172.20.20.0/24
||
p:172.20.20.10 | |
t:192.168.1.210 | | t:192.168.1.220
+---o-o---+
| IPMP |
+---------+
Solaris 8 and 9 - Some "host routes" will be defined with a start script in /etc/rc2.d/S70staticroutes (refer to appendix A). When IPMP
refers to the routing table it will choose the first five defined "host routes" as probe targets. This is due to the fact that normally the
"host routes" are before the default router in the routing table. If you have less than five "host routes", the default router (when
available in the test subnet) will be used as a probe target too.
Examples:
Please refer to the examples of section 1.3.
Solaris 10 - Use the "route -p" option to add persistent routes to the kernel (refer to appendix B). No additional script will be
necessary. If there is a default router defined in /etc/defaultrouter this will always be used as a probe target plus up to four additional
probe targets defined via "host routes".

Advantages:

test interfaces don't need IP addresses of the production subnet

allothers already mentioned in section 1.3

Disadvantages:

all others already mentioned in section 1.3

all others already mentioned in section 2.2

2.4 Configuration examples for 2.1, 2.2 and 2.3

/etc/hosts
127.0.0.1 localhost
172.20.20.10 host10 loghost
192.168.1.210 host10-test-qfe0
192.168.1.220 host10-test-qfe4
/etc/hostname.qfe0
host10 netmask + broadcast + group ipmp0 up \
addif host10-test-qfe0 netmask + broadcast + deprecated -failover up
/etc/hostname.qfe4
host10-test-qfe4 netmask + broadcast + deprecated -failover group ipmp0 up

ifconfig output:
qfe0: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 172.20.20.10 netmask ffffff00 broadcast 172.20.20.255
groupname ipmp0
ether 8:0:20:e8:88:dc
qfe0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
inet 192.168.1.210 netmask ffffff00 broadcast 192.168.1.255
qfe4: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 4
inet 192.168.1.220 netmask ffffff00 broadcast 192.168.1.255
groupname ipmp0
ether 8:0:20:e8:89:34

Note: The example describes the setup of an active-active IPMP configuration which is best practices. But you can also configure an
active-standby configuration. You simple have to add the "standby" flag to the /etc/hostname.qfe4 file. More details available in the
following Technical Instruction Document 1012450.1 "Solaris 8 and up: How to Disable Outbound Load Balancing in an IPMP group".
2.5 Configuration example for 2.2 if you use IPMP on the test subnet
/etc/hosts
127.0.0.1 localhost
#
# IPMP active IP address:
#
172.20.20.10 host10 loghost
#
# IPMP test interfaces
#
192.168.1.210 host10-test-ce0
192.168.1.220 host10-test-ce1
#
# additional active interface in the 'test subnet'
# to be able to respond to probe packets from other
# IPMP setups (See 2.2). Will also failover.
#
192.168.1.211 host10-prod-ce0

/etc/hostname.ce0
host10 netmask + broadcast + group ipmp0 up \
addif host10-prod-ce0 netmask + broadcast + up \
addif host10-test-ce0 netmask + broadcast + deprecated -failover up
/etc/hostname.ce1

host10-test-ce1netmask + broadcast + deprecated -failover group ipmp0 up

ifconfig output:
ce0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 172.20.20.10 netmask ffffff00 broadcast 172.20.20.255
groupname ipmp0
ether 8:0:20:e2:da:d5
ce0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 192.168.1.211 netmask ffffff00 broadcast 192.168.1.255
ce0:2: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
inet 192.168.1.210 netmask ffffff00 broadcast 192.168.1.255
ce1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
inet 192.168.1.220 netmask ffffff00 broadcast 192.168.1.255
groupname ipmp0
ether 8:0:20:e2:da:d6

A. Start script for adding static "host routes" on Solaris 8 and 9

----------- Begin of start script /etc/init.d/ipmp.targets -------------#!/sbin/sh

# /etc/rc2.d/S70ipmp.targets /etc/init.d/ipmp.targets
# Copyright (c) 2005 by Sun Microsystems, Inc.
# All rights reserved.
#
#ident "@(#)ipmp.targets 1.0.0
#
# Edit the following IPMP test TARGETS to suit your needs.
# To install:
# 1) cp ipmp.targets /etc/init.d
# 2) perform edits on the script as required (e.g: add TARGETS)
# 3) chmod 744 /etc/init.d/ipmp.targets
# 4) chown root:sys /etc/init.d/ipmp.targets
# 5) ln /etc/init.d/ipmp.targets /etc/rc2.d/S70ipmp.targets
#
TARGETS="172.20.20.117 172.20.20.127 172.20.20.137172.20.20.147172.20.20.157"
case "$1" in
'start')
/usr/bin/echo "Adding static routes for use as IPMP targets"
for target in $TARGETS; do
/usr/sbin/route add -host $target $target
done
;;
'stop')
/usr/bin/echo "Removing static routes for use as IPMP targets"
for target in $TARGETS; do
/usr/sbin/route delete -host $target $target
done
;;
esac
----------- End of start script /etc/init.d/ipmp.targets --------------

See also:
System Administration Guide: IP Services > Configuring IPMP Groups > Configuring Target Systems
B. Adding persistent "host routes" via 'route -p' option on Solaris 10

#
#
#
#
#

route
route
route
route
route

-p
-p
-p
-p
-p

add
add
add
add
add

-host
-host
-host
-host
-host

172.20.20.117
172.20.20.127
172.20.20.137
172.20.20.147
172.20.20.157

172.20.20.117
172.20.20.127
172.20.20.137
172.20.20.147
172.20.20.157

-static
-static
-static
-static
-static

Note: If there is a default router defined in /etc/defaultrouter this will always be used as a probe target plus up to four additional

probe targets defined via "host routes" (the first four).

If you don't like the default router to be used as a probe target at all, you'll have to delete the /etc/defaultrouter file and to define
the default router as the last static route, e.g.
# rm /etc/defaultrouter
#
#
#
#
#
#

route
route
route
route
route
route

-p
-p
-p
-p
-p
-p

add
add
add
add
add
add

-host 172.20.20.117 172.20.20.117

-host 172.20.20.127 172.20.20.127
-host 172.20.20.137 172.20.20.137
-host 172.20.20.147 172.20.20.147
-host 172.20.20.157 172.20.20.157
default 172.20.20.1 -static

-static
-static
-static
-static
-static

Note: Due to the missing /etc/defaultrouter file, Solaris will automatically start the routing daemon 'in.routed' at boot time, so you
should configure the "quiet mode" for RIP to not supply any routing information to other systems:
# routeadm -m route:default quiet_mode=true
C. Summary
It's not easy to give a general recommendation because it depends on the network infrastructure which the customers have.
Therefore you have to discuss the various possibilities with your customer. Maybe the most used setups are 1.1 and 2.2.
To discuss this information further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in
the My Oracle Support Community, Oracle Solaris Networking.

https://support.oracle.com/epmos/faces/DocumentDisplay?
_afrLoop=504864511487999&id=1010640.1&displayIndex=2&_afrWindowMode=0&_adf.ctrl-state=ottybr2y1_562
This document assumes the local-mac-address variable in the OpenBoot[TM] PROM (OBP) has been set to true prior to the most
recent boot.
This IBIS shows how to setup IPMP on the fly with both physical interfaces already configured. For a permanent solution consult
Document: 1010640.1
This document assumes the local-mac-address variable in the OpenBoot[TM] PROM (OBP) has been set to true prior
to the most recent boot.
This can be set from the command line:
# eeprom local-mac-address =true
# reboot
or from the ok prompt:
ok> setenv local-mac-address True
ok> reset-all
You need at least two interfaces in each failover group. For the purpose of this doc, hme0 and hme1 are the two interfaces.
Become superuser. Place each physical interface into the same group, using the command below:
# ifconfig hme0 group failover
# ifconfig hme1 group failover
Configure a test address for each interface that you want to failover.
# ifconfig hme0 addif <ip-address> <parameters> -failover deprecated up
For example,

# ifconfig hme0 addif 192.168.1.1 netmask + broadcast + -failover deprecated up

You can check the configuration by typing the following:
# ifconfig hme0:1
Note: You must mark a test address as deprecated to prevent applications from using the test address.
To configure a test address on hme1 with the following configuration:
# ifconfig hme1 addif ip-address netmask + broadcast + -failover deprecated up

This will setup hme1 for two-way failover