RMC (Resource Monitoring and Control): RMC is a distributed framework and architecture that allows the HMC to communicate

with a managed logical partition. RMC daemons should be running on AIX partition in order to be able to do DLPAR operations on HMC. For example "Dynamic LPAR Resource Manager" is an RMC daemon that runs inside the AIX (and VIO server). The HMC uses this capability to remotely execute partition specific commands. The daemons in the LPARs and the daemons on the HMC must be able to communicate to the AIX partition through an external network not through the Service Processor. An external network that the partition has access to and the HMC has acces to as well. For example, if HMC has a connection to a 9.x.x.x network and I put my AIX partition to that 9.x.x.x network and as long there is network connectivity (HMC is allowed to communicate to that partition over that network) and RMC daemon is running on the partition, then DLPAR operations are available. In order for RMC to work, port 657 upd/tcp must be open in both directions between the HMC public interface and the lpar. The RMC daemons are part of the Reliable, Scalable Cluster Technology (RSCT) and are controlled by the System Resource Controller (SRC). These daemons run in all LPARs and communicate with equivalent RMC daemons running on the HMC. The daemons start automatically when the operating system starts and synchronize with the HMC RMC daemons. Note: Apart from rebooting, there is no way to stop and start the RMC daemons on the HMC! ---------------------------------------HMC and LPAR authentication (RSCT authentication) (RSCT authentication is used to ensure the HMC is communicating with the correct LPAR.) Authentication is the process of ensuring that another party is who it claims to be. Authorization is the process by which a cluster software component grants or denies resources based on certain criteria. The RSCT component that implements authorization is RMC. It uses access control list (ACL) files to control user access to resources.

The RSCT authorization process in detail: 1. On the HMC: DMSRM pushes down a secret key and HMC IP address to NVRAM where AIX LPAR exists. 2. On the AIX LPAR: CSMAgentRM, reads the key and HMC IP address out from NVRAM. It will then authenticate the HMC. This process is repeated every five minutes on a LPAR to

detect new HMCs. 3. On the AIX LPAR: After authenticating the HMC, CSMAgentRM will contact the DMSRM on the HMC to create a ManagedNode resource then it creates a ManagementServer resource on AIX. 4. On the AIX LPAR: After the creation of these resources on the HMC and AIX, CSMAgentRM grants HMC permission to access necessary resources on the LPAR and changes its ManagedNode Status to 1 on the HMC. 5. On the HMC: After the ManagedNode Status is changed to 1, a session is established with the LPAR to query operating system information and DLPAR capabilities, and then waits for the DLPAR commands from users. ---------------------------------------RMC Domain Status When partitions have active RMC connections, they become managed nodes in a Management Domain. The HMC is then the Management Control Point (MCP) of that Management Domain. You can then use the rmcdomainstatus command to check the status of those managed nodes (i.e. your partitions). As root on the HMC or on the AIX LPAR you can execute the rmcdomainstatus command as follows: # /usr/sbin/rsct/bin/rmcdomainstatus -s ctrmc From HMC: You should get a list of all the partitions that the HMC server can reach on the public network on port 657. Management Domain Status: Managed Nodes O a 0xc8bc2c9647c1cef3 0003 9.2.5.241 I a 0x96586cb4b5fc641c 0002 9.2.5.33

From LPAR: You should get a list of all the Management Control Points Management Domain Status: Management Control Points I A 0xef889c809d9617c7 0001 9.57.24.139

1. column: -I: Indicates that the partition is "Up" as determined by the RMC heartbeat mechanism (i.e. an active RMC connection exists). -O: Indicates that the RMC connection is "Down", as determined by the RMC heartbeat

mechanism. 2. column: -A: Indicates that there are no messages queued to the specified node -a: Same as A, but the specified node is executing a version of the RMC daemon that is at a lower code level than the local RMC daemon. more info: https://www-304.ibm.com/support/docview.wss?uid=isg3T1011508 ---------------------------------------Things to check at the HMC: - checking the status of the managed nodes: /usr/sbin/rsct/bin/rmcdomainstatus -s ctrmc (you must be root on the HMC) - checking connection between HMC and LPAR: hscroot@hmc10:~> lspartition -dlpar <#0> Partition:<2*8204-E8A*0680E12 aix10.domain.com, 10.10.50.18> Active:<1>, OS:<AIX, 6.1, 6100-03-01-0921>, DCaps:<0x4f9f>, CmdCaps:<0x1b, 0x1b>, PinnedMem:<1452> <#1> Partition:<4*8204-E8A*0680E32 aix20.domain.com, 10.10.50.71> Active:<0>, OS:<AIX, 6.1, 6100-04-05-1015>, DCaps:<0x0>, CmdCaps:<0x1b, 0x1b>, PinnedMem:<656> For correct DLPAR function: - the partition must return with the correct IP of the lpar. - the active value (Active:...) must be higher than zero, - the decaps value (DCaps:...) must be higher 0x0 (The first line shows a DLPAR capable LPAR, the second line is anon-working LPAR) ---------------------------------------Things to check at the LPAR: - checking the status of the managed nodes: /usr/sbin/rsct/bin/rmcdomainstatus -s ctrmc - Checking RMC status: # lssrc -a | grep rsct ctrmc rsct 8847376 active <--it is a RMC subsystem IBM.DRM rsct_rm 6684802 active <--it is for executing the DLPAR command on the partition IBM.DMSRM rsct_rm 7929940 active <--it is for tracking statuses of partitions IBM.ServiceRM rsct_rm 10223780 active

IBM.CSMAgentRM rsct_rm 4915254 active <--it is for handshaking between the partition and HMC ctcas rsct inoperative <--it is for security verification IBM.ERRM rsct_rm inoperative IBM.AuditRM rsct_rm inoperative IBM.LPRM rsct_rm inoperative IBM.HostRM rsct_rm inoperative <--it is for obtaining OS information You will see some active and some missing (The key for DLPAR is the IBM.DRM) - Stopping and starting RMC without erasing configuration: # /usr/sbin/rsct/bin/rmcctrl -z <--it stops the daemons # /usr/sbin/rsct/bin/rmcctrl -A <--adds entry to /etc/inittab and it starts the daemons # /usr/sbin/rsct/bin/rmcctrl -p <--enables the daemons for remote client connections (This is the correct method to stop and start RMC without erasing the configuration.) Do not use stopsrc and startsrc for these daemons; use the rmcctrl commands instead! - recfgct: deletes the RMC database, does a discovery, and recreates the RMC configuration # /usr/sbin/rsct/install/bin/recfgct (Wait several minutes) # lssrc -a | grep rsct (If you see IBM.DRM active, then you have probably resolved the issue) - lsrsrc "IBM.ManagementServer" <--it shows HMCs via RMC