Avamar System Shutdown and Startup Procedures

EMC | Avamar Professional Services

Purpose and Scope: This document is provided for EMC/Avamar customers, which need to shutdown and restart the Avamar grid.

IMPORTANT: Verify you have a validated checkpoint

See the Avamar System Administrators Manual PDF for information about validating a checkpoint. You must have a recent validated (HFSCheck) checkpoint before shutting down the Avamar system. If there are complications with the shutdown, you may have to rollback the system to the last checkpoint. Please plan ahead, if possible, for the system shutdown and startup procedures. Validating a checkpoint, shutting down and starting up the GSAN can take many hours to complete depending on the size and capacity of the Avamar system.

To stop Avamar software and shutdown host systems

Log into the Avamar Administrator GUI and cancel any running backup jobs from the Activity screen. Quit the Administrator GUI. Using PuTTy from from the KVM console, log into the Avamar utility node as admin. (Default password is changeme) Load the admin OpenSSH key by entering: ssh-agent bash ssh-add ~admin/.ssh/admin_key (You are prompted to enter the admin passphrase; the default phrase is P3t3rPan) Suspend future cron job activity: suspend_crons The command will also warn about any currently running cron jobs. Any running cp_cron or gc_cron processes must be allowed to complete. A running hfscheck_cron process can be stopped, but stopping it also stops the validation of the associated checkpoint (making that a temporarily unreliable checkpoint for Avamar system rollbacks). Check for running crons and stop hfscheck_cron and/or repl_cron if necessary: To kill hfscheck_cron job: hfscheck_kill To kill replication, locate the avtar processes and kill all process IDs ps -efm | grep _cron kill <pid_of_repl_cron> ps -efm | grep avtar kill <process_id> <process_id> <process_id>

Check for any running cron jobs or if previous cron jobs have completed: ps -efww | grep _cron

When you can verify that no cron processes are running, then it is safe to continue.

Stopping the Avamar software processes Check current software status: dpnctl status

Stop software: dpnctl stop You will be asked if you are sure you want to do this. Type Y to continue with the shutdown. The gsan process may take a while to stop so be patient. Check software status to verify that the gsan / mcs / ems processes are stopped/down: dpnctl status

To Shutdown Linux OS on host systems Switch to the root user. (Default password is changeme) su Load the dpn OpenSSH key and run Linux shutdown: ssh-agent bash ssh-add ~dpn/.ssh/dpnid export SYSPROBEUSER=root mapall --parallel --bg poweroff (remember to use two dashes in front of each option)

This will shut down Linux on the Avamar host system nodes in parallel except for the spare node; KVM output on the systems would show "power down" when the host system is ready to power down. Shutdown the spare node: ssh x root@SPARENODE_IP poweroff

After shutting down the storage and spare nodes, bring down the utility node: poweroff Note: Datastore nodes will not actually power off but will remain at a halted state with power down on the screen. Customer supplied hardware should properly power off. Power down the individual Avamar host systems Once the power off process completes, the Avamar systems and any other components in the configuration can be powered off. (i.e. cabinet power, or any network switches used with the Avamar host systems)

To startup Avamar host systems and software

Startup Linux servers NOTE: Power up any dependant equipment (cabinet power systems, network switches, etc) before powering up the Avamar nodes. Power on all the Avamar host systems and wait a couple minutes for the systems to boot up. Ping the respective node IP addresses from another host to verify the Avamar systems are running. Once the Avamar host systems are up, login to the Avamar utility node as admin and load the admin SSH keys: ssh-agent bash ssh-add ~admin/.ssh/admin_key (You are prompted to enter the admin passphrase; the default phrase is P3t3rPan) Verify the storage nodes are up and can be reached by the utility node: mapall uptime

Verify the spare node has properly started up ssh x admin@SPARENODE_IP uptime

Start the Avamar software Check software status to verify processes are stopped/down: dpnctl status

Start software: dpnctl start You will be asked if you are sure you want to do this. Type Y to continue with the startup. The gsan process may take a while to start so be patient.

After you see the gsan / mcs / ems services startup messages, check the status to verify processes are running/up: dpnctl status

The Avamar system should now be up and running.