Recovering from a TSM 6.

x Crash
By Mark Anderson and David Ashworth
1. Verify that TSM is inaccessible. Wait at least one minute after entering your login before deciding it is hung. If at all possible do the following: a. Issue Can se all command b. Cancel processes c. Ensure mounts are released d. Issue halt command It is unlikely you will be able to do that, but if at all possible do so to speed up recovery. Make sure you are logged into AIX with sudo root privileges. 2. From the AIX command line, type ps eaf | grep dsm. If you see a dsmserv process still running, type kill <PID> (the process ID #). 3. Type df g. Look for the mount /tsmlog. What youre looking for is to see if it is full. Below you can see that in the case of this crash it is not, as it is only 60% full (which is very high, but not the cause of the crash):

4. Go to the tsm home directory (home/tsm02/tsm02) and type db2stop. Has to be done with tsm02 It will either say 'stopped' or something like it cant stop since db2 is still running. If it is stopped, go to step 6.

5. If the db2stop command does not kill db2, type ps eaf | grep db2. Look for a process called db2sysc. Kill the <PID>. Using the below picture as an example, you would type kill 7471140.

6. Check your work. Type ps eaf | grep db2, verify nothing is running. Type ps eaf | grep dsm, verify nothing is running. Su tsmadm Then run this command 7. From AIX type: /opt/tivoli/tsm/server/bin/dsmserv -u tsm02 -i /home/tsm02/tsm02 -quiet & This will start the TSM server. It will come up fairly quickly if there was not a hard lock/crash. You can determine how long it will take to recover with the following. From AIX, type db2pd. You will get a db2pd prompt. From that prompt type -recovery -db tsmdb1 (or whatever your db2 database name is. The one above is for OKCAXTS02) You will see the following:

This tells you the db2 recovery is in progress. It occurs in two phases, forward and backward. Phase one is forward, and takes the longest. You can monitor performance by comparing the Completed Work value to the Total Work Value. In this particular crash it took one hour to recover. The time it takes is totally dependent on how full the log is. If it is at capacity it can take two hours, while if it had only a couple of GB TSM should be up before you can even run this monitoring command. Type q to exit db2pd. Once this recovery is complete TSM will finish loading, and you can log in normally to verify.