Documente Academic
Documente Profesional
Documente Cultură
Page 1
Pa
Klaus-Dieter Krick
July 2016
Global Technology Services
PFE HighEnd Disk Solutions EMEA
IBM GTS IMT Germany, HW Storage Support,
Department 7906, Bldg. 55131-12-1, 5D
LNotes: Klaus-Dieter Krick1/Germany/IBM
eMail:kkrick@de.ibm.com
1
1. Authors comment ............................................................................................................... 3
2. Timestamp PE package ...................................................................................................... 4
3. Dual HMC .......................................................................................................................... 5
4. Data collection for slow and hung HMC, SRC E332FFF, and other CCFW issues .......... 6
5. Data Collection for Call home issues ................................................................................. 7
6. Large PE package ............................................................................................................... 8
7. Timestamp and Information for Statesaves / dumps .......................................................... 9
8. BEB00010 ........................................................................................................................ 10
9. BEB00011 ........................................................................................................................ 12
10. Individual File transfer upload ......................................................................................... 13
11. Content of a PE package important Information ................................................... 14
12. smart menu and issdumps ................................................................................................ 15
13. PE package information ................................................................................................... 16
14. PE package information ................................................................................................... 18
15. PE package and USB 9117-xxxx files ........................................................................ 20
16. USB and modem issue ..................................................................................................... 21
17. 5.1 PE Package Collection with Microcode Levels (75.1x.xx.xx) and above. ................ 22
18. 5.1 Smart Menu ................................................................................................................ 23
19. R7xData iqyylog.log GUI ................................................................................................ 24
20. IQYYLOG.LOG and using command line to upload ................................................. 25
21. BR 5 PE package information GUI .................................................................................. 26
22. BR5.1 PE package missing directory ............................................................................... 28
23. LPAR ODD dumps .......................................................................................................... 29
24. Statesave, forced by customer .......................................................................................... 31
25. DS8K File Manager ........................................................................................................ 32
26. Steps to offload a package with 2107 File manager......................................................... 35
27. Handling errors / microcode acuteness ............................................................................ 37
28. Handling errors 2 / microcode acuteness ......................................................................... 38
29. Modem busy? send queue issue ....................................................................................... 41
30. Incomplete PE package R6, R7 ........................................................................................ 43
31. missing lpar data in PE package ! Tip488 ........................................................................ 44
32. R8 manual pe package to memory stick ........................................................................... 45
33. Rules for filename ............................................................................................................ 45
34. Easy tier ............................................................................................................................ 46
35. DSCLI logs ....................................................................................................................... 47
36. ECUREP EMEA only .................................................................................................... 48
2
1. Authors comment
0B
This document is intended to give some hints for pe packages. It does not substitute
Official documentation.
One Important information is on chapter 8. (Content of a pe package )
If users check the zip file on there laptop and if the content is like described in chapter 8
then you can be almost sure that you have a good pe package.
You will find this book on DS8K Tips Database in tip 13624
http://ssgtech10.tucson.ibm.com/cress/TestDS8K.nsf
Special Thanks to
3
2. Timestamp PE package
1B
PE package on DFS
/.../ssd.sanjose.ibm.com/fs/home/dasdpe/fs148/pe/21079a2/ibm_ta_educatio/7516930/sea.081
211100249.cl9.pepck/21079A27516930.IBM_TA_educatio.666.NOPMH.081211100249.cl9.
pepck.zip
File Name
21079A27516930.Customername.666.NOPMH.081211100249.cl9.pepck.zip
On the HMC:
666 /extra/data/vr/666
081211100249
Year 08
Months 12
Day 11
Hours 10
Minutes 02
Seconds 49
4
3. Dual HMC
2B
Check for HMC configuration and provide PE package from both HMC.
lsrsrc IBM.EssHMC
Note: The service trace, or in other words the repair trace is the logfile where we can check
which actions have been performed by CE.
This service trace is located only on that HMC where the CE did the service action.
5
4. Data collection for slow and hung HMC, SRC E332FFF,
3B
There are certain problems with the HMC that can not be diagnosed with a PE Package. A tool needs
to run to collect additional log information to allow further analysis of these problems
6
5. Data Collection for Call home issues
There is blog posted by Carsten Haag,the content is like below.
See following link going to this blog.
https://ibm.biz/Bd4mLS
To address Call Home issues on recent code-levels please make sure to collect the proper
data.
Call Home problems, like
1. generic PE Package
2. HMC Debug Info
3. iqyylog.log
For items 1. and 2. you may refer to the following blog to see how this data is collected :
Item 3. can be collected either via the WUI or via SMART, individual file upload
/var/hsc/log/iqyylog.log
When using SMART, please look into TIP 13624 for the right process.
7
6. Large PE package
3B
Please contact Front End Support Center and ask for cleanup.
Front End member should then perform following procedure, available on
Squadrones-S DB (Lotus Notes Data Base)
Notice: Support Center can use /opt/esshmc/bin/smart . This ignores large files
and support center can check later on DFS which file is to large ( unpack l )
Notice: it is possible to send separated files to testcase , however you must document this in
PMH. Otherwise PMH might come back with comment like: damaged PE package, no valid
pe package available
Best practice to cleanup.
8
7. Timestamp and Information for Statesaves / dumps
4B
cpssdump01.081211_072331.1.1.tar
Year 08
Months 12
Day 11
___________
Hours 07
Minutes 23
Seconds 31
1.0 LPAR01
1.1 LPAR11
2.0 LPAR02
2.1 LAPR12
Notice: Dumps wrap around. This means , not the highest cpssdump number must be
necessarily the newest dump.
Notice: Dumps from different LPAR may have different numbers, even dumps are from
same time
example : cpssdump01.081211_072331.1.1.tar
cpssdump07.081211_072331.1.0.tar
9
8. BEB00010
5B
10
The following messages appears. Keep in mind that the zip file in our
Example 666 from the directory /extra/data/vr/666 will be listed
11
9. BEB00011
Problem:
When a PE package via the WebSM GUI is created, the sub-packages that are not selected for
offload will generate a Serviceable Event BEB00011. This SE will not automatically call home the
data, and is only intended for later manual offload when directed to do so by development.
For primary data collection, do not use the Serviceable Event with SRC BEB00011.
Code Bundle has introduced the new Serviceable Event BEB0 0011.
Example:
Upon pressing the Offload button, one serviceable event with SRC BEB00010 is created for the
selected packages.
When "Cancel" is pressed all packages that have not been previously selected and off-loaded will
create serviceable event BEB00011 for later offload if request by development.
12
10. Individual File transfer upload
Example:
Example:
210793175KB110.Customer_name.24.NOPMH.100226094208.cl8.pefil.zip
Support might store individual files in following directories, they are listed in Gui.
/extra/pepkg
/var/log
13
11. Content of a PE package
7B important Information
Please send pe packages and dumps only to testcase.boulder.ibm.com
/ssd/toibm/sharkdumps
the files below are zipped in the zip file from pe package.
In our example
666.zip
CasTrace
collectedCodeLevels.All.2107-9A2-7516930.htm
collectedCodeLevels.All.8676GHX-KKPVR1N.htm
ctsnap.SF7516930ESS01.12110953.tar.gz
ctsnap.SF7516930ESS11.12110957.tar.gz
ctsnap.taeduc1.12110952.tar.gz
HMCa.unzipthis
Lpar0.tar.gz
Lpar1.tar.gz
SFI-1.info
then please unzip this file on your laptop and upload the xxx.zip file,where xxx is
a number from problem event BEB00010
14
12. smart menu and issdumps
This tip is for support center.
ERROR:
The files appear to be on different SFIs...
This should not happen for DA pair StateSaves.
It is likely that these are not partner dumps.
Please return to the DA Statesave offload and select again.
If you select now all dumps you run into same problem.
21) Offload ALL these ISS DA dumps at once
Solution
Squadrones Database Document
How to generate/offload iss_looptatesaves
You can tar the files and upload them via smart and individual
File transfer
15
13. R1 PE package information
8B
Service Focal Point -> service utilities" -> "storage facilities" , then "selected" -> "get
storage facility images" -> "storage facilities" , then "selected"
Perform full Data Collection On Demand DCOD
16
If Auto offload check box is not selected ,then you must select all files
as indicated below.
Notice: This applies for R1 . Different on R3 and 4
CasTrace
collectedCodeLevels.All.2107-9A2-7516930.htm
collectedCodeLevels.All.8676GHX-KKPVR1N.htm << HMC data
ctsnap.SF7516930ESS01.12110953.tar.gz << LPAR data
ctsnap.SF7516930ESS11.12110957.tar.gz << LPAR data
ctsnap.taeduc1.12110952.tar.gz
HMCa.unzipthis << HMC data
Lpar0.tar.gz << LPAR data
Lpar1.tar.gz << LAPR data
SFI-1.info
17
14. R4 PE package information
9B
Notice:
PE package is within the description.xxx.zip file where xxx is the problem number from
BEB00010
18
Please select only files with "text in xxx.zip"
Notice: if all files are selected then a eed file will be generated and you might get
unpredictable results.
Just select one of the xxx in 231.zip (from example) will be enough
21079227520780.IBM_ATS_Mainz.231.NOPMH.090106101224.cl9.eed.zip
911757075AD2A0.IBM_ATS_Mainz.23.NOPMH.090309102503.cl9.pepck.zip
21079227520780.IBM_ATS_Mainz.231.NOPMH.090107101000.cl9. pepck.zip
19
15. R4 PE package and USB
10B 9117-xxxx files
On R4 machines we have another issue with USB.
Scanario.
Scenario:
CE is using FTP Manager to upload the file.
Since the type 9117 is not the right machine type, 2107 or 242x ftp manager will rename
the files to the correct one. You will get following message
Solution:
Unzip the file:
911757075AD2A0.IBM_ATS_Mainz.23.NOPMH.090309102503.cl9.pepck.zip
Then use FTP manager Program to rename the file
Use only this file >>>>>>>>> xxx.zip where xxx is any number .In our example it is 23
23 example: 23.zip
21079227520780.IBM_ATS_Mainz.23.NOPMH.090309102503.cl9.pepck.zip
20
16. USB and modem issue
There have been some cases identified where USB port on a Laptop HMC was used to save
data to
an USB stick where afterward the modem was in a hung condition.
It is a good idea to check modem function by doing a test call or to reboot HMC.
Make sure modem is working when dealing with this USB stuff.
21
17. R 5.1 PE Package Collection with Microcode Levels
1
If you have a copy services, dscli, GUI, TPC, or essni related issue, then you need to collect the
Customer User Interface Data Package. For all other problems, you need to collect a General PE
Package. In most cases, the problem type should easily determine which PE Package piece is the
right set of log files for problem determination.
Please note that these two pieces of the PE package are very different log files. If you cannot
determine the area of the machine that has the problem, then you will need to collect both pieces of
the PE Package to have a 'Full' PE Package. Regardless of which piece you select, both pieces are
actually prepared for offload; however, by default, only the piece you selected will be offloaded. The
other piece is there and the directions for offloading are provided on the screen. Or, you can repeat
the PE Package collection process and select the other piece.
The General PE Package will offload as a 'pepck' file type. The Customer User Interface Data
Package will offload as a 'pefil' file type.
It is very important to collect the right PE Package data on the first attempt; otherwise, significant delay
can result as the correct data will have to be collected.
The Smart tool for collecting a PE package was changed to allow the choice of which PE Package
Piece. To use the Smart tool, enter 'smart' at the hmc root command line. (Some people have
observed a message that smart is already running; this is an expected message as the smart tool
periodically polls for some information. Just enter the command (smart) a second time).
22
18. R 5.1 Smart Menu
Please select from the following options:
1) Exit SMART
2) Display the configuration of this Storage System
3) Data Collection and Offload Tools
4) View Configuration Files and Log Files
5) Diagnostic Tools
6) Recovery Tools
7) MicroCode and Firmware Tools
Selection (Hit Return to see choices) >
Choose option #3 for Data Collection and Offload Tools and you will see the following Menu of
choices:
Choose option #2 and you will see the option to choose which PE Package Piece that you would like
to collect:
Select 'General PEPackage' if you are collecting data for any service actions
including install/removal, MES, repair, code load and other actions. This package
contains data from RAS and functional components not including state saves.
Select 'Customer User Interface Data Package' if you are collecting data for problems encountered
with customer user interface applications such as DSGUI or TPC. This package contains data from
various software components running on the HMC such as DSGUI, CIM Agent, and ESSNI.
NOTE: DSCLI and TPC traces are not on the HMC
Option #3 will collect the Customer User Interface Data Package. This is the option you need to select
for copy services, dscli, GUI, TPC, and essni related problems.
Option #2 will collect the General PE Package for all other problem types.
If you connect via VPN and use the HMC GUI (Service Panels), the new R5.1 options exist with the
PE Package data collection
23
19. R 5 R7xData iqyylog.log GUI
24
20. IQYYLOG.LOG and using command line to upload
Check
#ls ltr /extra/phmcDebug
drwxrwxr-x 2 ccfw ccfw 4096 2014-04-03 10:53 20140403.1053
drwxrwxr-x 2 ccfw ccfw 4096 2014-04-04 10:54 20140404.1054
drwxrwxr-x 2 ccfw ccfw 4096 2014-07-17 16:44 20140717.1644
drwxrwxr-x 2 ccfw ccfw 4096 2014-07-17 17:05 20140717.1705 <<<<<< example
#cd 20140717.1705
# ls
#HMClogs.taeduc10717H58.pacman_no_unzip
/extra/phmcDebug/YYYYMMDD.TTTT/HMClogs.CUSTOMER.pacman_no_unzip
# rsSendFiles
Getting list of SFIs...
Enter filenames one line at a time. Use their full pathnames.
Type 'done' when finished: /extra/phmcDebug/20140717.1705/HMClogs.taeduc10717H58.pacman_no_unzip
Type 'done' when finished: done
Done (async) logging MM serviceable event.
On DFS
Files are then in sub directory tmp and file name is like
hmcdebuginfo.customer.20140717.165857.tgz
Note: If you do manual upload the file, rename the file on your laptop before
you use the FTP File Manager like:
HMClogs.taeduc10717H58.pacman_no_unzip
new
HMClogs.taeduc10717H58.pacman_no_unzip.zip
25
21. R 5 PE package information GUI
12B
Different to R4
26
ESSNI
Device Object Mismatch (BE340021)
Panics
Failed Service Action (Repair CDA)
CDA
Logical Configuration Errors
Machine down ( HMC still up )
FCIC Loop Issues
MES Power / Cache
MES Storage Enclsoure
MES Stoarge DA
hmcdebuginfo
27
22. R5.1 PE package missing directory
13B
CMVC
This is caused by running the following script live on the box to format the cda traces:
/opt/esshmc/bin/cdaFormat.sh
If you run this script , then remove always following files in /var/esshmc/log directory
cdaTraceBkpParallel*
cdaTraceParallel*
28
23. LPAR ODD dumps
This feature has been implemented and will be released in R6.1 June 2011
Starting with R6.1 we have the possibility to capture most of the machine data in a
Statesave-dump without forcing a warmstart.
This will avoid facing the known side-effects of a statesave, like timeouts, PPRC
suspends, etc.
These dumps can not be initiated / uploaded via smart menu at this time
It appears that these dumps are not the appropriate tool to collect FB LUN information or
Adapter traces. There is a limited copy services information included.
29
It is not possible to upload those odd dumps native via smart, you have to use individual
File upload
Example:
On the HMC
Hmca#cd /tmp
Hmca#mmhmcshowlparip
Hmca#ftp 172.16.1.101
output like:
example:
local: cpssdump00_ODDdump.120402_214502.1.0.tar.gz remote:
cpssdump00_ODDdump.120402_214502.1.0.tar.gz
200 PORT command successful.
150 Opening data connection for cpssdump00_ODDdump.120402_214502.1.0.tar.gz (18564753 bytes).
226 Transfer complete.
18564753 bytes received in 1.6 seconds (1.1e+04 Kbytes/s)
ftp> quit
Then use smart menu to upload those files (remove them from /tmp afterward
Hmca#/opt/esshmc/bin/smart
3) Data Collection and Offload Tools
9) Offload a single file (Interactive)
30
24. Statesave, forced by customer
Customer can generate statesave by using dscli
Example
dscli> diagsi -action warmstart IBM.2107-7516931
Date/Time: May 25, 2012 1:33:37 AM CDT IBM DSCLI Version: 5.4.21.280 DS:
IBM.2107-7516931
CMUD00023W diagsi: Are you sure you want to perform diagnostic control
warmstart? [y/n]:y
CMUD00026I diagsi: Diagnostic control warmstart is successfully submitted.
It will take some time for the system to complete the request.
31
25. DS8K File Manager
DS8K FTP File Program on DS8K Tips database tip 13073
32
Use windows explorer and search for java.exe
33
Modify the command then in following way
and run it
34
26. Steps to offload a package with 2107 File manager.
15B
This would be the first step and before you use to upload the file(s)
35
Assuming the files are already on a DVD or a Memory Stick
1.) insert the DVD or the Memory Stick into your laptop
2.) copy the file(s) into the directory. (default directory c:\2107ftp
3.) Push button send files
Select the file you want example 666.zip * important Notice
Note: only one files at a time
Note: If it is not in the right format you will be prompted to enter the right
information.
Messages appear.
This File will not be recognized by Testcase. Do you want to modify the file name? Yes
click:
Enter the 3-digit machine Model. Example: 9A2 OK
Enter the customersname. Example: Customer _ Gandalf OK
Enter the Problem number Example: 666
Note: Please use the one from Problem ID BEB00010 OK
Enter PMH # or NOPMH Example: NOPMH OK
Select PE package type.
Note: Important, select the right type:
pepck for pe package
pefil individual File transfer
sslpr for Statesave files,
ssphp for Phypdump
ssiss for issdump
dump for AIX dump
ssddm for DDM Statesaves
dumptrace if somebody can give me the answer then feel free to inform me.
You might then test the zip file : Hit the button test zip file
Select then the file you have created and hit the button send selected zip files
Note: you might create more files in the same schematic as it described on
previous steps and send it then in one step.
36
*Notice: If you have a filename like:
911757075AD2A0.23.NOPMH.090309110532.cl9.pepck.zip, then unzip the file and use the
xxx.zip where xxx is any number to offload the package via FTP Manager.
A CE used USB stick and used 2107FTP manager to upload the files.
He transferred the files to his Laptop and then he selected all the files
(9 files) and used FTP Manager to upload all of them in one
zip File *.
Problem.
DFS was not able to extract the files.
Solution.
Select only one file if using FTP Manager. In the example below see the file marked with blue
color.
Notice: the xxx.zip file is usually the pe package and there is no need to upload the other files.
Where xxx is any number between 1 and 999. Usually the problemnumber from
Service Event BEB0010
File *
210793175ABCD0.Customer.1.80322.081212091203.cl8.pepck.zip
/pepackage
82190436 Thu Dec 11 18:41:26 MST 2008 pepackage/22.zip
13415 Thu Dec 11 18:41:26 MST 2008 pepackage/actzuict.dat
4644 Thu Dec 11 18:41:26 MST 2008 pepackage/iqyvpd.dat
77 Thu Dec 11 18:41:26 MST 2008 pepackage/iqyvpdc.dat
4040029 Thu Dec 11 18:41:26 MST 2008 pepackage/iqyycom0.log
4055940 Thu Dec 11 18:41:26 MST 2008 pepackage/iqyycom1.log
4067290 Thu Dec 11 18:41:26 MST 2008 pepackage/iqyycom2.log
2637783 Thu Dec 11 18:41:26 MST 2008 pepackage/iqyylog.log
118945 Thu Dec 11 18:41:26 MST 2008 pepackage/problems.xml
37
28. Handling errors 2 / microcode acuteness
17B
21079A27516930.IBM_TA_educatio.666.01237.081212154535.cl9.pepck.zip
was created.
Unfortunately did the zip file contain just the following file: iqyycom2.log
How to fix it ?
1.) Please select just one of the files listed below.
xxx.zip where xxx is the problem number from BEB0010
Notice:
We have had another case with same sign and we have got a huge eed file with all
selected files in it.
38
That one was also not processable on DFS. This implies that we get unpredictable results if
we select all files.
39
The following message apeares for each file .
In our example 11 times
Dont worry
# ls -l /media/cdrom
total 162286
21079A27516930.IBM_TA_educatio.666.01237.081212154410.cl9.eed.zip
21079A27516930.IBM_TA_educatio.666.01237.081212154528.cl9.eed.zip
21079A27516930.IBM_TA_educatio.666.01237.081212154529.cl9.eed.zip
21079A27516930.IBM_TA_educatio.666.01237.081212154530.cl9.eed.zip
21079A27516930.IBM_TA_educatio.666.01237.081212154531.cl9.eed.zip
21079A27516930.IBM_TA_educatio.666.01237.081212154532.cl9.eed.zip
21079A27516930.IBM_TA_educatio.666.01237.081212154534.cl9.eed.zip
21079A27516930.IBM_TA_educatio.666.01237.081212154535.cl9.eed.zip
40
29. Modem busy? send queue issue
18B
Take this route (checked), so you are sure ALL Dont use this route as you might forget or fail to
needed data will be selected. select all needed data
Both options will ALWAYS have the PE-package being send via the Modem (remember no
VPN nor FTP).
So you MUST always clear the RSF queue afterwards, otherwise the modem will be
occupied for hours / days.
With the check or uncheck, you only select if you automatically have all PE-package data
offloaded (preferred), or if you want to make a selection yourself (not recommended, as this
could lead to incomplete PE-package data).
41
Keep in mind, a busy phone line prevents that support center
is able to dial into the machine at that time, respectively the
machine can not transmit service information for this time period.
Clearing the modem send queue will free up the line, so the support
center representative can immediately start online analyze, whilst
loading data to IBM.
42
30. Incomplete PE package R6, R7
PE Package generation does not collect all files due to a microcode problem. Reboot of the HMC or
generating a second PE Package will not correct the problem, this is a microcode error.
Actions:
Tip 13020
43
31. missing lpar data in PE package ! Tip488
19
44
32. R8 manual pe package to memory stick
1.) You can use a SDHC card to transfer pe package to your Laptop
2.) If no SDHC card available, remove the SDHC card from drive and insert
A USB memory into slot (2) . The machine will store then package on USB port drive.
Re-insert original SDHC into media slot (3) after this service action.
Preferred method is to use the 2107 FTP manager program , however if it is not available you
HU UH
XXXXYYYCCSSSSS.CUSTOMERNAME.NOPMH.yymmdd.clH.pepck.zip
where
45
NOPMH NOPMH Note: use uppercase
YYMM Date Year Months Day 01212010
January twenty-first 2010
clH cl8 for HMC1
cl9 for HMC2
pepck.zip details
HU U
Note:
Testcase.boulder.ibm.com or
testcase-blue.boulder.ibm.com *
ssd/toibm/sharkdumps
HU ftp://testcase.boulder.ibm.com/ssd/toibm/sharkdumps/ UH
Host Data like snap, drawings, customer mail and such things.
For EMEA only should be send to ECUREP. See also DASD Cookbook for
details. Other countries may have different FTP server for host data
*
Note: alternatively you might use testcase-blue.boulder.ibm.com
There has been a problem identifies sending files to testcase if you use Global Network
Dialer. It will not work from home for instance.
46
35. DSCLI logs
DSCLI logs can now be transfered to HMC and data collection can be made then via normal
procedure.
Upload then user interface log pe package
saveuilogs
47
The -action saveuilogs parameter offloads DS CLI logs to the Hardware Management
Console and saves DS Network Interface, DS CLI, DS Storage Manager, and CIM
logs. All of the user interface log files can then be retrieved with a PE package.
The ECuRep solution provides full life cycle management support for handling customer
problem documentation data. ECuRep helps the IBM support center personnel and IBM
customers to exchange problem documentation via a FTP server or email. The received data
48
get aligned to a specific RETAIN record and are stored in a centralized datastore which
provides common access to all IBM support. Automated RETAIN record updates will inform
the support team about the availability of the data. For selected platforms direct attached
debug systems are available which allow immediate analysis of the data.
HU http://itcenter.mainz.de.ibm.com/ecurep/emea.html U
testcase.boulder.ibm.com
49
50