Sunteți pe pagina 1din 31

SERVICE NOW- HOW TO UPDATE EXECUTIVE SUMMARY

=============================================

Issue Description : KIN Server Down


Observed Business Impact: RMU's, Punchtools, Layway are down
Observed Member Impact:: Store
Affected Business Unit(s): Store
Restoration Details: Bypassed UPS / Powered on KIN via BX / Restored RAID /
Booted KIN with F6 to accept array rearrangement / Rebooted KIN server / Restore
files, directories from Tape / Complete Restoration done from tape / Hotbox
replaced
Alerting:K_Server was notified by an automated monitoring alert / User Raised
Ticket

Issue Description : KIN DB Down


Observed Business Impact:
Observed Member Impact:: Store
Affected Business Unit(s): Store
Restoration Details: Rebooted KIN server / Replaced Processor card
Alerting:User Raised Ticket

Issue Description : Virtual KIN Server Down


Observed Business Impact: RMU's, Punchtools, Layway are down
Observed Member Impact:: Store
Affected Business Unit(s): Store
Restoration Details: Smith, Terence fixed the issue (started the KIN server in
VMware)
Alerting:K_Server was notified by an automated monitoring alert

Issue Description : UPS Down


Observed Business Impact: No battery backup for V-KIN Server (RMU's, Punchtools,
Layway will down)
Observed Member Impact:: Store
Affected Business Unit(s): Store
Restoration Details: New UPS installed for V_KIN
Alerting:K_Server was notified from V-KIN Server down alert

********************************************************************************

Issue Description : RX server Down


Observed Business Impact: Pharmacist not be able fill prescription
Observed Member Impact: Pharmacists
Affected Business Unit(s): Pharmacy
Restoration Details: Bypassed UPS and powered on RX server.
Alerting: K_Server was notified by an automated monitoring alert / User Raised
Ticket

Issue Description : RX System is thrashing -Pageln 13840:00 PageOut: 2739.00


Observed Business Impact: May be degrade the performance
Observed Member Impact: Pharmacists
Affected Business Unit(s): Pharmacy
Restoration Details: PageIN & PageOUT ratio came down normal
Alerting: K_Server was notified by an automated monitoring alert

Issue Description : PDX Down due to upgrade Failure


Observed Business Impact: Pharmacist not be able fill prescription
Observed Member Impact: Pharmacists
Affected Business Unit(s): Pharmacy
Restoration Details: Restored PDX from tape backup.
Alerting: User raised a ticket.

Issue Description : CPU Workload High. CPU Aggregate -1 is 99.99% busy in RX / IVR
Observed Business Impact: May be degrade the performance
Observed Member Impact: Pharmacists
Affected Business Unit(s): Pharmacy
Restoration Details: CPU utilization came down automatically and its normal now /
Rebooted RX
Alerting: Netcool Alert

********************************************************************************

Issue Description : IVR Server Down


Observed Business Impact: IVR is not available, No Automated Refill available.
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Bypassed UPS / Powered on via ILO / Replaced systemboard /
Replace Dialogic Card
Alerting: K_Server was notified by an automated monitoring alert. / User Raised
Ticket

Issue Description : IVR System -Too many defunct/zombie processes


Observed Business Impact: May be degrade the performance
Observed Member Impact: Pharmacists
Affected Business Unit(s): Pharmacy
Restoration Details: Killed Zombie / Defunct processes & Restarted the SEA agent
Alerting: Proactive alert

===================================================================================
================================
Issue Description : Daily backup not performed on Evault.
Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: L3 has Reinstalled the agent and Re-registered, sync'd and
updated job.
Alerting: Evault Portal alert

Issue Description : Hard drive failure in KIN / RX / IVR


Observed Business Impact: None
Observed Member Impact: NoneSERVICE NOW- HOW TO UPDATE EXECUTIVE SUMMARY
=============================================

Issue Description : KIN Server Down


Observed Business Impact: RMU's, Punchtools, Layway are down
Observed Member Impact:: Store
Affected Business Unit(s): Store
Restoration Details: Bypassed UPS / Powered on KIN via BX / Restored RAID /
Booted KIN with F6 to accept array rearrangement / Rebooted KIN server / Restore
files, directories from Tape / Complete Restoration done from tape / Hotbox
replaced
Alerting:K_Server was notified by an automated monitoring alert / User Raised
Ticket
SERVICE NOW- HOW TO UPDATE EXECUTIVE SUMMARY
=============================================
Issue Description : KIN Server Down
Observed Business Impact: RMU's, Punchtools, Layway are down
Observed Member Impact:: Store
Affected Business Unit(s): Store
Restoration Details: Bypassed UPS / Powered on KIN via BX / Restored RAID /
Booted KIN with F6 to accept array rearrangement / Rebooted KIN server / Restore
files, directories from Tape / Complete Restoration done from tape / Hotbox
replaced
Alerting:K_Server was notified by an automated monitoring alert / User Raised
Ticket

Issue Description : KIN DB Down


Observed Business Impact:
Observed Member Impact:: Store
Affected Business Unit(s): Store
Restoration Details: Rebooted KIN server / Replaced Processor card
Alerting:User Raised Ticket

Issue Description : Virtual KIN Server Down


Observed Business Impact: RMU's, Punchtools, Layway are down
Observed Member Impact:: Store
Affected Business Unit(s): Store
Restoration Details: Smith, Terence fixed the issue (started the KIN server in
VMware)
Alerting:K_Server was notified by an automated monitoring alert

Issue Description : UPS Down


Observed Business Impact: No battery backup for V-KIN Server (RMU's, Punchtools,
Layway will down)
Observed Member Impact:: Store
Affected Business Unit(s): Store
Restoration Details: New UPS installed for V_KIN
Alerting:K_Server was notified from V-KIN Server down alert

********************************************************************************

Issue Description : RX server Down


Observed Business Impact: Pharmacist not be able fill prescription
Observed Member Impact: Pharmacists
Affected Business Unit(s): Pharmacy
Restoration Details: Bypassed UPS and powered on RX server.
Alerting: K_Server was notified by an automated monitoring alert / User Raised
Ticket

Issue Description : RX System is thrashing -Pageln 13840:00 PageOut: 2739.00


Observed Business Impact: May be degrade the performance
Observed Member Impact: Pharmacists
Affected Business Unit(s): Pharmacy
Restoration Details: PageIN & PageOUT ratio came down normal
Alerting: K_Server was notified by an automated monitoring alert

Issue Description : PDX Down due to upgrade Failure


Observed Business Impact: Pharmacist not be able fill prescription
Observed Member Impact: Pharmacists
Affected Business Unit(s): Pharmacy
Restoration Details: Restored PDX from tape backup.
Alerting: User raised a ticket.
Issue Description : CPU Workload High. CPU Aggregate -1 is 99.99% busy in RX / IVR
Observed Business Impact: May be degrade the performance
Observed Member Impact: Pharmacists
Affected Business Unit(s): Pharmacy
Restoration Details: CPU utilization came down automatically and its normal now /
Rebooted RX
Alerting: Netcool Alert

********************************************************************************

Issue Description : IVR Server Down


Observed Business Impact: IVR is not available, No Automated Refill available.
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Bypassed UPS / Powered on via ILO / Replaced systemboard /
Replace Dialogic Card
Alerting: K_Server was notified by an automated monitoring alert. / User Raised
Ticket

Issue Description : IVR System -Too many defunct/zombie processes


Observed Business Impact: May be degrade the performance
Observed Member Impact: Pharmacists
Affected Business Unit(s): Pharmacy
Restoration Details: Killed Zombie / Defunct processes & Restarted the SEA agent
Alerting: Proactive alert

===================================================================================
================================
Issue Description : Daily backup not performed on Evault.
Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: L3 has Reinstalled the agent and Re-registered, sync'd and
updated job.
Alerting: Evault Portal alert

Issue Description : Hard drive failure in KIN / RX / IVR


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy / Store
Restoration Details: Replaced defective hard drive
Alerting: K_Server was notified by an automated monitoring alert. / User Raised
Ticket

Issue Description : Tape backup issue in RX server


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Replaced Tape drive / Cleaned Tape drive / Replaced
systemboard, scsi cable, cage etc
Alerting:K_Server was notified by an automated monitoring alert. / User Raised
Ticket

Issue Description : RX / IVR server not reporting in SRM


Observed Business Impact: Server not reporting in SRM
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Restarted the SRM agents
Alerting: K_Server was notified by an Monitoring team

Issue Description : ITM Monitoring Offline Agent Restart - RX / IVR server


Observed Business Impact: Unable to monitoring the server
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Restarted the ITM agents
Alerting: K_Server was notified by an Monitoring team

===================================================================================
======

Hi Terry,

The store having new KIN server (Dell - 430)


Spoke with Office Associate - Pamela Shaffer,
She said this morning VKIN was hung, hence they rebooted...
After reboot VKIN session not invoking

Please do the needful.

INC0363236

Tivoli team
Eduardo Manenti De Araujo XXX
Fabio Antonio Baldim XXX
Dipsmita Das2/India/IBM
Isaque Benatti Souza

=====================

Knowledge article KB0022536:

Symptoms
Symptoms

a. Procedure to install the new V-KIN server.

b. Procedure for V-KIN conversion.

c. Procedure to return the old KIN, Digi, monitor and tapes

Issue Description : PDX Down due to upgrade Failure


Observed Business Impact: Pharmacist not be able fill prescription
Observed Member Impact: Pharmacists
Affected Business Unit(s): Pharmacy
Restoration Details: Restored PDX from tape backup.
Alerting: User raised a ticket.

Issue Description : Unable to Archive backup.


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Backup Archived to IVR
Alerting:Netcool Alert / User Raised Ticket / None
Information:
1. The KIN conversion is to be performed two business days after the new V-KIN
server was installed.Hardware Deployment will call store the day before conversion
to confirm the conversion will take place the next morning.

2. Click here for the roll out schedule

3. The servers are configured at SHI, this includes imaging and installing a USB
flash drive for both the server and NAS.

4. The existing KIN server is on IP .202, after the conversion the new VKIN server
will be on .202

CauseSymptoms

a. Procedure to install the new V-KIN server.

b. Procedure for V-KIN conversion.

c. Procedure to return the old KIN, Digi, monitor and tapes

Issue Description : PDX Down due to upgrade Failure


Observed Business Impact: Pharmacist not be able fill prescription
Observed Member Impact: Pharmacists
Affected Business Unit(s): Pharmacy
Restoration Details: Restored PDX from tape backup.
Alerting: User raised a ticket.

Issue Description : Unable to Archive backup.


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Backup Archived to IVR
Alerting:Netcool Alert / User Raised Ticket / None
Information:

1. The KIN conversion is to be performed two business days after the new V-KIN
server was installed.Hardware Deployment will call store the day before conversion
to confirm the conversion will take place the next morning.

2. Click here for the roll out schedule

3. The servers are configured at SHI, this includes imaging and installing a USB
flash drive for both the server and NAS.

4. The existing KIN server is on IP .202, after the conversion the new VKIN server
will be on .202

CauseSymptoms

a. Procedure to install the new V-KIN server.

b. Procedure for V-KIN conversion.

c. Procedure to return the old KIN, Digi, monitor and tapes

Issue Description : PDX Down due to upgrade Failure


Observed Business Impact: Pharmacist not be able fill prescription
Observed Member Impact: Pharmacists
Affected Business Unit(s): Pharmacy
Restoration Details: Restored PDX from tape backup.
Alerting: User raised a ticket.

Issue Description : Unable to Archive backup.


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Backup Archived to IVR
Alerting:Netcool Alert / User Raised Ticket / None
Information:

1. The KIN conversion is to be performed two business days after the new V-KIN
server was installed.Hardware Deployment will call store the day before conversion
to confirm the conversion will take place the next morning.

2. Click here for the roll out schedule

3. The servers are configured at SHI, this includes imaging and installing a USB
flash drive for both the server and NAS.

4. The existing KIN server is on IP .202, after the conversion the new VKIN server
will be on .202

CauseSymptoms

a. Procedure to install the new V-KIN server.

b. Procedure for V-KIN conversion.

c. Procedure to return the old KIN, Digi, monitor and tapes

Issue Description : PDX Down due to upgrade Failure


Observed Business Impact: Pharmacist not be able fill prescription
Observed Member Impact: Pharmacists
Affected Business Unit(s): Pharmacy
Restoration Details: Restored PDX from tape backup.
Alerting: User raised a ticket.

Issue Description : Unable to Archive backup.


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Backup Archived to IVR
Alerting:Netcool Alert / User Raised Ticket / None
Information:

1. The KIN conversion is to be performed two business days after the new V-KIN
server was installed.Hardware Deployment will call store the day before conversion
to confirm the conversion will take place the next morning.

2. Click here for the roll out schedule

3. The servers are configured at SHI, this includes imaging and installing a USB
flash drive for both the server and NAS.
https://searshc.sysaidit.com/ShowFAQ.jsp?print=true&articleId=9...
Kmart Retail > Resolutions
New HP Restore Process For Pharmacy
Servers
Information
This article contains the updated procedure to handle an Rx Server Restore
Procedure with the new network backup
process.
Symptoms
a. The Rx server is not performing as expected.
Cause
a. Certain hardware errors like hardware replacement can require a server restore.
Resolution
Service Desk Resolution
Follow these steps to fix the issue
1. If a full restore is required, please assign the incident to IBM_IR_STASERVER.
L2/L3 Resolution
Follow these steps to fix the issue
NOTE 1: Whenever possible, BEFORE doing a restore, you need to save a copy of the
rxque and the phrx files.
This is the Multiple RX Queue file and the phone doctor files. These files contain
multiple fill and e-prescribing refill
requests that have not been processed. If this files aren’t saved, the information
will be lost. This means a customer
may come in for their prescription and the Pharmacist won't have the refill ready
because the information was lost. The
information can be sent to KIN. For details, see Article #515.
1. If you can get online with the server prior to starting the restore, you must
perform the following steps:
A. Save the rxque, phrx and phrxhist file (multi-fill and phone doctory queue). For
details, see solution Article
#515.
B. Generate the Re-Key Data, generate the Point of sale (POS) activity,
comprehensive, 3rd party plan analysis
and multi-fill queue reports. For details, see solution Article #1525 prior to
starting the restoration.
1 of 13
7/9/19, 11:07 PMhttps://searshc.sysaidit.com/ShowFAQ.jsp?print=true&articleId=9...
Starting the PXE Boot process to initiate the restore
2. Reboot the RX server. As the system is going through the restart, ask the
pharmacist to monitor the messages on the
screen and once they see the screen display “Press Ctr+S to Enter Configuration
Menu ...”, began to hold down
the F12 key to PXE boot the serv.
RESULT: The pharmacy will see the system come up to the following screen:
3. At the “boot:” prompt, ask the pharmacist to type: rebuild and press <Enter>
2 of 13
7/9/19, 11:07 PMhttps://searshc.sysaidit.com/ShowFAQ.jsp?print=true&articleId=9...
RESULT: It will then begin to wipe the hard drives and prep the system to do a
restore.
4. When it’s done, it will reboot and come up with the Kmart Pharmacy PXE Restore
screen and the pharmacist will be
prompted to enter the ARMS code for the day.
5. Determine the correct ARMS code and then ask the pharmacist to type the ARMS
code for today on the screen and
press Enter. For details on how to determine the ARMS code of the day, see Article
#672.
RESULT: You will see the screen below with 4 options.
3 of 13
7/9/19, 11:07 PMhttps://searshc.sysaidit.com/ShowFAQ.jsp?print=true&articleId=9...
6. Ask the pharmacist to Choose option '2 – Restore tape and Re-boot the system'
and press Enter.
RESULT: The pharmacist will be prompted to insert the PDX Backup tape into the tape
drive.
7. Ask the pharmacist to insert one of the old backup tapes (they should be stored
in the C2 cabinet) into the tape drive
and then press Enter.
RESULT: A screen to above picture will display. The tape label will be displayed;
please make sure that the tape
label reflects the correct date/time of the data that you want to restore to the
system.
8. Ask the pharmacist to type RESTORE and press Enter and the system will begin to
restore the tape and then reboot
the system when complete.
4 of 13
7/9/19, 11:07 PMhttps://searshc.sysaidit.com/ShowFAQ.jsp?print=true&articleId=9...
NOTE: After the reboot, the system should recover and the pharmacist will get the
login prompt. Do not allow the
pharmacist to login at this time. You will now need to re-install the Evault
Network backup software so you can restore
the latest data files to the system.
9. Get online with the RX server as root.
10. Type: cd /tmp.
11. Search the /tmp folder to see if the Evault backup software install package is
present. Type: ls -lt Agent-Linux-
8.60.7471.tar.gz kmportal.txt <Enter>
Result:
rx0851:/tmp # ls -lt Agent-Linux-8.60.7471.tar.gz
-rw-r--r-- 1 root root 17079510 Dec 31 17:25 Agent-Linux-8.60.7471.tar.gz
-rw-r--r-- 1 root root 95 Nov 29 16:17 kmportal.txt
rx0851:/tmp #
a. If the install package and .txt file are present, go to Step 12.
b. If the install package and .txt are not present in /tmp, you can copy them from
the store's rxivr
server at the store doing the following:
scp root@rxivr:/convert/kmportal.txt /tmp
scp root@rxivr:/convert/Agent-Linux-8.60.7471.tar.gz /tmp
RESULT: Once the files have been copied to the system, you can install the
software.
Go to Step 12.
12. Confirm that you are still in the /tmp directory. Type the following command:
tar -zxf Agent-Linux-8.60.7471.tar.gz
<Enter> to extract files from the installation package. The screen will return to a
prompt after roughly 30 secs once the
5 of 13
7/9/19, 11:07 PMhttps://searshc.sysaidit.com/ShowFAQ.jsp?print=true&articleId=9...
install files have been extracted.
13. Type: cd Agent* <Enter> You will be in the Agent-Linux-8.60.7471 directory.
Type: ./install.sh -defaults -web-
registration=/tmp/kmportal.txt <Enter> to install the Evault software and register
the server with the vault.
Result:
rx3484:/tmp/Agent-Linux-8.60.7471 # ./install.sh -defaults -web-
registration=/tmp/kmportal.txt
Verifying installer integrity... OK.
Verifying prerequisites... OK.
Install started at 15:38:22 2019.02.08
use_defaults implies user has read and agreed to the EULA.
Installing Backup Agent
Installing Backup Agent 8.60 for Linux.
Directory
: /usr/BUAgent
Disk Space Required : 133 MB (estimated)
Available
: 34118 MB
Fresh install. Skipping version check.
Preparing for installation ...
Using en-US [English (US)] as the default language.
6 of 13
7/9/19, 11:07 PMhttps://searshc.sysaidit.com/ShowFAQ.jsp?print=true&articleId=9...
The Portal address: portal.evault.com
The Portal port: 8086
The Portal username: rxupgrade@searshc.com
Registering with portal.evault.com:8086 using login of rxupgrade@searshc.com
Registered to The Portal.
/tmp/Agent-Linux-8.60.7471
By default, the Agent encrypts data using an encryption method that is integrated
in the Agent. For
audit purposes, some organizations require the Agent to use an external encryption
library that is provided.
Using the external encryption library can degrade Agent performance.
Please select one of the following:
[A] Encrypt data using the Integrated encryption method. Select this encryption
method for the best
Agent performance.
[B] Encrypt data using the External encryption library. Select this encryption
method if it is required
for audit purposes.
Note: To change the encryption method that is used, you must reinstall the Agent.
selecting A
/tmp/Agent-Linux-8.60.7471
Starting Agent: /etc/init.d/vvagent start quiet
Installation complete. Agent started successfully.
7 of 13
7/9/19, 11:07 PMhttps://searshc.sysaidit.com/ShowFAQ.jsp?print=true&articleId=9...
Install finished at 15:38:36 2019.02.08
The installation has been recorded in /usr/BUAgent/Install.log.
copy EULA to installation directory.
14. You will now need to register the system with the Carbonite Vault. For details,
see Article #9790.
15. Once the registration is complete, you can restore the /usr/pdx/*, /usr/kmart/*
and /usr/ldm/* files from the most
recent backup safeset in the Carbonite Vault. For details, see Article #9791.
16. Once the Carbonite restore is complete. you will need to restore the rxque and
phrx files from the KIN server, the
steps that were performed in NOTE 1 at the beginning of this knowledge article. For
details, see Article #515.
Result 1: If any scripts were lost, we need to bump numbers prior to the pharmacist
logging back in. To be safe,
we need to bump the RX numbers by 500 and the TX number by 1000 for each day of
typing the store has to re-type.
Result 2: No longer depend on the store to give us the correct numbers. You should
be able to tell from when the
store problem started and the date/time on the restore tape how many days of work
the store will have to re-do.
Example 1: If today is 1/12 and the tape label from Step 7 shows 1/11 22:30 then no
previous day's data
was lost, any typing between backup and the time the system was unavailable would
still need to be retyped.
Example 2: If today is 1/12 and the tape label from Step 7 shows 1/10 22:30 1 day
of data was lost in
addition to any typing done between the time of backup and the time the system was
unavailable.
NOTE: In the case the date/time of the last successfully filled script.
17. Log into the system as pdxsh and launch the PDX app. For details, see Article
#1034.
18. Bring up the RX filling screen.
20. Press ;L (semi-colon, letter L).
RESULT: This is the last script filled, note the date on the script in the LAST
field.
8 of 13
7/9/19, 11:07 PMhttps://searshc.sysaidit.com/ShowFAQ.jsp?print=true&articleId=9...
21. Press F5 (Profiles)
22. Press T (transaction profile) and Enter.
23. Press F5 (detail) on the script whose date matches above. Note the data from
the Filled: field in your case.
a. Use the combination of Example 1 and Example 2 in step 9 above, to determine if
and how many the RX and
TX numbers need to be bumped by.
NOTE 1: If there is any doubt at all, it is best to bump ahead the numbers. Contact
the Pharmacy Support
Desk or SHC_IR_RX_STORE_Appls with any questions on how to proceed.
NOTE 2: The Pharmacist needs to work with the Pharmacy Support Desk to verify all
re-typing is done
correctly.
NOTE 3: Failure to follow these procedures can cause any of the following major
problems for the store:
Same Rx number assigned to different customers (can affects IVR refills).
Third party information not matching carrier (fill date, co-pay, etc).
Accounting issues - difficulty matching payment.
Different Tx numbers if not filled in original order.
Different fill dates.
Customer profile co-pay inaccurate.
Claims dump out when brought in from store if store has communicated.
The carrier fill date does not match the store fill date.
9 of 13
7/9/19, 11:07 PMhttps://searshc.sysaidit.com/ShowFAQ.jsp?print=true&articleId=9...
24. Use the following steps to document and bump the numbers.
a. Pharmacist must be logged off from all terminals.
b. Choose PHARMACY --> MAINTENANCE --> ASSIGN STARTING NO
c. Copy the screen into your report or get a print screen of the numbers before
changing anything. Remember,
the numbers displayed are the last
RX used and the next RX number will be 1 higher. If you type an
incorrect value press ENTER and answer N to update the starting numbers. No changes
will be made.
d. Bump the number for each Schedule (Schedule 2, Sched 3-5, Schedule 6, OTC
drugs). You should bump the
number up by 1000 for each day of data lost.
e. Bump up the Tx number to ensure that a duplicate transaction number is not
assigned. The tx number must be
bumped up by 1000 for eachday of data lost.
EXAMPLE: The tx number after the restore is 1220259.
If the store has to re-key 1 day of work, the tx number should be changed to
1221259.
If the store has to re-key 2 days of work, the tx number should be changed to
1222259.
If the store has to re-key 3 days of work, the tx number should be changed to
1223259
and so on.
f. Bump Rx/Tx Audit up 1000 for each day of data lost.
g. Bump Services up 1000 for each day of data lost.
h. Bump PO Number up 100 for each day of data lost.
i. Bump Phone Doctor up 500 for each day of data lost.
j. Bump Rx Queue up 1000 for each day of data lost.
10 of 13
7/9/19, 11:07 PMhttps://searshc.sysaidit.com/ShowFAQ.jsp?print=true&articleId=9...
k. Bump Order up 100 for each day of data lost.
l. Bump Return/Adj up 100 for each day of data lost.
m. Bump Xmit Claim up 1000 for each day of data lost.
n. Bump Comm Jobs up 100 for each day of data lost.
o. Bump Store Xfer up 100 for each day of data lost.
p. Bump TP Submit up 1000 for each day of data lost.
q. Bump TP Recon up 1000 for each day of data lost.
NOTE: If the TP Recon field is blank, just skip it.
r. PAGE 2 - PRESS (F8) MORE.
s. Bump Rx Entry up 1000 for each day of data lost.
t. Bump Order Entry up 1000 for each day of data lost.
u. Bump Workflow up 1000 for each day of data lost.
v. Bump PHI Audit Log up 1000 for each day of data lost.
w. Press ENTER.
RESULT: Update starting number y/n will display.
x. Type y but do not press ENTER.
11 of 13
7/9/19, 11:07 PMhttps://searshc.sysaidit.com/ShowFAQ.jsp?print=true&articleId=9...
25. Go back into ASSIGN STARTING NO and verify the numbers are correct.
26. Tell the Pharmacist he will need to open a Help Ticket with the Pharmacy
Support Desk when ready to re-type
scripts. They have to work with the store to verify all re-typing is done properly.
Failure to follow these procedures can
cause any of the following major problems for the store:
Same Rx number assigned to different customers (can affects IVR refills).
Third party information not matching carrier (fill date, co-pay, etc).
Accounting issues - difficulty matching payment.
Different Tx numbers if not filled in original order.
Different Fill dates.
Customer profile co-pay inaccurate.
Claims dump out when brought in from store if store has communicated.
The carrier fill date does not match the store fill date.
27. After numbers have been bumped and verified working have the store go into
Patient query and send a Patient
query to rx.com (F7), if they get unable to communicate with rx.com, this indicates
the rx.com certification file got
corrupted and will need to be registered and escalated to central services via
email. For details see Article #3095 .
28. Type an email to PharmacySupportDesk@searshc.com and
SupportCentral/PharmacyHostTeam@searshc,com.
Include in the email the following:
A. Store number.
B. Case number.
12 of 13
7/9/19, 11:07 PMhttps://searshc.sysaidit.com/ShowFAQ.jsp?print=true&articleId=9...
C. That the system was restored.
D. How many days of business that have to be re-typed.
E. The rx and tx number range (starting number after restore and ending number once
you bumped rx and tx
numbers in Step 16).
F. Last script filled (step 12).
Group: SHC_IR_RX_STORE_APPLS
ARTICLE INFORMATION
TAGS
Created: 11-03-2019 Updated: 11-03-2019
Created by: Jose Cruz Updated by: Jose Cruz
Views: 41 Expires: 01-03-2020
backup, hard drive, HP,
network, system board,
tape, restore
Published to End Users: No
Related Items (0)
History (6)
13 of 13
7/9/19, 11:07 PM
4. The existing KIN server is on IP .202, after the conversion the new VKIN server
will be on .202

Cause
a. Procedure to install the new V-KIN server.

b. Procedure for V-KIN conversion.

c. Procedure to return the old KIN, Digi, monitor and tapes

Issue Description : PDX Down due to upgrade Failure


Observed Business Impact: Pharmacist not be able fill prescription
Observed Member Impact: Pharmacists
Affected Business Unit(s): Pharmacy
Restoration Details: Restored PDX from tape backup.
Alerting: User raised a ticket.

Issue Description : Unable to Archive backup.


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Backup Archived to IVR
Alerting:Netcool Alert / User Raised Ticket / None
Information:

1. The KIN conversion is to be performed two business days after the new V-KIN
server was installed.Hardware Deployment will call store the day before conversion
to confirm the conversion will take place the next morning.

2. Click here for the roll out schedule

3. The servers are configured at SHI, this includes imaging and installing a USB
flash drive for both the server and NAS.

4. The existing KIN server is on IP .202, after the conversion the new VKIN server
will be on .202

Cause

New V-KIN Server is replacing the IBM Netfinity 5600 Server

Resolution

Follow these steps to fix the issue

***IMPORTANT***: If you have any issues during the V-KIN Server


installation/conversion, call Hardware Deployment – Do NOT call the Help Desk.

1. Click here for the procedure to install the new V-KIN server

2. Click here for V-KIN conversion procedure

3. Click here for the Old KIN Removal procedure

========================

Issue Description : KIN Server Down


Observed Business Impact: when kin is down, Store cant end cash, End day, Can't do
online orders, RMUs offline, they can't punch in or out.
Observed Member Impact:: Associates can't perform any kin related activities
Affected Business Unit(s): Kmart
Restoration Details: Bypassed UPS / L3 Powered on kin VM / Rebooted KIN server /
Restore files, directories from Tape / Complete Restoration done from tape /
Powered on the UPS.
Alerting: Netcool Alert / User Raised Ticket / None

Issue Description : RX server Down


Observed Business Impact: Pharmacy can't able to work on pdx application.
Observed Member Impact: Pharmacists not able to fill prescriptions
Affected Business Unit(s): Kmart Pharmacy
Restoration Details: Bypassed UPS and powered on RX server / Powered on the UPS /
Complete Restoration done using Lifeboat CD with 10 Oct Tape / Rebooted RX server
due to hung state.
Alerting: Netcool Alert / User Raised Ticket / None

Issue Description : IVR Server Down


Observed Business Impact: IVR is not available, No Automated Refill available.
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Bypassed UPS / Powered on via ILO / Replaced systemboard /
Replace Dialogic Card
Alerting: Netcool Alert / User Raised Ticket / None

Issue Description : Tape backup issue in RX server


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Replaced Tape drive / Cleaned Tape drive / Reseated Tape drive
/ Replaced systemboard, SCSI cable, cage etc
Alerting: User Raised Ticket / None

Issue Description : zombie / Defunct Process in IVR / RX server


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Killed Defunct processes and restarted the SEA agents.
Alerting: Netcool Alert

Issue Description : High CPU workload RX server


Observed Business Impact: No impact to business as it happen during non business
hours.
Observed Member Impact: No impact to business as it happen during non business
hours.
Affected Business Unit(s): Pharmacy
Restoration Details: Restarted lcfd agents, now CPU workload is normal
Alerting:Netcool Alert

Issue Description : New UPS to be installed.


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Store
Restoration Details: With help of store person, connected the v-kin and nas to the
new UPS.
Alerting: Proactive ticket

Issue Description : Need to check the UPS status for V-KIN


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Store
Restoration Details: store person confirmed that V-KIN power cords are connected
to UPS.
Alerting: Proactive ticket

Issue Description : Hard drive failure in KIN / RX / IVR


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy / Store
Restoration Details: Replaced defective hard drive
Alerting: Netcool Alert / User Raised Ticket / None

Issue Description : Hard drive failure in V-Host (V-KIN)


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Store
Restoration Details: Replaced defective hard drive
Alerting: vCenter Alert

Issue Description : Server not reporting in SRM


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): None
Restoration Details: Restarted SEA Client & SRM Agent
Alerting: Proactive auto alert by BEM

Issue Description : Server not reporting in SRM


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): None
Restoration Details: issue resolved automatically.
Alerting: Proactive auto alert by BEM

Proactive auto alert by BEM

Issue Description : ITM6 Monitoring Agents Offline


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Restarted ITM Agent
Alerting:K_Server was notified by Tivoli team.

Issue Description : KIN DB Down


Observed Business Impact:
Observed Member Impact:: Store
Affected Business Unit(s): Store
Restoration Details: Rebooted KIN server / Replaced Processor card
Alerting:User Raised Ticket

Issue Description : PDX Down due to upgrade Failure


Observed Business Impact: Pharmacist not be able fill prescription
Observed Member Impact: Pharmacists
Affected Business Unit(s): Pharmacy
Restoration Details: Restored PDX from tape backup.
Alerting: User raised a ticket.

Issue Description : Unable to Archive backup.


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Backup Archived to IVR
Alerting:Netcool Alert / User Raised Ticket / None

Issue Description : High CPU workload RX server


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Resolved automatically and CPU workload is normal
Alerting:Netcool Alert / User Raised Ticket / None

Issue Description : System is Thrashing


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Rebooted RX server / Resolved automatically
Alerting:K_Server was notified by an automated monitoring alert.

Issue Description : NAS not pinging.


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Store
Restoration Details: L3 worked with store and fix the power source issue.
Alerting:K_Server was notified by Tivoli team.

not required. end user would not know if the device were functioning properly or
not.

Issue Description : iDrac not pinging.


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Store
Restoration Details: Reset the iDrac setting as per article #KB0031179
Alerting:K_Server was notified by Tivoli team.

Issue Description : HOST & iDrac not pinging.


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Store
Restoration Details: Bounced the n/w Port by Network Team and Reset the iDrac
setting
Alerting: L3 Alert
Issue Description : KIN Power Supply is not Redundant
Observed Business Impact: None
Observed Member Impact:: None
Affected Business Unit(s): Store
Restoration Details: Reseated power cables.
Alerting: Vcenter Alert

Issue Description : Tomcat appears to be stuck


Observed Business Impact: They can't punch in or out.
Observed Member Impact:: Associates can't perform punch in / out activities
Affected Business Unit(s): Kmart
Restoration Details: Issue resolved automatically by itself / Restarted Tomcat and
its running fine.
Alerting: BEM Alert

\Use Generic CI if we dont find the CI in ServiceNow.

PowerEdge T430 Server-GENERICASSET01178


HP ProLiant ML350 G3 Server - GENERICASSET01147
HP ProLiant ML350 G4 Server - GENERICASSET00873
HP ProLiant ML310 G5 IVR Server - GENERICASSET00858
T750 IVR UPS-GENERICASSET01169
EATON 1500 UPS - GENERICASSET01158
EATON Powerware 5125 UPS - GENERICASSET00058

Information:

You can use the System Management Homepage to check the status of the hard drives
(and other hardware within the server).
Open Internet Explorer
In the address field, key: https://rx.NNNN.st.kmart.com:2381 Where NNNN is the
store number (kmart is the pass)
Click Smart Array 6balkuma1@in.ibm.com41 Controller in Slot N (2 for G3, 3 for G4),
under Storage, for hard drive info.

2381 Where NNNN is the store number (kmart is the pass)


Click Smart Array 641 Controller in Slot N (2 for G3, 3 for G4), under Storage, for
hard drive info.

Issue Description : NAS backup not happening


Observed Business Impact: There is no impact to the business.
Observed Member Impact:: There is no impact to the Members
Affected Business Unit(s): Kmart
Restoration Details: Replaced defective hdd in NAS device and configured nas.
Alerting: proactive ticket.

NCC configured fes2u1 port 16 to vlan9. pxe booted into redhat install.

425118 HP ProLiant ML310 G5 IVR Server - GENERICASSET00858


425170 HP ProLiant ML350 G3 Server - GENERICASSET01147
425123 HP ProLiant ML350 G4 Server - GENERICASSET00873
511265 PowerEdge T430 Server-GENERICASSET01178
660505 PowerEdge T430 Server-GENERICASSET01178
141566 DS715 Disk Station NAS server-1540MUN539805
CI #435465 - NAS STORAGE

Initiate the Manual Backup on Carbonite Server Backup portal

rx7413 -Backup Failed -Carbonite Serverevault Backup

Issue Description : Backup issue in RX servevaulter


Observed Business Impact : There is no impact.
Observed Member Impact : There is no impact.
Affected Business Unit(s): Pharmacy
Restoration Details:Ran manual backup
Alerting: eMail Alert

Issue Description : KIN DB Down


Observed Business Impact:
Observed Member Impact:: Store
Affected Business Unit(s): Store
Restoration Details: Rebooted KIN server / Replaced Processor card
Alerting:User Raised Ticket

Issue Description : Virtual KIN Server Down


Observed Business Impact: RMU's, Punchtools, Layway are down
Observed Member Impact:: Store
Affected Business Unit(s): Store
Restoration Details: Smith, Terence fixed the issue (started the KIN server in
VMware)
Alerting:K_Server was notified by an automated monitoring alert

Issue Description : UPS Down


Observed Business Impact: No battery backup for V-KIN Server (RMU's, Punchtools,
Layway will down)
Observed Member Impact:: Store
Affected Business Unit(s): Store
Restoration Details: New UPS installed for V_KIN
Alerting:K_Server was notified from V-KIN Server down alert

********************************************************************************

Issue Description : RX server Down


Observed Business Impact: Pharmacist not be able fill prescription
Observed Member Impact: Pharmacists
Affected Business Unit(s): Pharmacy
Restoration Details: Bypassed UPS and powered on RX server.
Alerting: K_Server was notified by an automated monitoring alert / User Raised
Ticket

Issue Description : RX System is thrashing -Pageln 13840:00 PageOut: 2739.00


Observed Business Impact: May be degrade the performance
Observed Member Impact: Pharmacists
Affected Business Unit(s): Pharmacy
Restoration Details: PageIN & PageOUT ratio came down normal
Alerting: K_Server was notified by an automated monitoring alert

Issue Description : PDX Down due to upgrade Failure


Observed Business Impact: Pharmacist not be able fill prescription
Observed Member Impact: Pharmacists
Affected Business Unit(s): Pharmacy
Restoration Details: Restored PDX from tape backup.
Alerting: User raised a ticket.

Issue Description : CPU Workload High. CPU Aggregate -1 is 99.99% busy in RX / IVR
Observed Business Impact: May be degrade the performance
Observed Member Impact: Pharmacists
Affected Business Unit(s): Pharmacy
Restoration Details: CPU utilization came down automatically and its normal now /
Rebooted RX
Alerting: Netcool Alert

********************************************************************************

Issue Description : IVR Server Down


Observed Business Impact: IVR is not available, No Automated Refill available.
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Bypassed UPS / Powered on via ILO / Replaced systemboard /
Replace Dialogic Card
Alerting: K_Server was notified by an automated monitoring alert. / User Raised
Ticket

Issue Description : IVR System -Too many defunct/zombie processes


Observed Business Impact: May be degrade the performance
Observed Member Impact: Pharmacists
Affected Business Unit(s): Pharmacy
Restoration Details: Killed Zombie / Defunct processes & Restarted the SEA agent
Alerting: Proactive alert

===================================================================================
================================
Issue Description : Daily backup not performed on Evault.
Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: L3 has Reinstalled the agent and Re-registered, sync'd and
updated job.
Alerting: Evault Portal alert

Issue Description : Hard drive failure in KIN / RX / IVR


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy / Store
Restoration Details: Replaced defective hard drive
Alerting: K_Server was notified by an automated monitoring alert. / User Raised
Ticket

Issue Description : Tape backup issue in RX server


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Replaced Tape drive / Cleaned Tape drive / Replaced
systemboard, scsi cable, cage etc
Alerting:K_Server was notified by an automated monitoring alert. / User Raised
Ticket
Issue Description : RX / IVR server not reporting in SRM
Observed Business Impact: Server not reporting in SRM
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Restarted the SRM agents
Alerting: K_Server was notified by an Monitoring team

Issue Description : ITM Monitoring Offline Agent Restart - RX / IVR server


Observed Business Impact: Unable to monitoring the server
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Restarted the ITM agents
Alerting: K_Server was notified by an Monitoring team

===================================================================================
======

Hi Terry,

The store having new KIN server (Dell - 430)


Spoke with Office Associate - Pamela Shaffer,
She said this morning VKIN was hung, hence they rebooted...
After reboot VKIN session not invoking

Please do the needful.

INC0363236

Tivoli team
Eduardo Manenti De Araujo XXX
Fabio Antonio Baldim XXX
Dipsmita Das2/India/IBM
Isaque Benatti Souza

=====================

Knowledge article KB0022536:

Symptoms

a. Procedure to install the new V-KIN server.

b. Procedure for V-KIN conversion.

c. Procedure to return the old KIN, Digi, monitor and tapes

Information:

1. The KIN conversion is to be performed two business days after the new V-KIN
server was installed.Hardware Deployment will call store the day before conversion
to confirm the conversion will take place the next morning.

2. Click here for the roll out schedule

3. The servers are configured at SHI, this includes imaging and installing a USB
flash drive for both the server and NAS.
4. The existing KIN server is on IP .202, after the conversion the new VKIN server
will be on .202

Cause

New V-KIN Server is replacing the IBM Netfinity 5600 Server

Resolution

Follow these steps to fix the issue

***IMPORTANT***: If you have any issues during the V-KIN Server


installation/conversion, call Hardware Deployment – Do NOT call the Help Desk.

1. Click here for the procedure to install the new V-KIN server

2. Click here for V-KIN conversion procedure

3. Click here for the Old KIN Removal procedure

========================

Issue Description : KIN Server Down


Observed Business Impact: when kin is down, Store cant end cash, End day, Can't do
online orders, RMUs offline, they can't punch in or out.
Observed Member Impact:: Associates can't perform any kin related activities
Affected Business Unit(s): Kmart
Restoration Details: Bypassed UPS / L3 Powered on kin VM / Rebooted KIN server /
Restore files, directories from Tape / Complete Restoration done from tape /
Powered on the UPS.
Alerting: Netcool Alert / User Raised Ticket / None

Issue Description : RX server Down


Observed Business Impact: Pharmacy can't able to work on pdx application.
Observed Member Impact: Pharmacists not able to fill prescriptions
Affected Business Unit(s): Kmart Pharmacy
Restoration Details: Bypassed UPS and powered on RX server / Powered on the UPS /
Complete Restoration done using Lifeboat CD with 10 Oct Tape / Rebooted RX server
due to hung state.
Alerting: Netcool Alert / User Raised Ticket / None

Issue Description : IVR Server Down


Observed Business Impact: IVR is not available, No Automated Refill available.
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Bypassed UPS / Powered on via ILO / Replaced systemboard /
Replace Dialogic Card
Alerting: Netcool Alert / User Raised Ticket / None

Issue Description : Tape backup issue in RX server


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Replaced Tape drive / Cleaned Tape drive / Reseated Tape drive
/ Replaced systemboard, SCSI cable, cage etc
Alerting: User Raised Ticket / None

Issue Description : zombie / Defunct Process in IVR / RX server


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Killed Defunct processes and restarted the SEA agents.
Alerting: Netcool Alert

Issue Description : High CPU workload RX server


Observed Business Impact: No impact to business as it happen during non business
hours.
Observed Member Impact: No impact to business as it happen during non business
hours.
Affected Business Unit(s): Pharmacy
Restoration Details: Restarted lcfd agents, now CPU workload is normal
Alerting:Netcool Alert

Issue Description : New UPS to be installed.


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Store
Restoration Details: With help of store person, connected the v-kin and nas to the
new UPS.
Alerting: Proactive ticket

Issue Description : Need to check the UPS status for V-KIN


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Store
Restoration Details: store person confirmed that V-KIN power cords are connected
to UPS.
Alerting: Proactive ticket

Issue Description : Hard drive failure in KIN / RX / IVR


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy / Store
Restoration Details: Replaced defective hard drive
Alerting: Netcool Alert / User Raised Ticket / None

Issue Description : Hard drive failure in V-Host (V-KIN)


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Store
Restoration Details: Replaced defective hard drive
Alerting: vCenter Alert

Issue Description : Server not reporting in SRM


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): None
Restoration Details: Restarted SEA Client & SRM Agent
Alerting: Proactive auto alert by BEM

Issue Description : Server not reporting in SRM


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): None
Restoration Details: issue resolved automatically.
Alerting: Proactive auto alert by BEM
Proactive auto alert by BEM

Issue Description : ITM6 Monitoring Agents Offline


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Restarted ITM Agent
Alerting:K_Server was notified by Tivoli team.

Issue Description : KIN DB Down


Observed Business Impact:
Observed Member Impact:: Store
Affected Business Unit(s): Store
Restoration Details: Rebooted KIN server / Replaced Processor card
Alerting:User Raised Ticket

Issue Description : PDX Down due to upgrade Failure


Observed Business Impact: Pharmacist not be able fill prescription
Observed Member Impact: Pharmacists
Affected Business Unit(s): Pharmacy
Restoration Details: Restored PDX from tape backup.
Alerting: User raised a ticket.

Issue Description : Unable to Archive backup.


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Backup Archived to IVR
Alerting:Netcool Alert / User Raised Ticket / None

Issue Description : High CPU workload RX server


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Resolved automatically and CPU workload is normal
Alerting:Netcool Alert / User Raised Ticket / None

Issue Description : System is Thrashing


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Rebooted RX server / Resolved automatically
Alerting:K_Server was notified by an automated monitoring alert.

Issue Description : NAS not pinging.


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Store
Restoration Details: L3 worked with store and fix the power source issue.
Alerting:K_Server was notified by Tivoli team.

not required. end user would not know if the device were functioning properly or
not.

Issue Description : iDrac not pinging.


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Store
Restoration Details: Reset the iDrac setting as per article #KB0031179
Alerting:K_Server was notified by Tivoli team.

Issue Description : HOST & iDrac not pinging.


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Store
Restoration Details: Bounced the n/w Port by Network Team and Reset the iDrac
setting
Alerting: L3 Alert

Issue Description : KIN Power Supply is not Redundant


Observed Business Impact: None
Observed Member Impact:: None
Affected Business Unit(s): Store
Restoration Details: Reseated power cables.
Alerting: Vcenter Alert

Issue Description : Tomcat appears to be stuck


Observed Business Impact: They can't punch in or out.
Observed Member Impact:: Associates can't perform punch in / out activities
Affected Business Unit(s): Kmart
Restoration Details: Issue resolved automatically by itself / Restarted Tomcat and
its running fine.
Alerting: BEM Alert

\Use Generic CI if we dont find the CI in ServiceNow.

PowerEdge T430 Server-GENERICASSET01178


HP ProLiant ML350 G3 Server - GENERICASSET01147
HP ProLiant ML350 G4 Server - GENERICASSET00873
HP ProLiant ML310 G5 IVR Server - GENERICASSET00858
T750 IVR UPS-GENERICASSET01169
EATON 1500 UPS - GENERICASSET01158
EATON Powerware 5125 UPS - GENERICASSET00058

Information:

You can use the System Management Homepage to check the status of the hard drives
(and other hardware within the server).
Open Internet Explorer
In the address field, key: https://rx.NNNN.st.kmart.com:2381 Where NNNN is the
store number (kmart is the pass)
Click Smart Array 6balkuma1@in.ibm.com41 Controller in Slot N (2 for G3, 3 for G4),
under Storage, for hard drive info.

2381 Where NNNN is the store number (kmart is the pass)


Click Smart Array 641 Controller in Slot N (2 for G3, 3 for G4), under Storage, for
hard drive info.

Issue Description : NAS backup not happening


Observed Business Impact: There is no impact to the business.
Observed Member Impact:: There is no impact to the Members
Affected Business Unit(s): Kmart
Restoration Details: Replaced defective hdd in NAS device and configured nas.
Alerting: proactive ticket.

NCC configured fes2u1 port 16 to vlan9. pxe booted into redhat install.

425118 HP ProLiant ML310 G5 IVR Server - GENERICASSET00858


425170 HP ProLiant ML350 G3 Server - GENERICASSET01147
425123 HP ProLiant ML350 G4 Server - GENERICASSET00873
511265 PowerEdge T430 Server-GENERICASSET01178
660505 PowerEdge T430 Server-GENERICASSET01178
141566 DS715 Disk Station NAS server-1540MUN539805
CI #435465 - NAS STORAGE

Initiate the Manual Backup on Carbonite Server Backup portal

rx7413 -Backup Failed -Carbonite Server Backup

Issue Description : Backup issue in RX server


Observed Business Impact : There is no impact.
Observed Member Impact : There is no impact.
Affected Business Unit(s): Pharmacy
Restoration Details:Ran manual backup
Alerting: eMail Alert

Affected Business Unit(s): Pharmacy / Store


Restoration Details: Replaced defective hard drive
Alerting: K_Server was notified by an automated monitoring alert. / User Raised
Ticket

Issue Description : Tape backup issue in RX server


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Replaced Tape drive / Cleaned Tape drive / Replaced
systemboard, scsi cable, cage etc
Alerting:K_Server was notified by an automated monitoring alert. / User Raised
Ticket

Issue Description : RX / IVR server not reporting in SRM


Observed Business Impact: Server not reporting in SRM
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Restarted the SRM agents
Alerting: K_Server was notified by an Monitoring team

Issue Description : ITM Monitoring Offline Agent Restart - RX / IVR server


Observed Business Impact: Unable to monitoring the server
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Restarted the ITM agents
Alerting: K_Server was notified by an Monitoring team
===================================================================================
======

Hi Terry,

The store having new KIN server (Dell - 430)


Spoke with Office Associate - Pamela Shaffer,
She said this morning VKIN was hung, hence they rebooted...
After reboot VKIN session not invoking

Please do the needful.

INC0363236

Tivoli team
Eduardo Manenti De Araujo XXX
Fabio Antonio Baldim XXX
Dipsmita Das2/India/IBM
Isaque Benatti Souza

=====================

Knowledge article KB0022536:

Symptoms

a. Procedure to install the new V-KIN server.

b. Procedure for V-KIN conversion.

c. Procedure to return the old KIN, Digi, monitor and tapes

Information:

1. The KIN conversion is to be performed two business days after the new V-KIN
server was installed.Hardware Deployment will call store the day before conversion
to confirm the conversion will take place the next morning.

2. Click here for the roll out schedule

3. The servers are configured at SHI, this includes imaging and installing a USB
flash drive for both the server and NAS.

4. The existing KIN server is on IP .202, after the conversion the new VKIN server
will be on .202

Cause

New V-KIN Server is replacing the IBM Netfinity 5600 Server

Resolution

Follow these steps to fix the issue

***IMPORTANT***: If you have any issues during the V-KIN Server


installation/conversion, call Hardware Deployment – Do NOT call the Help Desk.

1. Click here for the procedure to install the new V-KIN server
2. Click here for V-KIN conversion procedure

3. Click here for the Old KIN Removal procedure

========================

Issue Description : KIN Server Down


Observed Business Impact: when kin is down, Store cant end cash, End day, Can't do
online orders, RMUs offline, they can't punch in or out.
Observed Member Impact:: Associates can't perform any kin related activities
Affected Business Unit(s): Kmart
Restoration Details: Bypassed UPS / L3 Powered on kin VM / Rebooted KIN server /
Restore files, directories from Tape / Complete Restoration done from tape /
Powered on the UPS.
Alerting: Netcool Alert / User Raised Ticket / None

Issue Description : RX server Down


Observed Business Impact: Pharmacy can't able to work on pdx application.
Observed Member Impact: Pharmacists not able to fill prescriptions
Affected Business Unit(s): Kmart Pharmacy
Restoration Details: Bypassed UPS and powered on RX server / Powered on the UPS /
Complete Restoration done using Lifeboat CD with 10 Oct Tape / Rebooted RX server
due to hung state.
Alerting: Netcool Alert / User Raised Ticket / None

Issue Description : IVR Server Down


Observed Business Impact: IVR is not available, No Automated Refill available.
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Bypassed UPS / Powered on via ILO / Replaced systemboard /
Replace Dialogic Card
Alerting: Netcool Alert / User Raised Ticket / None

Issue Description : Tape backup issue in RX server


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Replaced Tape drive / Cleaned Tape drive / Reseated Tape drive
/ Replaced systemboard, SCSI cable, cage etc
Alerting: User Raised Ticket / None

Issue Description : zombie / Defunct Process in IVR / RX server


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Killed Defunct processes and restarted the SEA agents.
Alerting: Netcool Alert

Issue Description : High CPU workload RX server


Observed Business Impact: No impact to business as it happen during non business
hours.
Observed Member Impact: No impact to business as it happen during non business
hours.
Affected Business Unit(s): Pharmacy
Restoration Details: Restarted lcfd agents, now CPU workload is normal
Alerting:Netcool Alert

Issue Description : New UPS to be installed.


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Store
Restoration Details: With help of store person, connected the v-kin and nas to the
new UPS.
Alerting: Proactive ticket

Issue Description : Need to check the UPS status for V-KIN


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Store
Restoration Details: store person confirmed that V-KIN power cords are connected
to UPS.
Alerting: Proactive ticket

Issue Description : Hard drive failure in KIN / RX / IVR


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy / Store
Restoration Details: Replaced defective hard drive
Alerting: Netcool Alert / User Raised Ticket / None

Issue Description : Hard drive failure in V-Host (V-KIN)


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Store
Restoration Details: Replaced defective hard drive
Alerting: vCenter Alert

Issue Description : Server not reporting in SRM


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): None
Restoration Details: Restarted SEA Client & SRM Agent
Alerting: Proactive auto alert by BEM

Issue Description : Server not reporting in SRM


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): None
Restoration Details: issue resolved automatically.
Alerting: Proactive auto alert by BEM

Proactive auto alert by BEM

Issue Description : ITM6 Monitoring Agents Offline


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Restarted ITM Agent
Alerting:K_Server was notified by Tivoli team.

Issue Description : KIN DB Down


Observed Business Impact:
Observed Member Impact:: Store
Affected Business Unit(s): Store
Restoration Details: Rebooted KIN server / Replaced Processor card
Alerting:User Raised Ticket
Issue Description : PDX Down due to upgrade Failure
Observed Business Impact: Pharmacist not be able fill prescription
Observed Member Impact: Pharmacists
Affected Business Unit(s): Pharmacy
Restoration Details: Restored PDX from tape backup.
Alerting: User raised a ticket.

Issue Description : Unable to Archive backup.


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Backup Archived to IVR
Alerting:Netcool Alert / User Raised Ticket / None

Issue Description : High CPU workload RX server


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Resolved automatically and CPU workload is normal
Alerting:Netcool Alert / User Raised Ticket / None

Issue Description : System is Thrashing


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Pharmacy
Restoration Details: Rebooted RX server / Resolved automatically
Alerting:K_Server was notified by an automated monitoring alert.

Issue Description : NAS not pinging.


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Store
Restoration Details: L3 worked with store and fix the power source issue.
Alerting:K_Server was notified by Tivoli team.

not required. end user would not know if the device were functioning properly or
not.

Issue Description : iDrac not pinging.


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Store
Restoration Details: Reset the iDrac setting as per article #KB0031179
Alerting:K_Server was notified by Tivoli team.

Issue Description : HOST & iDrac not pinging.


Observed Business Impact: None
Observed Member Impact: None
Affected Business Unit(s): Store
Restoration Details: Bounced the n/w Port by Network Team and Reset the iDrac
setting
Alerting: L3 Alert

Issue Description : KIN Power Supply is not Redundant


Observed Business Impact: None
Observed Member Impact:: None
Affected Business Unit(s): Store
Restoration Details: Reseated power cables.
Alerting: Vcenter Alert

Issue Description : Tomcat appears to be stuck


Observed Business Impact: They can't punch in or out.
Observed Member Impact:: Associates can't perform punch in / out activities
Affected Business Unit(s): Kmart
Restoration Details: Issue resolved automatically by itself / Restarted Tomcat and
its running fine.
Alerting: BEM Alert

\Use Generic CI if we dont find the CI in ServiceNow.

PowerEdge T430 Server-GENERICASSET01178


HP ProLiant ML350 G3 Server - GENERICASSET01147
HP ProLiant ML350 G4 Server - GENERICASSET00873
HP ProLiant ML310 G5 IVR Server - GENERICASSET00858
T750 IVR UPS-GENERICASSET01169
EATON 1500 UPS - GENERICASSET01158
EATON Powerware 5125 UPS - GENERICASSET00058

Information:

You can use the System Management Homepage to check the status of the hard drives
(and other hardware within the server).
Open Internet Explorer
In the address field, key: https://rx.NNNN.st.kmart.com:2381 Where NNNN is the
store number (kmart is the pass)
Click Smart Array 6balkuma1@in.ibm.com41 Controller in Slot N (2 for G3, 3 for G4),
under Storage, for hard drive info.

2381 Where NNNN is the store number (kmart is the pass)


Click Smart Array 641 Controller in Slot N (2 for G3, 3 for G4), under Storage, for
hard drive info.

Issue Description : NAS backup not happening


Observed Business Impact: There is no impact to the business.
Observed Member Impact:: There is no impact to the Members
Affected Business Unit(s): Kmart
Restoration Details: Replaced defective hdd in NAS device and configured nas.
Alerting: proactive ticket.

NCC configured fes2u1 port 16 to vlan9. pxe booted into redhat install.

425118 HP ProLiant ML310 G5 IVR Server - GENERICASSET00858


425170 HP ProLiant ML350 G3 Server - GENERICASSET01147
425123 HP ProLiant ML350 G4 Server - GENERICASSET00873
511265 PowerEdge T430 Server-GENERICASSET01178
660505 PowerEdge T430 Server-GENERICASSET01178
141566 DS715 Disk Station NAS server-1540MUN539805
CI #435465 - NAS STORAGE
Initiate the Manual Backup on Carbonite Server Backup portal

rx7413 -Backup Failed -Carbonite Server Backup

Issue Description : Backup issue in RX server


Observed Business Impact : There is no impact.
Observed Member Impact : There is no impact.
Affected Business Unit(s): Pharmacy
Restoration Details:Ran manual backup
Alerting: eMail Alert

S-ar putea să vă placă și