GU I DE

M
ajor incidents start to impact businesses within just minutes, often
before incident resolution teams can even be engaged. With this
immediate impact to business, meeting your major incident SLAs
is no longer enough. You are under a microscope from customers with
expectations that all systems are ‘always on,’ or you’ll inform them immediately
and communicate to them along the way.

What’s more, given the sensitive nature of incidents and

the variety of audiences requiring communication, knowing
Align Your Communications
how to adequately manage communication plans can be with Your Process During
tricky. Handling them poorly can be even worse than not
handling them at all. the Planning Stage
Having processes and systems in place to resolve major
When it comes to communicating during major incidents,
incidents means you’re already ahead of many large
business customers are like Goldilocks: They want
enterprises. Monitoring, contact, triage, resolution, post
information that’s just right. They don’t want too little, they
mortem all constitute areas where pre-planning can bring
don’t want too much, they want it relevant to them, and
value.
they want it right on time.

Major incidents happen. The processes and systems in place

This paper outlines best practices to empower and engage
will determine how successfully you resolve them. The right
your key stakeholders during major incidents.
technology, proper planning and configuration can produce

THE REAL COST OF POOR COMMUNICATION

S ending notifications too late, to the wrong
stakeholders, or with the wrong message can
damage your organization (and your job security).
Executives kept in the dark can take your time with
inquiries, damage your internal reputation, and cost you
your job.
Extend your communication beyond resolvers:
Third-party service providers who lack information
Customers will contact you by the thousands. Customer
cannot help with resolution. A service outage starts to
satisfaction drops by 15 percent every time a customer
affect most businesses within just 15 minutes.2
has to call back about the same issue.1

PROACTIVE COMMUNICATIONS DURING MAJOR INCIDENTS www.xMatters.com | 2


a turnkey, repeatable process that does not require manual
communications and produces faster resolution times.
Each of these steps produces a unique set of data. It would
be easy to use that data to send one communication to
incident resolvers and all stakeholders. Instead, parse the
data to produce multiple messages for unique audiences. For
instance, in the Triage step, messages might look like this:
To incident
•  resolvers: Servers SRDC-RHL09
through RHL16 in the third bank in Server Room 1
at Headquarters are failing intermittently. Please test
the servers and all the connections to determine the
exact cause so we can take corrective action ASAP.
To the IT department head: Servers SRDC-RHL09
• 
through RHL16 in the third bank in Server Room 1
at Headquarters are failing intermittently. We are
beginning tests. We will let you know when we have
determined the exact cause and can give you an
estimated completion time.
To executives: We’re having an issue with some
• 
servers at Headquarters. We are testing to determine
the cause, and we will let you know the estimated
completion time as soon as possible.
To customers: We are experience some service
• 
issues. We are working on a resolution diligently, and
will restore service as soon as possible.
Automate Messages with Templates
You need at least four distinct messages. There could
also be messages to partners, utilities, PR, corporate
communications, and more. Creating all these messages
on the fly in the midst of trying to resolve a major incident
would cause delay and increase the risk of error.
Instead, create a series of templates to automate the
tailoring of messages at each stage of the process.
PROACTIVE COMMUNICATIONS DURING MAJOR INCIDENTS
GU I DE
Choose a template, update recipients if necessary, update
messaging if necessary, add relevant data, and send. Even
better, tight integration between monitoring, ticketing, and
other systems with communication systems could allow for
messages to auto-populate based on the type of incident.
Right-Sizing
Communications:
How to Get it Just Right
As an IT professional, you likely didn’t sign up for one of the
trickiest communication jobs in the business, but getting
the communications right is now officially part of your job
description. A good rule of thumb on tackling this new task:
communicate as broadly as the impact of your incident, as
detailed as required (but no more) for any given audience,
as far ahead of impact as possible.
BREADTH
Major incidents are no longer isolated to back-end batch
processing nor even live systems that impact a small set
of constituents. With so many applications and services
interconnected, a single incident could have a domino effect
impacting multiple stakeholders. Consider the following
audiences when building communication plans:
Direct Stakeholders – Communications should be sent to
anyone whose business processes may be impacted by the
incident. Depending on the type of incident, this could be
owners of a process dependent on an impacted service,
an internal customer reliant on the interrupted service or
even end-customers (business or consumer) who will feel
the impact in a significant manner. For example, an incident
that might simply slow the services of an end consumer
might not require communications to consumers, but a
business customer reliant on a service that experiences an
outage will be more significantly impacted and should be
considered in your communications plan.
www.xMatters.com | 3

Help Desks and Call Centers – When services or applications
are slow or disrupted, your typical 2015 worker seems
to have trigger finger when it comes to calling your help
desk. Even a 20 minute e-mail outage can result in a flood
of hundreds of tickets and phone calls in the help desk of
a typical large enterprise. Alerting the staff manning the
lines not only prevents them from opening duplicate tickets,
but it’s also an opportunity to arm them with outbound
communication to alleviate the concerns of inbound callers.
Management – While extra inbound calls have hard dollar
costs associated with them, an out-of-the-loop executive
has an immeasurable reputation cost. Your highest value
customers most likely have direct access to company
executives and when they feel a disruption, chances are
they’re not calling into a call center. Executives should be
kept in the loop in real time and never caught off-guard by
in inbound client.
Customer-Facing Staff – In addition to call centers and
executives, any customer-facing staff should be kept
informed of incidents that might impact their customers. This
might include field staff, account executives, sales staff, etc.
Downstream Dominoes – In today’s inter-connected
enterprise, a single incident may trigger a multitude of
additional incidents. Understanding interdependencies could
help prevent or mitigate damage downstream. For example,
a power outage might not have an immediate impact outside
of restoration teams if a back-up generator kicks in. But if a
generator gets below a certain threshold and restoration is not
imminent, giving application owners ample time to shut down
apps properly could save hours in restart time once the power
is back up and running.
TIMING
The modern business customer - be they internal or
outside - is forgiving yet impatient. Intermingled with their
reliance on digital systems is an acknowledgement that
they will break, but the forgiveness comes packaged with
PROACTIVE COMMUNICATIONS DURING MAJOR INCIDENTS
GU I DE
expectations that communications will be transparent and
timely. As 451 Research Analyst Donnie Berkholz stated in a
blog post “We all understand that sometimes things break,
because clouds are incredibly complex systems. We’re only
really looking for two things out of it: (1) don’t have the
same problem twice, and (2) keep us informed.”
Optimal timing would allow you to reach stakeholders and
clients before they feel the impact of an incident. Let the
call center know before an influx of calls, let the client know
before their own systems are impacted, let your executives
know before someone else tells them.
CONTENT
As mentioned earlier in the paper, altering the content of
your communication to specific audiences is key. Resolution
teams need to have specific details as well as calls to action
to engage and inform them rapidly. External clients don’t
need or want details. They simply want to know that the
problem is being addressed and, when possible, what
to expect next. Don’t confuse transparency with over-
communication. Too much information can result in lost
confidence. The following guidelines can help with crafting
incident messaging for a variety of audiences:
1: Is the information of value to the recipient?
2: Is the information actionable by the recipient?
3: Does the information help to reduce dissatisfaction
or confusion? For example, updates at timed
intervals can be beneficial in assuring constituents
that teams are working on the issue, even if there is
no known additional information.
4: Does the information use vernacular familiar to the
audience? For example, don’t bog down executives
with fancy tech acronyms and internal process
abbreviations. Use plain language that they can
relay as necessary.
www.xMatters.com | 4
 GU I DE

Tactics for Communicating you know they can be fickle with specific needs. Allowing
stakeholders to subscribe to communications lets them

During an Incident control frequency of alerts, mode of communication and

subject matter on which they want to be informed.
When a major incident occurs, time is rarely your friend. The
following tactics help you to use communication techniques
Use Communication to Trigger Collaboration
to actually save time versus wasting precious minutes.
Resolution teams still need to take action quickly. In today’s
distributed, heterogeneous environments that often requires
Utilize Message Templates
collaboration. Something as simple as a conference call for
During the heat of an incident, the last thing your teams
a team representing a dozen skill sets can take well over
need to be doing is crafting prose. Message templates can
an hour to assemble. Automating notification to conference
save time by providing messaging and response options.
bridge recipients (with escalation processes for alternates)
Templates can be rapidly altered during incidents for
can assemble the same team in 3-5 minutes, drastically
specifics.
reducing resolution times.

Integrate When Possible

Partner with Your Communication Team
Even better, when communications templates are integrated
Once communications leave the walls of your enterprise, it’s
with monitoring systems, details can be auto-populated
fair game. While transparency and honesty are important
into message templates.
aspects of communicating to stakeholders, you may want
a professional to craft the exact wording. Proactively set
Automate When Possible
up meetings with your client or corporate communications
Tight integration between your communication and ticketing
team to create the templates and messages to keep in your
& monitoring systems allows for messages to automatically
arsenal for when a major incident hits.
be sent without taking your resolution and ops teams off
the tasks of restoring services. Enterprise-grade systems
will allow for multiple messages to be triggered for different
audiences with different messages.
“T
 he processes and systems
in place will determine how
Synchronicity
Synchronize contact information via data sync from successfully you resolve
systems of record. Augment basic contact information
with contextual information like on-call schedules, groups
them. The right technology,
and skill sets. During incidents, you are rarely looking for proper planning and
a specific person, but for a role or skill set. For example,
you don’t want ‘Joan,’ you want the ‘call center manager on
configuration can produce a
duty’. turnkey, repeatable process
Give Stakeholders Control that does not require
We’ve already covered the requirement of communicating
to stakeholders beyond resolution teams. We’ve also let
manual communications.”

PROACTIVE COMMUNICATIONS DURING MAJOR INCIDENTS www.xMatters.com | 5


Communication Goes Both Ways
Individuals impacted by a major incident can be some of
your best field agents. Use proactive communications as
a chance to gather data points if you think it might help
with troubleshooting or post mortem. Your customers
want to know that you’ll learn from the experience and feel
confident that the same issue won’t happen again. Including
their input as part of that process gives them a sense of
partnership in improving the services you provide.
After an Incident:
Post Mortem
During an incident, your company communicates as events
change, which is crucial. But it’s only after an incident is over
– without the distraction of firefighting - that one can gain
true perspective and start to sort out all of the facts. With
this 20/20 hindsight, it’s beneficial to give stakeholders the
confidence that you have identified root cause, that any
processes are now in place to mitigate similar incidents and
reduce risk.3
First-call resolution: how important is it, really?, https://www.
1. 
atlassian.com/help-desk/first-call-resolution, 2014
Business Impact of IT Incident Communications, http://info.xmatters.
2. 
com/rs/alarmpoint/images/xMatters-2015-Survey-Report.pdf, 2015
CONTACTS
MO R E ON LI N E RESOUR C ES
https://www.xmatters.com/resources/
INT ERN ATI ON A L
+1 925.226.0300 and press 2
U S /C AN TOLL FREE
+1 877.XMATTRS (962.8877)
EMEA
+44 (0) 20 3427 6333
AU ST RA LI A /A PJ SUP P ORT
+61-2-8038-5048 opt 2
WHIT E PAP E R
From resolution teams and IT management to executives,
from service desks and call centers to partners and clients,
a major incident sends ripples of impact throughout your
extended organization. Learning how to trigger and tier
proactive communication during times of disruption allows
for resolution teams to focus on resolution, mitigates
impact to client-facing teams, and can drastically increase
reputation and stakeholder satisfaction.
Cloud Outages, Transparency, and Trust,http://redmonk.com/
3. 
dberkholz/2015/01/12/cloud-outages-transparency-and-trust, 2015
ABOUT US
xMatters’ cloud-based solutions enable any business process or
application to trigger two-way communications (text, voice, email,
etc.) throughout the extended enterprise during time-sensitive
events. With over a decade of experience in rapid communication,
xMatters serves more than 1,000 leading global firms to ensure
business operations run smoothly and effectively during incidents
such as IT failures, product recalls, natural disasters, dynamic staffing,
service outages, medical emergencies and supply-chain disruption.
xMatters is headquartered in San Ramon, CA with additional offices
in London and Sydney.
Copyright 2015 xMatters. All rights reserved. All other products and brand names are
trademarks or registered trademarks of their respective holders.