RSIT Hardware and Infrastructure

Engineering
15120 NE 40th Street
Redmond, WA 98052

Date:
To:
From:
Subject:

October 11, 2015

Charlie McNerny
General Manager
Microsoft RSIT Division
Joe Reese, Engineering Senior Program Manager
RSIT Hardware and Infrastructure Engineering
Memo: Technology Deployment Research Study

Purpose
The purpose of this memo is to present preliminary findings regarding the formal
handoff inefficiencies between our Engineering and Operations groups. In addition,
we will provide recommendations based on the research findings, aptly aimed to
make improvements in several areas of these handoffs immediately.

Summary
On July 1, 2015, our engineering department implemented a new delivery
methodology, known as Agile, which subsequently changed our Engineering release
model to our Operations group. To recap, our engineering efforts design and build
IT solutions that are packaged and turned over to Operations for release into
Microsoft Production environments. Beginning with the Agile implementation, this
handoff has suffered numerous issues affecting our ability to smoothly delivery IT
improvements to our business. In an effort to understand these inefficiencies,
Charlie McNerny authorized a research project to review the entire process. I am
leading the review along with Pedro Silva, Engineering Manager, who is assisting
with this effort.
A number of research activities revealed that there are large inefficiencies in the
areas of preplanning and release notifications to Operations from Engineering, lack
of skills in certain Operations deployment groups, and gaps in the deployment
process. The activities to review internal Microsoft team releases process and
explore outside release management processes has yet to be completed. These
remaining tasks are set for completion in the following weeks.
Based on the findings, we strongly recommend two courses of action immediately:

New Process - Implement a process that sends a full Engineering plan for
the iteration, along with deployment estimates to Operations ~30 days in
advance of any release
Training Conduct release training for the ERP and CRM in Database, patch
management, and general maintenance
Deprecate Unnecessary Processes Stop use of the legacy processes
that were serve no purpose under the new methodology

These three efforts should increase transitions effectiveness by an estimated 20%

and require minimal investment ($7500) for training our resources to deploy
releases into production.

Introduction
Essentially, every month our Software & Hardware Engineering teams (roughly 30
individual teams, 150 resources) turn over their completed work, at the same time
and same day, to the Operations group for production deployment. This mass
turnover inundates the Operations group, who then scrambles to organize and
schedule all the work for release into the Production environments.
An efficient release plan and tighter coordination between Engineering and
Operations is lacking, along with no clear solution for path for this effort.
Observation identified that neither group communicates to the other group during
the early stages of development. Due to the lack of communication, Operations is
unable to prepare a release schedule nor understands the level of effort required to
deploy an individual component. In addition, the amount of solutions deployed
each month varies making it very difficult to forecast.
Over the past few weeks, the research project team performed the following
research activities:

Conducted interviews with key Operations and Engineering staff to determine

how handoffs are being used currently
Observed and documented the current release process steps, including
notification of release from Engineering thru the Operation planning and
scheduling steps to implement into production environments
Researched the release management process to uncover inefficiencies

In the following sections, we provide detail to the research methods, results,

conclusions, and recommendations.

Research Methods
We began our research by interviewing Linh Le, Operations Director, who provided a
breakdown of the entire process including release into production. Our interview
used a mix of structured and unstructured questions, designed to understand
Engineering to Operations touch points, resources employed, and processes
executed to schedule and deploy the IT solutions.

Based on this information, we formulated a series of tasks to conduct the research

needed to have a full end-to-end picture of this work. This structure includes the
following three tasks:

Conduct interviews with Operations and Engineering

Observe and document the processes
Review and analyze the processes

Task 1 - Conduct interviews with key Operations and Engineering staff

to determine how handoffs are being used currently
Linh Le has provided a resource and capacity matrix for Operations teams, including
his engineering teams handoff contacts. Linh has created a release team specific to
each engineering team, for which is considered dedicated to these counterparts;
there is no cross-team resource utilization amongst Operations.
The deployment resources include 30 dedicated deployment teams, one master
scheduler, and number of other resources who provide oversight and governance.
Each team develops a deployment playbook outlining procedures and sequences
of activities performed to deploy the solution into production.
Engineering interviews uncovered similar processes for promoting solutions to
Operations, along with similar readiness and sequencing activities. We selected
three individuals for 30 minutes interviews, three in Operations, and two in
Engineering. Both Pedro and I were present for each of the interviews, individually
gathering notes for later comparison. We combined our individual notes into a
single One Note document thus signal completion of this task. We allowed each
team time to review the One Note document, providing them an opportunity to
identify any discrepancies or inaccuracies.

Task 2 - Observe and document the current release process steps,

including notification of release from Engineering thru the planning and
scheduling steps to implement into production environments
Areas observed include:

Engineering notification
Operations notification triage, handling and routing to specific deployment
teams
Deployment team encapsulation of deployment steps, resources, and time
constraints
Review of triaged works, prior to scheduling
Scheduling artifact and final publishing

We selected the five largest of the 30 teams as a sample of the population for
observation and data collection. We focused on heavily used handoff steps and
ignored less utilized practices as means to focus on the recurring and repetitive
areas of the processes. We collected data for each of the above steps, categorizing
each step, noting if it played a critical path role, and time required for execution.

We recorded the collected data into an Excel spreadsheet for easy reference and
later analysis in Task 3.

Task 3 - Research both Engineering and release management process

steps to uncover flexibility in each item or process
Pedro and I used the data collected in Task 1 and 2, reviewing each step carefully
and accessing its overall value and efficiency level (amount of work divided by
resources used). For each deployment, we calculated total time by adding each
individual steps together. For deliveries that experienced issues in scheduling, we
noted and categorized primary reasons for the issue along with calculating the
additional time devoted to resolving the issue. We recorded all information into the
spreadsheet and reviewed with Linh Le for accuracy.

Results
In this section, we present the results of our research findings. Please note that we
are presenting the most important data we acquired during the research activities.

Task 1 - Conduct interviews with key Operations and Engineering staff

to determine how handoffs are being used currently
Each interview produced similar findings regarding specific areas of concern with
the new process. Operations cited lack of notification as the number one reason of
inefficient handoff. The second reason cited by interviewees is poor understanding
of what/how to deploy upcoming releases. The last issued cited by interviewees is
lack of available resources from Engineering to complete scheduling tasks.

Task 2 - Observe and document the current release process steps,

including notification of release from Engineering thru the planning and
scheduling steps to implement into production environments
For each of the 30 teams, we discovered 14 different handoff procedures, many of
which had similarities but not identical processes. Each team autonomously
created their own processes to handle the handoffs, not based on efficiency, but
simply based on reactionary responses to the new Agile process. There is no
sharing of processes between teams, nor did we identify any resources who review
the processes and procedures from an efficiency standpoint.
Scheduling requires a minimum of three resources to complete the outlined steps,
two resources from Operations, and one resource from Engineering. There of the 15
resources on each of the five teams that mistakenly executed the process
incorrectly, adding an additional eight hours to the time required to complete
scheduling.

Task 3 - Research both Engineering and release management process

steps to uncover flexibility in each item or process
Thirty-eight processes are in use across the five major scheduling and deployment
areas. Twelve tasks identified as unnecessary, serving no obvious purpose in the
new Agile process. Due to time constraints, that major tasks received a cursory
review and not full review with deep analysis. The team leads each reviewed the

findings for verification and accuracy. In addition, each team had an opportunity to
confirm/deny if the process met critical needs.

CrConclusions
Analysis of the execution time indicates five long pole tasks, which consumed over
30% of the total time. These tasks indicate a high level of complexity, lack of
resource proficiency, and or too much dependency on the individual item strongly
suggesting a process redesign.
Data indicates it takes on average four days to develop the scheduling for an
individual deployment based on the calculations of the five teams that we observed.
Further time to review other Microsoft team releases process and explore outside
release management processes is required. These remaining tasks are set for
completion in the following weeks, per our research plan proposal. However, we
strongly believe that there is major room for improvement based on the conclusive
evidence discovered to date.

Recommendations
Based on the findings of research, we strongly recommend three immediate courses
of action:

New Process - Implement a process that sends a full Engineering plan for
the month along with deployment estimates to Operations 30 days in
advance of any release
Training Conduct release training for the ERP and CRM in Database, patch
management, and general maintenance
Deprecate Unnecessary Processes Stop use of the legacy processes
that were serve no purpose under the new methodology

The new notification process will signal to Operations to begin preplanning of

deployment, allowing early access to technical information required for optimal
efficiency. This new process also provides Engineering a better understanding of
Operations planning needs, which allows Engineering to tailor the delivery to meet
these needs. The training initiative will improve release efficiency, better preparing
release resources to handle the complexity of our most demanding applications.
Finally, removing unnecessary processes will shorten end-to-end release time and
allow for better utilization for resourcing, focusing resources on relevant processes.
Overall, these efforts should increase transitions effectiveness by 20% and require
minimal investment ($7500) for training our resources to deploy releases into
production.