Data Center Relocation PDF

Data Center Relocation Guide
By: Peter Harrison
Friday, November 30, 2007

A LinuxHomeNetworking.com © White Paper
About The Author
Peter Harrison has over ten years of data center experience in both the private and public sectors
providing Internet and corporate services. He is a Cisco Certified Internetworking Engineer (CCIE) and
currently consults on data center design and network architectures. He is also the author of the Linux
Quick Fix Notebook published by Prentice Hall now available on Amazon.com.
He was the founding president of PCJAM, Jamaica's first computer user group, and was the principal
systems engineer responsible for the computerization of the island's tax collection and social security
systems.
He then sought new opportunities as the western Caribbean representative for a Fortune 500
pharmaceuticals firm and later became the international sales manager for a West Indian rum
company. Before moving to Silicon Valley he ran Trinidad and Tobago's first industrial trade office to
Latin America.
Peter has since worked extensively in the Internet sector deploying large-scale data centers and Web
sites. His extensive technical experience combined with his varied business background has helped
him create this highly readable guide for project managers, techies, and their bosses.
Peter also is the creator of LinuxHomeNetworking.com, a site dedicated to IT white papers and
discussion on Linux, Cisco products and data center activities.
In his quieter moments, Peter enjoys the art and literature of the Caribbean and Latin America. Long
rides on his bicycle provide another guilty pleasure. Peter likes to relax with his family on short weekend
trips to the many attractions of the San Francisco Bay Area.
Copyright
© Peter Harrison 2002-2007, All rights reserved.
Unless otherwise stated, the material published within this document is copyright of the author, Peter
Harrison. No part of this document, including page design, interior design, cover design and icons may
be reproduced or transmitted in any form, by any means, (electronic, photocopying, recording, or
otherwise) without the prior consent of the publisher/author.
As a sole exception, the author allows purchasers of this document to create a single printed version for
their own personal, non-commercial use.
Disclaimer – The Website and Manual

While every effort will be made to ensure that the information contained within this website and manual
is accurate and up to date, the author makes no warranty, representation or undertaking whether
expressed or implied, nor does it assume any legal liability, whether direct or indirect, or responsibility
for the accuracy, completeness, or usefulness of any information.
Disclaimer - Other sites

Hypertext links to sites outside this website are provided as a convenience to users and should not
necessarily be construed as an endorsement. Although every care is taken to provide links to suitable
material from this site, the nature of the Internet prevents the author from guaranteeing the suitability or
accuracy of any of the material that this site may be linked to. Consequently, the author can accept no
responsibility for unsuitable or inaccurate material that may be encountered and accepts no liability
whether direct or indirect for any loss or damage a person suffers because that person had directly or
indirectly relied on any information stored in the hypertext links.
Further, the author is not and can not be responsible for the accuracy or legitimacy of information found
elsewhere on the Internet and there is therefore no guarantee or warranty that any of the sites listed will
be available at any particular time. The author does not guarantee or warrant any services that might be
announced - use at your own risk.
Table of Contents
Chapter 1...........................................................................................................................................7
Why Relocate Your Web Site? .........................................................................................................7
When to Migrate From Virtual Hosting ...........................................................................................7
When to Migrate Between Data Centers ........................................................................................8
Factors That Affect Virtual and Self-Hosting ...................................................................................9
How to Analyze Migration Costs..................................................................................................10
Potential Increased Profits.......................................................................................................10
Net Changes in Monthly Expenses ..........................................................................................11
Capital Outlays .......................................................................................................................12
Conclusion.................................................................................................................................13
Chapter 2.........................................................................................................................................14
Preparing for Server Relocation......................................................................................................14
Data Center Selection Criteria.....................................................................................................14
The Relocation Project Plan........................................................................................................17
Coordination Preparation.........................................................................................................17
Customer Communications Preparation ...................................................................................19
Server Area Preparation..........................................................................................................20
Network Preparation ...............................................................................................................22
Server Preparation..................................................................................................................23
DNS Preparation ....................................................................................................................25
Transportation Preparation ......................................................................................................25
Conclusion.................................................................................................................................26
Chapter 3.........................................................................................................................................27
Post Relocation Activities ...............................................................................................................27
Activities During the Relocation...................................................................................................27
Post Relocation Activities............................................................................................................28
Appendix I ........................................................................................................................................31
Relocation Check ..........................................................................................................................31
Data Center Rating Form............................................................................................................32
ISP Rating Form ........................................................................................................................36
Cost Justification Work Sheet .....................................................................................................39
Potential Monthly Profit Improvement .......................................................................................39
Relocation versus Data Center Upgrade Capital Outlays ...........................................................40
Monthly Costs ........................................................................................................................41
Coordination Preparation Check Sheet ........................................................................................43
Customer Communication Check Sheet ......................................................................................45
Server Area Preparation Check Sheet .........................................................................................46
Network Preparation Check Sheet...............................................................................................48
Server Preparation Check Sheet .................................................................................................49
DNS Preparation Check Sheet....................................................................................................50
Transportation Preparation Check Sheet .....................................................................................50
Activities During the Relocation Check Sheet...............................................................................51
Post Relocation Check Sheet......................................................................................................52
Individual Server Worksheet .......................................................................................................53
Individual Server Worksheet (Part II) ...........................................................................................54
Additional Server Routes.........................................................................................................54
Network Enabled Applications List (netstat -a output) ................................................................54
Contractor Qualification Check Sheet ..........................................................................................55
Data Circuit Check Sheet............................................................................................................56
Vendor / Purchasing Check Sheet ...............................................................................................57
Post Mortem Analysis Sample Form............................................................................................58
Customer Information..............................................................................................................58
Incident Summary...................................................................................................................58
Incident Detail ........................................................................................................................58
Customer Impact ....................................................................................................................58
Root Cause Analysis...............................................................................................................59
Corrective Action Plan.............................................................................................................59
Appendix II .......................................................................................................................................60
How to Choose a Data Center ISP .................................................................................................60
Data Circuit Pricing ....................................................................................................................60
Internet Services.....................................................................................................................60
Non-Internet (Carrier) Services ................................................................................................61
Data Circuit Types......................................................................................................................61
Table A3.1 - Common Data Circuit Terminologies .....................................................................61
Data Circuit Provisioning.............................................................................................................62
IP Address Ownership................................................................................................................63
Routing Protocols .......................................................................................................................64
Border Gateway Protocol ........................................................................................................64
Table A3.2 - Common BGP Routing Terms...........................................................................65
Determining a BGP Autonomous System Number .................................................................66
Administrative Tasks Needed to Advertise BGP Routes .........................................................66
Conclusion.................................................................................................................................67
Introduction
The commoditization of many aspects of IT infrastructure, from software and servers to Internet access
and data storage, has created an explosive growth in the use of computers in day to day life. Now
almost every business has considered or uses the Internet as a sales, marketing or operational tool.
The supply and demand of IT services is therefore very fluid as the business and technology cycles
change. The result is a constant reevaluation of IT costs and functionality with the server farm often
being at the center of the analysis.
This book focuses on the need to relocate a website of servers to a new physical location where the
destination data center is managed by a third party. It covers common reasons for such actions, the
factors to consider in choosing a data center, the actual server migration and post relocation activities.
It is a short guide aimed at the busy professional who needs to be aware of the most important
operational activities that need to be done and gives many concrete examples with explanations of
many terms used in these types of projects.
If you have any comments or suggestions about this document, please feel free to contact me via the
LinuxHomeNetworking.com Web site.
Chapter 1
Why Relocate Your Web Site?

Businesses that need to have a Web presence usually begin with a cheap virtual hosting provider
(VHP) that constantly aim to reduce their costs via standardization. Each VHP web server potentially
handles hundreds of web sites, with access to only a single type of application server, database,
shopping cart, blog, web mail or message board forums software suite. Customization usually occurs
through a standard web GUI interface which is usually geared towards altering the work flow features of
the software and not its overall performance. Support is usually only given through instant messaging.
For a simple website with the aim of providing supplemental information to newspaper or web
advertising then basic virtual hosting services, which start at about $10 per month, should be sufficient.
The cost advantage of this service declines as you require additional high end services or
customization. The challenge is to determine the point at which do-it-yourself self-hosting becomes
more attractive than using a VHP.
If you decide to migrate to self-hosting, the next challenge is to determine the daily tasks you will need
to do yourself and those you intend to outsource to third party service providers. You will constantly find
yourself adjusting these responsibilities for a variety of reasons and you may even have to consider
migrating your Web site between physical locations to achieve these goals.
Always remember that the decision to migrate your Web site should be strictly based on business
needs. Embark on it when your service provider threatens the future growth of your company. Plan we ll,
only use proven stable technologies, go slowly, have a backup plan, inform your customers and
minimize your exposure to downtime risk at every step of the way.
This chapter will describe a number of scenarios in which physically migrating a Web site can become
desirable and will cover ways of determining a financial justification for doing so. The rest of the book
will cover the logistical problems of data center selection and preparation, planning, the migration itself,
testing and post migration procedures.
When to Migrate From Virtual Hosting

The lack of support for customization is the VHP's greatest weakness. There are many cases in which
this can become painfully obvious. Here are some typical examples:
1. Expensive Shopping Carts: Sometimes you want visitors to be able to search your website for a
list of available products by name, by category, in a particular price range or from a specific
manufacturer. This requires web pages to be generated dynamically using application server
software that queries a database. This can cost about $100 per month, and if you need the person
to buy the product using a shopping cart, then the price can reach as much as $150 for an entry
level service. In comparison, you can lease a dedicated server for $200 per month in a collocation
data center and if you choose to use Linux, your software procurement costs would be negligible.
Self hosting in this scenario can become desirable if you already have a capable IT staff with
sufficient resources to complete the project wi thin your budget and on time.
2. Unpredictable Software Updates: With virtual hosting you are dependent on your service provider
to provide software updates or patches to fix security, performance or functionality problems. There
may be delays in completing your requests especially if you are one of many customers on a
server. The service provider also has to ensure that the upgrade won’t affect any of the other
websites and this can add delays.
Additionally, there may be times when you need to implement software that needs to be installed
external to your home directory that isn't supported by your hosting provider. Examples of this
8 Relocating Servers Between Data Centers - © LinuxHomeNetworking.com
include a new database product and centrally managed server logins using LDAP. In these cases
the justification for self-hosting becomes even stronger.
3. Server Overload: With hundreds of websites on a server, you run the risk of slow response times
due to one of the URLs owned by another company suddenly becoming popular. The cause of this
latency is often difficult to determine, and correct especially in a shared environment where you
don't have access to many systems tools.
4. Lack of Redundancy: Many businesses rely on a web presence for the majority of their revenues
and cannot afford to have extended periods of downtime. With the use of load balancing devices it
is possible to spread your web hits across two or more servers. The load balancer regularly probes
your servers and automatically steers traffic away from any server that appears to be
malfunctioning or down. This is a useful offering if you need to take an application offline for
maintenance. Many virtual hosting providers don't offer such a service to individual customers.
5. Inflexible Security Services: You may want your applications to run on unique TCP/IP ports and
be accessible only to certain IP address ranges or you may want communications with these
ranges to be fully encrypted over a virtual private network (VPN). This will usually require some
form of VPN or firewall service that your provider may not offer. This requirement may open a
vulnerability to other web sites. For example, allowing FTP access to the virtual server potentially
opens the door to unrestricted file transfers to all sites on the server that share the same IP
address. This may be viewed as a security risk for your neighbors. If you don’t want to risk this type
of exposure, then consider self-hosting.
6. Restricted Supplier Management: You may require highly customized reporting or have
complicated inventory listings which have to track parts, sub assemblies and finished products.
There may be the need to link your shopping cart order entry system, which contains all your
customers' credit card information, with the inventory system of a supplier. Your virtual hosting
provider may be able to do this, but it may expose more of your business to this provider thereby
increasing your risk.
7. Poor Site Availability: All service providers need to schedule maintenance of their servers, but the
time they choose may be inconvenient to you. They may also have unreliable equipment that
adversely affects your site's performance and may not have adequate backups of your data in the
event of catastrophic failure.
8. Insufficient Language Capabilities: VHPs often provide technical support in only a few
languages. If you can't get adequate support for billing, engineering, and customer care services in
your preferred language, then an in house solution may be better. There are many other scenarios
in which a VHP may become undesirable but these examples have provided some of the main
ones you will most likely face.
The next section discusses how migrating existing self-hosted Web site can have very different issues
than those associated with migrating from a VHP.
When to Migrate Between Data Centers

The decision to migrate an existing self-hosted site to a new physical location often has very different
criteria than those associated with migrating from a VHP. This is because the desire for improved
services extends beyond an individual server and encompasses the entire data center facility. Here are
some common reasons for considering this option whether the facility is owned by your company or
provided by a third party.
1. Poor Cooling: As a data center becomes increasingly occupied the thermal load it needs to handle
becomes greater. Your service provider may not have adequate computer room air-conditioning
(CRAC) units to cool the entire floor space and may be unwilling to upgrade due to financial
constraints such as a lack of funding or an anticipated inadequate return on investment. There may
also be cases in which your section of the floor just has poor circulation and other better ventilated
Why Relocate Your Website? 9
locations within the facility are too small or geographically fragmented to comfortably accommodate
your servers.
2. Unstable Power: This can cover a variety of factors from fluctuating voltage, insufficient UPS
backup, ignored preventative maintenance, insufficient circuits in your server area and non-
redundant power feeds to your server area. In all these cases, both the status quo and upgrades of
the service could threaten the reliability of your electricity supply.
3. Insufficient Power: As servers become smaller and more powerful their demand for power can
become enormous. It is now possible to install up to 80 servers in a standard 19" x 72" x 36" rack
with a power load of over 20 kilowatts, that's as much as 20 microwave ovens running 24x7.
4. Unsuitable Physical Security: You may feel that this is limited to access to your server area, but it
should also include accurate entrance logs whether they be manual or electronic, video recording of
physical access to your area, the use of unique keys for all cabinets and customers, and the
deployment of suitable fire detection and suppression equipment. You may want to consider
migrating if any of these factors don't meet your requirements.
5. Inadequate Floor Space: The most obvious shortcoming is a lack of floor space but people often
forget that, in a data center owned by the business itself, floor space used for servers could be
better used for other purposes. For example, a server room located in a downtown office where
rents are high may be more cheaply located in an industrial area nearby.
6. Unreliable Data Networks: The data center you use may have a number of connectivity
shortcomings. The internet service providers (ISPs) used may be unreliable, the networking gear
could be poorly maintained or over-utilized, staff could be inadequately trained increasing the risk of
human error, cabling work could be sloppy, maintenance work that could cause outages may be
unannounced or fall outside your desired time frame, and the equipment used my be obsolete
making it unable to provide desired features that could improve your site's performance.
7. Using a New ISP: Sometimes the Internet IP addresses your site uses came bundled with the data
circuit provided by your ISP. If you are forced to use a new ISP, you may be forced to change your
addresses. This can be a complicated problem which will require many of the steps used in a
physical relocation. It is for this reason that some IT managers may use the opportunity to consider
a complete re-evaluation of the data center strategy which may include a migration of servers to a
more suitable data center.
8. High Cost: If your data center meets all your needs and yet costs more than a comparable facility
nearby, then it may be time to consider your alternatives such as negotiating better monthly rates
from your existing provider or simply go to the competition.
Be especially careful if the unsuitable data center is your own. Though the cost of making
improvements to your data center may be much greater than the cost of moving the servers to a data
center owned by a third party, the recurring costs there could be more than the ones you currently
incur.
Factors That Affect Virtual and Self-Hosting

There are some relocation needs that affect companies that do either virtual or self-hosting.
1. Data Center Consolidation: Companies often inherit data centers when they purchase other
companies. Servers and Web sites can be repurposed or retired to reduce overhead costs and
management complexity. Sometimes data center space can be used more productively for another
purpose.
2. Disaster Recovery: A disaster can debilitate your company especially if all your IT resources are in
a single location. Catastrophes can be caused by huge events such as hurricanes, floods, snow
storms, and earthquakes, or they can be more localized in cases such as hazardous spills and
lightening strikes. Even the simplest things can have a huge impact. Leaking water pipes in floors
above your data centers, faulty fire alarms that force complete building evacuations, employee
sabotage triggered by layoffs, and overloaded circuit breakers can all contribute to downtime that
could be avoided by having a failover data center.
3. Changing Customer Demands: Increased competition and the opening of new markets can shift
the demographics of your customer base. This may require the use of additional IT tools that are
more efficient, less costly, have better performance or have more features. Your existing facilities
may not be able to accommodate these increased demands and you may find yourself having to
expand beyond your existing boundaries.
4. Cost Reduction: New technologies and business processes can reduce many IT recurring
expenses but can also require relocation of your servers. For example it may be possible to
outsource some back office operations such as monitoring, backups and networking services more
cheaply in another facility. Also, as mentioned before, some overhead costs can be lowered by
reducing the number of data centers or servers you operate.
5. Failed Outsourcing: Your IT business partners may fail to meet your disaster recovery,
performance and cost expectations forcing you to seek alternatives such as new outsourcing
providers or to take the work in-house where you have more control.
6. Obsolescence: Outdated facilities can increase your recurring maintenance costs but they can
also not meet industry standards or government regulations. In such cases relocation may be
cheaper than upgrading.
The reasons for relocating can be complex and should be approached carefully. The cost justification,
and risk analysis for doing so can be difficult to quantify. This will be addressed in the sections to follow.
How to Analyze Migration Costs

If you decide to do self-hosting, you should also consider its consumption of your business resources,
namely time, talent and money. The financial cost of the equipment is obvious, but there are resource
costs related to installation, training, staff shortages, consulting, security and long term maintenance.
With an existing IT staff, the strain would be less but if the company is small the price of customization
could be a high proportion of your business overhead expenses thereby making self-hosting
uneconomical.
If you are a small company with limited IT staff, or you have capable staff that would be better utilized
expanding the business, then it may be best to adjust your requirements to fit the services offered by a
virtual hosting provider. You can reconsider self-hosting in future when the customization needs of the
company are more pressing. At this time create a pilot project using only the most essential
customization and if successful, gradually migrate over to a production version of the pilot site. Convert
your pilot to a general testing and staging area and then add modifications to the production site when
you are satisfied they work.
Sometimes businesses should accept the fact that self-hosting, though desirable, may be beyond the
budget and capabilities of their organizations. Switching to a more expensive fully managed hosting
provider that specializes in customizations may have to be considered in such a case.
Migration cost analysis should focus on three broad areas; the potential for increased profits, expected
reductions in monthly expenses and the capital outlay to carry out the migration. These are covered
next.
Potential Increased Profits

Most businesses try to forecast their profit growth. As part of these plans they purchase software
that can facilitate expansion into new markets or save costs and they may even consider testing the
limits of existing IT resources to achieve these goals. Sometimes the driving force to invest in IT
isn't profit growth, but guaranteed revenue. Calculating the potential lost profits by not investing in
information systems can also be a deciding factor. This can sometimes easily be calculated by
determining the impact per hour of downtime on sales and the expected amount of downtime during
the financial year due to inadequate resources. Once you have this information it becomes easier to
calculate the expected return on the IT investment in terms of increased profits or liberated cash
flow.
Net Changes in Monthly Expenses

Net expense reductions can be calculated as the eliminated monthly costs that stem from the
migration subtracted from the forecasted monthly expenses in the new facility. Recurring expenses
to consider in both the new and old facilities should include the following:
1. Maintenance Contracts: Most people primarily think of computer equipment maintenance but
if you manage your own server room there will be many other maintenance figures to consider.
The upkeep of CRAC units, security systems, UPSs, standby generator plants and fire systems
as well as the retention of janitorial and security guard services may need to be included. You
may also have to consider additional IT services such as networking, DNS and database
services.
2. Power: Electrical costs in your own server room may be directly tied to actual consumption, but
third party data centers usually charge for power by the number of circuit breakers and/or
power outlets you decide to use.
3. Cooling: As power consumption increases so do your cooling requirements. Many third party
data centers include the cooling cost into their power charges unless your installation requires
major expansion of their facilities. In this case you may be expected to either directly contribute
to the expansion or commit to a long term contract.
4. Floor Space: In your own server room this is a fairly straightforward cost. With a data center
provider you may be offered floor space as a full computer cabinet or only part of it. In some
cases you may be offered an existing caged area which may not exactly meet your needs or
you may have to sign a long term contract before the facility will agree to creating a custom
space for you.
5. Bandwidth: When you convert to self-hosting the cost of your Internet connection will most
likely become more variable in nature as it will fluctuate with the amount of bandwidth you use
each month. Appendix II discusses many of the factors in selecting a data center ISP in more
detail.
6. Staffing: Self-hosting will inevitably require additional staffing resources. You may have to hire
additional employees, either as staff or contractors, or you may have to train existing staff
members continuously and raise their salaries to match their new skill set. In the event of data
center consolidation you may have less staffing needs too.
7. Security: Physical security expenses will largely be covered under various security system
maintenance contracts and the hiring of security guards. If you are using a third party data
center, then this may be included in your floor space expenses. Data security is similar. This
may be provided as a contractual service or handled by yourself; it may also be included in one
or more maintenance contracts. Your budgeting should include the monthly cost of backup
services, replacement tapes or other media, off site storage, software update and anti-virus
services, and firewall and intrusion detection services.
8. Travel Time: It may seem trivial, but always try to determine the cost of commuting to your new
server location. Most website maintenance can be handled remotely, but wh en things go wrong
or you have new physical installation work to do the cost of travel delays can become
significant. You may decide on a cheap data center located an hour away by car but if a device
begins to intermittently fail, sometimes during rush hour, the duration of outages can jump
dramatically and so will the cost of lost revenue.
9. Equipment Leasing: Migrating to a new facility may also require leasing additional hardware to
accommodate the move. Some data center providers will also throw in equipment leases at
favorable rates to get your business but they won't allow you to leave with the equipment
should you decide to go to a competitor. It may also be a good time to renegotiate the
replacement of existing obsolete equipment with newer models. This could affect your
operational costs at the new location and should be factored into your calculations.
10. Systems Development: In a virtual hosting environment the software used to make the
management of your site easy is rolled into the cost of the service. With self-hosting this could
become an additional expense as you may have to begin a continuing project to provide the
same features to your internal users.
With the management of your website more firmly within your control you may also find yourself
having to integrate its operation more closely with other business systems within the
organization. This could add additional hidden expenses to the project. Always try to separate
the migration savings from the additional expenses of this type of work. It is easy for a profitable
change of web service providers to be weighted down by the allocation of expenses not directly
related to the activity.
Subtracting the net change in monthly expenses from your profit projections will provide an overall
expected monthly cash flow from the project. This then can be used to justify the one time capital
outlays required to get the job started.
Capital Outlays
There are a number of one time costs that will be encountered at the beginning of the migration that
you will have to account for. These include the following:
1. Leasehold Improvements: You may have install the infrastructure to support your web site
being moved to the new location if you don't have it already or if it's not supplied by your new
hosting facility. This could include additional power, cooling, fire systems, backup systems,
UPSs, computer cabinets, racks and shelving, fencing, patch panel cabling, standby generator
plants, cameras, keyless entry systems and raised flooring to name a few.
2. Transportation: The cost of physically moving should also be included. You could use a
professional moving firm or you could rent a truck. In the latter case you may also have to rent
secured shelving, on which the computer equipment will be placed, and mount it in the truck.
You may also be forced to rent or buy boxes in which to transport your equipment and other
miscellaneous items.
3. Overtime: Most web site migrations are usually done after business hours at night and on
weekends which may require the payment of shift premiums or additional vacation time to
compensate your employees for the inconvenience of the working on the project.
4. Staffing: In the event of data center consolidation you may be faced with increased staff costs
due to layoffs or rapid attrition which may demand unexpected re-hiring costs. Relocation or
expansion activities will also increase staffing costs due to the need for increased numbers of
employees.
5. Contractor Fees: Contractors may not just be used to install leasehold improvements but they
could provide services specifically for the move. These could include project management and
data migration and recovery skills your organization may not already have.
6. Training: This is a frequently over looked expense that is often rolled into monthly costs, but
you may require your staff to get several foundation training courses to allow them to handle
the new tasks they will be expected to achieve. Some people will treat it as a recurring
expense, others may want to account for it as part of the moving cost.
7. Equipment Acquisition and Temporary Leases: If you are moving between data center
providers you may be faced with incompatible equipment at the new location. For example, the
data backup formats used at the old and new facilities may be different forcing you to lease a
tape backup unit just in case you have to restore data in the event of a server failure during the
migration.
You may also be forced to use new servers which may add unexpected problems. The software
you currently use may be incompatible with newer technologies or you may not have any
remaining staff that knows how to reinstall your existing software on the new hardware. This
could force you to purchase brand new versions of the software before you can proceed.
8. Cleanup Costs: You may have to restore the original server area to its original condition. This
could require the removal of racks, cabinets, raised flooring, power feeds and networking gear.
You may have to get professional cleaning help not just from janitorial services, but also from
technical professionals who are capable of decommissioning IT infrastructure.
9. Penalty Fees: Some contracts have early termination fees and these may apply to some of
your IT infrastructure. Be aware of the cost and plan your migration to possibly limit the extent
of your obligations.
10. Movers Insurance: You may have to insure some of your equipment against damage that
could occur during the relocation. This may add further delays and costs.
Once you have an idea of the capital investment required for the migration, you can compare it with
the expected returns in the form of monthly savings or increased profitability. If the return on
investment is better than the interest you can get at a bank, then the project is probably worthwhile.
Even if the project has marginal profitability, it may be justifiable on the basis of making your
company more responsive to market pressures especially if you are moving from a VHP.
This type of financial analysis is often ignored and yet it is so important in making a reasonable
decision to switch service providers. The extra time it takes to do it could save your company
thousands of dollars in the long run.
Conclusion
The decision to migrate your Web site can be difficult. You have to weigh the benefits of increased
control, improved flexibility and reduced costs with that less complicated management and technical
skill requirements. Whatever you choose to do, plan carefully. Always get a professional opinion, even if
it's informal, and always be aware of the potential risks of the decision you make. If you decide to do it,
Chapter 2, "Preparing for Server Relocation" will provide a lot of guidance in completing a successful
project.
Chapter 2
Preparing for Server Relocation

The rationale for deciding to relocate or consolidate data centers was discussed at length in Chapter 1
"Why Relocate Your Web Site?" This chapter explains in detail the criteria you should use to select your
new data center and create a project plan. A lot of information will be covered and the numerous action
items mentioned are included in the work sheets in Appendix I to help make the process easier.
Data Center Selection Criteria

There are two broad categories of data center providers. The first only supply computer room floor
space, access to an ISP, basic monitoring and power. These are called collocation providers. The
second group provides more comprehensive management that may include all possible IT services
related to your site including systems development. These are called managed hosting providers. There
are a wide range of varying service levels in between and the interpretation of the terms within the
industry can often be very loose. Always request a very specific list of the services your data center
provides as part of your selection process.
As expected, the selection of a suitable data center will play an important role in any data center or web
farm relocation project. There are many factors related to the facility and its services that need to be
considered that are often overlooked. These include:
1. Location: The data center should be positioned away from zones at risk from natural disasters
such as flooding from rivers and dams, hurricanes and earthquakes. It should also be no closer
than a quarter kilometer away from major highways and railroads to reduce the evacuation risk from
toxic spills. Locations close to hazardous production facilities and aircraft flight corridors should be
avoided.
Your employees may have other personal interests in the location such as the presence of
reasonably priced housing nearby, recreational attractions in the area, access to public
transportation, and the availability of amenities such as schools and parks in the neighborhood. You
should monitor how traffic patterns affect the ease of accessibility to the site to see whether they
are unsuitable.
The immediate vicinity of the site is also important. Rainwater should drain away from the building
and then off site to prevent localized flooding. In high security environments the building should be
surrounded by embankments and perimeter fencing, reducing the risk physical attack.
2. Communications: The facility should have access to multiple ISPs with the cable entering from
different points of the building. This reduces the risk of outages due to a technical failures as well as
construction and landscaping accidents. Verify the roof access rights in the event you need to have
a satellite or microwave line of sight antenna installed.
It is also extremely important to visually verify the type of connectivity you have. Be certain that both
the ISPs that enter the building and the types of data circuits they can provide are suitable. Don't
sign a contract with an ISP where you are held hostage to unsuitable or otherwise inadequate
connectivity. This is discussed in greater detail in the appendix.
3. Electrical: Power should be supplied from multiple feeds from different substations. The facility
should also be able to run without interruption if its largest standby generator or UPS are offline for
maintenance. Ensure that the building has sufficient excess capacity to handle future growth.
In large facilities, the UPS feeds a network of power distribution units (PDUs) to supply each
section of the floor with a series of circuit breaker panels. Make sure that every rack or cabinet you
intend to use has access to outlets from at least two PDUs and that each PDU is operating at no
more than 45% so that it can handle the full load of the other one if it fails.
Appendix II - How to Choose a data Center ISP 15
Request a history of outages or other irregularities in the feeds from the site's utilities and ask how
you'll be notified by the facility of any electrical maintenance work to be carried out by either
themselves or their providers. The facility's staff should also be automatically notified by monitoring
equipment of any disruptions in the power supply to the area.
Ask how quickly the generators respond to an electrical outage and how long the UPS batteries can
last. Inquire about whether the UPSs have ever supplied the full load of the data center and when
last the system, including the batteries, was last maintained. Standby generators can be regularly
started without revealing any apparent problems, ask whether testing includes the use of a load
bank to simulate the power consumption of the data center. Investigate how frequently the
equipment is tested and how often it is maintained.
Verify that the power per square foot that the data center can provide meets your needs. Racks of
densely packed servers and data storage can be power hungry.
4. Cooling: Most data centers try to maintain a 75F/25C air temperature, verify this. On your plant
tour be on the lookout for computer room air conditioning (CRAC) units that squeak or rattle loudly
as it could be a sign of poor maintenance. Condensation from CRAC units should be drained away
immediately through piping, be on the lookout for water leaks.
5. Security: Verify that there is 24/7 security enforcement. This should include offices and common
areas being isolated from the data center floor, mandatory visitor/employee registration or
electronic ID access and interior/exterior video surveillance. Some data centers also link visitor ID
cards with a person's biometric information through the use of a palm reader. This helps to deter ID
card fraud.
6. Fire Protection: Not only should there be smoke and heat detectors, but they should be linked to
an alarm panel that graphically shows the location of the fire on the building's floor plan. The first
line of defense should be a gaseous system that suffocates the fire by displacing the oxygen in the
air. These systems are less damaging than water based ones but they are usually designed for
fires of short duration.
Larger fires will often require a pre-action water based system. Here the pipe lines are pre-filled
with pressurized air to reduce the risk of flooding during normal operation. Water only enters the
piping after an alarm signal has been detected, then the sprinklers release the water only after a
pre-defined temperature has been reached. False alarms are minimized by requiring two events to
occur before the system is activated. This is an industry standard method of fire prevention and it
should be on your checklist.
If your data center is situated on raised floor tiles, you should ask whether there are liquid detectors
underneath. This helps to prevent problems due to extinguisher and CRAC unit leaks. Also in this
case, make sure that the cabling lies in trays above the floor out of harms way from minor flooding.
If possible, the server area should also be isolated using fire proof doors.
7. Network Connectivity: Not all data centers will provide you with Internet connectivity. Some will
only have a demarcation point where ISPs have placed their equipment. You will then have to
contract with the ISPs to extend a data circuit to your server area. Connectivity can become more
complex than it first appears. There are different types of data circuits requiring varying types of
adapters on your network equipment.
If you require only one link, then you'll need to configure a single default gateway on your network
equipment to get to the Internet. When multiple links are required, you'll need to configure a
dynamic routing protocol on your network equipment. This will automatically calculate which of the
many links will get to the data to its final destination most quickly. It can also be used to bias traffic
to and from your web site on the cheapest ISP link and will automatically fail traffic over to the
remaining ISP circuits if one of the other circuits fail.
Detailed discussion of typical network connectivity issues usually requires the services of a network
engineer and is beyond the scope of this chapter. Appendix II will cover many frequently used
terminologies and scenarios to help you evaluate your options better.
8. Network Monitoring: It is often taken for granted that your data center provider continuously
monitors its equipment for failure. Ask about the frequency of the checks. A polling cycle of five
minutes or less is generally acceptable. Also ask about the types of checks done, ICMP (Internet
Control Message Protocol) or "ping" tests only check basic network connectivity and server
response. The facility should also use SNMP (Simple Network Management Protocol) to track CPU,
memory, error and data throughput rates. It is possible for SNMP enabled systems to send
notifications, or "traps", when components fail, or a predefined event, such as high CPU usage,
occurs. This information should be fed into some form of a job ticketing system that will ensure that
the problem is fixed quickly. Ask about the number of failed polls that will trigger an alarm and
whether they too will automatically generate a ticket.
9. DNS: Ensure that your data center uses multiple DNS servers, behind different firewalls, in multiple
locations to prevent your web site from being affected by one of the servers going down. Some
facilities will provide not just caching DNS for the exclusive use of your servers, but also
authoritative DNS services to handle Internet queries for your Web domain. With authoritative
services, ask about the procedures for updating DNS, the lead time for requesting changes and the
format of the DNS data the provider will need to enter it into their systems.
10. Customer Support: Ask about the availability of a web portal through which you can view
statistics, billing, contact, and server information related to your site. Also ask about the times
during which scheduled maintenance is done and the types of notifications that are provided.
Request a summary of escalation procedures used when problems occur and whether there is a
formalized means of documenting and permanently fixing problems. From time to time you may
need simple services such remote hands on help in rebooting a server or changing a backup tape.
Ask about the availability of such services and possibly more complex ones through an as-needed
contractual or longer term retainer based agreements.
11. Data Backups: The backup system you are using at your current location may be different from the
one used at the new facility. This could be the source of difficulties if you have to restore historical
data during or after the relocation due to server failure or human error. Verify whether the new
facility can handle data backed up using your software on your backup media. If not, you may have
to invest in data conversion services with a third party. Good backup services usually store data for
a predetermined period of time before reusing the media. They should also store most of the data
at a secured secondary facility. This protects the data from catastrophic events at the main data
center. Verify that this type of extra data security exists.
12. Floor Space: For improved safety, the data center floor should use anti static tiles to reduce the
risk of electrostatic shock damaging your equipment components. Water pipes, steam lines,
bathrooms, kitchens and other sources of moisture should all be located a safe distance away.
Also, they should not be directly above the area. You should also determine whether the location
has sufficient floor space to handle our current and future needs. Some facilities allow you to
reserve the area immediately surrounding your server area for future expansion.
13. Facility Cost: This factor can present itself in many different ways to include pricing for bandwidth,
power, cooling, security, floor space rental and custom services. It is a good idea to determine what
the total costs would be over the time period you expect your current website architecture to be
used as the costs can be presented as recurring and/or one time expenses. Lower recurring costs
can easily give the perception of cheaper operating expenses but the price may become
unfavorable when higher setup fees are taken into account.
Remember that this is a perfect wish list. The data centers in your vicinity may not meet all the criteria
but the list should allow you to reduce your final candidates to a manageable number. Data center
selection is only the first phase of the physical planning for the relocation and will largely be the
responsibility of your facilities and networking teams. The work that will follow will demand a lot more
from your IT support staff and will have to be carefully coordinated as we will see in the following
sections.
The Relocation Project Plan

Detailed logistical planning of all the steps related to the relocation needs to be started well in advance
of the deadline date. You'll probably need to start with a number of meetings to inform each of the
affected groups about the project. These will have to be followed by project planning meetings in which
roles and responsibilities are assigned and progress reports given. As the deadline date draws near or
as the complexity increases, be prepared to schedule daily and sometimes twice daily meetings to
achieve your goals.
There are many aspects of the migration that need to be thought about prior to arranging the first
meeting. Some of the most pressing ones will now be discussed.
Coordination Preparation
There are a number of things that need to be considered prior to setting up specific functional
groups for each aspect of the relocation. These are discussed next.
1. Project Management: Have a single overall project manager for the activity. If the project
starts to become complicated invest the time in tracking it with software tools such as Microsoft
Project. Spreadsheets can track static information well but do relatively poorly in monitoring the
status of dynamically changing deadlines.
Constantly changing priorities can be disruptive. Plan to include deadlines after which time no
further changes may be made. Set up meetings on these days to determine whether the project
or sub-project should be aborted, continued as planned or given a preparation time extension.
2. Roles and Responsibilities: Create an activity checklist that assigns each member in the
team clearly defined roles and time frames in which to get activities done. Specifically assign
someone the task of keeping track of the problem equipment that may fail. The project
manager will inevitably be distracted by other events and this will help to ensure that forgotten
technical issues don't threaten the success of the migration. There should be persons to lead
transportation, networking, server shutdown/startup, application testing, customer
communications and the locking of the doors at both the old and new facilities once the
relocation is complete.
An often overlooked role is one of the "gofer", someone who will go for anything that you have
forgotten. It could be to buy cables you forgot to order, pickup catered food, or to find the
software CDs that "must be somewhere over there in those boxes". Remember to give them
some small reward when it's all over as it is one of the most thankless jobs.
3. Disaster Recovery Team: Have a group of persons assigned to disaster recovery. It should
include staff that is familiar with systems administration, database administration, networking
and backups. These persons don't necessarily have to be sitting idly by waiting for something
to break. They can still play important roles in the preparation steps, but should be given a
reduced workload during the relocation itself that will allow them to dedicate their time to such
activities.
4. Procedures Documentation: There are three types of procedural documentation that will have
to be up to date. The first relates to those used by your existing systems which won't change as
a result of the relocation. The second would obviously be the documentation for systems that
will change after the project is over.
The third type is equally important. It is the documentation of the steps each participant is
expected to do during the relocation. As part of the definition of the roles and responsibilities,
some participants will need to have a detailed task list to help prevent them from making errors.
These would include step by step commands that a technician or engineer would need to
execute as part of the process. This person can cross this activity off their check list when the
tasks are completed for better control of the change process.
5. Interpersonal Communications: Make preparations to have a permanent conference bridge

open so that all members of the team can be better coordinated in the event of a crisis. Make
sure all active participants at the time of the migration all have mobile phones on their person.
6. Participant Lists: Have a complete list of participants in the relocation. This should include
their work, mobile, and if possible, their home phone numbers. It should also include contact
information for all third parties involved with the activity such as movers, technicians and
contractors. This list should be distributed to the entire team.
7. Contractors: You may need to use contractors to do some of the work your staff may have
neither the time nor ability to do. They should be qualified, experienced and authorized by the
manufacturers they represent. Contractors should also use the correct tools and be able to test
the quality of their work. A check list for contractors is provided in Appendix I.
8. Vendors / Purchasing: One of the most difficult aspects of a data center move is the
coordination of purchases from your vendors. There are many things to track. Items have
varying delivery lead times, you may forget to order something, items may have to be returned
or replaced, and deadlines may shift. A sample purchasing check list for purchases is provided
in Appendix I. You may want to adapt it to a spreadsheet format to make it easier to share with
your vendor.
9. Inventory: You will have to do a complete inventory of all the equipment to be moved. This will
also have to include "before" and "after" data related to the network connectivity and physical
location of each device. The actual required information for each type of equipment will be
covered later in the chapter and accompanying check lists are provided in Appendix I.
Record this information in a database if possible. It will allow for very flexible reporting including
individual status and data sheets for each device. Not everyone will have access to the
application, so ensure that it has the capability of creating word processing or spreadsheet
versions of the reports for more universal distribution.
10. Equipment Leases: Some relocations cannot afford any downtime at all and you be forced to
purchase or lease equipment to create a duplicate environment at the new location. You may
have to assign the acquisition of such equipment to a team lead and adjust your budget
accordingly.
11. Relocation Date and Time: Determine the best time for the migration. If it takes place at night,
and/or over an extended period, allow for overtime, catered food and possibly compensatory
time off. For nighttime moves, make sure your daytime skeleton staff is capable of handling
regular business issues and can relieve the night staff of some of the technical problems that
may arise. Verify that there won't be delays due to rush hour traffic or road maintenance at the
planned time.
12. Failed Equipment Identification: Have some way of marking equipment that isn't working. A
brightly colored sticky note stuck to a server and the rack or cabinet in which it is located is
usually sufficient. This makes it very easy to identify broken equipment from a distance. Make
everyone aware of this process.
13. Plan of Retreat: Create a plan of retreat in the event that things go dreadfully wrong. Create a
shortlist of scenarios during the actual relocation under which the project cannot go forward.
Have some mechanism of informing everyone of the decision. Create a list that defines the
sequence in which servers should be returned. Also identify a point of no return at which you
cannot roll back your changes. In this case create a minimum list of servers that need to be
functioning for the website to be adequately operational. If things go wrong, ensure that these
servers are functioning correctly.
14. Practice Migrations: Plan to do a practice relocation of some non critical servers to see
whether you are really prepared for the full scale operation. These are some of the general
preparatory tasks that need to be done and you may have to add a few that cater to your
unique needs. It is a good first step before proceeding with more specific plans.
Customer Communications Preparation

Notifying your internal and external customers of the expected changes will be critical to the
success of the project. Consider these activities as part of the plan whether the relocation is
successful or not.
1. Customer Notifications: Provide ample warning of the impending activities so that your
customers can plan for the change.
2. Targeted Messages: Have a single message with varying degrees of detail for each customer
group depending on their information needs. For example, web surfers may need to know that
your site may be unavailable for maintenance for a specific time period but business partners
may need to know about any new procedures the change may create. The message should
explain the reasons, expected features and associated benefits of the change. Plan to have this
message delivered in person to your most valued customers. Make sure it is delivered to all
internal departments that could be affected by the planned activities.
3. Contingency Planning: Create a communications plan in the event of a failure. Here is a list of
guidelines I strongly recommend be followed:
> Create a web page or blog to provide updates on the progress of problem resolution. This
should also include contact information for your crisis spokesperson.
> Provide sufficient information to make all affected parties aware that you're taking the
matter seriously. Describe the extent of the problem and a statement that addresses any
possible concerns of those who may not be affected.
> Update this page at predictable stated intervals even when there's nothing to report.
Significant events outside this schedule should be reported immediately.
> If the issue has public visibility, have a video or audio clip of the company President/CEO
making a factual statement about the matter that expresses remorse, and a promise to
have quick resolution.
> Create lists of potential questions, answers and discussion points for those parties that are
affected, not affected, business prospects, the press, and analysts.
> In times of crisis, personalized verbal assurances and updates can be valuable. Attempt to
give every customer a personal phone call. Use the Q&A discussion points. The callers will
have to be prepared to take some abuse. Have them allow the customers vent while
sticking to the script. Senior management should make some calls too to gain first hand
understanding of the pain everyone is feeling.
> Don't forget to contact customers that are not impacted. Focus on the most important ones
to provide them with personalized verbal service assurances. Send e-mails to all other non-
impacted customers with statements about service continuity.
> Have senior management call important sales prospects to provide justifications to choose
your company in spite of the disruption.
> Use a company wide all-hands meeting to provide a first hand situation status. It should be
brief as there will be lots of work to do. Provide regular email updates to employees via e-
mail.
> You should also have someone monitoring the web media and blogosphere to get
feedback on what people are saying, and who's saying it. When comments are necessary,
simply reference your updates web page. Use this updates page to correct misstated facts
neither making excuses nor being defensive. Only authorized persons do such limited
commentary on blogs.
> Contact key press and analysts. They will appreciate hearing directly from you even if they
remain highly critical.
4. Lessons Learned: Some aspects of the migration may fail and documenting the issues can
lead to much better experiences in future. Develop an easy to use post-mortem template that
can be used to document any failures related to the migration. It should include the persons
involved, dates and times the events occurred, reasons for the failure, error messages, the final
solution and steps that will be taken to prevent the recurrence of the problem. This information
should be made available to all members of staff involved in the migration and to customers
who may demand detailed explanations of the cause of disruptions.
The establishment of clear channels of communication with your customers is always important but
especially so during projects of high risk. Always include them as part of any relocation plan to
ensure a more complete success.
Server Area Preparation

The health of your server farm depends on the quality of your physical infrastructure. A poorly
prepared area can cause unacceptable delays and even a complete site shutdown. Follow these
steps to reduce your risk.
1. Power Management: Your servers and disk storage will consume the most amount of power in
your environment. Do an audit of the power consumption to determine how much you will
require in the new location. Get a total figure and an estimate of what you expect to consume
per rack or cabinet. Verify that the servers in each cabinet or rack won't overload the power
circuits supplying them. Some power hungry devices may need unusual voltages or electrical
connectors, double check this information ahead of time.
Make sure each rack or cabinet can receive power from redundant PDUs and that there is
adequate excess capacity on the PDUs to support not only your server farm but also the failure
of one of the PDU units.
Have the facility's management prove that you are getting UPS protected power in your area.
It can be very frustrating to arrive at the new facility to discover that the server power cords
used at the old location are too short for the racks in the new one. The problem usually occurs
when converting from fixed racks to ones in which the servers are mounted on rails. This allows
you to slide the servers out into the aisles for better access but requires cables that can stretch
the distance. Verify that you have sufficient quantities of adequate cables.
2. Cooling: Verify that the area has an adequate number of CRAC units to cover your anticipated
power load. The rule of thumb is that each watt of power consumed by a server requires a watt
of cooling. Once the migration is completed you'll have to test air temperatures and humidity to
ensure they meet the requirements of your equipment.
In raised computer room flooring, hot air is extracted from the room, chilled and then returns to
the server area under the floor blowing up into the cabinets through vents in the floor. The floor
under the tiles should therefore be clean and generally clear of obstructions such as cabling
and ducting. If possible baffles can be placed under each CRAC unit to guide the air flow in the
direction of the cabinets it needs to cool. In some cases the floor under the cooling zone of the
unit may need to be sealed off to force the air only to the required cabinets.
You may also find yourself in a situation where the overall cooling requirements of the server
farm are within the specification of your combined CRAC units but certain concentrations of
servers within the farm could overtax the capacity of individual units. Plan to spread these high
power density racks across the server floor to help balance the load across all CRAC units.
3. Labeling: This is very important. Ensure the power outlets are labeled with the PDU and circuit
breaker number. This is especially important for systems with dual power supplies that should
be plugged into separate power sources. Make sure power cables are labeled with the name of
the server at both ends too.
Make provisions to have all servers labeled on the front and the back to reduce the risk of
incorrect cabling and likelihood of making a mistake with a hard (power cycle) reboot in the
event of an unexpected server failure. Also make sure that all network cables are labeled at
both ends.
How do you start to number? Numbering schemes for cabinets and racks are usually straight
forward. Split up the server area into zones serviced by the same patch panels or switches.
Each zone will have a number of rows of racks and/or cabinets. A location number such as 1-
11-4 could mean zone 1, row 11, cabinet 4. You should also label patch panels in a similar way
so that 1-11-4 p7-2 would refer to the 2nd port of the 7th patch panel in cabinet 1-11-4.
4. Diagrams: Create diagrams that map the precise layout of servers in the racks and pre-install
shelving and rack kits at the expected locations. It is a good idea to put the heavier equipment
at locations near the bottom as this will make them easier to insert and remove. Make room for
monitors and their KVM (Keyboard, video monitor, mouse) switches also.
5. Mobile Monitors: Video monitors on carts should also be available at the new facility for
troubleshooting servers that need to be removed from racks or cabinets.
6. Rack Usage and Orientation: Install the servers in the same direction in the racks. This will
make all power and network cabling connections reside neatly on one side of the racks.
Servers on opposite sides of an aisle should either face each other or be back to back. This
creates a better cooling environment as the hot power supply exhausts of one server won't be
sucked in by the front facing air of the server behind it. CRAC units extract air through filters on
the top of the unit and blow chilled air through vents at the bottom. It is for this reason that
CRAC units should be placed in line with the hot aisles so that the air can easily be extracted
from them. When regular flooring is used, you may require ducting to blow the chilled air into
the cold aisle. With raised floors, the CRAC unit vents are physically below the floor level
blowing air up into the server cabinets. In this case the baffles and sealed floor techniques
mentioned in this chapter would help channel the air flow better. Sometimes with raised
flooring, the air blown up through the cabinets is insufficient to cool the servers and perforated
floor tiles need to be placed in the cold aisles for added cooling. Remember that perforated tiles
located in hot aisles are counter productive as they will help to cool air the servers never use.
As expected, the servers should be stacked vertically. When cabinets are used you should
insert unperforated blanking panels in any spaces between the servers to better channel the
cooling air from the front of the cabinet to the hot aisle in the back. Without the blanking panels,
the usual swirling vortices of exhaust air can easily be blown back to the front of the rack
through the spaces.
Server cabinets come in a variety of widths, the most common one being 19 inches wide.
Sometimes the walls between adjacent cabinets are removed to facilitate cabling. This can
affect the correct channeling of the cooling airflow through the servers and can usually be
avoided through better patch panel layouts ahead of moving time.
Remember to make the aisles wide enough to allow people to easily mount and dismount
servers in them.
Finally, some types of equipment may be too heavy for regular server cabinets and will require
the use of racks as an alternative source of support. This equipment will need to be identified
and located accordingly.
7. Network Cabling: Determine your network cabling requirements based on your server layout.
You may have to install patch panels to connect the server racks and cabinets to those
containing your network equipment. You would then connect your server to a patch panel port
in its rack with a standard network cable. This port is in turn connected to an equivalent port on
a patch panel in the network rack. By using another standard length cable you can extend the
connection to your network gear from the network rack patch panel. You may have to plan for
the purchase and installation of such a system.
Remember to consider the use of both copper and fiber connections. Copper Ethernet cable
used for 100 Mbps communication can be no longer than 100m in length. Make sure that the
combined length of your connections, via your patch panel system, does not exceed this length.
Multimode fiber has a maximum distance of 2Km when running at 100 Mbps and between
220m and 500m when running at 1 Gbps.
Glass fiber cables for servers are delicate in comparison to copper. Wherever possible ensure
that they are run in separate cable trays to help prevent possible damage. Also make sure that
the power cables run in separate trays or conduits from the data cables to reduce the risk of
damage and electrical interference. To further reduce the risk of damage, cables shouldn't hang
in the air or be stretched taught.
Bundled data cables should be wrapped together with Velcro, and not plastic, tie wraps to
make it easier to add additional wiring to the bunch. The bundles should be run to the sides of
racks and cabinets so as not to impede airflow.
8. Tools: Make sure each person responsible for the racking of the servers has a correct set of
tools. The most noticeable time saving tools will be electric screwdrivers. Have many and also
have lots of charged replacement batteries.
9. Facility Access: Verify that each person that is going to have access to the server area has
key access and parking rights at the data center beforehand.
10. Internet Connectivity: Ensure that your Internet connectivity to the area has been secured.
Verify that the network links have been installed and tested prior to your migration date. Some
sites require T1 data circuit links to credit card facilities or VPNs to remote offices. Make sure
these are in place and tested before the move too.
The entire relocation depends on the proper preparation of the server area but fortunately you can
save time by simultaneously preparing for other aspects of the move. These will be explored next.
Network Preparation
One of the most obvious reasons for having redundant network hardware is to help protect against
hardware failure causing your Internet connectivity to fail. Another equally important reason is to
help in server farm relocations.
Redundancy allows you to shutdown network equipment, move it to the new location, and
preconfigure it in anticipation of the server migrations. That isn't all, there are more preparations
that need to be done.
1. Equipment List: As mentioned previously, do a complete inventory of all your networking
equipment. Create a comprehensive list of all the important networking information that will
change as a result of the move.
2. Diagrams: Ensure you have a complete set of network diagrams that include each server and
network device that will be relocated. They should include every IP address, switch port,
gateway, route, and ISP circuit number.
Have separate drawings that clearly show how the network cables plug into the switches from
each server. This will help illustrate whether too many of your servers are vulnerable to the
failure of a single network device. Servers that play similar roles, such as database servers,
should be directly connected to different switches.
3. Preconfigure: Setup your new network equipment at the target data center and test
connectivity ahead of time. Connectivity should include tests from the Internet and practice
servers at the new location. Make sure your routing, access control lists, VPN tunnels and
firewall rules all take the IP addressing scheme you will be using at the new location into
account.
4. Monitoring: Special attention should be given to network monitoring. Verify that you'll be able
to switch monitoring from your old server address to the new one seamlessly.
5. Data Circuits: Keep close track on the provisioning of data circuits for the new location so that
they are installed prior to the migration date. These circuits should not only be sized to capably
handle your expected data transfer rates but also tested at various times of the day to ensure
your ISP has met their contractual commitments. Some types of equipment require modem
lines to provide emergency out of band technician access in the event of an emergency. This
would require the additional installation of one or more POTS telephone lines.
6. Cabling: Make sure that network cables are all re-labeled to reduce the risk of human error
when the servers are reconnected to their new network.
7. Logging: Many managed networks have centralized error logging and authentication servers.
Make sure your relocated network devices can continue to do so.
8. Backups: Create copies of all your network configurations, both old and new prior to the
relocation. The old ones will be helpful if you have to quickly roll back the work to the original
data center. The new ones will help protect against hardware failure in your new facility.
9. VPNs: Some corporate offices use VPNs to gain access to their Web server farm. If possible,
terminate some test VPN tunnels on the network equipment at the new location ahead of time.
Once the migration is complete you'll have to plan for recreating a redundant network
architecture in the new location. This will be covered in Chapter 3, "Post Relocation Activities".
Server Preparation
Server preparation for the migration is probably the most complicated task because there are
usually many of them with each running multiple applications that rely on the functioning of varying
components. Follow these simple steps to make the job easier.
1. Server List: Do a complete inventory of all your servers. Create a comprehensive list of all
important server information that will change as a result of the move. This can be recorded in a
simple spreadsheet and would include IP addresses, subnet masks, routing gateways, backup
server IP addresses, the switch ports the servers will use and server rack locations. It should
also include information such as the server's name and serial number for inventory purposes.
Each server should also have its own separate worksheet document that contains all its
relocation information. This would be attached to the server so that the engineers working on it
would be able to instantaneously reconfigure it when it arrives at the new location. Samples of
both documents are available in Appendix I.
2. TCP/IP ports: It may seem tedious, but get a printout of all the TCP/IP ports on which the
server is listening and also which clients have active connections to your server. This can be
done with the netstat -a command in most operating systems, including Windows. This will
help you to identify the applications that should be running before and after the relocation and
can be used as a quick check to detect any unexpected failures. It will also be helpful in more
precisely restricting the TCP/IP access between servers on your new networks. Finally, it will
help to identify inter-network application dependencies between servers which can be used to
determine the servers that should be relocated together as part of the same group.
Most operating systems can also give you a snapshot of services or applications that should be
running on startup. Get printouts of these for each server as part of the server's more
comprehensive post migration system check.
3. Routing Tables: Note the routing tables of all servers before the migration using the netstat
-nr command. Determine what the new routes should look like at the new location and note it
down. This is especially important for noting the default gateways and also for analyzing routes
on servers with either multiple NICs or routers.
Note: A server should have only one default gateway. In Linux and UNIX systems there is only
a single place to enter this value. Windows servers provide the option of having a default
gateway per NIC. In this case, make sure that only one NIC has a default gateway configured.
4. Data Backups: Archive all your server data. Make sure that you have a data restoration unit at
the remote location that will be able to restore your data from your backup media using the
same software. The recent advent of portable external hard disks using USB 2.0 connections
could simplify smaller backup and restoration work.
Large databases are often stored on storage area network (SAN) and network attached storage
(NAS) devices and are too large for the USB solution. In these cases data restoration by tape
can be excessively long which can make this an option of last resort. With SAN and NAS data
bases you may need to lease a duplicate device and clone your data to it. Disaster recovery
can be much faster as the secondary device will be preconfigured to replace the failed primary
one.
Another option is to restore the data ahead of time on the new NAS / SAN equipment located at
the new facility. You can then set up a data circuit between the old and new data centers so
that any new transactions can be replicated between the two. At any moment in time the data
bases at the two locations will be synchronized.
Always remember to do practice data backups and restorations for key servers and
applications.
5. RAID BIOS Settings: An often ignored item is the server's BIOS settings. The regular
parameters are usually easy to determine as the defaults are usually sufficient. The real
problem is with the BIOS metadata on hardware RAID cards. This metadata lists all the drives
in each RAID set, the order in which they are accessed in the RAID set and the type of RAID
being used. This often cannot be guessed. Schedule a server reboot before the relocation and
enter the RAID controllers BIOS setup to record this information. Without this simple plan, a
sudden jolt of a RAID card's loose onboard battery backup could cause you hours of downtime.
6. Spare Parts: If possible, have a set of spare servers that can be used for spare parts or
complete system replacements in case of failure.
7. Customized Procedures: In many cases applications on servers have to be started or
shutdown in a particular sequence. Sometimes co-dependent servers have to booted in a
special order. You will need to document these special procedures wherever they exist and
make note of them in your project plan. It may also influence the order in which the servers are
relocated to the new data center.
8. Migration Timetable: Perform a careful audit can help to determine the number of days over
which the migration should be spread and the sequence of the server moves. This will require
you to account for each application within your environment which should also include their
interdependencies with every other application, their software interfaces, the firewall rules that
protect them and any application batch or cron jobs they rely on. The audit will also help to
determine the groupings of systems that should move together.
9. Base Equipment List: Create a minimum list of servers that absolutely have to be up and
running in order to maintain the web farm's functionality. It will help to focus the minds of the
team members in the event of multiple failures.
10. Functionality Testing: Create a short list of tests for each server that will be used to verify that
it is functioning correctly. Don't limit this to just network connectivity, but also check that all the
required applications have started correctly and that some simple business operations can be
successfully completed.
11. Application Code Surveys: Test to make sure your applications don't use IP addresses to
access information on remote servers but use DNS names instead. The relocation may force
you to change the IP addresses of devices and could cause some programs to fail unless this
precaution has been taken.
The general steps required to prepare your servers for the migration are not hard but the process
can become difficult due to the sheer volume of information you need to track. Plan well and you
should be able to have a successful project.
DNS Preparation
If the relocation requires the IP addresses of your site or servers to change then you'll have to
make plans to adjust your DNS settings during the relocation.
There are two things to remember with DNS. The first is that it will take at least 48 hours for any
DNS change to propagate across the Internet. The second is that every DNS entry has an
associated time to live (TTL) value which defines how long DNS caching servers should store the
entry for local use before being required to query the entry's authoritative DNS server to see
whether there have been any changes. With this in mind, here is what needs to be done:
1. Set The TTL: There is no magic bullet that will allow you to tell all the caching DNS servers in
the world to simultaneously flush their caches of your zone file entries. Your best alternative is
to request your existing service provider to set the TTL on your web site, for example
www.myweb- site.org, in the DNS zone file to a very low value, say one minute. As the TTL
is usually set to a number of days, it will take at least three to five days for all remote DNS
servers to recognize the change. Once the propagation is complete, it will take only one minute
to see the results of the final DNS configuration switch to your new server. If anything goes
wrong, you can then revert to the old configuration, knowing it will rapidly recover within
minutes rather than days.
2. Server Based Testing: Set up your test server in house. Edit the /etc/hosts file to make
www.my-web-site.org refer to its own IP address, not that of the www.my-web-site.org
site that is currently in production. This file is usually given a higher priority than DNS, therefore
the test server wi ll begin to think that www.my-web-site.org is really hosted on itself. You
may also want to add an entry for mail.my-web-site.org if the new Web server is going to
also be your new mail server.
Test your server based applications from the server itself. This should include mail, Web, and
so on.
3. Client Based Testing: Test the server from a remote client. You can test the server running as
www.my-web-site.org even though DNS hasn't been updated. Just edit your /etc/hosts
file on your Web browsing Linux PC to make www.my-web-site.org map to the IP address
of the new server. In the case of Windows, the file would be
C:\WINDOWS\system32\drivers\etc\hosts. You may also want to add an entry for
mail.my-web-site.org if the new Web server is going to also be your new mail server.
Your client will usually refer to these files first before checking DNS, hence you can use them to
predefine some DNS lookups at the local client level only.
4. Check All Domains: Make sure similar steps are taken for all your DNS domains. Remember
to also update the DNS entries for your mail servers, they are generally located in a different
section of the DNS zone file and can be easily overlooked.
5. Prepare to Switch: Once testing is completed, coordinate with your Web hosting provider to
update your domain registration's DNS records for www.my-web-site.org to point to your
new Web server at the time of the relocation.
Plan to change your DNS TTL at least a week before the expected migration to limit it risking the
success of your project. DNS management is probably the easiest task to accomplish but poor DNS
planning can unexpectedly delay your project with your only recourse being to sit and wait for the
changes to propagate.
Transportation Preparation
A relocation would certainly not succeed without adequate transportation therefore it should be
planned well. Here are some factors to consider.
1. Moving Company Selection: Get multiple quotations from movers, preferably with each
provider giving a guaranteed maximum price for the job. Cost shouldn't be the only factor.
Investigate the reputation, staff training, moving van cleanliness and safety, performance
record, reliability, and the claims settlement customer service of each moving company. Visit
the mover's office to verify they have a business. Determine whether the company belongs to a
trade organization that requires a code of ethics and operation. Carefully consider whether the
staff are people you want to do business with.
2. Reserve Transportation: If you choose to rent or use movers, have guarantees that the
transportation will arrive on time.
3. Physical Protection: For large quantities of servers you'll need to have racks or shelving
preinstalled to accommodate as many servers as possible. You should also ensure that the
servers are securely fastened to prevent shifting in transit. Do not stack servers one on top of
the other as this increases the risk of damage.
Some of the more delicate devices may have to be specially wrapped for their protection in
bubble wrap or foam. Some may have to be bolted to shipping pallets and mechanically moved.
Finally, some vendors with full coverage maintenance contracts my stipulate that their staff be
the only persons authorized to package the equipment. Make adequate preparations in
advance.
4. Internal Logistics: Servers can be heavy. Get access to hand carts or wheeled dollies on
which the servers can be manually pushed within the buildings. If practical, rent ramps to
reduce the need to manually transfer servers at the various stages of transportation along the
way.
5. Insurance: You may have to insure the equipment prior to the relocation. Check to make sure
the selected moving company carries the required insurance coverage. If they break or lose
something, you should request that it be fixed or replaced to the limits of their liability. There
can be many clauses to this type of coverage, make sure the mover clearly explains the extent
of your exposure.
Transportation is often given the least amount of thought and servers will inevitably be carried on
the back seat of cars. Avoid this as much as possible, renting a truck or using professional movers
will be much faster, easier to track, less prone to equipment damage and easier to insure. You may
be tempted to save money here, but the few dollars spent on ensuring proper transportation can
save thousands in potential down time.
Conclusion
The preparative tasks for a server farm relocation can be complex but with the right tools and planning
it can be very manageable. Sample check lists, and post mortem forms are available in Appendix I.
Chapter 3, "Post Relocation Activities" will begin by discussing what needs to be done during the
relocation and will end with a number of activities that need to be completed once the project appears
to be over. Most importantly, it specifically outlines what to do if things start to go wrong.
Chapter 3
Post Relocation Activities

The deadline day has finally arrived. What should you do? It may not be as straightforward as it first
appears. This chapter outlines some precautions you'll need to take.
Activities During the Relocation

It may appear that with good planning the relocation should progress smoothly. That is largely true but
the challenge is in the coordination of the activities when they go right and when they go wrong.
14. Customer Communications: Send out a notification to all affected parties that the migration is
about to begin. Move the servers in the predefined order on your check list.
15. Team Allocations: You'll probably need to split your team into four groups. Those responsible for
the shutdown of the old site, the physical movement of the servers, the booting up of the servers in
the new location and testing.
16. DNS Updates: You'll have to synchronize the update of your DNS entry with the movement of the
servers. Coordinate with your DNS provider to update your domain registration's DNS records for
www.my-web-site.org to point to your new Web server. As the TTLs were set to one minute
previously, you'll be able to see results of the migration within minutes. Once the migration is
complete, you can set the TTL back to the original value to help reduce the volume of DNS query
traffic hitting your DNS server. Delete any test entries in your server /etc/hosts files to make
sure they don't unexpectedly interfere with future migrations.
17. Testing: The overall project manager should have a check list, white board or some other highly
visible tracking system to monitor the basic status of every device being migrated.
The testing teams should report the status of each device periodically. They should place one of
three markers on each system. It may be something as simple as a sticky note, or self adhesive
dots that are commonly available in office supply stores. These should include status colors for
"untested", "successfully tested" and "failed" with only one color being assigned at a time. The red,
green and yellow traffic light colors are one obvious choice.
18. Troubleshooting and Recovery: Things will go wrong. Make sure you have a conference call
bridge open throughout the activity period. Try to limit the attendees to only managers once the
initial problem has been identified as your technical staff will function best if left relatively
undisturbed during troubleshooting.
Assign a technical lead to be responsible for the full recovery of the situation even if the solution
wanders outside of their area of expertise. The advantage of doing this is that it ensures that there
is at least one technical person on the line who has a complete understanding of the history of the
problem. This makes the transfer of troubleshooting responsibility from group to group much
smoother. It is also a good practice to have an equivalent managerial lead for overall responsibility
of the issue.
If more than one problem occurs create another conference bridge and assign another pair of
technical and managerial leads to reduce confusion. The use of email to track the minute by minute
progress of a problem is generally counter productive as many of the participants will not have
access to email during the activities. This can cause delays. Assign someone to issue general
status messages when milestones in the troubleshooting have been reached. The abundant use of
the "Reply All" button should be avoided as many team members will miss one of the many email
threads. In times of crisis real time communication is always best, make sure everyone has access
to a mobile phone.
If things are going really badly, and you have passed your point of no return, focus your resources
on getting your minimum set of servers functioning as quickly as possible. If not, execute your roll
back plan to return your servers to the original data center.
19. Monitoring: Verify that the monitoring and error logging are working correctly and that your
predefined basic functionality testing is progressing as planned. You will need to execute more
detailed testing later and this will be covered in sections to follow.
Without good contingency planning the success of the project could quickly falter. The steps outlined
previously should help to further guarantee that your preparations help to achieve your objectives.
Post Relocation Activities

Relocation activities don't end with the passing of the deadline date. There are many other factors to
consider.
1. Thorough Testing: Test to make sure your customers and partners can gain access to the
expected areas of your server farm. Run through some predefined transactions to test the health of
your applications in the new environment.
2. Relocate Redundant Equipment: In many migration plans, half of a redundant pair of devices is
moved first as part of the main migration with the second device in the pair being moved at a later
date. This can create unexpected problems, for example, the newly inserted equipment may
overwrite the configuration of its partner with the configuration used at the old location. Plan for this
eventuality by reading your equipment handbooks and having a copy of the valid configurations
close at hand. The new equipment may also have to be configured for the IP address, logging,
monitoring and authentication schemes used at the new facility.
3. Contract Negotiations: Contracts relating to the old server facility will have to be terminated or
reapplied to the new location. This would include maintenance contracts, contractor services,
leases and data circuits.
4. Service Level Verification: Verify that all the promises made by the management of the new
facility have been met. Measure temperatures, test maximum data throughput rates, do practice
data backups and restorations, confirm the use of redundant power feeds, make sure the
information in your online customer care portal is correct, and test the facility's response time to do
standard changes.
5. Redundancy Testing: Plan to test the redundancy of the site. This may be as simple as
disconnecting one of your Internet links and seeing whether traffic reaches your site through an
alternative ISP. It could also be more complicated such as doing a database cluster failover or
network device reboot. It's never a good thing to merely assume that your architecture is redundant.
6. Monitoring and Reporting: Verify that your logging and monitoring are working correctly. Simulate
some known but non fatal errors and see whether your systems can detect them correctly.
7. Documentation: Modified environments usually require updated documentation. Diagrams and
procedures are often tedious to change but the time spent confirming the accuracy of your
information can greatly improve the efficiency of your operations staff and make troubleshooting
much quicker.
8. Examine Architecture Changes: It is generally not a good idea to incorporate major changes in
the network architecture of your server farm as part of the relocation. It will tend to increase the
complexity of the tasks and the likelihood of failure. A project to re-architect your farm after the
relocation is much more desirable.
There is one type of simultaneous re-architecture that can gain many rewards with relatively small
risk. Older server farms tend to have all servers directly connected to pairs of high capacity
switches located in a central location. This can make cabling costs high. A simple alternative is to
maintain your investment in these central switches, but have them connect to smaller "pizza box"
sized switches in each rack or cabinet. Servers can then be connected to the switch in their rack
and the switch can then aggregate, or "trunk", the required networks/VLANs to the main central
switches. Very little routing changes are required in this architecture. The cabling becomes much
cheaper and less complicated, the addition of a new rack only requires one or two cable runs to the
central switches and short standard length cables connect the server to their local switch.
Expansion can be done using cheaper commodity switches instead of more expensive blades that
need to be fitted into the central departmental switches. There is the risk of having increased points
of failure, but the failure of one of these small switches is less likely to threaten your entire site and
if they do fail, the replacement cost and time is usually less. There is also the complexity of
managing additional switches but the use of automated network device backup software such as
RANCID can significantly reduce some aspects of administration. Another risk is that backup traffic
on one network can choke out Web traffic traveling on the same trunk, in departmental switches the
backplane or master bus can usually handle this type of situation easily. You can largely avoid this
problem by using Gigabit instead of 100 megabit per second fast Ethernet trunk uplinks. In extreme
cases you may have to consider using separate switches for Internet facing and internal
networks/VLANs. This solution may not be suitable for your existing environment, but it may be
worth investigating for newer projects.
Remember that in this architecture the firewalls, load balancers and all other networking equipment
except the rack based switches should be directly connected to the departmental switches. This
prevents data from bouncing around from rack to rack as it passes from firewalls, to load balancers
and then to servers. If this is not done, a single saturated trunk to one of the racks could take your
entire site offline instead of only a small group of servers. With Gigabit links this is usually an
unlikely scenario, but prevention is always better than the cure.
9. Rental Returns: Remember to return all items rented as part of the migration. This would include
moving trucks, dollies, trolleys and shelving.
10. Infrastructure Checks: Verify the ambient temperature and humidity in the aisles meet the
specifications of the CRAC unit vendor. Also check that the temperature at the front and rear of
each server conform to the requirements of the manufacturer.
11. Documentation Storage: Store all documentation related to the project in electronic form in one
location so that it cal be reused and accessible by all.
12. Security Audits: Now that the relocation is complete you should perform a complete security audit
which should include a remote network vulnerability scan; verification that all installed software is
authorized, correctly patched, and regularly scanned for viruses if applicable; confirmation that only
authorized persons have physical and network access to your systems; and checks to see whether
firewall rules need to be adjusted to match the new needs of the organization.
13. Project Audit: You should conduct a post-move audit and review of the data center relocation
project. This should compare the results to the initial business case, conformity to schedule and
cost estimates and feedback from team members and key stakeholders. The lessons learned from
the data center move/relocation project can be used to enhance future projects. The audit will also
highlight areas of day to day processes that will need attention and so it should be used as a
means of improving the universal strategic goals of improved simplicity, lower cost and increased
speed.
14. Party Time! When it's all over have some form of recognition of the hard work done by all your
staff. Incentives such as extra time off, commendations for people who genuinely saved the day, an
outdoor barbeque, a pot-luck, a fun day, cinema tickets for the whole office to see the latest
blockbuster movie, or a small party can all be arranged at little extra cost. They should have an
element of surprise and most importantly should be held outside of the office, for example on the
lawn, in a nearby park or at a restaurant. Special events aren't usually viewed as being special if
they are held in the usual everyday surroundings.
If the migration required weeks of late nights in preparation and execution, and there is a sufficient
budget allocation, invitations to the event should be extended to the immediate families of your
staff. They had to bear the absence of loved ones working under stress over extended periods of
time. Without their inclusion the project would only be an economic victory not a morale building
exercise. It is important to recognize the talents and new skills learned by the members of the
company and the tolerance of their families. These are a few of the activities that need to be
executed after the dust has settled but they are no less important than any other task. The
relocation is only finished once there is complete peace of mind.
Conclusion
Server relocations can be daunting projects but the tools and knowledge provided in this book
should allow you to ably complete the project with few unexpected complications. Remember that,
as with any project, careful planning is critical to success and should be taken seriously by
everyone involved.
Appendix I
Relocation Check
This appendix has samples of all the various forms you may need as part of the relocation. They cover all the
preparation, execution and cleanup tasks mentioned in earlier chapters.
Data Center Rating Form

In this table, give each item a weight (W) from 1 to 10 depending on how important it is to you. When visiting the
data center give each item a score (S) from 1 to 10. At the end of the visit multiply W and S to get a sub total.
The data center with the highest grand total will be the most desirable.
Note: Data center ISP related factors are covered in the ISP rating form
Data Center Name: ________________________ Engineer: ___________________

Date: ________________________ Contact: ___________________
Score (S) Sub- Total (W x S)
Item Score Sub-

Item Description Comments Weight (S) Total
(W) (W x S)
1. External flood risk
2. Proximity to highways. Greater than

250 meters
3. Proximity to railways. Greater than

250 meters
4. Proximity to hazardous production

facilities
5. Proximity to reasonably priced

housing
6. Proximity to good schools
7. Proximity to recreational areas
8. Connectivity to public transportation
9. Proximity to airport flight paths
10. Grounds susceptible to localized

flooding
11. Embankments and security fencing

surrounding the building
12. Availability of roof access rights.
13. Power feeds from multiple substations
14. Building has sufficient excess power

capacity for growth
15. Adequate UPS coverage for expected

load
16. Staff automatically alerted when poor

power line conditioning is detected.
17. UPS N+1 redundancy. Sufficient extra

Item Score Sub-

(W) (W x S)
units to allow for offline maintenance

without jeopardizing coverage.
18. Standby generator N+1 redundancy.

Sufficient extra units to allow for offline
maintenance without jeopardizing
coverage.
19. Redundant PDUs supply your desired

floor space.
20. Redundant PDUs are less than 50%

loaded.
21. Facility can support the power per

square foot required by all our
equipment.
22. Facility has sufficient power to support

server load expansion into the
foreseeable future.
23. Each rack or cabinet is fed by both

PDU units.
24. Standby generators are regularly

tested under load
25. UPSs are regularly tested
26. Air temperature at 75F / 25C
27. Signs of leakage or rattling from the

CRAC units.
28. Offices and common areas isolated for

data center floor.
29. Mandatory visitor registration or

electronic ID access.
30. Biometric access to data center.
31. Smoke and heat detectors.
32. Pre-action fire suppression system.
33. Liquid detectors under the raised floor

tiles.
34. Graphical fire alarm panel with map of

data center floor.
35. Job tickets automatically created when

facility's equipment fails.
36. Lead times for change requests meet

your minimum response times.
37. Availability of web based customer

Item Score Sub-

(W) (W x S)
care portal.
38. Problem escalation procedures meet

your needs.
39. Availability of 24/7 remote hands

access.
40. Power and cooling costs
41. Floor space costs
42. Monitoring costs
43. Automatic data backup services

provided
44. Backup media compatible with

existing systems at current data
center.
45. Backup media with valid data is stored

off site
46. Backup media is reused after an

acceptable period in off site storage.
47. Technicians on site to provide "remote

hands" access to servers.
48. Antistatic tiles used on floor
49. Water pipes, steam pipes, kitchens

and bathrooms located away from the
computer area.
50. Facility allows you to reserve

neighboring space for future
expansion.
51. Data center area of the facility

protected by fire proof doors.
52. Site has 24/7 video surveillance
53. Site has biometric or some other form

of keyless entry system to gain access
to the server area.
54. Facility allows all your required types

of data circuits
55. Facility allows access to all your

required ISPs and data circuit carriers.
56. Facility provides access to multiple

desired ISPs or data services carriers.
57. ISPs and carriers at facility have

sufficient capacity to meet your future
Item Score Sub-

(W) (W x S)
needs for each type of circuit.
58. Facility has multiple ISP / carrier local

loop options. (Reduce single point of
failure or pricing negotiation risk)
59. Facility local loops use diverse paths

into facility. (Reduce single point of
failure risk)
60. Facility's CFA process meets timing,

procedural needs.
61. The provisioning of data circuit cross

connects meets timing, procedural
and cost requirements.
Grand Total
ISP Rating Form

In this table, give each item a weight (W) from 1 to 10 depending on how important it is to you. When visiting the
data center give each item a score (S) from 1 to 5
At the end of the visit multiply W and S to get a sub total. The data center with the highest grand total will be the
most desirable.
Data Center Name: ________________________ Engineer: ___________________
Date: ________________________ Contact: ___________________
Item Score Sub-

(W) (W x S)
1. 5 minute or better network monitoring

polling cycle
2. Number of failed polling cycles that

trigger alarms (Usually 3)
3. Job tickets automatically created when

polling fails?
4. SNMP based network monitoring.
5. Use of SNMP traps to generate job

tickets automatically.
6. Multiple DNS servers in multiple

geographic locations.
7. Adequate lead times for DNS change

requests.
8. Lead times for regular change

requests meet your minimum
response times.
9. Availability of web based customer

care portal.
10. Problem escalation procedures meet

your needs.
11. Bandwidth costs (Local Loop)
12. Bandwidth costs (95th percentile)
13. Lead time for provisioning data circuit.
14. CIR acceptable
15. CIR can be incremented to meet your

projected needs
16. Signs of congestion on network that

would prevent the ISP form meeting
the contractual CIR
Item Score Sub-

(W) (W x S)
17. SLA guaranteed uptime (Percent)
18. Scheduled maintenance windows

meet your timetable needs.
19. Scheduled maintenance windows give

adequate advance notification.
20. Are NOC staff certified in the

equipment they monitor?
21. NOC escalation procedures meet your

needs.
22. Type of data circuit to be provided.

(T1, E1, DS3, OC3, HSSI, Ethernet,
HSSI, etc.)
23. Does ISP have out of band access

(eg. Modem connection) to gain
access to their equipment if their data
circuit fails.
24. How often are error rates monitored to

trigger maintenance?
25. At what error rate triggers mandatory

maintenance?
26. What is your procedure for scheduling

technician visits to fix line degradation
problems?
27. Do you provide full BGP routes?
28. We have our own BGP AS number

with our own networks. Will you be
able to advertise these to the Internet
if we peer with you?
29. Do you have fully redundant paths to

each of your BGP peering transit
providers and the router to which we
will be peering?
30. We may need to adjust BGP routing to

amount of bandwidth we push through
your circuits. Do you support AS path
prepending?
31. Is your networking infrastructure fully

protected with UPS and standby
generator power?
32. Who do you peer with for Internet

transit?
33. Facility allows all your required types

of data circuits for this ISP
34. Facility allows access to this ISP / data

circuit carriers.
Item Score Sub-

(W) (W x S)
35. ISP / data circuit carrier has sufficient

capacity at facility to meet your future
needs for each type of circuit.
Grand Total
Cost Justification Work Sheet

Calculate the expected monthly change in profit and expenses over the expected life of the servers in the new
data center and off set this against the expected startup capital outlay costs to determine the economic merit of
the relocation/consolidation project.
Potential Monthly Profit Improvement

Done By: ________________________ Date: ___________________
Old New Net

Item Description Comments Facility Facility Change
1. Expected increase in profits due to

increased sales facilitated by
improved data center activities.
2. Eliminated risk to profits by moving

to a more suitable data center.
(Less revenue loss due to
infrastructure being incapable of
supporting customer needs, or
systems outages)
3. Liberated cash flow due to

consolidated operations
Grand Total
Relocation versus Data Center Upgrade Capital Outlays

Done By: ________________________ Date: ___________________
Old New Net

1. Leasehold improvements cost
2. Transportation moving costs
3. Overtime expenses
4. Increased staff hiring expenses
5. Contractor fees
6. Training fees
7. Equipment acquisition
8. Temporary leases for the relocation

or upgrade
9. Cleanup costs
10. Penalty fees
11. Staff layoff expenses SUB
Grand Total
Monthly Costs
Done By: ________________________ Date: ___________________
Old New Net

1. Air conditioning lease
2. Air conditioning maintenance
3. Data backup equipment leases
4. Data backup services
5. DBA services
6. Fire systems maintenance
7. Floor space lease
8. ISP fees
9. Janitorial services
10. Network equipment leases
11. Network management services
12. Power
13. Data backup replacement media

expenses
14. Security services fees
15. Security systems maintenance
16. Server leases
17. Server maintenance contracts
18. Server systems administration

services
19. Software licenses
20. Software maintenance contracts
21. Staffing expenses
22. Standby generator lease
23. Standby generator maintenance

Old New Net

24. Systems development expense
25. UPS maintenance fees
26. Web site monitoring expenses
27. Staff expense
Grand Total
Net Change
Description Old Facility New Facility Net Change

Coordination Preparation Check Sheet

Done By: ________________________ Date: ___________________
Person Deadline Status

Item Description Priority Responsible
1. Project manager assigned
2. Networking team lead identified
3. Transportation team lead identified
4. Server area team lead identified
5. Server team lead identified
6. Customer communication lead identified.
7. Catered food lead identified (Pizza, Veggie, Kosher,

Halal, etc.)
8. Post mortem team created to analyze any failures in the

process.
9. Gofer identified
10. Disaster recovery team lead identified
11. All members of the relocation team will have functioning

mobile phones on the day or the migration.
12. Conference call bride arranged
13. Best time and date for relocation identified
14. Team aware of method to be used to identify

malfunctioning equipment.
15. Short list of scenarios during which the relocation will

have to be rolled back has been created.
16. Point of no return identified.
17. Minimum list of functional servers for successful

migration created.
18. Sequence list of which equipment will be moved, and

when, created.
19. Roll back plan created that includes the sequence in

which equipment will be returned.
20. Server list / inventory spreadsheet has been created

and distributed to all teams.
21. Application audit performed for all servers to account for

interdependencies with other applications.

22. Procedures documentation completed for each server /

network device
23. Equipment that will have to be leased or purchased to

facilitate the relocation has been obtained
Customer Communication Check Sheet

Done By: ________________________ Date: ___________________

1. Single message of varying degrees of detail created for

each customer group.
2. Internal customers alerted of plan
3. External customers alerted of plan
4. Draft notifications to be used in the event of failure and

success created.
5. Short list of customers to be kept updated by phone created.
6. Post mortem template identified

Server Area Preparation Check Sheet

Done By: ________________________ Date: ___________________

1. Racks and cabinets installed in new location.
2. Adequate power supplied to each rack and cabinet
3. Each PDU in the server area can handle the failure of the
other redundant one supplying the area.
4. Adequate cooling provided for expected server heat load.
5. Power outlets are labeled to indicate the PDU that supplies

them.
6. Verify that you have power cables in sufficient quantities and

adequate lengths.
7. Power cables are labeled at both ends.
8. Power supplied is of the correct voltage using the desired

connectors.
9. Servers in racks and cabinets won't overload the power

circuits that supply them.
10. Diagrams created to show the server layout in the racks or

cabinets.
11. Shelving and rack kits for servers pre installed in cabinets
and racks.
12. KVM (Keyboard, Video Monitor, Mouse) switches

preinstalled.
13. Rack mounted keyboards and monitors installed in cabinets

or racks.
14. Racks and cabinets clearly indicate the server orientation to

create hot and cold aisles.
15. Keys given to all persons who require access to the server
area.
16. Staff doing the server "Rack and Stack" work have adequate
tools especially electric screwdrivers.
17. Data circuits for internet connectivity have been run to the
racks to be used by the networking equipment.
18. Monitors on carts available for troubleshooting servers that

need to be removed from racks.
19. Antistatic tiles installed on floor


20. Water pipes, steam pipes, kitchens and bathrooms located

away from the computer area.
21. Patch panels installed correctly
22. Patch panels have the correct connectors.
23. Cable lengths through the patch panels are under 100m in
length.
24. Glass fiber and copper cables run in separate cable trays.
25. Server area is large enough to accommodate future

expansion.
26. Aisles between racks wide enough to allow people to easily

mount and dismount servers in them.
27. Server area has 24/7 video surveillance
28. Server area has biometric or some other form of keyless

entry system access system.
29. Copper data cables, fiber data cables and power cables run
in separate conduits or trays.
30. Data cables not hanging in the air or pulled taught.
31. Patch panel cable bundles run to the sides of racks and
cabinets so as not to impede airflow.
32. Adjacent server cabinets do not have their walls removed.
33. Few air flow obstacles under the raised floor area.
34. Baffles used under the raised floor area to improve airflow.
35. Raised floor under the cooling zone of each CRAC unit is
sealed to force air through desired cabinets.
36. Perforated floor tiles only used in cold aisles.

Network Preparation Check Sheet

Done By: ________________________ Date: ___________________

1. Network equipment with dual power supplies are plugged

into electrical circuits from each PDU.
2. Groups of servers with similar function (eg. DB servers) are

connected to redundant switches in a 50/50 fashion.
3. Each server is labeled on the front and back.
4. Data circuits to be used by the network gear have been

tested.
5. Sample configurations have been created for all networking

devices with the new network addresses to be used at the
new location.
6. The standby devices in redundant pairs of networking

equipment at the old location have been shutdown, shipped
to the new location and preconfigured.
7. Complete network diagrams of the new location have been

created.
8. Test VPNs to remote offices have been created.
9. Network devices are labeled in the front and back.
10. All network cables labeled at both ends.
11. POTS lines installed for out of band access.

network device
13. Data circuit capacity requirements identified for both

production Web access and possible data replication
between data centers.
Server Preparation Check Sheet

Done By: ________________________ Date: ___________________

1. Servers with dual power supplies are plugged into electrical

circuits from each PDU.
2. Groups of servers with similar function (eg. DB servers) with

single power supplies are connected to redundant PDUs in
a 50/50 fashion.
3. All server power cords labeled at both ends.
4. Each server's "netstat -an" output has been recorded.
5. Each server's "netstat -nr" output has been recorded with

the expected output at the remote location prepared.
6. Server data backed up.
7. Practice data backups and restorations done for key servers

and applications.
8. All RAID BIOS settings recorded.available (replication

scenario)
9. Spare servers identified as possible sources of spare parts

or as "hot standby" devices.
10. Sequence of servers to be moved identified.
11. Sequence of how the servers will be returned moved in the

event of a roll back identified.
12. Testing short list created for each server.
13. Applications vetted to make sure they refer only to DNS

names not IP addresses.
14. Minimum list of servers for correct site functionality created.
15. Special operating procedures for key servers created. (eg.

Documentation of the sequence in which servers should be
booted or applications started)
16. Snapshots of the actively running services or applications

for each server has been created.

network device
18. Amount of data to be backed up per server has been

determined.
19. Equivalent amount of production data storage

DNS Preparation Check Sheet

Done By: ________________________ Date: ___________________

1. TTL on old DNS zone files set to 5 minutes and propagated

to the Internet.
2. New DNS zone files created with new IP addresses. (TTLs

should remain at 5 minutes)
3. Testing of application using new DNS entries in the test

server's hosts file completed.
4. Testing of website using new DNS entries in the web

browser client PC's hosts file completed.
Transportation Preparation Check Sheet

Done By: ________________________ Date: ___________________

1. Moving truck rented / Professional movers hired
2. Mover has adequate insurance coverage
3. Adequate method for stacking and securing servers in the

truck found.
4. Equipment insurance up to date.
5. Dollies and carts available at both locations for moving the

servers across the floor.
6. Boxes purchased or rented for moving equipment and

miscellaneous items.
7. Special packaging for the more delicate items.
8. Items under maintenance contract prepared for

transportation by the equipment vendor, not internally.
Activities During the Relocation Check Sheet

Done By: ________________________ Date: ___________________

1. Notifications sent to all customers that the activity is about to

start.
2. DNS using the new zone files has been propagated to the
Internet.
3. Emergency conference call bridge has been created for

communication with team.
Post Relocation Check Sheet

Done By: ________________________ Date: ___________________

1. All power cables labeled at both ends
2. All network cables labeled at both ends
3. Moving truck, dollies, carts, boxes and other transportation

related items returned.
4. New diagrams and other documentation created for

adequate description of the new server environment.
5. Error logging and monitoring occurring correctly.
6. Contracts related to old facility terminated or re-negotiated

for the new facility.
7. All equipment at the new location has been restored to fully

redundant operation.
8. Redundancy testing of new server environment completed.
9. Testing of data throughput rates at new location completed.
10. Customers and partners can get adequate access to the

servers at the new location.
11. Test data backups of servers successful.
12. DNS TTLs on new zone files have been reverted to a couple
days.
13. Security audits planned and executed.
14. Blanking panels inserted between servers
15. Server cable bundles run to the sides of racks and cabinets
so as not to impede airflow.
16. Ambient air temperature within design of the CRAC unit
17. Ambient air humidity within design of the CRAC unit
18. Intake temperature at the front of servers in the rack is

within specification.
19. Exhaust temperature at the front of servers in the rack is

within specification.
Individual Server Worksheet
Server Name: Backup Done:
Make: Model:
Serial Number: Inventory No.:
Operating System: Backup Done:
Engineer: Date:
Old Data Center New Data Center

Item Description
1. Default gateway
2. Server rack location
3. NIC #1 IP address
4. NIC #1 Subnet Mask
5. NIC #1 Connected to switch named:
6. Backup server IP address
7. DNS server #1 IP address
8. DNS server #2 IP address
9. NIC #1 Connected to switch port number:
10. NIC #1 Connected to VLAN number:
11. NIC #2 IP address
12. NIC #2 Subnet Mask
13. NIC #2 Connected to switch named:
14. NIC #2 Connected to switch port number:
15. NIC #2 Connected to VLAN number:
16. Attached Devices

Individual Server Worksheet (Part II)
Additional Server Routes

Item Network Subnet Mask Gateway
1.
2.
3.
4.
Network Enabled Applications List (netstat -a output)

(Will be same as that expected in New Data Center)
Old Data Center
Old Data Center

Contractor Qualification Check Sheet

Done By: ________________________ Date: ___________________
Item Description Yes No
1. Contractor has sufficient experience in the industry.
2. Contractor has sufficient certified staff.
3. Contractor has provided evidence of staff certification
4. Contractor has the correct installation and testing tools
5. Contractor lists the type and manufacturer of all testing tools
6. Contractor is willing to guarantee the quality of their work in writing and back it
with a one year warranty.
7. Contractor has provided proof of doing similar work previously.
8. Contractor has tested their work to your satisfaction and provided a test plan as
proof.
Data Circuit Check Sheet

Done By: ________________________ Date: ___________________
Carrier Circuit Intermediate POP Address

Type Carrier
Qwest T1
PRI
DS3
GigE
Sprint T1
PRI
DS3
GigE
ATT T1
PRI
DS3
GigE
Verizon T1
PRI
DS3
GigE
Level3 T1
PRI
DS3
GigE
etc. T1
PRI
DS3
GigE
etc. T1
PRI
DS3
GigE
etc. T1
PRI
DS3
GigE
Vendor / Purchasing Check Sheet
Done By: ________________________ Date: ___________________
Item Shipment Product Order Shipped Status Scheduled Date Shipment Data Center Tracking Number of
Group Quantity Quantity Shipped Shipped Method Ticket Number Pieces
Date Number
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
Post Mortem Analysis Sample Form
Customer Information
Customer Name List all affected customers
Location Data Center where incident occurred.
Date Report Date
Author Engineer working issue
Ticket Number(s) Related work order or trouble ticket numbers
Incident Date Date incident occurred
Post Mortem # Assigned by the IR Administrator
Responsible Support Team Engineer’s Department
Incident Summary
A brief, factual description of the issue:
Example: We received an alert from the monitoring system for customer X indicating sites were unavailable. The
troubleshooting process indicated that the primary local-director (load balancer) had failed over to the secondary.
There were also traps that indicated VLAN state changes on the Internet facing VLAN of the load-balancer. However,
even after the load-balancer failed over to the secondary, the sites were still not accessible
Incident Detail
Details from the ticket with time stamps (include event start time, escalation times & end time when
customer confirmed resolution). Ensure all parties involved in the troubleshooting, escalation or
communication are referenced, but do not use the individual’s names
Example:
Eastern Standard Time:

Date: xx/xx/2004
05:24pm – Event noted

05:29pm – Project Manager paged, site down
05:30pm – Escalated to Networking
05:34pm – Networking troubleshooting event
07:24pm – Confirmed with Customer site up
Customer Impact
Duration of time the customer was impacted and aspect of the site that was not available.
Example: Customer sites were inaccessible for 2hrs, from 5:24 pm to 7:24 pm.
Root Cause Analysis

Underlying factor causing the incident. Ensure complex technical terms and company specific terminology
is defined for the customer.
Example: The local-director load balancer has etherchannel setup on its interfaces. The fail over load-balancer had its
interfaces plugged into switch ports, which were not setup for ether-channel and were in the wrong VLAN. When the
device failed over to the secondary device, it came active but was unable to pass traffic. When the primary device was
made active, the sites were accessible
Corrective Action Plan

Corrective actions taken to immediately address the issue and any planned actions to prevent
reoccurrences and estimated completion dates of planned actions.
Example:
o Interfaces of the local-director were connected to the correct switch ports that had ether channel setup.
o Tested Fail over and the status on the local-directors show no errors.
o Will run an audit of the Customers net work diagrams to confirm correct setup.
o Customer care representative will schedule a meeting to review site specs with customer in 2 weeks.
Appendix II
How to Choose a Data Center ISP

In Chapter 2, "Preparing for Server Relocation", I outlined the importance of ISP preparation. This
chapter will discuss the many technical factors that govern the selection process in detail. These factors
include:
1. Pricing
2. Determining the type of data circuit to use.
3. Deciding on whether to use IP addresses issued by your ISP or addresses owned by your
company.
4. Configuring a routing protocol to use with your ISP.
Check lists are also included in Appendix I to help you make a better decision and facilitate your
monitoring of the status of all the required tasks. First let's discuss these factors in more detail.
Data Circuit Pricing

Pricing varies depending on the type of service you purchase. Internet circuits typically require you to
commit to a minimum data rate and charge a variable fee for usage above that rate to a defined
maximum. Non-Internet dedicated point to point services from data carriers usually charge a fixed fee
that allows transfers up to the maximum data rate. There is no variable component. This will be
discussed in more detail next.
Internet Services
A very common ISP billing technique used is called the 95th percentile method. Here the internet
service provider provides an absolute maximum data rate, also known as a committed information
rate (CIR), but you are billed based on actual usage. The ISP samples your data rate every five
minutes and sorts all the sample readings for the month from high to low. They then discard the top
5% of the samples with the highest utilization. You are then billed at the rate of the highest sample
that remains, not the average of those remaining. One of the advantages of this billing method is
that it allows you to download files, a usually bandwidth intensive process, for up to about an hour a
day without it affecting your bill.
In addition to bandwidth, you may also be charged a local loop rate which amounts to a monthly fee
that covers your connection from your facility to the nearest telephone / Internet exchange. This is
frequently proportional to the distance between the exchange and your facility. Sometimes this fee
is also related to your CIR and you may find that you can reduce this monthly fixed cost my
negotiating a lower CIR. You may also be able to reduce your 95th percentile rate by committing to a
longer contract or by convincing your ISP that you will be generating sufficient traffic to justify a bulk
discount.
Non-Internet (Carrier) Services

Data carriers typically will charge a flat fee for circuits with a pre-defined maximum data rate. You
will also be charged a local loop rate. The complexities of a CIR and 95th percentile are usually
absent.
Data Circuit Types

The selection of the type of data circuit to be used will depend upon the amount of bandwidth you
expect to use, the equipment available to your ISP in the area and the capabilities of your networking
equipment. The most commonly used data circuit technologies include those listed in Table A3.1.
Table A3.1 - Common Data Circuit Terminologies
Term Description
T1 1.544 Mbps link that can be split into up to 24 x 64 Kbps channels. Channels can be
aggregated together to increase throughput up to the T1 maximum in a configuration
called a "fractional T1". Channels may be used for voice or data. Typically runs over
copper
E1 2.048 Mbps link that can be split into up to 32 x 64 Kbps channels. Two channels are
used for signaling. Channels can be aggregated together to increase throughput up to the
E1 maximum in a configuration called a "fractional E1". Channels may be used for voice
or data. Typically runs over copper.
T3 Circuit configured to carry DS3 formatted data at up to 44.736 Mbps. A DS3 can be
fractionalized with up to 30 T1 circuits. Two circuits are used for signaling. This means a
T3 can up to 672 x 64 Kbps voice/data channels. Typically runs over copper.
DS3 See T3
HSSI High Speed Serial Interface capable of supporting up to 52 Mbps.
Packet Over SONET Methodology carrying TCP/IP traffic over SONET networks. The TCP/IP packets are
(POS) inserted into ATM packets which are then placed on the SONET circuit. You will most
likely need a POS interface on your router if you intend to transmit TCP/IP, Voice over IP
(VoIP ) or Internet packets on an OC type circuit. ATM interfaces may look the same, but
are designed to strictly carry traditional voice traffic. Typically runs over fiber optics.
SONET Synchronous International standard for transmitting digitized voice over optical fiber circuits. There are
Optical Network. a number of SONET circuit types, the most commonly used ones being optical carrier
levels 3 and 12 mentioned in this table. (OC-3, OC-12). Data traveling on SONET
networks use asynchronous transfer mode (ATM) formatted packets which were originally
designed to carry voice traffic. Typically runs over fiber optics.
Fast Ethernet Primarily copper based version of Ethernet that operates at 100 Mbps.
Gigabit Ethernet (Fiber) Optical version of Ethernet that operates at 1 Gbps.
Gigabit Ethernet (Copper) Copper based version of Ethernet that operates at 1 Gbps.
Term Description
Term Description
Wireless Circuits Uses a variety of methods to transmit data through the air. In most cases the ISP
manages the antenna equipment and hands off a physical cable link to their customer.
This link may be of any of the circuit types mentioned in this table. Wireless links can be
quickly installed but they tend to be subject to interference that can reduce their reliability.
DSL An adequate solution for low volume web sites. Usually offers a maximum bandwidth of 2
Mbps.
Cable Modem Unsuitable for high bandwidth websites as your circuit is shared by many other
subscribers who could affect your performance. Your bandwidth usage could also affect
video quality of other subscribers. Some ISPs purposely restrict traffic to web servers on
their cable links for this reason
It is good to note that data services are sometimes asymmetrical in nature, especially with DSL and
cable modem circuits. This means that the incoming downstream data rate from the Internet is different
from the reverse outgoing upstream speed. You should be most concerned about the upstream speed
for your Web site to the Internet. Inbound Web browser queries don't use a lot of data bandwidth, but
the Web pages that contain the outbound replies do. Internet service providers provide asymmetric
services for residential users and the downstream rate is almost always higher than the upstream. They
reserve symmetrical data circuits for businesses which usually need high bandwidth to both surf the
web and serve Web pages and more reliable guaranteed service. The ISP will usually provide the
business with a fixed range of Internet addresses as part of the service; residential customers usually
get a dynamic address allocation which is unsuitable for most businesses. Whenever possible always
opt for symmetrical services for your business.
Remember that there are many ways to plan the expansion of your data circuit bandwidth. You can:
1. Add more circuits.
2. Order a high speed circuit and throttle it with a lower valued CIR. Increasing the CIR increases your
bandwidth.
3. Use a factional or channelized service and expand your usage one channel at a time till the
maximum capacity of the circuit is reached.
Select your data circuit with care. A wrong decision could inhibit the growth of your business.
Data Circuit Provisioning

You should always be aware of the environment in which data circuit providers work. In most cases
neighborhoods are grouped into geographic zones which receive data communication services from a
central office (CO). COs can also be called telephone or Internet exchanges.
Usually the CO is owned and operated by a single incumbent carrier (eg. AT&T) that owns the wiring
infrastructure all the way to the neighborhoods' homes and businesses. Competing carrier can
sometimes arrange with the incumbent to provide competing services over the wired infrastructure for a
fee. The connection between a CO and your business or home is often called the local loop.
Ideally, a dedicated point to point data circuit between two neighborhoods should have a local loop in
neighborhood "A", which then connects to the carrier's backbone network. The backbone should then
provide services to the CO in neighborhood "B", which connects to the remote business via another
local loop. For Internet services, there need only be a single local loop to your ISPs Internet
infrastructure.
Not all ISPs are present in all COs. In order to provide services to all neighborhoods in a city, ISPs may
have to negotiate interconnections between COs. Therefore it is possible to purchase services from an
ISP who then has to negotiate multiple local loops for the circuit to finally reach its backbone
infrastructure.
It is best to minimize the number of local loops in your circuit design. Coordinating the installation and
troubleshooting activities of one ISP can be difficult. Extending this to multiple ISPs can be tricky.
You should also realize that not all data centers allow access to all carriers and in some cases there
may be only a limited number of circuit types available. Make sure you understand how your desired
types of circuits and carriers will gain access to the facility before making a final data center decision.
The relationship between carriers and ISPs in a CO leads to a variety of additional terminologies you'll
need to know:
LOA-CFA (Letter of Authority and Customer Facility Assignment): This document does two things.
Firstly, it allows a carrier to have access to another carrier's facility to do work (LOA). Secondly the
carrier that issues the document also provides a facility assignment (CFA) which indicates the specific
interconnection point within the CO for the other carrier to use. Work cannot proceed without a LOA-
CFA for the local loops. The more local loops you have, the more LOA-CFAs are required. It is
important to keep a very close eye on this process.
DLR (Design Layout Record): This document describes all the details of the circuit path from one end
to the other. It can include physical information such as location, floor, row, rack, panel and port. It can
also refer to virtual circuits, in other words, circuits that are securely shared with other customers, such
as a channelized DS3. A DLR can also mention interconnections with other known circuits, which can
help reduce the complexity of the document. You should always verify that a DLR has been created on
time in order for it not to hold up the rest of your operation.
FOC (Firm Order Commitment): It may sound rude, but FOC is a common term used in the industry. It
is the date your carrier will commit to having a fully functional circuit delivered to you. Always ask what
additional tasks will be required after the FOC date. You will almost certainly have to coordinate your
engineers and those of the carrier to harmonize and test their configurations before data flows correctly.
It is very possible for carriers to test their local loops correctly but make a mistake on the CFA with an
incorrect cross-connection.
MPOE (Main Point of Entry): Carriers and ISPs need to deliver data circuits to a specific room at a
business address. It is typically the same room in which all telephone lines enter the building.
MDF (Main Distribution Frame): Is usually a rack in the MPOE in which carriers will install the
equipment required to terminate the circuit's local loop coming from the CO. This rack and equipment is
usually the property of the carrier / ISP. Your equipment will usually connect to the MDF gear through a
patch panel provided by you carrier / ISP.
IDF (Intermediate distribution frame): In buildings with multiple tenants it is common to extend
connectivity from the MPOE to each tenant's premises. Each tenant location, (eg. a server room or the
location of their PBX) will have their own IDF for their own equipment. Connectivity between gear in the
MDF and the IDF is usually achieved by using patch panels.
Data center Cross-connects: A carrier or ISP will deliver your circuit to the MPOE, but you'll need to
have a cross-connect created to link your server room's IDF to the MPOE's MDF. Remarkably, data
centers often charge for this accessibility on a per circuit basis. It can be an unexpected hidden cost.
With the knowledge of these terminologies you should be in a much stronger position when talking to
your ISP and carriers.
IP Address Ownership
In a data center environment you will normally request a block of IP addresses (the data equivalent of a
telephone number) from your ISP for use by your servers. The ISP will assign a range of addresses to
you and will configure their equipment to route traffic to this range via the data circuit they provide.
There is a disadvantage to this. If you cancel your ISP data circuit, you will lose the IP addresses they
assigned to you. This could force you to reassign brand new IP addresses to your servers.
Always consider applying for your own IP addresses from your Regional Internet Registry (RIR). Here is
a useful list of RIRs you can use for your area:
1. AfriNIC (African Network Information Centre) - Africa region

2. APNIC (Asia Pacific Network Information Centre) - Asia Pacific region
3. ARIN (American Registry for Internet Numbers) - Americas and Southern Africa
4. LACNIC (Regional Latin-American and Caribbean IP Address Registry) - Latin America and some
Caribbean
5. RIPE NCC (Réseaux IP Européens Network Coordination Centre) - Europe and surrounding areas
will recognize your operation as being similar to that of an ISP and will assign you your own AS and
IP addresses. The circumstances for doing so are slightly different for each affiliate but the main
factors are that you can prove that your routing policy is different from that of your ISP and/or that
your connectivity requires links to multiple ISPs.
If you cannot obtain your own IP block then you will have to ensure that all your applications use DNS
names to refer to other servers in your environment and not their actual IP addresses. When new IP
addresses are required, you can just modify the DNS name to map to the new address. This minimizes
the impact of forced IP address changes on your operation.
Routing Protocols
Internet routing can be quite complicated and you will often need a network engineer to configure your
equipment to get access. This section will provide an overview for project managers of the most
common Internet routing challenges data center based web sites face. It will provide insights into what
can be done if things go wrong during your data center relocation. ISPs usually use two methods to
provide internet access to their clients. The first is by providing a simple default gateway through which
all network traffic should pass. This is the usual option when only a single link is provided. The second
method relies on the border gateway protocol (BGP) and is used primarily when Internet connectivity is
provided via multiple ISP links.
Border Gateway Protocol

BGP is a dynamic protocol that can be adjusted relatively easily to influence traffic to and from your
site in order to reduce bandwidth costs when your ISPs charge different rates, or to divert excess
traffic from an overloaded circuit to a lesser utilized one. Unlike the configuration of a static route
that can never change even if a link fails, BGP routes adjust themselves automatically depending
on the availability of network links to reach target destinations. This section will cover BGP for use
by project managers in some detail and Table A3.2 summarizes many of the terms that will be used
later.
Table A3.2 - Common BGP Routing Terms
Term Description
BGP Autonomous A BGP routing management domain usually owned by ISPs. Autonomous systems are
System (AS) assigned blocks of IP addresses which the ISP advertises to neighboring autonomous
systems. The ISP is also responsible for the routing of traffic destined to their AS or
passing through it to another AS.
Autonomous System A unique number assigned to the ISP for their autonomous system.
Number (ASN)
AS Path A sequential list of ASs traffic must pass through in order to reach its destination network.
AS Path Prepending A method of repeatedly adding on your AS number to the beginning of the AS path to
your network to bias traffic away from the link that is advertising the prepended list.
Local Preference A method of biasing the desirability of routes to the internet via a particular link in favor of
an alternative one within your BGP AS.
Multi-Exit Discriminator A method in which an ISP, with multiple links to your AS, can bias your routers to select
(MED) one of its paths to the Internet over another.
With BGP, each ISP is provided with a BGP autonomous system (AS) number from the Internet
Assigned Numbers Authority (IANA). The ISP then associates the IP addresses it owns to this AS.
Routes are exchanged with other ASs in the form of "I am AS number X and in my AS I have the
following networks". This results in BGP routers having long lists of all the available networks on the
Internet tied to a sequence of ASs that need to crossed in order to get to each one. This sequence
is called an AS path list.
BGP routers update their neighbors of changes they detect in the Internet. If a BGP router loses
connectivity with a peer responsible for advertising networks for a particular AS, it then notifies its
remaining neighbors of the failure and instructs them to remove their routes to the failed AS from
their routing tables.
ISPs can advertise default routes to your routers via BGP. This can be useful when your equipment
doesn't have sufficient memory to store the entire Internet routing table. Some manufacturers
recommend a minimum of 512 KB of RAM to support full routes.
No matter what types of routes you receive you can influence how traffic leaves your site with a
number of commonly used techniques. If the links terminate on the same router you can use a
system of weighting to route traffic completely over one link versus the other. Weights aren't
exchanged between routers, so when the links terminate on different routers within your control
you'll need to use BGP's local preference feature help them negotiate the preferred link. When
both your links are provided by the same ISP, you can also have them advertise a unique multi-exit
discriminator (MED) metric value in the advertisements on each link which will bias BGP on your
router to route its traffic on one link versus the other.
The methods previously discussed only refer to outbound traffic. Inbound traffic can be influenced
too. One method uses AS path prepending in which you repeatedly add on your own AS number in
your BGP advertisements. This lengthens your AS path list without making the traffic pass through
any additional ASs. Prepending can be applied on a per link basis so that internet routers will feel
that the AS path to your network on your preferred link is much shorter than the one to your network
via the less favorable link.
These modifications don't have to apply to all Internet routes. You can bias traffic on a per-network
and per-AS basis. This can be very useful. Say for example you have to email a weekly newsletter
to thousands of customers but the additional traffic saturates one of your ISP links, you can use
local preferences to make traffic to Hotmail, MSN and Yahoo! go through the original link, but traffic
to AOL and Gmail pass through the other. Another common example would be a situation in which
BGP automatically passes most of your outbound traffic over your most expensive link. You can
use some of the techniques mentioned to make BGP favor the cheaper link for your traffic until your
safe link bandwidth threshold is reached. You can usually guess the most popular ISPs from which
web surfers would be coming, if not there are automated tools such as netflow on routers, and
webalizer on web servers, that can provide more accurate insights. You can also figure out an AS
number manually using the method in the following section.
Determining a BGP Autonomous System Number
Determining the AS number of an ISP or mail service manually is usually straight forward. In
this case I'll attempt to determine the AS to which the IP address of mail.aol.com belongs.
1. Use the nslookup or host command on a Windows or Linux server to determine the IP
address of mail.aol.com
[root@bigboy tmp]# host mail.aol.com mail.aol.com has address 64.12.168.119
mail.aol.com has address 64.12.193.249 mail.aol.com has address 205.188.160.249
[root@bigboy tmp]#
2. Use a search engine to find a site that will provide access to "BGP looking glass" routers.
Log on to one of the looking glass routers listed on the site.
3. Enter one of the AOL IP addresses, in this case 64.12.168.119, and select "BGP", not
"BGP summary". Click on the submit button and you will get output looking like this.
TELIA (1299) AOL (1668) 8176, (aggregated by 8176 172.21.44.GENUITY/BBN (1))
213.248.86.53 (metric 7) from 216.218.252.149 (216.218.252.149)
Origin IGP, metric 46, localpref 100, valid, internal, atomic-aggregate
Community: 6939:2000
The output shows that the AS path list is 1299, 1688, 8176. This means to get to
mail.aol.com the looking glass router had to pass through AS 1299, then 1688 and finally
AS 8176. The final AS, 8176, is the AS for mail.aol.com. You will now have to do this again for
all the other IP addresses returned by the hosts command.
Administrative Tasks Needed to Advertise BGP Routes
Network engineers will have to configure BGP on their routers, but the project manager will
have to contact the ISP to make sure they are prepared to receive your routes. You will have to
inform them ahead of time of your AS number, the networks you wish to advertise and the
possibility of using AS path prepending. Once they receive this they will configure their
equipment and then provide you with:
1. A data circuit.
2. IP address assignments for both your and their equipment connected to the circuit.
3. Their BGP AS number.
Your network engineer will then be able to configure your equipment to provide correct BGP
connectivity to the Internet.
Note: When you use multiple ISPs, make sure your network engineer's BGP configuration
guarantees that data traveling from ISP #1 to ISP #2 doesn't pass through your AS. If this
happens, you will find yourself paying for traffic that wasn't destined to your site. The volume of
traffic passing through your AS could cripple your data circuits too.
Conclusion
Data center ISP selection is a very important part of any relocation activity. This Appendix has provided
a summary of the issues that need to be addressed in the process and will make the overall task much
simpler to complete. Activity checklists are provided in Appendix I to help facilitate this further.
A LinuxHomeNetworking.com © White Paper

Data Center Relocation PDF

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Data Center Relocation PDF

Încărcat de

Drepturi de autor:

Formate disponibile

Data Center Relocation Guide

By: Peter Harrison

Friday, November 30, 2007

Disclaimer – The Website and Manual

Disclaimer - Other sites

Why Relocate Your Web Site?

When to Migrate From Virtual Hosting

When to Migrate Between Data Centers

Factors That Affect Virtual and Self-Hosting

How to Analyze Migration Costs

Potential Increased Profits

Net Changes in Monthly Expenses

Preparing for Server Relocation

Data Center Selection Criteria

The Relocation Project Plan

5. Interpersonal Communications: Make preparations to have a permanent conference bridge

Customer Communications Preparation

Server Area Preparation

Post Relocation Activities

Activities During the Relocation

Post Relocation Activities

Data Center Rating Form

Data Center Name: ________________________ Engineer: ___________________

Item Score Sub-

1. External flood risk

2. Proximity to highways. Greater than

3. Proximity to railways. Greater than

4. Proximity to hazardous production

5. Proximity to reasonably priced

6. Proximity to good schools

7. Proximity to recreational areas

8. Connectivity to public transportation

9. Proximity to airport flight paths

10. Grounds susceptible to localized

11. Embankments and security fencing

12. Availability of roof access rights.

13. Power feeds from multiple substations

14. Building has sufficient excess power

15. Adequate UPS coverage for expected

16. Staff automatically alerted when poor

17. UPS N+1 redundancy. Sufficient extra

Item Score Sub-

units to allow for offline maintenance

18. Standby generator N+1 redundancy.

19. Redundant PDUs supply your desired

20. Redundant PDUs are less than 50%

21. Facility can support the power per

22. Facility has sufficient power to support

23. Each rack or cabinet is fed by both

24. Standby generators are regularly

25. UPSs are regularly tested

26. Air temperature at 75F / 25C

27. Signs of leakage or rattling from the

28. Offices and common areas isolated for

29. Mandatory visitor registration or

30. Biometric access to data center.

31. Smoke and heat detectors.

32. Pre-action fire suppression system.

33. Liquid detectors under the raised floor

34. Graphical fire alarm panel with map of

35. Job tickets automatically created when

36. Lead times for change requests meet

37. Availability of web based customer

Data Center Name: ______ Engineer: _

Data Center Name: ______ Engineer: _

Date: ______ Contact: _