afcom how to move a data center

©2014 Extreme Networks, Inc. All rights

reserved.

How to move a data center in 35 days

Rich Casselberry

@richcasselberry

www.linkedin.com/in/richcasselberry


reserved.

Hmmm

Don’t take this the wrong way, but….

2


reserved.

Lesson 1: Read the fine print….

3

pursuant to section 3, paragraph c


reserved.

Lesson 2

“Don’t throw out the dirty water until you have clean water”

4


reserved.

Lesson 3: Crazy isn’t always wrong

Just because you’re crazy doesn’t mean you’re wrong

5


reserved.

Week 1 – Retire, Consolidate, Virtualize

Communicate

Retired anything and everything we could

Start virtualizing

Develop IT application test plans

Met with other data centers to see if there was any other places to

go.

6


reserved.

Week 2 (2/10 to 2/16)

Designed new backup architecture

Finished cabinet layout and

ordered cabinets

Started planning for production

move, ordered trucks and quote

from EMC

Moved 35 servers from Boston

(mostly non-production iSCSI)

8


reserved.

Week 3 (2/17 to 2/24)

Ordered backup hardware

Received new cabinets and redesigned the room for placement

More power issues, and resolutions

Servers retired and virtualized this week

Network up and running in some cabinets

9


reserved.

Space and cooling planned out

By week 3 we had the space pretty well figured out, though we soon found out some cabinets would need to move so that the permanent AC could be installed. Though it won’t be used until June, we needed to make sure it could be put in the permanent spot.

10


reserved.

Week 4 (2/24 to 3/1)

Backup gear arrives and gets installed

Move schedule close to finalized

Temporary cooling units in and online

Network gear up and running

Cabinet layout finalized

First production move

11


reserved.

Backups arrived, setup, running and tested

Since we had used managed backups we had no gear to backup or

restore. Working with our vendors we were able to get it on site,

configured, cabled, powered and tested for the move this weekend.

13

Week 4 - Backups


reserved.

Cooling units arrive

Four 3 ton cooling units arrive and get setup and running to keep

the room at a constant 65. Some spots are warmer but using fans

we are able to keep the servers cool enough.

14


reserved.

Servers pulled and ready to load on the truck

Friday night we started at 2:00PM and had the servers pulled and on the truck by 9:00PM. Each server went in a separate pile depending on the cabinet it would be racked in, once in Andover. This allowed us to quickly get things running again.

15


reserved.

Week 5 (3/2 to 3/9)

Reviewed what worked/what didn’t

Reviewed power usage and moved engineering servers to lab

Finalized schedule for final move

Built DMZ for externally facing boxes

Defined outage plan for web sites

17


reserved.

Some servers moved to the lab

19


reserved.

Saturday AM

We had most of the servers racked, powered and network

attached Saturday night. These are all cabled and ready to be

turned on.

20


reserved.

Sunday testing

Of course the last thing we wanted to see was a power truck at the end of Minuteman road on Sunday, but nevertheless that’s what happened. High winds caused power issues in the park. Luckily our generator and UPS allowed us to continue testing.

23


reserved.

Data Center update

Move was successful

– Virtualized 37 servers

– Retired approximately 15 servers

– Moved 110 servers from Boston to Andover and another 36 in Andover

– Removed and reinstalled approximately 160 fiber connections, 700

copper connections and 260 power cords.

– Designed, installed, deployed and tested backups for 80 servers (23

TB)

– Virtual servers (all 37) moved in under 2 hours. 90 minutes was waiting

for the snapshots to finish.

– Over 2500 extra hours of planning, designing, testing, moving and

troubleshooting were involved in this move.

24


reserved.

Tip 1 - Plan everything

“Plans are useless, planning is essential”

25


reserved.

Tip 2 - The plan is only as good as the

documentation.

“If the documentation is 90% correct, 10% of the stuff won’t

work”

26


reserved.

Tip 3 – Communicate early and consistently.

“When people don’t hear, they assume the worst”

27


reserved.

Tip 4 – People are, well people

“Sleep is for the weak”

28


reserved.

Tip 5 – Check supplies

“Getting a roll of tape Tuesday afternoon is easy. At 3:00AM on

Saturday in the middle of nowhere is not”

29


reserved.

Tip 6 – Review process

“Wait for the admin to shutdown the box first”

30


reserved.

Tip 7 - Timing

A team of two people will take

–5 minutes to rack a server in the cabinet

–5 minutes per Ethernet cable

–2 minutes per power cable

31


reserved.

Tip 8 – Always have plan B and C

“If the elevator breaks can you really move all of that up the stairs?”

32


reserved.

Tip 9 – Do what you can early

“Anything that can already be done, should be”

33


reserved.

Tip 10 – Test and troubleshoot

“You aren’t done until it’s tested”

34


reserved.

Tip 11 – review and recognize

“You never know when the next move will be”

35


reserved.

Bonus Tip– Because 12 tips sounds better

than 11

Be there

36


reserved.

afcom how to move a data center

Technology

data center networks

colo data center space

virtual data centers

wrong way

dumb things

fine printeveryone

facility lesson

disaster avoidance