fire drills: an analog for verifying your solution's emergency response plans
TRANSCRIPT
Drew Cox on twitter @DrewCoxSA
Fire Drills: Do Them UNLESS YOU’RE INTO TRUSTING YOUR FUTURE TO FATE.
#FireDrill
Who am I?❖ Father of two !
❖ Husband of one!
❖ Born & Raised Texan!
❖ San Antonian for 9 years!
❖ Big Picture Person!
❖ Very Contextual!
❖ Passions: Exploration, Community, Flavor, Soccer, Urbanity & Libations
#FireDrill
Professional Background❖ 4 Years @ Rackspace, all in the cloud!
❖ Weekly show focused on best practice!
❖ Worked with hobbyists & Fortune 500!
❖ Passion for startups !
❖ Major award shows!
❖ Shark Tank!
❖ ‘Reddit’ AKA ‘Oprah Effect’!
❖ Super Bowl Ads!
❖ Strengths: Ideation, Maximizer, Arranger, Connectedness, & Intellection
#FireDrill
Goal: I hope to instill in each of you a healthy fear
& a healthier curiosity
#FireDrill
Goal: I hope to instill in each of you a healthy fear
& a healthier curiosity
#FireDrill
Analogies are like…
#FireDrill
What if…
#FireDrill
Or what if…
a server fails?
#FireDrill
Or what if…
a server fails?
you get hacked?
#FireDrill
Or what if…
a server fails?
you get hacked?
you hit the front page of reddit?!?
#FireDrill
Or what if…
a server fails?
you get hacked?
you hit the front page of reddit?!?
you miss an alert because you phone was off?
#FireDrill
Or what if…
a server fails?
you get hacked?
you hit the front page of reddit?!?
auto scale = auto fail?
you miss an alert because you phone was off?
#FireDrill
Or what if…
a server fails?
you get hacked?
you hit the front page of reddit?!?
you get DDoS’d?
auto scale = auto fail?
you miss an alert because you phone was off?
#FireDrill
Or what if…
a server fails?
you get hacked?
you hit the front page of reddit?!?
your domain expires?
you get DDoS’d?
auto scale = auto fail?
you miss an alert because you phone was off?
#FireDrill
Or what if…
a server fails?
you get hacked?
you hit the front page of reddit?!?
your domain expires?
you get DDoS’d?
auto scale = auto fail?
you miss an alert because you phone was off?
my developer goes rogue?
#FireDrill
Or what if…
a server fails?
you get hacked?
you hit the front page of reddit?!?
backups are corrupted?
your domain expires?
you get DDoS’d?
auto scale = auto fail?
you miss an alert because you phone was off?
my developer goes rogue?
#FireDrill
What if…
#FireDrill
“Safety Rules”
#FireDrill
Elementary School
#FireDrill
#FireDrill
Middle School
#FireDrill
#FireDrill
High School
#FireDrill
#FireDrill
College
#FireDrill
#FireDrill
#FireDrill
“Trust, But Verify” -Ronald Reagan
#FireDrill
“Trust, But Verify” -Ronald Reagan
#FireDrill
“Trust, But Verify” -Ronald Reagan’s !
Translation of a Russian Proverb
#FireDrill
#FireDrill
#FireDrill
Draw a map of your home. Show all doors and
windows.Know your site
#FireDrill
Visit each room. Find two ways out.
Review Each Component &
Have a Fall Back#FireDrill
All windows and doors should open easily. You should be able
to use them to get outside.Identify Choke
Points
#FireDrill
Make sure your home has smoke alarms. Push the test button to
make sure each alarm is working.Verify Monitoring
Solution
#FireDrill
Pick a meeting place outside. It should be in front of your home.
Everyone will meet at the meeting place.
Agree on a Collaboration
Platform#FireDrill
Make sure your house or building number can be
seen from the street.Brand a !
“Sorry Page”
#FireDrill
Talk about your plan with everyone in your home.
Your team needs to know the plan
#FireDrill
Learn the emergency phone number for your fire
department. Know Your
Escalation Path
#FireDrill
Practice your home fire escape drill!
I think you’ve got this one…
#FireDrill
❖ Monitoring!
❖ Recovery Time (reboot)!
❖ Fail Over!
❖ Backup!
❖ Scale Up!
❖ Scale Down!
❖ Traffic Spike!
❖ DDoS Attack !
!
!
What Needs Verification?❖ Domain Expiration!
❖ SSL Expiration!
❖ Payment Gateway is down!
❖ Higher Employee!
❖ Fire Employee!
❖ Compromise !
❖ Re-Seller Conflict!
❖ Etcetera
#FireDrill
❖ Monitoring!
❖ Recovery Time (reboot)!
❖ Fail Over!
❖ Backup!
❖ Scale Up!
❖ Scale Down!
❖ Traffic Spike!
❖ DDoS Attack !
!
!
What Needs Verification?❖ Domain Expiration!
❖ SSL Expiration!
❖ Payment Gateway is down!
❖ Higher Employee!
❖ Fire Employee!
❖ Compromise !
❖ Re-Seller Conflict!
❖ Etcetera
#FireDrill
Fire Drill Process1. Make a plan!
2. Document your plan!
3. Schedule the fire drill (off hours)!
4. Run the fire drill!
5. Identify room for improvement!
6. Implement changes!
7. Repeat
#FireDrill
Tips & TricksHistorical issues are great places to start!
!
!
!
!
!
#FireDrill
Tips & Tricks
#FireDrill
Tips & TricksTest run all patches & upgrades pre-prod!
!
!
!
!
!
#FireDrill
Tips & TricksIdentify high value/risk portions of your site!
!
!
!
!
!
#FireDrill
Tips & TricksModular solutions can be tested in isolation
#FireDrill
Tips & TricksLoad test to mitigate for high traffic events
#FireDrill
Fire Drills: Do Them!
#FireDrill
#FireDrill