keeping it cool in a crisis (relate live sydney)

#RelateLive

#RelateLive

Keeping it Cool in a Crisis

Dave DysonZendesk Senior Customer Service Evangelist

@dave_dyson

Crisis CommunicationWhat we’ll cover

• Definitions • Goals • Ingredients for Success • Our Red Alert Process • Metrics

Definitions

What is this “Red Alert” that you speak of?Categories of Bad

• Service Disruption • Security Incident • Legal Entanglement • Public Relations Nightmare • Physical Emergency

Goals

The “Prime Directive”Your Mission

• Repair Trust with Customers • Restore Service ASAP • Consistent Response • Efficiency

WARMTH

CO

MPE

TEN

CE

WARMTH

CO

MPE

TEN

CE

Ingredients for Success

Planning AheadA Haiku

A Red Alert is not the time to figure out how to handle one.

KEEPCALM

AND

HAVE A PLANDOCUMENT IT

TRAINTAKE OWNERSHIP

AND

COMMUNICATE

Stay CalmEmotions are Contagious

As a leader, your emotions carry extra weight.

If you’re nervous, your team will be too.

EmpathizeShow You Care

Active ListeningHelps Defuse Anxiety

• Leave your ego behind • Frustration is normal • Let them speak • Verbal “nods” • Reflect what they say • Validate their emotions

“If you fail to plan, you are planning to fail” - Benjamin Franklin

“vaj nab luj SoH, luj nab SoH!” - Benjamin Franklin (original Klingon)

Red Alert PlansWhat to Include

• Process • Roles & Duties • Staffing • Tools • Communications • Special Cases • Metrics

StakeholdersThis isn’t just about you.

• Support Team • Engineering • Operations • Security • Marketing / PR • Sales • Account Management • Customer Success • Executives

Your Plan is Not Set in StoneHone & Refine

DocumentationIf it’s not written, it won’t be remembered

• Complete • Process & People • Clear • Accessible • Up to Date

TrainingPractice Makes Perfect

• Onboarding • Shadowing • Video • Drills

Take Ownership

• Empathize • Apologize • Don’t Shift Blame

Communication“All decks, this is the bridge…”

• Internal • External • Timely • Accurate • Compassionate • Honest • Transparent • One Voice

Red Alerts at ZendeskThe Plan in Action

Red Alert Process DocumentationKeep it secure, keep it safe

• Process Documentation • Checklist • Twitter & Flowdock • How to get Notified • Training Materials • On-Call Schedules

Zendesk Red Alert Process Overview

1. Assess the Situation 2. Alert the Incident Team 3. Communicate to Stakeholders 4. Public Acknowledgement 5. Status Updates 6. Resolve the Issue 7. Wrap Up

Assessment - CriteriaIs This a Red Alert?

Security Event A current, ongoing threat to the security of customer or Zendesk data

OR

Service Event Loss of critical Zendesk functionality

- AND -

The event impacts multiple customers

Impact:

Scope:

AssessmentSpecific Criteria

Critical Functionality: • Account Access • Performance Degradation • Channels • Core Partner Services • Business Rules / Routing • Agent Collision • Reporting • Apps Framework

Security Threat: • Compromised data • Spoofing • Stolen passwords • DDoS Attack

Internal AlertsTools and recipients

PagerDuty: Incident Response Team

(Support, Operations)

Email: Internal Stakeholders

(Engineering, Operations, Marketing)

Flowdock:Wider Team Visibility (Support, Operations)

Company-Wide Visibility

The Incident Response Team

Customers

Customer Advocates

Incident Lead

Support Duty

ManagerOperations

Manager

Engineering & Operations

Staff

Incident Response Team

Incident Response Team - StaffingOn-Call Duty Rotation

Weekly on-call shifts for each role • 8 hours x 7 days – based in AMER, APAC, EMEA

Avoid consecutive duty shifts • Standard: one shift every 6 weeks

Support Duty Manager has on-call Backup as well • Allows for escalation when primary duty manager is unavailable

PagerDuty used for scheduling & alerts • Dashboard shows who is on call & upcoming schedule • Individuals can customizable alerts for incidents and upcoming shifts

The Zencident Room

Red Alert - Problem & Incident Ticket

Problem Ticket • Created internally, owned by Incident Lead • Uses Service Disruption ticket form • Technical updates & discussions recorded here • No risk of inadvertent customer communication • Solves attached incidents automatically

Incident Tickets • Customer reports - attached to the Problem ticket • Proactive tickets attached as well

Service Disruption Ticket FormAdditional Fields

Added to the Problem Ticket: • Impacted Pods • Alert Duration • Number of Incidents • Link to public Help Center article • Link to internal Operations incident record • Checkbox: Post-mortem published

Communicating to CustomersKey Points

• Acknowledge Quickly • Take Ownership • Provide Scope and Impact • Regular Progress Updates • Be Transparent

Communication CadenceWhat We Share, and When

Within 15 Minutes: • @ZendeskOps: Acknowledgement • Create internal Problem ticket • Respond to customers & attach to Problem ticket

Within 30 Minutes: • @ZendeskOps: Scope & Impact • Publish Help Center article • Proactive Notifications to top customers

Every 30 Minutes: • @ZendeskOps: Status update • Update Help Center article • Update Top MRR Customers • (Reset cadence if necessary)

“Tell them what you would want to know”

- Susan Griffin-Black, EO Products

Resolution & Wrap-UpOnce service is deemed restored by the incident team:

1. Ticket Resolution • Solving the Problem ticket solves all attached Incidents • Include link to Help Center article for post-mortem

2. Send “All Clear” Tweet • Include link to Help Center article for post-mortem

3. Post-Mortem (within 3 business days) • Operations team writes internal version • Support Operations Incident Manager edits for public

consumption, and publishes to Help Center article

Service Disruption Post-MortemWhat should it include?

• Scope and Customer Impact • Incident Duration • Communication Timeline • Process / Training Gaps • Recommendations for Improvement

Special SituationsExceptions to the Rule

Security Incidents Involve the Security Team

Zopim Outage Communicate via Zopim Twitter & Facebook

Partner Outage (Twilio, GoodData)

Report incident to Partner Refer to Partner’s system status page

Zendesk Support Instance is Down

Use backup voice service (IfByPhone) Outgoing communication via Tweets only

Shift Handoff Live handoffs only - no email

Overlapping Incidents Separate Red Alerts & Problem Tickets

The Red Alert AppCustom App in our Zendesk Instance:

1. Displays links to active Red Alerts 2. Turn a ticket into a Red Alert ticket

• PagerDuty, Ticket Form & fields 3. Send Social Media Communications

• Twitter, Facebook • Composition, templates, character counts • Salutation and closing • “All Clear” option adds link to Help Center article • Creates or updates Help Center article

4. Proactive Ticket Communications • Lists top customers, with Pod & feature info • Composition, templates, all-clear option • Creates or updates Incident tickets • Updates Yammer

5. All actions add Internal Notes to Problem ticket

MetricsRed Alert Impact Report (Insights)

• Number of Red Alerts • Number of attached Incidents • Total Support Handle Time • Estimated Support Cost ($/ticket) • Number Proactive Tickets • Top Red Alerts (by # of incidents) • Red Alerts by “About” field • Customer Satisfaction over time

• Process Scorecard

Customer SatisfactionRed Alert Tickets

CSAT (Last 7 weeks):

100%

The Future!Where No One Has Gone Before

System Status Page (released) • More detail, account-specific Improved Red Alert App • More capabilities • Possible public version Wider Proactive Communication • More channels • More customers Incident Management as a Job • Not just a role

Conclusion

WARMTH

CO

MPE

TEN

CE

KEEPCALM

AND

PLAN AHEADDOCUMENT IT

TRAINTAKE OWNERSHIP

AND

COMMUNICATE

Smooth Sailing“Second star to the right, and straight on ‘till morning…”

#RelateLive

Q & A