managing remote operation teams

1

Managing Remote Operation Teams

Sagi Brody, CTO@webairsagisagi@webair.com

2

Definitions

• Mixed in-house and remote employees

• Fulltime remote employees

• Remote office with remote employees

• Remote contractors

• Dedicated technicians via 3rd party support vendor

• Shared technicians via 3rd party support vendor

• 24/7 Technical Support (L1/L2)

• Infrastructure Support (Network/Cloud/Storage)

• Provisioning

• Development

• Management

Who?

Doing What?

3

Why Consider Remote Operation Teams?

• Take advantage of a larger talent pool• Enables Strategic Placement: • Infrastructure/Colocation (Boots on the ground)• Customer-base or language• Emerging markets• 24/7 Coverage

• Sets up company for scale & growth• Employee Flexibility:• Full or Partial remote work• Select from available locations• Start a new location strategically

• Lower Costs• Built in management layer and accountability (3rd party)

4

The Challenges

• Language & Culture Differences• Time Zones• Creating an Inclusive Environment• Real-time Communication• Monitoring• Response Time & SLAs• Customer Interaction• Rewards & Compensation• Long Term Employee Engagement • Technical Knowledge, Training, and standards• Liability, Tax, and Financial implications

5

Our Environment

Founded: 1996Headquarters: Long Island, NYServices Offered: Public, Private & Hybrid Cloud, Dedicated Servers, Colocation, CDN, Security, DRaaS, and moreCustomers: Enterprise, Healthcare, SMBs, VoIP providers, eCommerce, Information Technology, and more

The Webair Value:  Over 18 years providing customers with best-in-class Managed Hosting solutions High-touch Support Full ownership of our customer’s infrastructure stack so they can focus on their core

business. Manages most secure and redundant facility east of Manhattan with trans-Atlantic bypass

(Tier III, HIPPA/SSAE16)

6

Global Presence

Los Angeles, California• Technicians local to Colo• Infrastructure Engineers

Bodrum, Turkey• DevOps

Amsterdam, Netherlands• Technicians local to Colo• Systems Administrator

Boston, Massachusetts• Systems Administrators

Coimbatore Tama Nadu, India• Technical Support

Managers• Systems Migration

Specialists• Systems AdministratorsMontreal, Canada• Data Center Technicians

Long Island, New York (Flagship)• Customer Service

Specialists• Infrastructure Engineers• Project Managers• Account Representatives• PHP / Linux Developers• System Administrators

7

8

Language & Culture

• Make a conscious decisions you’re willing to deal with the nuances

• Understand local holidays• Implications of vacation time• Phone support & accents• Provide ESL training where applicable (even in house..)• Be respectful of local cultures, religions, and acceptable

chat topics• Language Differences:• South Africa: Floppy disks vs

Stiffy Disks • Indian English: “Do One

Thing..”• “I have a doubt about ssh

keys” (question)

9

Team Inclusion

• Do whatever you can to promote the human factor. They’re not just names on a chat screen

• Don’t change people’s names!! Promote personalization.• Engage them like in-house employees (small talk, build a

rapport)• ‘Good’ Video chat for regular in-house meetings, stand ups,

and scrums• Opportunistic meeting (conventions)• Meet face to face at least yearly to maintain relationship

and train• Swag & Business cards• Challenge them• Understand you may not be

approachable• Compensation, Rewards, and performance

reviews

10

Managing & Monitoring Complex Infrastructure Remotely

• How can our remote teams support & troubleshoot complex setups with little to no information on them?

• LBs, App, Web, DB, FW, NM, CDN.. (non Cpanel..)

• Distributed, virtualized, and abstracted infrastructure - Network, Storage, Compute.

• Self-healing & Auto scaling

• Microservices based architecture

• Troubleshooting this == An Art.

11

Communication & Context

Make it part of your culture.

12

Centralized Monitoring

13

Centralized Monitoring & Context

14

Monitoring for Success

• Fast resolution starts with a solid monitoring strategy.• Empower DevOps to create layered alerts.• Create simple interfaces, scripts, APIs to allow for easy

additions of monitoring to standardized systems.• Make it part of development & build process.• Monitor the Big and the Small:• Big - The finished product. Look for expected result via end

user interface to ensure all services are properly functioning.• Small - Every layer! OK/FAIL monitoring for multi-

dimensional services and 3rd party providers.• If setup properly, big and small alerts should trigger

simultaneously allowing instant localization.• Build automated actions triggered on alerts (log monitoring,

splunk..)

15

Remove Roadblocks..

• Don’t relegate remote Level1 teams to only observe. Empower them to participate in the long term success of your customers.

• Balance standard methods & procedures with flexibility.

16

Smart Monitoring & Troubleshooting

17

3rd Party Monitoring (NIH)

”In God we trust. Everyone else we Monitor.”

18

Aggregated Statistics

Cluster Wide

CPU

Network

Memory

Disk

19

Aggregated Statistics

Network by Host

Host 1

Host 2

Host 3

Host 4

20

Aggregated Statistics

Network by VM (Host 1)

VM 1

VM 2

VM 3

VM 4

21

Training & Technical Knowledge

• Documentation software & strategy• Owners for various sections• Automated article expiration• Built in diagram creation• Tie ins to other systems• Part of your culture, comments, and chat

• Knowledge sharing & training strategy• Prescheduled internal training• Mentor program, buddy system• Remove Silos• Build policy which discourages over-escalation

• Don’t forget the soft skills – Troubleshooting, Technical writing, Resourcefulness, Googling, Customer interactions

22

Documentation

“If it wasn’t documented, it didn’t happen.”

23

Security

• References & Background checks• VPN• Ability to quickly cut access • Build trust over time, tiered access levels• Activity based alerts (strange queries)• Historical logging (SSH & other), auditing• Unique IPs & hostnames• Device policies• File change monitoring (Tripwire)• Out of Band access for remote teams

24

Legal and Tax liabilities

• Customer privacy concerns (ie ePHI)• Remote team strategy in line with audits & compliance?

(SSAE16, HIPPA, etc)• Data stored in other countries • Impact on BAA agreements• Sufficient Cybersecurity insurance which covers remote

team members or outsourced?

• Tax deductions when paying out of state employees • Are NDAs enforceable?• Payment methods & Currency• Sanctions??

25

Do it Yourself!

• Expand Opportunistically• Strong remote team starts with trust –

Send someone you trust to start one!• Employees moving ‘back home’ can

start a remote office and recruit locally• Ensure remote management and

accountability• Are they capable of it? Do they want it?

26

Working with 3Rd Party Companies

• Presence in your country is a bonus• Do you receive dedicated support or shared?• What management options are available?• What is their employee retention rate?• Do they rotate employees or do you get to keep them

long term?• Who is accountable for that team, who can you trust?• Do they offer contracts?• Will they sign an NDA, who is liable for security?• Will local laws help or hurt you?• References

27

THANK YOU!

Sagi Brody@webairsagisagi@webair.com