case study: embratel delivers the gold medal for infrastructure monitoring at rio 2016

World®’16

CaseStudy:EmbratelDeliverstheGoldMedalforInfrastructureMonitoringatRio2016

WandersonCoelhodeSouza- OperationsManager- Embratel

DO4X121S

DEVOPS– AGILEOPERATIONS

2 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.

Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.

ForInformationalPurposesOnlyTermsofthisPresentation


Abstract

In2016,RiodeJaneirowillbecomethefirstSouthAmericancitytohosttheOlympicandParalympicGames.CapitalizingonitsexperiencehostingthePanAmericanGamesin2007,andtheWorldCupin2014,the‘MarvelousCity’surpassedChicago,TokyoandMadrid.TheprojectisbudgetedatBrazilianReal28.8bn($7.6bn).ThispresentationwillsharetheexperienceofhowtoplanandoperatetheOlympicInfrastructureusingCAUUnifiedInfrastructureManagement.Therewillbe10typesofservice,includingdatacenters,internetaccess,high-capacitydata,voice,mobility,CATVandwifi.Duringtheevent,RiodeJaneirowillhostrepresentativesfrom206countriesandmillionsofvisitors,anditisestimatedthatmorethan21,000journalistswillcovertheevent– allofwhichrequiredanunprecedentedtelecomsandITinfrastructure.TheRio2016OlympicGameswillusemorethan60,000networkpointsconnectingalltheOlympicarenas,andapproximately18,000fixedlines,16,000SIMcards,threemillionSMSmessagesand15,000pointsofCATV.TheOlympicbackbone(370km)isequivalentto971lapsofastandardfootballpitch,orthelengthof6,800Olympicswimmingpools.

WandersonCoelhodeSouzaOperationsManagerEmbratel


Agenda

OLYMPICSGAMESINFRASTRUCTUREREQUIREMENTOVERVIEW

ROLEOFCAUNIFIEDINFRASTRUCTUREMANAGEMENT(CAUIM)

KEYDASHBOARDS

MOBILEAPPLICATIONUSECASE

1

2

3

4

32

41

BARRA

DEODOROMARACANÃ

COPACABANA

OLYMPIC GAMES 42PARALIMPIC GAMES 23

65 CHAMPIONSHIPS44 Test events4 Ceremonies (openings and closures)100 DAYS Torch relay across the country

CHAMPIONSHIPS IN 12 DAYS

CHAMPIONSHIPS IN 17 DAYS

+

4.8Bworldwideviewersin220countries30Kjournalists70Kvolunteers15Kathletes206countries9Mtickets300Kaccreditations

Alltheadminnetworks,gamesandwebhostingwerehostedinEmbrateldatacentersandmonitoredwithCAUIMtechnology

1600 servers (physical/virtual)40 network DC devices1 Pb used data54 M on real-time games processing

Thewholeenvironmentmonitored24x7

Olympicvenuesinfrastructure

33,380hardwaremodules113,472LANports440UCSservers5,159wifi accesspoints177securitydevices(firewalls,IPS,IDS)

Whatwashostedatthedatacenter

ü Accesscontrolcoreequipmentofallvenues(ACS)ü ControllingaccesstotheAPsofallvenuesü Accesscontrolforallusersofthegames,forwiredand

wirelessnetworks(ISE)ü DHCPserver/DNS/VPNü Firewalls,IPS,IDS,balancerstrafficü AdminRio2016network(servers/storage/backup)ü Gamesnetwork- gameinformationdisclosuresystem

(servers/storage/backup)

Duringthegames,approximately2,200 ticketswereopenedforthedatacentersupportteam

SLA/GamesTime ResponseTime FixTime

Severidade1 5min(90%) 1hr(90%)



Severidade4 1hr(90%) 6hr(90%)

ServiceRequest 1hr(90%) 8hr(90%)

AggressiveSLAlevelswereagreedtocovernormaloperatingandcriticalperiods

• Events test• Technical simulations• Games time

CAUnifiedInfrastructureManagementUsedtomonitorthestatusofallgameserversandnetworkdevicesinvenuesforRio2016

CAUIMwasusedbecauseitisascalable,easy-to-use,cloudresilient,andpowerfulITmonitoringtoolandprovides360degreevisibilityintosystemsandinfrastructureperformance

CA Unified Infrastructure Management (CA UIM)Providespossibilityofcontrollingandmonitoringthefollowingparameters:

ü System Monitoringü %CPU usage, processor queue length, processor timeü Disk usage (% andMb), disk time, disk transfers/read/write per sec, pagingü Memory usage (% andMb)ü Windows services, Windows event logsü Scanning directories, files

ü Network Monitoringü SNMP queries are sent towards network equipment (switches and firewalls)ü Interface traffic from network equipmentü ICMP ping of all the devices, servers and VMs

ü Application Monitoringü URL availabilityü Jboss/JAVAü Application processes (CPU usage, memory usage, threads);ü Application servicesü SQL/Oracle

Applications Support Team

CreationofinteractivedashboardswithCAUIMwaskeytoproactiveissuedetection/prevention

The badges below represent some of the manyportals supportedNote that one of the badges has a major (orange) alarm; the team at TOC24x7 with the help of the right screens and alerts in front of them, was ableto verify with agility and act quickly to solve the problem

WithCAUIM,itwaspossibletoverifythemetricsinfothroughthemainmanagerinterface

MIBshavebeendefinedformonitoring(CPU,interfaces,uptime,etc.)AllappliedinSNMPcommunities,includingtheloadbalancerappliances

Probesandthresholdswereessentialingamestime,providingassertiveroleinidentifyingtherootcauseandproblemresolution

VMwareVMax LoadAverage

Database

It waspossibletomonitorthe

VmotionsandDRSsofVMs.

Identifydiskfailures,generatingalerts,even

beforetheidentificationinthestoragetoolitself

1-5minutes,identifyquicklythegrowthoftheapplicationqueues

Transactionslogandincreaseoftablespace

StatisticsfromAug.5toAug.17ü MobileApp

ü Totalvisitors:12,768,127ü Totalpageviews:781,877,340

ü Overallü TotaltrafficintheRio2016backbone:1,794PBsü Totalinternettraffic:632TBsü Uniquedevicesinthenetwork:155,226ü Peakofsimultaneouslyconnectedwifi devices:14,520ü Peakofinternettraffic:5.35Gbps

CAUIMhelpedtrackuserexperienceissueswithmobileapplication

ü The mobile application launch to follow the torch relayü Internet statistics showed that every year the internet consumption has undergone access via desktop

to mobile devices; in March 2012 the number of users was 2,336,000 in December 2015 around 3.366million and estimated growth of approximately 51% to games time

ü Mobile application launch for games timeü Withthenewapplication,itwaspossibletotracktheresultsofonlinegames,receivebreakingnews,

informationonVenues,athletesandmonitoringoftheentireevent,usingmobile

LoadBalanceTraffic InternalNetworkTraffic

OracleMemoryUtilization AllErrors@CDN

ContinousCAUIMdatasharingw/appdevfromAug.10-14resolvedseveralperformanceissues


SummarybenefitswithCAUIM

BETTERGAMESANDAPP

EXPERIENCE

SINGLEVIEWINTORIOOLYMPICSITINFRASTRUCTURE

IMPROVEDMOBILE

APPLICATIONPERFORMANCE

PROACTIVEISSUERESOLUTION

RAPIDROOTCAUSEANDTRIAGE

TicketsResolvedDuringRioOlympics

2200


Thankyou.

Stayconnectedatcommunities.ca.com


DevOps– AgileOps

FormoreinformationonDevOps– AgileOps,pleasevisit:http://cainc.to/wYXSg6

case study: embratel delivers the gold medal for infrastructure monitoring at rio 2016

Technology