11111P.ESCAPE

FROM•

Robust backups empowergovernment agencies to

avoid viral threats andprevent them from paying

hefty ransoms.

1111111b°by DAVID RATHS

photography by;=DWARD LINSMIER

v T „A‘.77 •

4001 .111 .0

90%Increase in detectedransomware attacksglobally in 2017 overthe previous year'

-401011*

VAREH e r m i n i o Rodriguez recalls

exactly where he was andwhat he was doing on themorning in February 2016when the city of Sarasota,

Fla., was hit by ransomware.Rodriguez, who had been the city's

IT director since November 2014, wasmeeting with Sarasota County officialsin their offices nearby City Hall. Hestarted receiving text messages fromstaff. The files on the city servers hadbecome encrypted and inaccessible. "Istepped outside and had a phone calland realized that there was somethinghorribly wrong," he says.

Racing back to City Hall, he and hissecurity analyst took the next 45 min-utes to verify that the city was under aransomware attack. Cybercriminalshad encrypted millions of files andwere demanding the equivalent of $34million in bitcoin to decrypt them.

Largely because of the VeeamSoftware Backup & Replication solu-tion that Sarasota had put in place onlyfour months earlier, Rodriguez wasconfident that the city's IT infrastruc-ture could be restored quickly, and henever considered paying a ransom. "Icould not have accepted a job where Iwould have to pay a bad guy to get myfiles back," Rodriguez says. "Thatwould mean I wasn't doing my job."

Source: 'Maiwarebytes Labs. "Cybercrime tactics and techniques: 2017 state of malware," 2018 SPRING 2019 • STATETECHMAGAZINE.COM 2 7

Many state and local governmentagencies are less prepared than Sarasota.Ransomware attacks highlight key mis-takes governments tend to make regard-ing disaster recovery planning: nottesting the operations of their backupsand not ensuring that backups areremote but accessible.

"Too many people think ransomwareonly happens to others — until they fallvictim themselves," says Alan Shark,executive director of the PublicTechnology Institute. "Experts recom-mend comprehensive backups. Whilethis is certainly sound advice, one mustbe very careful!'

"Data must be backed up separatelyfrom the application software andstored offsite. It must be cataloged andgo through a separate automated datascan that seeks out any malicious codeand flags anomalies," he adds.

VERIFY BACKUPSAlthough the city of Sarasota IT teammembers were confident they couldrestore their data, city workers mean-while were unable to do their jobs.

"Our finance, inspections and utili-ties departments were all down. It was ahuge wake-up call," Rodriguez says.

The ransomwarehad arrived as a fakeinvoice in an emailattachment, the ITteam determined. By4 p.m., the team iden-tified affected serversand began restoringthem with Veeam."The previous solutionhad an old virtual tapelibrary, but nobodyverified the solutionworked;' he says."Backups werenonexistent."

In addition to theVeeam solution, thecity had bought a newdisk target — a DellEMC Data Domainstorage appliance."Once we went to Veeam and DataDomain, everything changed,"Rodriguez says. "We crafted this newsolution and were just getting the hangof it when the ransomware hit."

Rodriguez knew the city's backupswere reliable thanks to regular testing,but he did not know how long it wouldtake for the restoration.

"Although we had done plenty of testrestores, we hadn't really done a majorrestore of this fashion," Rodriguez saysof the emergency. "We worked all nightand didn't go home!'

The next day, after talking to Veeamrepresentatives, Sarasota conducted anInstant VM Recovery, which allowedthem to mount a virtual machine image

A DECLINE OVERALL IN RANSOMWARE SAMPLESThe number of ransomware samples detected by McAfee globally declined over 2018. McAfee speculates thedecline was due in part to ransomware actors switching to cryptomining for more money.'

2,500,000 •

2,000,000 •

1,500,000 •

1,000,000 •

500,000 •

2016Q1 Q2 Q 3 Q 4

2 8 STATETECHMAGAZ1NE.COM • SPRING 2019 Source: 'McAfee, "McAfee Labs Threat Report," December 2018

"I could not haveaccepted a jobwhere I wouldhave to pay abad guy to getmy files back."

— Herminio Rodriguez, ITDirector for Sarasota, Fla

to a production host from a compressedand deduplicated backup file.

"By doing instant restores, we wereable to get back two servers almostimmediately," Rodriguez says. "We hadrestored a third server manually. Thatallowed us to bring all the data backonline within 14 business hours."

CONTINUITY FOR THE CITYWhen Chris McMasters became CIO ofCorona, Calif., several years ago, one ofthe first conversations he had was withthe police chief, who wanted to knowhow the city was going to protect itselffrom a ransomware attack. At the time,ransomware was shifting from targetingcorporations to governments, especiallypolice departments.

McMasters accelerated the city'sshift to Microsoft Government Cloud,including using Azure Backup andAzure Site Recovery to protect mission-critical applications and data. "That wasdefinitely one of the reasons we wentwith Azure — for the continuity wecould have to restore very quickly andnot have to pay ransom," McMasterssays. "That is also how we set priorities.We pushed mission-critical apps to thecloud first."

Corona is a city of approximately167,000 in Riverside County. WhenMcMasters arrived, he saw that thebackup was only a mile away, whilemost of the applications were hosted

ASSESSINGYOUR RISK

->

Deborah Snyder, CISO for thestate of New York, suggestsseveral tactics to protect againstthe threat of ransomware.• Assess business risk. "Take the

time to actually assess yourbusiness risk and the impact acyberattack would have on yourorganization. Know where yoursensitive data is — not just theobvious data and your trustedsystem of record, but unstruc-tured data as well," Snydersays. Identifying the location ofdata is the first step to protect itfrom ransomware.

• Put In data loss preventiontechnology. "Make sure youhave encryption everywherethat you have sensitiveinformation," Snyder says."Pay attention to employeetraining. You have to persuadeusers to do the right thing.They continue to be the mostchallenging element of ourstructure to harden." Bad actorsoften target human behaviorwith phishing attacks and othermeans to gain access to launchransomware.

• Pay attention to insiderthreats. "In this day and age,when heuristics and analyticsare available to implement,identifying undesirable andsuspicious end-user behavioris easier than it ever was,and there is no excuse fornot doing it," Snyder says.

on-premises. "That alarmed me," hesays. "What happens if we have anearthquake? The San Andreas Fault isnot that far from here. Having thebackup just on the other side of thefreeway wouldn't help us much."

His effort to find a new backup solu-tion dovetailed with other changes hewanted to make with Microsoft at theoutset. "We went to Office 365 andOneDrive and pushed our critical infra-structure — Active Directory andDomain Name System environments —into the cloud first," McMasters says.

In his first year in Corona, his teammoved at least 80 percent of servers intothe cloud. "The principal reason for thatdecision was continuity for the city," hesays. The main data center is 300 milesaway in Arizona, with a site recoverysystem set up in Texas.

REMOTE RECOVERYIn 2017, Arizona Strategic EnterpriseTechnology, the state's shared technol-ogy services organization, began movingits on-premises mainframe services tohosted services provided by IBM's Zcloud. In addition to the increased speedand computing power and reduction incapital expenditures, the move hasenhanced the state's disaster recoverycapability, says Suzan Tasvibi-Tanha,assistant director and chief of managedservices in the Arizona Department ofAdministration.

"In the past, we would copy ourmainframe logical partitions to anotherduplicate mainframe from theDepartment of Economic Security, butto fully recover in the case of an event, itwould take us a good 10 days. With IBM,we have brought that down to 72 hours,"she says.

IBM's Z cloud has a main data pro-cessing center in Colorado and a disas-ter recovery site in Raleigh, N.C. "Ourdata gets transmitted to both places andour disaster recovery logical partitionsare ready and configured. In case of adisaster, we could switch communica-tion to Raleigh from Colorado:' Tasvibi-Tanha says. "We did not have thosecapabilities in the past." ■

Read how Colorado mounted an effective ransomware defense withbackups at statetechmag.com/ColoradoRansomware.

SPRING 2019 • STATETECHMAGAZINE.COM ) 9