matt carroll - "security patching system packages is fun" said no-one ever

Join the conversation #devseccon

By Matt Carroll

“Security patching system packages is fun!” said no-one ever.

Matt Carroll - Site Reliability Engineermattc@yelp.com

@oholiab

“Security patching system packages is fun!”

said no-one ever.

Some security tasks are just a dragSocial engineering is for protagonists too!

Use tooling to minimise pain and maximise motivation

You CAN use technology to help solve people problems

Lol package management.

The Takeaway

WHO?

mattcQualifications:●Worries too much

Who’s this then?

https://www.flickr.com/photos/ajc1/10994593713

Yelp’s MissionConnecting people with great

local businesses.

Yelp StatsAs of Q2 2016

92M 3272%108M

Building our PaaS: PaaSTA!Managing our edgeSupporting deploys and developer workflows

Server/instance maintenanceToolingThe kitchen drawerBackronyming badly

Not just rebooting and saying no.

Operations team

KC Green

heartbleed.com

KC Green

Patching packages is hard and boringThings that are difficult to upgrade in place without downtime

Technical debt and edge casesPackage freezingYelppacksFrankenLucidDockerTrying to do clever things with apt

So what’s the problem?

Doing clever things with apt

http://scarfolk.blogspot.co.uk/2013/05/the-dont-campaign-and-kak-1973.html

#!/bin/bash[ -z "$1" ] && exit 2

library_regex="$@"

mapped_deleted=""while read process; do pid=$(echo $process | awk '{print $1}') cmd=$(echo $process | awk '{print $2}') map=$(sudo grep -E "$library_regex" /proc/$pid/maps 2>/dev/null) if ! [ "$map" = "" ]; then echo -e "\n${process}\n------------" echo "$map" if echo "$map" | grep -q "(deleted)"; then mapped_deleted="$mapped_deleted\n$cmd" fi fidone< <(ps --no-header -eo pid,comm)

if [ "$mapped_deleted" = "" ]; then exit 1else echo echo "NEEDS RESTART" echo -n "=============" echo -e "$mapped_deleted" | sort | uniq exit 0fi

IN B4 APT-GET UPGRADE

For tech debtFor reproducible buildsBecause apt does silly thingsTo generally be aware of impact of CVEs on critical components

At least until we’ve built confidence

But it’s really really dull!

Needs eyes on from an engineer

A JIRA projectUbuntu Security Notice emails straight to tickets

Wheel of Misfortune

Enter AUTOSEC

Engineers tend to like:

Interesting workNew thingsTo know where they standA tight feedback loopAgency

Tedious stuff is tedious

Engineers tend to like:

Interesting workNew thingsFeedbackAgency

Tedious stuff is tedious

https://pixabay.com/p-155981

Engineers tend to like:

Interesting work ❌New things ❌Feedback ❌Agency ❌

Tedious stuff is tedious

Engineers tend to like:

Interesting work ❌New things ❌Feedback ❌Agency ❌

Tedious stuff is tedious

I am not a wizard:

Interesting work ❌New things ❌Feedback ✅Agency ✅

What do I think we can fix?

Aim to reduce MTTR for security tickets within Q2Resolution within 2 weeks unless critical

Close out all pre-Q2 2016 tickets

We had organizational buy in

Already we have a better idea of where we stand as a team.

Enter AUTOSEC OKR

Automate distribution of workIncluding deadlines

Tighten up feedback loop with metrics and frequent reports

Make the critical path to decisive action more explicitImprove documentationMake it easy to get helpImprove perceived agency

How?

Asking non-security specialists to make security decisions

“Won’t Fix” is against engineering naturePrevent naive interventionismYou are making tradeoffs (absolute security vs moving faster than competitors)

You only find out if you did the wrong thingEmpower people to make hard decisions with little payoff

autosec-review mail group (leveraging JIRA again)Anything you can do to make it less painful

Increase Agency: Recognise the futility

JIRA gives us a bunch of stuff for free

We totally have a PaaS to put the Wheel of Misfortune on!(You could totally do this with a cron job)

The AUTOSEC service

Also hooks into JIRA

Work distribution

github.com/Netflix-Skunkworks/go-jira for ad-hoc metrics and mailouts

Helps team members know that they’re helping and what progress on the goals are like

Pretty much the only feedback you get

Feedback: Metrics and reporting

Proactively security patching system packages often feels more like an arcane ritual to satisfy the script kiddie gods than it does engineering. In part, this is because of a feedback loop that’s more of a feedback line… Post completion, you’re safe in the knowledge that you still haven’t been hacked that you’re aware of. Probably.But it’s still important – if your OS vendor has gotten round to announcing and fixing vulnerabilities to you, then they’ve landed in everyone else’s inbox too!This talk will address some of the problems inherent in defensive infrastructure security. It will give examples of how to change the problem space in order to motivate engineers toward being proactive in a field that is “everyone’s responsibility”. Hopefully this should give some insight into how you can leverage technology and pragmatism to instigate change in your security culture. By reducing the pain and uncertainty of taking action, you can make infrastructure security more rewarding and more effective as a result!

Clear up documentation on processEven so, a well defined process on paper is difficult to follow

Did it anywayIncrease agency by REMOVING extraneous information

Break points should happen as early as possibleShould ideally be scripted

Increase Agency: Critical path

Clear up documentation on processEven so, a well defined process on paper is difficult to follow

Did it anywayIncrease agency by REMOVING extraneous information

Break points should happen as early as possibleShould ideally be scripted

Increase Agency: Critical path

Deadlines really help you prioritize work

JIRA and cron(ish) againExtension of AUTOSEC serviceTells you when you’re nearing deadline

“I need it done now/ASAP/yesterday” are not deadlines

Helps to balance against the actually rewarding work

Increase Agency: nagbot

DID IT WORK?

WHAT NEXT?

Those stats are probably liesI hope none of our servers run on Snapdragon kernels…

Scrape information out of USNsAuto-triage information

Introspect with mcollective and what’s in our reposMaybe even auto-close?Feed information to documentation scripts to remove some of the questions

Pointless overhead

When processes are in flux, docs change

I don’t even want to read the docs once, and neither do you

Continue on with scripting workNo extraneous informationProcess is more interactiveProcess changes can be reviewed for greater confidence!

Mental caching

http://www.express.co.uk/finance/crusader/623732/Crusader-act-now-victim-PPI

Remove our old cruft (obviously)

Make puppet dpkg pin versions of packages we install via puppet

apt-get upgrade from upstream security becomes safer

Less complicated process means less can fall through the gaps

Fix packaging

Some security tasks are just a dragSocial engineering is for protagonists too!

Use tooling to minimise pain and maximise motivation

You CAN use technology to help solve people problems

Lol package management.

The Takeaway

Thanks for listening!

@YelpEngineering

fb.com/YelpEngineers

engineeringblog.yelp.com

github.com/yelp

Join the conversation #devseccon

Matt CarrollSRE at Yelp

mattc@yelp.com@oholiab

oholiab on Freenode