matt carroll - "security patching system packages is fun" said no-one ever
TRANSCRIPT
Join the conversation #devseccon
By Matt Carroll
“Security patching system packages is fun!” said no-one ever.
Matt Carroll - Site Reliability [email protected]
@oholiab
“Security patching system packages is fun!”
said no-one ever.
Some security tasks are just a dragSocial engineering is for protagonists too!
Use tooling to minimise pain and maximise motivation
You CAN use technology to help solve people problems
Lol package management.
The Takeaway
WHO?
mattcQualifications:●Worries too much
Who’s this then?
https://www.flickr.com/photos/ajc1/10994593713
Yelp’s MissionConnecting people with great
local businesses.
Yelp StatsAs of Q2 2016
92M 3272%108M
Building our PaaS: PaaSTA!Managing our edgeSupporting deploys and developer workflows
Server/instance maintenanceToolingThe kitchen drawerBackronyming badly
Not just rebooting and saying no.
Operations team
KC Green
heartbleed.com
KC Green
Patching packages is hard and boringThings that are difficult to upgrade in place without downtime
Technical debt and edge casesPackage freezingYelppacksFrankenLucidDockerTrying to do clever things with apt
So what’s the problem?
Doing clever things with apt
http://scarfolk.blogspot.co.uk/2013/05/the-dont-campaign-and-kak-1973.html
#!/bin/bash[ -z "$1" ] && exit 2
library_regex="$@"
mapped_deleted=""while read process; do pid=$(echo $process | awk '{print $1}') cmd=$(echo $process | awk '{print $2}') map=$(sudo grep -E "$library_regex" /proc/$pid/maps 2>/dev/null) if ! [ "$map" = "" ]; then echo -e "\n${process}\n------------" echo "$map" if echo "$map" | grep -q "(deleted)"; then mapped_deleted="$mapped_deleted\n$cmd" fi fidone< <(ps --no-header -eo pid,comm)
if [ "$mapped_deleted" = "" ]; then exit 1else echo echo "NEEDS RESTART" echo -n "=============" echo -e "$mapped_deleted" | sort | uniq exit 0fi
IN B4 APT-GET UPGRADE
For tech debtFor reproducible buildsBecause apt does silly thingsTo generally be aware of impact of CVEs on critical components
At least until we’ve built confidence
But it’s really really dull!
Needs eyes on from an engineer
A JIRA projectUbuntu Security Notice emails straight to tickets
Wheel of Misfortune
Enter AUTOSEC
Engineers tend to like:
Interesting workNew thingsTo know where they standA tight feedback loopAgency
Tedious stuff is tedious
Engineers tend to like:
Interesting workNew thingsFeedbackAgency
Tedious stuff is tedious
https://pixabay.com/p-155981
Engineers tend to like:
Interesting work ❌New things ❌Feedback ❌Agency ❌
Tedious stuff is tedious
Engineers tend to like:
Interesting work ❌New things ❌Feedback ❌Agency ❌
Tedious stuff is tedious
I am not a wizard:
Interesting work ❌New things ❌Feedback ✅Agency ✅
What do I think we can fix?
Aim to reduce MTTR for security tickets within Q2Resolution within 2 weeks unless critical
Close out all pre-Q2 2016 tickets
We had organizational buy in
Already we have a better idea of where we stand as a team.
Enter AUTOSEC OKR
Automate distribution of workIncluding deadlines
Tighten up feedback loop with metrics and frequent reports
Make the critical path to decisive action more explicitImprove documentationMake it easy to get helpImprove perceived agency
How?
Asking non-security specialists to make security decisions
“Won’t Fix” is against engineering naturePrevent naive interventionismYou are making tradeoffs (absolute security vs moving faster than competitors)
You only find out if you did the wrong thingEmpower people to make hard decisions with little payoff
autosec-review mail group (leveraging JIRA again)Anything you can do to make it less painful
Increase Agency: Recognise the futility
JIRA gives us a bunch of stuff for free
We totally have a PaaS to put the Wheel of Misfortune on!(You could totally do this with a cron job)
The AUTOSEC service
Also hooks into JIRA
Work distribution
github.com/Netflix-Skunkworks/go-jira for ad-hoc metrics and mailouts
Helps team members know that they’re helping and what progress on the goals are like
Pretty much the only feedback you get
Feedback: Metrics and reporting
Proactively security patching system packages often feels more like an arcane ritual to satisfy the script kiddie gods than it does engineering. In part, this is because of a feedback loop that’s more of a feedback line… Post completion, you’re safe in the knowledge that you still haven’t been hacked that you’re aware of. Probably.But it’s still important – if your OS vendor has gotten round to announcing and fixing vulnerabilities to you, then they’ve landed in everyone else’s inbox too!This talk will address some of the problems inherent in defensive infrastructure security. It will give examples of how to change the problem space in order to motivate engineers toward being proactive in a field that is “everyone’s responsibility”. Hopefully this should give some insight into how you can leverage technology and pragmatism to instigate change in your security culture. By reducing the pain and uncertainty of taking action, you can make infrastructure security more rewarding and more effective as a result!
Clear up documentation on processEven so, a well defined process on paper is difficult to follow
Did it anywayIncrease agency by REMOVING extraneous information
Break points should happen as early as possibleShould ideally be scripted
Increase Agency: Critical path
Clear up documentation on processEven so, a well defined process on paper is difficult to follow
Did it anywayIncrease agency by REMOVING extraneous information
Break points should happen as early as possibleShould ideally be scripted
Increase Agency: Critical path
Deadlines really help you prioritize work
JIRA and cron(ish) againExtension of AUTOSEC serviceTells you when you’re nearing deadline
“I need it done now/ASAP/yesterday” are not deadlines
Helps to balance against the actually rewarding work
Increase Agency: nagbot
DID IT WORK?
WHAT NEXT?
Those stats are probably liesI hope none of our servers run on Snapdragon kernels…
Scrape information out of USNsAuto-triage information
Introspect with mcollective and what’s in our reposMaybe even auto-close?Feed information to documentation scripts to remove some of the questions
Pointless overhead
When processes are in flux, docs change
I don’t even want to read the docs once, and neither do you
Continue on with scripting workNo extraneous informationProcess is more interactiveProcess changes can be reviewed for greater confidence!
Mental caching
http://www.express.co.uk/finance/crusader/623732/Crusader-act-now-victim-PPI
Remove our old cruft (obviously)
Make puppet dpkg pin versions of packages we install via puppet
apt-get upgrade from upstream security becomes safer
Less complicated process means less can fall through the gaps
Fix packaging
Some security tasks are just a dragSocial engineering is for protagonists too!
Use tooling to minimise pain and maximise motivation
You CAN use technology to help solve people problems
Lol package management.
The Takeaway
Thanks for listening!
@YelpEngineering
fb.com/YelpEngineers
engineeringblog.yelp.com
github.com/yelp
Join the conversation #devseccon
Matt CarrollSRE at Yelp
[email protected]@oholiab
oholiab on Freenode