monitor the unmeasurable
TRANSCRIPT
devopsdays Portland 2016 Jennifer Davis Twitter: @sigje
Monitor the Unmeasurable
monitored, resilient to failure, and increase value to our organizationheartbleed in 2014 struck across all organizations, one vector of fragility emerged. Assessing and monitoring fragility will allow us to more proactively monitor our vulnerabilities.
CC Image courtesy of Fruit with Swedish Pancake by Janet Hudson on Flickr
Monitoring should be viewed as stack. Maybe not a pancake stack with tasty fruit, although integrated pancake delivery with pagerduty alerts would rock. While I wait for my event to resolve, I can eat tasty pancakes. Everything in your stack should be monitored, and made up of layers
CC Image courtesy of Concentrated warning by Anders Sandberg on Flickr
In all my time at Yahoo, I saw a number of signals that told me that something was wrong. When I went into different environments as a Chef consultant, I saw that it was something that impacted all environments large and small. This made me want to start talking in a bigger forum with others. What are the signals that we aren’t monitoring? How do we start monitoring them and proactively act on these rather than react?
Technology Optional
Monitoring doesn’t have to be technology driven. For example, as a manager I could track the quality of 1-1s with my reports, track who is making it to meetings regularly, and how they are spending their time. If one person (our diamond in this case) is doing all the grunt work and doesn’t any amount of time on projects, that may be impacting overall happiness. Too much toil work leads to unhappiness.
• Technology
• Organization
• Process
Monitor these 3 Types of signals.
CC Image courtesy of Train Signal at Brogdale Farm courtsey of Oast House Archive
I’m going to talk about 3 signals that are important to monitor. Easy to remember because it’s “TOP”. Technology organization and process.
• Dependencies,
• Consumers to producers, and
• Value generation.
Monitor Technology Signals
The missing technology monitoring aren’t the availability, error counts, latencies. These are important, but signals that we may ignore. Three examples of these signals are dependencies, consumers to producers and value generation.
Monitor dependencies
Monitoring dependencies is about monitoring the versioned artifacts that my artifacts depend on. In this example I’m using the berks dependency to see what chef cookbooks depend on the chef-client chef cookbook. There are 3 top level dependencies of cron, logrotate, and windows. Windows has an additional dependency on chef-handler. Ideally I pin my versions so I know exactly what works and doesn’t.
module.exports = leftpad; function leftpad (str, len, ch) { str = String(str); var i = -1; if (!ch && ch !== 0) ch = ' '; len = len - str.length; while (++i < len) { str = ch + str; } return str; }
left-pad
How many people impacted by left pad? This is the entire left pad module. It’s essentially a function that implements a basic left-pad string. Many packages depended on this simple package, including Babel, and React. In march of 2016, the author unpublished all of his work. This led to a lot of individuals who didn’t host their own artifacts getting impacted.
Monitor Consumers to Producers.
When I talk about monitoring consumers to producers I’m not talking about the the software algorithm. In this example, consumers are people who use the software but don’t contribute. Producers are people who are actively collaborating with the maintainers to produce reusable solutions, i.e. solutions that help the community and not just themselves. Whether software is opensource or properitary, producers are the people working on the software.
Monitor Consumers to Producers.
An example of a danger and why we need to monitor this is looking at Heartbleed. In 2014, the OpenSSL Software Foundation published information about receiving $2000 in donations, and one full time individual working on openssl. With these kind of investments supporting the software, it's not suprising that a vulnerability existed in this critical software that secures hundreds of thousands of web servers. If there is important software, we need to be monitoring consumers to producers. It doesn’t mean that we should be inventing software ourselves, because that software will have the same problem. In general, if you don’t have adequate producers to support projects you depend on, pay some producers to do that work whether it’s donating money or other resources to the open source projects.
Monitor Value Generation.
• Affinity,
• Single points of knowledge, and
• Burnout.
Monitor Organization Signals
Monitor Affinity.
• Shortens time to get work done.
• Reduces communication barriers.
• Build trust based on regard.
Value of Affinity
Monitor Affinity.
Monitor Single points of Knowledge.
CC Image courtesy of the trick is to keep breathing by Guillaume on Flickr
Productivity
40-hours-a-week(steady)
60-hours-a-week(declining)
2 week 4"week 6 week 8-weekstart
Graphing"productivity"and"overtime
Image courtesy of Laws of Productivity
Graphing)recovery)from)crunch
Productivity
1.--Crunching
2.--Crunching-ends
3.--Team-recovers
4.--Return-to-baselineA
B
Typically-A-≤-B
Image courtesy of Laws of Productivity
• Excessive gating of processes,
• Life cycle of products,
• Hiring and Termination.
Monitor Process Signals
Monitor Excessive Gating
CC Image courtesy of The Gates, 2005 by jschauma on Flickr
level of value versus level of effort
Shadow HR, Marketing, IT
Image courtesy of Ryan McGuire by Gratisography/
/
npm unpublish software issue
Monitor SLC Processes.
Monitor Hiring and Termination Process.
• Technology
• Organization
• Process
Monitor these 3 signals
CC Image courtesy of Train Signal at Brogdale Farm courtsey of Oast House Archive
Thanks! Twitter: @sigje