meaningful security metrics

Download Meaningful Security Metrics

If you can't read please download the document

Post on 28-Nov-2014




3 download

Embed Size (px)


Dr. John D. Johnson delivers a presentation on security metrics to the Cyber Security Strategies Summit, Washington, DC, May 2011.


  • 1. John Johnson, PhD, CISSPSr. Security Program Manager
  • 2. } Setting the Stage Defining Terms and Setting Expectations} Exploring the Problem A historical perspective Reaching out to other disciplines} Gathering Data Both qualitative and quantitative data sources} Models for Making Sense of the Data Making sense of large data sets Asking the right question, getting the right answer} Deriving Value and Driving Improvements Examples of Operational, Strategic & Business Metrics Building Your Security Metrics Program} Audience Discussion
  • 3. } Performance metrics: measure how well an organization performs and drive process improvements and demonstrate value-add} What are we actually measuring? Are we more secure today than yesterday? How do we compare to our peers?} We are often stuck with what our tools provide The Cycle: Detect Report Prioritize Remediate} Security metrics be made meaningful; this means they should provide value to stakeholders We need to learn to ask the right questions, if our results are going to be meaningful The best metrics are SMART: Specific, Measurable, Attainable, Repeatable & Time-Dependent
  • 4. } Coming up with meaningful security metrics is an inherently difficult problem, but we can: Draw upon examples from other disciplines Realize that there are many ways to tell a story so that it is meaningful to stakeholders, focus on impact and outcomes Recognize what we do not know and cannot measure, and still do our best to account for these threats Find ways to quantify security activities Gather more data from different sources Work together to define standard frameworks for analyzing data and drawing conclusions
  • 5. From: Stars in the Water by Lesley DuTemple, Illustration by Jack Oyler
  • 6. } born 1546 / died 1601} A Danish nobleman, known for his precise and comprehensive planetary and astronomical observations. } Ive studied all available charts of the planets and stars and none of them match the others. There are just as many measurements and methods as there are astronomers and all of them disagree. Whats needed is a long term project with the aim of mapping the heavens conducted from a single location over a period of several years. does this sound familiar?} Also known for wearing a golden nose, after losing his real nose in a duel, and for dying due to bladder complications after throwing a really good party, but refusing to leave to use the bathroom.
  • 7. } born 1571/ died 1630} Worked with Tycho Brahe, until Brahes untimely death.} Kepler had access to volumes of quantitative data on the planets and he developed a scientific theory (a model of planetary motion).} Keplers Laws provided a foundation for Newtons Law of Gravity and transformed forever the way we see the night sky.
  • 8. } Born 1564 / died 1642} He built the first telescope for observing the heavens. (early adopter)} He drew what he observed, but had no model, so his results turned out to be nonsense (later corrected by Huygens).} Nevertheless, he did not give up and his many observations (phases of Venus, Jupiters moons) vindicated the heliocentric model and led to the birth of modern science.
  • 9. Galileo did the best he could As the tools and modelswith what he had improve, knowledge improvesGrainy Data, No Model Good Data, Good Model
  • 10. Graphic showing oil prices ( Pedro Monteiro of the What Type blog): Coarse Data, Confusing
  • 11. Jon Peltier added better and finer-grained data, and asked more meaningful questions to come up with graphics that were easier to interpret. Good data visualization should be easy to read. Which of these would you take to your management?
  • 12. This is a very simplistic graph, showing the kind of data you could get fromAV tools in 2006. Reporting was limited and data was coarse, but it wascombined with pertinent facts to explain fluctuations.
  • 13. } Ordinal numbers are used to rank, or create stop light graphics Red, Yellow, Green High, Medium, Low No units or scales} Cardinal numbers have units and can better be interpreted to add value More precise Can compare across business units, companies Can be used to establish a baseline
  • 14. } Motivators Regulations - Compliance Audits (both internal and external) Money (security is rarely a profit center) Responding to new threats Enabling new technology and business processes} Qualitative and Quantitative Data Traditionally, ordinal data and storytelling has been good enough Quantitative data can be automated & more consistent New threats and shorter exploit times means detection and response needs to be quicker A mix of both types of data and more standard models are needed to respond
  • 15. } Good standard risk assessment frameworks exist to address some of our concerns Examples: FAIR, VERIS Provide standard taxonomy for describing risk Standard for gathering and expressing data in consistent manner Allows for analyzing complex risk scenarios} SEIM projects can add value, if you are ready to do something with all the data you collect, and if someone is going to look at it Large data sets can be difficult to filter and reduce, while maintaining integrity} Not all industries have the same risks and priorities} There can be legal issues, when dealing with various data types, that vary by country} Data visualization and mining tools can help discover issues by looking at data from different vantage points, and prompting drill-down and asking better questions
  • 16. VERIS is a set of metrics designed to provide a common language for describing security incidents in a structured and repeatable manner. The overall goal is to lay a foundation on which we can constructively and cooperatively learn from our experiences to better manage risk. Each incident is a chain of events, composed of: Agent, Action, Asset, Attribute. 2011 Verizon. All Rights Reserved. MC14949 04/11. The Verizon and Verizon Business names and logos and all other names, logos, and slogans identifying Verizons products and services are trademarks and service marks or registered trademarks and service marks of Verizon Trademark Services LLC or its affiliates in the United States and/or other countries. All other trademarks and service marks are the property of their respective owners.
  • 17. } Most companies have various regulatory reporting requirements (i.e. SOX, PCI). Suggested metrics: Manager sign-off on access controls A&A control artifacts Audit reports/findings (number, severity, BU) Exception reporting/tracking PCI Complia


View more >