carolyn engstrom - it data analytics: why the cobbler's children have no shoes
TRANSCRIPT
Carolyn M. Engstrom
Gain a new perspective on the problem of IT Data Analytics
Leave with inspiration and information about how to apply data analytics to achieve value
The cobbler is the IT department which uses his skills and tools to
make shoes.
Shoes are metrics, output, analysis, etc.
Shoeless children are internal processes.
IT doesn’t apply tools and skills to
meet it’s own goals
Audit and Compliance are child protective
services. “Your children have no shoes!!”The broader
organization helps design them and
uses them.
Big Data centric Metrics focused Necessary evil of compliance Effectiveness dominates Efficiency lags Structured, centralized data Enterprise solutions Security Event and Incident Management Analytics are afterthoughts of implementation
Data quality worries Data efficiency worries Need to predict, forecast Historical reporting Siloed knowledge of business process “Gartner Says Power Shift in Business
Intelligence and Analytics Will Fuel Disruption”
CIO: 21 Data and analytics trends that will dominate 2016
• Statistical• Predictive Models
• Really big data!• Lots of sources!• Really important
issues to solve!
• End-user focused• Reporting• Summarize• Drill Down
• Outside Data Sources
• Unstructured Data• Extract, Transform,
Load
Source: “What Kind of Big Data Problem Do You Have?” SAS, 2014
Reac
tive
Proa
ctiv
e
Large Big Data
Dat
a Ca
pabi
lity
Data Size
Big Analytics Big Data Analytics
Business Intelligence
Big Data Business
Intelligence
They come in many sizes Big: $$$◦ Aggregated from External Sources◦ Primarily Big Data Business Intelligence
Medium: $$-$$$◦ Aggregated from Internal and External Sources◦ Operational and Security Information
Small: $◦ Internal Accumulation◦ Risk Assessment◦ Context, Calibration, Criticality
“Actionable Security Intelligence From Big, Midsize and Small Data “ by C. Warren Axelrod, Ph.D., CISM, CISSP – ISACA Journal, 2016
Achieve Insight Uncover Meaning Improve Assurance/Effectiveness Improve Efficiency Identify Trends Demonstrate Progress Prototype Requirements Improve Data Integrity Unlock Knowledge Management
“A Practical Approach to Data Analytics”, ISACA, 2011
Black box auditing◦ Frameworks◦ Methodology◦ Audit procedures◦ Standards of fieldwork
Evolving data analytics skillset Reports lack a persuasive story, meaning or
context Unique exposure to data, processes, and
risks
Source: CEB Audit Leadership: Peer Feedback- Data Analytics Vendors 2014
Define a population◦ Controls or risks
Information Provided by Client (IPE)◦ Population integrity
Non-statistical Sampling: based on frequency◦ Annual, Semi-annual, Quarterly, Automated = 1◦ Monthly = 2◦ Weekly = 5◦ Daily = 15◦ Many times daily = 25
Statistical Sampling◦ Confidence Intervals, 90% or 95%◦ Mathematical function identifies sample size◦ Not frequently used
100% population analysis◦ 1 source- IPE◦ 2 sources or more- Data Integrity◦ Removes population bias◦ Provides quantifiable measure of effectiveness
Assessment of exceptions All of these techniques support an auditor’s
conclusion
Source: CEB Audit Leadership: Peer Feedback- Data Analytics Vendors 2014
Expand your perspective on data◦ Transaction◦ Trending◦ Continuous Monitoring
If data is valuable, for the love of goodness, DO NOT USE A WORD DOCUMENT as a source of truth… EVER.
Control: For SOX in-scope apps job completion is monitored and abends are recorded in ticket software and resolved.
Batch process extracts job
fails from log
Employee selects a sample of
25
Employee searches for ticket
Employee records results in a Word doc. Embeds job
log object(s)
Monthly Quarterly 25 manual times 4Q x n apps
Audit used logs to
select their own
sample
Testing covers only failures◦ How many jobs ran successfully?
Only applied to SOX applications Manual process◦ Required about 2-3 hours per quarter per app◦ Multiple control owners
Audit coverage was minimal % of population Files maintained all over the network
Batch process extracts job completions
and fails from log
Monthly
Use data prep software to format logs
User extracts tickets
Compares
Exceptions: not timely, no ticket
Sends Exceptions
Report sent to control owner
Redesign cost about $2000 for data prep Time investment of about 40 hours Quantitative assurance◦ 100% SOX population coverage◦ 100% exception coverage
Context of success, failures, exceptions (%) Correct data quality issues Centralize file storage Increase frequency to monthly from quarterly
but decrease time
Build a table for jobs and attributes◦ Interfaces Data flow of confidential data Data flow of financial data◦ Report Integrity◦ Job number◦ Criticality
Build knowledge management Use data visualization rather than reports
Narratives about 6-20 pages long Topics◦ Access Controls◦ Change Management◦ Interfaces ◦ Job Resolution◦ Infrastructure Identification (asked to update xls) App servers Database servers and instances Servers (OS, location)
Identified Business Processes, but not financial statement accounts or disclosures
Narrative of an actual process Identify financial statement accounts and
disclosures Identify key controls May identify key reports by name Identify information on interfaces
Productionize Application Narrative◦ Change management application attributes◦ Created report out of the application◦ Improved population for change management
controls Foster Audit Knowledge Management ◦ Key Reports◦ Interface information to Chart of Accounts◦ Financial Statement Line Items◦ Custom Report for Review
Create relationships among data that was previously locked
Transform unstructured data Enforce consistency Content is more accessible Less data to maintain Improve efficiency and effectiveness of
existing tools
No previous defined vulnerability management process
Select a large-scale tool for vulnerability identification
Delays in projects due to incomplete network topography
Use Nessus to scan sample of servers (20) Collect data to baseline scores Use scripts to collect ◦ Patch levels from servers ◦ Event log entries◦ Registry settings◦ Customized reporting
Use data to clarify business requirements◦ Roles◦ Communication requirements◦ Documentation
More quantifiable data than initial business case
Established expected baselines Resourcing and timelines Calculated revised Return on Investment Defined a process Verified business requirements
1. Map regulatory/oversight requirements to internal controls
2. Inventory and leverage existing data sources 3. Use existing, free, or low cost tools4. Analyze Baseline◦ Data Flow◦ Data Integrity◦ Return on Investment
5. Re-baseline and productionize (governance)◦ Automation◦ Workflow
Don’t overlook unstructured data Unlock your small data◦ Gather and update effectively◦ Focus on context and criticality
Audit can be great sources of small data, but know the audit approach
Leverage the same data sources for different risks and insights
Data-Driven Security: Analysis, Visualization and Dashboards by Jay Jacobs and Bob Rudis (book)
Threat Modeling: Designing for security by Adam Shostack (book)
Database Debunkings Fabian Pascal (blog) Dresner Advisory Services 2016 End User
Preparation Market Study (Market Research) Storytelling with Data by Cole Nussbaumer
Knaflic (book and blog)