mark zangari, ceo, quantellia at mlconf sea - 5/01/15
TRANSCRIPT
Agency
Agency’s aims:
1. Extend using machine learning into a new domain of problems.
2. The beginnings of a formal technique for linking data science to business levers.
3. Instead of answering “Given input data A, what output data B does my model predict?”, Agency answers “If I do A to complex dynamical system S, with the intention of achieving objective B, what is the probability that S’s outputs are closer to B?”
Agency
The four pillars for effective machine learning:
1. Availability of training data
2. System stability over characterstic times(a) training data period (past)(b) prediction period (future)
3. Availability of input data at the time the prediction is to be made.
4. Sufficient signal-to-noise in data for pattern recognition to be possible.
Agency
The four pillars for effective machine learning:
What can we do if one or more of the four pillars are not present?
Agency
The four pillars for effective machine learning:
What can we do if one or more of the four pillars are not present?
Example from finance:Commercial lending vs. consumer lending.
Agency
Consumer Credit:1. Ample, accessible data2. Large, consistent, classifiable population 3. Relevant variables easily measured4. Strong correlations between input variables and credit risk.
Consumer
Credit Transactions
Data Warehouse
Rating System Credit Score
Bill Fair & Earl Isaac
Contacted 58 of the nation’s top lending institutions in 1958 offering to show them how using data would help them make better credit decisions...
Only one responded.
Image composed from stock images and portraits of Fair and Isaac at http://www.fico.com/en/about-us#our_history
Agency
Consumer Credit:1. Ample, accessible data2. Large, consistent, classifiable population 3. Relevant variables easily measured4. Strong correlations between input variables and credit risk.
Commercial Credit for Small/Medium Businesses:1. Little data available, hard to obtain2. Each business different, many are very new with little history.3. Complex inter-entity relationships affect credit risk.4. Many correlations , hard to isolate those that are good predictors
of credit risk.
Agency
Fully automated credit rating is rarely used for scoring Small/Medium Business Loans.
The process commonly used is a good example of Agency in action…
Agency
Type of Business
Loan Amount Probability of Default
Typical Model Decomposition for Commercial Loan Decision
Grant this Loan?
Profitable Loan Book
Desired ObjectiveBusiness Lever
ExternalInputs
Agency
Agency Principle One:
Learning and other analytics are designed to discriminate actions that will increase the probability of the objectives being met, from actions that do not.
So…
What is an “objective?”
Agency
Objective:
Map every element in the set of outcomes to a measure of the favorability of that outcome occurring, relative to the other outcomes.
Probability of default
Des
ired
Lo
an F
req
uen
cy
Agency
Objective:
Map every element in the set of outcomes to a measure of the favorability of that outcome occurring, relative to the other outcomes.
Probability of default
Des
ired
Lo
an F
req
uen
cy
Agency
Type of Business
Loan Amount Probability of Default
Typical Model Decomposition for Commercial Loan Decision
Grant this Loan?
Profitable Loan Book
Desired ObjectiveBusiness Levers
ExternalInputs
Measure Measure Learn
Interest Rate
Agency
Agency Principle Two:
If a system does not ideally support machine learning because it does not satisfy the four pillars, decompose it using a dynamic system model until each link between each node either:(a) Satisfies the four pillars, or(b) Can be described using known domain-specific
relationships (formulas, rules, approximations, etc.)
Agency
Financial Spreading
Values
Market & Industry Data
Management Team Bios & Experience
Relationships to other Entities
Key Ratios
Aggregate Statistics
Scoring
Probability of Default
Known mathematical relationships
Machine Learning(intangibles)
Rating
Machine Learning
Transform Tables
Typical Model Decomposition for Business Credit Rating
Feature Engineering
Agency
Management Team
Experience
Agency gives us insight into decisions via the model levers.
E.g. What interest rate do we charge and how does thisaffect our success?
Interest Rate 1
Management Team
ExperienceInterest Rate 2
Interest Rate 1 Interest Rate 2
Agency
Summary of characteristics where Agency becomes highly relevant and useful:
• Sparse data
• Complex emergent dynamics, including feedback loops, phase changes, and other non-linear effects.
• Provides a way of including intangibles with few measureables in the model
• Can utilize relationships that the training data does not make apparent.
Agency
A (brief) mathematical representation of Agency:
We draw analogy from the information measure:
The “Agency” A of a lever L, which maps some part of the output space to favorable outcomes, f and some other part to unfavorable outcomes, with measure M(f) and M(u) is given by:
Agency
Agency:
• Helps business users connect real-world decisionswith support from data.
• Provides a architecture for using machine learning in situations that are not typically well suited to ML solutions.
• Allows ML results to be extrapolated beyond information contained in the data, by integrating knowledge of system dynamics.