10 measures and KPIs for ML successBuilding the business case for MLOps and management

algorithmia.com | @algorithmia

We’ll get started in just a moment. While we wait for everyone to join, enjoy getting to know our speaker better as we ask him questions about his youth 😊

Send over your questions! 🎉

Diego Oppenheimer, CEO, Algorithmia

Diego Oppenheimer is co-founder and CEO of Algorithmia. Previously, he designed, managed, and shipped some of Microsoft’s most used data analysis products including Excel, Power Pivot, SQL Server, and Power BI. He holds a Bachelor’s degree in Information Systems and a Master’s degree in Business Intelligence and Data Analytics from Carnegie Mellon University.@doppenhe | linkedin.com/in/diego

Speaker

Survey questions while we wait for people to join…

1. How many ML models do you have in production?

a) None yetb) 1-5c) 5-25d) 25+e) Do not know

2. What’s the typical time it takes you to deploy a new ML model?

a) 0-7 daysb) 8-30 daysc) 31-90 daysd) 91-365 dayse) 1+ yearsf) Do not know

3. What is your infrastructure cost of operations for all your models in production?

a) Under $50kb) $50k-$250kc) $250k-$500kd) $500k or moree) Do not know

2020 report findings

From the 2020 state of enterprise ML report:

● 55% of companies developing machine learning have never deployed a model.

● 22% of companies spend between 31 and 90 days deploying a model.

● 25% of data scientist time is spent on deployment.

algorithmia.com/state-of-ml

The machine learning lifecycle

Data, network, model securityInfosec complianceRegulatory compliance

Customerapplications

ValidationVerificationPreventive controls

Measures and KPIs

Leverage – Measure impact

● Shorten time to market

● Reduce IT ops overhead

● Rapid iteration and model optimization

● Model reusability

● Infrastructure scalability and cost

Risk – KPIs to measure value

● Time to deployment

● Models in production

● New models deployed each month

● API calls

● Chargebacks and showbacks

1. Shorten time to market

● Automate deployment paths

● Impact of model in production

● Monitoring and reporting 2020 State of Enterprise Machine Learning Survey

31+ days Average time to deploy for 59% of organizations.

1. Shorten time to marketChallenge: A customer wanted to use ML for fraud detection and compliance, but was taking up to 18 months to deploy a single model.

Criminal patterns and behavior would change during that time, making any models obsolete at deployment.

Solution:● Algorithmia streamlined the developer

workflow for machine learning.

● Enabled modular pipeline for model reuse, allowing quick editing and deployment into client applications.

● Deployed 100+ models with 10 different open-source frameworks and more than 75 libraries in 5 months.

Top 4 consulting firm

Platform approach to ML ops provides KPIs:

● Time to deploy

● Business result

● Performance

2. Reduce IT Ops overhead

Model registration

● Self-service

● Documentation

● Model operations

2x growthNumber of AI projects doubling in the next year by 2021Gartner predicts the future of AI technologies, Feb 2020

Gartner predicts the future of AI technologies, Feb 2020

2x Number of new AI projects by 2021.

2. Reduce IT Ops overhead

Operational KPIs:

● Models in production

● Call and errors

● Hardware utilization

Challenge: In experimenting with computer vision and ML for rapid prototyping, Toyota’s insurance group, MS&AD, found that deploying each model as a “one-off” was time-consuming and resource-intensive.

Solution:● Toyota increased models in production

by 75% with a consolidated ML architecture running on Algorithmia.

● The Algorithmia platform provided a central space to serve the company’s ML innovation needs and made collaboration across teams easier.

3. Rapid iteration and model operation

● Rapid iteration

● Freedom to choose

● Pipelining80% will face production delaysFor enterprises moving ML to production, due to lack of collaboration and IT process immaturity.

“Accelerate your ML & AI Journey” Gartner, 2019

v.1.0.0

v.2.0.0

CONSUMING APPS:

API

“Accelerate your ML & AI Journey” Gartner, 2019

80%

Of enterprises will face production delays due to lack of collaboration and IT process immaturity.

Production model KPIs:

● New models deployed each month

● Models in production

● Model versioning

3. Rapid iteration and model operation

Challenge: HappyMoney needed to integrate user data and apply ML models in real time to provide advice on how to pay down personal debt efficiently.

Solution:● Algorithmia integrated real-time

and batch data from financial services firms to run the models and generate savings recommendations.

● Algorithmia chained versions of HappyMoney’s cash-flow model projections using different frameworks from real-time bank transaction data and risk analysis from different data science tools.

4. Model reusability

● Reusability

● Future-proofing

● Model maintenance

2020 State of Enterprise Machine Learning Survey

Top 3 use casesReduce company costsProcess automation for internal organizationImprove customer experience

4. Model reusability

Model activity KPIs:

● API calls

● Avg call duration

● Models in production

Challenge: Wanted to deliver customer engagements without long consulting engagements, software purchases, sourcing external data, or repeatedly building out applications.

Solution:● They added Algorithmia into the

workflow as a repository for reusable models and a model deployment and serving engine.

● Their customers can now integrate data connectors and pipeline models into one end-to-end analytics process using proprietary industry algorithms.

Logistics Consulting company

Traditional capacity planning

● Wasteful and expensive

● Hard capacity limits

Local maximum planning

● Still inefficient

● Increased management overhead

Serverless (Algorithmia)

● Massive efficiency boost

● Substantially lower cost

● Infinite scalability

5. Infrastructure cost and scalability

5x increaseBy 2023, cloud-based AI will increase 5x, making AI one of the top cloud services.Gartner predicts the future of AI technologies, Feb 2020

Gartner predicts the future of AI technologies, Feb 2020

5xThe amount cloud-based AI will increase by 2023. Making AI one of the top cloud services.

5. Infrastructure cost and scalability

Infrastructure consumption KPIs:

● Chargebacks and showbacks reports

● Auditability: Who called what model,

at what time, with which data

Measures and KPIs

Leverage – Measure impact

● Shorten time to market

● Reduce IT ops overhead

● Rapid iteration and model optimization

● Model reusability

● Infrastructure scalability and cost

Risk – KPIs to measure value

● Time to deployment

● Models in production

● New models deployed each month

● API calls

● Chargebacks and showbacks

Algorithmia Enterprise

Model deploymentConnect, load, catalog, version, and validate models for production

Model operationsManage costs, control infrastructure usage, monitor operations, deliver models and services at high velocity

Governance and securityAdvanced security enabling rule-based control over users, data, infrastructure and models

Building versus buying:how to achieve ML operations

Join us on 9 June for our next webinar that will help you evaluate a build vs buy decision on investing in your ML operations and management capacity.

Webinar

Register today: info.algorithmia.com/build-vs-buy-webinar

Kenny DanielCTO Algorithmia

Sam CharringtonTWIML AI Podcast Host

Q&A