Systems of Intelligence:The Biggest Change in Enterprise Applications in 50 Years

George Gilbert@GGilbert41

Big Data Analyst

StoreE-Mail

SocialMedia IM

Where Customers Should be InvestingAnalogous to Pipelines in Systems of Record

Operational apps

Customer interactions

Agility: Speed of improving reportsweeks, months, quarters

Latency: Speed of reportingDays, weeks, months

ETL Development

OperationalData

Customer“Breadcrumbs”

Production ETL

StoreE-Mail

SocialMedia IM

Where Customers Should be InvestingSystems of Intelligence Pipelines are all About Speed and Agility

Retail

ConsumerMobile

Call Center

eCommerce

Operational apps

Customer interactions

Agility: Improving predictions seconds to days

Latency: Speed of predictionsms to seconds

Machine learning pipeline

OperationalData

Customer“Breadcrumbs”

Predictions,Recommendations

ImprovingPredictions(Machine Learning)

Intelligence

Critical new data and analytic skills are required• Accuracy of predictions (Revenue)

Modernizing SoR can accelerate the journey• Speed of predictions (latency)• Speed of improving predictions (agility)

Choice of new platform• TCO/operational complexity• Development complexity• Existing infrastructure – technology and

skills

What Trade-Offs Should Customers Consider When Deploying Systems of Intelligence

Incremental Revenue

Planning Your Customer Journey: Skills and Platform Progress

Smart Grid

Fraud prevention

Real-time loyaltyomni-channelmulti-touchpoint

Predictive model learns from and anticipates consumer in near real-time

Continuously updated prediction of energy supply, demand tunes end-point consumptionIntelligent

systems management

System learns “normal” behavior of apps and infrastructure and flags or fixes anomalies

Identify spending behavior out of the norm

ApplicationsApplications

Technology Maturity, Enterprise Capabilites

Technology Maturity, Enterprise CapabilitesTime

Big Data Platform Evolution – Simplifying Operations and Development:Storage Consolidation, API’s > OSS Products, Spark Hollowing out Hadoop

Machine Learning

SQL: Join, filter, aggregate

Streaming

HDFS-Compatible File System

Resource + Workload Manager

HBase-Compatible DB

Polyglot Data API: SQL, KV, JSON

YARN Resource Manager

HDFS, HBase

StreamingSpark, Flume, Flink, SamzaDataflow

Kafka-Like Messaging

SQLImpala, Drill, Hive, HAWQ…

Machine LearningMahout,

Spark MLlib

Hadoop 2.0 Big Data 3.0

In-Memory Storage (DB or File System)

Dev Tools Dev Tools Dev Tools

Dev Tool

Inte

grat

ed D

evop

s

Man

agem

ent,

Gove

rnan

ce

Operational Simplicity vs. Complexity:Single Pane of Glass vs. Familiar Legacy Sprawl

Development Simplicity vs. Complexity: Spreadsheet vs. Many Development Tools

Many data managers – optimal functionality(Cassandra, Aerospike, MongoDB, Neo4j…)

Single vendor data platform(Azure, AWS, Google Cloud Platform, Bluemix, Pivotal)

Single multi-purpose engine(Oracle, Spark)

Customers Must Balance Operational Simplicity and Development SimplicityRelative to Existing Skills and Infrastructure

Ope

ratio

nal

Sim

plic

ity

Development Simplicity

Hadoop ecosystem(Hortonworks, Cloudera, MapR)

SaaS: AirBnB, Uber, Commerce

Fortune 500

Mainstream

Pro: Optimal functionalityCon: ComplexityCustomer sweet spot• Leading-edge Internet-

centric companies• Netflix, Uber, ad-tech,

gaming, ecommerce

Many Data Managers – Optimal Functionality: Highest Development and Operational Complexity

Many optimized data managers(Cassandra, Aerospike, MongoDB, Neo4j…)

Single vendor data platform(Azure, AWS, Google Cloud Platform, Bluemix, Pivotal)

Single multi-purpose engine(Oracle, Spark)

Ope

ratio

nal

Sim

plic

ity

Development Simplicity

Hadoop ecosystem(Hortonworks, Cloudera, MapR)

Hadoop 2.0

Big Data 3.0

Customers Must Balance Operational Simplicity and Development SimplicityRelative to Existing Skills and Infrastructure

Critical new data and analytic skills are required• Accuracy of predictions

Modernizing SoR can accelerate the journey• Speed of predictions (latency)• Speed of improving predictions (agility)

Choice of new platform• TCO/operational complexity• Development complexity• Existing infrastructure – technology and

skills

Systems of Intelligence P&L Statement: Designing with “Budgetary” Constraints

Technology maturity:• lowers cost • opens new application possibilities

Incremental Revenue